Migrate all your messages to Gmail or other mail account using imapsync

This article covers:

• all the information, that I managed to find out or figure out,
• all the bottlenecks I had to resolve and all the walls, I hit,
• some additional info or alternative methods,

about migrating all messages from some mail account to Gmail, using imapsync and IMAP protocol.

The imapsync is an extremely flexible and powerful tool, that also has an on-line counterpart (so you don’t get your hands too dirty), and its documentation simply surpasses imagination, so it is certainly able to transfer all the messages between virtually any IMAP servers, not just between Gmail.

But this article focuses on external IMAP server –> Gmail scenario. For other scenarios, you should find some tips here, but you may need to consult imapsync documentation and the Internet or to use another solution.

Before you start

I have decided to use imapsync tool and I encourage you to use it, even though this is a paid tool, because:

• it supports many different operating systems and 75+ different mail servers, accounts or clients,
• it is extremely flexible and can cover a lot of different scenarios or migration processes,
• it has both off-line and on-line versions,
• documentation is simply amazing,
• author is very helpful.

The imapsync, as its name suggests, works in a full synchronisation mode. Meaning that (as long as your source mail server produces a valid message signatures you can use this over and over again or run your migration process in batches and still don’t get any duplicates (again… as long as…).

This is especially important since migrating most of the accounts will last days, not hours, mostly due to bandwith limits on your IMAP servers and due to your Internet or PC performance.

Preliminary assumptions for this article (some for my case only):

• source is a custom IMAP account and destination is Gmail account,
• source account was used by 3+ past years, producing 2.9 GB of data and 26 830 messages in total,
• messages are encrypted in the source account and thus may be (most likely will be) not readable after migration (*),
• source was accessed for past 3+ years by Microsoft Outlook 2013, which could mess a little bit (imapsync should handle this),
• Gmail bandwidth limits can cause whole process to last long hours or even days (!).

(*) Program can decipher messages on-the-fly, but this process requires a lot of knowledge and time and is not covered here; please, contact imapsync author for a possible support in this case.

When I run imapsync for the first time (actual messages sync, after a number of dry runs), it reported me:

• average speed of 0.77 messages per second,
• average speed of 79.850 KiB/s (on actual 14.47 Mbps network — due to Gmail limts),
• ETA: 10.5 hour (34 599 s)

on that 2.9 GB of data / 26 830 messages source account.

You may use above numbers to try to estimate parameters for the sync process in your case.

The price

This article focuses on imapsync (though some untested alternatives are given in the end), which is a paid tool and in general you have to buy the off-line version or donate in order to use the on-line version.

Prices for imapsync are:

• 60 EUR lifetime for off-line version,
• 59 EUR for using on-line version to migrate many accounts,
• 6 EUR for single migration process using on-line version.

But, before you start shaking yourself that paying 60 EUR for "probably one-time used tool" is madness, keep in mind that in many scenarios you will not pay anything.

This is, because:

• imapsync ships with "do anything" license, so you can buy it and then… sell it to someone else,
• Gilles (imapsync author) refunds money for any reason, "30 days money back guarantee",
• Gilles offers a copy for just an nice email sent to him (I’ve got my copy this way!),
• you can offer a smaller donation / price, if 60 EUR is too much for you,
• imapsync is free on GitHub, you only have to build it for yourself,
• the on-line version works just fine even without donation.

So, you must admit that there are quite a lot of options for using imapsync completely free of charge, right?

But, there’s a downside (as always!). Gilles invested a nineteen years of his life into imapsync development and support and even though he still finds that funny, too many people using imapsync too often free of charge may eventually cause its development to cease. The on-line version eats server resources, someone must pay for. And the off-line version’s development eats hours of development, someone must get profit for, right?

First steps

Before you begin, enable IMAP protocol in Gmail (it is disabled by default):

• on your computer, open Gmail,
• click the "gear" icon in the top right of the screen,
• pick Settings item from menu,
• go to the Forwarding and POP/IMAP tab,
• in the IMAP access section, select Enable IMAP option,
• confirm changes by clicking Save Changes.

Also make yourself 100% sure that Windows settings won’t stop your sync process when you’re off:

Some resources to consult before your start:

• imapsync main information,
• Windows FAQ
• possible Gmail issues,
• handling message duplicates,
• Gmail and G suite bandwidth limits.

Off-line version of the imapsync tool is a command-line utility that you run directly or through .bat files.

I won’t discuss here matters of downloading and unpacking of the software or basic knowledge about how .bat files work or what to take care off. All these issues are clarified in an online documentation. I assume that you are able either to execute all presented commands by hand or change .bat file and execute it.

Dry run

As you may expect all we actually need is:

• source IMAP server’s URL, login and password and
• destination IMAP server’s URL, login and password.

And the command is:

imapsync.exe --host1 test1.lamiral.info --user1 test1 --password1 "secret1" ^
             --host2 test2.lamiral.info --user2 test2 --password2 "secret2" ^
             --justfolders ^
             --automap ^
             --dry

Switches used here:

• host1, user1, password1, host2, user2, password2 — obvious,
• dry — forces imapsync to do nothing, just print what would be done without this switch,
• justfolders — migrates folders only and ignore messages, so you can check folder mapping etc.
• automap tries to guess folders mapping for folders like "Sent", "Junk" etc., which may use i.e. localised names on one of servers (i.e. "Sent" = "Enviada", "Gesendet", "Wysłane", "Gönderildi", "Inviati", "Skickat", "Verzonden" etc.).

These are actual values and working test servers (no fake), provided by imapsync author, so as long as you have everything working and configured on your end of the Internet, executing the above command should actually end with a success.

Put this command to a .bat file or directly into command line to test local PC and network connection.

If everything is OK then you can replace above values with your one, dry run again and then:

• verify, if the folder mapping is correct,
• remove --dry and have a run to create folders on your destination,
• if everything goes well so far then remove --justfolders to start syncing messages.

Example dump from command line about folder mapping:

Folders mapping from --automap feature (use --f1f2 to override any mapping):
[Wysłane]           -> [[Gmail]/Sent Mail]
[Wiadomości-śmieci] -> [[Gmail]/Spam]
[Szablony]          -> [[Gmail]/Drafts]
[Usunięte]          -> [[Gmail]/Bin]

If folder mappings is incorrect, add a serie of --f1f2 folder1=folder2 switches, as instructed above.

Less secure app access

Now, let’s try using the same with Gmail as the destination:

imapsync.exe --host1 imap.sourceserver.com --user1 one.zero.null --password1 "someW31RDpass#" ^
             --host2 imap.gmail.com --user2 account@gmail.com --password2 "someW31RDpass#" ^
             --automap ^
             --justfolders ^
             --dry

On contrary to above, these are truly fake, so don’t even bother trying them! :]

First thing that you will most likely hit will be "Less secure app access", disabled by default in Gmail, and a nice warning message landing in your mail account:

Get rid of this little problem:

• go to your Google Account,
• on the left navigation panel, click Security,
• scroll down to the bottom of the page,
• locate the Less secure app access panel,
• click Turn on access,
• confirm everything what requires confirmation.

Expect some nasty warnings (e-mail messages, notifications on connected mobile devices etc.), because Google considers enabling this functionality as a serious security risk to your account and data. For this reason, strictly disable this feature again, once your message migration process is completed.

Once this limitation is lifted, imapsync should be able to access your Gmail account without any problems and executing a command similar to above should actually result in a success.

I can confirm that (as of writing this — March 2019) with Less secure app access option enabled, default imapsync configuration (i.e. port, using SSL etc.) are correct and the tool is able to connect Gmail account without any additional work needed.

If, on contrary, you’re still not able to get access to your destination Gmail account then:

• this article in Google Help and
• this FAQ section in imapsync‘s documentation

might be a good starting points.

Next steps?

As instructed above, you can now (step by step):

• verify folder mapping and
• remove --dry to actually create folders in Gmail.

Last step is to remove --justfolders to start actual messages syncing process.

But, since we’are talking about Gmail as the destination, before you go that far first, please, review following chapter. This is necessary, because Gmail is very restrictive in both bandwidth limits and some special characters in messages’ titles or content. And you need a serie of imapsync switches to handle all these issues.

If your destination is not Gmail, you may safely skip the next part and try the actual sync process right now.

Actual sync process

Here there are:

imapsync.exe    --host1 imap.sourceserver.com ^
                --user1 one.zero.null ^
                --password1 someW31RDpass# ^
                --host2 imap.gmail.com ^
                --user2 account@gmail.com ^
                --password2 someW31RDpass# ^
                --maxbytespersecond 20_000 ^
                --maxbytesafter 1_000_000_000 ^
                --maxsleep 2 ^
                --maxsize 35_651_584 ^
                --automap ^
                --expunge1 ^
                --addheader ^
                --exclude "\[Gmail\]$" ^
                --regextrans2 "s/[ ]+/_/g" ^
                --regextrans2 "s/['\^\"\\\\]/_/g"

Switches used here (details):

• host1, user1, password1, host2, user2, password2 — obvious,
• maxbytespersecond — avoid exceeding Gmail maximum data transfer speed limits,
• maxbytesafter — tests has proven that in most cases transfer speed limits are applied after 1 GB,
• maxsleep — avoid too long pause caused by using maxbytespersecond,
• maxsize — skip messages bigger than 25 MB due to Gmail message size limits,
• automap — try to guess folders mapping for folders like "Sent", "Junk" etc.
• expunge1 — deletes messages marked as deleted by the source host,
• addheader — add missing "Message-Id:" and "Received:" headers for messages in "Sent" folder,
• exclude — avoid a weird error in Gmail (details) that may sometimes occur,
• regextrans2 — convert (illegal in Gmail) blank characters and ^ character to underscore (_).

Notice, that justfolders and dry switches are gone and executing this command will trigger a complete sync process.

If you’d like to use them, then keep in mind that they must appear before regextrans2. This is because Windows has some problems executing command that includes switches without quotation mark after switches with quotation marks. Thus, you have to include regextrans2-like switches (having quotation marks) as the last one.

The imapsync‘s FAQ section about Gmail says that Gmail has problems with space (blank) characters in folder names and that is why you must include --regextrans2 "s/[ ]+/_/g" ^ switch as well.

My tests proven that this is not true. You can go without this switch and Gmail should handle folder names with spaces as well.

Breaking and continuing

The imapsync, both versions, fully supports breaking and restarting sync process at any moment.

For the reasons explained here (Power Options not changed), my computer was turned off after half hour since I left the office. Instead of seeing process completed, when I came back next morning, I saw that it was interrupted after transferring only 30% of the messages.

Fortunately, imapsync handled sync process gracefully and picked up its job, where it left it.

There’s one, quite very important thing you need to remember when such thing happens to you. The imapsync is run again and has its limits "reset" to default (actually: to what you told it in run parameters), bo your Gmail’s or other IMAP host’s bandwidth limits are most likely not lifted of.

I.e. If you restart imapsync the very same day, it was stopped, you’re still eligible for daily bandwidth limits. This means that you may consider lowering maxbytesafter value to some minimal or increasing maxsleep value to i.e. 5 seconds. A matter of tries.

The imapsync can give you some idea here. With every copied message you see something like this:

msg [0] Atlassian.Confluence/2788 {34222} copied to [0]_Atlassian/Confluence/2781 0.81 msgs/s
155.371 KiB/s 1.036 GiB copied ETA: Wed Mar  6 15:50:45 2019  16610 s  13487/26838 msgs left

The middle value here ("155.371 KiB/s 1.036 GiB") tells you that your entire process will use 1 GB of data.

So, if you:

• are starting a first sync process this day (or to be more precise: during last 24 hours) and,
• compare this value to Gmail and G suite bandwidth limits

then you may learn that you are exactly at 200% of your daily bandwidth limit ("POP and IMAP bandwidth limits — Upload with IMAP — 500 MB"; the first one says about uploading via web client and this doesn’t count for imapsync). And thus, using some additional command-line switches when launching imapsync may be necessary.

But, if you would be, i.e. at half of that limit, you could start imapsync without any limit-related switches.

The value of estimated data to transfer is given with every synced message, so you’re actually risking nothing. You can start sync process without any limits only to see calculated data to be transferred, and if you learn that it exceeds your daily quota, defined either at source or destination host, you can break imapsync at any time, i.e. immediately (to restart it with some limit-related switches) or near exhausting your daily quota (and continue next day).

Summary or the Wall of Pain

Things that may fail or things to look for:

1. Gmail has IMAP access disabled by default. Enable it (details).
2. Gmail blocks "less secure" apps from accessing your account. imapsync is one of such. Enable this access (details).
3. Your network configuration may limit usage of IMAP protocol for certain programs only (i.e. Microsoft Outlook) and imapsync will fail on connecting to any server. Change network configuration or change network itself to resolve this.
4. Due to hosts or network limits the entire process may last hours or even days. I don’t think you can fix this (details).

I must admit that I was quite scared to do this and the whole idea of pushing 28k messages to Gmail was terrifying me. Turned out, this was completely not necessary. The imapsync did all the dirty work for me.

The cleanup process

Things to consider after your mail sync process is completed:

1. Disable "less secure apps access" in Gmail (details).
2. Bring back Windows’ Power Options to its original state (some details).

Of course, the most important part is to actually verify that all messages was transferred correctly.

Alternatives

As you may get from the whole article, I am actually totally in love with imapsync.

But, since you may find it too expensive or too complex (it is still a command-line tool with a lot of switches and a documentation written as it would be written by a true Linux geek! :]), here are some alternatives:

• install Mozilla Thunderbird, add both source and destination IMAP accounts and use its "Copy To" feature,
• use Mail.app, if you wish to do this on-line with no software installed (details),
• use direct Outlook .ost file import to Gmail (39 USD) — here and here.

These solutions were not tested by me. If none of above solution is for you, then ask the Internet.