Revision as of 15:00, 1 September 2015

Overview

This can be the most robust method of migration, though it's only limited to mail. IMAPSync will compare source and destination hosts, and copy any messages not on the destination server from the host server. You can run this multiple times and generally not result in any duplicate messages in the destination mailbox.

It is a command line-based perl script, so it requires either some sort of unix-based system or cygwin (or other compatible Perl setup on Windows). Consequently, it does require some level of technical proficiency to be used.

Previously, this was normal free OSS software, but the author has since started requiring payment for it. Yet, not only was it previously released under an open source license, but it continues to be licensed as OSS with the newer paid versions, so you can still get reasonably up-to-date versions legally. If you wish to purchase the software, you can get a windows executable. All others will want to install from the git repository.

• IMAPSync Homepage: http://imapsync.lamiral.info/
• Open source repository: https://github.com/imapsync/imapsync

You will want to read the documentation and determine what commands are appropriate to your situation, but the general idea is that you provide the source host/user/password, the destination host/user/password, imapsync logs into both and transfers the mail. These are delineated as host1/user1/password1 and host2/user2/password2 in the command line options.

Here is an example IMAPSync command:

imapsync --host1 mail.some_mail_server.com \
--user1 some_username \
--password1 some_password \
--host2 zimbra.xmission.com \
--user2 account_on_zimbra@domain.com \
--password2 zimbra_password \
--syncinternaldates --subscribe

IMAP Admin Account Logins

Also note that many mail servers (including our Zimbra servers) support the ability for an admin account to auth as a user. This is very useful when migrating an entire domain, as it frees you from needing to know each individual user's password. IMAPSync supports this with the "authuser" command. The same example command, except using the admin account, would go something like this:

imapsync --host1 mail.some_mail_server.com \
--user1 some_username \
--authuser1 admin_account_name \
--password1 admin_account_password \
--host2 zimbra.xmission.com \
--user2 account_on_zimbra@domain.com \
--authuser2 admin_account_on_zimbra@domain.com\
--password2 admin_account_zimbra_password \
--syncinternaldates --subscribe

Google Apps Admin Account Logins

Google Apps currently supports OAuth v1 and v2 to admin authenticate to other accounts, however OAuth v1 was deprecated on April 15th, 2015 and may stop working at any time. You can use either of these to have something like a "master password" for accessing Google Apps accounts without the administrator needing to know each individual user's password. OAuth v1 is a little easier to use, and is recommended while it is still available.

OAuth v1

To use OAuth1 with imapsync, either from or to Google Apps, you will need to generate an OAuth consumer key, and then use this key as the password while setting the "authmech" imapsync flag to "xoauth". Also note that gmail requires ssl over imap.

To generate the OAuth1 key, in Google Apps, go to the admin console and choose Security -> Advanced Settings -> Manage OAuth Domain Key. Click "Enable this consumer key" if it's not already enabled. The string of random characters under "OAuth Consumer Secret" is the "master password". Make sure "Two-Legged OAuth Access Control" is checked.

To use this key, you want the password to be the master key, and the authmech to be xoauth. If your master key was "blork", your imapsync command might look like this:

imapsync --host1 imap.gmail.com -ssl1 \
--user1 user@domain.com \
--password1 blork \
--authmech1 xoauth \
--host2 zimbra.xmission.com \
--user2 account_on_zimbra@domain.com \
--authuser2 admin_account@domain.com \
--password2 admin_account_password
--syncinternaldates --subscribe

Note, once again, that OAuth1 is deprecated and support may be removed at Google's discretion.

OAUTH v2

The newest version of IMAPSync has support for OAUTH v2. The process is a little more involved.

Provisioning OAUTH2 in Google Apps

1.) Setup a "service account" in the google developer api web site 2.) Download an encrypted key file and note the service account's email address form 3.) Associate the service account with your google apps domain

In the following link, "Google APIs Console" and "Google Apps Dashboard" sections illustrate setting it up:

http://www.limilabs.com/blog/oauth2-gmail-imap-service-account

Using OAUTH2 in IMAPSync

1.) Set the authmech for the user to "xoauth2" 2.) Make sure the key file google created for your service account is present on the local filesystem 3.) Provide the service account address, encrypted keyfile location and optionally the keyfile password (google's default password will be assumed, so you probably never need to do this) in the following format: <service account>;<keyfile location>;<keyfile password>

An example of the arguments for my google apps domain: --authmech2 xoauth2 --password2 '587056876087-mjdh0aiajc0ru7gpb7434si8nlitrbuo@developer.gserviceaccount.com;notasecret.p12'

Example session (using --justlogin, AUTHENTICATED is the confirmation of successful login):

[server :11:36PM] imapsync>./imapsync --justlogin --host1 imap.gmail.com --ssl1 --authmech1 xoauth2 --user1 testuser@domain.com --password1 '587056876087-mjdh0aiajc0ru7gpb7434si8nlitrbuo@developer.gserviceaccount.com;notasecret.p12' --host2 zimbra.xmission.com --user2 testuser@domain.com --authuser2 admin_account@domain.com --password2 somepassword
Transfer started at Fri Jan 30 23:37:05 2015
PID is 31730
Log file is LOG_imapsync/2015_01_30_23_37_05_testuser@notearthday.org.txt
$RCSfile: imapsync,v $ $Revision: 1.607 $ $Date: 2014/11/14 16:25:06 $
Here is a [linux] system (Linux server 3.2.0-58-generic #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013 x86_64)
With perl 5.14.2 Mail::IMAPClient  3.30
Command line used:
./imapsync --justlogin --host1 imap.gmail.com --ssl1 --authmech1 xoauth2 --user1 testuser@domain.com --password1 MASKED --host2 zimbra.xmission.com --user2 testuser@domain.com --authuser2 admin_account@domain.com --password2 MASKED
Temp directory is /tmp
PID file is /tmp/imapsync.pid
Modules version list:
Mail::IMAPClient     3.30
IO::Socket           1.32
IO::Socket::IP       ?
IO::Socket::INET     1.31
IO::Socket::SSL      1.53
Net::SSLeay          1.42
Digest::MD5          2.51
Digest::HMAC_MD5     1.01
Digest::HMAC_SHA1    1.03
Term::ReadKey        2.30
Authen::NTLM         ?
File::Spec           3.33
Time::HiRes          1.972101
URI::Escape          3.31
Data::Uniqid         0.12
Unicode::String      2.09

Info: turned ON syncinternaldates, will set the internal dates (arrival dates) on host2 same as host1.
Info: will try to use LOGIN authentication on host1
Info: will try to use XOAUTH2 authentication on host2
Info: imap connexions timeout is 120 seconds
Host1: IMAP server [imap.gmail.com] port [993] user [testuser@domain.com]
Host2: IMAP server [mail.xmission.com] port [143] user [testuser@domain.com]
Host1: * OK Gimap ready for requests from 199.104.120.118 qu16mb267177216pab
Host1: imap.gmail.com says it has CAPABILITY for AUTHENTICATE XOAUTH2
Host1: success login on [imap.gmail.com] with user [testuser@notearthday.org] auth [XOAUTH2]
Host2: * OK IMAP4 ready
Host2: zimbra.xmission.com says it has NO CAPABILITY for AUTHENTICATE LOGIN
Host2: success login on [zimbra.xmission.com] with user [testuser@domain.com] auth [LOGIN]
Host1: state Authenticated
Host2: state Authenticated
Host1 capability: IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT
Host2 capability: IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SORT SORT=MODSEQ SORT=DISPLAY THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE LIST-EXTENDED WITHIN QRESYNC SCAN XLIST URLAUTH URLAUTH=BINARY LOGINDISABLED COMPRESS=DEFLATE IDLE