Zimbra Migration: IMAPSync
- Data Type(s) Migrated: Mail
- Recommended for: XMission Zimbra customers with xmbase mailboxes, or customers migrating from non-Exchange mail servers
- Some technical proficiency required (command line-based, requires Linux, Mac OS X or cygwin)
This can be the most robust method of migrating, though it's only limited to mail. IMAPSync will compare source and destination hosts, and copy any messages not on the destination server from the host server. You can run this multiple times and generally not result in any duplicate messages in the destination mailbox.
It is a command line-based perl script, so it requires either some sort of unix-based system or cygwin (or other compatible Perl setup on Windows). Consequently, it does require some level of technical proficiency to be used.
Previously, this was normal free OSS software, but the author has since started requiring payment for it. Yet, not only was it previously released under an open source license, but it continues to be licensed as OSS with the newer paid versions, so you can still get reasonably up-to-date versions legally. If you wish to purchase the software, you can get a windows executable. All others will want to install from the git repository.
- IMAPSync Homepage: http://imapsync.lamiral.info/
- Open source repository: https://github.com/imapsync/imapsync
You will want to read the documentation and determine what commands are appropriate to your situation, but the general idea is that you provide the source host/user/password, the destination host/user/password, imapsync logs into both and transfers the mail. These are delineated as host1/user1/password1 and host2/user2/password2 in the command line options.
Here is an example IMAPSync command:
imapsync --host1 mail.some_mail_server.com \ --user1 some_username \ --password1 some_password \ --host2 zimbra.xmission.com \ --user2 email@example.com \ --password2 zimbra_password \ --syncinternaldates --subscribe
IMAP Admin Account Logins
Also note that many mail servers (including XMission's Zimbra servers) support the ability for an admin account to auth as a user. This is very useful when migrating an entire domain, as it frees you from needing to know each individual user's password. IMAPSync supports this with the "authuser" command. The same example command, except using the admin account, would go something like this:
imapsync --host1 mail.some_mail_server.com \ --user1 some_username \ --authuser1 admin_account_name \ --password1 admin_account_password \ --host2 zimbra.xmission.com \ --user2 firstname.lastname@example.org \ --authuser2 email@example.com\ --password2 admin_account_zimbra_password \ --syncinternaldates --subscribe
We suggest considering the following options for use with your script.
When importing to Zimbra we find that IMAPsync seems to struggle with flags:
The exclusion of the Trash and Junk mail folders in particular can be quite a time saver depending on the originating server settings and your organizational requirements:
--exclude "Emailed Contacts" --exclude "Trash" --exclude "Junk"
If you retain Hangouts messages those may not transfer cleanly to an IMAP folder:
The following two options together help imapsync determine a message uniquely:
Because of the way Gmail and Google Apps use labels instead of traditional imap folders it is possible for an email to be duplicated across multiple folders. This can use quite a lot of additional storage space if you make heavy use of labels. The following option will see a unique message (in conjunction with the two uniqueness options above) and only copy it only once. If you rely or prefer a message to have multiple labels omit this option, just note that you will have multiple copies of that message, one in each imap folder:
The option above will copy the message into the folder of the corresponding label that imapsync encounters first. For that reason imapsync has several options to change what order folders/labels are processed. You will likely want the following option since Gmail will put all messages under this label and you will lose any other label:
--folderlast "[Gmail]/All Mail"
If you have already run your script and decide you do not want duplicate messages after all the following option will delete any duplicate messages that it finds on the "host2" host (note the "2" in the option name). This option may be dangerous without the two uniqueness options mentioned above:
Google Apps Admin Account Logins
Migrating from Google Apps to Zimbra at XMission.
Google Apps support using OAuth v1 and v2 to admin authenticate to other accounts. (Note: OAuth v1 was deprecated by Google on April 15th, 2015 and may stop working at any time) You may use either of the processes outlined below to have something like a "master password" for accessing Google Apps accounts without the administrator needing to know each individual user's password. The OAuth v1 process is a little easier to use, and is recommended while it is still available.
Migrate from Google Apps to Zimbra with OAuth v1
To use OAuth1 with imapsync, either from or to Google Apps, you will need to generate an OAuth consumer key, and then use this key as the password while setting the "authmech" imapsync flag to "xoauth". Also note that gmail requires ssl over imap.
To generate the OAuth1 key, in Google Apps, go to the admin console and choose Security -> Advanced Settings -> Manage OAuth Domain Key. Click "Enable this consumer key" if it's not already enabled. The string of random characters under "OAuth Consumer Secret" is the "master password". Make sure "Two-Legged OAuth Access Control" is checked.
To use this key, you want the password to be the master key, and the authmech to be xoauth. If your master key was "blork", your imapsync command might look like this:
imapsync --host1 imap.gmail.com -ssl1 \ --user1 firstname.lastname@example.org \ --password1 blork \ --authmech1 xoauth \ --host2 zimbra.xmission.com \ --user2 email@example.com \ --authuser2 firstname.lastname@example.org \ --password2 admin_account_password --syncinternaldates --subscribe
Note, once again, that OAuth1 is deprecated and support may be removed at Google's discretion.
Migrate from Google Apps to Zimbra with OAUTH v2
The newest version of IMAPSync has support for OAUTH v2. The process is a little more involved.
Provisioning OAUTH2 in Google Apps
1.) Setup a "service account" in the google developer api web site 2.) Download an encrypted key file and note the service account's email address form 3.) Associate the service account with your google apps domain
In the following link, "Google APIs Console" and "Google Apps Dashboard" sections illustrate setting it up:
Using OAUTH2 in IMAPSync
1.) Set the authmech for the user to "xoauth2" 2.) Make sure the key file google created for your service account is present on the local filesystem 3.) Provide the service account address, encrypted keyfile location and optionally the keyfile password (google's default password will be assumed, so you probably never need to do this) in the following format: <service account>;<keyfile location>;<keyfile password>
An example of the arguments for my google apps domain: --authmech2 xoauth2 --password2 'email@example.com;notasecret.p12'
Example session (using --justlogin, AUTHENTICATED is the confirmation of successful login):
[server :11:36PM] imapsync>./imapsync --justlogin --host1 imap.gmail.com --ssl1 --authmech1 xoauth2 --user1 firstname.lastname@example.org --password1 'email@example.com;notasecret.p12' --host2 zimbra.xmission.com --user2 firstname.lastname@example.org --authuser2 email@example.com --password2 somepassword Transfer started at Fri Jan 30 23:37:05 2015 PID is 31730 Log file is LOG_imapsyncfirstname.lastname@example.org $RCSfile: imapsync,v $ $Revision: 1.607 $ $Date: 2014/11/14 16:25:06 $ Here is a [linux] system (Linux server 3.2.0-58-generic #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013 x86_64) With perl 5.14.2 Mail::IMAPClient 3.30 Command line used: ./imapsync --justlogin --host1 imap.gmail.com --ssl1 --authmech1 xoauth2 --user1 email@example.com --password1 MASKED --host2 zimbra.xmission.com --user2 firstname.lastname@example.org --authuser2 email@example.com --password2 MASKED Temp directory is /tmp PID file is /tmp/imapsync.pid Modules version list: Mail::IMAPClient 3.30 IO::Socket 1.32 IO::Socket::IP ? IO::Socket::INET 1.31 IO::Socket::SSL 1.53 Net::SSLeay 1.42 Digest::MD5 2.51 Digest::HMAC_MD5 1.01 Digest::HMAC_SHA1 1.03 Term::ReadKey 2.30 Authen::NTLM ? File::Spec 3.33 Time::HiRes 1.972101 URI::Escape 3.31 Data::Uniqid 0.12 Unicode::String 2.09 Info: turned ON syncinternaldates, will set the internal dates (arrival dates) on host2 same as host1. Info: will try to use LOGIN authentication on host1 Info: will try to use XOAUTH2 authentication on host2 Info: imap connexions timeout is 120 seconds Host1: IMAP server [imap.gmail.com] port  user [firstname.lastname@example.org] Host2: IMAP server [mail.xmission.com] port  user [email@example.com] Host1: * OK Gimap ready for requests from 22.214.171.124 qu16mb267177216pab Host1: imap.gmail.com says it has CAPABILITY for AUTHENTICATE XOAUTH2 Host1: success login on [imap.gmail.com] with user [firstname.lastname@example.org] auth [XOAUTH2] Host2: * OK IMAP4 ready Host2: zimbra.xmission.com says it has NO CAPABILITY for AUTHENTICATE LOGIN Host2: success login on [zimbra.xmission.com] with user [email@example.com] auth [LOGIN] Host1: state Authenticated Host2: state Authenticated Host1 capability: IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT Host2 capability: IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SORT SORT=MODSEQ SORT=DISPLAY THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE LIST-EXTENDED WITHIN QRESYNC SCAN XLIST URLAUTH URLAUTH=BINARY LOGINDISABLED COMPRESS=DEFLATE IDLE
Common errors in transitioning accounts include:
- Incorrect email address or password
- Gmail Account locked down to not allow IMAP access
- Change this in user mailbox settings
- Gmail account will allow you to login via webmail but not with other apps
- We know the account works. https://support.google.com/mail/answer/14257?rd=1
- Likely need to allow less secure apps access https://support.google.com/accounts/answer/6010255?hl=en