Difference between revisions of "Zimbra Migration: IMAPSync"

From XMission Wiki
Jump to: navigation, search
(Created page with "<strong> * Data Type(s) Migrated: Mail * Required for: POP users not using Outlook * Recommended for: POP users not using Outlook, users that use a mail client </strong> This...")
 
(Break "TIPS" out into own section; add a few imapsync flags helpful for working with Gmail)
 
(10 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 +
= Overview =
 
<strong>
 
<strong>
 
* Data Type(s) Migrated: Mail
 
* Data Type(s) Migrated: Mail
−
* Required for: POP users not using Outlook
+
* Recommended for: XMission Zimbra customers with xmbase mailboxes, or customers migrating from non-Exchange mail servers
−
* Recommended for: POP users not using Outlook, users that use a mail client
+
* Some technical proficiency required (command line-based, requires Linux, Mac OS X or cygwin)
 
</strong>
 
</strong>
  
−
This can often be the easiest method for users to migrate their own data, particularly if they already have a configured mail client. For users relying on POP3, this can often be the <i>only</i> method to migrate downloaded mail.
+
This can be the most robust method of migrating, though it's only limited to mail. IMAPSync will compare source and destination hosts, and copy any messages not on the destination server from the host server. You can run this multiple times and generally not result in any duplicate messages in the destination mailbox.
  
−
Simply configure the destination mail server as an additional IMAP mail server, and then copy the existing mail from their original folders to the folders of the destination mail server
+
It is a command line-based perl script, so it requires either some sort of unix-based system or cygwin (or other compatible Perl setup on Windows). Consequently, it does require some level of technical proficiency to be used.
 +
 
 +
Previously, this was normal free OSS software, but the author has since started requiring payment for it. Yet, not only was it previously released under an open source license, but it continues to be licensed as OSS with the newer paid versions, so you can still get reasonably up-to-date versions legally. If you wish to purchase the software, you can get a windows executable. All others will want to install from the git repository.
 +
 
 +
* IMAPSync Homepage: http://imapsync.lamiral.info/
 +
* Open source repository: https://github.com/imapsync/imapsync
 +
 
 +
You will want to read the documentation and determine what commands are appropriate to your situation, but the general idea is that you provide the source host/user/password, the destination host/user/password, imapsync logs into both and transfers the mail. These are delineated as host1/user1/password1 and host2/user2/password2 in the command line options.
 +
 
 +
Here is an example IMAPSync command:
 +
 
 +
imapsync --host1 mail.some_mail_server.com \
 +
--user1 some_username \
 +
--password1 some_password \
 +
--host2 zimbra.xmission.com \
 +
--user2 account_on_zimbra@domain.com \
 +
--password2 zimbra_password \
 +
--syncinternaldates --subscribe
 +
 
 +
= IMAP Admin Account Logins =
 +
 
 +
Also note that many mail servers (including XMission's Zimbra servers) support the ability for an admin account to auth as a user. This is very useful when migrating an entire domain, as it frees you from needing to know each individual user's password. IMAPSync supports this with the "authuser" command. The same example command, except using the admin account, would go something like this:
 +
 
 +
imapsync --host1 mail.some_mail_server.com \
 +
--user1 some_username \
 +
--authuser1 admin_account_name \
 +
--password1 admin_account_password \
 +
--host2 zimbra.xmission.com \
 +
--user2 account_on_zimbra@domain.com \
 +
--authuser2 admin_account_on_zimbra@domain.com\
 +
--password2 admin_account_zimbra_password \
 +
--syncinternaldates --subscribe
 +
 
 +
= TIPS =
 +
 
 +
We suggest considering the following options for use with your script.
 +
 
 +
When importing to Zimbra we find that IMAPsync seems to struggle with flags:
 +
 
 +
  --regexflag "s/:FLAG/_FLAG/g"
 +
 
 +
The exclusion of the Trash and Junk mail folders in particular can be quite a time saver depending on the originating server settings and your organizational requirements:
 +
 
 +
  --exclude "Emailed Contacts"
 +
  --exclude "Trash"
 +
  --exclude "Junk"
 +
 
 +
If you retain Hangouts messages those ''may'' not transfer cleanly to an IMAP folder:
 +
 
 +
  --exclude "Chats"
 +
 
 +
The following two options together help imapsync determine a message uniquely:
 +
 
 +
  --useheader="X-Gmail-Received"
 +
  --useheader="Message-Id"
 +
 
 +
Because of the way Gmail and Google Apps use labels instead of traditional imap folders it is possible for an email to be duplicated across multiple folders. This can use quite a lot of additional storage space if you make heavy use of labels. The following option will see a unique message (in conjunction with the two uniqueness options above) and only copy it only once. If you rely or prefer a message to have multiple labels omit this option, just note that you will have multiple copies of that message, one in each imap folder:
 +
 
 +
  --skipcrossduplicates
 +
 
 +
The option above will copy the message into the folder of the corresponding label that imapsync encounters first. For that reason imapsync has several options to change what order folders/labels are processed. You will likely want the following option since Gmail will put all messages under this label and you will lose any other label:
 +
 
 +
  --folderlast "[Gmail]/All Mail"
 +
 
 +
If you have already run your script and decide you do not want duplicate messages after all the following option will delete any duplicate messages that it finds on the "host2" host (note the "2" in the option name). This option may be dangerous without the two uniqueness options mentioned above:
 +
 
 +
  --delete2duplicates
 +
 
 +
= Google Apps Admin Account Logins =
 +
 
 +
'''Migrating from Google Apps to Zimbra at XMission.'''
 +
 
 +
Google Apps support using OAuth v1 and v2 to admin authenticate to other accounts. (Note: OAuth v1 was deprecated by Google on April 15th, 2015 and may stop working at any time)  You may use either of the processes outlined below to have something like a "master password" for accessing Google Apps accounts without the administrator needing to know each individual user's password. The OAuth v1 process is a little easier to use, and is recommended while it is still available.
 +
 
 +
== Migrate from Google Apps to Zimbra with OAuth v1 ==
 +
 
 +
To use OAuth1 with imapsync, either from or to Google Apps, you will need to generate an OAuth consumer key, and then use this key as the password while setting the "authmech" imapsync flag to "xoauth". Also note that gmail requires ssl over imap.
 +
 
 +
To generate the OAuth1 key, in Google Apps, go to the admin console and choose Security -> Advanced Settings -> Manage OAuth Domain Key. Click "Enable this consumer key" if it's not already enabled.  The string of random characters under "OAuth Consumer Secret" is the "master password". Make sure "Two-Legged OAuth Access Control" is checked.
 +
 
 +
To use this key, you want the password to be the master key, and the authmech to be xoauth. If your master key was "blork", your imapsync command might look like this:
 +
 
 +
imapsync --host1 imap.gmail.com -ssl1 \
 +
--user1 user@domain.com \
 +
--password1 blork \
 +
--authmech1 xoauth \
 +
--host2 zimbra.xmission.com \
 +
--user2 account_on_zimbra@domain.com \
 +
--authuser2 admin_account@domain.com \
 +
--password2 admin_account_password
 +
--syncinternaldates --subscribe
 +
 
 +
Note, once again, that OAuth1 is deprecated and support may be removed at Google's discretion.
 +
 
 +
== Migrate from Google Apps to Zimbra with OAUTH v2 ==
 +
 
 +
The newest version of IMAPSync has support for OAUTH v2. The process is a little more involved.
 +
 
 +
=== Provisioning OAUTH2 in Google Apps ===
 +
 
 +
1.) Setup a "service account" in the google developer api web site
 +
2.) Download an encrypted key file and note the service account's email address form
 +
3.) Associate the service account with your google apps domain
 +
 
 +
In the following link, "Google APIs Console" and "Google Apps Dashboard" sections illustrate setting it up:
 +
 
 +
http://www.limilabs.com/blog/oauth2-gmail-imap-service-account
 +
 
 +
 
 +
=== Using OAUTH2 in IMAPSync ===
 +
 
 +
1.) Set the authmech for the user to "xoauth2"
 +
2.) Make sure the key file google created for your service account is present on the local filesystem
 +
3.) Provide the service account address, encrypted keyfile location and optionally the keyfile password (google's default password will be assumed, so you probably never need to do this) in the following format: <service account>;<keyfile location>;<keyfile password>
 +
 
 +
An example of the arguments for my google apps domain: --authmech2 xoauth2 --password2 '587056876087-mjdh0aiajc0ru7gpb7434si8nlitrbuo@developer.gserviceaccount.com;notasecret.p12'
 +
 
 +
 
 +
Example session (using --justlogin, AUTHENTICATED is the confirmation of successful login):
 +
 
 +
<code><pre>
 +
[server :11:36PM] imapsync>./imapsync --justlogin --host1 imap.gmail.com --ssl1 --authmech1 xoauth2 --user1 testuser@domain.com --password1 '587056876087-mjdh0aiajc0ru7gpb7434si8nlitrbuo@developer.gserviceaccount.com;notasecret.p12' --host2 zimbra.xmission.com --user2 testuser@domain.com --authuser2 admin_account@domain.com --password2 somepassword
 +
Transfer started at Fri Jan 30 23:37:05 2015
 +
PID is 31730
 +
Log file is LOG_imapsync/2015_01_30_23_37_05_testuser@notearthday.org.txt
 +
$RCSfile: imapsync,v $ $Revision: 1.607 $ $Date: 2014/11/14 16:25:06 $
 +
Here is a [linux] system (Linux server 3.2.0-58-generic #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013 x86_64)
 +
With perl 5.14.2 Mail::IMAPClient  3.30
 +
Command line used:
 +
./imapsync --justlogin --host1 imap.gmail.com --ssl1 --authmech1 xoauth2 --user1 testuser@domain.com --password1 MASKED --host2 zimbra.xmission.com --user2 testuser@domain.com --authuser2 admin_account@domain.com --password2 MASKED
 +
Temp directory is /tmp
 +
PID file is /tmp/imapsync.pid
 +
Modules version list:
 +
Mail::IMAPClient    3.30
 +
IO::Socket          1.32
 +
IO::Socket::IP      ?
 +
IO::Socket::INET    1.31
 +
IO::Socket::SSL      1.53
 +
Net::SSLeay          1.42
 +
Digest::MD5          2.51
 +
Digest::HMAC_MD5    1.01
 +
Digest::HMAC_SHA1    1.03
 +
Term::ReadKey        2.30
 +
Authen::NTLM        ?
 +
File::Spec          3.33
 +
Time::HiRes          1.972101
 +
URI::Escape          3.31
 +
Data::Uniqid        0.12
 +
Unicode::String      2.09
 +
 
 +
Info: turned ON syncinternaldates, will set the internal dates (arrival dates) on host2 same as host1.
 +
Info: will try to use LOGIN authentication on host1
 +
Info: will try to use XOAUTH2 authentication on host2
 +
Info: imap connexions timeout is 120 seconds
 +
Host1: IMAP server [imap.gmail.com] port [993] user [testuser@domain.com]
 +
Host2: IMAP server [mail.xmission.com] port [143] user [testuser@domain.com]
 +
Host1: * OK Gimap ready for requests from 199.104.120.118 qu16mb267177216pab
 +
Host1: imap.gmail.com says it has CAPABILITY for AUTHENTICATE XOAUTH2
 +
Host1: success login on [imap.gmail.com] with user [testuser@notearthday.org] auth [XOAUTH2]
 +
Host2: * OK IMAP4 ready
 +
Host2: zimbra.xmission.com says it has NO CAPABILITY for AUTHENTICATE LOGIN
 +
Host2: success login on [zimbra.xmission.com] with user [testuser@domain.com] auth [LOGIN]
 +
Host1: state Authenticated
 +
Host2: state Authenticated
 +
Host1 capability: IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT
 +
Host2 capability: IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SORT SORT=MODSEQ SORT=DISPLAY THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE LIST-EXTENDED WITHIN QRESYNC SCAN XLIST URLAUTH URLAUTH=BINARY LOGINDISABLED COMPRESS=DEFLATE IDLE
 +
</pre></code>
 +
 
 +
 
 +
=== Troubleshooting ===
 +
 
 +
Common errors in transitioning accounts include:
 +
 
 +
* Incorrect email address or password
 +
* Gmail Account locked down to not allow IMAP access
 +
** Change this in user mailbox settings
 +
* Gmail account will allow you to login via webmail but not with other apps
 +
** We know the account works. https://support.google.com/mail/answer/14257?rd=1
 +
** Likely need to allow less secure apps access https://support.google.com/accounts/answer/6010255?hl=en
 +
 
 +
[[Category: Zimbra]]

Latest revision as of 18:30, 31 March 2019

Overview

  • Data Type(s) Migrated: Mail
  • Recommended for: XMission Zimbra customers with xmbase mailboxes, or customers migrating from non-Exchange mail servers
  • Some technical proficiency required (command line-based, requires Linux, Mac OS X or cygwin)

This can be the most robust method of migrating, though it's only limited to mail. IMAPSync will compare source and destination hosts, and copy any messages not on the destination server from the host server. You can run this multiple times and generally not result in any duplicate messages in the destination mailbox.

It is a command line-based perl script, so it requires either some sort of unix-based system or cygwin (or other compatible Perl setup on Windows). Consequently, it does require some level of technical proficiency to be used.

Previously, this was normal free OSS software, but the author has since started requiring payment for it. Yet, not only was it previously released under an open source license, but it continues to be licensed as OSS with the newer paid versions, so you can still get reasonably up-to-date versions legally. If you wish to purchase the software, you can get a windows executable. All others will want to install from the git repository.

You will want to read the documentation and determine what commands are appropriate to your situation, but the general idea is that you provide the source host/user/password, the destination host/user/password, imapsync logs into both and transfers the mail. These are delineated as host1/user1/password1 and host2/user2/password2 in the command line options.

Here is an example IMAPSync command: 

imapsync --host1 mail.some_mail_server.com \
--user1 some_username \
--password1 some_password \
--host2 zimbra.xmission.com \
--user2 account_on_zimbra@domain.com \
--password2 zimbra_password \
--syncinternaldates --subscribe

IMAP Admin Account Logins

Also note that many mail servers (including XMission's Zimbra servers) support the ability for an admin account to auth as a user. This is very useful when migrating an entire domain, as it frees you from needing to know each individual user's password. IMAPSync supports this with the "authuser" command. The same example command, except using the admin account, would go something like this: 

imapsync --host1 mail.some_mail_server.com \
--user1 some_username \
--authuser1 admin_account_name \
--password1 admin_account_password \
--host2 zimbra.xmission.com \
--user2 account_on_zimbra@domain.com \
--authuser2 admin_account_on_zimbra@domain.com\
--password2 admin_account_zimbra_password \
--syncinternaldates --subscribe

TIPS

We suggest considering the following options for use with your script.

When importing to Zimbra we find that IMAPsync seems to struggle with flags: 

  --regexflag "s/:FLAG/_FLAG/g"

The exclusion of the Trash and Junk mail folders in particular can be quite a time saver depending on the originating server settings and your organizational requirements: 

  --exclude "Emailed Contacts" 
  --exclude "Trash" 
  --exclude "Junk"

If you retain Hangouts messages those may not transfer cleanly to an IMAP folder: 

  --exclude "Chats"

The following two options together help imapsync determine a message uniquely: 

  --useheader="X-Gmail-Received"
  --useheader="Message-Id"

Because of the way Gmail and Google Apps use labels instead of traditional imap folders it is possible for an email to be duplicated across multiple folders. This can use quite a lot of additional storage space if you make heavy use of labels. The following option will see a unique message (in conjunction with the two uniqueness options above) and only copy it only once. If you rely or prefer a message to have multiple labels omit this option, just note that you will have multiple copies of that message, one in each imap folder: 

  --skipcrossduplicates

The option above will copy the message into the folder of the corresponding label that imapsync encounters first. For that reason imapsync has several options to change what order folders/labels are processed. You will likely want the following option since Gmail will put all messages under this label and you will lose any other label: 

  --folderlast "[Gmail]/All Mail"

If you have already run your script and decide you do not want duplicate messages after all the following option will delete any duplicate messages that it finds on the "host2" host (note the "2" in the option name). This option may be dangerous without the two uniqueness options mentioned above: 

  --delete2duplicates

Google Apps Admin Account Logins

Migrating from Google Apps to Zimbra at XMission.

Google Apps support using OAuth v1 and v2 to admin authenticate to other accounts. (Note: OAuth v1 was deprecated by Google on April 15th, 2015 and may stop working at any time) You may use either of the processes outlined below to have something like a "master password" for accessing Google Apps accounts without the administrator needing to know each individual user's password. The OAuth v1 process is a little easier to use, and is recommended while it is still available.

Migrate from Google Apps to Zimbra with OAuth v1

To use OAuth1 with imapsync, either from or to Google Apps, you will need to generate an OAuth consumer key, and then use this key as the password while setting the "authmech" imapsync flag to "xoauth". Also note that gmail requires ssl over imap.

To generate the OAuth1 key, in Google Apps, go to the admin console and choose Security -> Advanced Settings -> Manage OAuth Domain Key. Click "Enable this consumer key" if it's not already enabled. The string of random characters under "OAuth Consumer Secret" is the "master password". Make sure "Two-Legged OAuth Access Control" is checked.

To use this key, you want the password to be the master key, and the authmech to be xoauth. If your master key was "blork", your imapsync command might look like this: 

imapsync --host1 imap.gmail.com -ssl1 \
--user1 user@domain.com \
--password1 blork \
--authmech1 xoauth \
--host2 zimbra.xmission.com \
--user2 account_on_zimbra@domain.com \
--authuser2 admin_account@domain.com \
--password2 admin_account_password
--syncinternaldates --subscribe

Note, once again, that OAuth1 is deprecated and support may be removed at Google's discretion.

Migrate from Google Apps to Zimbra with OAUTH v2

The newest version of IMAPSync has support for OAUTH v2. The process is a little more involved.

Provisioning OAUTH2 in Google Apps

1.) Setup a "service account" in the google developer api web site 2.) Download an encrypted key file and note the service account's email address form 3.) Associate the service account with your google apps domain

In the following link, "Google APIs Console" and "Google Apps Dashboard" sections illustrate setting it up:

http://www.limilabs.com/blog/oauth2-gmail-imap-service-account


Using OAUTH2 in IMAPSync

1.) Set the authmech for the user to "xoauth2" 2.) Make sure the key file google created for your service account is present on the local filesystem 3.) Provide the service account address, encrypted keyfile location and optionally the keyfile password (google's default password will be assumed, so you probably never need to do this) in the following format: <service account>;<keyfile location>;<keyfile password>

An example of the arguments for my google apps domain: --authmech2 xoauth2 --password2 '587056876087-mjdh0aiajc0ru7gpb7434si8nlitrbuo@developer.gserviceaccount.com;notasecret.p12'


Example session (using --justlogin, AUTHENTICATED is the confirmation of successful login): 

[server :11:36PM] imapsync>./imapsync --justlogin --host1 imap.gmail.com --ssl1 --authmech1 xoauth2 --user1 testuser@domain.com --password1 '587056876087-mjdh0aiajc0ru7gpb7434si8nlitrbuo@developer.gserviceaccount.com;notasecret.p12' --host2 zimbra.xmission.com --user2 testuser@domain.com --authuser2 admin_account@domain.com --password2 somepassword
Transfer started at Fri Jan 30 23:37:05 2015
PID is 31730
Log file is LOG_imapsync/2015_01_30_23_37_05_testuser@notearthday.org.txt
$RCSfile: imapsync,v $ $Revision: 1.607 $ $Date: 2014/11/14 16:25:06 $
Here is a [linux] system (Linux server 3.2.0-58-generic #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013 x86_64)
With perl 5.14.2 Mail::IMAPClient  3.30
Command line used:
./imapsync --justlogin --host1 imap.gmail.com --ssl1 --authmech1 xoauth2 --user1 testuser@domain.com --password1 MASKED --host2 zimbra.xmission.com --user2 testuser@domain.com --authuser2 admin_account@domain.com --password2 MASKED
Temp directory is /tmp
PID file is /tmp/imapsync.pid
Modules version list:
Mail::IMAPClient     3.30
IO::Socket           1.32
IO::Socket::IP       ?
IO::Socket::INET     1.31
IO::Socket::SSL      1.53
Net::SSLeay          1.42
Digest::MD5          2.51
Digest::HMAC_MD5     1.01
Digest::HMAC_SHA1    1.03
Term::ReadKey        2.30
Authen::NTLM         ?
File::Spec           3.33
Time::HiRes          1.972101
URI::Escape          3.31
Data::Uniqid         0.12
Unicode::String      2.09

Info: turned ON syncinternaldates, will set the internal dates (arrival dates) on host2 same as host1.
Info: will try to use LOGIN authentication on host1
Info: will try to use XOAUTH2 authentication on host2
Info: imap connexions timeout is 120 seconds
Host1: IMAP server [imap.gmail.com] port [993] user [testuser@domain.com]
Host2: IMAP server [mail.xmission.com] port [143] user [testuser@domain.com]
Host1: * OK Gimap ready for requests from 199.104.120.118 qu16mb267177216pab
Host1: imap.gmail.com says it has CAPABILITY for AUTHENTICATE XOAUTH2
Host1: success login on [imap.gmail.com] with user [testuser@notearthday.org] auth [XOAUTH2]
Host2: * OK IMAP4 ready
Host2: zimbra.xmission.com says it has NO CAPABILITY for AUTHENTICATE LOGIN
Host2: success login on [zimbra.xmission.com] with user [testuser@domain.com] auth [LOGIN]
Host1: state Authenticated
Host2: state Authenticated
Host1 capability: IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT
Host2 capability: IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SORT SORT=MODSEQ SORT=DISPLAY THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE LIST-EXTENDED WITHIN QRESYNC SCAN XLIST URLAUTH URLAUTH=BINARY LOGINDISABLED COMPRESS=DEFLATE IDLE


Troubleshooting

Common errors in transitioning accounts include:

Retrieved from "http://wiki.xmission.com/index.php?title=Zimbra_Migration:_IMAPSync&oldid=9713"