dovecot_migration.md

Email Migration with Dovecot

Notes on migrating 20 years of personal email from dovecot on an iMac to Synology Mail Server on a DS920+.

Preparation

setting	FROM	TO
host	corroboree.hrsoftworks.net	REDACTED
version	2.2.18	2.3.5.2 (a83ad21)
install dir	/usr/local/	/volume1/@appstore/MailServer aka /var/packages/MailServer/target
mail_location	~/Library/Mail/Maildir	~/.Maildir

• Working on the assumption that Synology Mail Server is actually dovecot for all purposes I care about.
  • There's various scattered references suggesting they're closely related, and this old reference suggests that at least some point in time the mail serving part of it is.
  • I also have my fingers crossed tightly.
• Compared Synology Mail Server to Synology MailPlus Server. Seems the latter is the same, except lots of extra user features. Since I just want it to serve mail, none of them appear to be relevant.
  • Mail is free. MailPlus is paid.
  • 6 years ago Synology said they intend to keep Mail going "for now".
  • Ah, MailPlus has a migration feature! Oh well, still not worth it for me.

• Consulted Migration To Dovecot and imapsync, and concluded that dovecot's built in dsync is the way to go.
• Main reference: Migrating from any IMAP/POP3 server to Dovecot via dsync

• Compared dovecot.conf FROM and TO.
  • No auto generated mailboxes, and special_use mailboxes are the same, so that's a big plus for a smooth migration.
  • Waaaaaaaay more default configuration in TO, but happy to inherit new defaults as long as they're compatible.
  • Other than namespace inbox, which is effectively the same, the FROM configuration is just:

   disable_plaintext_auth = no
   mail_location = maildir:/Users/liteyear/Library/Mail/Maildir
   passdb {
     driver = pam
   }
   ssl = no
   userdb {
     driver = passwd
   }
  

  • All those are either auth related and mail_location , both of which I'm happy to be different.
  • I don't appear to have any master users set up. Just the mail user.
• Create IMAP migration configuration as `/volume1/@appstore/MailServer/etc/dovecot/conf.d/dovecot-migration.conf:

imapc_host = corroboree.hrsoftworks.net

# Authenticate as masteruser / masteruser-secret, but use a separate login user.
# If you don't have a master user, remove the imapc_master_user setting.
imapc_user = %u
#imapc_master_user = masteruser
imapc_password = masteruser-secret

imapc_features = rfc822.size
# If you have Dovecot v2.2.8+ you may get a significant performance improvement with fetch-headers:
imapc_features = $imapc_features fetch-headers
# Read multiple mails in parallel, improves performance
mail_prefetch_count = 20

# If the old IMAP server uses INBOX. namespace prefix, set:
#imapc_list_prefix = INBOX

# for SSL:
#imapc_port = 993
#imapc_ssl = imaps
#ssl_client_ca_dir = /etc/ssl
#imapc_ssl_verify = yes
# for <2.2.0: change ssl_client_ca_dir=/etc/ssl to imapc_ssl_ca_dir=/etc/ssl

• Ended up very simple.
• Note this adds the migration configuration, rather than replacing what's there. The reference says to put it straight in etc/dovecot, assumedly so it becomes the config, rather than adding to it, but I don't want to have to figure out what additional settings are required.
  • Oh I see, migrating_mailboxes is assumed knowledge.
• Then apparently to execute you just do:
  • doveadm -o mail_fsync=never backup -R -u user@domain imapc:
• But I'm confused about whether this is on TO (surely, since no assumption is made that FROM is dovecot) and if so, who user@domain is.
• I'm also confused about what "dsync" is after all, since it doesn't appear in the actual commands.
• Eh, according to migrating_mailboxes it turns out with v2.1.14+ on the old server I can use the doveadm protocol instead of imap to do the transfer. But is that better?
• Okay, so cancel that, I'm going with the migrating_mailboxes method. None of dovecot-migration.conf is necessary. Only need to enable a doveadm listener on FROM with:

service doveadm {
   inet_listener {
     port = 12354
   }
}

doveadm_password = supersecret

• And then also add a (same?) doveadm password to TO??
• Note doveadm is in bin/ while dovecot is in sbin/.
• Ah, I see!
  • -u specifies the mail username (%u) to use
  • doveadm_password on TO does indeed need to match FROM, so in mind that makes more sense as a -o option passed on the doveadm command line.
  • The doveadm protocol itself is a fine alternative to IMAP for migration, so will just forward that port instead.
  • The last parameter to doveadm specifies the protocol: tcp for doveadm and imapc for IMAP (client).
  • Easy! Once you know.

Action

# On FROM
sudo vi /usr/local/etc/dovecot/dovecot.conf # add "service doveadm" section
sudo launchctl unload /Library/LaunchDaemons/net.hrsoftworks.dovecot.plist
sudo launchctl load /Library/LaunchDaemons/net.hrsoftworks.dovecot.plist
tail /var/log/mail.log # confirm dovecot restarted
nc 127.0.0.1 12354 # confirm the service is listening
launchctl unload ~/Library/LaunchAgents/net.hrsoftworks.getmail.plist # stop incoming mail

# On TO
ssh -L 12354:localhost:12354 -N -f corroboree.hrsoftworks.net # to save having to forward ports on router, tunnel over SSH instead. I figure the overhead is mostly compute, not bandwidth, so will still be bandwidth constrained with minor impact.
sudo bin/doveadm -o doveadm_password=supersecret -D backup -Ru liteyear tcp:localhost:12354

Progress

• Running flat chat at 600kB/s down (capped at uplink capacity of FROM).
• Files and folders appearing in ~/.Maildir
• Logs show that nothing existing was found and everything is being created using defaults.
• No progress indicator, but can figure it out from size of maildir:
  • $ du -s Library/Mail/Maildir = 31178504 (512 byte blocks) = 15GB.
  • (31178504*512) / (600*1024) = 25982 seconds = 7.2 hours

Now we wait!

Result

• Unsatisfying. Appears to have finished but:
  • last message is a "client disconnect" error, and
  • byte size is 15100463038 instead of 15963394048.
• Checking it's all there...
  • Running du -a | sed '/.*\.\/.*\/.*/!d' | cut -d/ -f2 | sort | uniq -c in FROM and TO.
  • Mostly the same. Lots with 1 file less, that seems to be a dovecot.index.log.2 file which is fine by me.
  • Some with 2 files less: dovecot-keywords and dovecot.index. I think this is okay.
  • Junk folder is way smaller. Weird, but okay.
  • .Newie Ventures.Meeting Room has 7 more files. Don't know why.
  • Otherwise all accounted for!
• Changed IMAP settings email client and crossed fingers it would re-connect and then do basically nothing because the state is the same.
  • Seems to have connected fine, but stuck on "Downloading 1 new message".
  • Finally went away and seems happy and stable. Haven't tried changing anything to trigger a new read/write yet.

Have since added, deleted, moved and marked emails as read, and not issues found.