Backup mails – backup and restore

After setting up a Dovecot instance and having imapsync running, how could I do the main task, the backup?

I use tar with its incremental option (infos about can be found here and here). Also I compress the backup, if you have pigz on your computer, use it, it makes the backup a lot faster because it uses all cpu cores.

Create first backup

sudo su -
cd /var/lib/docker/volumes/dovecot-server_dovecot-server-home-volume/_data/testaccount
tar -cvzf /root/testaccount-mail-backup-20240817.tgz -g /root/testaccount-mail-backup-20240817.inc Maildir/

To use all cores while compression:
tar -cvf /root/testaccount-mail-backup-20240817.tgz -g /root/testaccount-mail-backup-20240817.inc --use-compress-program=pigz Maildir/

Create next incremtal backup

cp /root/testaccount-mail-backup-20240817.inc /root/testaccount-mail-backup-20240818.inc
tar -cvf /root/testaccount-mail-backup-20240818.tgz -g /root/testaccount-mail-backup-20240818.inc --use-compress-program=pigz Maildir/

Restore mails vom backups

cd /var/lib/docker/volumes/dovecot-server_dovecot-server-home-volume/_data/testaccount
tar -xvf /root/testaccount-mail-backup-full-20240817.tgz --use-compress-program=pigz Maildir/ -g /dev/null
tar -xvf /root/testaccount-mail-backup-full-20240818.tgz --use-compress-program=pigz Maildir/ -g /dev/null
ls -la
# (1003:1003 is the group and user-id of the testaccount, change to the values for your account)
chown -R 1003:1003 Maildir/

Backup mails – setup imapsync

For backup of my mails, I have started with setup a local Dovecot instance (Backup mails – setup of a dovecot server).

Back in the days, I already searched for a solution and used imapsync. Because of my own examples how to use it and good experience with it, I used it again. It will be used to sync the mails between my main mailaccount and the local Dovecot accounts.

Install imapsync

To install imapsync, some software needs to be installed. From the official install documentation, you get the for different os the instructions, what and how to install. For me it is Ubuntu.

sudo apt-get install  \
libauthen-ntlm-perl     \
libclass-load-perl      \
libcrypt-openssl-rsa-perl \
libcrypt-ssleay-perl    \
libdata-uniqid-perl     \
libdigest-hmac-perl     \
libdist-checkconflicts-perl \
libencode-imaputf7-perl     \
libfile-copy-recursive-perl \
libfile-tail-perl       \
libio-compress-perl     \
libio-socket-inet6-perl \
libio-socket-ssl-perl   \
libio-tee-perl          \
libjson-webtoken-perl   \
libmail-imapclient-perl \
libmodule-scandeps-perl \
libnet-dbus-perl        \
libnet-ssleay-perl      \
libpar-packer-perl      \
libproc-processtable-perl \
libreadonly-perl        \
libregexp-common-perl   \
libsys-meminfo-perl     \
libterm-readkey-perl    \
libtest-fatal-perl      \
libtest-mock-guard-perl \
libtest-mockobject-perl \
libtest-pod-perl        \
libtest-requires-perl   \
libtest-simple-perl     \
libunicode-string-perl  \
liburi-perl             \
libtest-nowarnings-perl \
libtest-deep-perl       \
libtest-warn-perl       \
make                    \
time                    \
cpanminus
sudo cpanm Mail::IMAPClient

git clone https://github.com/imapsync/imapsync.git

cd imapsync

./imapsync --testslive

Setup imapsync

For each account, I want to sync, I setup a shell script with the commands to start the sync.

#!/usr/bin/bash
./imapsync \
--host1 imap.example.com --port1 993 --user1 username --password1 secretpassword --ssl1 \
--host2 localhost --port2 143 --user2 testaccount --password2 secretpassword2 --nossl2 --delete2 \
--nofoldersizes --exclude Trash
# --dry --justlogin
If you want to try out first, if login to the two servers with the account data works, uncomment the last line and add a \ at the end of the „Trash“. In that case, imapsync will try to connect and give you some information about, if connections were successful.

With the shell script you should be able to sync mails between first and second mail account. First sync can take some time, depending on the amount of mails.
 

Gmail options

If you have a gmail account, there is an option, which optimize the handling for gmail. If gmail is the first (source) account, add –gmail1. Information about handling and options can be found in the imapsync documentation for gmail. Important is the part to receive an app password for your gmail/google account! This is my shell script to sync gmail into the Dovecot instance.
 
#!/usr/bin/bash
./imapsync \
--host1 imap.gmail.com --port1 993 --user1 yourmail@gmail.com --password1 apppassword --ssl1 --gmail1 \
--host2 localhost --port2 143 --user2 verena --password2 secretpassword --nossl2 --delete2 \
--nofoldersizes --exclude Trash
# --dry --justlogin


 

Backup mails – setup of a dovecot server

Over the years, I have received a lot of mails, not all of them were moved to trash. Time to think about a solution to backup them and have way to restore them, if ever needed.

After a little thinking, I decided to use the following technics:

  1. Setup a local imap server (Dovecot) in Docker
  2. Setup local users in the dovecot server to sync the mails into it.
  3. Backup the mails stored in Maildir format by Dovecot with incremental backups.
  4. Try to restore the backup into a testaccount, to validate, the this is working

Currently I will execute the backup and restore steps manually. If I think after some time, it is bulletprof, I will think about how to automate. Right now the focus is on setup the infrastructure and commands to make backups possible.

How to setup a local dovecot server with Docker

I will run the Dovecot instance on one of my computers, currently it is running on my notebook.

Dockerfile

I use Ubuntu 24.04 for the Dovecot instance. 

FROM ubuntu:24.04

RUN apt-get update \
&& apt-get upgrade -y \
&& apt-get install -y dovecot-imapd locales \
&& apt-get clean \
&& locale-gen de_DE.UTF-8 \
&& update-locale LANG=de_DE.UTF-8 LC_MESSAGES=POSIX
COPY ./files/dovecot.conf /etc/dovecot/dovecot.conf
COPY ./files/addDovecotUser.sh /addDovecotUser.sh

RUN chmod +x /addDovecotUser.sh

ENV LANG=de_DE.utf8

EXPOSE 143

ENTRYPOINT ["dovecot", "-F"]

addDovecotUser.sh

To setup a user in the Dovecot instance, I have written the following script, which should be placed in the folder files in the folder, where the Dockerfile is placed.

#!/bin/bash

adduser $1
usermod -aG users $1
usermod -aG dovecot $1

mkdir -p /home/$1/Maildir/cur
mkdir -p /home/$1/Maildir/new
mkdir -p /home/$1/Maildir/tmp

chown -R $1:$1 /home/$1

dovecot.conf

Because I run and access the Dovecot instance only locally, I did not setup a configuration with SSL.

listen = *
auth_mechanisms = plain login
#log_path = /var/log/dovecot.log
log_path = /dev/stdout
mail_location = maildir:~/Maildir
protocols = imap
service imap-login {
inet_listener imap {
port = 143
}
}
ssl = no
disable_plaintext_auth = no

userdb {
driver = passwd
}
passdb {
driver = pam
}
protocol imap {
imap_idle_notify_interval = 2 mins
imap_max_line_length = 64 k
imap_client_workarounds = tb-extra-mailbox-sep
}
mail_max_userip_connections = 20

Docker Compose

At the end I use Docker Compose to setup and run the Dovecot instance. I limit the ressources for the Dovecot, for me it was enough.

services:
dovecot-server:
image: dovecot-server:latest
restart: always
cpus: 0.50
mem_limit: 1024m
mem_reservation: 256m
volumes:
- dovecot-server-home-volume:/home
networks:
- dovecot-net
ports:
- "143:143"
networks:
dovecot-net:
volumes:
dovecot-server-home-volume:

With

docker compose up 

you can start the local dovecot instance (I use Portainer with stacks to setup the infrastructure).

Setup a mailaccount in Dovecot

To create a new mailaccount in your Dovecot instance, you can execute bash inside the container and use it to execute the shell script. First execute the bash:

docker exec -it dovecot-server-dovecot-server-1 bash

Then execute the shell script and give the account name as an argument. It will ask you some more infos, which are not needed (except the password) and can be left empty (just press return):

root@f41d9f10fa99:/# ./addDovecotUser.sh test
info: Adding user `test' ...
info: Selecting UID/GID from range 1000 to 59999 ...
info: Adding new group `test' (1004) ...
info: Adding new user `test' (1004) with group `test (1004)' ...
info: Creating home directory `/home/test' ...
info: Copying files from `/etc/skel' ...
New password:
Retype new password:
passwd: password updated successfully
Changing the user information for test
Enter the new value, or press ENTER for the default
Full Name []: testaccount
Room Number []:
Work Phone []:
Home Phone []:
Other []:
Is the information correct? [Y/n]
info: Adding new user `test' to supplemental / extra groups `users' ...
info: Adding user `test' to group `users' ...
root@f41d9f10fa99:/#
Now you should have an imap account which could be used.
In Thunderbird I used the following configuration:
Server: localhost
Port: 143
Connection security: none
Auth method: password, unencrypted

Next step will be setup of imapsync, which will be used to sync the mails by imap protocol.

IMAP-Server syncen

In den letzten Wochen sind mir mehrmals Festplatten meiner „Server“ verreckt. Auf den Servern lief unter anderem mein Mailserver mit Dovecot. Nachts werden die Daten zwar immer gesichert, aber der letzte Crash fand genau zum Zeitpunkt des Backups statt. Daher konnte ich dieses Backup nicht nutzen. Zum Glück konnte ich auf die Festplatte noch per externem Festplattengehäuse zugreifen und den aktuellen Stand des Mailservers manuell sichern und in der neuen Installation wieder einspielen.

Aber wie kann man die Daten dann zeitnah sichern? Mir fiel ein, dass auf meinem Synology-NAS ein Mailserver aktiviert werden kann. Idee war nun, den Hauptmailserver mit dem Mailserver auf der Synology zeitnah zu syncen. Da auf der Synology auch noch ein Webmailer installiert werden kann, hätte ich so eine Notbehelfslösung beim nächsten Crash.

Ich hatte mir zuerst Offlineimap angeschaut. Leider zickt es mit selbstgenerierten Zertifikaten bei SSL Verbindungen. Dann Imapsync gefunden. Es kann komplett über die Kommandozeile gesteuert werden. Und, da ich meine Rechner mittlerweile alle unter Arch Linux laufen lasse, es findet sich ein passendes Installationspaket in AUR, dem Programmrepository welches von Nutzern gepflegt wird.

Nach der Installation von Imapsync habe ich mir ein Shell-Skript angelegt, welches den Aufruf von Imapsync kapselt. Da auf dem Mailserver nur mein Account liegt, ist es eine ganz simple Lösung.

Das Shellskript findet sich unter ~/imapsync_calibanatspace.sh und hat folgenden Inhalt:

#!/bin/bash
imapsync --host1 localhost --port1 993 --user1 benutzername1 --password1 geheimespasswort1 --ssl1 --host2 backupmailserver --port2 993 --user2 benutzername2 --password2 geheimespasswort2 --ssl2 --delete2

Das Shellskript wird per Cron regelmässig aufgerufen. Ich lasse den Syncvorgang testweise erstmal alle drei Stunden durchführen. Daher habe ich folgendes für meinen Benutzer in der Crontab eingetragen:

0 0,3,6,9,12,15,18,21 * * * ~/imapsync_calibanatspace.sh