Space Panda

Keeping Mail Local

In order to run mutt independently from my mail provider I keep my mail synchronised between my machine and the server(s).

This is a somewhat different approach from both IMAP and POP3, because I neither have to decide to keep my mail exclusively offline (the POP3 approach) nor do I have to keep it on the server (the IMAP approach).

Instead there’s a full copy of all mails on the server and on my machine. The synchronisation is done based on message IDs, as provided by the Maildir standard.

Yes, that means that the mail is stored locally in Maildir folders.

Overview

Suppose you have two mail providers:

  • Private mail account (here: example.org)

  • Work (here: bigcompany.com)

You might want to keep your private mail on the server as long as it’s an active conversation, but after a conversation is done, it should be no longer on the server.

The work email however should always stay on the server and never be deleted from there.

For the actual synchronisation step we’ll be using isync.

Folders

On the local machine I’m using a folder structure like this:

~
└── Mail
   ├── local
   ├── example.org
   └── bigcompany.com

The local folder contains the offline emails that are no longer on the server, example.org and bigcompany.org contain the folder structure as dictated by the respective mail server.

On the left is a laptop, below it the folder structure as described in Folders. In the middle a cloud represents the internet. On the right two servers represent the example.org and bigcompany.com mail servers.

Pretty much the example situation.

isync Configuration

isync’s configuration is supposed to be stored at ~/.mbsyncrc.

It contains a set of IMAP accounts, named Store definitions (locations that contain email, like IMAP servers, Maildir folders), and Channel configurations that connect the stores and define the synchronisation behaviour.

Given the example above, the configuration file could look like this:

# mbsyncrc

IMAPAccount private
Host imap.example.org
User baldrick
PassCmd "echo -n $PW_PRIVATE"
SSLType IMAPS
CertificateFile /etc/ssl/certs/ca-certificates.crt

IMAPAccount work
Host imap.bigcompany.com
User baldrick@bigcompany.com
PassCmd "echo -n $PW_WORK"
SSLType IMAPS
CertificateFile /etc/ssl/certs/ca-certificates.crt

IMAPStore private-remote
Account private

IMAPStore work-remote
Account work

MaildirStore private-local
Path ~/Mail/example.org
Inbox ~/Mail/example.org/Inbox
SubFolders verbatim

MaildirStore work-local
Path ~/Mail/bigcompany.com
Inbox ~/Mail/bigcompany.com/Inbox
SubFolders verbatim

Channel work
Master :work-remote:
Slave :work-local:
Create Slave
SyncState *
CopyArrivalDate yes
Expunge Slave

Channel private
Master :private-remote:
Slave :private-local:
Create Slave
SyncState *
CopyArrivalDate yes
Expunge Both

For details on what each options means, please see the manpage of mbsync.

As you can see there’s some weird stuff going on with the PassCmd directive (which tells isync how to get the password when querying the account). In this case it pretty much tells isync to get the password from the environment variables $PW_PRIVATE or $PW_WORK respectively.

An alternative is to provide a program that prints out the password; for example if you are using pass:

IMAPAccount work
PassCmd "pass email/baldrick@bigcompany.com"

If you just store your password separately in GPG encrypted files, you could also simply query GPG for the password:

IMAPAccount private
PassCmd "gpg -o - --for-your-eyes-only -d ~/.passwords/private-mail.gpg 2>/dev/null"

As soon as your command becomes a bit more elaborate, it’s probably a good idea to wrap it in a shell script and put it in ~/bin/:

IMAPAccount private
PassCmd "~/bin/getpassword.sh private-mail"

Depending on what you have available when being asked for the password you could have the script first check if the password file is there and, if that’s not the case, use pinentry to ask the user for the password:

#!/bin/bash
# ~/bin/getpassword.sh

NAME=$1

if [ -e "$HOME/.passwords/$NAME.asc" ]
then
    gpg -o - --for-your-eyes-only -d $HOME/.passwords/$NAME.asc 2> /dev/null
else
    echo -e "SETDESC Please enter password for $NAME\nSETPROMPT Password:\nGETPIN" | \
        pinentry | \
        grep 'D ' | \
        cut -c 3- -
fi

Once you found a nice way to have isync prompt you for your password, you can try and pull (-L) mails from all (-a) channels:

$ mbsync -L -a

If this succeeded you should have all your mail in ~/Mail.

Automatic Mail Fetching

All these password fetching solutions have one major disadvantage: everytime you trigger the email synchronisation (which still has to be done manually), the programm will either ask you for your password or query GPG which might again prompt for your PIN.

Instead I have a shell script running in the background (in a tmux) that synchronises automatically in intervals.

The shell script asks once for all passwords (or get’s them from the corresponding encrypted password files), keeps them in a variable and exports them for mbsync when they are required:

#!/bin/bash
# ~/bin/syncmail.sh

PRIVATEPWD=$(~/bin/getpassword.sh private-mail)
WORKPWD=$(~/bin/getpassword.sh work-mail)

if [ -z "$PRIVATEPWD" && -z "$WORKPWD" ]
then
    echo "No passwords. Exiting."
    exit
fi

while [ true ]
do
    echo -n "~"
    channels=""

    if [ -n "$PRIVATEPWD" ]
    then
        PWD_PRIVATE=$PRIVATEPWD
        export PWD_PRIVATE
        channels+=" private"
    fi
    if [ -n "$WORKPWD" ]
    then
        PWD_WORK=$WORKPWD
        export PWD_WORK
        channels+=" work"
    fi

    mbsync -q $channels
    result=$?

    unset PWD_PRIVATE
    unset PWD_WORK

    echo -en "\x8"
    if [ "$result" = "0" ]
    then
        echo -en "."
    else
        echo -en "!"
    fi

    # wait for 5m
    sleep 5m
done

The script will put out a ~ when it is busy synchronising. The ~ will be replaced with a . when the operation was successful or a ! if there was a problem. mbsync will be rather verbose with the output anyway though, so the ! will likely be drowned in other noise.

Caveats

Keeping your passwords in a long living shell script is not safe. Especially if there may be other people connecting to this machine that have access to a debugger.