Personal Mail Storage

Introduction

The Personal Mail Storage aims to solve problems with reading one or more e-mail and news feed accounts from multiple computers, retaining the read status meta data, and allowing full operation when being off-line.

Note	TODO This is at present only a specification, an implementation will hopefully become available later.

Requirements

Messages MUST NOT under any circumstances get corrupted or lost.
Messages MUST be recoverable even if data structures are corrupted.
Messages MUST be handled in a secure manner (transfer, access control).
Only meta-data is allowed to change (read/delete flags, tags, etc).
Messages, once stored, SHALL NOT change (unless expunged)
Deleted e-mails MAY be expunged at any point.
Synchronization between storages must be fully automatic.
Must be fast and memory efficient with 100K+ messages.

Note	TODO Requirements will change, and more will quite possibly be added to this specification.

Discussion

What's wrong with POP3?

POP3 has the obvious problem of e-mails being "popped" and deleted by one mail user agent, and is therefore not readable by the other.
One can choose to leave messages on the server, but this still leaves problems synchronizing meta data between mail user agents.

What's wrong with IMAP4?

IMAP4 relies on having an active connection with the server, however some mail user agents can cache messages for off-line reading.
IMAP4 is a very complex protocol.

Diagram

               [ POP3 ]  [ IMAP4 ]  [ NNTP ]
                    \        |        /
                    ------------------
   [ RSS/Atom ] --- | fetch messages | --- [ Other ]
                    ------------------
                        /
                       /
           ---------------------
           | spam/virus filter | (optional)
           ---------------------
                   /
                  /
              -------       ---------------
              | PMS | <---> | Synchronize |
              -------       ---------------
               /
              /
      ------------------
      | Serve messages | --- [ Other ]
      ------------------
     /        |      \
[ IMAP4 ]  [ RSS ]  [ POP3 ]

Design

The design is intended to be simple. The only way to communicate with the storage is through these core operations:

Add a message
List all messages
Retrieve a message
Change meta-data
Synchronize storages

Separate tools

The intention is to use a separate tool for fetching e-mail messages, and serving mail to local mail user agents.

All interaction with the storage is done using the storage API functionality. Tools can be written in a variety of languages such as Python scripts, or anything preferred.