Offline mailing method using rImap by Rimon Barr v1.2.2, 18 Jan 2002 --------------------------------------------------------------------- Description: A mechanism that allows one to perform all mail operations on an offline replica and later synchronise changes. Table of Contents: 1. About this document 1.1 Purpose and scope 1.2 Comments and feedback 1.3 Copyrights and trademarks 1.4 Acknowledgements and thanks 1.5 Disclaimer 1.6 Activity log 2. Overview 2.1 Original configuration 2.2 Problem description 2.3 Solution description 3. Setting up rImap 3.1 Installation 3.2 Configuration 3.3 Use 3.4 The synchronization algorithm 3.5 Upgrading rImap 3.6 Further information 3.7 CAUTION 4. Setting up an outgoing mail server 4.1 Sendmail 4.1.1 Installation 4.1.2 Configuration 4.1.3 Use 4.1.4 Further information 4.2 Qmail 4.2.1 Installation 4.2.2 Configuration 4.2.3 Use 4.2.4 Further information 4.3 Other SMTP servers 5. Setting up your mail user agent 5.1 Pine 5.2 Other agents 6. Automation 6.1 Crontab 6.2 Dialup scripts 6.3 Other 7. Concluding remarks --------------------------------------------------------------------- 1. - About this document 1.1 - Purpose and scope This document was written with the express purpose of explaining how to use rImap, and sendmail, to easily setup offline mail operation. Basically, the author wrote a mail replication system, found it to be immensely useful and wanted to tell the world about it too. The newest version of this document is available at: http://rimap.sourceforge.net/rimap-offline-mailing.txt This document is not meant to solve everyone's email problems. It is merely a description of a setup that the author finds to be extremely useful. Hopefully, you will too. You are welcome to adapt these ideas to your liking, and I'd love to hear about what you come up with. 1.2 - Comments and feedback Comments on this HOWTO, or about rImap, are encouraged and may be directed to the author, Rimon Barr . If you find this script useful, if you think of a neat feature to be added, or if you spot a mistake, then please let me know. 1.3 - Copyrights and trademarks (c) 2001 Rimon Barr This manual may be reproduced in whole, or in part, only without fee, and subject to the following restrictions: * The copyright notice above and this permission notice must be preserved complete on all complete or partial copies * Any translation or derived work must be approved by the author in writing before distribution. * If you distribute this work in part, instructions for obtaining the complete version of this document must be included. * Small portions may be reproduced as illustrations for reviews or quotes in other works without this permission notice if proper citation is given. Exceptions to these rules may be granted for academic purposes: write to the author and ask. These restrictions are here to protect authors, not to restrict learning and education. Any trademarks mentioned in this document are the property of their respective trademark owners. 1.4 - Acknowledgements and thanks I'd like to thank the Python guys for an excellent scripting language that greatly simplified my work. Plug: Go learn Python at http://www.python.org. 1.5 - Disclaimer The author is not responsible in any way for any damage of any kind incurred directly or indirectly by following the suggestions and examples in this text, or by using the rImap software. This document represents the best of his knowledge and experience, but you are warned to make backups and take other precautions to prevent possible data loss and other damage. In short, I am not responsible or accountable for your actions. 1.6 - Activity log 03/24/01 - v1.0 - Initial writeup 29/10/01 - v1.1 - Feature updates, Qmail 27/11/01 - v1.2 - Synchronisation algorithm, minor corrections 16/01/02 - v1.2.1 - minor corrections, changed document url 18/01/02 - v1.2.2 - rimap upgrade section --------------------------------------------------------------------- 2. - Overview 2.1 - Original configuration As of the time of this writing, I receive my email at the Computer Science Department of Cornell University. Users can receive their email in one of two locations. The first is a standard Unix mailbox in a mail spool directory. The second is within a department IMAP server. I use Pine, which can read mail from both of these mail sources. I have chosen to drop my mail in the IMAP server for a number of reasons, which are outside the scope of this document. I have a laptop and would like to read and process my mail when I am offline. When I am at the department, I would like to use my desktop machine, and check mail from there. I want all my mail folders to be available at both locations and be automatically synchronised. One solution is to find an email client which supports disconnected IMAP, but I don't want to switch clients. Moreover, at the time of this writing there is a dearth of good disconnected IMAP Linux mail user agents. I have not found anything similar to Outlook Express or Eudora. The closest is a program called Mulberry, but it is akward to use and is not free. Another solution is to replicate the email in two places, and that is the approach I have chosen. I store messages both on an IMAP server, and they are also stored in a replica on my laptop to be used offline. The problem arises of keeping the replicas synchronized. Before writing rImap, I would connect and process email online against the IMAP server. Today, I process email against an offline replica, and quickly syncronise later via dialup. 2.2 - Problem description Define a mail replica to be a set of mail folders, which each contain messages. Needed, a program that is capable of synchronising the mail messages between two mail replicas. This is rImap. Needed, a mechanism for deferring sent email. This is built into sendmail, Qmail or some other mail server. 2.3 - Solution description I keep an offline replica of my mail on my laptop. That means that for every IMAP folder on the server, I keep a mailbox file on my laptop in ~/mail/sync/. rImap keeps these synchronised with the folders on the IMAP server. I configured Pine to run in two modes. On my desktop, connected machine, I run Pine pointing at the IMAP server. On my laptop, which is usually disconnected, I run Pine pointing at ~/mail/sync/. Likewise, I have two modes for sending emails. On my desktop, connected machine, I tell pine to use the department SMTP server. On my laptop, I tell Pine to use a locally running SMTP server, which defers delivery in a mail queue. Sendmail, Qmail and other mail servers automatically take care of delivery. --------------------------------------------------------------------- 3. - Setting up rImap 3.1 - Installation rImap installation is simple. Take the rimap file, and put it anywhere, preferrably in your path. Make sure it is set to be executable. Ensure that Python version 2.0 or above is installed on your system, and that the script interpreter is at: /usr/bin/python2. Running "rimap -v" should emit a single line with version information. Run "rimap -h" for command-line help. 3.2 - Configuration rImap reads needs to find a .rimap configuration file in your home user directory. The configuration file basically simplified the command-line parameters. The .rimap file defines two objects used by the replication engine: replicas and folder sets. Replicas are mail locations, and rImap currently understands how to deal with two types of replicas: mailbox files and imap. (If you store your mail otherwise, then consider extending rImap, or contact the author. The new replicas types are easily plugged into the system, and Python provides excellent mail support libraries.) Replicas are defined by a url registered with the engine. A mailbox url takes the following form: mailbox:// An imap url takes the following form: imap://[username[:password]@][server[:port][/path] username defaults to the $USER environment variable. password is prompted, if not provided server defaults to 127.0.0.1 (localhost) port defaults to 143 (standard IMAP port) Folder sets are named lists of folders to replicate. For convenience, the .rimap file is really Python source code, which defines two variables: replica and folderset. Note the commas between replica and folderset entries. These are really just Python associative arrays. A sample .rimap file: (replace 'xxx' below with your IMAP password) # Sample .rimap file replica={ 'cucs': 'imap://barr:xxx@imapserver.cs.cornell.edu/', 'local': 'mailbox://mail/sync' } folderset={ 'quick': ['INBOX', 'Drafts'], 'important': ['INBOX', 'Drafts', 'list', 'Sent Items', 'later', 'store'], 'all': ['INBOX', 'Drafts', 'list', 'Sent Items', 'later', 'store', 'Bin', 'bag', 'jokes', 'music'], 'test': ['rim'] } Because this is really Python code, we don't have to repeat folder names, and can rewrite the folderset section using some temporary variables, as below. The result is identical, producing the same folderset definition. quick=['INBOX', 'Drafts'] important=quick+['list', 'Sent Items', 'later', 'store'] all=important+['Bin', 'bag', 'jokes', 'music'] folderset={ 'quick': quick, 'important': important, 'all': all, 'test': ['rim'] } The configuration file exists at ~/.rimap. Make sure that the $HOME environment variable is defined. 3.3 - Use Once you have made your .rimap file, you can run rimap as follows: rimap If it doesn't work, and you get an error message, then you have set something up incorrectly. There are some error messages, but not much support here... Try run: rimap --debug That will print out an exception stack. If you read through it, you should be able to figure out what property the system expects and you have not satisfied. If you are having troubles, please email me and I will make the error messages more instructive, so that others will be as confused by common mistakes. Time for an example. To synchronise the 'important' set of folders between the 'cucs' and 'local' replicas (as defined in the .rimap sample configuration file above), we would execute: rimap local cucs important You can also limit the size of messages that are transferred. This is really useful when you are calling in long-distance, and really don't need to see a joke email message with a 1.5Mb attachment. Only transfer messages smaller than 200k: rimap -s=200 local cucs important One can also perform uni-directional transfers, if one is interested only in downloading or uploading messages. The -1 switch only transfers messages into the first replica, ie. replica1 <- replica2. The -2 switch specifies the opposite direction. Restricting message transfers only from cucs to local is achieved by the following command: rimap -1 local cucs important 3.4 - The synchronisation algorithm The synchronizer brings the folders on two replicas into synchronization in a very simple manner. It reads the index of messages in each folder, and determines the differences. It then performs the required actions to bring them into synchronization. There are only three actions: deleting a messaging, copying a message and updating its flags. Given replica A and B, and some message which exists on A but is missing on B. This can be explained in one of two ways. Either the message is a new message on replica A, or it was deleted on replica B. In the former case, we need to copy the message over to replica B, and in the latter, we need to delete the message from replica A. Discerning between these two alternative actions for synchronization requires a knowledge of the prior state of the folders. rImap keeps a list of the messages in each folder in a state file. It is created automatically, and kept up-to-date by the system. The state files are named .rimap--, where and are replaced with the replica names specified on the command-line, and defined in the configuration file. The replica names are sorted in lexicographically, so command line order is irrelevant. For example, the state file used for the command: rimap local cucs important is .rimap-cucs-local. To see the contents of a state file, simply use: rimap --dump local cucs This will output the message identifiers in each folder. These message identifiers are globally unique values extracted from the Message-ID message header. They are generated in rare cases when mail clients do not produce them. (I have seen this behaviour in Outlook Express 5.5!) You can restrict output to only one folder, and delete state using the --delete switch. The general syntax is: rimap <--dump|--delete> replica1 replica2 [folder] The only time that rImap will delete a message is when it sees it in the prior state of a folder, but not in the folder currently. This means that the absence of a state file is not harmful. It will simply merge the contents of the two replicas with message copies. But if you want your deletes to be propagated then leave the state file alone. Lastly, rImap also checks that message flags are identical. rImap applies a simplified algorithm for merging conflicts. It simply takes the union of all possible flags. If a message is 'Answered' on one replica, the 'Answered' flag is set; if the 'Seen' (ie. not new) on one replica then the 'Seen' flag is set; etc. During regular operation you will not have to bother with any of this... This entire section is provided for your interest. It works well in practice. 3.5 - Upgrading rImap The rImap state file format may change with each rImap release. As such, it is advisable to delete the state files when upgrading to a new version of rImap. In future, the format may be version-proofed, but this is not implemented as yet. The rImap configuration file is not expected to change much, and should be forward compatible. New replica types may be added with time, which will not affecting current users. A release announcement will be made if forward compatibility of the configuration file is broken. 3.6 - Further information For more detail, please read the documentation at: http://www.cs.cornell.edu/barr/repository/rimap/ To understand the rImap algorithm, also read the documentation, or go straight to the code (primarily the synchFolder function). 3.7 - CAUTION I recommend backing up folders before rImap is used for the first time. I'm not responsible if something goes wrong with your specific setup, and you lose all your email. All I can tell you is that it works for me. I have been using the program without any message loss for over six months. If something does go wrong, please let me know what happened so that I can try prevent it from happening to others. Please provide as much pertinent detail as possible. --------------------------------------------------------------------- 4. - Setting up an outgoing mail server 4.1 - Sendmail 4.1.1 - Installation For details refer to: Mail-Administrator-HOWTO Mail-User-HOWTO http://www.sendmail.org I run RedHat Linux, and just installed the sendmail rpm. Make sure that sendmail is running at your runlevel. 4.1.2 - Configuration I use the default RedHat configuration file for sendmail, with two minor modifications. Make the following changes to: /etc/sendmail.cf Search for 'DeliveryMode', and make the line read: O DeliveryMode=queue What this does is force sendmail to simply queue all SMTP requests, rather than trying to send them. I prefer this, because it makes Pine work faster, and most of the time I really don't care to send the messages *immediately* even if I am online. I have a cron job that takes care of pushing mail every ten minutes. Next, search for 'queuewarn', and make the line read: O Timeout.queuewarn=2d (The other queuewarn lines are commented in my file.) What this does is suppress warning messages for two days. Usually this value is set to 5 hours. This means that if you leave a message in your queue for more than 5 hours, sendmail will send you a warning message telling you about this. I find this annoying, because I often write messages at night, and only go online in the morning to send them. For me, 2 days is a more reasonable value. That's it. Now just make sure that sendmail is running. (You can try a telnet to port 25 on localhost to make sure.) 4.1.3 - Use When you are online, you need to initiate mail queue processing as follows: sendmail -q -v The -q switch tells sendmail to process the queue. The -v switch is just for verbose output. Your choice. 4.1.4 - Further information You don't really need to know more about sendmail to use it, but you are welcome to go and learn more. 4.2 - Qmail 4.2.1 - Installation For installation details refer to: http://www.qmail.org You can find HOWTOs and FAQs there. Currently, I have installed the qmail-1.03-102memphis rpm. 4.2.2 - Configuration I basically use the default configuration of qmail. In the /var/qmail/control directory, I have: defaultdomain: u.cs.cornell.edu locals: rimpad.u.cs.cornell.edu me: rimpad.u.cs.cornell.edu plusdomain: cornell.edu rcpthosts: rimpad.u.cs.cornell.edu and the rc file looks like: exec env - PATH="/var/qmail/bin:$PATH" \ qmail-start ./Mailbox splogger qmail No other modifications are necessary... and most of these few are likely unneeded. That's all you need in order to send mail using the mail program from the command-line. Qmail will not start up an SMTP server. 4.2.3 - Use If you would like to view the outgoing mail queue, run /var/qmail/bin/qmail-qread as root. When you are online, you can restart qmail processing via the following commands: killall qmail-remote killall -ALRM qmail-send 4.2.4 - Further information You don't really need to know more about qmail to use it, but you are welcome to go and learn more. I have recently switched from sendmail to qmail on all my machines, because I find it more flexible and easier to understand. 4.3 - Other SMTP servers It's also possible to perform the same thing with another SMTP server. I've used qmail before for other tasks and I really like it. It should be up to the task. If someone wants to document the above process for other SMTP servers, please send it to me. Try to mimick the format above, and I'll just include it in this document. --------------------------------------------------------------------- 5. Setting up your mail user agent 5.1 - Pine Assuming the folders of the local replica are stored in ~/mail/sync, that the inbox folder is INBOX, the fcc folder is 'Sent Items', the save folder is 'Bin', we setup Pine as follows: Set the following options on the Pine Setup Configuration screen: inbox-path = mail/sync/INBOX default-fcc = Sent Items default-saved-msg-folder = Bin smtp-server = 127.0.0.1 If you are not running an SMTP server, and would like to use the command-line mail program as an interface to your outgoing mail queue, then leave: smtp-server = Set a folder collection list to point to: mail/sync/ 5.2 - Other mail user agents It is not difficult to get most agents to use the replica folders and the local SMTP server when sending email. However, it someone has experience with other user agent, please consider writing a short explanation of how to get them to work on a local replica. --------------------------------------------------------------------- 6. - Automation 6.1 - Crontab Cron is useful for to try synchronise every once in a while. I try to perform a quick synchronise every 10 minutes: rimap local cucs quick And a full synchronise (all folders) every two hours: rimap local cucs all It is helpful to pipe output to null, and perhaps to only run when you are connected to the network (as opposed to via modem), as below: tcsh -c "isroute eth0 rimap local cucs quick >& /dev/null" isroute is the following script: #! /bin/tcsh # syntax: isroute route cmd # eg. isroute eth0 ls; only runs ls if eth0 appears in route table set route = `cat /proc/net/route | grep $1` if ("$route." != ".") then shift exec $* endif 6.2 - Dialup scripts Often, I dialup from home, and I would like to outgoing mail queue to get processed as soon as I connect. The following lines in /etc/ppp/ip-up.local do the trick: For sendmail: /usr/sbin/sendmail -q For qmail: killall qmail-remote killall -ALRM qmail-send 6.3 - Other While I'm describing tips and tricks, below is the list of the aliases that I use regularly. alias syncmailquick '~/rimap/rimap -s=200 local cucs quick' alias syncmailquickbig '~/rimap/rimap local cucs quick' alias syncmailimportant '~/rimap/rimap local cucs important' alias syncmail '~/rimap/rimap local cucs all' alias sm 'syncmail' alias smi 'syncmailimportant' alias smq 'syncmailquick' alias smqb 'syncmailquickbig' alias smqh 'smq; hangup' alias smqbh 'smqb; hangup' alias mailq 'su -- /var/qmail/bin/qmail-qread' alias watchmailq 'watch tcsh -c mailq' And these are really useful: alias cit 'pppd call cit' (this is one of my alias for dialup) alias hangup 'killall pppd' alias dialmail 'cit & sleep 60; smqh' alias dm 'dialmail' --------------------------------------------------------------------- 7. - Concluding remarks As stated in the introduction, the method I have just described is how I manage my email, and use rimap. If you find it useful, I am glad. If you would like to contribute a suggestion, an idea, a fix, a comment or just a thank you... please drop me a line at . All the best, Rimon. ---------------------------------------------------------------------