cyrus

Ouch! Just got a call from a client wondering that he hadn’t got any emails for a few days and if I could look into the matter. And sure enough: his server side filter was set to sort out any email with a recipient numerically greater than 7! Which, apparently, was true for every email he received. What had happened? Upgrading to avelsieve-1.9.4 had introduced a bug that would set the header to the default value (To:) when saving changes to a rule. So after modifying any rule that used, say, the X-Spam-Score: header would then use the “To:”-header instead upon saving (and spew out an error message Warning: Cannot modify header information - headers already sent.)

Having not noticed either of that, his filter now happily shoved everything into quarantine... (Where they luckily were easily retrieved)

The solution (found here)? Remove the print_r($c); statement in line 53 of the avelsieve/include/process_user_input.inc.php.

Note to myself: more testing!!

Ah, the unsung heroes who do their job so well, that nobody notices... In this case I'm speaking (singing?!) of Squirrelmail (and its multitude of (largely) useful plugins). I've been using Squirrelmail (oops, sorry, I keep forgetting)... we have been using Squirrelmail now for well over three years and pretty much from the start it was something simple and useful and just worked[tm].

It's been a selling-point for quite a few of my hosting customers over time (well, along with the virtues of IMAP in general and Cyrus in particular) but only today did I finally venture into the plugins to enhance the experience. In the end I installed just three: Quota support, New User Wizard and (most importantly) the insanely useful avelsieve.

Now all our customers can enjoy serverside filtering with (relatively) easily clickable rules. The screenshot shows the perhaps most useful of all: discarding mail according to its X-Spam-Score (provided by SpamAssassin).

I spent well over five hours exploring and testing various other plugins and the best usages for the ones I decided to stick with, but it was a thoroughly enjoyable geek experience... you know... it works!

Update: Actually the rule shown in screenshot below is incorrect - it's comparing the X-Spam-Status header, not X-Spam-Score. That header is conspicioulsy absent from the default setup of avelsieve. However, in order to add it, all it takes is editing avelsieve's config.php (search for '$header').

Part of the process when migrating Cyrus IMAP accounts is running the reconstruct tool on the destination machine to add the mail to the database. Using the -r -f flags ensures, that the command will pick up any folders and subfolders.

Unless - as a client and I found out this morning - a folder contains no emails but just subfolders. In that case reconstruct ignores that folder (as well as its subfolders). To enforce its inclusion simply copy (any) cyrus.header file from another folder into it and re-run reconstruct (an empty file created with touch won’t work - it needs to be of this particular binary format that Cyrus expects).

When it rains, it pours, I guess. So just after spending a weekend procrastinating work and finally getting back to it, the drive of the machine hosting this site broke! After getting home at 2 a.m. I noticed that it’s not reachable anymore and immeditately issued a reboot request through HostEurope’s management interface. And what do you know? At 02:20 I get a call from the technician on duty telling me that the machine was frozen and was now in the process of checking the drive and that he would get back when he had a prompt. At 02:30 I noticed that the machine could be pinged again. At 02:32 I got another call and we went through the list of files, that fsck had reported broken - phew! nothing important, just a couple of log files… To make a long story short: today co-admin Cryx and I spent a full ten hours migrating some 106 Cyrus based IMAP accounts weighing in at 8Gb that still had been residing there to the new mail server. (We had been meaning to do this for ages but their was no real pressing need - until yesterday!)

Despite being somewhat of a seasoned admin meanwhile, I still have a lot of respect for any operations involving changes in IMAP and DNS (we had to change MX-records for a few dozen domains)- it’s complicated stuff and people can get really upset if it doesn’t work… We modified my existing script to handle the new task and actually managed to pull off the whole stunt between 22:00 and midnight after spending the day preparing everything - and as far we can tell, no emails have been lost at all - due to our ever-so-nifty migration strategy ;-).

So, if you happen to be one of my customers: sorry for not having given you prior notice, but with the damaged drive we just didn’t want to take any risk! I’ll be sending out a detailed explanation of what happened during the day. Meanwhile most providers have already picked up the new IP-Address for mail.tomster.org and after sending a gazillion test mails from each and every server we’ve got an account on I’m pretty confident, that come this morning nobody will know that anything has happened at all - knock on wood! All in all, though, I’m quite pleased how things went down and how we handled the situation. Also, I couldn’t help but notice how much fun it was using TextMate and Python in handling this real-world problem…

And to Cryx: you’re one hell of a guy to work with, thanks for everything!!

One of the great advantages of IMAP is, that the server can do all of the sorting and filtering instead of the client. One of the greatest disadvantages of IMAP is, that the server can do all of the sorting and filtering instead of the client. (Yes, that was a copy & paste job).

In practice this means, that while it's great to use sieve to let the server sort all incoming mail into (IMAP-)folders (i.e. for mailing-lists) it also means, that it's (currently!) still pretty tedious to maintain these settings. And nowadays one can plain forget about trying to tame spam using regex.

For me, this meant, that my mail client's spam filter only eliminated spam in my inbox, but none of them in my many subfolders (because sieve had already delivered them there, the sneaky bastard!). This, essentially, rendered some lists (i.e. ds@ccc.de) unusable. However, deactivating the sieve filter wasn't an option, either. My inbox would simply be swamped with list mails if I would have to read them via webmail for a while.

The solution to this dilemma, was rather easy in the end and one of those things that I had been procrastinating for months and then just took a mere minutes. The idea is to let Spamassassin tag each message with a Spam-Score header and then let sieve filter out messages according to that score. The problem was, though, that for some unfathomable reason sieves numerical comparator cannot deal with negative values, so lots of list traffic with highly negative spam scores would falsely be locked away without trial. Outrage!

Well, all you really got to do is check for the occurrence of the '-' character in advance and then use the 'discard' action, if appropriate. So here's the beginning of my personal sieve script that's now keeping my inbox nice and clean. Whereas previously I would have to deal with ca. 200+ spam mails per day in my junk box and mailinglists (plus ca. 10 to 20 that actually ended up in my inbox) I now get about 10 to 15 in the junkbox and just a couple in the inbox and lists. Hooray! I feel like I've rediscovered email again. It's become really useable now!

if not header :contains "X-Spam-Score" "-" {

	if header :value "ge" :comparator "i;ascii-numeric" \
			["X-Spam-Score"] ["10"]
	        {
	                discard;
	                stop;
	        }
	if header :value "ge" :comparator "i;ascii-numeric" \
			["X-Spam-Score"] ["6"]
	        {
	                fileinto "INBOX.Junk";
	                stop;
	        }
}