When it rains, it pours, I guess. So just after spending a weekend procrastinating work and finally getting back to it, the drive of the machine hosting this site broke! After getting home at 2 a.m. I noticed that it’s not reachable anymore and immeditately issued a reboot request through HostEurope’s management interface. And what do you know? At 02:20 I get a call from the technician on duty telling me that the machine was frozen and was now in the process of checking the drive and that he would get back when he had a prompt. At 02:30 I noticed that the machine could be pinged again. At 02:32 I got another call and we went through the list of files, that fsck had reported broken - phew! nothing important, just a couple of log files… To make a long story short: today co-admin Cryx and I spent a full ten hours migrating some 106 Cyrus based IMAP accounts weighing in at 8Gb that still had been residing there to the new mail server. (We had been meaning to do this for ages but their was no real pressing need - until yesterday!)

Despite being somewhat of a seasoned admin meanwhile, I still have a lot of respect for any operations involving changes in IMAP and DNS (we had to change MX-records for a few dozen domains)- it’s complicated stuff and people can get really upset if it doesn’t work… We modified my existing script to handle the new task and actually managed to pull off the whole stunt between 22:00 and midnight after spending the day preparing everything - and as far we can tell, no emails have been lost at all - due to our ever-so-nifty migration strategy ;-).

So, if you happen to be one of my customers: sorry for not having given you prior notice, but with the damaged drive we just didn’t want to take any risk! I’ll be sending out a detailed explanation of what happened during the day. Meanwhile most providers have already picked up the new IP-Address for mail.tomster.org and after sending a gazillion test mails from each and every server we’ve got an account on I’m pretty confident, that come this morning nobody will know that anything has happened at all - knock on wood! All in all, though, I’m quite pleased how things went down and how we handled the situation. Also, I couldn’t help but notice how much fun it was using TextMate and Python in handling this real-world problem…

And to Cryx: you’re one hell of a guy to work with, thanks for everything!!