Had a long day yesterday again. Then a PTA meeting at Billabong. Over at 9:30 p.m. (Actually quite nice seeing everybody, I think, we're doing a terrific job keeping our kita up and running...)

Finally Feierabend. Head over to Carsten, where he and Jörg are watching the final three hours of 24 on DVD. Great!

As we were watching, I got a call from a client with a mail problem. Okay, since I've been doing some springcleaning anyway lately, I figured I'll build another world and update everything while I'm at it. Big mistake! So with my powerbook on the lap I started out with the usual procedure. make buildworld; make buildkernel; make installkernel; make installworld;. Been there, done that. And afterall, FreeBSD is rocksolid, right? (Seven of the nine world's most reliably running servers run it)

So I started building the world (takes about 45 minutes on that machine) and started off home. Got there at one o'clock. One last beer, while building and installing the kernel. Takes about ten minutes. Then install the world, reboot and off to bed...

Wrong! If I had read the freaking README I would have know that with this update I absolutely must boot the new kernel before installing the world or else I would be "in dead water". Which is exactly where I found myself when the install process of the world stopped with an error... I was stuck with a kernel not matching a world, that was part 5.1 and part 5.2! Even pwd coredumped! I was lucky I still had access at all! So here I was, dead tired at 1 a.m. with a hosed production machine of a company with 20 users who would show up for work in a couple of hours...

First I shut down Exim, since it was happily accepting mails from the outside world which it then instantly lost by segfaulting. Luckily Erdgeist was still up. He walked me through some manouvers to first copy some 5.1 binaries from another machine (/bin, /sbin etc.) refetching the sources and then rebuilding the tools that make buildworld itself depends on manually such as make, install, ls and cat. We spent almost three hours on the phone until finally I had a running buildworld process again! One hour later I had an installed world with non-segfaulting tools. Phew!

Of course, Exim still accepted mails at the door, meticulously signing the receipts and then, the minute the messenger had left he ran down into the basement where he laughed maniacally and burned the received mail in an open fire made of unused furniture. Then he would calmly go back upstairs and answer any questions about messages with "No, Sir, nothing Sir".

Piece of cake now. Updated the ports collection, rebuilt exim (thus updating it from 4.2.2 to 4.3.0) and mails sent from outside showed up in peoples INBOXes again. Three hours left, till the first people would show up for work. Apart from various lessons learned and the good feeling of having toughed something out I really felt (once again!) good about being a geek end experiencing this kind of solidarity. Thanks again, Erdgeist!