Being responsible for live production servers can be quite invasive in your daily life, when something goes wrong. But it's part of the job. And just like Cryx I love my job, too. Being responsible for other people being able to do their job also brings a sense of pride with it.

However, sooner ot later something will go wrong. It's just a matter of time and how well you are prepared ;-)

And when something goes wrong it comes in two flavours: either you can fix it remotely or not.

Well, tonight it fell into the second category and again it had to do with building a world on FreeBSD, although this time I had learned my lesson. You see, just recently a security advisory concerning a possible Denial-of-Service vulnerability had been released and I have been dutifully upgrading my affected systems. After doing this on four of six systems something went wrong, though. Upgrading the kernel was sufficient according to the instructions but I thought I had seen some userland updates, as well during cvsup so I decided to build the world, as well. And while it was chomping through make installworld I got a kernel panic...

To make a long story shorter: I had to kiss my wife good night and head back to the offices of that particular client. I was just lucky, it wasn't any of the machines hosted in Cologne!

Upon arrival the system was pretty much hosed with a broken world and missing essentials such as make, install or even /usr/bin/login!

In the end I had to scp those binaries from another machine after booting into single user mode and bringin up the network manually and then attempted to rebuild the world. Kernel panic again! Recopy of /usr/bin/*, now booting into normal mode without attempting further builds of the world. Phew, all services (mainly exim, Cyrus Imap and Apache) are running again!

There's another thing I learned: take things slowly. The machine is now again operational, though still in a dubious state: it can't build any world at the moment... But instead of pulling an all-nighter to get that fixed, too right away I simply initiated a download of the 5.2.1 ISO-Images. Over the weekend I will reinstall the world from CD and then run the upgrade again. Good night.