This week, I spent a long time mucking about in the mail relays. Freshclam skipped over 8 of the 9 mirrors it knew about, and the remaining one was down, so it spun for an hour trying to fetch new virus definitions. While it was busy with that, clamd woke up, tried to refresh the database, couldn’t acquire the lock (since freshclam had it), and shut itself down. That broke the two clamdscan processes that amavisd-new was using, and 6000 messages piled up in the Postfix incoming queue. I managed to get the whole mess resolved with the help of debian-volatile, which provides rolling stable packages for ClamAV and other frequently-changing projects. I also put in place more comprehensive monitoring for Cacti and Nagios, so next time the queues explode, we’ll know about it sooner.

The upgrade to ClamAV prompted me to go through and fix all of amavisd-new, including making it talk to SpamAssassin again. The upside is that all incoming mail is now thoroughly filtered for spam and viruses before hitting our Exchange servers, which really cuts down on load and junk in people’s inboxes.

This morning was tough! I accidentally knocked Brian’s bottle of Coke right into his laptop keyboard. Again, man, I’m really sorry! Right after that, I spent an hour writing and debugging a memory check plugin for Nagios, then promptly forgot which box I was on and overwrote it with the broken plugin I was testing out from NagiosExchange. Had to rewrite the plugin from memory.

Luckily, work has smoothed out a little since then, and the rest of the day has gone okay. Outside of the office, life is going great! Looking forward to maybe going to the Blue October concert at the Crystal Ballroom tomorrow night!