This week, I spent a long time mucking about in the mail relays. Freshclam skipped over 8 of the 9 mirrors it knew about, and the remaining one was down, so it spun for an hour trying to fetch new virus definitions. While it was busy with that, clamd woke up, tried to refresh the database, couldn’t acquire the lock (since freshclam had it), and shut itself down. That broke the two clamdscan processes that amavisd-new was using, and 6000 messages piled up in the Postfix incoming queue. I managed to get the whole mess resolved with the help of debian-volatile, which provides rolling stable packages for ClamAV and other frequently-changing projects. I also put in place more comprehensive monitoring for Cacti and Nagios, so next time the queues explode, we’ll know about it sooner.
The upgrade to ClamAV prompted me to go through and fix all of amavisd-new, including making it talk to SpamAssassin again. The upside is that all incoming mail is now thoroughly filtered for spam and viruses before hitting our Exchange servers, which really cuts down on load and junk in people’s inboxes.