My Saturday in Sendmail Hell

Sendmail problems are like Glenn Close in Fatal Attraction. You have to kill them a couple of times just to make sure they're really dead.

I returned home after the jaunty 1,104-mile drive from Dallas to find this server with an 100 percent full disk, which was 70 percent more than the day before. The culprit was Sendmail, which bounced the same e-mail back and forth until it filled the disk.

I burned a Saturday at the keyboard as feeling slowly returned to my butt cheeks -- one of these days, I'm afraid I will break one by driving for hundreds of miles.

Solving a problem with no available disk space takes hours. Commands can take 10 minutes or more to execute, because the CPU's throttled by the process filling the disk. I began by killing the most likely cause, Sendmail: /etc/rc.d/init.d/sendmail stop.

Although I've been running a Linux box for years, I know little about Sendmail, which I regard charitably as the worst software that will ever be written in any language, open source or commercial, under any programming methodology on any operating system. Unlike Apache, MySQL, and other programs on which I rely, Sendmail can't easily be reconfigured. The software's configuration file is so complex that it requires a configuration file of its own!

When Sendmail sends a message, the e-mail ends up in two files in /var/spool/clientmqueue. I deleted each 100+ megabyte e-mail as it showed up, rebooting Sendmail and the server several times, but it took 24 hours to stop coming back.

Like all Sendmail problems that I fix, I was left with absolutely no knowledge of what I did that worked. My best guess is that I could prevent a recurrence if there's a way to tell Sendmail to reject all incoming or outgoing e-mail larger than a set size, such as 10 megabytes.

Linux · 2005/07/31 · 5 COMMENTS · Link

Comments

I'd recommend taking the time to learn Postfix or Qmail. The time invested in learning to administer the new MTA is paid back several times over by the fact that once you've finished, you can say goodbye to Sendmail forever.

(Personally I'd recommend Postfix, because Dan Bernstein scares me.)

As an aside, from a discussion in Jamie Zawinski's blog:

I'm kinda surprised that after all this time, sendmail's config is still based on ancient hieroglyphs. (To enable sender-validation in sendmail, enable this option: bird, squiggley line, sideways man, fish)

#1 · Charles Miller · 2005/07/31

I would second Postfix.

It is a piece of cake to set up and administer. The config file is actually human readable and there is an excellent support email list.

www.postfix.org

The author works at IBM and this project has IBM's full support so it's not going away at any time.

Lot's of great third-party software too (anti-SPAM, etc...)

#2 · DaveH · 2005/08/01

I've also had this problem of Sendmail filling up the disk with files in /var/spool/clientmqueue, seems like a few others too, judging by a Google search.

But no-one ever came up with an explanation of how it started.

#3 · Mike W · 2005/08/16

In my case, I had an exceptionally large error log being e-mailed to me. The file was too large, so it bounced back, and the bounce was even larger. Repeat until the disk is full.

#4 · Rogers Cadenhead · 2005/08/17

Hey just googled my problem with my sendmail and guess what. no idea how it happens but i know it builds up in my disk after a mail is sent from my sendmail... which came pre-installed on my OS and i kinda need for a script.

#5 · Nathan Howard · 2005/11/17