I returned from a trip out of town Monday to crashing web servers that ate my lunch all week long. For several days, I used the top command in Linux and watched helplessly as two servers ground to a halt with load averages higher than 100.

Top reports the processes that are taking up the most CPU, memory and time. On the server running Workbench, the culprit was always httpd, the Apache web server. This didn't make sense, because Apache serves web pages, images, and other files with incredible efficiency. You have to hose things pretty badly to make Apache suck.

If you know the process ID of a server hog, Apache can tell you what that process is doing in its server status report, a feature that requires the mod_status module. The report for Apache's web site shows what they look like.

Using this report, I found the culprit: A PHP script I wrote to receive trackback pings was loading the originating site before accepting the ping, which helps ensure it's legit:

// make sure the trackback ping's URL links back to us
$handle = fopen($url, "r");
$tb_page = '';
while (!feof($handle)) {
$tb_page .= fread($handle, 8192);
}
fclose($handle);
$pos = strpos($tb_page, "http://www.cadenhead.org/workbench");
if ($pos === false) {
$error_code = 1;
send_response(1, "No link found to this site.");
exit;
}

Most trackback pings are not legit -- I've received 600 from spammers in just the past three hours. Each ping required Apache to check the spammer's site, download a page if it existed, and look for a link to Workbench. A single process performing this task could occupy more than 50 percent of the CPU and run for a minute or more.

I'm surprised Apache ran at all after I added trackback a couple months ago. I was beginning to think the web server software was idiot-proof, but I've proven otherwise.

-- Rogers Cadenhead

Comments

I was beginning to think the web server software was idiot-proof, but I've proven otherwise.

I'm confused. Why is it Apache's fault that you wrote code causing it to crash? A search through all the source of a page would be extremely processor intensive with hundreds of requests an hour.

Can I blame IIS if I wrote an infinite loop in VBScript?


 

Who said it was Apache's fault?


 

Ah, my apologies! I mis-read and took the quote (that I quoted) as meaning it was Apache's fault. Silly me glossed over:

I found the culprit: A PHP script

Keep up the good work!


 

> I was beginning to think the web server software was idiot-proof, but I've proven otherwise.

It's like that saying in Poker... If you've been playing for a while and you haven't spotted the chump, then you are the chump.


 

Just accept the trackbacks (returning a "202 Accepted") and queue them into a file or database table, then separately via a cronjob or some other process, run the queue and purge the 99.999957281% of the crap that is spam from the spammers.

I changed my trackback script to be a PHP script which always replies "202 Accepted" but files all the data way in files to be processed later. Out of maybe 5000 trackbacks I received last year not a single one was valid.


 

I'm having trouble figuring out how the script you posted was using up 50% of the CPU for up to a minute. It shouldn't be nearly that bad. What kind of hardware are we dealing with? The download of the page is essentially computationally free, and scanning one string for a single occurrence of another string is very efficient. I'm just not seeing why it would be running slowly.


 

This server is an Intel 2.4 Ghz P4 Celeron with 1 gig of DDR 266 memory running Redhat Enterprise Linux 3.


 

I would bet it's the DNS lookup on the fopen of the URL. DNS is typically thread safe but not reentrant. Under a threaded server like Apache/2 each DNS call will block until the hostname's been resolved, if you get a slew of trackbacks at the same time the following requests will stall waiting for the hostname to resolve. Assuming everything is working perfectly this shouldn't be a problem, but if you're getting slammed by trackbacks which list fake web sites for the trackback URL, your DNS may hang just long enough on each request to cause the server to fall over.

This is pure speculation on my part, I haven't stared at apache or resolver source code in years.


 

It it helps, I just crashed a server too by simply requesting 100 or so PHP files in just a few minutes via AJAX. Seems each of th PHP files had loops which ran a thousand+ or so times, causing the crash by overloading the machine?


 

It's not apache's fault the server crashed? How preposterous! The software shouldn't crash. It should issue some form of useful error message then carry on.


 

Post #9. I too just carshed a W2K3 server by running a php script that processes a lot of data. Crashdump shows that php is at fault. This is is rather annoying.


 

The problem with the example code is that it does not check for errors returned by fopen().

If fopen() returns an invalid handle, then the while loop will run forever, since feof($handle) will always be false.


 

Thanks really practical. Will share website with my good friends.


 

I benefit from browsing your web sites. Cheers!
fotbollskl?der www.afterglowvp.com


 

Wow, gorgeous website. Thnx ...
fotbollstr?jor barn www.afterglowvp.com


 

I like perusing your web sites. Regards!
fotballdrakter barn www.travelstobhutan.com


 

Maintain the awesome job !! Lovin' it!


 

Thanks, this website is extremely beneficial.
maglie calcio www.synchronizationofus.com


 

Amazing web page you have in here.
maglie calcio www.synchronizationofus.com


 

Thanks extremely beneficial. Will certainly share website with my friends.
fotballdrakter barn www.nottheitgirls.com


 

Appreciate it for sharing your superb web page.
maglie calcio a poco prezzo www.glivytech.com


 

Great web-site you've gotten going here.
fotballdrakter barn www.nottheitgirls.com


 

Amazing....such a important online site.
maglie calcio poco prezzo www.glivytech.com


 

Great website! It looks very expert! Keep up the good job!
billige fodboldtr?jer www.zamanthajoeh.com


 

Many thanks! This is an outstanding website!
billiga fotbollstr?jor www.tampabaygoodliving.com


 

thank so a lot for your web site it assists a whole lot.
fotbollstr?jor med eget tryck www.tampabaygoodliving.com


 

Sustain the spectacular job !! Lovin' it!


 

Great web website! It looks very expert! Sustain the helpful job!
fodboldtr?jer med tryk www.crimsontee.com


 

Greetings, good web site you've got here.
fodboldtr?jer b?rn www.crimsontee.com


 

Just wanted to point out I am just relieved I came on the web page.|


 

I appreciate checking your internet site. Thanks for your time!
magliette calcio www.doralcentermall.com


 

Thanks a ton! It is an awesome website!
maglie calcio bambino www.doralcentermall.com


 

Great internet site! It looks extremely good! Maintain the excellent job!
Magliette Calcio Poco Prezzo www.wendtsporkpalace.com


 

You're a really beneficial website; could not make it without ya!
Magliette Calcio Poco Prezzo www.wendtsporkpalace.com


 

Rather helpful, looking forth to coming back.


 

Incredibly individual pleasant site. Enormous information offered on few clicks on.
maglie calcio www.haeroo.com


 

Many thanks very practical. Will certainly share website with my friends.
Maglie Calcio a poco prezzo www.niveauradio.com


 

Wow because this is very helpful job! Congrats and keep it up.


 

Keep up the excellent work and generating the crowd!


 

Thanks a bunch! This a wonderful web site.
Barcelona Tr?jer www.shannonmdrealty.com


 

Thanks extremely practical. Will certainly share site with my buddies.
Manchester United Tr?jer www.shannonmdrealty.com


 

Passion the website-- extremely user friendly and great deals to see!
Real Madrid Tr?jor www.fashionradicalsnews.com


 

Wow, gorgeous site. Thnx ...
Manchester City Tr?jor www.fashionradicalsnews.com


 

Wow because this is excellent work! Congrats and keep it up.


 

Many thanks really practical. Will share site with my friends.
Bayern Munchen Drakt www.socalhardwoodflooring.com


 

You're a really beneficial web site; couldn't make it without ya!
Manchester United Tr?jer www.asleftasfound.com


 

Wow cuz this is great work! Congrats and keep it up.


 

Sustain the great job and delivering in the crowd!


 

Great internet site! It looks very professional! Maintain the great job!
Maglia Athletic Bilbao Poco Prezzo www.zeropartypolitics.com


 

What's up, great web page you have presently.
Liverpool Tr?jer www.asleftasfound.com


 

Hey, well put together webpage you've presently.
Arsenal Tr?jor www.bettermensolutions.com


 

Astonishingly user friendly website. Enormous information offered on couple of clicks.
Bayern Munchen Tr?jor www.bettermensolutions.com


 

You're a really practical website; could not make it without ya!
Manchester City Drakt www.mydescubrimientos.com


 

Passion the site-- extremely individual pleasant and whole lots to see!
fotbollströjor shop.tampabaygoodliving.com


 

Definitely amazing! m8chty72.cf m8chty72.cf's really very good and looks fantastic too. Also fantastic for folks.


 

I feel these 7y0gfkir.gq 7y0gfkir.gq are soo friggin adorable!!!<


 

Wonderful Web page, Maintain the very good work. Thanks!|
maglie calcio poco prezzo landisdocumentary.com


 

Just desired to express I am pleased I stumbled on your web site.|
maglie calcio poco prezzo nada.or.kr


 

I adore this website - its so usefull and helpfull. sites:ggglese.com
Billige Fodboldtrøjer ams.uy.to


 

Exceptionally individual pleasant site. Great information available on few clicks on. sites:ggglese.com
Günstige fußballtrikots agropromnika.dp.ua


 

Extremely instructive, look frontward to visiting again. sites:ggglese.com
maglie calcio poco prezzo styleplus4u.net


 

Add a Comment

These HTML tags are permitted: p, b, i, a, and blockquote. A comment may not include more than three links. Participants in this discussion should note the site's moderation policy.