\n

I returned from a trip out of town Monday to crashing web servers that ate my lunch all week long. For several days, I used the top command in Linux and watched helplessly as two servers ground to a halt with load averages higher than 100.

Top reports the processes that are taking up the most CPU, memory and time. On the server running Workbench, the culprit was always httpd, the Apache web server. This didn't make sense, because Apache serves web pages, images, and other files with incredible efficiency. You have to hose things pretty badly to make Apache suck.

If you know the process ID of a server hog, Apache can tell you what that process is doing in its server status report, a feature that requires the mod_status module. The report for Apache's web site shows what they look like.

Using this report, I found the culprit: A PHP script I wrote to receive trackback pings was loading the originating site before accepting the ping, which helps ensure it's legit:

// make sure the trackback ping's URL links back to us
$handle = fopen($url, "r");
$tb_page = '';
while (!feof($handle)) {
$tb_page .= fread($handle, 8192);
}
fclose($handle);
$pos = strpos($tb_page, "http://www.cadenhead.org/workbench");
if ($pos === false) {
$error_code = 1;
send_response(1, "No link found to this site.");
exit;
}

Most trackback pings are not legit -- I've received 600 from spammers in just the past three hours. Each ping required Apache to check the spammer's site, download a page if it existed, and look for a link to Workbench. A single process performing this task could occupy more than 50 percent of the CPU and run for a minute or more.

I'm surprised Apache ran at all after I added trackback a couple months ago. I was beginning to think the web server software was idiot-proof, but I've proven otherwise.

-- Rogers Cadenhead

Comments

I was beginning to think the web server software was idiot-proof, but I've proven otherwise.

I'm confused. Why is it Apache's fault that you wrote code causing it to crash? A search through all the source of a page would be extremely processor intensive with hundreds of requests an hour.

Can I blame IIS if I wrote an infinite loop in VBScript?


 

Who said it was Apache's fault?


 

Ah, my apologies! I mis-read and took the quote (that I quoted) as meaning it was Apache's fault. Silly me glossed over:

I found the culprit: A PHP script

Keep up the good work!


 

> I was beginning to think the web server software was idiot-proof, but I've proven otherwise.

It's like that saying in Poker... If you've been playing for a while and you haven't spotted the chump, then you are the chump.


 

Just accept the trackbacks (returning a "202 Accepted") and queue them into a file or database table, then separately via a cronjob or some other process, run the queue and purge the 99.999957281% of the crap that is spam from the spammers.

I changed my trackback script to be a PHP script which always replies "202 Accepted" but files all the data way in files to be processed later. Out of maybe 5000 trackbacks I received last year not a single one was valid.


 

I'm having trouble figuring out how the script you posted was using up 50% of the CPU for up to a minute. It shouldn't be nearly that bad. What kind of hardware are we dealing with? The download of the page is essentially computationally free, and scanning one string for a single occurrence of another string is very efficient. I'm just not seeing why it would be running slowly.


 

This server is an Intel 2.4 Ghz P4 Celeron with 1 gig of DDR 266 memory running Redhat Enterprise Linux 3.


 

I would bet it's the DNS lookup on the fopen of the URL. DNS is typically thread safe but not reentrant. Under a threaded server like Apache/2 each DNS call will block until the hostname's been resolved, if you get a slew of trackbacks at the same time the following requests will stall waiting for the hostname to resolve. Assuming everything is working perfectly this shouldn't be a problem, but if you're getting slammed by trackbacks which list fake web sites for the trackback URL, your DNS may hang just long enough on each request to cause the server to fall over.

This is pure speculation on my part, I haven't stared at apache or resolver source code in years.


 

It it helps, I just crashed a server too by simply requesting 100 or so PHP files in just a few minutes via AJAX. Seems each of th PHP files had loops which ran a thousand+ or so times, causing the crash by overloading the machine?


 

It's not apache's fault the server crashed? How preposterous! The software shouldn't crash. It should issue some form of useful error message then carry on.


 

Post #9. I too just carshed a W2K3 server by running a php script that processes a lot of data. Crashdump shows that php is at fault. This is is rather annoying.


 

The problem with the example code is that it does not check for errors returned by fopen().

If fopen() returns an invalid handle, then the while loop will run forever, since feof($handle) will always be false.


 

This is just the information I am finding everywhere. Thanks for your blog, I just subscribe your blog. This is a nice blog..


 

Jordan 12 Gym Red www.jordan12gymred.us.com
nike factory outlet store online www.nikefactoryoutletstoreonline.com
nike factory outlet store online www.nikefactoryoutletstoreonline.us
Nike Outlet www.nikestores.us.com
retro jordan 33 www.jordan33.us
cheap jerseys from china www.cheapjerseysfromchina.us
nfl jerseys www.customnfljerseys.us
air jordan 11 concord www.jordan11concord.us.com
Jordan 12 Gym Red 2018 www.jordan12gymred.us
Jordan 12 Gym Red 2018 www.redjordan12.us
Yeezy Shoes www.yeezy.com.co
Yeezys www.yeezys.us.com

Pandora Rings Official Site www.pandorarings-jewelry.us


 

Thanks for posting this info. I just want to let you know that I just check out your site and I find it very interesting and informative. I can't wait to read lots of your posts.


 

Nike Air Vapormax Flyknit www.nikeairvapormaxflyknit2.us
Nike VaporMax Flyknit,Nike Air Vapormax Flyknit,Nike Air Vapormax Flyknit 2 www.nikevapormaxflyknit.us
Nike VaporMax Plus,Nike Air VaporMax Plus,VaporMax Plus,Nike Air Vapormax Flyknit,Nike Air Vapormax Flyknit 2 www.nike-vapormaxplus.us
Nike Air Max 2019,Air Max 2019 www.max2019.us
Nike Air Max 2019,Air Max 2019,Nike Air Max www.air-max2019.us
Nike Air Max 2019,Air Max 2019,Nike Air Max www.nike-airmax2019.us
Nike Air Zoom Pegasus 35,Nike Pegasus 35,Nike Air Zoom Pegasus www.nikeairzoompegasus35.us
Nike Pegasus 35,Nike Air Zoom Pegasus 35,Nike Air Zoom Pegasus,Nike Pegasus www.nikepegasus-35.us
Nike Zoom,Nike Air Zoom www.nike-zoom.us
Nike Air Max 270,Air Max 270,Nike Max 270 www.nikemax270.us
Nike Shox,Nike Shox Outlet,Cheap Nike Shox Outlet www.nikeshoxoutlet.us
Adidas Outlet www.outlet-adidas.us
adidas originals www.originalsadidas.us
adidas ultra boost,ultra boost www.adidasultra-boost.us
Adidas Shoes www.shoesadidas.us
Pandora Rings,Pandora Ring,Pandora Rings Official Site www.pandorarings-jewelry.us
Pandora Official Site,Pandora Jewelry Official Site,Pandora Rings Official Site www.pandora-officialsite.us
Pandora.com,Pandora,Pandora Official Site,Pandora Jewelry Official Site www.pandora-com.us
Pandora jewelry Outlet,pandora charms,,pandora bracelets,pandora rings,pandora outlet www.pandora-jewelryoutlet.us
pandora outlet,pandora jewelry outlet,pandora charms outlet,pandora jewelry www.pandoraoutlet-jewelry.us
Cheap NFL Jerseys,NFL Jerseys Cheap,Cheap Sports Jerseys www.cheapsportsnfljerseys.us
Cheap NFL Jerseys,NFL Jerseys Outlet,Cheap Jerseys www.cheapoutletnfljerseys.us
Cheap NFL Jerseys,NFL Jerseys Cheap,Cheap Sports JerseysNFL Jerseys,Cheap NFL Jerseys,NFL Jerseys Wholesale,Cheap Jerseys www.nfljerseyscheapwholesale.us
Cheap NFL Jerseys,NFL Jerseys Cheap,Cheap Sports Jerseys www.cheapjerseyselitenfl.us
NFL Jerseys,NFL Jerseys 2019,Cheap NFL Jerseys,NFL Jerseys Wholesale www.nfljerseys2019.us
Pittsburgh Steelers Jerseys,Steelers Jerseys,Steelers Jerseys Cheap www.pittsburghsteelers-jerseys.us
Dallas Cowboys Jerseys,Cowboys Jerseys,Cowboys Jerseys Cheap www.dallascowboysjerseyscheap.us
NFL Jerseys,NFL Jerseys 2019,Cheap NFL Jerseys, Cheap Authentic Nfl Jerseys www.nflauthenticjerseys.us
NFL Jerseys Wholesale,NFL Jerseys 2019,Cheap NFL Jerseys, Cheap Nfl Jerseys Wholesale www.wholesalenfljerseysshop.us
NFL Jerseys Wholesale,Cheap Nfl Jerseys Wholesale,,Cheap NFL Jerseys,NFL Jerseys 2019 www.authenticnflcheapjerseys.us
Pandora Sale, Pandora Jewelry, Pandora UK www.pandorasale.org.uk
Pandora Charms,Pandora UK,Pandora Charms Sale Clearance www.pandoracharmssaleuk.me.uk
Pandora Bracelets,Pandora Bracelet,Pandora Jewelry www.pandorabraceletsjewellry.me.uk
Pandora UK, Pandora Sale, Pandora Jewelry UK www.pandorauk-sale.org.uk

Salomon UK www.salomons.me.uk


 

It is a great website.. The Design looks very good.. Keep working like that!.
how to preform a website audit


 

Add a Comment

These HTML tags are permitted: p, b, i, a, and blockquote. A comment may not include more than three links. Participants in this discussion should note the site's moderation policy.