I returned from a trip out of town Monday to crashing web servers that ate my lunch all week long. For several days, I used the top command in Linux and watched helplessly as two servers ground to a halt with load averages higher than 100.

Top reports the processes that are taking up the most CPU, memory and time. On the server running Workbench, the culprit was always httpd, the Apache web server. This didn't make sense, because Apache serves web pages, images, and other files with incredible efficiency. You have to hose things pretty badly to make Apache suck.

If you know the process ID of a server hog, Apache can tell you what that process is doing in its server status report, a feature that requires the mod_status module. The report for Apache's web site shows what they look like.

Using this report, I found the culprit: A PHP script I wrote to receive trackback pings was loading the originating site before accepting the ping, which helps ensure it's legit:

// make sure the trackback ping's URL links back to us
$handle = fopen($url, "r");
$tb_page = '';
while (!feof($handle)) {
$tb_page .= fread($handle, 8192);
}
fclose($handle);
$pos = strpos($tb_page, "http://www.cadenhead.org/workbench");
if ($pos === false) {
$error_code = 1;
send_response(1, "No link found to this site.");
exit;
}

Most trackback pings are not legit -- I've received 600 from spammers in just the past three hours. Each ping required Apache to check the spammer's site, download a page if it existed, and look for a link to Workbench. A single process performing this task could occupy more than 50 percent of the CPU and run for a minute or more.

I'm surprised Apache ran at all after I added trackback a couple months ago. I was beginning to think the web server software was idiot-proof, but I've proven otherwise.

-- Rogers Cadenhead

Comments

I was beginning to think the web server software was idiot-proof, but I've proven otherwise.

I'm confused. Why is it Apache's fault that you wrote code causing it to crash? A search through all the source of a page would be extremely processor intensive with hundreds of requests an hour.

Can I blame IIS if I wrote an infinite loop in VBScript?


 

Who said it was Apache's fault?


 

Ah, my apologies! I mis-read and took the quote (that I quoted) as meaning it was Apache's fault. Silly me glossed over:

I found the culprit: A PHP script

Keep up the good work!


 

> I was beginning to think the web server software was idiot-proof, but I've proven otherwise.

It's like that saying in Poker... If you've been playing for a while and you haven't spotted the chump, then you are the chump.


 

Just accept the trackbacks (returning a "202 Accepted") and queue them into a file or database table, then separately via a cronjob or some other process, run the queue and purge the 99.999957281% of the crap that is spam from the spammers.

I changed my trackback script to be a PHP script which always replies "202 Accepted" but files all the data way in files to be processed later. Out of maybe 5000 trackbacks I received last year not a single one was valid.


 

I'm having trouble figuring out how the script you posted was using up 50% of the CPU for up to a minute. It shouldn't be nearly that bad. What kind of hardware are we dealing with? The download of the page is essentially computationally free, and scanning one string for a single occurrence of another string is very efficient. I'm just not seeing why it would be running slowly.


 

This server is an Intel 2.4 Ghz P4 Celeron with 1 gig of DDR 266 memory running Redhat Enterprise Linux 3.


 

I would bet it's the DNS lookup on the fopen of the URL. DNS is typically thread safe but not reentrant. Under a threaded server like Apache/2 each DNS call will block until the hostname's been resolved, if you get a slew of trackbacks at the same time the following requests will stall waiting for the hostname to resolve. Assuming everything is working perfectly this shouldn't be a problem, but if you're getting slammed by trackbacks which list fake web sites for the trackback URL, your DNS may hang just long enough on each request to cause the server to fall over.

This is pure speculation on my part, I haven't stared at apache or resolver source code in years.


 

It it helps, I just crashed a server too by simply requesting 100 or so PHP files in just a few minutes via AJAX. Seems each of th PHP files had loops which ran a thousand+ or so times, causing the crash by overloading the machine?


 

It's not apache's fault the server crashed? How preposterous! The software shouldn't crash. It should issue some form of useful error message then carry on.


 

Post #9. I too just carshed a W2K3 server by running a php script that processes a lot of data. Crashdump shows that php is at fault. This is is rather annoying.


 

The problem with the example code is that it does not check for errors returned by fopen().

If fopen() returns an invalid handle, then the while loop will run forever, since feof($handle) will always be false.


 

Nice to read your article! I am looking forward to sharing your adventures and experiences.
muslim marriage site


 

The website is looking bit flashy and it catches the visitors eyes. Design is pretty simple and a good user friendly interface.
class t shirt printing


 

The way to break Apache server in PHP is impressive and effective. I appreciate this approach.
fnaf world


 

Nice to read your article! I am looking forward to sharing your adventures and experiences.
ออแกไนซ์


 

The website is looking bit flashy and it catches the visitors eyes. Design is pretty simple and a good user friendly interface.
บท็อกซ์


 

Excellent article. Very interesting to read. I really love to read such a nice article. Thanks! keep rocking.
แปลภาษา


 

Hey there! This is my first comment here so I just wanted to give a quick shout out and tell you I really enjoy reading through your posts. Can you suggest any other blogs/websites/forums that cover the same topics? Appreciate it!|


 

Thank you for sharing your thoughts. I truly appreciate your efforts and I will be waiting for your next write ups thank you once again.|


 

I really wanted to write down a simple note so as to appreciate you for all of the unique facts you are showing at this site. My time consuming internet search has now been paid with good quality content to talk about with my classmates and friends. I 'd express that most of us visitors are unquestionably endowed to exist in a decent network with many special people with valuable ideas. I feel very fortunate to have seen the site and look forward to tons of more cool times reading here. Thanks a lot again for all the details.
yeezy boost tiny.gl


 

I enjoy you because of each of your work on this website. Gloria delights in going through investigations and it's simple to grasp why. My spouse and i hear all relating to the dynamic means you deliver simple thoughts on the website and as well as inspire contribution from some other people about this topic plus our own daughter is now understanding a lot. Take advantage of the remaining portion of the new year. You are performing a good job.
cheap nfl jerseys www.nfljerseys.us.org<>


 

I would like to show some thanks to this writer just for bailing me out of this challenge. As a result of looking throughout the search engines and meeting ideas which are not pleasant, I figured my life was done. Existing minus the solutions to the difficulties you have sorted out as a result of your good post is a serious case, as well as ones which might have negatively damaged my entire career if I hadn't encountered the blog. Your own natural talent and kindness in taking care of all the details was tremendous. I am not sure what I would've done if I had not come across such a thing like this. I'm able to at this point relish my future. Thanks so much for this professional and sensible guide. I will not be reluctant to refer your blog post to any individual who would need guidance about this subject.


 

I wish to show my thanks to this writer for rescuing me from such a condition. Right after surfing through the the web and finding solutions that were not pleasant, I believed my life was well over. Existing without the presence of strategies to the issues you have fixed by means of this posting is a critical case, and ones that could have negatively affected my entire career if I hadn't come across your web page. Your main training and kindness in dealing with all the pieces was tremendous. I'm not sure what I would've done if I had not come upon such a stuff like this. I am able to at this moment relish my future. Thanks a lot so much for this specialized and effective guide. I won't think twice to propose your web blog to anyone who should get tips on this issue.
kate spade handbags www.katespadehandbags-outlet.us.com<>


 

Thank you so much for providing individuals with an extremely wonderful possiblity to read critical reviews from this blog. It is usually so pleasing and also full of a lot of fun for me personally and my office peers to search your blog the equivalent of thrice in one week to learn the latest things you will have. Not to mention, I am just always happy concerning the effective inspiring ideas served by you. Some 4 points on this page are easily the most suitable we have all ever had.
off white clothing www.offwhiteclothing.us.com<>


 

A lot of thanks for every one of your work on this web site. Debby take interest in working on investigations and it's simple to grasp why. A lot of people notice all of the lively way you provide important tactics through your web blog and as well as attract participation from the others on that area while my daughter is really understanding a whole lot. Have fun with the rest of the year. You are carrying out a really good job.
zx flux www.adidaszxflux.com<>


 

A lot of thanks for your whole work on this web site. My mother really likes setting aside time for investigations and it's really easy to see why. Many of us learn all concerning the powerful tactic you present rewarding ideas on the blog and as well increase response from some other people on the situation while our favorite child is always starting to learn a lot of things. Take pleasure in the remaining portion of the year. You are always conducting a pretty cool job.
yeezy boost 350 v2 www.yeezyboost350v2.org.uk<>


 

I precisely had to thank you so much all over again. I'm not certain what I would've gone through in the absence of these suggestions discussed by you over that industry. It actually was a real frightening condition in my opinion, however , observing a new well-written approach you solved it took me to cry with happiness. I am thankful for this guidance and thus believe you realize what a great job you happen to be accomplishing educating the mediocre ones using your blog. More than likely you have never got to know any of us.


 

Thanks for all your valuable effort on this blog. Ellie really loves going through research and it's really simple to grasp why. My partner and i hear all regarding the compelling method you give rewarding guides via your web blog and therefore foster contribution from other individuals on this area while our favorite child is without question being taught so much. Enjoy the remaining portion of the year. You are always doing a terrific job.
nike basketball shoes www.nikebasketballshoes.us.com<>


 

I precisely needed to thank you so much all over again. I'm not certain the things I would have done in the absence of the entire thoughts shown by you regarding my subject matter. Previously it was an absolute traumatic circumstance in my opinion, but understanding a new expert style you handled that made me to jump over contentment. Now i'm happy for your advice as well as sincerely hope you recognize what an amazing job you are undertaking teaching the others with the aid of a site. More than likely you haven't got to know any of us.
ultra boost urlr.be


 

Thank you so much for providing individuals with an extraordinarily memorable possiblity to read in detail from this blog. It really is so kind and packed with a lot of fun for me personally and my office co-workers to visit your web site more than thrice a week to read through the newest guidance you will have. And of course, we are at all times happy with all the mind-blowing inspiring ideas served by you. Certain 2 facts in this posting are truly the most efficient we've had.
jordan retro www.retro-jordans.us.com<>


 

I and my friends were found to be examining the best techniques found on your web page and then at once developed a terrible suspicion I had not expressed respect to you for those tips. All of the young men appeared to be totally warmed to study them and have now without a doubt been tapping into them. Thank you for getting quite considerate and for utilizing these kinds of perfect issues most people are really desperate to understand about. Our sincere apologies for not saying thanks to you sooner.
off white www.offwhite.us.com<>


 

I want to point out my affection for your generosity giving support to those people that really want help on this particular idea. Your very own commitment to passing the solution all through has been quite practical and have always helped regular people just like me to achieve their endeavors. Your own interesting guide signifies much a person like me and a whole lot more to my colleagues. Thank you; from all of us.


 

I want to get across my passion for your kindness giving support to women who actually need help with this particular theme. Your very own dedication to getting the solution across has been rather powerful and have usually permitted ladies like me to attain their aims. Your own informative help and advice can mean a whole lot a person like me and still more to my office colleagues. Thanks a lot; from each one of us.


 

I definitely wanted to post a small message so as to say thanks to you for all the fantastic tips you are showing at this website. My rather long internet lookup has at the end of the day been paid with excellent ideas to go over with my two friends. I would assert that many of us visitors actually are undoubtedly lucky to live in a fantastic community with very many special professionals with helpful plans. I feel truly privileged to have used your website and look forward to really more excellent minutes reading here. Thank you again for all the details.
golden goose sale www.golden-goose.us.com<>


 

I have to express thanks to the writer for bailing me out of this particular trouble. After browsing through the world-wide-web and coming across views that were not productive, I thought my entire life was gone. Being alive minus the strategies to the difficulties you've sorted out through this post is a crucial case, and the kind which might have in a wrong way affected my career if I hadn't discovered your web site. Your primary ability and kindness in handling almost everything was useful. I am not sure what I would have done if I had not discovered such a step like this. I can at this point relish my future. Thank you very much for the high quality and result oriented guide. I won't think twice to endorse your web page to any person who will need assistance about this matter.
air max 270 www.airmax270.us.com<>


 

I actually wanted to write down a brief word so as to say thanks to you for all of the magnificent tips you are giving at this site. My extensive internet lookup has finally been honored with wonderful information to go over with my friends. I 'd point out that many of us site visitors are unequivocally lucky to be in a great place with many marvellous people with very beneficial points. I feel extremely fortunate to have seen your website page and look forward to plenty of more excellent times reading here. Thanks again for a lot of things.
adidas zx flux www.adidaszxflux.com<>


 

Add a Comment

These HTML tags are permitted: p, b, i, a, and blockquote. A comment may not include more than three links. Participants in this discussion should note the site's moderation policy.