My Name Server is Totally Lame

I started the day with a dead name server that knocked more than 100 sites offline, including Workbench, the Drudge Retort and all of the Buzzword.Com bloggers.

I've been using BIND for years and thought I had run out of interesting new ways to break it. Overnight, most name requests failed and my server log filled up with errors like this:

lame server resolving 'www.cadenhead.org' (in 'cadenhead.org'?): 67.19.3.218#53

A lame server is one that's not responding to a name request it is expected to handle. Requests began failing several days ago, growing in number as name servers around the Internet contacted my malfunctioning server when their cached data expired.

After using DNS Report to test my name server and poring over the configuration file /etc/named.conf and individual domain zone files, all of which appeared correct, I realized that something was missing from my server log each time I rebooted BIND. Lines like this:

named[30731]: zone cadenhead.org/IN: loaded serial 4418

This log entry shows that a name server has properly loaded a zone and can take its requests. When it's missing, the server has no idea it should handle that domain. I suspected that BIND had stopped looking at /etc/named.conf, so I checked the process status monitor:

ps auxw | grep named

This revealed that BIND was running out of a new directory: /var/named/chroot. Somehow, BIND had been moved inside a chroot jail, a popular security technique on Linux that limits a server to a directory and its subdirectories. If crackers break the server, they only can see the files inside the jail. The rest of the box is invisible to them.

I found the new location of named.conf in the jail, added all of my domain zones to the file and rebooted BIND. My server is no longer lame.

I've saved thousands of dollars over the years by running Internet servers on Linux instead of Windows, but there's a significant cost in work time, because something unusual goes fubar at least once a month. I can't tell whether this BIND change happened because of security patches I installed last week or an action taken by my hosting company without my knowledge.

In any case, it now works, so I won't be touching anything until something else breaks.

Security · Linux · Domains · 2005/11/04 · 13 COMMENTS · Link

Comments

Looks like it's time to fire up a new DNS server called "DNS Retort" to combat the vast right-wing conspiracy that is hell-bent on silencing you.

I'm just messing with you, R. :)

#1 · David Knighton · 2005/11/04

Rogers;

I'd like to know who stuck your BIND in a chroot jail? Is this your server, or does someone else manage it?

Just wondering...

#2 · Joe · 2005/11/04

My servers are comanaged. I think this means that I do all of the work, except when I ask for help or they decide to jump in and put a security update on my box to protect the integrity of the host.

It looks like they got a bind-chroot rpm last week, installed it all over their network for security reasons, and didn't realize it would move BIND. But I'm still digging into it.

#3 · Rogers Cadenhead · 2005/11/04

I use tinydns and have never had a problem.

#4 · Guan Yang · 2005/11/05

Hey Rogers,

I just signed up with cari.net and got a dedicated colocated server (as in: I am root on my own private machine, nobody else has access to it) for $99 setup and $59 a month. They install FC4 for you and turn it over to you. They'll remote-hands for free up to a point if you have a software problem and can't boot. Since the hardware is theirs, they fix anything that breaks there, free. 4Mbit/s or 1200 GB/month bandwidth included in that price. I have no idea how they make money at this ... the next bandwidth tier gets a bit pricey, but at the point I'll just buy another server instead, because it's cheaper. (In fact, the guy I talked to even recommended this strategy.) No contract for a year or anything like that. I couldn't imagine some other organization having the ability to install software on my server!

I've called them a few times for support questions and they've been quite responsive. I am not affiliated with cari.net.

Rich

#5 · Rich Drewes · 2005/11/07

RC,

My suggestion:

Use Windows 2003 Servers

Yes, maybe they cost a little more, but not too much more. I've been serving 200+ websites for several years now, and still no problems. Then again, I'm a .net guru and I use Visual Studio.Net 2003 for all my developing, so I am pretty much stuck on Windows.

The real cost is MS SQL Server 2000.

Holla!

#6 · YAAAYUH · 2005/11/08

Gotta agree with the last commenter. Windows 2003 is so much easier to setup, configure and maintain than Linux.

Ohwell, to each their own.

#7 · Yabba Dabba Doo · 2005/11/18

This article saved my day. Thanks!

Turns out chroot jail is widely used in RHEL as well. The deceiving part was that named.conf is located both under /etc and under /named/var/chroot/etc

The first one totally useless, and deceiving.

#8 · adriano · 2005/11/20

Yeah, I know how that is. Sometimes ThePlanet (servermatrix) people aren't even that helpful, I mean sometimes.

They never explain really good details when things happen, and they say UNFORTUNATELY as much as we say FUCK.

#9 · Josh · 2005/12/06

if you want to remove the chroot... this is setup in file:
/etc/sysconfig/named

remove (comment) the ROOTDIR variable. make sure all of you files are accessible by named - this might mean removing any symlinks and putting files back to the directory you specified named to use (default: /var/named)

#10 · fredlongwood · 2005/12/07

to YAAAYUH

so your idea of fixing a problem is to replace the OS?

#11 · toYAAAYUH · 2006/04/10

Thanks for this entry, it solved my problem. I have a monthly a vps account and their support team was taking too long to figure it out. Sure enough, named was running in chroot jail with the wrong config file. Cpanel however was reading the non chroot config file.

Thanks again!

#12 · Jim · 2006/09/07

I had this problem with Fedora Core, which clearly inherited it from its RH roots. As AdrianO says if there weren't another set of files in the 'normal' location, that in effect did nothing, it wouldn't be so confusing.

I guess RH took some stick over this because later Fedora releases and CENTOS 4 have replaced those dead files with symbolic links pointing into the jail.

#13 · john v · 2006/10/11