Sambal

As I mentioned last time, I ran into some challengesincredibly annoying problems setting up a dhcpd failover server.

1. Dhcpd doesn’t record full data for foreign leases.

At first I thought it would be enough to run dhcpd with a failover peer and use djb_update.pl to create the host list for tinydns.  But then I discovered that when the dhcpd master grants a lease, it communicates to the dhcpd slave.  But this communication apparently consists only of the slave’s IP address and ethernet address, and not its hostname.  So it’s not possible to rely on the dhcpd failover protocol to keep the name-IP mappings synchronized on the dhcpd servers.

2. Dhcpd servers race to respond.

Like little virtual fire engines and paramedics, the dhcpd servers race to answer a poor IPless host’s cry for help.  It took me a while to see this one because taurus and draco were initially on the same segment.  Then we did a server move and got hung up halfway, with taurus in its new location but draco still in the old one.  Then all of a suddent a bunch of hosts disappeared from dns.

The problem appears to be that the nearest dhcpd server usually wins in assigning IP addresses, and the way dhcpd failover is implemented is a lot more like load-balancing than I would have thought.  In my opinion, the failover peer should be quiet unless its peer is down (which it is aware of as part of the failover protocol).  In any case, the failover server is live and ready to fill requests, and if it happens to be the nearest server to a given host, it gives out the IP address.

So it’s not enough to be able to get a host list from the failover peer if the main server is down.  At any time, some IPs may be assigned from the master host and some from the failover host.

3. DHcpd doesn’t put hardcoded hosts into its leases file.

This is arguably not a design defect in dhcpd — rather, I can’t imagine that this was an accident — but it sure is annoying.  When you assign a permanent IP to a host using this syntax:

host foobar { hardware ethernet de:ad:be:ef:ca:fe; fixed-address 10.1.1.15; }

then the host suddenly disappears from DNS.

So I managed to briefly lose taurus, my secondary dhcpd server.  Because taurus would be the backup source of internal DNS data, I decided to fix its IP address.  (dnscache can’t accept hostnames in its servers file, for the obvious reason.)  But I neglected to add an entry for taurus in the static host file, and when the old lease expired, dhcpd did not record a new one.  Even though taurus was still getting all its host configuration information from dhcp, including IP address, routing, name servers, etc., dhcpd did not consider it to have a lease since its address could not expire.  Since taurus did not have a valid lease in dhcpd.leases, the djb_update script didn’t put an entry for taurus into the dhcp file.

The most annoying thing about this one is that in order to fix the IP address of a host it’s not enough to change the dhcpd.conf; you also better remember to change the static host table.  And the penalty for error is a delayed failure, because the host won’t disappear from DNS until the old lease expires.

4. djb_update lets the last live lease it reads win.

This is a reasonable design decision since djb_update expects to run on one servers dhcpd.leases file, not on the concatenation of two servers’ files.  But it means that if a host gets a lease from the failover server and then the master recovers, if we manually renew the lease and now get a response from the master, there appear to be two live leases for the host.  (I’d call that a bug in the dhcp protocol, that the host can abandon the lease without notifying the failover server.)

Sinec djb_update doesn’t expect that to happen, it just runs through the leases file making a name -> IP hash using all the valid leases.  If it sees two valid leases for the same name, it uses whichever lease it sees last.  That means that order matters when concatenating the lease files.

A more robust solution would be to honor the most recently written active lease; to add another condition when updating the lease data that the start time of the lease under consideration must be greater than the start time of the saved lease.

This is one of those problems where it takes more time to complain about it than to fix it.  I plan to hack this into my already-hacked-up djb_update.pl script.

Next time: the file layout, Makefiles, scripts, duct tape and baling wire necssary to make this thing work.

Here are the before and after pictures. We used to have two independent single points of failure.

If boris was down, nobody could get an IP address, and internal name resolution didn’t work because the cache couldn’t see the tinydns server (on boris) that is authoritative for the internal network.

If draco the firewall was down, not only could nobody connect to the internet (after all, that is the point of having a firewall), and external name resolution didn’t work. But internal name resolution didn’t work either, because the single caching name server (dnscache) lived on draco.

That’s a result of the “djb way,” where name servers are separate from caches. Tinydns won’t answer queries for a domain which it’s not authoritative over, and dnscache won’t ever give an authoritative answer, but only process requests from clients, get answers from authoritative servers, and cache the results.

In the new setup, there are no single points of failure for dhcp or name service. (There’s still a single firewall host.) We moved all functions off of boris, which is slated for decommissioning.

Draco keeps a dnscache, since it has easy access to outside name servers. It picks up a dhcpd server and associated tinydns. I made it the dhcpd master, because it’s the least-frequently-rebooted server.

Since dnscache and tinydns use the same port (53/udp, the domain service), I had to configure an ethernet alias on draco to serve the tinydns off of. (I couldn’t use localhost, because it needs to be visible to the other dnscache server if the other tinydns server is down.)

Taurus, a new machine still figuring out its identity (external web/intranet/wiki/login/database/…?), gets the dhcpd slave server and an associated tinydns. At first I thought it made sense to separate the dhcpd from the tinydns server, but eventually I recognized that if one dhcpd server was down, it would not do any good to be trying to fetch its dhcpd.leases file to build a dns table. The functions are so closely associated — dhcp gives named hosts an IP address while tinydns reports what the IP address is.

Scorpio, the backup server (BackupPC/snapback), picks up an additional function as an internal name cache. That was a last-minute decision when I realized I needed a second name cache, and I’m not entirely happy about it. Since scorpio carries backup data, I’d like to be able to unplug it and take it offsite (e.g., if I’m reconstructing a crashed machine from its backups) without disrupting the network. So that’s going to have to change. I suppose there’s no deep reason why I can’t put the dnscache on taurus, but I’d have to configure another alias…

That’s the overall layout for redundant dhcpd and internal name service using tinydns and dnscache. Next post: “challenges” (aka annoying problems) I encountered making this work.

A compiler warning tells you where you are probably making a mistake.  Changing the code to eliminate the warning may not remove the mistake.