Apche failover across subnets - Update DNS

[Karthik S S]

Hi,

We have two hp-ux boxes with identical configuration running Apache and they are placed in different subnets. The data is replicated manually between these two machines (haweb1 & haweb2). We want to configure a sort of failover b/w these two systems so that when haweb1 fails haweb2 can take over which is not possible using Service guard as these machines are in different subnet. SO I am planning to write a generic monitoring script which can to some extent automate this. Also I want to have a common host name for these two machines (haweb). Now the problem here is to update the DNS automatically such that when haweb1 fails the common hostname entry - haweb in DNS should be assigned with the IP of haweb2 and vise versa. Kindly suggest me how do I go about configuring this.

Thanks in advance,
Karthik S S

[Steven E. Protter · Accepted solution]

You should be able to use virtual networking to handle the ip address situation.

If the main lan of the backup machine is lan0, then have a script take over the errant ip address for lan0:1

You need to run a test and make sure that locks up networking.

For the rest, I'll tell you how I try and handle it for a real web server shop that combines two Linux Servers with a very old HP-UX box thats been drafted into service.

I have three boxes
dns1
dns2
dns3

dns2 is the hp-ux box.

All three are master servers for the entire domain. In practice, a couple of times a day, dns2 and dns3 take the zone files from /var/named via scp and restart the named daemon.

Assuming I don't do something stupid and not test a dns change, i get consistency. I generally test changes on dns2 and then push the files back to dns1 when tested.

When dns1 decides to fail as it happened this morning, I have a script that takes over the IP addresss of dns1. Assuming the box is good enough to drop totally off the net, the web data is also copied to all three servers so once the ip address takeover occurs, the outside world does not know my server went down.

Now the thing to deal with is when the web server stops responding and the IP address is hunky dory. That happened today. The only way I am sure would work would be MC/ServiceGuard with the apache application clusetered on three nodes. I don't know if the subnet is an issue because I've not been to ServiceGuard class yet.

I hope I've at least given you additional issues to consider.

I'm thinking of a script on dns2 and dns3 that does an ssh "service httpd status" and somehow reports back the status to the other two servers. If that command fails its safe to assume that dns1 is no longer serving the web and a takover scrip is called for.

I'm also working on this problem and think we should share results.

SEP

[Steven E. Protter]

Inspired by your question I ran the following test:

ssh dns1 "service httpd status"
# you'll need to write this for hp-ux, I've started it and would be happy to share.

I ran this command against dns1 and then did this:
echo $?

Got a zero

Then I shut down the httpd server on dns1

ran the same command again

echo $? returned a 1

So, even if the IP address is up and running its possible to reliabily detect whether the httpd server is running on another server.

Of couse you must have exchanged public keys but its worth it for this functionality.

So how to deal with what happened to me this morning. The video card on the server failed, the IP address was happily taking pings, then named daemon was happily taking and answering requrests. The httpd server was dead as a doornail.

For either of the other two servers to take over service under my scheme they need to take over the IP adress of the web server.

Or they need to shut down the named daemon on the dns1 box.

What I'm thinking is this:

All three servers have a block of ip addresses they can use for service.

The three Master DNS servers have differing databases.

If our check script detects httpd down and named up on the main server, that named daemon needs to be clobbered.

That might be accomplished by logging on and running a service named stop command, but when things are messed up connectivity is a problem.

What could be done is one server sacrifices itself to bring down the main server.

It could try and allocate the same ip address as dns1. That would effectively lock both servers and bring all services for dns1 off the network.

That would allow the remaining server to be the goto server.

Remember dns1 provides answers so long as its there. The whole net is supposed to go to dns2 if dns1 isn't answering requests.

The issue of dns latency can be handled by updated the serial portion of the authority record.


$TTL 86400
@ IN SOA @ dns1.investmenttool.com (
2003122901 ; serial

See the date 2003122901

If you change that to todays date and restart the named daemon dns servers even on the public Internet will recognize the change immediately and dns2 will be to the goto server.

The question is how would a public dns setup react. You don't have to worry about that because you are running an internal server.

I think we should correspond on this issue.

If I'm totally half cocked, zero point me and ignore me. I've had a really strange day. But right now, I think I'm on to something.

SEP

[Karthik S S]

Hi SEP,

Thanks for the wonderful reply and giving me some direction on addressing the problem. I have a meeting now and should be able to test the setup only by afternoon :-( .. I will keep you updated on this.

Thanks again,
Karthik S S

[Karthik S S]

Hi SEP,

Sorry for the delay in replying you back. Finally I got few test machines and the time to test this configuration out. I have written a small script (I am not very good at it) that runs on the DNS server which will monitor apache on two hp-ux boxes (fep1 and fep2).

fep1 - main apache server
fep2 - backup apache server
fepha - common host name given for the "Highly Available Apache" (???!!) which at any point in time runs either on fep1 or fep2.

If it finds fep1 is not running the Apache services then it starts it on fep2 and updates the IP of fepha with fep2's IP address and vise versa.

I am sure that still I need to tune this script to take care of all other aspects. For instance it only check if the services are alive but not the host itself.

If you have time kindly check this script and give me some more inputs ...

Thanks,
Karthik S S