N-Class server & Network Duplicate IP Address

[Simon R Wootton]

On Friday last week, our network failed causing 600 nationwide users to be off the systems for 3 hours. We eventually found the cause of the network problems but I'm looking for ways to stop it from happening again.

The problem was caused by a dumb terminal having the same IP address as the main N-Class server, the terminal had been swapped by an engineer & re-configured correctly. It took us 2.5 hours to find the duplicate after tracing routes etc.

My question is how do I combat this again - is there a way to stop the network from crashing and affecting the main servers if a piece of kit has the same IP ?

Any suggestions/comments welcome.
Simon

[Patrick Wallek]

I don't know of any way to prevent duplicate IP addresses from royally screwing things up. The only way I can think of to combat this is to make sure that ANYONE that assigns an IP address to ANYTHING checks to make sure that the IP address isn't already in use by anything else.

If you aren't using DHCP for granting IP addresses it should be fairly easy to set up some kind of policy to prevent this in the future. Even if you were using DHCP, there are ways around having duplicate IPs.

[Tim Malnati]

I had a similar experience with a pc tech. It's astounding how quickly the network falls apart. It was a case where he applied the server IP address to Windows networking instead of the telnet application configuration.

First, I recommend that servers be on a separate subnet than workstations with totally separate networking equipment. In this case, a person applying a duplicate IP has the chance of hosing the local subnet, but not your servers. If he/she duplicates the server IP nothing happens where the configured IP does not match the subnet that it's connected to and the router should stop any traffic.

DHCP and a Windows system policy locking out user networking configuration are the next line of defense. If the machine needs to have a static IP address, let DHCP handle this with a reservation instead of configuring it locally on the workstation.

All of this is more expensive to implement, but how much did that 2+ hour outage cost? Many end users have learned enough to be dangerous these days, and corporate policies removing IT types from the field does not help. Add to this the criticality of inter-machine server communications and you've probably justified the expenditure.

[paul courry · Accepted solution]

This is trivial, but necessary.

In the programming world it is standard to require that TWO, repeat, TWO digits of a critical number be different to prevent mistyped digits from causing major problems.
I myself prefer all three numbers of a segment to be different, thus reducing the possibility of an error by a factor of 1000.

Example:

138.84.101.xxx
138.84.854.xxx

In the above example none of the digits in the segments in the class B address above (101 versus 854) are the same. It would require some effort to mistype three numbers and thus cause a problem.

Of course YMMV and the world is constantly designing and improving faster, dumber idiots to get around our efforts at making things idiotproof.

[rick jones]

one additinal low-tech (and thus perhaps minimal effect) thing you can do is give your server more than one IP address. in theory, if another device on the net absconds with one of them, people can be told to try the backup ip/name.

certainly though, it would not be a complete solution - there could still be problems with connections initiated from the server with the stolen IP address.

[Tim Malnati]

Although Rick has a good idea, be careful! You need to consider the impact of bringing the users back into the machine immediately, particularly in a transaction oriented environment. The situation is even worse in a multi-server distributed database application using such things as NFS, dedicated sockets, etc. In some environments, loss of a connection does not necessarily mean total loss of the processes attached.

[Cheryl Griffin]

This message will get even more responses if you post it under the Networking & Openview --> HP-UX Networking forum.

[Shannon Petry]

Hard to come up with a bullet proof solution for laziness, stupidity, and lack of attention to detail, but here is one thing I did which may help.

On all critical servers I hardcode the ARP table.
I.E.
DNS server,
NFS servers,
App servers.

Build a table of
IP MAC-ADDRESS

I made a little boot script that looks at a control file (/etc/rc.config.d/hardarp)
The boot script then does this...
if [ "${LOAD_ARP_TABLE}" == "1" ] ; then
if [ -f "${ARP_TABLE}" ] ; then
/etc/arp -f "${ARP_TABLE}"
echo "Caution ARP table loaded from $ARP_TABLE"
sleep 1
fi
fi

While it may not affect clients, at least my servers never loose communication....

Regards,
Shannon