Frustrating network problem

[Matt Hearn]

I'm having fits over a problem we're having. We have a system that is for all intents and purposes identical to another system. Both systems have three NICs, one of which is on the production LAN, one of which is on the backup (as in, we run the backups over it) LAN, and one is unused.

Both boxes were recently hard crashed due to a power outage. One box came back up and reappeared on both networks with no problems. The other came back up on the production network, but doesn't seem to work right on the backup network.

It doesn't seem that HP has the same -i option for ping as everyone else, and I can't find any way of specifying I want to ping from the backup NIC, so I'm basically flying blind. If I ping the broadcast address for the backup NIC, I get a response from another server that's on that LAN, that I shouldn't be able to see at all. No response from the media server, which I should be able to see, and the media server can't see my box either.

I've compared routing, it's the same (except for the device-specific IP stuff); I've checked the ARP tables, and they're a mess, and I don't understand what I'm seeing there.

I can't find any tools that will tell me if the NIC is simply broken. The link lights on it are on. I'm not, unfortunately, near the box at the moment, so switching the cable over to the unused NIC to see if I can bring the connection up there isn't an option yet.

Any suggestion of things to try would be ENORMOUSLY helpful!

The NIC in question is a A5230A/B5509BA PCI card, sitting in an L2000.

[Rick Garland · Accepted solution]

A 1st step - compare the netconf files on each system. Aside from the typical differences of hostname and IP addresses, the remainder of the file can be pretty close to being the same for each system.

Is anything appearing as "questionable" in the configuration? Example, check the subscript numbers for the various LAN interfaces.

[Florian Heigl (new acc)]

netstat -i before and after the ping will show interface usage, which should be enough to replace the ping -i.
also look at administrative and operational stati of the backup lan interface - any chance one prints 'down'?
even if not, try a lanadmin reset of the nic.

in netstat -rn are the device routes really, really ok?

throw away the current arp tables with
arp -a -d and then look if they become messy again.

run tcpdump on the backup interface while pinging.

[Matt Hearn]

netconf: They are the same, aside from the host IP addresses and names being different. Same gateways, same broadcasts and masks.

The netstat -i before and after seems to show that the packets are indeed going through the backup lan NIC (which contradicts what the network guy is saying; he's not seeing any go through).

netstat -rn: I don't know if it's okay, but on the bad box it's identical (aside from expected differences) to the one that works.

arp -a -d doesn't work, I get "arp: -d: bad namelist"

I don't currently have TCPdump installed, but I'll dig up a copy off my laptop here somewhere and put it on there.

Thanks!

[Devesh Pant_1]

Matt,
don't loose heart here. You might want to follow the folowing steps

1. do a lanscan
2.do a netstat -in

now you will have some details on the IP address and the PPA of the NICs

3. check the ifconfig lan0 ( or whatever the PPA number you want to check)

4. go throught the /etc/rc.config.d/netconf file to verify the settings.

I believe you have already done all this, not to worry

5. The greatest of all the tools handy to verify hardware leve connectivity between productio NICs and backup NIcs would be
linkloop, this will tell you if there is a problem below the IP level and if the network cable etc has issues
here is the command
linkloop -i 1 0x00306E057115
in the example 1 is your production NIC lan1 and 0x00306E057115 is the MAC address of the prodction NIC of the working Server You can get it using lanscan).
this command should come back with an OK
if not, there is a hardware level issue.

also you can check the speeds of each nic with

6.lanadmin -x 1

if you need more help please ask

thanks
DP

[Florian Heigl (new acc)]

then it's

arp -d -a


sorry :)

also - netstat -i - do You also see the counter increase for packets *coming back*?

[Pat Lieberg]

Is it possible the adapter in question was damaged by the power outage? Have you checked an ioscan output to see if the adapter is listed as CLAIMED?

Also, what version of HP-UX are you using? My 11.11 system has the -i option for ping.

[Matthew Ghofrani]

The linkloop suggession is the best, also check the validity of "/etc/rc.config.d/hpbtlanconf" and make sure your speed and duplex and handshakes match on what you see there vs. the port assignment on the switch. It is possible they have switched back to their original default.

Matthew from Boston