arp resolution issue

[steven Burgess_2]

Hi all,

Bit of a long one here

We have an ibm blade center with 2 blades running rhel3 update 4 (plus some aix blades) last week 2 of the blades disappeared off the network. I started a remote control session and noted that the lan cards were up, with no errors in the messages and kernel messages file. I could ping the card internally but couldn't ping out. The blade centers have internal d-link switches which are then connected to our customers lan. They have 2 on board ethernet cards, eth0 through on switch and eth1 through the other. When I moved the ip address to eth1, all was ok. This is the same for both blades ! I placed a call with IBM as I initially thought that 2 servers of the same type (HS20 blades) having the same issue, it must be hardware. We first of all went through updating bios, broadcom, ethernet and management module firmware before they agreed to send out an engineer. (nothing in any of the hardware logs by the way) 1 blade now has a new system board, there are 2 new switches in the center, still we can't ping the switch when connected through eth0. If i run tcpdump against eth0 I can see passing traffic etc. When I ping the switch I can see the arp request but no reply. The entry in the arp table is incomplete. If I check the arp table on the switch I see the mac address of eth0 on the external port, not the internal port. The reply is going out of the wrong port. If I manually update the arp table on the server with the correct details of the switch/server I am trying to ping, all is ok. We moved one of the servers into another blade center and we get the same issue !! We don't seem to be able to pinpoint exactly where the problem is. I asked the comms guy on site whether he could see anything advertising itself with the same mac addresss (unlikely) and whether any changes were made (confirmed not) Myself and IBM are now at a loss as per what we can do next.

Any Ideas ?

Thanks in advance, and for spending the time reading this thread

Steven

[steven Burgess_2]

The more I think about this, I am getting slightly worried that there is a possibility of arp redirection going on ? How can we detect this ?

Steve