NIC on DL380 having problems...maybe?

[Matt Hearn]

Hi all, we're running Red Hat Enterprise 2.4.21-47.0.1.EL on a DL380; recently they had some kind of application problem, and someone rebooted the box. Now we can't get in through the network.

I'm able to get in via the ILO, but my best OS is HP-UX, so I'm completely at a loss for how to check hardware. The NIC appears in ifconfig, and it can ping itself; if you ping broadcast, the NIC responds. My thinking is that means that the NIC itself is fine, and we're having a coincidental network problem.

As an added bonus, the server itself is in Puerto Rico, and getting support to it would be a MAJOR challenge. I'm a bit lost here. Any idea how I can verify that the NIC is good and the problem isn't on the server?

Thanks!

[Ivan Ferreira]

You can check the link status with:

ethtool ethX

You can also check if the firewall is enabled, if so, disable it to ensure that is not the problem:

service iptables status

Check if you have increasing input packet in the output of ifconfig -a or netstat -ni for the eth device.

[Matt Hearn]

I get some interesting info from ethtool eth0:

Settings for eth0:
Supported ports: [ MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: Unknown! (65535)
Duplex: Unknown! (255)
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Current message level: 0x000000ff (255)
Link detected: no

I'm not sure if the lack of a "Link detected," as well as the Unknown speed and duplex measurements, mean that the NIC itself is bad, or if it's just not getting a link to the switch (bad cable or switch port). Anybody else seen this?

[Matt Hearn]

Turns out I'm an idiot! (no surprise there) Due to my unfamiliarity with the server, it turns out they don't use eth0 anymore; they only have a connection plugged into eth2, which has been giving the following error on any attempt to start the network:

tg3 device eth2 does not seem to be present, delaying initialization.

There's some indications that I may have some kind of driver malfeasance going on, but I can't figure out what's conflicting.

[Ivan Ferreira]

What is the content if your files:

/etc/modprobe.conf
/etc/sysconfig/network-scripts/ifcfg-eth*

The ouput of:

ifconfig -a

[Matt Hearn]

There was no /etc/modprobe.conf, but I did find one elsewhere in /etc:

# cat /etc/log.d/conf/services/modprobe.conf | more
###########################################################################
# $Id: modprobe.conf,v 1.7 2002/10/12 02:08:09 kirk Exp $
###########################################################################

# You can put comments anywhere you want to. They are effective for the
# rest of the line.

# this is in the format of = . Whitespace at the beginning
# and end of the lines is removed. Whitespace before and after the = sign
# is removed. Everything is case *insensitive*.

# Yes = True = On = 1
# No = False = Off = 0

Title = "ModProbe"

# Which logfile group...
LogFile = messages

# Only give lines pertaining to the modprobe service...
*OnlyService = modprobe
*RemoveHeaders
########################################################
# This was written and is maintained by:
# Kirk Bauer
#
# Please send all comments, suggestions, bug reports,
# etc, to kirk@kaybee.org.
########################################################



# more /etc/sysconfig/network-scripts/ifcfg-eth*
::::::::::::::
/etc/sysconfig/network-scripts/ifcfg-eth0
::::::::::::::
DEVICE=eth0
ONBOOT=yes
BOOTPROTO=static
IPADDR=138.58.8.134
NETMASK=255.255.255.128
GATEWAY=138.58.8.129
::::::::::::::
/etc/sysconfig/network-scripts/ifcfg-eth2
::::::::::::::
# Broadcom Corporation|NetXtreme BCM5703X Gigabit Ethernet
DEVICE=eth2
ONBOOT=yes
BOOTPROTO=static
IPADDR=138.58.8.134
NETMASK=255.255.255.128
GATEWAY=138.58.8.129

# ifconfig -a | more
eth0 Link encap:Ethernet HWaddr 00:0D:9D:97:89:AD
inet addr:138.58.8.134 Bcast:138.58.8.255 Mask:255.255.255.128
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:11

eth1 Link encap:Ethernet HWaddr 00:0D:9D:97:89:AC
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:15

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:389 errors:0 dropped:0 overruns:0 frame:0
TX packets:389 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:41326 (40.3 Kb) TX bytes:41326 (40.3 Kb)

Thanks!!!

[Ivan Ferreira]

Lot of problems.

If /etc/modprobe.conf does not exists, then /etc/modules.conf should.

The output of ifconfig -a shows eth0 and eth1 (normal), but you are not receiving any packets, so this looks like a physical link problem.

If you want to use only one adapter, modify the ethX device file with the right values and delete the other, as eth2 does not exists, you can remove ifcfg-eth2 file. This is why you get "delaying initialization".

You can then rename and set the apropiate DEVICE entry the remaining file, according to the interface you want to configure.

[rick jones]

As you've probably already determined, seeing responses to a ping of a local IP address or broadcast address does not mean that the NIC itself is necessarily healthy. On the way out the stack notices that the packet will be coming to itself and may not even bother the driver (in the unicast case). btw, the same holds true for HP-UX - if you ping broadcast and see only your own IP in replies, it does not mean that anything made it out onto the net itself.

lspci is sometimes useful to run - it is the closest linux has to ioscan. you can see what it shows for interfaces.

there is a port of the HP-UX linkloop utility. with it you can check link-level connectivity - iirc the URL is:

http://freshmeat.net/linkloop

linux is somewhat notorious about renaming interfaces - device name persistence isn't nearly as robust (IMO) as it is under HP-UX. while I'm not sure it is there in a 2.4 kernel, udev is the stated direction for piling device name persistence on top of linux. often one will see a MAC address in those ifcfg-ethN files which is meant to cause a NIC with that MAC to have that interface name. that isn't necessarily udev, but could be using some earlier mechanisms.