topic Re: DL380 (G4) + Linux + High Load=Network Drop in ProLiant Servers (ML,DL,SL)

DL380 (G4) + Linux + High Load=Network Drop

Rick King_4 — Thu, 25 May 2006 17:24:18 GMT

Hello All!

I am in need of assistance with a DL380 G4 running Linux. The problem is that when the machine is under a high load, it will drop off of the network, and return to the network when the load is lowered. Has anyone else had this problem before, were you able to find a fix? Someone suggested to update the NIC driver, it just so happens HP released a NIC driver for the bcm5700 modules on May 23, 2006. I installed the new driver, but no dice, the machine still dropped from the network. If anyone can provide tips on debugging Linux, or if there's a silver bullet on a fix, any information is greatly appreciated. Additional info below:

My best to you,

Rick King, Boise, ID

Machine: DL380 G4 Bios Rev P51 (Latest)
Operating System: Novell Linux 9 (SuSe)
Uname: Linux boilx7 2.6.5-7.201-bigsmp #2 SMP Thu May 18 20:05:14 MDT 2006 i686 i686 i386 GNU/Linux

lsmod Output
mvfs 306256 192
vnode 129032 196 mvfs
edd 26008 0
joydev 26688 0
sg 54048 0
st 57500 0
sr_mod 33316 0
ide_cd 54788 0
cdrom 55196 2 sr_mod,ide_cd
nvram 25736 0
usbserial 48240 0
parport_pc 53568 0
lp 27780 0
parport 56520 2 parport_pc,lp
nfsd 119240 17
exportfs 22528 1 nfsd
speedstep_lib 20224 0
freq_table 21504 0
ipv6 326908 28
deflate 20352 0
zlib_deflate 39064 1 deflate
twofish 55424 0
serpent 31616 0
aes 45504 0
blowfish 26752 0
sha256 26240 0
sha1 25088 0
crypto_null 18816 0
af_key 48784 2
libafs 566464 2
tg3 101252 0
ehci_hcd 46468 0
hw_random 21780 0
uhci_hcd 48016 0
evdev 26112 0
usbcore 130272 5 usbserial,ehci_hcd,uhci_hcd
bcm5700 169112 0
e1000 116612 0
af_packet 39048 0
subfs 24448 1
dm_mod 73472 0
reiserfs 276688 3
cciss 69956 7
sd_mod 37888 0
scsi_mod 135364 5 sg,st,sr_mod,cciss,sd_mod

lspci Output
0000:00:00.0 Host bridge: Intel Corporation E7520 Memory Controller Hub (rev 0c)
0000:00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express Port A (rev 0c)
0000:00:06.0 PCI bridge: Intel Corporation E7520 PCI Express Port C (rev 0c)
0000:00:1d.0 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #1 (rev 02)
0000:00:1d.1 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #2 (rev 02)
0000:00:1d.2 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI #3 (rev 02)
0000:00:1d.3 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #4 (rev 02)
0000:00:1d.7 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB2 EHCI Controller (rev 02)
0000:00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c2)
0000:00:1f.0 ISA bridge: Intel Corporation 82801EB/ER (ICH5/ICH5R) LPC Interface Bridge (rev 02)
0000:00:1f.1 IDE interface: Intel Corporation 82801EB/ER (ICH5/ICH5R) IDE Controller (rev 02)
0000:01:03.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
0000:01:04.0 System peripheral: Compaq Computer Corporation Integrated Lights Out Controller (rev 01)
0000:01:04.2 System peripheral: Compaq Computer Corporation Integrated Lights Out Processor (rev 01)
0000:02:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge A (rev 09)
0000:02:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge B (rev 09)
0000:03:01.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10)
0000:03:01.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10)
0000:04:03.0 RAID bus controller: Compaq Computer Corporation Smart Array 64xx (rev 01)
0000:05:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge A (rev 09)
0000:05:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge B (rev 09)

Output from cat /proc/bus/pci/devices | grep bcm
0308 14e41648 19 fdef0004 00000000 00000000 00000000 00000000 00000000 00000000 00010000 00000000 00000000 00000000 00000000 00000000 00000000 bcm5700
0309 14e41648 1a fdee0004 00000000 00000000 00000000 00000000 00000000 00000000 00010000 00000000 00000000 00000000 00000000 00000000 00000000 bcm5700

Re: DL380 (G4) + Linux + High Load=Network Drop

Vitaly Karasik_1 — Sat, 27 May 2006 14:11:39 GMT

Do you see something interesting in /var/log/messages / dmesg?

Re: DL380 (G4) + Linux + High Load=Network Drop

Rick King_4 — Tue, 30 May 2006 09:56:39 GMT

Thank you for your question.

There are no obvious errors or fatal messages in the log file. The only thing I was able to capture was these two lines in dmesg. These lines just repeated themselves well over a hundred lines.

RPC: sendmsg returned error 101
nfs: RPC call returned error 101

Re: DL380 (G4) + Linux + High Load=Network Drop

Vitaly Karasik_1 — Tue, 30 May 2006 10:51:31 GMT

nothing about ethX?

"ifconfig" output?

Re: DL380 (G4) + Linux + High Load=Network Drop

Rick King_4 — Tue, 30 May 2006 10:57:08 GMT

As far as ifconfig, here's what I have (after a fresh reboot:

ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:11:86:BG:8C:63
inet addr:15.8.26.235 Bcast:15.8.21.255 Mask:255.255.248.0
inet6 addr: fg80::233:85ff:febe:8c63/64 Scope:Link
inet6 addr: 2001:1890:1109:401:211:85ff:febe:8c63/64 Scope:Global
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3120362 errors:0 dropped:0 overruns:0 frame:0
TX packets:3507228 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:778623905 (742.5 Mb) TX bytes:1865702949 (1779.2 Mb)
Interrupt:25 Memory:fdef0000-fdf00000

I'll try to capture the ifconfig command while the machine is unpingable.

Re: DL380 (G4) + Linux + High Load=Network Drop

Vitaly Karasik_1 — Wed, 31 May 2006 04:20:59 GMT

>I'll try to capture the ifconfig command >while the machine is unpingable.

It should be interesting.
In fact, it's very strange that we see nothing about NIC in the syslog.
BTW, which kernel/distro do you use?

Re: DL380 (G4) + Linux + High Load=Network Drop

SPa — Wed, 31 May 2006 04:37:52 GMT

What is the firmware version for the NIC and is this the onboard one.

There is a firmware update available for the gigabit adapters.

Re: DL380 (G4) + Linux + High Load=Network Drop

Rick King_4 — Wed, 31 May 2006 12:02:57 GMT

SumeshP, thank you for your reply.

The NIC we are using is the onboard NIC. I don't know what firmware version the NIC has, however if you know how to obtain the version number in Linux let me know. What I do know is that the bios version the machine is using is P51.

If there is a firmware upgrade for the NIC can you let me know how to get it?

Thanks again Sumesh.

Re: DL380 (G4) + Linux + High Load=Network Drop

Rick King_4 — Thu, 01 Jun 2006 10:44:24 GMT

Ok, at the beginning of the post I indicated the BIOS rev was P51 (latest). What I should have posted was P51 Release Date: 08/26/2004.

Since SumeshP mentioned there is an update available for the gigabit adapters, I've been searching HP's website for the update. What I have found was the "firmware maintenance 7.50" CD. I downloaded the ISO and updated the BIOS on one of the DL380s to P51 Release Date: 10/10/2005. So I have one machine with BIOS P51 Release Date: 08/26/2004, and the other machine with BIOS Release Date: 10/10/2005.

SumeshP, if you can provide the information on the firmware you mentioned, I would greatly appreciate it!

Re: DL380 (G4) + Linux + High Load=Network Drop

SPa — Fri, 02 Jun 2006 01:37:25 GMT

The online NIC firmware may be downloaded from the below link:

http://h18000.www1.hp.com/support/files/server/us/index.html

Select the server Model and the Operating system you have and you should get the Firmware-Network that updates all the firmwares of the Gigabit adapters.

The name is HP ProLiant NC10xx/67xx/77xx/150x/320x Gigabit Server Adapter Online Firmware Upgrade Utility for Linux.

The important thing to look at is the prerequisites that mentions the nic driver to be above a specific version

Re: DL380 (G4) + Linux + High Load=Network Drop

Rick King_4 — Fri, 02 Jun 2006 16:32:32 GMT

SurmeshP, thank you very much for the location of the NIC firmware update. I am running builds on the two DL380s, we'll see how they hold up. Thanks again.

Re: DL380 (G4) + Linux + High Load=Network Drop

Operaciones — Tue, 28 Nov 2006 09:56:02 GMT

Did you find a solution for this?... Im having the same problem

Re: DL380 (G4) + Linux + High Load=Network Drop

Rick King_4 — Tue, 28 Nov 2006 10:10:48 GMT

Operaciones,

Apparently, the firmware for the onboard NICs was the fix I needed. SurmeshP was kind enough to post a link to the firmware upgrades:
http://h18000.www1.hp.com/support/files/server/us/index.html

After I upgraded the firmware, I did not see a problem with the network drop anymore.