HP DL360G4, Broadcom drivers, and EL4 Update 4 drop off network

[Sergio J. Larrondo]

Hello all,

I'm seeing intermittent problems with DL360G4's dropping off the network. They will run fine for about a week, but then they decide not to talk anymore. There really doesn't seem to be a ryhme or reason to when they drop. The console is still up, but I can't log in, and the system requires a reboot to get it back on the network. I've tried different ports on the switch, but to no avail.

I am running RHEL3 ES Update 4 with the 2.4.21-27.ELhugemen kernel. I tried the errata kernel (2.4.21-32.ELhugemem) without any success. I am running the Broadcom drivers v7.4.6a. My DL360G3's do not have the problem with the identical software load.

Has anyone experienced this problem?

Thanks,
Sergio J. Larrondo

[xyko_1]

Hi Sergio,

What you mean about ... the console is still up but I can't login ....
You can use the system via console ? The console is open all the time and you can send commands to the system using the console ?
Or you can't access the system anyway ?

If all system is "dead" including console, I guess your problem is other than your nic. It seems all system went down.

Please give us more information.

regards,
xyko

[Rick Garland]

If you cannot log into the console, this would point to something other than the NIC.

[Sergio J. Larrondo]

The console is still up and responding (I run in text mode). I can hit enter as many times as I want and it will keep prompting me for a login. The link LED on the front is lit, but any attempts to ping the machine or access it fail.

If I try to log in as myself (via a NIS account) or root, the machine waits forever. If you're asking if the machine crashes, then technically no-- no panic, just no network connectivity until I reboot. It is not hung.

The machine will keep running, the log is full of ypbind[658]: broadcast: RPC: Timed out. Naturally, I would expect that to be the case because it's like the interface takes itself down and ypbind can't talk to the NIS server. This does this on three DL360G4's I have, all with identical hardware/software configurations. I would find it hard to believe I have bad hardware on all of them.

Before the problem happens I see (on the -27 kernel):

NETDEV WATCHDOG: eth0: transmit timed out
kernel: bcm5700: eth0 NIC Link is Down
kernel: bcm5700: eth0 NIC Link is Up, 1000 Mbps full duplex

I don't see these messages with the -32 kernel, but the interface still drops.

Thanks for your replies!

Sergio

[xyko_1]

Sergio,

it's possible for you to keep an openned session on console, as root if possible, to do a verification about network health when the system loose connectivity ? I know that it's a big securety hole but you are in a great trouble.

regards,
xyko

[Sergio J. Larrondo]

I don't remember if I tried that before (staying logged in as root) successfully, but it is worth a shot. It is a secured area so it'll be safe.

I am running a cronjob that takes statistics of /proc/meminfo, /proc/net/nicinfo/eth0.info, ifconfig, and lsmod. One of my trouble nodes has been up for seven days, so I think it's due to drop soon...

Thanks again!

Sergio

[Maxim Rozin]

Hello all,

We have the same problem. It happens once in a while (days/weeks/months). It happens on hp DL360, DL580 and on ML370. The OS is RHEL 3 ES/AS Update 1/2/3/4. Here is what we discovered so far:
# the system DOES NOT accept new connections via ssh and console.
# the system DOES accept new connections via http (if appache is up).
# open sessions (ssh and console) remain open.
# sometimes (not always) when the system hang, we saw that /var was 100% full (because of bin.X and save.X files in /var/log/audit/). after cleaning the file system, the problem disapeared.
# sometimes (not always) when the system hang, we saw that the signal queue was getting bigger and bigger - the signals were getting into the queue but could not get out.
# almost always there were dozens of CROND proccesses.
# almost always, syslog was not working - the proccess was up, but nothing was appended to /var/log/messages. We tried to stop and start syslog, but it didn't work. actually, none of the /etc/init.d scripts worked. we discovered that initlog command (which is called by most of the /etc/init.d scripts) didn't work, and because of that, all the /etc/init.d scripts failed. so we replaced all the occurences of "initlog" in /etc/init.d/functions with "echo", and this solved the problem (/etc/init.d scripted started to work).
# we also noticed that after rebooting the system (when it hangs), it takes about 10 minutes to start the system logger.

Well, our systems still hang once in a while, and every time it happens, we learn something new. But we still can't prevent this.

Do you have any ideas?

Max.

[Vitaly Karasik_1]

I suggest to upgrade to "Update 5".
From another side, I have DL380G4 with RHEL3 upd4 and it runs without problems. I use "original" tg3 from RHEL.

[Phil Crooker]

Yes, we too are having this problem - have had for a long time with different versions of kernel and tg3 driver, though very intermittent - for us once every few months across about 5 servers with this interface (DL360G2 and G3). I found a debian bug - #278119 that was posted October last year:

http://lists.debian.org/debian-kernel/2004/10/msg00373.html

This was supposed to be fixed by a later ip_conntrack_core but was untested and unconfirmed. Looking at the code we are on much later versions, so it obviously isn't fixed.

For the record, we are using tg3.c ver v3.8 (July 14, 2004) though this has happened with earlier versions, eg v2.2 (August 24, 2003).

The symptom with us is when the system is (for us) moderately to highly loaded we get the following error:

Oct 10 12:52:03 bastion kernel: tg3: eth0: transmit timed out, resetting
Oct 10 12:52:03 bastion kernel: tg3: tg3_stop_block timed out, ofs=2c00 enable_bit=2
Oct 10 12:52:03 bastion kernel: tg3: tg3_stop_block timed out, ofs=3400 enable_bit=2
Oct 10 12:52:03 bastion kernel: tg3: tg3_stop_block timed out, ofs=2400 enable_bit=2
Oct 10 12:52:03 bastion kernel: tg3: tg3_stop_block timed out, ofs=1800 enable_bit=2
Oct 10 12:52:03 bastion kernel: tg3: tg3_stop_block timed out, ofs=4800 enable_bit=2

All comms ceases to that interface. You can run modprobe -r tg3 to force a reload but this doesn't really help.

A number of others have posted similar problems across the distributions, etc but nothing appears to have been done.

[Walt Reed]

I'm also seeing this problem with kernel 2.4.21-37.ELsmp on my DL360G4's.

This is most likely not a hardware problem, but a bug in the driver. I read about a fix in the 2.6 line, with a workaround of using ethtool to turn off TSO. This doesn't work with the 2.4 driver:

[root@dpdb2 log]# ethtool -K eth0 tso off
Cannot set device tcp segmentation offload settings: Operation not supported

So far, I have not found any other workarounds... I'm close to upgrading to RHEL4 to see if I can get this problem solved. Id rather not of course if possible.

[Ivan Ferreira]

This problem:

NETDEV WATCHDOG: eth0: transmit timed out
kernel: bcm5700: eth0 NIC Link is Down
kernel: bcm5700: eth0 NIC Link is Up, 1000 Mbps full duplex


I solved by adding the noapic kernel option to the kernel boot parameters. Edit grub.conf and add the noapic option to the kernel options:

kernel /vmlinuz- ro root=LABEL=/ rhgb quiet noapic

[Phil Crooker]

In reply to the last two messages:

First about the TSO - as TSO was only available from the 2.5 kernel, how can it be a factor re the tg3 driver in the 2.4 (unless this has been backported)? Certainly it wasn't included in the Mandrake 2.4 kernel (and I don't think Debian does backporting).

Where it is supported/implemented, how can you tell it is on or off? ethtool doesn't say and it doesn't seem easy to find out otherwise, at least to my non-system-programming eyes.

Second, about the noapic kernel parameter workaround - is this about the same problem? The watchdog error could be relating to something else, are you getting these errors:

kernel: tg3: eth0: transmit timed out, resetting
kernel: tg3: tg3_stop_block timed out, ofs=2c00 enable_bit=2

many thanks.

[Julian Dunn]

We're experiencing this problem with both HP-DL380's, both G3 and G4 using SUSE Linux Enterprise Server 9. The problem seems to be in the SLES9 SP2 kernels and later and we are seeing the same kinds of errors as others have reported under high traffic (e.g. lots of NFS), e.g.

NETDEV WATCHDOG: eth1: transmit timed out
tg3: eth1: transmit timed out, resetting
tg3: tg3_stop_block timed out, ofs=2c00 enable_bit=2
tg3: tg3_stop_block timed out, ofs=3400 enable_bit=2
tg3: tg3_stop_block timed out, ofs=2400 enable_bit=2
tg3: tg3_stop_block timed out, ofs=1800 enable_bit=2
tg3: tg3_stop_block timed out, ofs=4800 enable_bit=2
tg3: eth1: Link is down.

Does anyone have working suggestions for how to proceed? What impact would booting with "noapic" have on the rest of the system?

[Phil Crooker]

In response to Julian, there is no fix as yet. It appears from this thread (though not confirmed) that this problem occurs with the broadcom driver as well as the tg3. The problem has been around since Dec '03 as a Redhat bug:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=112377

From reading various bits on the 'net the noapic kernel parameter has a "small" impact on performance.

My own problem is that this bug happens so rarely that it would be difficult for me to evaluate if noapic does anything.

We are looking at installing Intel GB cards as a work around. To be fair this has been confused with a number of other problems and (at least for me) impossible to replicate. Rather embarrassing though as Windows servers have no problems....

[Julian Dunn]

Booting with "noapic" did not help the issue nor did switching to the Broadcom vendor drivers (bcm5700 versus tg3)

We're having problems reproducing the problem on demand, although the problem appears to afflict older cards (BCM5700) more than the on-board controllers on the HP-DL380-G[34] (BCM5703X and BCM5704 respectively).

We are also looking at installing Intel cards as a workaround; however, we're raising this particular issue with our representatives at HP as I am personally surprised that this issue has gone on for so long, and been reported on so many flavours of operating system with no resolution.

[Sergio J. Larrondo]

For those of us damned with this problem, what version of the Broadcom chip is on your HP? Here's some of the output from my /proc/net/nicinfo/eth0.info:

Description HP NC7782 Gigabit Server Adapter
Driver_Name bcm5700
Driver_Version 7.4.12b
Bootcode_Version 5704-v3.27b
PCI_Vendor 0x14e4
PCI_Device_ID 0x1648
PCI_Subsystem_Vendor 0x0e11
PCI_Subsystem_ID 0x00d0
PCI_Revision_ID 0x10
PCI_Slot 2
PCI_Function 0
PCI_Bus 2
PCI_Bus_Speed 64-bit PCIX 66MHz

I want to add that it doesn't seem to affect the 5703 based adapters in the 360G3's we have-- if it does, I havent noticed.

So what gives? Is this a Broadcom issue? A driver issue? A kernel issue?

I've upgraded kernels, firmwares, and drivers but nothing seems to fix this. I'm furious because we now have DL585's we have to put in production and they are most likely going to be plagued with the same problem.

Redhat support told me to lock the card and switch to the speed instead of setting it auto. Anyone try this with any success?

Sergio

[Julian Dunn]

Hi Sergio,

We don't have a /proc/net/nicinfo on these servers (maybe that's specific to the kernel you're using) but we've seen the problem on the following cards that are identified in output of /sbin/lspci -vv:

0000:06:01.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5700 Gigabit Ethernet (rev 12)
Subsystem: 3Com Corporation 3C996-T 1000Base-T
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- SERR- Latency: 64 (16000ns min), cache line size 10
Interrupt: pin A routed to IRQ 15
Region 0: Memory at f7ff0000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [40] PCI-X non-bridge device.
Command: DPERE- ERO- RBC=0 OST=0
Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, DC=simple, DMMRBC=0, DMOST=0, DMCRS=0, RSCEM- Capabilities: [48] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable+ DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data
Capabilities: [58] Message Signalled Interrupts: 64bit+ Queue=0/3 Enable-
Address: d074defbf9f5759c Data: b61d


and


0000:02:02.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5703X Gigabit Ethernet (rev 02)
Subsystem: Compaq Computer Corporation NC7781 Gigabit Server Adapter (PCI-X, 10,100,1000-T)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- SERR- Latency: 64 (16000ns min), cache line size 10
Interrupt: pin A routed to IRQ 31
Region 0: Memory at f06e0000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [40] PCI-X non-bridge device.
Command: DPERE- ERO- RBC=2 OST=0
Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, DC=simple, DMMRBC=0, DMOST=0, DMCRS=0, RSCEM- Capabilities: [48] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data
Capabilities: [58] Message Signalled Interrupts: 64bit+ Queue=0/3 Enable-
Address: 41f10212c1060000 Data: 0050


Admittedly the former (5700) is an add-in PCI card, but the latter which also experienced problems is the on-board adapter to an HP-DL380-G3, so I'm certainly fearful of using this stuff in production if we can't easily swap out a potentially defective adapter.

I've done a bit of digging on the Internet regarding this issue and it seems like almost every Linux flavour -- RedHat, SUSE, Debian -- as well as two of the BSDs, NetBSD and FreeBSD, have open problem reports about this, some dating back several years.

As for locking the card to a particular speed, I'm told by our networking people that this isn't recommended for Gig-E. Part of the reason why we may only be seeing this now in our environment is that previously we were running 100Mbps-FDX on the backend and we recently upgraded to Gig-E.

[Chad Slater]

We're experiencing the exact same problem as the OP, but in RHEL4 Update 2.

Here's my lspci -vv (2 cards, the same, so to save space I've only included one).

Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10)
Subsystem: Compaq Computer Corporation NC7782 Gigabit Server Adapter (PCI-X, 10,100,1000-T)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- SERR- Latency: 64 (16000ns min), Cache Line Size 10
Interrupt: pin A routed to IRQ 217
Region 0: Memory at fde70000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [40] PCI-X non-bridge device.
Command: DPERE- ERO- RBC=2 OST=0
Status: Bus=2 Dev=2 Func=0 64bit+ 133MHz+ SCD- USC-, DC=simple, DMMRBC=2, DMOST=0, DMCRS=1, RSCEM-
Capabilities: [48] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable+ DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data
Capabilities: [58] Message Signalled Interrupts: 64bit+ Queue=0/3 Enable-
Address: 3ff357faabbf55f8 Data: abff

[Phil Crooker]

Hi Again,

We have it very infrequently on the BCM5703 (once every two months on our busier machines). We have them running on 100MHz.

Here is all I could dig out with lspci:

Ethernet controller: Broadcom Corporation NetXtreme BCM5703 Gigabit Ethernet (rev 2).
Bus 2
device 1
function 0: IRQ 11.
Master Capable. Latency=64. Min Gnt=64.
Non-prefetchable 64 bit memory at 0xf7ef0000 [0xf7efffff].

As to getting this fixed, as the redhat bug is open with "NEEDINFO" status and as one of the authors of the driver has a redhat email address, perhaps a few posts to bugzilla.rehat.com would be in order.

To be fair to Mesrs Miller and Garzik, the problem happens with the broadcom driver as well as with tg3 - perhaps some pressure from HP to release the driver details (or to fix the bloody thing themselves)?? Broadcom has apparently been uncooperative and the driver had to be reverse engineered.

[Vipulinux]

Hi

You need to use the broadcom modules rather than the default one (tg3)

download from http://www.broadcom.com/support/ethernet_nic/downloaddrivers.php

Steps:
rmmod tg3
Install the bcm module
edit /etc/modules.conf (modprobe.conf)and replace tg3 with bcm5700
modprobe bcm5700

service network restart
and you are done..

Hope this helps.

cheers!!

[Chad Slater]

I fixed a similar issue by upgrading the firmware and the IPMI.

The fixes in http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&objectID=PSD_EU050629_CW01&prodTypeId=329290&prodSeriesId=406606 fixed it for me.

[Steven E. Protter]

Segio,

This issue clearly shows poor or not testing on behalf of broadcom. A simple test of this very common OS and kernel combination would have revealed the problem and they could recommend the firmware upgrade.

Its quite annoying. There are now, new dirvers for the BCM5700 series card on broadcom's website. Quite annoying to install.

Also, I've bonded the NIC cards under RH 3 and 4. Don't try active-active setup, it will explode networking rather badly.

Please don't forget to reward those that help you with some points. Makes ITRC more fun than the average peer to peer forum.

Countdown to Linux digit 5
300
SEP

[Phil Crooker]

Regarding Chad Slater's comment: the upgrades appear to me to only apply to the 5704 chipset (perhaps later as well). Thus this wouldn't help for those such as me that have the 5703.

Regarding the new bcm5700 driver, ver 8.3.14, it sounds hopeful, I'll be installing it tomorrow on one of our servers. I note broadcom have distributed the driver under GPL, so they are doing the right thing. Anyway no mention of the bug in question (no CHANGELOG) so I bit the bullet and filled in a service request on the broadcom website - the link is: http://www.broadcom.com/support/ethernet_nic/supportrequest.php
(you must use this form or they don't reply) - and asked if this new driver does in fact fix the problem.

Someone wrote back wanting details, etc, though I had already included that info as well as a link to this thread. He/She also said "this version had some fixes related to transmit timeouts / NETDEV WATCHDOG errors".

After repeating the requested info, I offered to post their reply here or invited them to post something, but no reply.

So if anyone who has been had the problem with earlier bcm5700 versions would like to chase it up with broadcom, go ahead. They said they would like to reproduce the problem. If the 8.3.14 does fix the problem, can you please post that here???

cheers.

[Sergio J. Larrondo]

Hi All,

Thanks to all the replies with the tips. I'll be upgrading the IMPI bootcode as recommended and the Broadcom driver to 8.3.14. I should know by the end of the week if the changes hold.


Thanks,
Sergio

[Sergio J. Larrondo]

I upgraded to the Broadcom v8.3.14 drivers and still had problems. I then upgraded the IPMI firmware this past weekend, and I haven't seen the problem yet. So the verdict is still out.

Sergio