iLO Dropping off Network

[Marc Eastburn]

I'm in the process of working this issue with HP Second level support, but I was wondering if anyone is having a similar issue .. so here it is.

I have 6 DL580g2's and 5 DL360g2's (all using the Integrated Lights Out Adapter) where the iLO's (all of them) drop off of the network pretty much every 2 days, the iLO has a link light, the Switch has a link light, but I cant get to the iLO any way at all, no ping ect. Our network engineering group has looked at the switches (Cisco 2950's), and said that the MAC address is no longer registered with the switch. Resetting the iLO through the Management Agents on the machine brings the iLOs back to life, then another 2 days, they go down again.
I have RIB's RIB II's and servers all plugged into the same swithces, not a single one of them is having any problems at all.

Anyone else having this problem?

[John Bolene]

I have a single DL360G2 running Ilo version 1.06 and it has no problems.

[Marc Eastburn]

Mine are running 1.15, installed ths version trying to fix the problem. I also just installed 1.20a, which was released on Friday. This did not fix the problem.

Thanks for letting me know

[Tom Mucha_1]

Not really sure if this would help, but you can try locking down the iLo and the switches to a set speed and duplex.

[David C_3]

Besides forcing the iLO and switches to a particular speed, sometimes the DL360 G2 hood cover will interfere with the latch on the CAT 5 cable that is plugged in to the iLO network port. Check to make certain that the hood on the DL360G2 doesn't interfere with your network cable connection.

[Marc Eastburn]

Tom and David,

Thanks for the reply, I have already tried setting the port speed and duplex, have verified that it is not the cables coming loose and this is happening on all of my ilo's not just the 360's. Compaq/HP has replaced mother boards on both a 360 and a 580 to see if this would help, still having the problem on all ilo's.

[Ron Kinner · Accepted solution]

Have you looked on the switch interface ports to see if they are showing any errors? Lots of switches will kill a port with too many errors. If you are not monitoring your console you might want to turn on snmp-server host and point it at something that can receive and store the logs. Kiwi Enterprises syslog will work on a windows box or just use the builtin feature on any UNIX or LINUX system. Sometimes the logs may tell you what is happening.

You could also try putting in a static mac on the switch for each iLO. Treating the symptom if not the problem.

Ron

[David C_3]

Would it be possible to put one of the servers on an isolated network using different networking equipment (maybe in a lab environment with a different model switch) to see if there is a compatibility issue between the switch and the iLO nic?

[Marc Eastburn]

Ron, we have a seperate networking group here that is also looking into the issue. There were some errors on the switches, but according to the Net Engineer, not significant enough to down the port. Also, if this is referring to the errDisable state, my understanding is that this would actually shut the port off, requiring a manual reset of the port, it would kill the link light as well. In my case, the link light on both ends is still on and resetting the iLO, not the switch, will bring it back up.

Interesting idea ... I might just ask out network folks about hard coding the MAC.

On another note, we just got in some DL380 g3's, they have the iLO as well, can anyone tell me if the iLO chip in the 380's is the same as in the 580's and 360's ?

[Marc Eastburn]

David, I tried plugging one of the iLO's in to one of our Core switches, still Cisco, but a different model, can't remember off hand which model. Still had the same problem. The only way I did not have a problem was with a crossover cable plugged directly into another server.

[Marc Eastburn]

Answered my own question .... the 380g3 and the 580g2 both have the same iLO chip part number ... bummer, looks like I'll have 2 more servers to add to the list.

[Ron Kinner]

Marc,

errDisable is what I was thinking about. It depends what version of the IOS you are running whether you have to reset the port or not. Newer version can have a timeout feature where after some interval they reset themselves. There is also a command

set option errport enable

which oddly enough disables the errDisable function which could be used to see if that was the problem. The thing always reports disabling a port to either the console or the snmp server. "If you are running CatOS 5.4(1) or later, there is a feature called errdisable-timeout which, if enabled, will tell you why a port was disabled. Here is an example

Cat5500> (enable) show errdisable-timeout
ErrDisable Reason Timeout Status Port ErrDisable Reason
------------------- -------------- ---- ----------------
bpdu-guard enable 11/1 bpdu-guard
channel-misconfig disable
duplex-mismatch disable
udld disable
other disable

Interval: 30 seconds"

The port does turn Orange tho when it is errdisabled.

http://www.cisco.com/warp/public/473/20.html

What sort of errors are they seeing on the ports? Really shouldn't see any errors if your cable is good.

What version of IOS are they running on the cisco switch?

What OS are you running on the servers?

Ron

[Marc Eastburn]

Ron,

Not sure what errors they are seeing, I don't remember him saying what they were. But I do remember that he seemed to think it was no big deal

IOS is 12.1.9 EA1

OS is WIN2K server sp3, plus lots of patches :-)
At one point we checked out a few of the ports that were affected, they were not in the errdisable state (according to my network group.) from my perspective, the lights were not orange.

We are getting ready to do a trace and send it in to Compaq/HP, just need to wait for the network group to free up some time.

[David Ogden]

Was there ever a resolution? We are having the same problem.
All our switches are Nortel.
Ilo drops off the network after non uniform period of time which can be as short as a few minutes.
Have upgraded server and Ilo to latest firmware and replaced motherboard - no difference.

[Marc Eastburn]

Yes, finally got a resolution last week. The problem was the network buffers were being used by and not released. HP sent me new firmware (1.27) that has solved my problem (not one has gone down since the upgrade).

I Just looked at the Driver download site and the 1.27 Firmware is not there yet. Maybe a call to support would get you the updated version.

[David C_3]

iLO f/w 1.27 was posted to the FTP site as SP21992.exe (ftp://ftp.compaq.com/pub/softpaq/sp21501-22000/SP21992.EXE)

[Jason Brooke]

I'm having the same issue with iLO's dropping off the network, with firmware 1.50 on some dl380's. Mine take about 1-2 weeks before it happens. If I have an onsite support guy go in, reboot the machine and enter the iLO f8 setup and change/save the network settings to cause the iLO to reset, it comes good for another 1-2 weeks.

[Dougall Lynch]

Hi,

I have the identical issue - ML370 G3. ILO is re-configured and works fine. By the time we actually need to use it the IP will not ping. Networks cannot locate the MAC address registered with the switch. Any solution/ideas would be appreciated. I have yet to log a response call on the box.