ILO2 Disabled and Can't enable on DL380-G6 - anyone else?

[Jabkowski Matt]

On February 5th the ILO2 chips were disabled on five of my servers. We were not working on the servers. Two DL380-G6's, Two DL360-G5's and one ML370-G5 all have the same problem. THe problem is more obvious on the DL380-G6's becasue the cooling fans all went to hight speed. One of the tech support specialists tells me this is because all environemntal controls on the G6's have been moved to the ILO chip.

Is anyone else having this problem?

They are trying to change the motherboards but I can't believe 5 motherboards on different model servers and different OS's would all go bad at the same time.

[Jabkowski Matt]

And what is GCC?

The next day after this happend I was instructed to upload HPS Reports on all servers and upload them to an FTP site. It has been 3 days and I can't get an status update as to what if anything is being done with the reports or if anyone is even working on it. All I get is an email from a customer support person that the reports are uploaded, have been sent to GCC and they will keep me posted. But I never getting any updates.

[Wallace H Wyeth]

Yes we are having the same issue. For no apparent reason, the fans on all our G6 servers went to high speed last Friday, 4/23/2010.

Have you been able to resolve your issue?

[JPM79]

We began seeing this on all our G5s and G6s on 5/8. All apparently went offline at about the same time according to our logs, and is affecting Windows 2003/2008 (with event 57 showing up in the event logs) as well as vSphere 4 hosts. The problem appears to be limited to our native VLAN. Our other iLO 2 hosts at other sites are unaffected, so this leads me to believe that there is some sort of rogue network device causing the issue. If I shut down a given server, remove power completely, unplug iLO, and power it back on the server functions normally until I reconnect it to our native VLAN. iLO also functions if I cross-connect it directly to a laptop. Did you ever find a resolution to your issue? I have not contacted HP yet but plan to first thing next week. The 1.82 firmware has not helped.

[Wallace H Wyeth]

We are still working on this issue. HP has asked us to update the firmware on our switches which we plan to do later this week.

In the interim, we have connected the iLO ports of all our G6 servers to an older HP Procurve 2848 switch. We are not seeing the iLO lockup when connected to this older technology.

Please keep this thread updated if you learn more.

Wallace

[Wallace H Wyeth]

We have also applied the 1.82 update and as JPM stated it did not resolve the issue.

JPM I agree that you should log this with HP. Each additional report will help them learn more about the cause of this failure.

Can you post what switches you are using?

[JPM79]

Do you have a case # I can reference when calling?

I am seeing the issue when I connect an iLO 2 interface to any port configured for the native VLAN on our Cisco 6513 running IOS v 12.2(33)SXH2a or a Cisco 3548 runnign IOS v 12.0(5)WC3b. I did verify that if I place the iLO connection on another other than the native VLAN then the card continues to respond. However, as soon as I connect it to the native VLAN port the fans go high speed and it stops responding, even if it doesn't have a valid IP for the native VLAN. My next step will be to try placing a hub between the iLO and the switch.

I am wondering if spanning tree on the Cisco is coming in to play (we have everything set to portfast default and no ports are blocking, but I am wondering if the HP iLO 2 cards don't like the bpdus.)

[Wallace H Wyeth]

HP has created several cases but this seems to be the central case number.

CASE:4613617133

We do not have spanning tree enabled, but this a PoE switch. I wonder if PoE is in any way involved.

The most interesting thing is that most of these servers have been working fine for a long time. Nothing obvious changed at the time of failure. Clearly this is happening to a number of HP servers so what is the common factor?

[JPM79]

Thank you, hopefully this case # will convince the HP rep I am on hold with to assist me (they are trying to say that it is a network issue). I am having the issue on a non PoE switch (the Cisco 3500XL), but also observer it on the PoE line cards in our 6513.

[JPM79]

HP is saying that case is closed and that it was an issue the switch. No other details available.

[Wallace H Wyeth]

Please post your case number so I can share it with the engineers working our case.

Thanks,

[JPM79]

My case # is 4614588237. I ended up speaking with a supervisor and they agreed to elevate the case, but since they believe it is a network issue they promised nothing. Thanks.

[Kedd Guyton]

I noticed that you've tried updating the iLO 2 firmware to v1.82, but have you tried also updating the iLO 2 Management Controller driver to v1.12.0.0 (or later)?

The Event ID 57 is also mentioned in the HP advisory:
http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01802766&dimid=1053525778&dicid=alr_apr10&jumpid=em_alerts/us/apr10/all/xbu/emailsubid/mrm/mcc/loc/rbu_category/alerts

[JPM79]

On our DL360 G6's I have upgraded the system BIOS to 2010.03.30, the iLO2 firmware to 1.82, the HP ProLiant Integrated Lights-Out Management Interface Driver to 1.15.0.0B, and the HP ProLiant iLO 2 Management Controller Driver to 1.13.0.0, with no luck. The issue also affects our VMWare vSphere 4 servers, for which no driver updates are available/applicable.

[JPM79]

We managed to resolve our issue by tracing the problem down to a bad Cisco SD2005 unmanaged switch plugged in to a port on our core Cisco 6513 on the native VLAN.  We only found this by methodically isolating components of our network.  We began by isolating our access switches from our core switch to rule them out one by one.  We then went through and began disabling groups of ports on the 6513 until we identified the port range that the problem was coming from.  From there, we identified each device on the problem range and went about disabling each port one by one until we found the culprit.  The port on the core switch gave no indiciations that there were any troubles and the devices on the SD2005 were functioning fine.

GigabitEthernet5/17 is up, line protocol is up (connected)
  Hardware is C6k 1000Mb 802.3, address is 001e.be89.5368 (bia 001e.be89.5368)
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseT
  input flow-control is off, output flow-control is on
  Clock mode is auto
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:26, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 6000 bits/sec, 9 packets/sec
     190225 packets input, 15654604 bytes, 0 no buffer
     Received 168431 broadcasts (5370 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 2 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     1427293 packets output, 136340580 bytes, 0 underruns
     0 output errors, 0 collisions, 3 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

[Wallace H Wyeth]

That is very interesting.  I will look for possible correlations in our network.

We can avoid the issue by movling the ILO connections to an older HP switch rather than plugging them directly to the 5412.