iLO2 2.05 FW update causing remote console issues

[mdcarson]

I updated 15 iLO2's manually to test the new firmware. We're experiencing one of the problems described as being fixed in the release notes, so I'm eager to roll it out to the remaining 350 servers using the Lights Out Migration Utility, but I'm getting mixed results.

I have three systems so far (DL380 G5 and DL385 G5 models) where the remote consoles no longer recognize keyboard or mouse input. This includes the integrated/activeX, java, and full screen options. The consoles launch OK but the CAD/hotkey buttons don't work and the security screen saver rolls uninterrupted when I hit a key or try to click inside the console.

All of the servers are running WS2K3 SP2 (32-bit) and had the 1.13.0.0 iLO 2 Management Controller Driver before the FW update was applied.

Rolling back to version 2.01 doesn't resolve the problem. I can't reboot the servers with problems until the weekend.

Has anyone else epxerienced this yet?

[c.wolf]

Hi,
I don't have this problem. I installed FW 2.05 and after first connect to remote console it installed a new version of dvc.cab under IE 8.0. At first klick CAD IE crashed. But now it works fine.
We use 1.15.0.0 iLO 2 Management Controller Driver

Do you use VNC and had set the option "Disable local inputs"? Inputs per iLO are "local inputs", ans they are not allow with this option when a connect with VNC already in use.

[Oscar A. Perez]

Could you please test if you can mount virtual media in one of those servers?

If Virtual Media doesn't work, it would mean that the Windows UHCI driver is in a bad state and you will have to reboot the server.

[mdcarson]

Thanks for the replies.

There are two iLO related drivers:
HP ProLiant Integrated Lights-Out Management Interface Driver for Windows Server 2003/2008 (Version 1.15.0.0)
HP ProLiant iLO 2 Management Controller Driver for Windows Server 2003 (Version 1.13.0.0)

I mentioned the management controller driver in the previous post because there were some issues with previous versions causing crashes. The problems servers have the latest version of each installed. (PSP 8.60)

We don't use VNC.

The browser version I'm connecting from doesn't seem to make a difference as I have some systems that work OK after the upgrade, and some that don't.

Virtual Media goes through the motions from the integrated remote console, but the mounted drives never show up in RDP sessions.

I'll wait until after the scheduled weekend reboots and see what happens.

[Oscar A. Perez]

About the issue. This is a driver problem. Microsoft fixed it in W2K8. We added a workaround in iLO2 2.05 to prevent the driver from getting into a bad state when iLO2 reboots but, you have to be at 2.05 firmware level to not experience the issue anymore.
When iLO2 versions 2.01 or earlier reboot, Windows UHCI driver could randomly stop responding. This leads to no virtual KB/Mouse, no virtual media. The other known symptom is Windows 2003 hanging on shutdown. So, I would expect you to see this hang when rebooting those servers.
Also, I'm interested in knowing how many of your servers are experiencing the issue. We could only reproduce it ~5% of the time.

[mdcarson]

Windows 2003 SP2 hanging at shutdown is the reason I was anxious to apply the new firmware. I was aware of the issue as reported in advisory c02101827, but was under the impression that the problem had been resolved in iLO2 FW 1.82.

Roughly 200 of our physical servers have iLO2's and 2003 SP2. Nearly all of our systems have scheduled weekly reboots. (Scheduled weekly reboots were originally configured for patch deployment flexibility.) We generally have 5 - 15 servers in this category that lockup at shutdown each week. In most cases there are no errors, they just stop at a solid gray screen with mouse cursor. Some are repeat offenders almost every week, others are random.

5% is only a small number if you're not the oncall admin receiving a dozen calls throughout your weekend!

To ensure consistency in our RILOE2/iLO/iLO2 configurations we have several scheduled tasks in Insight Manager that push settings to all management processors (~700) once each week. The 2.05 FW release notes mentioned "resetting iLO 2" is part of the cause, so I have temporarily disabled those tasks as well.

As long as the three iLO2's with problems work normally again after the scheduled reboots, I'll update the rest next week.

[Oscar A. Perez]

I wrote the advisory c02101827 and I even though the symptoms mentioned there are the same, the root cause is quite different.
In that case, it was iLO2 UHCI controller that halted forever after detecting a recoverable PCI error. This caused the Windows UHCI driver to sit there waiting forever for iLO2 UHCI to come back. So far, that issue has only been reported in DL385G5p and DL385G6 servers.
In the new case, the W2K3 UHCI driver doesn't like the fact that iLO2 goes for a reboot and its UHCI controller is unavailable for ~30 seconds. If during iLO2 reboot, the UHCI driver was in the middle of a task, the driver could hang. We figured a way to work around this in 2.05 but, you need to be at 2.05 level in order to not see this issue again.