DL360g5 iLO upgrade cause vsp to go unresponsive

[Patrick Hemmer]

I went and upgraded the iLO on all our g5 servers (mostly dl360) from various versions (all around 1.4) to 2.0. Well this had one drastic side effect, the virtual serial port went completely unresponsive.

All our servers are running linux, and we have the console= parameter as console=/dev/ttyS1, which is the virtual serial port.

This caused major havoc as anything which writes to /dev/console had problems. Some processes hung completely, others slowed down. A couple of these critical processes were named and syslog-ng. Also logging into the iLO and using the vsp didnt work. You could get to it, but the getty wouldnt respond (even after restarting it).

Rebooting the boxes resolved the issue, but I didnt want to have to reboot ~80 servers. Turns out there was another solution to this. If you run "echo h > /proc/sysrq-trigger" it caused everything to start working again. I'm guessing when the kernel tries to write out to the virtual serial port, it has a problem, and so it reinitializes it.

I'm just posting this so that anyone else who runs into this issue can have a workaround without having to reboot.

[Fran Garcia]

hey Patrick,

Good post! We have a similar setup here and this stuff is great to know. We usually do our firmware upgrades on bundles and we wait for a maintenance window to fully reboot the box, but it's not rare to do a single iLo update.

Thanks again :-)

ps: isn't there any way to add points to a *question* ?!?!? :-D