iLo2 lost connection to blades

[TK8]

Hi

our firm is using HP xw460c and WS460 workstation blades and recently iLO on some of those blades lost connectivity to the blade H/W

it was random and there was no changes on the network side either.

Would anyone know what might cause this kind of behavior.

most of the blades on one chassis were fine while a few of them had this issue. reseating the blade resolved the issue for the affected blades though.

any insight into this would be much appreciated.

i have already checked the logs on the blade and iLo and there wasnt anything specific except for the array controller battery that was dying on the blade.

[SomeGuyWorking]

Are you running any vulnerability scanners.  I've seen some scanners with updates checking for the OpenSSL bug causing iLOs to crash requiring a power cycle of the host to bring the iLO back.

[BenoitVien]

We had the same issue this week-end.

Around 100 servers, blades and rackmount as been affected (all ILO2). Only the really old firmware version did not lose connexion.  Heartbleed security scan as been running in the same time the ILO become unresponsible.

For the Blades, doing a "reset server x" from a command line on the OA fix the issue, but this cause a cold reset of the server. We think that a rackmount servers need to be unplug from AC.

HP support confirmed the issue but dont have a better workaround than a reseat for now.

[Oscar A. Perez]

What iLO2 versions are not affected?  I need this info ASAP

[BenoitVien]

I have mixed result... The only one that are not affected are running 2.15 or 2.06, but thats not 100% of them that are OK.

[Oscar A. Perez]

Ok, I've reproduced the issue. I'm working on the fix right now.

[Oscar A. Perez]

I've patched the bug in the SSL library and I built an iLO2 v2.25 BETA.   You can download it from the link below.

ftp://ble3ding:H3artssl@ftp.usa.hp.com 

[TK8]

Thanks Oscar!
it seems that you're pretty clear on why this happened however, do you think having set up SIM on the network recently would have had anything to do with this issue?
even though it is deployed, we only have 1 chassis discovered on that SIM and that chassis had no issues whatsoever

Thanks

[Oscar A. Perez]

It has nothing to do with HPSIM

Ther bug was in the RSA SSLC library that iLO2 uses for its webserver.  For some reason, it doesn't happen every time the OpenSSL Heartbleed bug detection script is run against iLO.  

[Todd Lutz]

Thanks Oscar!  That seems to have fixed our problem.  Our ILO2s were crashing twice a day whenever IT did a vunerability scan.