HP BL460c Gen8 Reboots / goes unresponsive

[SaadLiaqat]

Hi, 

We have a RHEL 5.7 (x64) running with oracle RAC (3 nodes). 

We are continuesly facing issues where one random RAC node goes unresponsive . If I access it from iLo I still dont see anything (blank window). 

We have updated the iLo and System ROM, but have faced this issue again. 

Here is what I see on the IML logs.

ASR Detected by System ROM 9/13/2014 1:37PM 9/13/2014 1:37PM
An Unrecoverable System Error (NMI) has occurred (System error code 0x0000002B, 0x00000000) 9/13/2014 1:35PM 9/13/2014 1:35PM

If I look into the /var/log files, I dont see anything..... (16:40 was around the time I manually power reset the blade as it was un responsive.)

Sep 13 07:53:36 gsm-rac3 ntpd[13202]: synchronized to 172.31.17.1, stratum 1
Sep 13 16:40:08 gsm-rac3 syslogd 1.4.1: restart.
Sep 13 16:40:08 gsm-rac3 kernel: klogd 1.4.1, log source = /proc/kmsg started.

We previously had a similar setup with RHEL 5.4 with oracle RAC on bl465 gen6 , never encountered this problem

(this was the one replaced with gen8 blades and according to HP the infiniband was not under support anymore).

We also have a running RHEL5.7 Oracle RAC (2 nodes) running on bl460c Gen7 blades.. also never encountered this problem

apparently this problem started hapening with gen8 blades only. 

Has anyone faced this issue with bl460c gen8 blades?

any recommendation on how we can further debug this issue?

deeply appreciate all the help

Thanks

Saad 

[RbBsmn]

What iLO version do you have? There is a fix for this in iLo version 1.51.

[SaadLiaqat]

We recently upgraded to iLO 1.51, but faced this issue again... 

[RbBsmn]

Uhm, that's interesting. iLo 2.0 Was recently released, but I doubt this will resolve this as I don't think it's listed as a fix.
I suppose an NVRAM clear on the 3 blades has already been suggested?
Especially since you are experiencing it on multiple blades I would contact HP support and request them to elevate this to L2.

[SaadLiaqat]

Case is already excalated to L2 by HP... But they havent found anything yet... 

[RbBsmn]

If you don't mind, could you send me the case number as a private message?
I would love to follow the indept analysis of this for future references.

[Server-Support]

Yes please, and update us with the resolution.

I also suffer from the same problem where my HP BL 465c G7 configured as Windows Oracle RAC node 1 and 2 intermittently loses its connection to the shared storage.

Rebooting the physical blade server weekly is the work around so far before the problem happens.

[RbBsmn]

Did you ever get this resolved?

[Server-Support]

Hi,
For me the problem resolved after I apply the firmware upgrade on all of the blades in the enclosure into the latest edition firmware from HP SPP 2014.09 ISO file. All components of the Blade servers are now up to date and no problem any more.