OAs reboot on their own after 4.21 upgrade

[RUCNOK]

We have several HP C7000 BladeSystem enclosures and recently upgraded all of the Onboard Administrators (OAs) to 4.21 from 4.01.  After the upgrade, the OAs on 2 of our enclosures reboot on their own every 33 minutes exactly, sending out email alerts and causing unneeded alarm.

We contacted HP about the issue but they just keep scratching their heads.  Their only recommendation was to downgrade the OAs on these two enclosures back to 4.01 and then upgrade to 4.12 before upgrading to 4.21.  So we gave this a try... back at 4.01 the OAs never rebooted on their own... at 4.12 the OAs started rebooting again every 33 minutes and sending alerts and this also occured once we upgraded to 4.21 again.

We noticed in the OA System Log that just before the reboot occurs the following events are logged:

     - OA: DHCP Monitor: DHCPD or RADVD is not running. Restarting OA.
     - OA: Onboard Administrator is rebooting

All OAs (those affected and those that are not) are at hardware version B1 and part number 412142-B21.  Has anyone seen this issue before or have any suggestions on what can be done to correct it?  We'd like all of our OAs to be at the same firmware version (4.21) across the board.

[4dmir]

we got smth strange with OA 4.21 too
in about week after upgrade, SIM opened new case on OA's  without any understandeble info
ping of OA's shown packet loss (~10%), or even stop
on OA all blades shown critical, but acutally blades and interconnects worked
physicaly blades was "red" too

after reseting of OA's via web - we lost web access, only cli, and instable
changing of Active OA  few times lead to nothing


we unpluged both OA's, pluged OA1 - same story...
unplug OA1 - plug OA2, same story...

downgrade fw on OA1, while OA2 was unplaged - nothing
reseat one of blades - it started to show gren "front LED" but iLO, and interconnects - no ping no access

power off\on enclosure - iLO's started to ping and became accessible
in Encl settings - EBIPA settings were "default" so no IP's set etc
but iLO's shown IP and was accessible, interconnects - not

we set IP's to interconnects and OA's manualy via ebipa - and finaly got access to interconnects...


now evr works on 4.21 - hope there won't be any troubles anymore

[coeur_noir]

We are seeing the exact same thing.  Just started after the update to 4.21.  Has anyone given you an answer as to how to stop this?

[coeur_noir]

To add to it, our hardware version on the OAs are H0 and F0, but exact same behaviour - every thirty-three minutes, reboot with the same final error log message.

[DanBA]

Issue fixed by Version:4.30 (9 Sep 2014)

Problems Fixed

General

• In rare cases, the OA may fail to successfully identify server blades that have been either physically inserted into the enclosure or have been reset by executing the OA CLI RESET SERVER command.  When this issue occurs, the affected server blades are misidentified as an unsupported blade type.
• Restoring or configuring OA settings from a saved configuration script may fail to apply all network settings on the Standby OA module. The failing configuration script commands return the following error: “An error occurred while communicating with the other Onboard Administrator.” 
• The RSA key for SSH is not properly exchanged with the Standby OA during an OA failover event when Enclosure IP Mode is enabled. After an OA failover event, when logging into the OA, SSH clients using the Enclosure IP address to access the Active OA may display an RSA key fingerprint warning message or “man-in-the-middle” security breach warning message.  This issue only occurs on OA 4.1x or OA 4.2x versions of OA firmware when the OA has been factory reset or new SSH keys were manually generated.  
• The OA periodically reboots with the following entry logged in the OA system log:“OA: DHCP Monitor: DHCPD or RADVD is not running. Restarting OA.” 
• In an IPv6-only management network environment with the VLAN feature enabled, the CONNECT SERVER OA CLI command fails to connect to the server blade serial console with an error displayed similar to the following: Connecting to bay 1 ... iLO failed to respond: Interrupted system call (4)

[Server-Support]

My OA is on version 3.71, so can i just skip and update the firmware directly to version 4.30 with no issue or downtime ?

[Lincy_Zachariah]

Hi,

Yes you can directly go to version 4.30.
For the onboard administrator updates you do not require any downtime for the enclosure and that they can be performed any time.

[Server-Support]

Thanks Lincy.

Does updating the VC modules firmware using HP SPP bootable can also cause any outage to the network or fibre channel traffic on the enclosure ?

[Lincy_Zachariah]

Hi,
onboard administrator updates most of the time you do not require any downtime for the enclosure and that they can be performed any time. If you want to update the virtual connect, the first thing to do is run a health check. Once you validated that the virtual connect is healthy, you also need to make sure that your models are set up in a redundant configuration.

Thanks & regards,

Lincy

[Lincy_Zachariah]

Hi,

Blog on - Minimizing Downtime when Deploying Firmware and Driver Updates

http://h30507.www3.hp.com/t5/Coffee-Coaching-HP-and-Microsoft/Minimizing-Downtime-when-Deploying-Firmware-and-Driver-Updates/ba-p/106383#.VK_FTCuUdKZ

https://www.youtube.com/watch?v=DcHPe30Gz2s