topic Re: OAs reboot on their own after 4.21 upgrade in BladeSystem - General

OAs reboot on their own after 4.21 upgrade

RUCNOK — Wed, 07 May 2014 19:21:26 GMT

We have several HP C7000 BladeSystem enclosures and recently upgraded all of the Onboard Administrators (OAs) to 4.21 from 4.01. After the upgrade, the OAs on 2 of our enclosures reboot on their own every 33 minutes exactly, sending out email alerts and causing unneeded alarm.

We contacted HP about the issue but they just keep scratching their heads. Their only recommendation was to downgrade the OAs on these two enclosures back to 4.01 and then upgrade to 4.12 before upgrading to 4.21. So we gave this a try... back at 4.01 the OAs never rebooted on their own... at 4.12 the OAs started rebooting again every 33 minutes and sending alerts and this also occured once we upgraded to 4.21 again.

We noticed in the OA System Log that just before the reboot occurs the following events are logged:

- OA: DHCP Monitor: DHCPD or RADVD is not running. Restarting OA.
- OA: Onboard Administrator is rebooting

All OAs (those affected and those that are not) are at hardware version B1 and part number 412142-B21. Has anyone seen this issue before or have any suggestions on what can be done to correct it? We'd like all of our OAs to be at the same firmware version (4.21) across the board.

Re: OAs reboot on their own after 4.21 upgrade

4dmir — Mon, 12 May 2014 09:44:17 GMT

we got smth strange with OA 4.21 too
in about week after upgrade, SIM opened new case on OA's without any understandeble info
ping of OA's shown packet loss (~10%), or even stop
on OA all blades shown critical, but acutally blades and interconnects worked
physicaly blades was "red" too

after reseting of OA's via web - we lost web access, only cli, and instable
changing of Active OA few times lead to nothing

we unpluged both OA's, pluged OA1 - same story...
unplug OA1 - plug OA2, same story...

downgrade fw on OA1, while OA2 was unplaged - nothing
reseat one of blades - it started to show gren "front LED" but iLO, and interconnects - no ping no access

power off\on enclosure - iLO's started to ping and became accessible
in Encl settings - EBIPA settings were "default" so no IP's set etc
but iLO's shown IP and was accessible, interconnects - not

we set IP's to interconnects and OA's manualy via ebipa - and finaly got access to interconnects...

now evr works on 4.21 - hope there won't be any troubles anymore

Re: OAs reboot on their own after 4.21 upgrade

coeur_noir — Tue, 27 May 2014 20:54:00 GMT

We are seeing the exact same thing. Just started after the update to 4.21. Has anyone given you an answer as to how to stop this?

Re: OAs reboot on their own after 4.21 upgrade

coeur_noir — Tue, 27 May 2014 20:59:00 GMT

To add to it, our hardware version on the OAs are H0 and F0, but exact same behaviour - every thirty-three minutes, reboot with the same final error log message.

Re: OAs reboot on their own after 4.21 upgrade

DanBA — Mon, 15 Sep 2014 06:41:30 GMT

Issue fixed by Version:4.30 (9 Sep 2014)

Problems Fixed

General

In rare cases, the OA may fail to successfully identify server blades that have been either physically inserted into the enclosure or have been reset by executing the OA CLI RESET SERVER command. When this issue occurs, the affected server blades are misidentified as an unsupported blade type.
Restoring or configuring OA settings from a saved configuration script may fail to apply all network settings on the Standby OA module. The failing configuration script commands return the following error: “An error occurred while communicating with the other Onboard Administrator.”
The RSA key for SSH is not properly exchanged with the Standby OA during an OA failover event when Enclosure IP Mode is enabled. After an OA failover event, when logging into the OA, SSH clients using the Enclosure IP address to access the Active OA may display an RSA key fingerprint warning message or “man-in-the-middle” security breach warning message. This issue only occurs on OA 4.1x or OA 4.2x versions of OA firmware when the OA has been factory reset or new SSH keys were manually generated.
The OA periodically reboots with the following entry logged in the OA system log:“OA: DHCP Monitor: DHCPD or RADVD is not running. Restarting OA.”
In an IPv6-only management network environment with the VLAN feature enabled, the CONNECT SERVER OA CLI command fails to connect to the server blade serial console with an error displayed similar to the following: Connecting to bay 1 ... iLO failed to respond: Interrupted system call (4)

Re: OAs reboot on their own after 4.21 upgrade

Server-Support — Sun, 28 Dec 2014 23:11:19 GMT

My OA is on version 3.71, so can i just skip and update the firmware directly to version 4.30 with no issue or downtime ?

Re: OAs reboot on their own after 4.21 upgrade

Lincy_Zachariah — Mon, 05 Jan 2015 12:27:57 GMT

Hi,

Yes you can directly go to version 4.30.
For the onboard administrator updates you do not require any downtime for the enclosure and that they can be performed any time.

Re: OAs reboot on their own after 4.21 upgrade

Server-Support — Mon, 05 Jan 2015 13:31:48 GMT

Thanks Lincy.

Does updating the VC modules firmware using HP SPP bootable can also cause any outage to the network or fibre channel traffic on the enclosure ?

Re: OAs reboot on their own after 4.21 upgrade

Lincy_Zachariah — Fri, 09 Jan 2015 12:16:56 GMT

Hi,
onboard administrator updates most of the time you do not require any downtime for the enclosure and that they can be performed any time. If you want to update the virtual connect, the first thing to do is run a health check. Once you validated that the virtual connect is healthy, you also need to make sure that your models are set up in a redundant configuration.

Thanks & regards,

Lincy

Re: OAs reboot on their own after 4.21 upgrade

Lincy_Zachariah — Fri, 09 Jan 2015 12:18:36 GMT

Hi,

Blog on - Minimizing Downtime when Deploying Firmware and Driver Updates

http://h30507.www3.hp.com/t5/Coffee-Coaching-HP-and-Microsoft/Minimizing-Downtime-when-Deploying-Firmware-and-Driver-Updates/ba-p/106383#.VK_FTCuUdKZ

https://www.youtube.com/watch?v=DcHPe30Gz2s

Re: OAs reboot on their own after 4.21 upgrade

Server-Support — Fri, 09 Jan 2015 12:50:43 GMT

Wow,

I never knew that there is a validate option in HP Blade VC manager page. So is that test safe to be executed without causing any downtime to the production environment ?

Because I need to know if the VC module has been configured to be redundant or not.

Re: OAs reboot on their own after 4.21 upgrade

Lincy_Zachariah — Sat, 10 Jan 2015 01:49:34 GMT

Hi,

No this will not cause downtime.

Healthcheck Command - List the state of modules in the enclosure. This command is useful if an Administrator needs to know if performing an action, such as a failover or a VC or OA firmware upgrade, would cause an outage.

VSCU user guide - https://h20565.www2.hp.com/hpsc/doc/public/display?sp4ts.oid=3794423&docId=emr_na-c04384256&docLocale=en_US

http://h20564.www2.hp.com/hpsc/doc/public/display?docId=mmr_kc-0108485

Re: OAs reboot on their own after 4.21 upgrade

Server-Support — Sat, 10 Jan 2015 03:25:39 GMT

Many thanks for the helpful assistance and guidance Lincy.

I really appreciate your reply.

Have a good day.

Re: OAs reboot on their own after 4.21 upgrade

Server-Support — Tue, 28 Apr 2015 04:34:26 GMT

Version 4.40 has just been released

http://h20564.www2.hp.com/hpsc/swd/public/detail?sp4ts.oid=3188477&swItemId=MTX_dc5dc1fa9051424caa148ee23a&swEnvOid=4168#tab-history

Has anyone tried to upgrade into that version yet ?

Re: OAs reboot on their own after 4.21 upgrade

Server-Support — Thu, 25 Feb 2016 00:11:33 GMT

FYI,

I've done the update using manual firmware upload from the HP BladeSystem Onboard Administrator page.

It was working well with no issue to do the upgrade durin business hours. No production server was impacted by this upgrade.

Hope this helps.