Unable to update OA Firmware

[bob_moodytx]

While updating our VC's to version 3.18 (elminate the DNS bug that got us) I was able to update VC-Bay2 without issue. Then when I went to update VC-BAY1 the udpate hung. I found out the VC lost credentials with the OA. So I went to the OA Enclosure Status and attempted to reestablish credentials. However a message came back and said the OA firmware needed to be at 3.11 or greater. Our current OA firmware was 3.10.

I downloaded the firmware and applied to one of our two C7000 chassis. It worked without issue (firmware level and hardware level are identical). When I tried to apply the update to the newer chassis it would acted like it loaded the FW patch then would hang and timeout. I attempted the update again using a telnet session and the update image command with a TFTP link. I could see the bin file load but then it hung and would eventually fail.

I rebooted each of the OA's several times and even swapped thier roles (Act to SB and SB to Act) with no success. I cant complete the VC updates without getting the OA updates first.

Both OA modules are type 456204-B21 BUT 1 is at harware level A3 and the other is at B1 (ISSUE? )

I thought of pulling them one at a time and trying to update. Will this work?

I need help ASAP ...

Thanks to all,

Bob M.

[chuckk281]

I will see if Monty can help you.

[bob_moodytx]

Thank you Sir. I appreciate any help that I can get.

[chuckk281]

Monty needs to see the OA syslog files for both the active and standby OAs. That will tell him what is going on. He indicated that he doesn't see OA firmware update issues so something must be going on.

[bob_moodytx]

Here is the Active Log

May 24 00:57:04  OA: Log cleared by LDAP xxx\xxxxxxxxx

May 24 02:16:23  Kernel: Network link is up at 1000Mbps - Full Duplex
May 24 02:16:28  OA: LCD Status is: OK.
May 24 02:16:29  Enclosure-Link: Service started
May 24 02:16:38  Enclosure-Link: Initial topology scan completed successfully
May 24 02:17:06  OA: Blade 1 is reporting nominal health status.
May 24 02:17:07  OA: Blade in bay #1 status changed to OK
May 24 02:17:07  OA: Blade 1 is now configured for Virtual Connect Manager.
May 24 02:17:15  OA: Blade 2 is reporting nominal health status.
May 24 02:17:16  OA: Blade in bay #2 status changed to OK
May 24 02:17:16  OA: Blade 2 is now configured for Virtual Connect Manager.
May 24 02:17:16  OA: Blade in bay #2 status changed to OK
May 24 02:17:16  OA: Blade in bay #1 status changed to OK
May 24 02:17:17  Redundancy: Onboard Administrator is switching to Active mode
May 24 02:17:17  Redundancy: Onboard Administrator has now completed the Standby to Active transition
May 24 02:17:30  OA: Blade 3 is reporting nominal health status.
May 24 02:17:30  OA: Blade in bay #3 status changed to OK
May 24 02:17:30  OA: Blade 3 is now configured for Virtual Connect Manager.
May 24 02:17:39  OA: Blade 4 is reporting nominal health status.
May 24 02:17:39  OA: Blade in bay #4 status changed to OK
May 24 02:17:39  OA: Blade 4 is now configured for Virtual Connect Manager.
May 24 02:17:41  OA: Blade 5 is reporting degraded health status.
May 24 02:17:41  OA: Blade in bay #5 status changed from OK to Degraded
May 24 02:17:41  OA: Blade in bay #5 status changed to Degraded
May 24 02:17:41  OA: Blade 5 is now configured for Virtual Connect Manager.
May 24 02:17:44  OA: Blade 6 is reporting nominal health status.
May 24 02:17:44  OA: Blade in bay #6 status changed to OK
May 24 02:17:44  OA: Blade 6 is now configured for Virtual Connect Manager.
May 24 02:17:46  OA: Blade 7 is reporting nominal health status.
May 24 02:17:46  OA: Blade in bay #7 status changed to OK
May 24 02:17:46  OA: Blade 7 is now configured for Virtual Connect Manager.
May 24 02:17:47  OA: Blade in bay #7 status changed to OK
May 24 02:17:48  OA: Blade 8 is reporting nominal health status.
May 24 02:17:48  OA: Blade in bay #8 status changed to OK
May 24 02:17:48  OA: Blade 8 is now configured for Virtual Connect Manager.
May 24 02:17:49  OA: Blade in bay #8 status changed to OK
May 24 02:17:52  OA: Blade 13 is reporting nominal health status.
May 24 02:17:52  OA: Blade in bay #13 status changed to OK
May 24 02:17:52  OA: Blade 13 is now configured for Virtual Connect Manager.
May 24 02:17:54  OA: Blade 14 is reporting nominal health status.
May 24 02:17:54  OA: Blade in bay #14 status changed to OK
May 24 02:17:54  OA: Blade 14 is now configured for Virtual Connect Manager.
May 24 02:18:04  OA: Blade in bay #3 status changed to OK
May 24 02:18:13  OA: Blade in bay #4 status changed to OK
May 24 02:18:15  OA: Blade in bay #13 status changed to OK
May 24 02:18:15  OA: Blade in bay #14 status changed to OK
May 24 02:18:16  OA: Blade in bay #5 status changed to Degraded
May 24 02:18:17  OA: Blade in bay #6 status changed to OK
May 24 02:20:51  OA: bobm logged into the Onboard Administrator from 10.16.xxx.xxx
May 24 02:24:31  OA: Failed to copy firmware image to the Standby Onboard Administrator
May 24 02:24:31  OA: The Active and Standby Onboard Administrators may not be communicating reliably
May 24 02:24:31  OA: Consider resetting the Standby and/or Active Onboard Administrators and retrying
May 24 02:24:31  OA: Failed flashing Standby Onboard Administrator
May 24 02:24:31  OA: Please inspect the system log on the Standby Onboard Administrator for further details
May 24 02:24:31  OA: Active Onboard Administrator will not be flashed
May 24 02:28:14  OA: Authentication failure for user Administrator from 10.16.xxx.xxx, requesting sshd
May 24 02:28:27  OA: Failed login attempt with user Administrator from 10.16.xxx.xxx
May 24 02:28:52  OA: bobm logged into the Onboard Administrator from 10.16.xxx.xxx
May 24 02:32:05  OA: bobm logged out of the Onboard Administrator
May 24 02:51:40  OA: bobm logged into the Onboard Administrator from 10.16.xxx.xxx
May 24 02:55:18  OA: Failed to copy firmware image to the Standby Onboard Administrator
May 24 02:55:18  OA: The Active and Standby Onboard Administrators may not be communicating reliably
May 24 02:55:18  OA: Consider resetting the Standby and/or Active Onboard Administrators and retrying
May 24 02:55:18  OA: Failed flashing Standby Onboard Administrator
May 24 02:55:18  OA: Please inspect the system log on the Standby Onboard Administrator for further details
May 24 02:55:18  OA: Active Onboard Administrator will not be flashed
May 24 02:57:26  OA: bobm logged out of the Onboard Administrator
May 24 02:58:43  OA: bobm logged out of the Onboard Administrator
May 24 06:23:53  OA: LDAP dc1\rtadm logged into the Onboard Administrator from 10.16.xxx.xxx
May 24 06:33:27  OA: LDAP dc1\rtadm logged out of the Onboard Administrator
May 24 06:33:39  OA: LDAP dc1\rtadm logged into the Onboard Administrator from 10.16.xxx.xxx
May 24 06:49:09  OA: Protocol Link Loss Failover was enabled by user LDAP xxx\xxxx
May 24 06:52:35  OA: Link Loss Failover interval set to 30 seconds by user LDAP xxx\xxxx.
May 24 09:46:32  OA: bobm logged into the Onboard Administrator from 10.16.xxx.xxx

[bob_moodytx]

Here is the Standby Log

[bob_moodytx]

Attachments arent working and the log is to large to post (>20,000 chars)

[bob_moodytx · Accepted solution]

!! PROBLEM RESOLVED !!

I was able to successfully update the OA's on our C7000. While reseaching the issue I came across the following article

http://h30499.www3.hp.com/t5/HP-BladeSystem-Management/Onboard-Administrator-OA-Hardware-Versions-and-Firmware-Sync/td-p/2349325

There was a statement made in the response that clicked:

"The OA with KVM has a new hardware architecture that makes it incompatible for the FW upgrade proces as a ‘sync’ feature from the OA itself."

We do infact have the OA's with the KVM and yes i can attest that either the web based or the telnet method of updating these OA's does not work if they are both online.

I went to our server center and pulled the standby OA and ran the firmware update. Ran like a charm. When I reinserted the offline OA they came back as degraded (difference in the OA's firmware) . I then pulled the OA I updated and logged on to the remaining OA. I performed the same update without any errors or issues.

When I reinderted the OA, they came back and reported being at firmware version 3.30.

I proceeded to check on the status of the VC modules (recall trying to update these is what started all this). The VC/OA credentials had automatically been reestablished.

When I tried to update the remaining VC it told me that a previous update had not finished. I pulled and reinserted the VC module and when it came back online both were at 3.18.

Thanks to everyone for looking, and I hope this helps someone out.

Bob M

[chuckk281]

Great news Bob. I will forward to Monty that we are good for something.

Chuck

[bob_moodytx]

Chuck,

LOL.. not sure if you noticed or not but the post I referred to was yours anyway .. .

Sooooooo.. Heres another Kudos.

I really appreciate you being on the forum and responding so quick.

Bob