2 x C7000 stop recognize BL680 G5 and BL460 G1 after OA Upgrade

[Ai90]

Dear Community Members,

I have two decent 9 years old C7000 chassis with OA version 3.30. One chassis is equipped with 8 x BL460 G1 blades, second chassis is equipped with 8 x BL680 G5 respectibely. VC connect version is 3.50 is on both chassis.

Both chassis have:

2x HP 1/10GB VC-ENET Enet Modules P/N 399593-B22

2 x HP 1/10Gb VC-Enet Module 2x10Gb 8x1Gb P/N 399593-B22

After update of OA to version 3.71, then to 4.01 and ending on 4.23 version, as last version compatible with BL460 G1 and BL680 G5 respectively, blades started to be not recognizable by noth chassis after next power cycle. They do not respond on power cycle button and indicating red flash light. From OA point of view blades are displayed as blank slots in chassis overview with unidentified hardware with no chance to control them. All blades were updated to last supported iLo and ROM versions using appropriate packages for both models.

As result I have all 8 x 680 G5 not recognized in first chassis with blinking flash lights, 5 out of 8 BL460 G1 not recognized on second Chassis. 3 BL460 G1 still running, because I did not reboot them yet.

To migitate the issue I tried the following on first Chassis with BL680 G5:

1. Extract and insert all 8 Blades and OAs.

2. Mixing the slots for Blades

3. Downgrade of OA to previous versions

4. Executing E-FUSE reset to all 8 Blades from SSH CLI.

5. Complete power down of Chassis and Powering On.

6. Inserting OA module with version 3.50 from foreign chassis and upgrading to different 3.X and 4.X versions.

To migitate the issue I tried the following on second Chassis with BL460 G1:

1. Extract and insert all 5 Blades and OAs

2. Mixing the slots for Blades

3. Downgrade of OA to previous versions

4. Executing E-FUSE reset to 5 Blades from SSH CLI. 

5. Setting Pin 6 on MB on reset position and inserting blades back to Chassis.

Can You kindly advise how to bring back to life these systems?

Thank you in advance.

[Ai90]

Some illustrations to the issue:

Please find below videos with LED from fully faulty Chassis 1 with BL680 G5 and Chassis 2 with partially faulty BL460 G1 respectively.

https://streamable.com/f84t8q

https://streamable.com/ffafo5

Thank You in advance for Your assistance and ideas.

[DANDKS]

You have mentioned that all the devices were removed & reinserted in both the Chassis. Not sure if the OA Tray/Sleeve was removed & reseated. If not done, this is highly recommended.

OA Tray/Sleeve is the device on which the OA is seated or installed. This is a critical component in communication between the OA & the Blades.

Refer to the below links for instructions

https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-c01578191

If the Tray was already removed & reinserted, we would recommend to raise a support ticket with HPE for further assistance as there could a possibility of parts replacement needed.

Thank you

[Ai90]

As you can notice from intial post, I already brought OA from different Chassis, which had different OA version, so extraction and insertion of OA modules was performed several times at least. I did this even with total power off of chassis and manual switchover between Active / Standby OA.

Unfortunately systems are quite old - support contract is expired. I am not sure that official support will help in this case.

[DANDKS]

Both the Enclosures may require a field service with or without part replacement. Hence, we recommend to contact the HPE Support team to raise a support ticket for further assistance. There are chargeable services for support on products which are out of warranty or support contract.

The described issue is looks complex which requires deep troubleshooting.

Thank you

[DANDKS]

Please perform the below steps:

a. Power Cycle the complete c7000 Enclosure.
b. I understand OA modules were Swapped between enlcosures, however There is a OA Tray which Holds the OA MOdules, Please Reseat the OA Tray / OA Sleeve. Not just the OA MOdules.

c. Get the OA Tray from a working Enclosure & install it in the problematic Enclosure for testing
d. As you said you have different c7000 enclosure from where you extracted the OA MOdules, can you move the servers to the working enclosure and check the status of the servesr?

If nothing works, try gathering OAsupport dump logs to check further.

https://support.hpe.com/hpesc/public/docDisplay?docId=c02818404

Thank you