All blades offline in c3000

[Swarm]

Hi guys,

We just had an issue in which all of our blades had a red X and status unknown in our c3000 enclosure.

OA Firmware 3.6

Any ideas?  Log is below

Jul 26 05:45:44 Kernel: hub 1-1:1.0: over-current change on port 2
Jul 26 05:45:44 Kernel: hub 1-1:1.0: over-current change on port 2
Jul 26 05:45:45 Kernel: hub 1-1:1.0: over-current change on port 2
Jul 26 05:45:45 Kernel: hub 1-1:1.0: over-current change on port 2
Jul 26 05:46:28 Kernel: Network link is up at 1000Mbps - Full Duplex
Jul 26 05:46:30 OA: Time zone changed to SGT-8
Jul 26 05:46:32 OA: LCD Status is: OK.
Jul 26 05:46:32 OA: Device 2 power delayed during device discovery.
Jul 26 05:46:33 OA: Blade in bay #2 status changed to OK
Jul 26 05:46:33 Enclosure-Link: Service started
Jul 26 05:46:40 Enclosure-Link: Initial topology scan completed successfully
Jul 26 05:46:42 OA: Device 2 power delayed until potential partner devices are discovered.
Jul 26 05:46:44 Redundancy: Service started (ACTIVE)
Jul 26 05:46:44 OA: Onboard Administrator booted successfully
Jul 26 05:46:57 OA: HTTP server is up
Jul 26 05:47:01 OA: VLAN is disabled
Jul 26 05:47:50 OA: Internal health status of interconnect in bay 1 changed to OK
Jul 26 05:49:38 OA: Management Processor on Blade 1 appears unresponsive.
Jul 26 05:49:43 OA: Partner device 2 was denied power because the corresponding device is missing.
Jul 26 05:49:56 OA: Management Processor on Blade 4 appears unresponsive.
Jul 26 05:50:20 OA: Management Processor on Blade 8 appears unresponsive.
Jul 26 05:51:52 OA: Blade 1 has been allocated a default power value of 509W because iLO appears unresponsive.
Jul 26 05:51:52 OA: Blade 4 has been allocated a default power value of 509W because iLO appears unresponsive.
Jul 26 05:51:52 OA: Blade 8 has been allocated a default power value of 509W because iLO appears unresponsive.

[Torsten.]

The OA cannot "see" the ILOs.

I would first check and restart the OA, then the ILOs (blades) if needed.

[Swarm]

I did restart the OA but unfortunately this did not do anything for the blades.

I had to unplug the power on the enclosure, wait, then plug it back in.

Not sure if a newer firmware fixes this?