Virtual Connect FC issue

[Balde]

VC-FC lost communication with OA and VC Ethernet:

This issue began on July 2009, the blades originally came with:

OA firmware 2.32 on C3000 enclousure
VC firmware 1.31 (2 VC Ethernet and 2 4GB VC Fibre Channel)
8 Blades 460 G1 with ROM version I15 11/02/2008 & iLO2 1.70

We decided to upgrade the firmware of all blade c-class components so we did it.

First, with the firmware cd we did the blades
second by the OA GUI we upgrade OA 2.32 to 2.51
third also by VCM (GUI) we upgrade VC 1.31 to 2.10

All the process of firmware update was succesfull, no error messages, no warnings, no configuration lost, everything was cool. We used the blades and VC for several deliveries, until we had to turn off all the enclousures of the datacenter for aircooling manteinance. As we normally turn back on we noticed that the blades couldn't reach the SAN, we check all FC cables, Gbic's, switch and so on, we finally noticed that on VCM we had a warning for the VC-FC module:

VCETW280200PL vcmd: [FC:enc0:iobay3:4019:Major] FC Module state NO_COMM : Cannot communicate with component
VCETW280200PL vcmd: [FC:enc0:iobay3:4004:Info] FC Module power on
VCETW280200PL vcmd: [FC:enc0:iobay3:4011:Warning] FC Module state UNKNOWN : Component operational state cannot be determined

Also when we get OA information about bay3 (where VC-FC was plugged) didn't report Management IP Addr and Firmware version, temperature, status and everything else was ok.

We asked some field eng.for help and what to do in a case like this, they recomend us to downgrade the VC firmware little by little. We did it from VCFW 1.10, 1.15, 1.15b.... until 2.10. It didn't work
We checked SAW for information and ITRC, with no luck on a similar issue.
We tried FW update utility for VC in CLI mode
We already change those VC-FC's to other enclousures (C7000 and C3000), reset all the enclousure components and nothing changes

We are looking for information to discard that the firmware update process was the triger for this broken VC's, as with the same procedure other 2 VC-FC are operating normally.
What it could be? Could be the raise of temperature?, Could be voltage variations? The Virtual Connect Modules? Take in mind that 2 of the VC broke on a C3000 enclosure and other VC in other C3000 enclousure? Do you have any idea of what is happening? or how to restore functionallity of the VC-FC? How could we diagnose those VC do you have any tool? Does they have an internal "physical" reset, as we had tasted all resets available on the enclousure and on the VC Ethernet.

Tnks 4 help

[Adrian Clint]

Two things.

VC firmware 2.10 is a bundle comprising VC-Enet firmware (version 2.10) and VC-SAN firmware (1.34 I think). Check what version the VC-SAN firmware is at. If you update the VC firmware from a machine with FTP running it may not update the VC-SAN firmware. So check the firmware versions of VC-Enet and VC-SAN are of compatible versions. Try and use the VCFW update utility or the GUI or another machine if updating it does not work.

I've had issues with VC-SAN firmware being accepted but the VC-SANs dont activate it. What I do is putty into the OA, and then issue a restart/reset interconnect x command which will reboot the VC-SAN module and should reload the newer firmware and restablish comms with VC-enet. But make sure the top 2 VCs in bays 1 & 2 are not being reset at the same time - or they cannot communicate - is this what could have happened?

[The Brit]

This looks like the same problem posted by "Erick".

Additional comment

The latest OA firmware is 2.60, and the latest VC firmware is 2.30 (includes FC firmware 1.40).

As pointed out by Adrian, quite often (particularly with FC modules) the firmware loads but is not activated. I have noticed that a software reboot doesn't always activate the new FW, however a cold reboot (remove/replace) does.

Dave.

[Balde]

I already check that the FC module restarts, also i pull out from the enclousure wait a few seconds and put it back and nothing change. The big problem is that when i send the firmware by gui or using VC cli utility it says that the health of the module its OK, but it can't stablish communication (vcutil -a update OA password -l file). There's no way to send the firmware again to the module as it stucks at 2% by GUI, its something similar when you try to program a broken chip.

I'm going to try putty to OA to reset the FC module, I'll be back later.

[Balde]

VC Ether FW is at 2.10
VC FC FW is at 1.32

I updated both at the same time by GUI with the same .bin

[The Brit]

From your original post it sounds like there was a period (after the VCM upgrade to 2.10), when these modules were all functioning correctly. Can you confirm that?

It also seems that the problem began after the DC Power outage.

How long was the interval between the upgrade and the power outage??

Since it is a communication issue, my first instinct would be to check that the Enclosure Bay IP Addressing didn't get screwed up.

Dave

[Balde]

Yes all the VC modules were working, everything happened when we shutdown the enclousure a couple of hours and turn it back on. The servers are operating well, I'm working on firmware check and health status check...

No errors, no warnings when i applied the 2.10 firmware, the VC-FC stop "talking" with onboard and de VC-Ether.

[The Brit]

Check Enclosure Bay IP Addressing.

On OA, select "Enclosure Settings > Enclosure Bay IP Addressing". Select the "Interconnect Bays" tab.

Check that the FC modules are showing an IP address in the "Current Address" column, and that the addresses are in the same sub-net as the other IC modules and the OA's.

Dave.

[Balde]

Yes everything in the EBIPA it's ok, no duplicated address, the mask is in order... but the EBIPA assign no IP Addr to that module in particular, if you also change another VCFC module that works well, the EBIPA assign an IP addr, even when you change that module to different enclousures happen the same thing, no IP addr is assigned. I already erase OA and VC conf, but nothing change.

[The Brit]

It is beginning to sound like a bad module.

One final suggestion. I assume that you have two FC modules, one in bay3 (bad) and one in bay4 (good).

I suggest you remove both and swap. i.e. move Bay4 --> Bay3, and Bay3 --> Bay4.

I know it is illogical, however I have seen this work.

if the Good Module is placed in Bay3 and is still good, then Bay3 is OK. If the bad module is placed in bay4 and stays bad, then I would say that it is a bad Module.

However if the Good module is placed in Bay3 and goes bad, then I would say it is a bad bay, and you may have to have the backplane replace.

DAve.