Virtual Connect FC issue

[Balde]

VC-FC lost communication with OA and VC Ethernet:

This issue began on July 2009, the blades originally came with:

OA firmware 2.32 on C3000 enclousure
VC firmware 1.31 (2 VC Ethernet and 2 4GB VC Fibre Channel)
8 Blades 460 G1 with ROM version I15 11/02/2008 & iLO2 1.70

We decided to upgrade the firmware of all blade c-class components so we did it.

First, with the firmware cd we did the blades
second by the OA GUI we upgrade OA 2.32 to 2.51
third also by VCM (GUI) we upgrade VC 1.31 to 2.10

All the process of firmware update was succesfull, no error messages, no warnings, no configuration lost, everything was cool. We used the blades and VC for several deliveries, until we had to turn off all the enclousures of the datacenter for aircooling manteinance. As we normally turn back on we noticed that the blades couldn't reach the SAN, we check all FC cables, Gbic's, switch and so on, we finally noticed that on VCM we had a warning for the VC-FC module:

VCETW280200PL vcmd: [FC:enc0:iobay3:4019:Major] FC Module state NO_COMM : Cannot communicate with component
VCETW280200PL vcmd: [FC:enc0:iobay3:4004:Info] FC Module power on
VCETW280200PL vcmd: [FC:enc0:iobay3:4011:Warning] FC Module state UNKNOWN : Component operational state cannot be determined

Also when we get OA information about bay3 (where VC-FC was plugged) didn't report Management IP Addr and Firmware version, temperature, status and everything else was ok.

We asked some field eng.for help and what to do in a case like this, they recomend us to downgrade the VC firmware little by little. We did it from VCFW 1.10, 1.15, 1.15b.... until 2.10. It didn't work
We checked SAW for information and ITRC, with no luck on a similar issue.
We tried FW update utility for VC in CLI mode
We already change those VC-FC's to other enclousures (C7000 and C3000), reset all the enclousure components and nothing changes

We are looking for information to discard that the firmware update process was the triger for this broken VC's, as with the same procedure other 2 VC-FC are operating normally.
What it could be? Could be the raise of temperature?, Could be voltage variations? The Virtual Connect Modules? Take in mind that 2 of the VC broke on a C3000 enclosure and other VC in other C3000 enclousure? Do you have any idea of what is happening? or how to restore functionallity of the VC-FC? How could we diagnose those VC do you have any tool? Does they have an internal "physical" reset, as we had tasted all resets available on the enclousure and on the VC Ethernet.

Tnks 4 help

[Adrian Clint]

Two things.

VC firmware 2.10 is a bundle comprising VC-Enet firmware (version 2.10) and VC-SAN firmware (1.34 I think). Check what version the VC-SAN firmware is at. If you update the VC firmware from a machine with FTP running it may not update the VC-SAN firmware. So check the firmware versions of VC-Enet and VC-SAN are of compatible versions. Try and use the VCFW update utility or the GUI or another machine if updating it does not work.

I've had issues with VC-SAN firmware being accepted but the VC-SANs dont activate it. What I do is putty into the OA, and then issue a restart/reset interconnect x command which will reboot the VC-SAN module and should reload the newer firmware and restablish comms with VC-enet. But make sure the top 2 VCs in bays 1 & 2 are not being reset at the same time - or they cannot communicate - is this what could have happened?

[The Brit]

This looks like the same problem posted by "Erick".

Additional comment

The latest OA firmware is 2.60, and the latest VC firmware is 2.30 (includes FC firmware 1.40).

As pointed out by Adrian, quite often (particularly with FC modules) the firmware loads but is not activated. I have noticed that a software reboot doesn't always activate the new FW, however a cold reboot (remove/replace) does.

Dave.

[Balde]

I already check that the FC module restarts, also i pull out from the enclousure wait a few seconds and put it back and nothing change. The big problem is that when i send the firmware by gui or using VC cli utility it says that the health of the module its OK, but it can't stablish communication (vcutil -a update OA password -l file). There's no way to send the firmware again to the module as it stucks at 2% by GUI, its something similar when you try to program a broken chip.

I'm going to try putty to OA to reset the FC module, I'll be back later.

[Balde]

VC Ether FW is at 2.10
VC FC FW is at 1.32

I updated both at the same time by GUI with the same .bin

[The Brit]

From your original post it sounds like there was a period (after the VCM upgrade to 2.10), when these modules were all functioning correctly. Can you confirm that?

It also seems that the problem began after the DC Power outage.

How long was the interval between the upgrade and the power outage??

Since it is a communication issue, my first instinct would be to check that the Enclosure Bay IP Addressing didn't get screwed up.

Dave

[Balde]

Yes all the VC modules were working, everything happened when we shutdown the enclousure a couple of hours and turn it back on. The servers are operating well, I'm working on firmware check and health status check...

No errors, no warnings when i applied the 2.10 firmware, the VC-FC stop "talking" with onboard and de VC-Ether.

[The Brit]

Check Enclosure Bay IP Addressing.

On OA, select "Enclosure Settings > Enclosure Bay IP Addressing". Select the "Interconnect Bays" tab.

Check that the FC modules are showing an IP address in the "Current Address" column, and that the addresses are in the same sub-net as the other IC modules and the OA's.

Dave.

[Balde]

Yes everything in the EBIPA it's ok, no duplicated address, the mask is in order... but the EBIPA assign no IP Addr to that module in particular, if you also change another VCFC module that works well, the EBIPA assign an IP addr, even when you change that module to different enclousures happen the same thing, no IP addr is assigned. I already erase OA and VC conf, but nothing change.

[The Brit]

It is beginning to sound like a bad module.

One final suggestion. I assume that you have two FC modules, one in bay3 (bad) and one in bay4 (good).

I suggest you remove both and swap. i.e. move Bay4 --> Bay3, and Bay3 --> Bay4.

I know it is illogical, however I have seen this work.

if the Good Module is placed in Bay3 and is still good, then Bay3 is OK. If the bad module is placed in bay4 and stays bad, then I would say that it is a bad Module.

However if the Good module is placed in Bay3 and goes bad, then I would say it is a bad bay, and you may have to have the backplane replace.

DAve.

[Balde]

As i told you before, i already moved that module to another bay, also to another enclousure, i know that 1 bad module isn't lucky, but 3 bad modules... something is going wrong.... the 3 bad modules broke on different enclousures, we already check for high temp, energy stuff, changed modules to different bays, downgrade firmware of VC, checked EBIPA IP assign, power off and on the entire enclousure, reset everything that have a reset.... the logs says:
VCETW280200PL vcmd: [FC:enc0:iobay3:4019:Major] FC Module state NO_COMM : Cannot communicate with component
VCETW280200PL vcmd: [FC:enc0:iobay3:4004:Info] FC Module power on
VCETW280200PL vcmd: [FC:enc0:iobay3:4011:Warning] FC Module state UNKNOWN : Component operational state cannot be determined

[The Brit]

Maybe I'm missing something here, but it seems to me that what you are saying is that

1. You have ONE failed VC FC module (In Bay 3).
2. Your other VC FC modules are functioning correctly
3. You have tried the same "bad" module in different bays, and different enclosures, and in each case, the module failed.

So the only question I have is

Have you tried a "Good" VC FC module in BAY 3, and if so, what was the result????

[Balde]

Yes, also in diferent bays on different enclousures

[The Brit]

So you tried a "Good" module in Bay 3 and it worked fine ??

Do you have any reason to think that the original module from Bay 3, i.e. the "Bad" module, is really "GOOD" ???

I have re-read this thread several times and it seems to me that this module has not worked, (in any enclosure bay), since the power was taken off the enclosure. Correct me if I am wrong!!

So why don't you replace it ???

Dave.

[norbert rudholzer]

Hello,

we have exactly the same Problem on 2  C3000 Enclosures.
The 4Gb VC Modul ( Part Number 409513-B22)
worked already ~4 Years (FW 2.00 VirtualConnect 4.10)

We had to remove one Modul from Bay 4.
After reinserting it into the same slot, we get "No Communication" an no management IP Adress.

EPIBA settings are OK

On another C3000 Enclosure with a functioning Module, there was the same.

We removed a working Module and reinserted it

And now we have 2 Modules not working in Slot 4

Any suggestions?

[jdixon]

We are having this exact same problem? Any updates from anyone? I know it has been a long time snice this was posted.

[peyrache]

What are OA/VC firmware versions ?
Shall you post OA logs please ?
OA is very sensitive to broadcast storm
Did you connect OA network on separate Lan (isolated )
JY

[WilliamSmith11]

Exactly the same issue

But in may case it happen when we try to restore a backup configuration.

We have two C7000,

We moved the two VC-FC to a different enclosure and dont work.

We downgrade de OA firmware  version and dont work,

Some suggestion ?

[Torsten.]

You have VC FC modules only. This cannot work. You need VC ethernet modules to manage them.

[WilliamSmith11]

In th attachment all module are inserted, same result

[WilliamSmith11]

With the four module in place same result

[Torsten.]

can you connect to the modules in bay 1 and 2 now?

there is no management link to the FC modules.

[WilliamSmith11]

Sure I can connect to module 1 or module 2 , but can not reach module 3 or 4 cause there no communication,

I have try to set iP address to the module 3 and 4 manually but it is not taken by the modules.

[Torsten.]

you cannot connect to these modules, the ethernet modules will control them.

[aisaddi]

I have also the same problem on a C7000. One SAN Module on Interconnect Bay 3 did not get a management IP.  So no communiction between Enclosue and Module and therefore we get no connection to SAN switch outside. I reseted, un-/replug and changed the SAN module. Nothing helps.

Did anyone get meanwhile a solution for that problem?

Regards

Karlheinz