FlexFabric FC Uplink outage when perform a OA Failover

[marcelkoedijk]

Hi There,

When we do a OA Failover test, like as remove active OA or disable OA UTP link. VirtualConnect FLexFabric cannot communicate with the OA, offcourse this appens but what happend more....

The VCM becomes to degraded status and the FC Uplinks comes to unknown status and disconnect. All server profiles (with FCOE HBA configured) comes to pending state, what only can corrected by a hard boot 9 (offcourse)... but they were all fine before the failover test, assigned, operational, etc. But when we do de failover test of the OA all server profiles come to " pending status".

Some Facts:

OA Firmware: 3.60

VC Firmware: 3.70

ILO4 Firmware: 1.101.10

554FLB Firmware: 4.1.450.7

2x enclosures C7000 Gen2, redundant OA, with total 4 Virtual Connect FlexFabric Modules (bay1+bay2 off both enclosure configured as one VC Domain)

Servers are al BL460Gen8 with the most ESXi 5.1 (but operating system or drivers are not related to this kind off issues)

Problem Description:

When we do the OA failover tests (timestamp 07-01-2013 3:43pm, vclogfile) on the enclosure BLE101 and BLE102 we see that Virtual Connect Manager is lost connection with the OA. This should be normal, but.

At the same time we lost all Fiberchannel uplinks ports in the enclosure BLE102 Virtual Connect port X1,X2 on each module. Fiberchannel Storage to all affected ESX Physical Hosts.

In VCM we see now "degraded status", stacking links to the BLE102 are failed, no comm, no communication with active OA module. All FCOE SAN uplinks on the BLE102 are " ?" status.

All Virtual Connect Profiles that have FCOE HBA configured are in "pending status" mode. A cold boot will needed. This profiles where 100% green and ok before we do de OA failover tests. Only the FCOE profiles have a "pending state".

What we did to restore:

First off all we bring the onboard administrators in redundant status (put the disconnect UTP cable back in de OA). After VCM was reconnected with the OA (automatically) the OA communication was OK. Stackinglink redundancy lost come back to normal. And automatically the Fiberchannel Uplink ports come back online as " V" Ok/Green. For the Server Profiles that where " pending" we do a cold boot so the profiles come normal. And after that we start the VM's manually.

Somebody knowns why this happens? Or has simulary issues?

See attechment for VCLOG

Note...this issue also appaers on 27-12-2012 and 04-01-2012, related to network (OA) outage because network maintencance, also here SAN! problems, strange?

[marcelkoedijk]

anyone recognize this problem ?

[papa--noel]

Hello,

It seems that we face the same problem. 2 weeks ago, all systems lost the connection to the storage system (no oa failover). Today (only one, we have to pray!) one host has lost the connectivity, lot of errors in the flex log. The host has lost is profile (profile pending statut, but no change has been made!).

AO: 3.60

Flex: 4.01

HPUX 11iv3

Support is still analyzing logs...

Guy