I/O Stops in C-Class Blade Enclosure whith internal 4/24 Port Brocade Switches

[Oliver Brandt]

Hi there,

We have eight bl460c windows blades. Each comes with two dual core processors, 2 GB ram and dual Qlogic 4GB Fibre Channel.
The Enclosure has two interconnect LAN switches in bay 1 and 2 and two internal 4/24 FC-Switches in bay 3 and 4.
The C-Class blades are located in Slot 1,2,5,6 in the upper row and in Slot 9,10,13,14 in the lower row.
There is an EVA 4000 with XCS 6110 connected to the external switch-ports 17 and 18. All switch-ports are fixed to 4Gbit. When doing performance tests with IO-Meter on all eight blades we see performance problems and I/O stops on some of the blades.
Each blade has two LUNs and one worker for each LUN, each I/O-meter worker is doing about 25MB/s seq. write performance.
Results of the window of I/O-meter display an I/O-stop for some time, see attached sheet.

We see also lots of events in the application event log of the command view EVA server located in slot 1.

Date/Time Stamp: 17-Aug-2007/07:04:07
StorageCell WWN - Name: 5000-1FE1-5005-EFB0 BAMEVA01
Event Type: HSV Controller Event
Event Code and Description: 42044a08: CAC=4a - Fibre Channel Services Fibre Channel Port Link Failure A Fibre Channel port link has failed or a Drive Enclosure Environmental Monitoring Unit task has failed. A host Fibre Channel port has failed to respond.

Date/Time Stamp: 17-Aug-2007/07:03:45
StorageCell WWN - Name: 5000-1FE1-5005-EFB0 BAMEVA01
Event Type: HSV Controller Event
Event Code and Description: 09cdc305: CAC=c3 - Storage System Management Interface Entity State Change The state of a Storage System Management Interface entity has changed. A Fibre Channel port has transitioned to the FAILED state.

These Events occur 30 times per second and sometimes we get these "30 events a second" every second.

On the EVA the same Events occur and look like this:

07:04:07:007
17-Aug-2007
Controller B 09cdc305 #1340039
A Fibre Channel port has transitioned to the FAILED state.
Corrective action code: c3 More details

07:04:07:007
17-Aug-2007
Controller B 42044a08 #1340038
A host Fibre Channel port has failed to respond.
Corrective action code: 4a More details

This occurs on both controllers and both FC-ports. The Problem has been seen on three different C-Class Enclosures with different internal Brocade FC-Switches and on two different EVA 4000. The Problem seems to occur when using more than 4 Blades.

A workaround is to configure all blade FC-ports to 2Gbit. That solves the I/O hang and stop the problem. But the events above still occur.

Does anyone has the same problem?

[Mila Sojka]

Yes, we have the same problem.
We have similar configuration: EVA4100, c7000 with Brocade 4GBs FC switch, 4 x BL460c with Qlogic 4Gbs FC HBA.

It's very strange. In event log is written:" Fibre Channel Services Fibre Channel Port Link Failure A Fibre Channel port link has failed or a Drive Enclosure Environmental Monitoring Unit task has failed. A host Fibre Channel port has failed to respond"

But all FC port working fine and EMU is OK.
And also there is no warning or error in Command View GUI.

Does anybody knows, how to solve this problem?

[Oliver Brandt]

update the brocade sanswitch firmware to 5.3.0d

[Oliver Brandt]

switchfirmware 5.3.0d