OA firmware upgrade disrupts fibre channel

[Sarah Nordstrom]

I am watching this thread, all of our SAN switches are Brocade 4Gb 4/24. We have one chassis with FC pass-thrus instead of SAN switches, but it's a critical production system so I can't risk upgrading the OA on it right now.

[Andrew C. Brown]

Another possible factor: Most recently we had two enclosures exhibit this behavior about an hour or so apart. The enclosures were chained together via the link cable, which may explain how the problem jumped but is hardly a smoking gun.

Interestingly another enclosure in the same stack had no issues. Of note is the fact that the two affected enclosures are both OA hardware rev A1. None of our other enclosures have A1 rev OA's in both slots, although one enclosure has a rev A1 in the secondary slot.

Sarah: Regarding your production enclosure, I would try to upgrade it sooner rather than later. The reason for our 2.25 upgrade was an iLO bug that caused the OA's in the two aforementioned enclosures to spontaneously reboot, which in turn caused the FC issue. This bug is supposedly fixed in 2.25.

Of course I should point out that we haven't yet upgraded the iLO of the second affected enclosure, because when we upgraded the first enclosure to 2.25, the problem happened AGAIN. :P

[Sarah Nordstrom]

The chassis I experienced it on has a hardware rev. A1 OA in the left slot, and a rev. A0 OA in the right slot. I've upgraded two chassis so far (one c7k and one c3k) but only one has SAN switches in it, and it experienced the issue. All ours are linked with the link cable. We have seen the issue you're talking about in 2.25 also (the OA spontaneously rebooting), but in our case it hasn't caused the FC flap issue (yet).

[Sarah Nordstrom]

Actually, upon rereading your post it looks like I misread. We are already running 2.25 on all c-class chassis, and still experience the spontaneous reboot. We are upgrading to 2.31 to attempt to solve that as well as problems when linking c7000s with c3000s via the Enclosure-Link.

[Sarah Nordstrom]

After going back and forth a few times all we're getting out of HP is that their techs are aware of this but don't know why it happens yet. We've been told to do all OA updates in maintenance windows now.

[Andrew C. Brown]

Sarah,

Yes, I suspect HP is aware that our cases are related, as they told us that they don't know what's going on yet but that they knew of one other customer who was affected.

I know you're using the Brocade 4/24 switches, can you tell me what software version you're running on the problem enclosure(s)? We're at 5.3.0a right now, and I'm wondering if that might be a factor. Enclosures at another datacenter are running 6.1.0a and I don't believe they've been affected by this issue.

[Sarah Nordstrom]

Ours is also at 5.3.0a.

[Raghuarch]

Sarah,

Can you try updating your SAN Firmware to 6.1.0 and check do you see the problem.
I think you can't go straight to 6.1.0 you may need to update to 6.0.1a before.
SAN Firmware may be the Problem.

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareIndex.jsp?lang=en&cc=us&prodNameId=3185341&prodTypeId=3709945&prodSeriesId=3185340&swLang=8&taskId=135&swEnvOID=54

[Sarah Nordstrom]

The only other chassis I have available won't be able to be upgraded until at least March, unfortunately. I can't risk another production outage at this time.

[Andrew C. Brown]

I can confirm that Brocade OS 5.3.0a is the issue. I successfully replicated the problem on a test enclosure here by downgrading the Brocade switches to 5.3.0a and then performing a firmware upgrade on the OA's to force a reboot. I did the upgrade/reboot twice and the link flaps occurred each time.

I then upgraded to 6.0.1a and repeated the test. No link flaps.