"Loop down" in SANSurfer software - but VC shows all up

[NJK-Work]

Our virtual SAN failed over last night (FalconStor IPStor) and two of our three newly deployed blade servers crashed with a BSOD.

In troubleshooting the problem I noticed that on the two problem servers, each show a "loop" down on one of the QLogic mezzanine ports in the SANSurfer management software. However, in the VC manager software, I show all 8 ports of our SAN VC are connected and have a "Green" status. My SAN administrator also shows all 8 ports connected and logged into the fabric.

Does anyone know why the Qlogic SANSurfer software would report loop down, yet the VC shows everything as healthy?

Thanks
Nelson

[JKytsi]

Hi,
check this advisory

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&objectID=c00874480&prodTypeId=12454&prodSeriesId=3232030

in our enviroments, loops are down if don't enable point-to-point. without vc everything works fine.

[NJK-Work]

Thanks.

Interesting, I had seen that setting in the SANSurfer software and tried changing it to Point-to-Point (without knowing about the advisory) but it had no affect.

So I rebooted the server, like it says in the advisory, and made the change in the Ctrl-Q utility and then did a cold boot. I still see no disks on port 1 of the HBA. On port 2 I still see the same disks.

So this did not fix my problem. But thanks.

Anyone have any other ideas?
Nelson

[JKytsi]

okay ...
sometimes the san switch port next from vc does not go to NPIV state.

check from both fabrics the ports you are using with vc. if the port is not in NPIV go back to qlogic bios and do the scan

[NJK-Work]

Hi, Thanks.

I am not sure what you mean by "san switch port next from vc" means. Could you clarify?

Also, my SAN administrator is telling me NPIV is a global setting - it cannot be set per port (it is either ALL on or ALL off for the entire switch). So if it were off, wouldn't my 3rd blade (the one that is still working) be broken too?

Thanks
Nelson

[JKytsi]

I mean the SAN switch where vc is connected =)

NPIV is enabled globally, but the port must "go" to NPIV state. You have all the uplinks connected ? The VC divides the server bays to FC uplinks. Ask the SAN admin to check if all the ports are in NPIV
"switchshow" will show it in brocade SAN

[NJK-Work]

Thanks. I will check with him.

Nelson

[JKytsi]

Hope it help ... I am fresh out of ideas, maybe double check zoning next.

[JKytsi]

Or the good-ol-update-vc-firmare =)

[NJK-Work]

Thanks.

Per the SAN administration, "Nothing changed on the switches". The only thing that happened was the primary IPStor failed over to the second then failed back. At that same time, 2 of 3 blades BSODed and now only see disk down one of the 2 HBA ports. He also said NPIV is enabled on all ports.

I appreciate the help.
Nelson

[NJK-Work]

I tried updating the firmware via the SANSurfer software, and rebooted, but it still shows the old (at least I think it is old) firmware after the reboot. Is there a trick to updating the firmware.

In SANSurfer it shows:

BIOS Version: 1.26
Firmware Version; 4.00.27

What I downloaded says:
"hpqlx24xxaf1.64risc4.00.90.zip"

So I am guessing the Firmware should say 4.00.90 after the update...right?

Nelson

[JKytsi]

damn =)

Is there anything visible if you scan storage from the affected blades qlogic BIOS (both ports) ?

[JKytsi]

Hi,

I think with Qlogic the firmware comes with the driver not in BIOS update. Anyway in FW update CD 8.20 has newer BIOS to HBA (2.02 or something)

[NJK-Work]

When I scan for fibre devices down both ports from the QLogic Ctrl-Q utility at boot up, only one port shows disk. Prior to the BSOD, both ports showed disks (I verified this on all three blades when I built them).

Nelson

[JKytsi]

Then I would start to look for errors in SAN switch ports (porterrshow or portshow [port])

counters can be cleared with statsclear, there should not be any erros. Usually errors are generated due loose connections in fibers (check by pressing FC connectors) or faulty cable.

How is the storage side ?