I/O hang when changing SAN configuration. Brocade switch, HP-UX 11.31, DMX4

[Alejandro Tuya]

Hi all,

Here is a quite odd problem we are facing...

We have a rx6600 running hp-ux 11.31 connected to a DMX4 through 2 fiber channel cables that go to a switch and then double path to the DMX4. That gives us 4 paths which are seen in the machine.

Now we want to change configuration and move both fiber channel cables to a different switch, in which zoning has been done appropriately.

We connect the first cable, enable the zone config and everything goes fine and we are able to see the disks, but when we connect the second one, ioscan, powermt (yes, using powerpath) and any other i/o related command hangs...

Fcmsutil indicates that physically the link is fine:

fcmsutil /dev/fcd2

Vendor ID is = 0x1077
Device ID is = 0x2422
PCI Sub-system Vendor ID is = 0x103C
PCI Sub-system ID is = 0x12D7
PCI Mode = PCI-X 266 MHz
ISP Code version = 4.0.90
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 4Gb
Local N_Port_id is = 0x010800
Previous N_Port_id is = 0x010800
N_Port Node World Wide Name = 0x50060b000069f731
N_Port Port World Wide Name = 0x50060b000069f730
Switch Port World Wide Name = 0x200800051e8312ea
Switch Node World Wide Name = 0x100000051e8312ea
Driver state = ONLINE
Hardware Path is = 0/7/1/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) fcd B.11.31.0709 Jun 11 2007

But the most amazing is that running a ioscan -k demonstrates that the system is able to see the disks, but seems not to be able to communicate with them neither to create special files...:

ioscan -fnkC disk
Class I H/W Path Driver S/W State H/W Type Description
===============================================================================
disk 1 0/0/2/1.0.16 UsbScsiAdaptor CLAIMED DEVICE USB SCSI Stack Adaptor
/dev/deviceFileSystem/Usb/MassStorage/dsk/disk@hp-1008+294=A60020000001
/dev/deviceFileSystem/Usb/MassStorage/rdsk/disk@hp-1008+294=A60020000001
disk 264 0/3/1/0.1.11.0.0.0.0 sdisk CLAIMED DEVICE EMC SYMMETRIX
/dev/dsk/c25t0d0 /dev/rdsk/c25t0d0

(...)

disk 263 0/3/1/0.1.15.0.1.0.2 sdisk CLAIMED DEVICE EMC SYMMETRIX
/dev/dsk/c23t0d2 /dev/rdsk/c23t0d2

(...)

disk 285 0/7/1/0.1.11.0.0.0.0 sdisk CLAIMED DEVICE EMC SYMMETRIX

(...)

disk 326 0/7/1/0.1.15.0.1.0.2 sdisk CLAIMED DEVICE EMC SYMMETRIX


Disks seen through HBA 0/3/1 seem to be fine but those seen through 0/7/1 do not seem to be ok...

Syslog provides the following message:

Oct 29 23:53:57 rxiq vmunix: 0/7/1/0: Fabric Name Server rejected GPN_FT query.
Oct 29 23:53:57 rxiq vmunix: Reason code: unable to perform request;
Oct 29 23:53:57 rxiq vmunix: Explanation: FC-4 Type not registered.
Oct 29 23:53:57 rxiq vmunix: This may indicate that the HBA port is not part of a zone.
Oct 29 23:53:57 rxiq vmunix: Please check zone configuration on your switch.

But still we are sure zoning is fine (in fact, disks can be seen...).

Tracing ioscan with tusc provides the following:

(...)

00:15:49 [16030] open(0x2000000040024e80, O_WRONLY|O_CREAT, 0664) ..................................................................... [entry]
00:15:49 [16030] open("/etc/ext_ioconfig", O_WRONLY|O_CREAT, 0664) .................................................................... = 5
00:15:49 [16030] open(0x2000000040024ea0, O_WRONLY|O_CREAT, 0664) ..................................................................... [entry]
00:15:49 [16030] open("/etc/ioconfig", O_WRONLY|O_CREAT, 0664) ........................................................................ = 6
00:15:49 [16030] lockf(5, F_LOCK, 0) .................................................................................................. [entry]
00:15:51 [16030] lockf(5, F_LOCK, 0) .................................................................................................. [sleeping]


We are in the middle of a change, so any help or any comment or idea that maybe useful will be surely appreciated (and of course assigned points).

Thanks indeed!

Alejandro

[Johnson Punniyalingam]

I would go with Basic ideas,

# ioscan -fn
# insf -e
# ioscan -fnC disk

or optional choice , I would take a reboot

HTH,

Johnson

[Uday_S_Ankolekar · Accepted solution]

It may be bad port on SAN switch or bad fibre cable. Swap the cable or port and see if it makes any difference

Good luck

-USA..

[Alejandro Tuya]

Hi! (finally found the solution...)

Johnson, as mentioned above, ioscan and any other i/o related commands will hang... rebooting will not fix the problem... Instead, if we do not disconnect the fibre cables, the system will not be able to start up...

Uday, of course it seemed like a problema in the HBA side of the switch, but it was not that after a few tries....we tried switching cables, zoning and many more...

The problem we were having is the link speed. For some reason it would not work when configured to 4 Gbps (despite other almost-the-same machines work fine at 4Gbps with the same storage...). The difference between the system that worked and the one giving us all te problmes was only the date of the fibre channel driver... The one that works is from Jan08... the one that does not work properly is from Jun07... both cases with 11.31

Might that be the cause?!?

THnaks

[Michael Steele_2]

Hi

with 11.31 pvlinks are no longer needed.
Just use the new device file, e.g.
/dev/disk/disk7
this covers all available links.
Check with

ioscan -m dsf