ioscan shows NO_HW but disks available

[Ben Dehner]

I am in the process of setting up an RX6600 server to connect to an IBM N5200 disk array through a 2/16V (Brocade 3850) SAN.

The RX6600 has two HBAs, which are connected to seperate switches. The IBM has two controllers, which have two HBA's each, connected to the seperate switches. So when I map a single LUN from the array, I expect to see 4 hardware paths to access it. When I mapped my first LUN, it worked as I expected.

Now, however, two of my hardware paths are showing NO_HW in ioscan. However, 'pvdisplay' still can connect to the physical volume. If I create a volume group using only the 'NO_HW' hardware paths, it runs fine. So there seems to be a discrepancy between the ioscan status and the actual device availability.

I can't think of any changes that would have caused this. The allegedly missing paths belong to one of the RX6600 HBA's connected to one particular FC switch. I checked the swtich, and it shows F-port connections all around. The zoning hasn't been changed, and must have been correct at one time or I never would have seen the device in the first place. And since pvdisplay and LVM configurations can still access the device, the underlying fabric still must be valid.

Help??

[Sandman!]

Could you post the command and attach its output here. There could be a myriad of reasons: fabric zoning, LUN masking, true paths to name a few.

[Torsten.]

Hi,

did you use the "-H" flag with ioscan?

If you specify a device below the lba level with "ioscan -H ..." this will give you NO_HW - this is written in the man page.

Did you?

[A. Clay Stephenson]

I know what you are saying but I would prefer to know what the box itself is saying.

Please post the output of "ioscan -C disk -fn" and the output of diskinfo for the "NO_HW" /dev/rdsk/xxxx device nodes. Then post a pvdisplay output of one of these device nodes as well.

[Ben Dehner]

ioscan -fnC disk give:

Class I H/W Path Driver S/W State H/W Type Description
====================================================================================
disk 1 0/3/1/0/4/0.2.12.0.0.0.0 sdisk CLAIMED DEVICE NETAPP LUN
/dev/dsk/c5t0d0 /dev/rdsk/c5t0d0

disk 4 0/3/1/0/4/0.2.13.0.0.0.0 sdisk CLAIMED DEVICE NETAPP LUN
/dev/dsk/c7t0d0 /dev/rdsk/c7t0d0

disk 5 0/7/1/0/4/0.1.12.0.0.0.0 sdisk NO_HW DEVICE NETAPP LUN
/dev/dsk/c9t0d0 /dev/rdsk/c9t0d0

disk 6 0/7/1/0/4/0.1.13.0.0.0.0 sdisk NO_HW DEVICE NETAPP LUN
/dev/dsk/c10t0d0 /dev/rdsk/c10t0d0

As you can see, the hardware path 0/7/0/4/0 is allegedly not working. As a test, I physically disconnected the fiber card corresponding to hardware path 0/3/1/0/4/0. I can still access the LUN even though ioscan still shows all paths to this LUN as "NO_HW".

[Ben Dehner]

Should have put this in last post:

root:/home/btd>diskinfo /dev/rdsk/c10t0d0
SCSI describe of /dev/rdsk/c10t0d0:
vendor: NETAPP
product id: LUN
type: direct access
size: 1531024384 Kbytes
bytes per sector: 512

root:/home/btd>pvdisplay /dev/dsk/c10t0d0
--- Physical volumes ---
PV Name /dev/dsk/c10t0d0
PV Name /dev/dsk/c9t0d0 Alternate Link
VG Name /dev/vg01
PV Status available
Allocatable yes
VGDA 2
Cur LV 1
PE Size (Mbytes) 32
Total PE 46717
Free PE 46715
Allocated PE 2
Stale PE 0
IO Timeout (Seconds) default
Autoswitch On

[Ben Dehner]

Oops, looks like I typo'd in the previous post -- nonworking path is 0/7/1/0/4/0. Also, here's one last thing -- the VG that shouldn't be working:

root:/home/btd>vgdisplay -v vg01
--- Volume groups ---
VG Name /dev/vg01
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 1
Open LV 1
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 46723
VGDA 2
PE Size (Mbytes) 32
Total PE 46717
Alloc PE 2
Free PE 46715
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg01/apps
LV Status available/syncd
LV Size (Mbytes) 64
Current LE 2
Allocated PE 2
Used PV 1


--- Physical volumes ---
PV Name /dev/dsk/c10t0d0
PV Name /dev/dsk/c9t0d0 Alternate Link
PV Status available
Total PE 46717
Free PE 46715
Autoswitch On

[Sandman!]

Imho you should have only two paths (not four) to the storage LUNs since you have only two HBAs. One HBA is on 0/3/1/0/4/0 and another's on 0/7/1/0/4/0.

The fact that /dev/dsk/c9t0d0 is an alternate link to /dev/dsk/c10t0d0 is wrong as it creates a SPOF since both paths spring out of the same HBA. The other path should be out of your other HBA (on path 0/3/1/0/4/0) and through the alternate SAN switch. The probable device file for the alt link looks like /dev/dsk/c7t0d0.

~hope it helps

[Ben Dehner]

No, there should be 4 paths to the backend LUN. In front of the LUN there are two controllers; on each controller, there are two HBA's, so the target array has 4 ports connected to my SAN fabric. So the same LUN is being presented at 4 different PID's in the fabrics, and each host HBA can see two of these PID's because they are connected to different switches. You can also see this reflected in the hardware paths.

I have two seperate non-meshed switches, with domain ID's of 1 and 2. Since these switches are effectively seperate fabrics, there is no requirement that the domain ID be unique, but I prefer to assign them that way to aid in troubleshooting.

Host HBA 0/7 is conntected to Switch domain ID 1; the target HBA's are connected to ports 12 and 13 in this switch, leading to target hardware paths of
0/7/1/0/4/0.1.12.0.0.0.0
0/7/1/0/4/0.1.13.0.0.0.0

Host HBA 0/3 is connected to Switch domain ID 2; the target HBA's are connected to ports 12 and 13, leading to target hardware paths of
0/3/1/0/4/0.2.12.0.0.0.0
0/3/1/0/4/0.2.13.0.0.0.0

[Sandman!]

You have two paths between your fabric switch and the storage array but only one between the host HBA and the fabric switch. Sort of like a "fork in the road". However they are not true paths because if the host HBA goes down then you have lost that entire path to your LUN. So for the purpose of redundancy you should have only two paths to your LUNs.

[Ben Dehner]

Well, this maybe somewhat off topic, but hey ...

I agree that, if I lose a host HBA, all paths using that HBA will be down. So having 4 paths instead of 2 does nothing -- for that type of failure. However, what about other component failures -- say I lose an array HBA? With only two paths, that's equivalent to loosing a host HBA -- or losing an array controller entirely; I'm still running, but only with 50% IO bandwidth. However, four paths, I can still use both host HBAs and both array controllers.

And what about multiple component failures? Say I lose a fiber switch AND an array controller. Since both host HBAs see both array HBAs, I can still tolerate the failure.

[Ben Dehner]

Thanks to all who looked at this problem. Unfortunately, there seems to be no resolution at this point. One thing I did notice is that I could see the LUNs fine, but when I created a volume group using vgcreate, all of a sudden the LUNs would show NO_HW. After removing the volume group with vgexport, the LUNs would come back as CLAIMED.

I have opened a case with HP support on this, but so far no luck. The only thing I have heard is an obscure reference from NetApp (the IBM N5200 is an OEM'd NetApp) that this seems to be a problem with HP + NetApp. The only resolution I heard was that you have to re-boot the server to see new LUNs.

Again, thanks.

[Pradeep_29]

Hi Ben,

Did you get solution to this problem? I am facing similar problem as you.

Did you see any response from HP or NetApp?

Thanks,
Pradeep.

[Ben Dehner]

No, I didn't receive any resolution from either. After I rebooted the server, everything was visible as I expected. Since that time, I have not needed to modify the storage configuration, so I haven't seen whether or not the problem has recurred.

[Pradeep_29]

Yes, I am able to mount the filesystem and do reads/writes. I did copy large file into it.

Thx,
Pradeep.