EMC SAN Disk "NO_HW" on 11.31 - Fixed by reboot?

[Craig Johnson_1]

We're seeing this way too often on our 11.31 deployments. It happens for both Symmetrix and Clariion disk. "ioscan" finds the disk but often says "no_hw". A reboot clears the problem.

We cannot reboot production servers in order to add disk. What are we doing wrong?

Here's some output from the smh about one of the disks in question:

Detail View of Disk: 64000/0xfa00/0x8
-----------------------------------------------------------------------------------------------------------------
LUN H/W Path 64000/0xfa00/0x8
Persistent Device File /dev/disk/disk12
Legacy Device File(s) /dev/dsk/c31t0d0;/dev/dsk/c30t0d0;/dev/dsk/c32t0d0;/dev/dsk/c33t0d0;
Legacy H/W Path(s) 0/6/0/0/0/0/4/0/0/0.99.5.19.0.0.0;0/6/0/0/0/0/4/0/0/0.99.80.19.0.0.0;0/7/
Device Health offline
Device Identifier
Alias
World Wide ID 0x50060160bb20186150060160bb201861
Size 0 GB
Block Size (bytes)
Capacity (blocks)
Use unused
VG Name -
Device Type RAID
Instance Number 12
Total Path Count 4
Device Class disk
Driver esdisk
State UNOPEN
Vendor "DGC "
Product "LUNZ "
Serial Number "APM00091400581"
Firmware Revision "0428"
SCSI Protocol Revision

[Alzhy]

If you are NOT using Powerpath - there are SOME Symmetri and Clarions that are not supported natively by HP-UX 11.31's mass storage stack with respect to multipathing.

I thought I posted a few weeks ago a URL showing what Storage Arrays are officially supported under 11.31.

What are your EMC models?

[Mel Burslan]

Sounds like a SCSI reservation issue to me but I am not a SAN admin, so I can not assure you about that. Clariion series equipment that I had the displeasure of dealing with so far, always had initial operational issues with UNIX servers.

Feel free to call both HP and EMC support and explain them your dilemma and have them suggest some solutions.

[Craig Johnson_1]

We do use PowerPath.

[Craig Johnson_1]

Correction: We do NOT use PowerPath.

[Alzhy]

Monsieur.. give us your EMC Symmetrix and Clarion Models and we can tell you with "definiteness" if the 11.31 stack supports those arrays. NOT all EMC SAN arrays are supported sir.

[Craig Johnson_1]

Symmetrix DMX-2000, Symmetrix DMX-3 4500, CLARiiON NS-960

[Alzhy]

http://www.hp.com/products1/serverconnectivity/mass_storage_devices.html#EMC

Your EMC Arrays are all suported alright.

[Craig Johnson_1]

So, if they're supported, what is the problem?

[Alzhy]

Only areas I can think of are:

(1) Check the SAN --- find out from the Switch Logs if your FC HBA connections are getting logged off or have some disconnects. This should also be manifesting in syslog (dmesg) or /var/adm/syslog/syslog.log

(2) Verify (during times they work normally) if the disks are truly multipathed -- meaning: for each DSF persistend EMC device - /dev/rdsk/diskNNN -- that there are 2 devices. I can;t recall the syntax but it is one of the ioscan arguments. If your dsf devices are not multipathed -- then there may lie your issues. Consult EMC manuals to see if there are special "scsimgr" tweakables -- IF you are indeed NOT using PowerPath.

Question though -- what typical disk device do you usually use out of these EMC LUNs? Can you give a sample? Is it /dev/rdisk/diskNNN ?

[Alzhy]

For example -- on my rx6600 running HP-UX 11.31, 4 FC-HBAs hooked up to an HP EVA 8100.


# ioscan -k -m dsf /dev/rdisk/disk5634
Persistent DSF Legacy DSF(s)
========================================
/dev/rdisk/disk5634 /dev/rdsk/c14t0d4
/dev/rdsk/c18t0d4
/dev/rdsk/c12t0d4
/dev/rdsk/c16t0d4
/dev/rdsk/c34t0d4
/dev/rdsk/c32t0d4
/dev/rdsk/c28t0d4
/dev/rdsk/c30t0d4
/dev/rdsk/c48t0d4
/dev/rdsk/c52t0d4
/dev/rdsk/c44t0d4
/dev/rdsk/c46t0d4
/dev/rdsk/c64t0d4
/dev/rdsk/c66t0d4
/dev/rdsk/c62t0d4
/dev/rdsk/c60t0d4
# diskinfo /dev/rdisk/disk5634
SCSI describe of /dev/rdisk/disk5634:
vendor: HP
product id: HSV210
type: direct access
size: 524288000 Kbytes
bytes per sector: 512

It clearly shows my multipath works. SO I only need to use /dev/rdisk/disk5634 in LVM or VxVM or in any other storage usage. HP-UX 11.31's mass storage stack MPIO will take care of the load balancing and redundancy (ALUA in this case as this is an EVA)

[Craig Johnson_1]

Yes our multipathing works. We have two FC connections and I see two paths to each disk PLUS the agile address.

Using the agile address in LVM, that is what you see in /etc/lvmtab.

We HAVE to get this working. We cannot reboot boxes to see disk.

[Alzhy · Accepted solution]

Then CHECK your syslog / dmesg whenever you experience these episodes.

If you're lucky - it may tell what your issues really are.

Next - check your SAN- check the switch logs for port errors or if the switch ports serving your server's HBAs show a pattern of losing the connection(s).

[Alzhy]

Can you show us a sample of one of your disks in the above syntax showing indeed you have HP-UX 11.31 multipathing in play?

[Steven E. Protter]

Shalom,

rmsf -x

((11.31 only))

You should not need powerpath no matter what EMC says if the disk array is supported by HP-UX.

Making the OE current (September 2010) and installing the QPK is a very good idea.

Also make sure there are no srdf or clones inside the EMC sim frame holding the disks.

You might want to disable legacy paths as well.

SEP

[Craig Johnson_1]

OK, from dmesg:

class : lunpath, instance 3
Evpd inquiry page 83h/80h failed or the current page 83h/80h data do not match the previous known page 83h/80h data on LUN id 0x0 probed beneath the t
arget path (class = tgtpath, instance = 3) The lun path is (class = lunpath, instance 3).Run 'scsimgr replace_wwid' command to validate the change

class : lunpath, instance 3
An attempt to probe existing LUN id 0x0 failed with errno of 14.

class : lunpath, instance 11
Evpd inquiry page 83h/80h failed or the current page 83h/80h data do not match the previous known page 83h/80h data on LUN id 0x0 probed beneath the t
arget path (class = tgtpath, instance = 13) The lun path is (class = lunpath, instance 11).Run 'scsimgr replace_wwid' command to validate the change

class : lunpath, instance 11
An attempt to probe existing LUN id 0x0 failed with errno of 14.

class : lunpath, instance 12
Evpd inquiry page 83h/80h failed or the current page 83h/80h data do not match the previous known page 83h/80h data on LUN id 0x0 probed beneath the t
arget path (class = tgtpath, instance = 12) The lun path is (class = lunpath, instance 12).Run 'scsimgr replace_wwid' command to validate the change

class : lunpath, instance 12
An attempt to probe existing LUN id 0x0 failed with errno of 14.

class : lunpath, instance 4
Evpd inquiry page 83h/80h failed or the current page 83h/80h data do not match the previous known page 83h/80h data on LUN id 0x0 probed beneath the t
arget path (class = tgtpath, instance = 7) The lun path is (class = lunpath, instance 4).Run 'scsimgr replace_wwid' command to validate the change

class : lunpath, instance 4
An attempt to probe existing LUN id 0x0 failed with errno of 14.


We ran the "scsimgr wwid" command on the four then another "ioscan -fnNC disk" and voila, the disks were seen and claimed.

Another of our admins reports that this doesn't always work, but in this case it did.

[$t!$-]

it happened because of changes in zoning. You need to revalidate WWID for instance number which is already assigned to your lun.

with below command you can find out which are path or instance are showing auth_failed status

#scsimgr lun_map

After finding all instance number you need to revalidate those lun with below command
scsimgr â f replace_wwid â C lunpath â I

it will replace WWID with current one and make same path visible again.

[$t!$-]

replace command has not appear properly above ..pasting again

scsimgr -f replace_wwid -C lunpath -I