Autopath: Claimed devices failed

Henrik Rasmussen · ‎06-12-2013

On our EVA8000 I have two disk groups, one called "Default Disk Group" with 300 GB Fibre Channel Disks, and one called "FATA Disk Group" which is 1TB Near-Online FATA disks.

Our topology consists of two SAN switches cross-connected to both the two figure interfaces on the EVA and two HBAs on each server (redundant setup). Therefore each LUN are

presented to each of the servers 8 times (8 paths). Both servers are configured in the same member zone in the SAN switches.

I've just added a new LUN to the "Default Disk Group", presenting it to the HP-UX 11.23 servers (two servers in a Service Guard Cluster). I have been running ioscan and insf.

I have done this procedure many times before, adding the new LUNs to the LVM config, extending filesystems.

As usual, the devices (8 paths) are created and is Claimed (to debug the problem I unpresented the LUN which was confirmed by NO_HW on the device files). Re-presenting them

again also showed that they are Claimed. This confirms that the servers are actually able to see the LUN.

However, when I tried issuing the pvcreate on the device (no matter which of the 8 paths I chose), pvcreate displayd an error message: pvcreate: Couldn't open physical volume"/dev/rdsk/cXXt0d2": No such device or address.

When I run 'autopath display' it shows "Failed" on each 8 paths associated with some of the LUNs including the new one I lust presented, but Active for each of the 8 paths on other LUNs. And it is the same on the other server in the cluster.

I've tried deleting the new LUN (by rmsf -H device files prior to deletion), re-adding it, presenting it, and run ioscan and insf, but to no avail. Issuing autopath discover doesn't help either.

No systems seem to be affected by it, however I can't extend a logical volume as I can't pvcreate the new device. The SAN switches seems to be okay. If it was a fibre problem, only some path's to each LUN would be affected, not each path on only some LUNs.

The new LUN is managed by controller A on the EVA, the same controller as LUNs that do not have the problem LUNs that autopath shows as active).

Concentrating on one of my VGs, it contains 4 LUNS which autopath shows as active. On the EVVA they are all configured Default Disk Group, Vraid5, Read cache on, Mirrored,

Write cache: Write-back, and in LUNs of 30GB, 165GB, 25GB, 60GB, 100GB, 25GB, and the new LUN is 100GB configured as the other's.

I can't seem to figure out what is wrong. Anyone?

Torsten. · ‎06-12-2013

Maybe the access id not allowed after the unpresenting/presenting again. Anything related in syslog?

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.
__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!

Henrik Rasmussen · ‎06-12-2013

The problem was there before unpresenting. No associated entries prior to unpresenting. But when I unpresented (after removing the special files), the syslog of course recorded I/O errors. Also problem is not only regarding the unpresented LUN, but several others already being presented for months. I also tried re-adding the new LUN with a new free LUN number, but this didn't solve it either. Here is syslog info and some autopath outout

[root] artemis:/var/adm/syslog> grep vmunix syslog.log.0

Jun 10 09:34:27 artemis vmunix: 0/2/1/0.2.8.0.0.0.2 sdisk

Jun 10 09:34:27 artemis vmunix: 0/2/1/0.2.14.0.0.0.2 sdisk

Jun 10 09:34:27 artemis vmunix: 0/2/1/0.2.12.0.0.0.2 sdisk

Jun 10 09:34:27 artemis vmunix: 0/2/1/0.2.10.0.0.0.2 sdisk

Jun 10 09:34:30 artemis vmunix: 0/5/1/0.1.8.0.0.0.2 sdisk

Jun 10 09:34:30 artemis vmunix: 0/5/1/0.1.11.0.0.0.2 sdisk

Jun 10 09:34:30 artemis vmunix: 0/5/1/0.1.9.0.0.0.2 sdisk

Jun 10 09:34:30 artemis vmunix: 0/5/1/0.1.10.0.0.0.2 sdisk

Jun 10 09:47:07 artemis vmunix: DIAGNOSTIC SYSTEM WARNING:

Jun 10 09:47:07 artemis vmunix: The diagnostic logging facility has started receiving excessive

Jun 10 09:47:07 artemis vmunix: errors from the I/O subsystem. I/O error entries will be lost

Jun 10 09:47:07 artemis vmunix: until the cause of the excessive I/O logging is corrected.

Jun 10 09:47:07 artemis vmunix: If the diaglogd daemon is not active, use the Daemon Startup command

Jun 10 09:47:07 artemis vmunix: in stm to start it.

Jun 10 09:47:07 artemis vmunix: If the diaglogd daemon is active, use the logtool utility in stm

Jun 10 09:47:07 artemis vmunix: to determine which I/O subsystem is logging excessive errors.

Jun 10 09:47:19 artemis vmunix: DIAGNOSTIC SYSTEM WARNING:

Jun 10 09:47:19 artemis vmunix: The diagnostic logging facility is no longer receiving excessive

Jun 10 09:47:19 artemis vmunix: errors from the I/O subsystem. 2 I/O error entries were lost.

Jun 10 09:53:21 artemis vmunix: 0/5/1/0.1.10.0.0.0.2 sdisk

Jun 10 09:53:21 artemis vmunix: 0/5/1/0.1.11.0.0.0.2 sdisk

Jun 10 09:53:21 artemis vmunix: 0/5/1/0.1.8.0.0.0.2 sdisk

Jun 10 09:53:21 artemis vmunix: 0/5/1/0.1.9.0.0.0.2 sdisk

Jun 10 09:53:21 artemis vmunix: 0/2/1/0.2.8.0.0.0.2 sdisk

Jun 10 09:53:21 artemis vmunix: 0/2/1/0.2.10.0.0.0.2 sdisk

Jun 10 09:53:21 artemis vmunix: 0/2/1/0.2.12.0.0.0.2 sdisk

Jun 10 09:53:21 artemis vmunix: 0/2/1/0.2.14.0.0.0.2 sdisk

Jun 10 14:01:06 artemis vmunix: 0/2/1/0.2.12.0.0.3.6 sdisk

Jun 10 14:01:08 artemis vmunix: 0/2/1/0.2.10.0.0.3.6 sdisk

Jun 10 14:01:08 artemis vmunix: 0/2/1/0.2.14.0.0.3.6 sdisk

Jun 10 14:01:08 artemis vmunix: 0/2/1/0.2.8.0.0.3.6 sdisk

Jun 10 14:01:08 artemis vmunix: 0/5/1/0.1.8.0.0.3.6 sdisk

Jun 10 14:01:08 artemis vmunix: 0/5/1/0.1.11.0.0.3.6 sdisk

Jun 10 14:01:08 artemis vmunix: 0/5/1/0.1.9.0.0.3.6 sdisk

Jun 10 14:01:08 artemis vmunix: 0/5/1/0.1.10.0.0.3.6 sdisk

Jun 10 14:28:50 artemis vmunix: LUN deleted succesfully..Porceeding to update KRS -600508B400074442000170000CBE0000

Jun 10 14:30:23 artemis vmunix: AUTOPATH: Device 600508B400074442000170000CBE0000 added back

Jun 10 14:31:12 artemis vmunix: WARNING: Unable to configure alternate path for dump devices.

Jun 10 14:32:04 artemis vmunix: WARNING: Unable to configure alternate path for dump devices.

Jun 10 14:33:39 artemis vmunix: WARNING: Unable to configure alternate path for dump devices.

Jun 10 14:39:14 artemis vmunix: WARNING: Unable to configure alternate path for dump devices.

Jun 10 14:39:58 artemis vmunix: 0/5/1/0.1.11.0.0.0.2 sdisk

Jun 10 14:39:58 artemis vmunix: 0/5/1/0.1.8.0.0.0.2 sdisk

Jun 10 14:39:58 artemis vmunix: 0/5/1/0.1.9.0.0.0.2 sdisk

Jun 10 14:39:58 artemis vmunix: 0/5/1/0.1.10.0.0.0.2 sdisk

Jun 10 14:39:58 artemis vmunix: 0/2/1/0.2.10.0.0.0.2 sdisk

Jun 10 14:39:58 artemis vmunix: 0/2/1/0.2.12.0.0.0.2 sdisk

Jun 10 14:39:58 artemis vmunix: 0/2/1/0.2.8.0.0.0.2 sdisk

Jun 10 14:40:02 artemis vmunix: 0/2/1/0.2.14.0.0.0.2 sdisk

Jun 10 14:40:11 artemis vmunix: AUTOPATH : PPM Failing path <CB0E3600> error returned <5>

Jun 10 14:40:11 artemis vmunix:

Jun 10 14:40:11 artemis vmunix: AUTOPATH : Path 0xbc0e3600 failed! Rerouting to alternate path

Jun 10 14:40:11 artemis vmunix: AUTOPATH : PPM Failing path <CB213600> error returned <5>

Jun 10 14:40:11 artemis vmunix: AUTOPATH : Path 0xbc213600 failed! Rerouting to alternate path

Jun 10 14:40:11 artemis vmunix: AUTOPATH : PPM Failing path <CB123600> error returned <5>

Jun 10 14:40:11 artemis vmunix: AUTOPATH : Path 0xbc123600 failed! Rerouting to alternate path

Jun 10 14:40:11 artemis vmunix: AUTOPATH : PPM Failing path <CB233600> error returned <5>

Jun 10 14:40:11 artemis vmunix: AUTOPATH : Path 0xbc233600 failed! Rerouting to alternate path

Jun 10 14:40:11 artemis vmunix: AUTOPATH : PPM Failing path <CB253600> error returned <5>

Jun 10 14:40:11 artemis vmunix: AUTOPATH : Path 0xbc253600 failed! Rerouting to alternate path

Jun 10 14:40:11 artemis vmunix: AUTOPATH : PPM Failing path <CB273600> error returned <5>

Jun 10 14:40:11 artemis vmunix: AUTOPATH : Path 0xbc273600 failed! Rerouting to alternate path

Jun 10 14:40:11 artemis vmunix: AUTOPATH : PPM Failing path <CB293600> error returned <5>

Jun 10 14:40:11 artemis vmunix: AUTOPATH : Path 0xbc293600 failed! Rerouting to alternate path

Jun 10 14:40:11 artemis vmunix: AUTOPATH : PPM Failing path <CB2B3600> error returned <5>

Jun 10 14:40:11 artemis vmunix: AUTOPATH : Path 0xbc2b3600 failed! Rerouting to alternate path

Jun 10 14:40:11 artemis vmunix: AUTOPATH : All the paths to the end device 0xbc2b3600 failed!