rp7410 va7100 array high i/o wait and general question about va7100

[Jared Rudy]

We have a RP7410 with HPUX 11iv1 installed and two VA7100 arrays. Long story short, we've been experiencing high i/o waits as seen from running a sar command (average 35% to 40% %wio). I also noticed some SCSI read errors from 5 disks in logs. None of the disks are showing faulty so I though that all 5 disks bad was highly unlikely. Further investigation showed all those disks on the same i/o hardware address.

Couple questions.

First, each of our VA7100 arrays are full with 15 hard drives each. We have dual controller configuration. When I do an ioscan one controller address shows 12 disks and the other 13 disks for one array. I would think that each controller would show 15 disks right? Here's the output from ioscan:

# ioscan -funC disk |grep "0/0/6/0/0.8"
disk 1 0/0/6/0/0.8.0.110.0.0.0 sdisk CLAIMED DEVICE HP A6188A
disk 29 0/0/6/0/0.8.0.110.0.0.1 sdisk CLAIMED DEVICE HP A6188A
disk 31 0/0/6/0/0.8.0.110.0.0.2 sdisk CLAIMED DEVICE HP A6188A
disk 32 0/0/6/0/0.8.0.110.0.0.3 sdisk CLAIMED DEVICE HP A6188A
disk 33 0/0/6/0/0.8.0.110.0.0.4 sdisk CLAIMED DEVICE HP A6188A
disk 34 0/0/6/0/0.8.0.110.0.0.5 sdisk CLAIMED DEVICE HP A6188A
disk 35 0/0/6/0/0.8.0.110.0.0.6 sdisk CLAIMED DEVICE HP A6188A
disk 36 0/0/6/0/0.8.0.110.0.0.7 sdisk CLAIMED DEVICE HP A6188A
disk 37 0/0/6/0/0.8.0.110.0.1.0 sdisk CLAIMED DEVICE HP A6188A
disk 40 0/0/6/0/0.8.0.110.0.1.1 sdisk CLAIMED DEVICE HP A6188A
disk 42 0/0/6/0/0.8.0.110.0.1.2 sdisk CLAIMED DEVICE HP A6188A
disk 44 0/0/6/0/0.8.0.110.0.1.3 sdisk CLAIMED DEVICE HP A6188A
# ioscan -funC disk |grep "1/0/6/0/0.8"
disk 5 1/0/6/0/0.8.0.108.0.0.0 sdisk CLAIMED DEVICE HP A6188A
disk 30 1/0/6/0/0.8.0.108.0.0.1 sdisk CLAIMED DEVICE HP A6188A
disk 38 1/0/6/0/0.8.0.108.0.0.2 sdisk CLAIMED DEVICE HP A6188A
disk 39 1/0/6/0/0.8.0.108.0.0.3 sdisk CLAIMED DEVICE HP A6188A
disk 41 1/0/6/0/0.8.0.108.0.0.4 sdisk CLAIMED DEVICE HP A6188A
disk 43 1/0/6/0/0.8.0.108.0.0.5 sdisk CLAIMED DEVICE HP A6188A
disk 45 1/0/6/0/0.8.0.108.0.0.6 sdisk CLAIMED DEVICE HP A6188A
disk 46 1/0/6/0/0.8.0.108.0.0.7 sdisk CLAIMED DEVICE HP A6188A
disk 47 1/0/6/0/0.8.0.108.0.1.0 sdisk CLAIMED DEVICE HP A6188A
disk 48 1/0/6/0/0.8.0.108.0.1.1 sdisk CLAIMED DEVICE HP A6188A
disk 49 1/0/6/0/0.8.0.108.0.1.2 sdisk CLAIMED DEVICE HP A6188A
disk 50 1/0/6/0/0.8.0.108.0.1.3 sdisk CLAIMED DEVICE HP A6188A

The same is true for the other array. Why am I not seeing all 15 disks per array? Commandview SDM shows all 15 disks in the gui. Thinking maybe for some reason the other disks are on a different i/o address I did an ioscan for disk and grep'd the disk model number (A6188A). It returned back exactly 50 disks. One controller showing 12 and the other 13 for two arrays is 50. But Shouldn't I get back 30 since that's how many disks actually exist?

Second, how do I go about fixing these SCSI errors? I believe understanding the first part of the question is needed for me to figure out the second question.

[Jared Rudy]

Sorry, the above output from my ioscan both show 12. Here's the complete output for all 4 controllers.

# ioscan -funC disk |grep "0/0/6/0/0.8"
disk 1 0/0/6/0/0.8.0.110.0.0.0 sdisk CLAIMED DEVICE HP A6188A
disk 29 0/0/6/0/0.8.0.110.0.0.1 sdisk CLAIMED DEVICE HP A6188A
disk 31 0/0/6/0/0.8.0.110.0.0.2 sdisk CLAIMED DEVICE HP A6188A
disk 32 0/0/6/0/0.8.0.110.0.0.3 sdisk CLAIMED DEVICE HP A6188A
disk 33 0/0/6/0/0.8.0.110.0.0.4 sdisk CLAIMED DEVICE HP A6188A
disk 34 0/0/6/0/0.8.0.110.0.0.5 sdisk CLAIMED DEVICE HP A6188A
disk 35 0/0/6/0/0.8.0.110.0.0.6 sdisk CLAIMED DEVICE HP A6188A
disk 36 0/0/6/0/0.8.0.110.0.0.7 sdisk CLAIMED DEVICE HP A6188A
disk 37 0/0/6/0/0.8.0.110.0.1.0 sdisk CLAIMED DEVICE HP A6188A
disk 40 0/0/6/0/0.8.0.110.0.1.1 sdisk CLAIMED DEVICE HP A6188A
disk 42 0/0/6/0/0.8.0.110.0.1.2 sdisk CLAIMED DEVICE HP A6188A
disk 44 0/0/6/0/0.8.0.110.0.1.3 sdisk CLAIMED DEVICE HP A6188A
# ioscan -funC disk |grep "1/0/6/0/0.8"
disk 5 1/0/6/0/0.8.0.108.0.0.0 sdisk CLAIMED DEVICE HP A6188A
disk 30 1/0/6/0/0.8.0.108.0.0.1 sdisk CLAIMED DEVICE HP A6188A
disk 38 1/0/6/0/0.8.0.108.0.0.2 sdisk CLAIMED DEVICE HP A6188A
disk 39 1/0/6/0/0.8.0.108.0.0.3 sdisk CLAIMED DEVICE HP A6188A
disk 41 1/0/6/0/0.8.0.108.0.0.4 sdisk CLAIMED DEVICE HP A6188A
disk 43 1/0/6/0/0.8.0.108.0.0.5 sdisk CLAIMED DEVICE HP A6188A
disk 45 1/0/6/0/0.8.0.108.0.0.6 sdisk CLAIMED DEVICE HP A6188A
disk 46 1/0/6/0/0.8.0.108.0.0.7 sdisk CLAIMED DEVICE HP A6188A
disk 47 1/0/6/0/0.8.0.108.0.1.0 sdisk CLAIMED DEVICE HP A6188A
disk 48 1/0/6/0/0.8.0.108.0.1.1 sdisk CLAIMED DEVICE HP A6188A
disk 49 1/0/6/0/0.8.0.108.0.1.2 sdisk CLAIMED DEVICE HP A6188A
disk 50 1/0/6/0/0.8.0.108.0.1.3 sdisk CLAIMED DEVICE HP A6188A
# ioscan -funC disk |grep "0/0/8/0/0.8"
disk 2 0/0/8/0/0.8.0.108.0.0.0 sdisk CLAIMED DEVICE HP A6188A
disk 7 0/0/8/0/0.8.0.108.0.0.1 sdisk CLAIMED DEVICE HP A6188A
disk 8 0/0/8/0/0.8.0.108.0.0.2 sdisk CLAIMED DEVICE HP A6188A
disk 9 0/0/8/0/0.8.0.108.0.0.3 sdisk CLAIMED DEVICE HP A6188A
disk 10 0/0/8/0/0.8.0.108.0.0.4 sdisk CLAIMED DEVICE HP A6188A
disk 11 0/0/8/0/0.8.0.108.0.0.5 sdisk CLAIMED DEVICE HP A6188A
disk 13 0/0/8/0/0.8.0.108.0.0.6 sdisk CLAIMED DEVICE HP A6188A
disk 19 0/0/8/0/0.8.0.108.0.0.7 sdisk CLAIMED DEVICE HP A6188A
disk 20 0/0/8/0/0.8.0.108.0.1.0 sdisk CLAIMED DEVICE HP A6188A
disk 21 0/0/8/0/0.8.0.108.0.1.1 sdisk CLAIMED DEVICE HP A6188A
disk 22 0/0/8/0/0.8.0.108.0.1.2 sdisk CLAIMED DEVICE HP A6188A
disk 23 0/0/8/0/0.8.0.108.0.1.3 sdisk CLAIMED DEVICE HP A6188A
disk 51 0/0/8/0/0.8.0.108.0.1.4 sdisk CLAIMED DEVICE HP A6188A
# ioscan -funC disk |grep "1/0/8/0/0.8"
disk 6 1/0/8/0/0.8.0.110.0.0.0 sdisk CLAIMED DEVICE HP A6188A
disk 12 1/0/8/0/0.8.0.110.0.0.1 sdisk CLAIMED DEVICE HP A6188A
disk 14 1/0/8/0/0.8.0.110.0.0.2 sdisk CLAIMED DEVICE HP A6188A
disk 15 1/0/8/0/0.8.0.110.0.0.3 sdisk CLAIMED DEVICE HP A6188A
disk 16 1/0/8/0/0.8.0.110.0.0.4 sdisk CLAIMED DEVICE HP A6188A
disk 17 1/0/8/0/0.8.0.110.0.0.5 sdisk CLAIMED DEVICE HP A6188A
disk 18 1/0/8/0/0.8.0.110.0.0.6 sdisk CLAIMED DEVICE HP A6188A
disk 24 1/0/8/0/0.8.0.110.0.0.7 sdisk CLAIMED DEVICE HP A6188A
disk 25 1/0/8/0/0.8.0.110.0.1.0 sdisk CLAIMED DEVICE HP A6188A
disk 26 1/0/8/0/0.8.0.110.0.1.1 sdisk CLAIMED DEVICE HP A6188A
disk 27 1/0/8/0/0.8.0.110.0.1.2 sdisk CLAIMED DEVICE HP A6188A
disk 28 1/0/8/0/0.8.0.110.0.1.3 sdisk CLAIMED DEVICE HP A6188A
disk 52 1/0/8/0/0.8.0.110.0.1.4 sdisk CLAIMED DEVICE HP A6188A

[Mel Burslan]

Since your server can not see the drives you say exist according to commandview, I'd suggest you check the zoning of these drives on the SAN. Otherwise, if they are there and attached to the server, they should show up in the ioscan output.

[Patrick Wallek · Accepted solution]

You are not seeing the individual disks in your VA7100. What you are seeing are the LUNs that have been configured on the VA7100 and presented to your HP-UX servers.

If you look at the LUN information for the arrays you will see 12 LUNs for one array and 13 for the other.

As far as the SCSI errors go, that is somewhat surprising. I suppose it could be a controller issue on the array(s), but if that were the case you should see messages in Commandview that says there are problems with the array.

[Jared Rudy]

One array shows 11 luns and the other shows 12. They both have a spare so would that be the 12 and 13?

[Torsten.]

You cannot see disks in ioscan at all, just LUNs only. You also cannot see spares, because there is only reserved space, no spare disk drive.

[Torsten.]

Use

# armdsp -i

to get the aliases, then

# armdsp -a

to get all the details.

[Jared Rudy]

Okay sorry I was in a hurry and I didn't look good enough before I posted. You are right I do have 12 luns on one array and 13 on the other. Thanks for clarifying that.

Now why would I be getting high i/o waits and scsi read errors? Could it be something as simple as a flakey optical cable? Or maybe a flakey controller?

[Patrick Wallek]

High I/O waits could be related to how the data is laid out on the array itself and how it is being accessed. That is basically related to how long it takes the array to return the data being requested. This is not necessarily a problem, but could be a symptom.

The SCSI Read errors are another thing entirely, at least to me. This could be indicative of a problem. What exactly, I'm not sure.

[Jared Rudy]

Guys thanks for clarifying how luns are reported back. Now I'm still working on this high i/o problem. Looking in my messages file I get the following. Is it really possible that I have 5 drives that are bad? The funny thing is all these drives that are having issues all exist on the same i/o address:

Jun 4 13:30
...
device 0x1f0a0500 (with priority: 1, and current flags: 0x0).
LVM: VG 64 0x010000: PVLink 31 0x020100 Failed! The PV is still accessible.
LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000097902000), from raw device 0x1f020600 (with priority: 0, and current flags: 0x40) to raw device 0x1f0a0600 (with priority: 1, and current flags: 0x0).
LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000097906000), from raw device 0x1f020700 (with priority: 0, and current flags: 0x40) to raw device 0x1f0a0700 (with priority: 1, and current flags: 0x0).
LVM: VG 64 0x040000: PVLink 31 0x020500 Failed! The PV is still accessible.
LVM: VG 64 0x040000: PVLink 31 0x020600 Failed! The PV is still accessible.
LVM: VG 64 0x040000: PVLink 31 0x020700 Failed! The PV is still accessible.

SCSI: Read error -- dev: b 31 0x020200, errno: 126, resid: 1024,
blkno: 8, sectno: 16, offset: 8192, bcount: 1024.
LVM: Performed a switch for Lun ID = 0 (pv = 0x000000009786a000), from raw device 0x1f020200 (with priority: 0, and current flags: 0x40) to raw device 0x1f0a0200 (with priority: 1, and current flags: 0x0).
LVM: VG 64 0x020000: PVLink 31 0x020200 Failed! The PV is still accessible.
LVM: VG 64 0x010000: PVLink 31 0x020000 Failed! The PV is not accessible.
LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000097820000), from raw device 0x1f020000 (with priority: 0, and current flags: 0xc0) to raw device 0x1f0a0000 (with priority: 1, and current flags: 0x0).
LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000978b6000), from raw device 0x1f020400 (with priority: 0, and current flags: 0x40) to raw device 0x1f0a0400 (with priority: 1, and current flags: 0x0).
LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000978b2000), from raw device 0x1f020300 (with priority: 0, and current flags: 0x40) to raw device 0x1f0a0300 (with priority: 1, and current flags: 0x0).
LVM: VG 64 0x030000: PVLink 31 0x020300 Failed! The PV is not accessible.
LVM: VG 64 0x030000: PVLink 31 0x020400 Failed! The PV is still accessible.
LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000978fc000), from raw device 0x1f0a0500 (with priority: 1, and current flags: 0x0) to raw device 0x1f020500 (with priority: 0, and current flags: 0x80).
LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000097902000), from raw device 0x1f0a0600 (with priority: 1, and current flags: 0x0) to raw device 0x1f020600 (with priority: 0, and current flags: 0x80).
LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000097906000), from raw device 0x1f0a0700 (with priority: 1, and current flags: 0x0) to raw device 0x1f020700 (with priority: 0, and current flags: 0x80).
LVM: VG 64 0x040000: PVLink 31 0x020500 Recovered.
LVM: VG 64 0x040000: PVLink 31 0x020600 Recovered.
LVM: VG 64 0x040000: PVLink 31 0x020700 Recovered.
LVM: Performed a switch for Lun ID = 0 (pv = 0x000000009786a000), from raw device 0x1f0a0200 (with priority: 1, and current flags: 0x0) to raw device 0x1f020200 (with priority: 0, and current flags: 0x80).
LVM: VG 64 0x020000: PVLink 31 0x020200 Recovered.
LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000978b2000), from raw device 0x1f0a0300 (with priority: 1, and current flags: 0x0) to raw device 0x1f020300 (with priority: 0, and current flags: 0x80).
LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000978b6000), from raw device 0x1f0a0400 (with priority: 1, and current flags: 0x0) to raw device 0x1f020400 (with priority: 0, and current flags: 0x80).
LVM: VG 64 0x030000: PVLink 31 0x020300 Recovered.
LVM: VG 64 0x030000: PVLink 31 0x020400 Recovered.
LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000097820000), from raw device 0x1f0a0000 (with priority: 1, and current flags: 0x0) to raw device 0x1f020000 (with priority: 0, and current flags: 0x80).
LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000097824000), from raw device 0x1f0a0100 (with priority: 1, and current flags: 0x0) to raw device 0x1f020100 (with priority: 0, and current flags: 0x80).
LVM: VG 64 0x010000: PVLink 31 0x020000 Recovered.
LVM: VG 64 0x010000: PVLink 31 0x020100 Recovered.