Disk Enclosures
cancel
Showing results for 
Search instead for 
Did you mean: 

rp7410 va7100 array high i/o wait and general question about va7100

SOLVED
Go to solution
Jared Rudy
Advisor

rp7410 va7100 array high i/o wait and general question about va7100

We have a RP7410 with HPUX 11iv1 installed and two VA7100 arrays. Long story short, we've been experiencing high i/o waits as seen from running a sar command (average 35% to 40% %wio). I also noticed some SCSI read errors from 5 disks in logs. None of the disks are showing faulty so I though that all 5 disks bad was highly unlikely. Further investigation showed all those disks on the same i/o hardware address.

Couple questions.

First, each of our VA7100 arrays are full with 15 hard drives each. We have dual controller configuration. When I do an ioscan one controller address shows 12 disks and the other 13 disks for one array. I would think that each controller would show 15 disks right? Here's the output from ioscan:

# ioscan -funC disk |grep "0/0/6/0/0.8"
disk 1 0/0/6/0/0.8.0.110.0.0.0 sdisk CLAIMED DEVICE HP A6188A
disk 29 0/0/6/0/0.8.0.110.0.0.1 sdisk CLAIMED DEVICE HP A6188A
disk 31 0/0/6/0/0.8.0.110.0.0.2 sdisk CLAIMED DEVICE HP A6188A
disk 32 0/0/6/0/0.8.0.110.0.0.3 sdisk CLAIMED DEVICE HP A6188A
disk 33 0/0/6/0/0.8.0.110.0.0.4 sdisk CLAIMED DEVICE HP A6188A
disk 34 0/0/6/0/0.8.0.110.0.0.5 sdisk CLAIMED DEVICE HP A6188A
disk 35 0/0/6/0/0.8.0.110.0.0.6 sdisk CLAIMED DEVICE HP A6188A
disk 36 0/0/6/0/0.8.0.110.0.0.7 sdisk CLAIMED DEVICE HP A6188A
disk 37 0/0/6/0/0.8.0.110.0.1.0 sdisk CLAIMED DEVICE HP A6188A
disk 40 0/0/6/0/0.8.0.110.0.1.1 sdisk CLAIMED DEVICE HP A6188A
disk 42 0/0/6/0/0.8.0.110.0.1.2 sdisk CLAIMED DEVICE HP A6188A
disk 44 0/0/6/0/0.8.0.110.0.1.3 sdisk CLAIMED DEVICE HP A6188A
# ioscan -funC disk |grep "1/0/6/0/0.8"
disk 5 1/0/6/0/0.8.0.108.0.0.0 sdisk CLAIMED DEVICE HP A6188A
disk 30 1/0/6/0/0.8.0.108.0.0.1 sdisk CLAIMED DEVICE HP A6188A
disk 38 1/0/6/0/0.8.0.108.0.0.2 sdisk CLAIMED DEVICE HP A6188A
disk 39 1/0/6/0/0.8.0.108.0.0.3 sdisk CLAIMED DEVICE HP A6188A
disk 41 1/0/6/0/0.8.0.108.0.0.4 sdisk CLAIMED DEVICE HP A6188A
disk 43 1/0/6/0/0.8.0.108.0.0.5 sdisk CLAIMED DEVICE HP A6188A
disk 45 1/0/6/0/0.8.0.108.0.0.6 sdisk CLAIMED DEVICE HP A6188A
disk 46 1/0/6/0/0.8.0.108.0.0.7 sdisk CLAIMED DEVICE HP A6188A
disk 47 1/0/6/0/0.8.0.108.0.1.0 sdisk CLAIMED DEVICE HP A6188A
disk 48 1/0/6/0/0.8.0.108.0.1.1 sdisk CLAIMED DEVICE HP A6188A
disk 49 1/0/6/0/0.8.0.108.0.1.2 sdisk CLAIMED DEVICE HP A6188A
disk 50 1/0/6/0/0.8.0.108.0.1.3 sdisk CLAIMED DEVICE HP A6188A

The same is true for the other array. Why am I not seeing all 15 disks per array? Commandview SDM shows all 15 disks in the gui. Thinking maybe for some reason the other disks are on a different i/o address I did an ioscan for disk and grep'd the disk model number (A6188A). It returned back exactly 50 disks. One controller showing 12 and the other 13 for two arrays is 50. But Shouldn't I get back 30 since that's how many disks actually exist?

Second, how do I go about fixing these SCSI errors? I believe understanding the first part of the question is needed for me to figure out the second question.
11 REPLIES
Jared Rudy
Advisor

Re: rp7410 va7100 array high i/o wait and general question about va7100

Sorry, the above output from my ioscan both show 12. Here's the complete output for all 4 controllers.

# ioscan -funC disk |grep "0/0/6/0/0.8"
disk 1 0/0/6/0/0.8.0.110.0.0.0 sdisk CLAIMED DEVICE HP A6188A
disk 29 0/0/6/0/0.8.0.110.0.0.1 sdisk CLAIMED DEVICE HP A6188A
disk 31 0/0/6/0/0.8.0.110.0.0.2 sdisk CLAIMED DEVICE HP A6188A
disk 32 0/0/6/0/0.8.0.110.0.0.3 sdisk CLAIMED DEVICE HP A6188A
disk 33 0/0/6/0/0.8.0.110.0.0.4 sdisk CLAIMED DEVICE HP A6188A
disk 34 0/0/6/0/0.8.0.110.0.0.5 sdisk CLAIMED DEVICE HP A6188A
disk 35 0/0/6/0/0.8.0.110.0.0.6 sdisk CLAIMED DEVICE HP A6188A
disk 36 0/0/6/0/0.8.0.110.0.0.7 sdisk CLAIMED DEVICE HP A6188A
disk 37 0/0/6/0/0.8.0.110.0.1.0 sdisk CLAIMED DEVICE HP A6188A
disk 40 0/0/6/0/0.8.0.110.0.1.1 sdisk CLAIMED DEVICE HP A6188A
disk 42 0/0/6/0/0.8.0.110.0.1.2 sdisk CLAIMED DEVICE HP A6188A
disk 44 0/0/6/0/0.8.0.110.0.1.3 sdisk CLAIMED DEVICE HP A6188A
# ioscan -funC disk |grep "1/0/6/0/0.8"
disk 5 1/0/6/0/0.8.0.108.0.0.0 sdisk CLAIMED DEVICE HP A6188A
disk 30 1/0/6/0/0.8.0.108.0.0.1 sdisk CLAIMED DEVICE HP A6188A
disk 38 1/0/6/0/0.8.0.108.0.0.2 sdisk CLAIMED DEVICE HP A6188A
disk 39 1/0/6/0/0.8.0.108.0.0.3 sdisk CLAIMED DEVICE HP A6188A
disk 41 1/0/6/0/0.8.0.108.0.0.4 sdisk CLAIMED DEVICE HP A6188A
disk 43 1/0/6/0/0.8.0.108.0.0.5 sdisk CLAIMED DEVICE HP A6188A
disk 45 1/0/6/0/0.8.0.108.0.0.6 sdisk CLAIMED DEVICE HP A6188A
disk 46 1/0/6/0/0.8.0.108.0.0.7 sdisk CLAIMED DEVICE HP A6188A
disk 47 1/0/6/0/0.8.0.108.0.1.0 sdisk CLAIMED DEVICE HP A6188A
disk 48 1/0/6/0/0.8.0.108.0.1.1 sdisk CLAIMED DEVICE HP A6188A
disk 49 1/0/6/0/0.8.0.108.0.1.2 sdisk CLAIMED DEVICE HP A6188A
disk 50 1/0/6/0/0.8.0.108.0.1.3 sdisk CLAIMED DEVICE HP A6188A
# ioscan -funC disk |grep "0/0/8/0/0.8"
disk 2 0/0/8/0/0.8.0.108.0.0.0 sdisk CLAIMED DEVICE HP A6188A
disk 7 0/0/8/0/0.8.0.108.0.0.1 sdisk CLAIMED DEVICE HP A6188A
disk 8 0/0/8/0/0.8.0.108.0.0.2 sdisk CLAIMED DEVICE HP A6188A
disk 9 0/0/8/0/0.8.0.108.0.0.3 sdisk CLAIMED DEVICE HP A6188A
disk 10 0/0/8/0/0.8.0.108.0.0.4 sdisk CLAIMED DEVICE HP A6188A
disk 11 0/0/8/0/0.8.0.108.0.0.5 sdisk CLAIMED DEVICE HP A6188A
disk 13 0/0/8/0/0.8.0.108.0.0.6 sdisk CLAIMED DEVICE HP A6188A
disk 19 0/0/8/0/0.8.0.108.0.0.7 sdisk CLAIMED DEVICE HP A6188A
disk 20 0/0/8/0/0.8.0.108.0.1.0 sdisk CLAIMED DEVICE HP A6188A
disk 21 0/0/8/0/0.8.0.108.0.1.1 sdisk CLAIMED DEVICE HP A6188A
disk 22 0/0/8/0/0.8.0.108.0.1.2 sdisk CLAIMED DEVICE HP A6188A
disk 23 0/0/8/0/0.8.0.108.0.1.3 sdisk CLAIMED DEVICE HP A6188A
disk 51 0/0/8/0/0.8.0.108.0.1.4 sdisk CLAIMED DEVICE HP A6188A
# ioscan -funC disk |grep "1/0/8/0/0.8"
disk 6 1/0/8/0/0.8.0.110.0.0.0 sdisk CLAIMED DEVICE HP A6188A
disk 12 1/0/8/0/0.8.0.110.0.0.1 sdisk CLAIMED DEVICE HP A6188A
disk 14 1/0/8/0/0.8.0.110.0.0.2 sdisk CLAIMED DEVICE HP A6188A
disk 15 1/0/8/0/0.8.0.110.0.0.3 sdisk CLAIMED DEVICE HP A6188A
disk 16 1/0/8/0/0.8.0.110.0.0.4 sdisk CLAIMED DEVICE HP A6188A
disk 17 1/0/8/0/0.8.0.110.0.0.5 sdisk CLAIMED DEVICE HP A6188A
disk 18 1/0/8/0/0.8.0.110.0.0.6 sdisk CLAIMED DEVICE HP A6188A
disk 24 1/0/8/0/0.8.0.110.0.0.7 sdisk CLAIMED DEVICE HP A6188A
disk 25 1/0/8/0/0.8.0.110.0.1.0 sdisk CLAIMED DEVICE HP A6188A
disk 26 1/0/8/0/0.8.0.110.0.1.1 sdisk CLAIMED DEVICE HP A6188A
disk 27 1/0/8/0/0.8.0.110.0.1.2 sdisk CLAIMED DEVICE HP A6188A
disk 28 1/0/8/0/0.8.0.110.0.1.3 sdisk CLAIMED DEVICE HP A6188A
disk 52 1/0/8/0/0.8.0.110.0.1.4 sdisk CLAIMED DEVICE HP A6188A
Mel Burslan
Honored Contributor

Re: rp7410 va7100 array high i/o wait and general question about va7100

Since your server can not see the drives you say exist according to commandview, I'd suggest you check the zoning of these drives on the SAN. Otherwise, if they are there and attached to the server, they should show up in the ioscan output.
________________________________
UNIX because I majored in cryptology...
Patrick Wallek
Honored Contributor
Solution

Re: rp7410 va7100 array high i/o wait and general question about va7100

You are not seeing the individual disks in your VA7100. What you are seeing are the LUNs that have been configured on the VA7100 and presented to your HP-UX servers.

If you look at the LUN information for the arrays you will see 12 LUNs for one array and 13 for the other.

As far as the SCSI errors go, that is somewhat surprising. I suppose it could be a controller issue on the array(s), but if that were the case you should see messages in Commandview that says there are problems with the array.
Jared Rudy
Advisor

Re: rp7410 va7100 array high i/o wait and general question about va7100

One array shows 11 luns and the other shows 12. They both have a spare so would that be the 12 and 13?
Torsten.
Acclaimed Contributor

Re: rp7410 va7100 array high i/o wait and general question about va7100

You cannot see disks in ioscan at all, just LUNs only. You also cannot see spares, because there is only reserved space, no spare disk drive.

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Torsten.
Acclaimed Contributor

Re: rp7410 va7100 array high i/o wait and general question about va7100

Use

# armdsp -i

to get the aliases, then

# armdsp -a

to get all the details.

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Jared Rudy
Advisor

Re: rp7410 va7100 array high i/o wait and general question about va7100

Okay sorry I was in a hurry and I didn't look good enough before I posted. You are right I do have 12 luns on one array and 13 on the other. Thanks for clarifying that.

Now why would I be getting high i/o waits and scsi read errors? Could it be something as simple as a flakey optical cable? Or maybe a flakey controller?
Patrick Wallek
Honored Contributor

Re: rp7410 va7100 array high i/o wait and general question about va7100

High I/O waits could be related to how the data is laid out on the array itself and how it is being accessed. That is basically related to how long it takes the array to return the data being requested. This is not necessarily a problem, but could be a symptom.

The SCSI Read errors are another thing entirely, at least to me. This could be indicative of a problem. What exactly, I'm not sure.

Jared Rudy
Advisor

Re: rp7410 va7100 array high i/o wait and general question about va7100

Guys thanks for clarifying how luns are reported back. Now I'm still working on this high i/o problem. Looking in my messages file I get the following. Is it really possible that I have 5 drives that are bad? The funny thing is all these drives that are having issues all exist on the same i/o address:

Jun 4 13:30
...
device 0x1f0a0500 (with priority: 1, and current flags: 0x0).
LVM: VG 64 0x010000: PVLink 31 0x020100 Failed! The PV is still accessible.
LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000097902000), from raw device 0x1f020600 (with priority: 0, and current flags: 0x40) to raw device 0x1f0a0600 (with priority: 1, and current flags: 0x0).
LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000097906000), from raw device 0x1f020700 (with priority: 0, and current flags: 0x40) to raw device 0x1f0a0700 (with priority: 1, and current flags: 0x0).
LVM: VG 64 0x040000: PVLink 31 0x020500 Failed! The PV is still accessible.
LVM: VG 64 0x040000: PVLink 31 0x020600 Failed! The PV is still accessible.
LVM: VG 64 0x040000: PVLink 31 0x020700 Failed! The PV is still accessible.

SCSI: Read error -- dev: b 31 0x020200, errno: 126, resid: 1024,
blkno: 8, sectno: 16, offset: 8192, bcount: 1024.
LVM: Performed a switch for Lun ID = 0 (pv = 0x000000009786a000), from raw device 0x1f020200 (with priority: 0, and current flags: 0x40) to raw device 0x1f0a0200 (with priority: 1, and current flags: 0x0).
LVM: VG 64 0x020000: PVLink 31 0x020200 Failed! The PV is still accessible.
LVM: VG 64 0x010000: PVLink 31 0x020000 Failed! The PV is not accessible.
LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000097820000), from raw device 0x1f020000 (with priority: 0, and current flags: 0xc0) to raw device 0x1f0a0000 (with priority: 1, and current flags: 0x0).
LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000978b6000), from raw device 0x1f020400 (with priority: 0, and current flags: 0x40) to raw device 0x1f0a0400 (with priority: 1, and current flags: 0x0).
LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000978b2000), from raw device 0x1f020300 (with priority: 0, and current flags: 0x40) to raw device 0x1f0a0300 (with priority: 1, and current flags: 0x0).
LVM: VG 64 0x030000: PVLink 31 0x020300 Failed! The PV is not accessible.
LVM: VG 64 0x030000: PVLink 31 0x020400 Failed! The PV is still accessible.
LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000978fc000), from raw device 0x1f0a0500 (with priority: 1, and current flags: 0x0) to raw device 0x1f020500 (with priority: 0, and current flags: 0x80).
LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000097902000), from raw device 0x1f0a0600 (with priority: 1, and current flags: 0x0) to raw device 0x1f020600 (with priority: 0, and current flags: 0x80).
LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000097906000), from raw device 0x1f0a0700 (with priority: 1, and current flags: 0x0) to raw device 0x1f020700 (with priority: 0, and current flags: 0x80).
LVM: VG 64 0x040000: PVLink 31 0x020500 Recovered.
LVM: VG 64 0x040000: PVLink 31 0x020600 Recovered.
LVM: VG 64 0x040000: PVLink 31 0x020700 Recovered.
LVM: Performed a switch for Lun ID = 0 (pv = 0x000000009786a000), from raw device 0x1f0a0200 (with priority: 1, and current flags: 0x0) to raw device 0x1f020200 (with priority: 0, and current flags: 0x80).
LVM: VG 64 0x020000: PVLink 31 0x020200 Recovered.
LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000978b2000), from raw device 0x1f0a0300 (with priority: 1, and current flags: 0x0) to raw device 0x1f020300 (with priority: 0, and current flags: 0x80).
LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000978b6000), from raw device 0x1f0a0400 (with priority: 1, and current flags: 0x0) to raw device 0x1f020400 (with priority: 0, and current flags: 0x80).
LVM: VG 64 0x030000: PVLink 31 0x020300 Recovered.
LVM: VG 64 0x030000: PVLink 31 0x020400 Recovered.
LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000097820000), from raw device 0x1f0a0000 (with priority: 1, and current flags: 0x0) to raw device 0x1f020000 (with priority: 0, and current flags: 0x80).
LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000097824000), from raw device 0x1f0a0100 (with priority: 1, and current flags: 0x0) to raw device 0x1f020100 (with priority: 0, and current flags: 0x80).
LVM: VG 64 0x010000: PVLink 31 0x020000 Recovered.
LVM: VG 64 0x010000: PVLink 31 0x020100 Recovered.
Jared Rudy
Advisor

Re: rp7410 va7100 array high i/o wait and general question about va7100

Now if I'm understanding right, for example the disk at the beginning there 0x020100 which happens to be /dev/dsk/c2t0d1 isn't really the disk but the lun having issues?
Jared Rudy
Advisor

Re: rp7410 va7100 array high i/o wait and general question about va7100

That's a typo. It's 0x020200 which converts to /dev/dsk/c2t0d2.