POWERFAILED errors.........

[seanharrodine]

Hi everyone,

Can someone give me some pointers with this issue please that we are having with our A500.
The disk array is a 7100 by the way.

I have just noticed the following errors in /var/adm/syslog/syslog.log.......

Oct 21 15:42:29 jhwmss02 vmunix: LVM: Recovered Path (device 0x1f040200) to PV 0 in VG 2.
Oct 21 15:42:33 jhwmss02 vmunix: LVM: Recovered Path (device 0x1f050200) to PV 1 in VG 2.
Oct 21 15:42:33 jhwmss02 vmunix: LVM: vg[2]: pvnum=0 (dev_t=0x1f040200) is POWERFAILED
Oct 21 15:42:33 jhwmss02 vmunix: LVM: vg[2].lv[1] has 2 fault disks on : pvnum=0 (dev=0x1f040200) pvnum=1 (dev=0x1f050200)
Oct 21 15:42:46 jhwmss02 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000430e9800), from raw device 0x1f040200 (with priority: 0, and current flags: 0xd0) to raw device 0x1f0a0200 (with priority: 1, and current flags: 0x0).
Oct 21 15:42:46 jhwmss02 vmunix: LVM: Restored PV 0 to VG 2.
Oct 21 15:42:46 jhwmss02 vmunix: LVM: Restored PV 1 to VG 2.
Oct 21 15:42:48 jhwmss02 vmunix: LVM: Recovered Path (device 0x1f040200) to PV 0 in VG 2.
Oct 21 15:42:48 jhwmss02 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000430e9800), from raw device 0x1f0a0200 (with priority: 1, and current flags: 0x0) to raw device 0x1f040200 (with priority: 0, and current flags: 0x0).

Oct 21 15:43:40 jhwmss02 vmunix:
Oct 21 15:43:40 jhwmss02 vmunix: SCSI: Read error -- dev: b 31 0x040200, errno: 126, resid: 2048,
Oct 21 15:43:40 jhwmss02 vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 2048.
Oct 21 15:43:40 jhwmss02 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000430e9800), from raw device 0x1f040200 (with priority: 0, and current flags: 0x40) to raw device 0x1f0a0200 (with priority: 1, and current flags: 0x0).
Oct 21 15:43:40 jhwmss02 vmunix: LVM: Recovered Path (device 0x1f050200) to PV 1 in VG 2.
Oct 21 15:43:40 jhwmss02 vmunix: LVM: vg[2]: pvnum=0 (dev_t=0x1f0a0200) is POWERFAILED
Oct 21 15:43:40 jhwmss02 vmunix: LVM: vg[2].lv[1] has 2 fault disks on : pvnum=0 (dev=0x1f0a0200) pvnum=1 (dev=0x1f050200)
Oct 21 15:43:40 jhwmss02 vmunix: LVM: Restored PV 0 to VG 2.
Oct 21 15:43:42 jhwmss02 vmunix: LVM: Restored PV 1 to VG 2.

Oct 21 15:43:45 jhwmss02 vmunix: LVM: Recovered Path (device 0x1f040200) to PV 0 in VG 2.
Oct 21 15:43:45 jhwmss02 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000430e9800), from raw device 0x1f0a0200 (with priority: 1, and current flags: 0x0) to raw device 0x1f040200 (with priority: 0, and current flags: 0x80).

Oct 21 15:46:01 jhwmss02 vmunix:
Oct 21 15:46:01 jhwmss02 vmunix: SCSI: Read error -- dev: b 31 0x040200, errno: 126, resid: 2048,
Oct 21 15:46:01 jhwmss02 vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 2048.
Oct 21 15:46:01 jhwmss02 vmunix: LVM: vg[2]: pvnum=0 (dev_t=0x1f040200) is POWERFAILED
Oct 21 15:46:01 jhwmss02 vmunix: SCSI: Write error -- dev: b 31 0x040200, errno: 126, resid: 34816,
Oct 21 15:46:01 jhwmss02 vmunix: blkno: 1078, sectno: 2156, offset: 1103872, bcount: 34816.
Oct 21 15:46:01 jhwmss02 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000430e9800), from raw device 0x1f040200 (with priority: 0, and current flags: 0xc0) to raw device 0x1f0a0200 (with priority: 1, and current flags: 0x0).
Oct 21 15:46:01 jhwmss02 vmunix:
Oct 21 15:46:06 jhwmss02 vmunix: LVM: Recovered Path (device 0x1f040200) to PV 0 in VG 2.
Oct 21 15:46:06 jhwmss02 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000430e9800), from raw device 0x1f0a0200 (with priority: 1, and current flags: 0x50) to raw device 0x1f040200 (with priority: 0, and current flags: 0x80).
Oct 21 15:46:06 jhwmss02 vmunix: LVM: Recovered Path (device 0x1f0a0200) to PV 0 in VG 2.
Oct 21 15:46:06 jhwmss02 vmunix: LVM: Restored PV 0 to VG 2.

This system is part of a 2 node cluster and this machine is currently running the package.

Can anyone confirm please just which disk is showing an issue ?
Am i right in thinking its 0x1f040200 and 0x1f0a0200 which i think resolve to /dev/dsk/c4t0d2 and /dev/dsk/c10t0d2 ???

Thanks in advance.

Sean

[Alzhy]

Yes that is IF:

ls -l /dev/rdsk/*|egrep "1f040200|1f0a0200" MATCHES your cXtYdZ values.

[Alzhy]

Corrrection:

ls -l /dev/rdsk/*|egrep "040200|0a0200"

Man I am getting rusty....

[seanharrodine]

Hi there,

Here you go.......

#ls -l /dev/rdsk/*|egrep "040200|0a0200"
crw-r----- 1 bin sys 188 0x0a0200 Jun 18 2002 /dev/rdsk/c10t0d2
crw-r----- 1 bin sys 188 0x040200 Jun 18 2002 /dev/rdsk/c4t0d2

Does it mean that both disks are potentially faulty ?

Sean

[Alzhy]

Potentially yes.
But other things to check.

If Disks are SCSI disks... check the Enclosure. Likley power was accidentally turned off. We've many sad experiences with our DS21XX enclosures.

[seanharrodine]

The disk arrays are fibre connected.

We have been experiencing issues for some time now, and we see disks thrashing at 100% also, noticeably, c4t0d2.

Originally, as its a cluster, focus by our hardware engineers was on potential controller issues and then long wave fibre cables, but we have always suspected disk/s problems.

[seanharrodine]

The plot thickens now.

Both servers in the cluster were rebooted this afternoon.

On the server that we suspect has an issue, the VA7100 has 2 arrays of which 1 is now showing the following....

armdsp -a array_1

Status : Array Status
Device is not ready.

...and the local operator is saying that there is strange display of grenn and amber light patterns occuring.

array_2 looks ok (i think !)......

jhwmss01:/home/sharrod1/output# armdsp -a array_2


Vendor ID:______________________________HP
Product ID:_____________________________A6188A
Array World Wide Name:__________________50060b0000092e94
Array Serial Number:____________________00SG217J0072
Alias:__________________________________array_2
Software Revision:______________________01.07.00 - 0256 - 030925
Command execution timestamp:____________Oct 21, 2010 9:50:32 PM
------------------------------------------------------------

ARRAY INFORMATION

Array Status:_________________________Ready
Firmware Revision:____________________38370HP19P1031031142
Product Revision:_____________________HP19
Last Event Log Entry for Page 1:______34959

ENCLOSURES

Enclosure at M
Enclosure ID__________________________0
Enclosure Status______________________Good
Enclosure Type________________________HP StorageWorks Virtual Array 7100
Node WWN______________________________50060b0000092e94

FRU HW COMPONENT IDENTIFICATION ID STATUS
===========================================================================
M Enclosure 00SG217J0072 Good
M/P1 Power Supply 94020HC01729 Good
M/P2 Power Supply 94020HC01734 Good
M/MP1 MidPlane 000599090772 Good
M/C2 Controller 00PR04106492 Good
M/C2.H1 Host Port Good
M/C2.J1 BackEnd Port Good
M/C2.G1 Host GBIC A28BX0T Good
M/C2.B1 Battery 28295:MOLTECHPS:NI2040:2002/3/12 Good
M/C2.PM1 Processor HP:A6188A:HP19 Good
M/C2.M1 DIMM 256 Good
M/C1 Controller 00PR04106054 Good
M/C1.H1 Host Port Good
M/C1.J1 BackEnd Port Good
M/C1.G1 Host GBIC A226435 Good
M/C1.B1 Battery 33670:MOLTECHPS:NI2040:2002/6/4 Good
M/C1.PM1 Processor HP:A6188A:HP19 Good
M/C1.M1 DIMM 256 Good
M/D1 Disk 3EV04DJA Good
M/D2 Disk 3EV0288R Good
M/D3 Disk 3EV0J0LP Good
M/D4 Disk 3EV04LSQ Good
M/D5 Disk 3EV028RA Good
M/D6 Disk 3EV04CH7 Good
M/D7 Disk 3EV05PK3 Good

CONTROLLERS

Controller At M/C2:
Status:_______________________________Good
Serial Number:________________________00PR04106492
Vendor ID:____________________________HP
Product ID:___________________________A6188A
Product Revision:_____________________HP19
Firmware Revision:____________________38370HP19P1031031142
Manufacturing Product Code:___________IJMTU00013
Controller Type:______________________HP StorageWorks Virtual Array 7100
Battery Charger Firmware Revision:____4.3
Front Port At M/C2.H1:
Status:_____________________________Good
Port Instance:______________________0
Hard Address:_______________________112
Link State:_________________________Link Up
Node WWN:___________________________50060b0000092e94
Port WWN:___________________________50060b0000147708
Topology:___________________________Private Loop
Data Rate:__________________________1 GBit/sec
Port ID:____________________________112
Device Host Name:___________________JHWMSS01
Hardware Path:______________________0/6/0/0.8.0.112.0.0.0
Device Path:________________________/dev/dsk/c11t0d0
Host GBIC at M/C2.G1:
Status:_____________________________Good
Identification:_____________________A28BX0T
Battery at M/C2.B1:
Status:_____________________________Good
Identification:_____________________28295:MOLTECHPS:NI2040:2002/3/12
Manufacturer Name:__________________MOLTECHPS
Device Name:________________________NI2040
Manufacturer Date:__________________March 12, 2002
Remaining Capacity:_________________5687 mAh
Remaining Capacity:_________________94 %
Voltage:____________________________12223 mVolts
Discharge Cycles:___________________1
Processor at M/C2.PM1:
Status:_____________________________Good
Identification:_____________________HP:A6188A:HP19
DIMM at M/C2.M1:
Status:_____________________________Good
Identification:_____________________256
Capacity:___________________________256 MB

Controller At M/C1:
Status:_______________________________Good
Serial Number:________________________00PR04106054
Vendor ID:____________________________HP
Product ID:___________________________A6188A
Product Revision:_____________________HP19
Firmware Revision:____________________38370HP19P1031031142
Manufacturing Product Code:___________IJMTU00013
Controller Type:______________________HP StorageWorks Virtual Array 7100
Battery Charger Firmware Revision:____4.3
Front Port At M/C1.H1:
Status:_____________________________Good
Port Instance:______________________0
Hard Address:_______________________114
Link State:_________________________Link Up
Node WWN:___________________________50060b0000092e94
Port WWN:___________________________50060b000014696a
Topology:___________________________Private Loop
Data Rate:__________________________1 GBit/sec
Port ID:____________________________114
Device Host Name:___________________JHWMSS01
Hardware Path:______________________Unknown
Device Path:________________________Unknown
Host GBIC at M/C1.G1:
Status:_____________________________Good
Identification:_____________________A226435
Battery at M/C1.B1:
Status:_____________________________Good
Identification:_____________________33670:MOLTECHPS:NI2040:2002/6/4
Manufacturer Name:__________________MOLTECHPS
Device Name:________________________NI2040
Manufacturer Date:__________________June 4, 2002
Remaining Capacity:_________________5019 mAh
Remaining Capacity:_________________83 %
Voltage:____________________________12391 mVolts
Discharge Cycles:___________________2
Processor at M/C1.PM1:
Status:_____________________________Good
Identification:_____________________HP:A6188A:HP19
DIMM at M/C1.M1:
Status:_____________________________Good
Identification:_____________________256
Capacity:___________________________256 MB

PORTS

Settings for port M/C2.H1:
Port ID:______________________________112
Behavior:_____________________________HPUX
Topology:_____________________________Private Loop
Queue Full Threshold:_________________21
Data Rate:____________________________1 GBit/sec

Settings for port M/C1.H1:
Port ID:______________________________114
Behavior:_____________________________HPUX
Topology:_____________________________Private Loop
Queue Full Threshold:_________________21
Data Rate:____________________________1 GBit/sec

DISKS

Disk at M/D1:
Status:_______________________________Good
Disk State:___________________________Included
Vendor ID:____________________________HP 18.2G
Product ID:___________________________ST318452FC
Product Revision:_____________________HP06
Data Capacity:________________________16.689 GB (35000000 blocks)
Block Length:_________________________520 bytes
Address:______________________________111
Node WWN:_____________________________20000004cf75e3b3
Initialize State:_____________________Ready
Redundancy Group:_____________________1
Volume Set Serial Number:_____________000017A600000006
Serial Number:________________________3EV04DJA
Firmware Revision:____________________HP06

Disk at M/D2:
Status:_______________________________Good
Disk State:___________________________Included
Vendor ID:____________________________HP 18.2G
Product ID:___________________________ST318452FC
Product Revision:_____________________HP06
Data Capacity:________________________16.689 GB (35000000 blocks)
Block Length:_________________________520 bytes
Address:______________________________112
Node WWN:_____________________________20000004cf75385e
Initialize State:_____________________Ready
Redundancy Group:_____________________1
Volume Set Serial Number:_____________000017A600000006
Serial Number:________________________3EV0288R
Firmware Revision:____________________HP06
Recovery Maps are on this disk.

Disk at M/D3:
Status:_______________________________Good
Disk State:___________________________Included
Vendor ID:____________________________HP 18.2G
Product ID:___________________________ST318452FC
Product Revision:_____________________HP06
Data Capacity:________________________16.689 GB (35000000 blocks)
Block Length:_________________________520 bytes
Address:______________________________113
Node WWN:_____________________________20000004cf753979
Initialize State:_____________________Ready
Redundancy Group:_____________________1
Volume Set Serial Number:_____________000017A600000006
Serial Number:________________________3EV0J0LP
Firmware Revision:____________________HP06
Recovery Maps are on this disk.

Disk at M/D4:
Status:_______________________________Good
Disk State:___________________________Included
Vendor ID:____________________________HP 18.2G
Product ID:___________________________ST318452FC
Product Revision:_____________________HP06
Data Capacity:________________________16.689 GB (35000000 blocks)
Block Length:_________________________520 bytes
Address:______________________________114
Node WWN:_____________________________20000004cf75e0ba
Initialize State:_____________________Ready
Redundancy Group:_____________________1
Volume Set Serial Number:_____________000017A600000006
Serial Number:________________________3EV04LSQ
Firmware Revision:____________________HP06

Disk at M/D5:
Status:_______________________________Good
Disk State:___________________________Included
Vendor ID:____________________________HP 18.2G
Product ID:___________________________ST318452FC
Product Revision:_____________________HP06
Data Capacity:________________________16.689 GB (35000000 blocks)
Block Length:_________________________520 bytes
Address:______________________________115
Node WWN:_____________________________20000004cf75ddc0
Initialize State:_____________________Ready
Redundancy Group:_____________________1
Volume Set Serial Number:_____________000017A600000006
Serial Number:________________________3EV028RA
Firmware Revision:____________________HP06

Disk at M/D6:
Status:_______________________________Good
Disk State:___________________________Included
Vendor ID:____________________________HP 18.2G
Product ID:___________________________ST318452FC
Product Revision:_____________________HP06
Data Capacity:________________________16.689 GB (35000000 blocks)
Block Length:_________________________520 bytes
Address:______________________________116
Node WWN:_____________________________20000004cf75e04c
Initialize State:_____________________Ready
Redundancy Group:_____________________1
Volume Set Serial Number:_____________000017A600000006
Serial Number:________________________3EV04CH7
Firmware Revision:____________________HP06

Disk at M/D7:
Status:_______________________________Good
Disk State:___________________________Included
Vendor ID:____________________________HP 18.2G
Product ID:___________________________ST318452FC
Product Revision:_____________________HP06
Data Capacity:________________________16.689 GB (35000000 blocks)
Block Length:_________________________520 bytes
Address:______________________________117
Node WWN:_____________________________20000004cf75deec
Initialize State:_____________________Ready
Redundancy Group:_____________________1
Volume Set Serial Number:_____________000017A600000006
Serial Number:________________________3EV05PK3
Firmware Revision:____________________HP06

LUNS

LUN 0:
Redundancy Group:_____________________1
Active:_______________________________True
Data Capacity:________________________10.5 MB
WWN:__________________________________50060b0000092e940000000000000006
Number Of Business Copies:____________0

LUN 1:
Redundancy Group:_____________________1
Active:_______________________________True
Data Capacity:________________________102.5 MB
WWN:__________________________________50060b0000092e940001000000000007
Number Of Business Copies:____________0

LUN 2:
Redundancy Group:_____________________1
Active:_______________________________True
Data Capacity:________________________58.424 GB
WWN:__________________________________50060b0000092e940002000000000008
Number Of Business Copies:____________0

CAPACITY Totals for Redundancy Group 1:
REGULAR LUNs:_________________________58.534 GB
BUSINESS COPIES:______________________0 bytes

CAPACITY USAGE

Total Disk Enclosures:________________1

Redundancy Group:_____________________1
Total Disks:________________________7
Total Physical Size:________________116.825 GB
Allocated to Regular LUNs:__________58.534 GB
Allocated as Business Copies:_______0 bytes
Used as Active Hot Spare:___________16.687 GB
Used for Redundancy:________________41.602 GB
Unallocated (Available for LUNs):___512 KB

SUB-SYSTEM SETTINGS

RAID Level:___________________________HPAutoRAID
Auto Format Drive:____________________On
Hang Detection:_______________________On
Capacity Depletion Threshold:_________100%
Queue Full Threshold Maximum:_________4096
Enable Optimize Policy:_______________True
Enable Manual Override:_______________False
Manual Override Destination:__________False
Read Cache Disable:___________________False
Rebuild Priority:_____________________Low
Security Enabled:_____________________False
Shutdown Completion:__________________0
Subsystem Type ID:____________________0
Unit Attention:_______________________True
Volume Set Partition (VSpart):________False
Write Cache Enable:___________________True
Write Working Set Interval:___________8640
Enable Prefetch:______________________True
Disable Secondary Path Presentation:__False

RESILIENCY SETTINGS

Simplified Resiliency Setting:________Normal Performance (Default)
Enable Secure Mode:___________________True
Disable NVRAM on UPS Absent:__________False
Disable NVRAM on WCE False:___________False
Disable Read Hits:____________________False
Force Unit Access Response:___________1
Lock Write Cache On:__________________True
Performance Goal Configuration:_______Normal Performance
Resiliency Threshold:_________________4
Single Controller Warning:____________True

DISK SETTINGS

Auto Include:_________________________On
Auto Rebuild:_________________________On
Hot Spare:____________________________Automatic
Max Drives per Loop Pair:_____________15
Max Drives per Subsystem:_____________15

ENCLOSURE SETTINGS

Max Enclosures per Loop Pair:_________0
Max Enclosures per Subsystem:_________0

LUN SETTINGS

LUN Creation Limit:___________________128
Maximum LUN Creation Limit:___________128
Migrating Write Destination:__________False

SCRUB SETTINGS

Scrub Restart Period:_________________7190 minutes
Scrub State:__________________________Idle, initialized

OPERATIONS IN PROGRESS

The array is OPTIMIZING.

WARNINGS

None

Can anyone assist in what the problem may be please ?

TIA,

Sean

[Andrew Rutter · Accepted solution]

hi,

you probably have got bad disks.

If it was the controller i would suspect you would have been loosing the whole array not just getting errors about particular disk devices as you have.

As you say you have been experiencing this a while, so you maybe had one bad disk that kept dying and recovering, but because its an autoraid box, it keeps trying to move the data around to save it (hence disks thrashing). probably onto another disk - hot spare? depends upon your config. but now it looks like you have possibly lost another disk, and now it cannot startup the array properly because of too many bad disks in the group.

green/amber leds flashing? where? at the controller end? or on the disks? most likely the disks and this is telling you they disks dont seem to be able to pass there self tests or are failing.

Andy

[seanharrodine]

Hi Andrew,

Many thanks for your input.

That has been our thoughts that it is disk related.

We have green/amber lights flashing across all the disks in the array of which there are 7.
The 3 lights on both controllers around the rear are all green.

We have masses of errors flagging in logprn so we plan to save them away and clear the logs, and then restart the array and see if we get any further clarification of which disk/disks/devices are failing.

We are planning on doing a restart via "cvui".

Is this the same as a hard reset performed by an onsite operator.....i'm guessing so.

Thanks again
Sean

[seanharrodine]

So, we have powered down the server and the VA7100 and both have now come back up fine.

All lights are green so we have failed the cluser package across onto the node.

But, we now see that the 7 disks in the array are being thrashed once more.
This has been happening now for well over an hour.

(see enclosed sar output)

# sar -d 5 5

HP-UX jhwmss01 B.11.11 U 9000/800 10/22/10

18:32:16 device %busy avque r+w/s blks/s avwait avserv
18:32:21 c1t15d0 5.99 0.50 17 412 4.56 4.80
c3t15d0 2.99 0.50 14 400 4.93 4.21
c4t0d2 87.62 0.50 21 9463 4.18 43.45
c11t0d2 11.98 0.50 21 9463 5.36 4.98
18:32:26 c1t15d0 1.20 0.50 3 13 3.06 5.05
c4t0d2 89.80 0.50 20 9379 4.77 44.21
c11t0d2 10.00 0.50 20 9379 5.02 4.75
18:32:31 c1t15d0 1.00 0.50 5 22 2.40 3.01
c3t15d0 0.40 0.50 3 15 2.55 2.96
c4t0d2 90.00 0.50 20 9454 4.95 44.57
c11t0d2 9.60 0.50 20 9454 4.75 4.93
18:32:36 c1t15d0 1.60 0.50 7 46 2.64 3.67
c3t15d0 0.20 0.50 5 40 2.78 0.71
c4t0d2 90.80 0.50 21 9482 4.89 43.65
c11t0d2 9.20 0.50 21 9482 4.58 4.75
18:32:41 c1t15d0 2.20 0.50 6 47 2.70 4.67
c3t15d0 0.20 0.50 4 40 2.89 0.53
c4t0d2 87.80 0.50 21 9482 4.09 43.65
c11t0d2 11.40 0.50 21 9482 5.30 4.78

Average c1t15d0 2.40 0.50 8 108 3.52 4.36
Average c3t15d0 0.76 0.50 5 99 3.90 2.81
Average c4t0d2 89.20 0.50 20 9452 4.57 43.90
Average c11t0d2 10.44 0.50 20 9452 5.00 4.84

It always show disk c4t0d2 but i dont believe that this relates to 1 specific disk does it, as it is an autoraid array.

I have checked armdsp and all disks are showing as good and normal.

logprn also does not show any errors.

One thing that i have noticed is a firmware discrepancy on 2 of the disks......

Vendor Product ID Rev FRU Location
HP 18.2G ST318452FC HP06 M/D1
HP 18.2G ST318452FC HP06 M/D2
HP 18.2G ST318452FC HP06 M/D3
HP 18.2G ST318452FC HP06 M/D4
HP 18.2G ST318452FC HP06 M/D5
HP 18.2G ST318452FC HP05 M/D6
HP 18.2G ST318451FC HP10 M/D7

....and disks D6 & D7 evidently store the "Recovery Maps"

could this cause the disk thrashing because at the moment, we are unable to move the package across to this node.

TIA,
Sean

[Alzhy]

Well -- very evident your VA7100 has some issues which you must resolve. That's a very old array man... Can you even still engage HP Support?

[chris huys_4]

Hi Sean,

>the VA7100 has 2 arrays

You mean, the a500 is connected to 2 different VA7100 arrays.

Is the command "armdsp -a array_1" allready giving "a normal" output ? If so can you post the output ?

Greetz,
Chris