stale lv in vg00

[Clara Rowe]

I know there are a lot of threads on stale lv's. I have read them and I'm still not certain how to proceed.

Messages in syslog.log:
Feb 7 02:04:51 ed02 vmunix: SCSI: Async write error -- dev: b 31 0x01f000, errno: 5, resid: 24576,
Feb 7 02:32:11 ed02 vmunix: SCSI: Async write error -- dev: b 31 0x01f000, errno: 5, resid: 8192,
Feb 7 02:34:50 ed02 vmunix: SCSI: Async write error -- dev: b 31 0x01f000, errno: 5, resid: 16384,

strings /etc/lvmtab
/dev/vg00
/dev/dsk/c1t15d0
/dev/dsk/c3t15d0

vgdisplay -v vg00
--- Volume groups ---
VG Name /dev/vg00
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 11
Open LV 11
Max PV 16
Cur PV 2
Act PV 2
Max PE per PV 4350
VGDA 4
PE Size (Mbytes) 8
Total PE 8680
Alloc PE 7968
Free PE 712
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---

All available/syncd except:
LV Name /dev/vg00/lvol9
LV Status available/stale
LV Size (Mbytes) 15120
Current LE 1890
Allocated PE 3780
Used PV 2

lvdisplay -v /dev/vg00/lvol9
--- Logical volumes ---
LV Name /dev/vg00/lvol9
VG Name /dev/vg00
LV Permission read/write
LV Status available/stale
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 15120
Current LE 1890
Allocated PE 3780
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict
IO Timeout (Seconds) default

lvdisplay -v /dev/vg00/lvol9|grep stale
LV Status available/stale
00000 /dev/dsk/c1t15d0 01374 stale /dev/dsk/c3t15d0 01374 current
00128 /dev/dsk/c1t15d0 01502 stale /dev/dsk/c3t15d0 01502 current
00134 /dev/dsk/c1t15d0 01508 stale /dev/dsk/c3t15d0 01508 current
00136 /dev/dsk/c1t15d0 01510 stale /dev/dsk/c3t15d0 01510 current
00142 /dev/dsk/c1t15d0 01516 stale /dev/dsk/c3t15d0 01516 current
00152 /dev/dsk/c1t15d0 01526 stale /dev/dsk/c3t15d0 01526 current
00156 /dev/dsk/c1t15d0 01530 stale /dev/dsk/c3t15d0 01530 current
00208 /dev/dsk/c1t15d0 01582 stale /dev/dsk/c3t15d0 01582 current
00220 /dev/dsk/c1t15d0 01594 stale /dev/dsk/c3t15d0 01594 current
00232 /dev/dsk/c1t15d0 01606 stale /dev/dsk/c3t15d0 01606 current
00236 /dev/dsk/c1t15d0 01610 stale /dev/dsk/c3t15d0 01610 current
00244 /dev/dsk/c1t15d0 01618 stale /dev/dsk/c3t15d0 01618 current
00280 /dev/dsk/c1t15d0 01654 stale /dev/dsk/c3t15d0 01654 current
00284 /dev/dsk/c1t15d0 01658 stale /dev/dsk/c3t15d0 01658 current
00288 /dev/dsk/c1t15d0 01662 stale /dev/dsk/c3t15d0 01662 current
00292 /dev/dsk/c1t15d0 01666 stale /dev/dsk/c3t15d0 01666 current
00296 /dev/dsk/c1t15d0 01670 stale /dev/dsk/c3t15d0 01670 current
00629 /dev/dsk/c1t15d0 02003 current /dev/dsk/c3t15d0 02003 stale

vgsync results:
vgsync: Couldn't re-synchronize stale partitions of the logical volume:
Device offline/Powerfailed

vgsync: Couldn't resynchronize logical volume "/dev/vg00/lvol9".
vgsync: Couldn't resynchronize volume group "vg00".

pvdisplay /dev/dsk/c3t15d0
--- Physical volumes ---
PV Name /dev/dsk/c3t15d0
VG Name /dev/vg00
PV Status available
Allocatable yes
VGDA 2
Cur LV 11
PE Size (Mbytes) 8
Total PE 4340
Free PE 356
Allocated PE 3984
Stale PE 1
IO Timeout (Seconds) default
Autoswitch On

pvdisplay /dev/dsk/c1t15d0
--- Physical volumes ---
PV Name /dev/dsk/c1t15d0
VG Name /dev/vg00
PV Status available
Allocatable yes
VGDA 2
Cur LV 11
PE Size (Mbytes) 8
Total PE 4340
Free PE 356
Allocated PE 3984
Stale PE 14
IO Timeout (Seconds) default
Autoswitch On

diskinfo /dev/rdsk/c1t15d0
SCSI describe of /dev/rdsk/c1t15d0:
vendor: HP 36.4G
product id: MAN3367MC
type: direct access
size: 35566480 Kbytes
bytes per sector: 512

diskinfo /dev/rdsk/c3t15d0
SCSI describe of /dev/rdsk/c3t15d0:
vendor: HP 36.4G
product id: MAN3367MC
type: direct access
size: 35566480 Kbytes
bytes per sector: 512

Any help will be much appreciated.

Clara

[Deoncia Grayson_1]

did you try doing a lvsync?

lvsync /dev/vg00/lvol9

[James R. Ferguson · Accepted solution]

Hi Clara:

You have all indications of a failed disk or connection to it:

"Device offline/Powerfailed"

In order to be sure which disk, do:

# ioscan -fnC disk

...look for "NO_HW" (possiblly)

# dd if=/dev/rdsk/cXtYdZ of=/dev/null bs=64k count=100

...look for an unequal number of records in/out

You will find this document very useful:

http://www.docs.hp.com/en/5991-1236/When_Good_Disks_Go_Bad.pdf

Regards!

...JRF...

[Ajitkumar Rane]

Hello Clara,

Since only a chunk of PE's are shown as stale, the disk has media fault. You should get the disk replaced. do a lvreduce to remove the mirror copies and also vgreduce to get rid of the faulty disk. You already have the document attached in one the previous links for LVM disk replacement.



Rgds,

Ajit

[Nguyen Anh Tien]

Hi Clara.
I think that one of your disks have NO_HW state.
pls check and make sure that it is ok
#ioscan -fnuC disk
then
#lvsync /dev/vg00/lvol9

[Sivakumar TS]

Hi,

From the syslog, we understand that the disk c1t15d0 has h/w issue.

pls check the disk by

#dd if=/dev/rdsk/c1t15d0 of=/dev/zero

IF it gives error..

REPLACE the disk.


With Regards,

Siva.

[Arunvijai_4]

Hi Clara,

It can be a fault disk in your server, You can check this by executing

# dd if=/dev/rdsk/c1t15d0 /dev/null bs=64k count=1000

-Arun

[shridhar_1]

hi

u hav to replace the disk

just reduce all the mirrors
from faulty hdd
before removing the disk




shri

[Clara Rowe]

I want to thank all of you for your help on this. Points are forth coming. Here is the rest of the information you requested:

lvsync /dev/vg00/lvol9
lvsync: Couldn't re-synchronize stale partitions of the logical volume:
Device offline/Powerfailed

lvsync: Couldn't resynchronize logical volume "/dev/vg00/lvol9".

dd if=/dev/dsk/c1t15d0 of=/dev/null
dd read error: I/O error
24629384+0 records in
24629384+0 records out

dd if=/dev/dsk/c3t15d0 of=/dev/null
71132960+0 records in
71132960+0 records out




ioscan -fnC disk
Class I H/W Path Driver S/W State H/W Type Description
======================================================================
disk 0 0/0/1/1.15.0 sdisk CLAIMED DEVICE HP 36.4GMAN3367MC
/dev/dsk/c1t15d0 /dev/rdsk/c1t15d0
disk 1 0/0/2/1.15.0 sdisk CLAIMED DEVICE HP 36.4GMAN3367MC
/dev/dsk/c3t15d0 /dev/rdsk/c3t15d0


I'll be getting the disk replaced today.

Thank you all so very much. I love this forum and recommend it to often. I get dirty looks from our IBM reps when I sing it's praises. But you guys never fail to help.

Thanx again.

Clara

[Devender Khatana]

Hi,

Caution - Take ignite backup first.

This disk replacement may lead to some more problems as both of the mirrors have some stale PEs. The first disk (C1t15d0)anyway is faulty and need replacement but notice PE no. 629 which is current on the faulty disk and stale on the good disk. Replacing faulty disk will make it stale for ever and causing file system inconsistency. The only possibility to overcome this is if during lvreduce OS mark the PE as current on the good disk and if this does not happen you will need ignite recovery.

HTH,
Devender

[Clara Rowe]

Thanx to everyone for all the help.

Clara