Logical Volume Problem

[hahmedma]

we have issue here .. when they try to take a full backup from the System it faild due the Hard Desks in stale status for almost of the logical volme...our investigation as follow: 
1-checked the output of the vgdisplay -v...really i had find more than 9 logical volume on the stale state... 
2-then try to do manuall synch between the primary and the mirror by the command lvsync but it show this error message"lvsync: Couldn't re-synchronize stale partitions of the logical volume: Device offline/Powerfailed" This error it might be related to Hardware issue (i.e. Bad sectors or failure on HardDisk) or software (mirroring problem) 
3-from our side we had take some action as recreate a new logical volume and make it's mirroring and mount the old data from the old logical volume to the new logical volume and rename it as the old one then remove the old one...the 2 logical volume are typically identical...i tried this in the 2 Logical volume one of them is synched but the other one is stale... Really i didn't know te problem is come from there is a problem on the Hard desks it self or there were some Problem on the Mirror configuration. here are some O/P of some commands: Class I H/W Path Driver S/W State H/W Type Description ========================================================================= disk 0 0/0/3/0.0.0.0 sdisk CLAIMED DEVICE TEAC DV-28E-N /dev/dsk/c0t0d0 /dev/rdsk/c0t0d0 disk 1 0/1/1/0.1.0 sdisk CLAIMED DEVICE HP 73.4GST373455LC /dev/dsk/c2t1d0 /dev/rdsk/c2t1d0 disk 2 0/1/1/1.0.0 sdisk CLAIMED DEVICE HP 73.4GMAX3073NC /dev/dsk/c3t0d0 /dev/rdsk/c3t0d0 

srvvodnm,sys,root # vgdisplay -v 
--- Volume groups ---
 VG Name /dev/vg00 VG Write Access read/write VG Status available Max LV 255 Cur LV 19 Open LV 19 Max PV 16 Cur PV 2 Act PV 2 Max PE per PV 5372 VGDA 4 PE Size (Mbytes) 64 Total PE 2186 Alloc PE 2038 Free PE 148 Total PVG 2 Total Spare PVs 0 Total Spare PVs in use 0
 --- Logical volumes --- LV Name /dev/vg00/lvol1 LV Status available/syncd LV Size (Mbytes) 256 Current LE 4 Allocated PE 8 Used PV 2 
LV Name /dev/vg00/lvol2 LV Status available/syncd LV Size (Mbytes) 8192 Current LE 128 Allocated PE 256 Used PV 2 
LV Name /dev/vg00/lvol3 LV Status available/stale LV Size (Mbytes) 256 Current LE 4 Allocated PE 8 Used PV 2 LV Name /dev/vg00/lvol4 LV Status available/syncd LV Size (Mbytes) 1216 Current LE 19 Allocated PE 38 Used PV 2 LV Name /dev/vg00/lvol5 LV Status available/stale LV Size (Mbytes) 5504 Current LE 86 Allocated PE 172 Used PV 2
 LV Name /dev/vg00/lvol6 LV Status available/syncd LV Size (Mbytes) 256 Current LE 4 Allocated PE 8 Used PV 2 LV Name /dev/vg00/lvol7 LV Status available/stale LV Size (Mbytes) 6528 Current LE 102 Allocated PE 204 Used PV 2
 LV Name /dev/vg00/lvol9 LV Status available/syncd LV Size (Mbytes) 2048 Current LE 32 Allocated PE 64 Used PV 2 
LV Name /dev/vg00/lvol10 LV Status available/syncd LV Size (Mbytes) 512 Current LE 8 Allocated PE 16 Used PV 2 
LV Name /dev/vg00/lvol11 LV Status available/syncd LV Size (Mbytes) 512 Current LE 8 Allocated PE 16 Used PV 2
LV Name /dev/vg00/lvol12 LV Status available/stale LV Size (Mbytes) 5056 Current LE 79 Allocated PE 158 Used PV 2 
LV Name /dev/vg00/lvol13 LV Status available/stale LV Size (Mbytes) 23552 Current LE 368 Allocated PE 736 Used PV 2 
LV Name /dev/vg00/lvol15 LV Status available/stale LV Size (Mbytes) 4736 Current LE 74 Allocated PE 148 Used PV 2 
LV Name /dev/vg00/lvol16 LV Status available/stale LV Size (Mbytes) 1216 Current LE 19 Allocated PE 38 Used PV 2 
LV Name /dev/vg00/lvol17 LV Status available/stale LV Size (Mbytes) 384 Current LE 6 Allocated PE 12 Used PV 2 LV Name /dev/vg00/lvol18 LV Status available/stale LV Size (Mbytes) 2752 Current LE 43 Allocated PE 86 Used PV 2 
LV Name /dev/vg00/lvol19 LV Status available/stale LV Size (Mbytes) 1344 Current LE 21 Allocated PE 42 Used PV 2
LV Name /dev/vg00/lvol20 LV Status available/syncd LV Size (Mbytes) 128 Current LE 2 Allocated PE 4 Used PV 2 LV Name /dev/vg00/lvol21 LV Status available/stale LV Size (Mbytes) 768 Current LE 12 Allocated PE 24 Used PV 2 
--- Physical volumes --- PV Name /dev/dsk/c2t1d0 PV Status available Total PE 1093 Free PE 74 Autoswitch On PV Name /dev/dsk/c3t0d0 PV Status available Total PE 1093 Free PE 74 Autoswitch On 
--- Physical volume groups --- PVG Name pvg0 PV Name /dev/dsk/c2t1d0 PVG Name pvg1 PV Name /dev/dsk/c3t0d0 the output of the scbackup: srvvodnm,sys,root # 
tail scbackup.log NOTE: > scbackup -R 
=====>>>>> 18-Apr-2012 13:15:40 MERGE-DELETE process activated on splitted Logical Volumes ERROR: MERGE-DELETE process cannot be performed. Command file not found: /SCINSTALL/data/backup/scbackup_recover_tmp =====>>>>> 18-Apr-2012 18:53:58 Starting SCBACKUP procedure ( Backup-Restore Ver. 3.1.4 ) NOTE: > scbackup -R =====>>>>> 18-Apr-2012 18:53:58 MERGE-DELETE process activated on splitted Logical Volumes ERROR: MERGE-DELETE process cannot be performed. Command file not found: /SCINSTALL/data/backup/scbackup_recover_tmp

rvvodnm,sys,root # lvdisplay /dev/vg00/lvol3 --- Logical volumes --- LV Name /dev/vg00/lvol3 VG Name /dev/vg00 LV Permission read/write LV Status available/stale Mirror copies 1 Consistency Recovery MWC Schedule parallel LV Size (Mbytes) 256 Current LE 4 Allocated PE 8 Stripes 0 Stripe Size (Kbytes) 0 Bad block off Allocation PVG-strict/contiguous IO Timeout (Seconds) default 

also note some error on the system log may be releated to the SCSI (Tape)!! pr 18 18:36:26 srvvodnm vmunix: Apr 18 18:36:26 srvvodnm vmunix: SCSI: Read error -- dev: b 31 0x021000, errno: 126, resid: 1024, Apr 18 18:36:26 srvvodnm vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 1024. Apr 18 18:36:17 srvvodnm su: + tty?? root-root Apr 18 18:36:26 srvvodnm above message repeats 2 times Apr 18 18:36:26 srvvodnm vmunix: LVM: VG 64 0x000000: PVLink 31 0x021000 Failed! The PV is not accessible. Apr 18 18:36:26 srvvodnm vmunix: LVM: VG 64 0x000000: PVLink 31 0x021000 Recovered. Apr 18 18:36:43 srvvodnm EMS [2761]: ------ EMS Event Notification ------ Value: "SERIOUS (4)" for Resource: "/storage/events/disks/default/0_1_ 1_0.1.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 180944898 -r /storage/ev ents/disks/default/0_1_1_0.1.0 -n 180944897 -a Apr 18 18:36:46 srvvodnm su: + tty?? root-root Apr 18 18:37:37 srvvodnm su: + tty?? root-axadmin

[Torsten.]

The output is NOT readable (bad format), but there must be a bad disk that needs replacement.

[hahmedma]

here is the investiagation on the attached on the .txt formate

[Torsten.]

Investigate disk /dev/rdsk/c2t1d0 according to

When_Good_Disks_Go_Bad_WP
http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c01911837/c01911837.pdf


also do

/opt/resmon/bin/resdata -R 180944898 -r /storage/ev
ents/disks/default/0_1_1_0.1.0 -n 180944897 -a

Confirm the disk is bad and replace!

[hahmedma]

Thanks for your quick replay....

please find attached the Output of the command which you send...

and you confirm that the Problem on hard desk is faluier and need to replaced... or we need to check it's connection to the SCSI...

or may be some problem on the SCASI Cable need to be replace...

thanks for your Feedback...

[Torsten.]

Since this is an internal disk with a write failure, i don't think there is a cable problem. Do the "dd" test according to the manual and proceed with a disk replacement.

[hahmedma]

this the O/P of the dd comand

 dd if=/dev/rdsk/c2t1d0 of=/dev/null bs=1024k &
[1]     28954

[Torsten.]

The output would look like

2352+0 records in
2352+0 records out

or

dd read error: I/O error

you just sent the process to background - watch out for the result.

[hahmedma]

this is the O/P:

srvvodnm,sys,root #  dd if=/dev/rdsk/c2t1d0 of=/dev/null bs=1024k
1214+0 records in
1214+0 records out

srvvodnm,sys,root #  dd if=/dev/rdsk/c2t1d0 of=/dev/null bs=1024k count=64 &
64+0 records in
64+0 records out

thanks for feedback

[Torsten.]

I would replace the disk anyway, because of all the write errors.

Obviously it cannot sync the mirror because of these errors.

[hahmedma]

ok thanks for your feedback we will try to replace the hard desk....