unable to remove the failed disk from LVM control

[Anil_32]

Hi All Gurus,

I have a mirrored dsiks and one disk is bad now. I wanna replace it. ioscan -fnC disk is not showing this bad disk. when I am trying to remove it from lvm control it is giving the below error messages.
# lvreduce â m 0 /dev/vg03/lvol1 /dev/dsk/c1t12d0

lvreduce: Physical volume "/dev/dsk/c1t12d0" does not belong
to volume group "/dev/vg03".
Logical volume "/dev/vg03/lvol1" is not reduced.

Any help in how to proceed further is highly appreciated...


Thanks in advance,
An

[Franky_1]

Hi Anil,

first you should break the mirror - use :

lvreduce -m 0 /dev/vg00/lvol1 /dev/dsk/

Regards

Franky

[Dietmar Konermann]

Looks like the disk is not currently attached to the active VG.

1. Check for lvdisplay -v ... should see "???" instead of /dev/dsk/c1t12d0.

2. Then use lvdisplay -v -k ... no you should see the PV key instead of the "???".

3. Try to reduce the mirror with this PV key:
# lvreduce -k -m 0 /dev/vg03/lvol1

(replace with the key found in step 2)

Best regards...
Dietmar.

[Anil_32]

HI Dietmar,

Thanks a lot for your command. It should work now. Before doing that I would like to clarrify one more doubt....
After executing the lvreduce with the key number, will the below command work for reducing the volume group???

#vgreduce /dev/vg03 /dev/dsk/c1t12d0

if not, will it work with "-f" along with vgreduce????

if not, pls let me know, how to remove this bad disk for replacement. I would highly appreciate, if u can provide me the steps...

Thanks in advance,
Anil.

[Dietmar Konermann]

The "vgreduce /dev/vg03 /dev/dsk/c1t12d0" shoud work fine. Please ensure that all lvols with extents mapped to that disk have been reduced before.

Best regards...
Dietmar.

[Anil_32]

Thanks again ...
But how to check what are all the lvols with extents mapped to that disk ???

[Geoff Wild]

lvdisplay -v on each lvol to see the extents mapped to that disk.

for i in `vgdisplay -v vg03 |grep "LV Name"`
do
echo $i
lvdisplay -v $i |grep c1t12d0
done


Rgds...Geoff

[Anil_32]

Hi Geoff,

lvdisplay -v /dev/vg03/lvol1 |grep c1t12d0
is not giving anything. Because this physical disk c1t12d0 is not showing on the system scan now.

Is there any other method to find out this???

But the same above command on the mirrored disk c0t12d0 is showing a very lengthy display as below:
/home/root> lvdisplay -v /dev/vg03/lvol1 |grep c0t12d0
/dev/dsk/c0t12d0 4340 4340
0000 /dev/dsk/c0t12d0 0000 current ??? 0000 stale
0001 /dev/dsk/c0t12d0 0001 current ??? 0001 stale
0002 /dev/dsk/c0t12d0 0002 current ??? 0002 stale
0003 /dev/dsk/c0t12d0 0003 current ??? 0003 stale
0004 /dev/dsk/c0t12d0 0004 current ??? 0004 stale
0005 /dev/dsk/c0t12d0 0005 current ??? 0005 stale
0006 /dev/dsk/c0t12d0 0006 current ??? 0006 stale
0007 /dev/dsk/c0t12d0 0007 current ??? 0007 stale
0008 /dev/dsk/c0t12d0 0008 current ??? 0008 stale
0009 /dev/dsk/c0t12d0 0009 current ??? 0009 stale
0010 /dev/dsk/c0t12d0 0010 current ??? 0010 stale
0011 /dev/dsk/c0t12d0 0011 current ??? 0011 stale
0012 /dev/dsk/c0t12d0 0012 current ??? 0012 stale
0013 /dev/dsk/c0t12d0 0013 current ??? 0013 stale


upto


4335 /dev/dsk/c0t12d0 4335 current ??? 4335 stale
4336 /dev/dsk/c0t12d0 4336 current ??? 4336 stale
4337 /dev/dsk/c0t12d0 4337 current ??? 4337 stale
4338 /dev/dsk/c0t12d0 4338 current ??? 4338 stale
4339 /dev/dsk/c0t12d0 4339 current ??? 4339 stale ..


Actually it is 18GB disk and is fully used by this volume..


pls clarify this !!!

[Geoff Wild]

The "??? 0000 stale " is the missing disk...

Has the disk been physically removed/no longer in ioscan?

IE: diskinfo /dev/rdsk/c1t12d0 returns nothing?


In that case, try:

vgreduce -f /dev/vg03

"force reduction of missing physical volume(s) in a given volume group."

Rgds...Geoff

[Oved]

Did the "lvreduce -m 0" worked for the lvols ?

Because if it did work you are not supposed to see the "current ??? 0000 stale" parts.

Happy Weekend
Oved

[Dietmar Konermann]

Anil,

as I wrote above, you *first* need to deallocate all extents from the disk in question (e.g. using the lvreduce -k -m 0 ... for all lvols where your PV is shown with lvdisplay -k -v ...). *Then* you should be able to perform the vgreduce.

Your lvdisplay -v output shows that LVM still assumes a mirror copy on that disk.

Best regards...
Dietmar.

[Geoff Wild]

Yes - Dietmar's got it:

-k This option should be used only in the special instance when you want to reduce a
mirrored logical volume on a physical volume that is missing or has failed.

lvreduce -k -m 0 /dev/vg03/lvol1

Rgds...Geoff

[Anil_32]

Hi Gurus,

The disk is still on the system. the log messages says that POWER FAILED for this disk and I am yet to replace the disk. Henec it is not appearing in ioscan and lvdisplay.

My doubt is how to freeup the LVM db to take the disk out for replacement and how to put the new disk back in the mirrored copy without loosing the data. Because I do not have much knowledge about HP's LVM.

I would highly appriciate, if any of the Gurus could provide me all the steps from removing the disk for replacement to putting back the new disk in morror without loosing the data....


Thanks,
Anil.

[Dietmar Konermann]

Anil,

if you see POWERFAILED messages in syslog.log then the PV should be still attached... hence it should be visible in lvdisplay -v.

Could you please post a recent extract of your syslog.log along with ioscan -fnkCdisk and strings /etc/lvmtab.

Best regards...
Dietmar.

[Anil_32]

Hi,

Pls find below is the ioscan -fnkC disk
ioscan -fnkC disk
Class I H/W Path Driver S/W State H/W Type Description
=====================================================================
disk 0 8/4.0.0 sdisk CLAIMED DEVICE SEAGATE ST39173WC
/dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
disk 1 8/4.1.0 sdisk CLAIMED DEVICE SEAGATE ST39173WC
/dev/dsk/c0t1d0 /dev/rdsk/c0t1d0
disk 2 8/4.2.0 sdisk CLAIMED DEVICE SEAGATE ST39173WC
/dev/dsk/c0t2d0 /dev/rdsk/c0t2d0
disk 3 8/4.3.0 sdisk CLAIMED DEVICE SEAGATE ST39173WC
/dev/dsk/c0t3d0 /dev/rdsk/c0t3d0
disk 4 8/4.5.0 sdisk CLAIMED DEVICE SEAGATE ST34371N
/dev/dsk/c0t5d0 /dev/rdsk/c0t5d0
disk 5 8/4.8.0 sdisk CLAIMED DEVICE SEAGATE ST34573WC
/dev/dsk/c0t8d0 /dev/rdsk/c0t8d0
disk 11 8/4.12.0 sdisk CLAIMED DEVICE SEAGATE ST318275LC
/dev/dsk/c0t12d0 /dev/rdsk/c0t12d0
disk 6 8/8.0.0 sdisk CLAIMED DEVICE SEAGATE ST39173WC
/dev/dsk/c1t0d0 /dev/rdsk/c1t0d0
disk 7 8/8.1.0 sdisk CLAIMED DEVICE SEAGATE ST39173WC
/dev/dsk/c1t1d0 /dev/rdsk/c1t1d0
disk 8 8/8.2.0 sdisk CLAIMED DEVICE SEAGATE ST39236LC
/dev/dsk/c1t2d0 /dev/rdsk/c1t2d0
disk 9 8/8.3.0 sdisk CLAIMED DEVICE SEAGATE ST39173WC
/dev/dsk/c1t3d0 /dev/rdsk/c1t3d0
disk 10 8/16/5.2.0 sdisk CLAIMED DEVICE TOSHIBA CD-ROM XM-5701TA
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0



Jul 19 03:51:51 BSQAH94 vmunix: LVM: vg[3]: pvnum=1 (dev_t=0x1f01c000) is POWER
FAILED

1f01c000 is the disk ----> c1t12d0

Today I saw few more new errors in syslog.log file after rebooting this machine:


Jul 16 12:08:45 BSQAH94 last message repeated 39 times
Jul 16 12:08:45 BSQAH94 vmunix: SCSI: Target detected parity error -- lbolt: 27
7287, dev: bc01c000
Jul 16 12:08:45 BSQAH94 last message repeated 9 times
Jul 16 12:08:46 BSQAH94 vmunix: ected parity error -- lbolt: 277311, dev: bc01c
000
Jul 16 12:08:46 BSQAH94 vmunix: SCSI: Target detected parity error -- lbolt: 27
7311, dev: bc01c000
Jul 16 12:08:46 BSQAH94 last message repeated 17 times
Jul 16 12:08:46 BSQAH94 named[1160]: ns_req: no address for root server
Jul 16 12:08:46 BSQAH94 vmunix: Target detected parity error -- lbolt: 277319,
dev: bc01c000
Jul 16 12:08:46 BSQAH94 vmunix: SCSI: Target detected parity error -- lbolt: 27
7319, dev: bc01c000
Jul 16 12:08:46 BSQAH94 vmunix: arget detected parity error -- lbolt: 277329, d
ev: bc01c000
Jul 16 12:08:46 BSQAH94 vmunix: SCSI: Target detected parity error -- lbolt: 27
7329, dev: bc01c000
Jul 16 12:08:46 BSQAH94 last message repeated 8 times
Jul 16 12:08:46 BSQAH94 vmunix: Target detected parity error -- lbolt: 277337,
dev: bc01c000
Jul 16 12:08:46 BSQAH94 vmunix: SCSI: Target detected parity error -- lbolt: 27
7337, dev: bc01c000
Jul 16 12:08:46 BSQAH94 last message repeated 26 times
Jul 16 12:08:46 BSQAH94 vmunix: SCSI
Jul 16 12:08:46 BSQAH94 vmunix: parity error -- lbolt: 277363, dev: bc01c000
Jul 16 12:08:46 BSQAH94 vmunix: SCSI: Target detected parity error -- lbolt: 27
7363, dev: bc01c000
Jul 16 12:08:46 BSQAH94 last message repeated 19 times
Jul 16 12:08:46 BSQAH94 vmunix: detected parity error -- lbolt: 277383, dev: bc
01c000
Jul 16 12:08:46 BSQAH94 vmunix: SCSI: Target detected parity error -- lbolt: 27
7383, dev: bc01c000
Jul 16 12:08:46 BSQAH94 last message repeated 12 times
Jul 16 12:08:46 BSQAH94 vmunix: : Target detected parity error -- lbolt: 277389
, dev: bc01c000
Jul 16 12:08:46 BSQAH94 vmunix: SCSI: Target detected parity error -- lbolt: 27
7389, dev: bc01c000





Pls let me know if you need anymore info.

Thanks,
Anil.

[Dietmar Konermann]

pvdisplay -v /dev/dsk/c1t12d0

[Anil_32]

Here is the o/p:


pvdisplay -v /dev/dsk/c1t12d0
pvdisplay: Couldn't find the volume group to which
physical volume "/dev/dsk/c1t12d0" belongs.
pvdisplay: Cannot display physical volume "/dev/dsk/c1t12d0".


Thanks,
Anil.

[Dietmar Konermann]

Just checked... the NO_VGFORPV error happens if the PV cannot be found in /etc/lvmtab! Please post "strings /etc/lvmtab" as requested before.

Best regards...
Dietmar.

[Anil_32]

Thanks a lot for spending time on this.
here is the o/p of lvmtab file:
strings /etc/lvmtab
/dev/vg02
/dev/dsk/c0t0d0
/dev/dsk/c1t0d0
/dev/vg99
/dev/dsk/c0t1d0
/dev/dsk/c1t1d0
/dev/vg01
/dev/dsk/c0t2d0
/dev/dsk/c1t2d0
/dev/vg05
/dev/dsk/c0t3d0
/dev/vg00
/dev/dsk/c0t5d0
/dev/dsk/c0t8d0
/dev/dsk/c1t3d0
/dev/vg03
/dev/dsk/c0t12d0

Thanks,
Anil.

[Dietmar Konermann]

Anil,

so it looks like /dev/dsk/c1t12d0 is attached to vg03, but missing in /etc/lvmtab.

Did you perform a vgscan to rebuild lbmtab? Please restore the lvmtab that was present at vg03 activation time. The again check with lvdisplay and pvdisplay.

I assume that your lvreduce commands will work with that restored lvmtab. Please specify also -A n... otherwise the automatic vgcfgbackup my get hung.

Best regards...
Dietmar.

[Anil_32]

Thanks for that suggession.

Could you pls provide me the sequence of steps for removing this bad disk and putting back the new disk in LVM control for mirrored one with out loosing the data (online). It would be great if you could provide me the steps from the scratch.

Thanks in advance,
Anil.

[Dietmar Konermann · Accepted solution]

1) restore /etc/lvmtab file that contains c1t12d0 for vg03.

2) pvdisplay -v /dev/dsk/c1t12d0
note all lvols that have a mirror copy in this disk

3) lvreduce -m 0 -A n
for all lvols noted in step 2

4) vgreduce -A n vg03 /dev/dsk/1t12d0

5) replace defective disk

6) ioscan -fCdisk
verify that new disk is recognized

7) pvcreate /dev/rdsk/c1t12d0

8) vgextend vg03 /dev/dsk/c1t12d0

9) lvextend -m 1 /dev/dsk/c1t12d0
for all lvols noted in step 2

[Anil_32]

HI,
Thanks a lot for the steps.

we put back the lvmtab file back (original) file and could able to reduce the lvol1 successfully. Now the lvdisplay shows as below:
lvdisplay -v /dev/vg03/lvol1 |pg
lvdisplay: Warning: couldn't query physical volume "/dev/dsk/c1t12d0":
The specified path does not correspond to physical volume attached to
this volume group
lvdisplay: Warning: couldn't query all of the physical volumes.
--- Logical volumes ---
LV Name /dev/vg03/lvol1
VG Name /dev/vg03
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 17360
Current LE 4340
Allocated PE 4340
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c0t12d0 4340 4340

--- Logical extents ---
LE PV1 PE1 Status 1
0000 /dev/dsk/c0t12d0 0000 current
0001 /dev/dsk/c0t12d0 0001 current
0002 /dev/dsk/c0t12d0 0002 current
0003 /dev/dsk/c0t12d0 0003 current
0004 /dev/dsk/c0t12d0 0004 current
0005 /dev/dsk/c0t12d0 0005 current



After that I am not able to reduce the vg.
vgreduce command is not able to reduce it. Below is the o/p for that command u gave me. I tried with vgreduce -f ,even it didn't work.

# vgreduce -f /dev/vg03
vgreduce: Couldn't query physical volume "/dev/dsk/c1t12d0":
The specified path does not correspond to physical volume attached to
this volume group



# vgreduce -A n /dev/vg03 /dev/dsk/c1t12d0
vgreduce: Couldn't query physical volume "/dev/dsk/c1t12d0":
The specified path does not correspond to physical volume attached to
this volume group


Pls let me know how to go further???

thanks,
Anil.

[Dietmar Konermann]

Anil,

to be honest, your system is in a quite weird state. However, please try to copy back that lvmtab file *without* c1t12d0 and perform then a "vgreduce -f vg03".

Hope, that finally helps.
Dietmar.

[Anil_32]

Hi

Thanks a lot/... It worked !!!!!

Now I could able to reduce the vg...

Thanks a lot for your valuable help..

I have yet to get the ordered disk.
Once I get it I will do the further step. I will get back to you, if I have any issues there...

Once again thanks a lot...

Regards,
Anil.