How to release PE's not assigned to any LV

[Carl Houseman]

I've got a PV with 4000 PE's not assigned to any LV, but not available either. If I pvdisplay -v I get this for the PE's at the end:

03812 current ??? 00000
03813 current ??? 00001
03814 current ??? 00002
03815 current ??? 00003
etc.

This PV was in this condition before my time so I don't know what might have caused it.

How can I release these PE's so they can be incorporated into a LV? If my only choice to blow away the PV and re-create?

I did find another thread discussing this problem (http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1064012) but the only potentially helpful answer was a link to a thread in the Europe itrc site which isn't available in the America/Asias itrc site.

[Senthil Kumar .A_1]

Hi Carl,

I tried that EU link, but it says the requested doc is not found. Could be my ID is not previledged enough to read it. Anyway,

please run "pvdisplay -v /dev/dsk/c#t#d# | more",.... you need to observe the following part of the output
.
.
.

--- Distribution of physical volume ---
LV Name LE of LV PE for LV
/dev/vg00/lvol1 30 30
/dev/vg00/lvol2 128 128
/dev/vg00/lvol3 375 375
/dev/vg00/lvol4 50 50
/dev/vg00/lvol5 768 768
/dev/vg00/lvol6 75 75
/dev/vg00/lvol7 200 200
/dev/vg00/lvol8 319 319
/dev/vg00/lvol9 250 250
/dev/vg00/lvol10 250 250
/dev/vg00/lvol11 1750 1750

--- Physical extents ---
PE Status LV LE
00000 current /dev/vg00/lvol1 00000
00001 current /dev/vg00/lvol1 00001
.
.
.
.


NOTE: notice the "Distribution of physical volume" part. If you do not find any valid LV entry in this section and also after you can cross check by running vgdisplay -v on the concerned VG and make sure no LV's are in-consistent state. If not, then feel free to do the following..

vgreduce vg## /dev/dsk/c#t#d#
vgextend vg## /dev/dsk/c#t#d#

...I think this should solve your problem.

Senthil

[Senthil Kumar .A_1]

Hi Carl,

On a second thoought, it seems you don't have this state for the entire list of PE in that PV, do you?...

Why not try,

vgscan -pv
vgscan -a

May be it might sync the data in VGDA's and PVRA's with the lvmtab file. Its worth a try, though not sure it might fix it or not; before doing anything drastic.

Senthil

[Jollyjet]

I've got a PV with 4000 PE's not assigned to any LV----- it say's that the physical volume is not used at all.

better create a new pv in that pvcreate -f /dev/rdsk/C#t#t# and create volume group and logical volume's.

[Mridul Shrivastava]

Locate the Vg/ LV which contains this disk, execute vgdisplay and If you find difference between active and current PV that means this vg contains the disk.
then execute the lvdisplay of lvs to find out which LV is showing ??? for a mirrored disk ( idealy it should show the disk name). Once this is done follow the below mentioned procedure.

Execute the following commands for the affected LV:

lvdisplay -v /dev/vgxx/
lvdisplay -v -k /dev/vgxx/

Then compare the outputs.. you will get a key value of the disk for which it is showing ???.. then you can use the following command to remove the disk from the lv

lvreduce -k -m 0 /dev/vgxx/

disk is showing as ??? because it is failed or missing so this the only way out to remove the disk from lv/vg... once this is done execute the following command.

vgreduce -f

This will remove all the disk which are not present in /etc/lvmtab for this Vg.. once lv is reduced..

-k This option should be used only in the
special instance when you want to reduce a
mirrored logical volume on a physical volume
that is missing or has failed.

The vgreduce command with -f option removes all missing physical volume from the volume group.

[Carl Houseman]

Hi all, I'm just coming back to this as it has now become an important problem to solve. :)

Also, I kind of gave up fixing it the last time because the responses here had no clear agreement as to the fix, and I am very concerned that I "first do no harm" in attempting the fix.

Regarding the lvdisplay -v and lvdisplay -v -k, the lv which isn't mirrored to the second disk doesn't show up as having PE's on the second disk. For example, in a normal mirrored lv, I get this:

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c0t6d0 1000 1000
/dev/dsk/c3t6d0 1000 1000

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 0 02812 current 1 02812 current
00001 0 02813 current 1 02813 current

For the unmirrored lv I get just this:

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c0t6d0 528 528

--- Logical extents ---
LE PV1 PE1 Status 1
00000 1 03812 current
00001 1 03813 current

Regarding pvdisplay -v for each pv, the differences are:

--- Distribution of physical volume ---
LV Name LE of LV PE for LV
/dev/vg01/lvol1 50 50
/dev/vg01/lvol2 512 512
/dev/vg01/lvol3 50 50
/dev/vg01/lvol4 1000 1000
/dev/vg01/lvol5 200 200
/dev/vg01/lvol6 500 500
/dev/vg01/lvol7 500 500
/dev/vg01/lvol8 1000 1000
/dev/vg01/lvol9 528 528

and
--- Distribution of physical volume ---
LV Name LE of LV PE for LV
/dev/vg01/lvol1 50 50
/dev/vg01/lvol2 512 512
/dev/vg01/lvol3 50 50
/dev/vg01/lvol4 1000 1000
/dev/vg01/lvol5 200 200
/dev/vg01/lvol6 500 500
/dev/vg01/lvol7 500 500
/dev/vg01/lvol8 1000 1000

All of the lv's show "available/syncd" in vgdisplay -v. Here's the end of that display:

LV Name /dev/vg01/lvol8
LV Status available/syncd
LV Size (Mbytes) 8000
Current LE 1000
Allocated PE 2000
Used PV 2

LV Name /dev/vg01/lvol9
LV Status available/syncd
LV Size (Mbytes) 4224
Current LE 528
Allocated PE 528
Used PV 1

--- Physical volumes ---
PV Name /dev/dsk/c0t6d0
PV Status available
Total PE 4340
Free PE 0
Autoswitch On

PV Name /dev/dsk/c3t6d0
PV Status available
Total PE 4340
Free PE 0
Autoswitch On


So it looks like I can try either:

vgreduce vg01 /dev/dsk/c3t6d0

or

vgreduce -f vg01

and either should have the same results, wouldn't you agree?

Final thing, I want to be absolutely sure that either vgreduce command above will not remove anything in use or otherwise damage anything. Everyone in agreement, please say "aye".

[Torsten.]

Look at this:

--- Distribution of physical volume ---
LV Name LE of LV PE for LV
/dev/vg01/lvol1 50 50
/dev/vg01/lvol2 512 512
/dev/vg01/lvol3 50 50
/dev/vg01/lvol4 1000 1000
/dev/vg01/lvol5 200 200
/dev/vg01/lvol6 500 500
/dev/vg01/lvol7 500 500
/dev/vg01/lvol8 1000 1000
/dev/vg01/lvol9 528 528

and
--- Distribution of physical volume ---
LV Name LE of LV PE for LV
/dev/vg01/lvol1 50 50
/dev/vg01/lvol2 512 512
/dev/vg01/lvol3 50 50
/dev/vg01/lvol4 1000 1000
/dev/vg01/lvol5 200 200
/dev/vg01/lvol6 500 500
/dev/vg01/lvol7 500 500
/dev/vg01/lvol8 1000 1000


The first PV has 4340 PEs for lvol1 ... 8 + lvol9 ;
the second has 3812 PEs for lvol1 ..8.

But both reporting 4340 PEs in use!

How about lvol9?

What does lvdisplay reports about this?


Your vgreduce command will fail, because there are still valid PEs on this disk.

[Carl Houseman]

lvol9 exists only on the one pv... lvdisplay -v /dev/vg01/lvol9:

--- Logical volumes ---
LV Name /dev/vg01/lvol9
VG Name /dev/vg01
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 4224
Current LE 528
Allocated PE 528
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c0t6d0 528 528

--- Logical extents ---
LE PV1 PE1 Status 1
00000 /dev/dsk/c0t6d0 03812 current
00001 /dev/dsk/c0t6d0 03813 current
00002 /dev/dsk/c0t6d0 03814 current
00003 /dev/dsk/c0t6d0 03815 current
00004 /dev/dsk/c0t6d0 03816 current
...
00524 /dev/dsk/c0t6d0 04336 current
00525 /dev/dsk/c0t6d0 04337 current
00526 /dev/dsk/c0t6d0 04338 current
00527 /dev/dsk/c0t6d0 04339 current

[Torsten. · Accepted solution]

I can imagine there is another LVOL on this disk, but someone removed the lvol device file.

Please post

# ll /dev/vg00

[Torsten.]

Look at my test on my test system:

The "normal" output of pvdisplay:
======================================

PV Name /dev/dsk/c0t6d0
VG Name /dev/vg00
PV Status available
Allocatable yes
VGDA 2
Cur LV 9 <<===
PE Size (Mbytes) 4
Total PE 2168
Free PE 0
Allocated PE 2168
Stale PE 0
IO Timeout (Seconds) default
Autoswitch On
Proactive Polling On

--- Distribution of physical volume ---
LV Name LE of LV PE for LV
/dev/vg00/lvol1 75 75
/dev/vg00/lvol2 96 96
/dev/vg00/lvol3 50 50
/dev/vg00/lvol4 50 50
/dev/vg00/lvol5 125 125 <<===
/dev/vg00/lvol6 853 853
/dev/vg00/lvol7 558 558
/dev/vg00/lvol8 311 311
/dev/vg00/lvol9 50 50


Now I removed the file /dev/vg00/lvol5 (NEVER DO THIS!):

pvdisplay now shows:
======================================

PV Name /dev/dsk/c0t6d0
VG Name /dev/vg00
PV Status available
Allocatable yes
VGDA 2
Cur LV 8 <<===
PE Size (Mbytes) 4
Total PE 2168
Free PE 0
Allocated PE 2168
Stale PE 0
IO Timeout (Seconds) default
Autoswitch On
Proactive Polling On

--- Distribution of physical volume ---
LV Name LE of LV PE for LV
/dev/vg00/lvol1 75 75
/dev/vg00/lvol2 96 96
/dev/vg00/lvol3 50 50
/dev/vg00/lvol4 50 50
/dev/vg00/lvol6 853 853
/dev/vg00/lvol7 558 558
/dev/vg00/lvol8 311 311
/dev/vg00/lvol9 50 50

...
01985 current ??? 00047
01986 current ??? 00048
01987 current ??? 00049
01988 current ??? 00050
01989 current ??? 00051
01990 current ??? 00052


Note the difference:

Cur LV 9
PE Size (Mbytes) 4
Total PE 2168

and

Cur LV 8 <<====
PE Size (Mbytes) 4
Total PE 2168


Now it looks exactly like yours!

If I reverse my changes, everything goes back to normal.

[Carl Houseman]

It seems my lvol9 device file is present and accounted for (and reflects the date when I first created it, previously the space was unallocated to any LV):

root:/ ll /dev/vg01:

total 0
crw-rw-rw- 1 root sys 64 0x010000 Aug 13 2004 group
brw-r----- 1 root sys 64 0x010001 Jun 11 18:28 lvol1
brw-r----- 1 root sys 64 0x010002 Aug 13 2004 lvol2
brw-r----- 1 root sys 64 0x010003 Aug 13 2004 lvol3
brw-r----- 1 root sys 64 0x010004 Aug 13 2004 lvol4
brw-r----- 1 root sys 64 0x010005 Aug 13 2004 lvol5
brw-r----- 1 root sys 64 0x010006 Aug 13 2004 lvol6
brw-r----- 1 root sys 64 0x010007 Aug 13 2004 lvol7
brw-r----- 1 root sys 64 0x010008 Aug 13 2004 lvol8
br--r----- 1 root sys 64 0x01000a Sep 26 2006 lvol9
crw-r----- 1 root sys 64 0x010001 Aug 13 2004 rlvol1
crw-r----- 1 root sys 64 0x010002 Aug 13 2004 rlvol2
crw-r----- 1 root sys 64 0x010003 Aug 13 2004 rlvol3
crw-r----- 1 root sys 64 0x010004 Aug 13 2004 rlvol4
crw-r----- 1 root sys 64 0x010005 Aug 13 2004 rlvol5
crw-r----- 1 root sys 64 0x010006 Aug 13 2004 rlvol6
crw-r----- 1 root sys 64 0x010007 Aug 13 2004 rlvol7
crw-r----- 1 root sys 64 0x010008 Aug 13 2004 rlvol8
cr--r----- 1 root sys 64 0x01000a Sep 26 2006 rlvol9

[Torsten.]

Look:

brw-r----- 1 root sys 64 0x010008 Aug 13 2004 lvol8
br--r----- 1 root sys 64 0x01000a Sep 26 2006 lvol9


lvol8 ==> 0x010008

lvol9 ==> 0x01000a

Was there another lvol with 0x010009 in the past? Likely.



One of the most biggest problems are created if someone removing LVOLs or VG by only deleting the device files ... maybe this happens here too.



A very clean way to solve this issue would be to remove all the mirrors from the disk in question, remove disk from VG, pvcreate the disk and start over with mirroring.

[Carl Houseman]

Hoping to find a less disruptive solution than backing out all the mirrors and destroying/re-creating the PV.

[Torsten.]

There is definitively another LVOL on this disk. You may consider to recover the related device files ... and see what happens with this "new" lvol.

Command would be

mknod /dev/vg01/lvol10 b 64 0x010009
mknod /dev/vg01/rlvol10 c 64 0x010009

[Carl Houseman]

Yep, I think you go to the heart of the matter... originally there was an lvol9 whose device files 0x010009 were deleted. I didn't know about that and I had made an lvol9 which came in at 0x01000a.

So the fastest solution in 20/20 hindsight would have been to delete the 0x01000a lvol9 via normal LV removal method, and then re-create the devices files as you've indicated for 0x010009. Since the space is now free on the volume which formerly had the 0x01000a version of lvol9, it can now be extended and mirrored.

BTW I had HP working on this too, and the first thing they told me was to delete the device files for (0x01000a) lvol9, and then they had me create the ones at 0x010009. This restored the original lvol9 but moved the ??? allocated space to the other drive! So between the two of you I now know more about LV device files than I ever wanted....