Moving PVs from PVG2 to PVG1 ??

[Johnny Damtoft]

Hello,

I have a volumegroup called VG02 with only one LVOL /dev/vg02/lvol10.
Under VG02 there is 2 PVGs. Currently the system is setup to use PV from PVG1 due to PVG-Strict. This gives me a few problems, now that there is no more space in PVG1.

PVG1 is out of free space, so i want to use the PVs from PVG2. For some reason the PVs in PVG2 is also already partial in use by VG02s lvol10. (i dont know how this happend)

So, how can i more the PVs from PVG2 to PVG1?
I have tried to edit the /etc/lvmpvg file, without any luck.

Any ideas??



Here is a vgdisplay -v vg02:

# vgdisplay -v vg02
--- Volume groups ---
VG Name /dev/vg02
VG Write Access read/write
VG Status available, exclusive
Max LV 255
Cur LV 1
Open LV 1
Max PV 50
Cur PV 16
Act PV 16
Max PE per PV 13000
VGDA 32
PE Size (Mbytes) 8
Total PE 83328
Alloc PE 65000
Free PE 18328
Total PVG 2
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg02/lvol10
LV Status available/syncd
LV Size (Mbytes) 520000
Current LE 65000
Allocated PE 65000
Used PV 16


--- Physical volumes ---
PV Name /dev/dsk/c108t0d0
PV Status available
Total PE 5208
Free PE 0
Autoswitch On

PV Name /dev/dsk/c114t0d0
PV Status available
Total PE 5208
Free PE 0
Autoswitch On

PV Name /dev/dsk/c108t0d1
PV Status available
Total PE 5208
Free PE 0
Autoswitch On

PV Name /dev/dsk/c114t0d1
PV Status available
Total PE 5208
Free PE 0
Autoswitch On

PV Name /dev/dsk/c108t0d2
PV Status available
Total PE 5208
Free PE 0
Autoswitch On

PV Name /dev/dsk/c114t0d2
PV Status available
Total PE 5208
Free PE 0
Autoswitch On

PV Name /dev/dsk/c108t0d3
PV Status available
Total PE 5208
Free PE 0
Autoswitch On

PV Name /dev/dsk/c114t0d3
PV Status available
Total PE 5208
Free PE 0
Autoswitch On

PV Name /dev/dsk/c108t0d4
PV Status available
Total PE 5208
Free PE 2289
Autoswitch On

PV Name /dev/dsk/c114t0d4
PV Status available
Total PE 5208
Free PE 2289
Autoswitch On

PV Name /dev/dsk/c108t0d5
PV Status available
Total PE 5208
Free PE 2289
Autoswitch On

PV Name /dev/dsk/c114t0d5
PV Status available
Total PE 5208
Free PE 2289
Autoswitch On

PV Name /dev/dsk/c108t0d6
PV Status available
Total PE 5208
Free PE 2293
Autoswitch On

PV Name /dev/dsk/c114t0d6
PV Status available
Total PE 5208
Free PE 2293
Autoswitch On

PV Name /dev/dsk/c108t0d7
PV Status available
Total PE 5208
Free PE 2293
Autoswitch On

PV Name /dev/dsk/c114t0d7
PV Status available
Total PE 5208
Free PE 2293
Autoswitch On


--- Physical volume groups ---
PVG Name pvg1
PV Name /dev/dsk/c108t0d0
PV Name /dev/dsk/c114t0d0
PV Name /dev/dsk/c108t0d1
PV Name /dev/dsk/c114t0d1
PV Name /dev/dsk/c108t0d2
PV Name /dev/dsk/c114t0d2
PV Name /dev/dsk/c108t0d3
PV Name /dev/dsk/c114t0d3

PVG Name pvg2
PV Name /dev/dsk/c108t0d4
PV Name /dev/dsk/c114t0d4
PV Name /dev/dsk/c108t0d5
PV Name /dev/dsk/c114t0d5
PV Name /dev/dsk/c108t0d6
PV Name /dev/dsk/c114t0d6
PV Name /dev/dsk/c108t0d7
PV Name /dev/dsk/c114t0d7

[Doug O'Leary]

Hey;

I'm not that familiar with pvgs; I tend to avoid them. I do know, however, that the /etc/lvmpvg file is optional. What happens if you just make the thing go away?

mv /etc/lmvpvg /etc/lvmpvg1

My understanding is that the vg won't be using pvgs anymore...

HTH;

Doug

[Johnny Damtoft]

Still not working, when removing the /etc/lvmpvg file, the server goes like this:

lvextend: Error detected when reading from file "/etc/lvmpvg".

The VG02/lvol10 is setup to use PVG-Strict

# lvdisplay /dev/vg02/lvol10
--- Logical volumes ---
LV Name /dev/vg02/lvol10
VG Name /dev/vg02
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 520000
Current LE 65000
Allocated PE 65000
Stripes 0
Stripe Size (Kbytes) 0
Bad block NONE
Allocation PVG-strict/distributed
IO Timeout (Seconds) default

[NMory]

Are you using mirroring on this lvol 10
(Mirror Allocation Policy: PVG-strict)?
Are you using Distributed allocation also?
All the disks are assinged to this lvol10?
I think is better if you show us a lvdisplay -v /dev/vg02/lvol10 but just to the point before it shows the Logical Extents distribution, because it could get too long.

Probably you do have mirroring, that's why you have PVG2 already partial in use by VG02s lvol10....

If that's the case you should reduce the mirror on lvol10 in order to reassign the other disks to VG01 with vgextend.

[NMory]

Ok, I see that there is no mirroring, but you do have it configured as distributed.
It's weird that you have some info lvol10 in VG02, because Distributed allocation means that the information will be distributed across all the disks within a PVG.
You should be able to edit the /etc/lvmpvg file and then do a vgextend of the new disks added to PVG1:

#vgextend â g pvg1 vg02 [dev/dsk/c#t#d# ..]
#vgchange -a y vg02

[NMory]

The commands got messed up, don't know why, here they are again:

#vgextend -g pvg1 vg02 [dev/dsk/c#t#d# .....]
#vgchange -a y vg02

[Mridul Shrivastava]

1. Determine which logical volume mirrors to remove from the volume group.(identifying disk)

# vgdisplay -v | more

Located mirrored logical volumes by looking for non-matching Current LE and Allocated PE lines for each logical volume. Mirrors
exist where the Allocated PE count is 2 or 3 times the Current LE's assigned.

Determine the physical disk which will have it's mirrors removed:

# lvdisplay -v /dev/vg07/lvol1 | more

Look for the section near the top that shows something like this:

Mirror copies 1

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c0t3d0 2169 2169
/dev/dsk/c1t3d0 2169 2169


2. Select a disk to be reduced from the logical volume. This example will reduce /dev/dsk/c1t3d0.

# lvreduce -m 0 /dev/vg07/lvol1 /dev/dsk/c1t3d0

Repeat for all mirror disk that need to be removed.


3. Reduce the disk out of the volume group:

# vgreduce vg07 /dev/dsk/c1t3d0

Repeat for all disks that need to be reduced.

4. Add it again in vg using standard pvg option.

[Sundar_7]

Johnny,

With the distributed allocation and PVGs, The PVG groups are simply used to restrict the number of LUNS the LV should span across.

For example, in the above configuration, had you not used the PVG-strict, the LVs will be distributed across all the 16 PVs.

Since you have 2 PVGs, the LV will first be distributed to the first 8 LUNS in the PVG1 and then the LV will start using the LUNS from PVG2.

So, as per my understanding, you should be able to extend the LV as long as you have space in PVG-2.

What exactly is the problem you are having with PVG1 being full ?

Sundar.

[Johnny Damtoft]

** Luis Nassar:
There is no striping or mirroring using LVM, the only reason "PVG-strict/distributed" is used are to spread-out the data. (check the command: man lvchange, and find -D and -s g)

I have not added new disks to the system or PVG2. The luns in PVG2 has been there for some time and used for other lvols, which not has been deleted. The luns in PVG2 should not be used for vg02/lvol10, however the disks are in use by vg02/lvol10. (and i don't understand how)

** Mridul Shrivastava:
In my post from today at 14:14, you can see that there is no mirrors. (Mirror copies 0)
I have thought about doing lvreduce and vgreduce to remove the luns from PVG2, but they are currently used my vg02/lvol10 as mentioned previously.

** Sundar:
Yep, you should be correct. But this is not the case. I have tried to do lvextend several times, even with options to use PVG2 or any other luns that is not in PVG1. Please remember that all the disks/luns in PVG1 are used 100% (no more PE left).

So with PVG1 all used-up all i have left is PVG2, but i can't touch that, for whatever reason?

** ALL:
Your posts got me thinking. What would happen if i was to do a vgextend of the luns from PVG2, into the VG02 volumegroup and PVG1?

All i get, when i do a lvextend is:
# lvextend -L 600000 /dev/vg02/lvol10
lvextend: "LogicalExtentsNumber" is bigger than the maximum value allowed.

In trying to add another 80GB to vg02/lvol10, when vgdisplay tells me that there is 146GB free the above error is confusing.

Please give me your input. Thanks.


// Johnny



# lvdisplay -v /dev/vg02/lvol10
--- Logical volumes ---
LV Name /dev/vg02/lvol10
VG Name /dev/vg02
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 520000
Current LE 65000
Allocated PE 65000
Stripes 0
Stripe Size (Kbytes) 0
Bad block NONE
Allocation PVG-strict/distributed
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c108t0d0 5208 5208
/dev/dsk/c114t0d0 5208 5208
/dev/dsk/c108t0d1 5208 5208
/dev/dsk/c114t0d1 5208 5208
/dev/dsk/c108t0d2 5208 5208
/dev/dsk/c114t0d2 5208 5208
/dev/dsk/c108t0d3 5208 5208
/dev/dsk/c114t0d3 5208 5208
/dev/dsk/c108t0d4 2919 2919
/dev/dsk/c114t0d4 2919 2919
/dev/dsk/c108t0d5 2919 2919
/dev/dsk/c114t0d5 2919 2919
/dev/dsk/c108t0d6 2915 2915
/dev/dsk/c114t0d6 2915 2915
/dev/dsk/c108t0d7 2915 2915
/dev/dsk/c114t0d7 2915 2915

[Sundar_7 · Accepted solution]

oh gee, you should have posted the error message earlier !! :-) .

The error message doesnt indicate lack of space but you are hitting the max PE limit 65535.

The LV is currently occupying 65000 PEs, now extending it by 80 GB means, the number of PEs to be allocated is more than 65535.

To confirm this, try extending the LV to the max

# lvextend -l 65535 /dev/vg02/lvol10

This should work.

The only work around for this problem is to recreate the VG with bigger PE size (say for ex 32 MB).

Sundar

[Johnny Damtoft]

So, are you saying that no matter what the "Max PE per PV" are in this case 13000, the max PEs are limited to 65535??

Damn, and i thought that i was able to extend this lvol beoynd 1TB.

So, i have to create a new VG to fix this?

// Johnny

[Johnny Damtoft]

If i should create a new VG for this lvol, what is the max PE Size? Currently im using 8MB, and you mentioned 32MB - but what is the maximum?

Could i set 128MB or 256MB, or even more?

// Johnny

[Sundar_7]

My understanding is that the Max PEs to a LV is limited to 65535.

Some one year back , the only way to fix this was to recreate the VG with bigger PE size.

But you could give try the following

* get the vgmodify. This is the unsupported utility to modify the VGDA and the VG parameters. But still , the chances of increasing the PE size is only very minimal

* Check for the latest LVM cumulative patches. I dont know if HP has fixed this limitation as yet

If you dont like any of these above options

* backup the FS,destroy the VG, recreate with the bigger PE size, recreate the LV/FS and restore.

Good luck !

Sundar.

[Johnny Damtoft]

Hi Sundar,

Please check this link:
http://docs.hp.com/en/6054/Limits_wp.htm

I have not yet tried to do the lvextend -l 65535 /dev/xx, byt when i read the contect of the link i'm not so sure that there is a max of 65535 PE per LV. There is a max of 65535 PE per PV.

Is this correct, or what is the deal?

// Johnny

[Johnny Damtoft]

Sundar:

After reading this:

-l le_number Allocate space to the logical volume, specified in logical extents. le_number is a decimal value in the range 1 to 65535 (the implementation limit). The default is described above.

It seems that you are right.
The max is 65535 LE, and i'm currently using 65000.

Thanks.

[Mridul Shrivastava]

pe_size :Sets the number of megabytes in each physical extent, where pe_size is expressed in units of megabytes (MB) in the range 1 to 256.pe_size must be equal to a power of 2 (1, 2,4, 8, etc.). The default value for pe_size is 4 (four megabytes).

max_pe: Set the maximum number of physical extents that can be allocated from any of the physical volumes in the volume group. The default value for max_pe is 1016. However, if the size of any physical volume exceeds 1016 times the pe_size, the default value for max_pe is adjusted to match the physical volume size. The maximum number of physical extents can be a value in the range 1 to 65535.

So even after using 65535 extents space is full the other way out is to recreate vg with greater PE size.

[Sundar_7]

Yes, so now you know :-). But I will not go overboard and set the PE size to too high. 32 or 64 MB should be just fine, if you ask me.