Cannot lvextend - not enough free extents - again!

[ConnieK]

HPUX 11.23
Added 10 32GB LUNS to existing VG
One LV in VG
vgdisplay output

--- Volume groups ---
VG Name /dev/vg02
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 1
Open LV 1
Max PV 128
Cur PV 57
Act PV 57
Max PE per PV 50000
VGDA 114
PE Size (Mbytes) 128
Total PE 15333
Alloc PE 12643
Free PE 2690
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg02/lvol1
LV Status available/syncd
LV Size (Mbytes) 1618304
Current LE 12643
Allocated PE 12643
Used PV 47

lvdisplay:
--- Logical volumes ---
LV Name /dev/vg02/lvol1
VG Name /dev/vg_vg02
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule striped
LV Size (Mbytes) 1618304
Current LE 12643
Allocated PE 12643
Stripes 47 <---- ?
Stripe Size (Kbytes) 64
Bad block on
Allocation strict
IO Timeout (Seconds) default

When I try to extend the LV by available MB, I get this error:

ohons002:/dev/rdsk# lvextend -L 1960000 /dev/vg02/lvol1
Warning: rounding up logical volume size to extent boundary at size "1960064" MB.
Warning: rounding up logical volume size to extent boundary at size "1961216" MB for striping.
lvextend: Not enough free physical extents available.
Logical volume "/dev/vg02/lvol1" could not be extended.
Failure possibly caused by strict allocation policy


What am I doing wrong? Please help!

[Berd]

ConnieK,

I think you need to use vgdisplay -v vg02 to see how your spare disk space is allocated. I guess you do not have sufficient spare disk space on enough physical volumes to satisfy the strict allocation policy for striping.

HTH

Berd

[ConnieK]

The vgdisplay is there at the top. Here it is again:
--- Volume groups ---
VG Name /dev/vg02
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 1
Open LV 1
Max PV 128
Cur PV 57
Act PV 57
Max PE per PV 50000
VGDA 114
PE Size (Mbytes) 128
Total PE 15333
Alloc PE 12643
Free PE 2690
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg02/lvol1
LV Status available/syncd
LV Size (Mbytes) 1618304
Current LE 12643
Allocated PE 12643
Used PV 47

Here is just one of the used PV's:
PV Name /dev/dsk/c18t9d1
PV Name /dev/dsk/c16t9d1 Alternate Link
PV Status available
Total PE 269
Free PE 0
Autoswitch On
Proactive Polling On

Here is one of the newly added PV's:
PV Name /dev/dsk/c16t6d2
PV Name /dev/dsk/c18t6d2 Alternate Link
PV Status available
Total PE 269
Free PE 269
Autoswitch On
Proactive Polling On

[ConnieK]

More info. I found out this one lvol contains a Seibel database and it was specifically set up this way - striped across all existing PV's.

What I need an answer on is how can I extend the LV? Should I not be able to add all the space added to the VG? (320GB).

Please reply soon as I am sort of under the gun.... :-)

Thanks!

[Patrick Wallek]

Since the LV is set up as a striped LV, you need disk space available on ALL 47 PVs that the LV is striped across.

If even 1 PV does not have sufficient free disk space, then your lvextend will fail.

[chris huys_4]

Hi,

Patrick means to add 320GB of diskspace on a logical volume with 47 stripes, you dont need to add 10 luns of 32GByte, but 47 luns of (320 GB /47 stripes=)6.9GByte.

Greetz,
Chris

[ConnieK]

Chris and Patrick. Thank you for your insight. Could you also give me the command structure for adding the LUNs back in? I will vgreduce the 10 out first. I just don't want to have to do this again and I am not sure of the specific way to do it. Please?

[chris huys_4 · Accepted solution]

Hi Connie,

#lvdisplay -v /dev/vg02/lvol1
The 47 first logical extents of the /dev/vg02/lvol1 should give an idea what "lun" device files were used as "primary devicefiles" of the 47 "existing luns".

"In the same way" vgextend should then also be extended with 47 "device files" of the 47 new luns. And this with the goal, to spread IO as much as possible "over" all the hba controllers. When the primary device files all are added in lvmtab for the 47 newly added luns, offcourse alternate device files will need also to get added.

adding primary devicefiles

#vgextend /dev/ /dev/dsk/ ... /dev/dsk/

adding alternate devicefiles
#vgextend /dev/ /dev/dsk/ ... /dev/dsk/


Followed then offcourse with the lvextend command lvextend -L 1960000 /dev/vg02/lvol1
.

To explain it a bit better.
f.e. with a lvol of 2 stripes. (oh yes did I mention that 47 stripes is totally over the top. ;) )

gsesrv8 # strings /etc/lvmtab
/dev/vgtest
:o}B
/dev/dsk/c4t0d1
/dev/dsk/c8t3d3

The first lun of volumegroup vgtest, is c4t0d1 and thus sends its IO through controller c4, the second lun is c8t3d3 and thus sends its IO through controller c8

NOTE: normally after the above primary device files also the alternate links should have been added in lvmtab for /dev/vgtest, so a c8t0d1 and c4t3d3, but apparantly I forget to add them..

gsesrv8 # lvcreate -i 2 -l 30 /dev/vgtest

gsesrv8 # lvdisplay -v /dev/vgtest/lvol3
--- Logical volumes ---
LV Name /dev/vgtest/lvol3
VG Name /dev/vgtest
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule striped
LV Size (Mbytes) 960
Current LE 30
Allocated PE 30
Stripes 2
Stripe Size (Kbytes) 8
Bad block on
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c4t0d1 15 15
/dev/dsk/c8t3d3 15 15

--- Logical extents ---
LE PV1 PE1 Status 1
00000 /dev/dsk/c4t0d1 00030 current
00001 /dev/dsk/c8t3d3 00030 current
00002 /dev/dsk/c4t0d1 00031 current
00003 /dev/dsk/c8t3d3 00031 current

>> CONCLUSION -- the adding of c4t0d1 as primary device file for the first lun in lvmtab and then c8t3d3 as primary device file for the second lun, will cause the striped volume lvol3 to effectively use c4t0d1 for the first/third/fifth/.. extent and device file c8t3d3, for its second/fourth/sixth extent, causing the IO to be spread equally over the 2 controllers

Greetz,
Chris

[ConnieK]

Chris,

I am not sure if I or you are not completely understanding.... I need to add 10 LUN's to an existing VG - which I already have. Then I need to lvextend the LV. There is only one LV in this VG. RE: lvdisplay you referenced above - here is a snippet of mine -
----- snip -------------
--- Logical volumes ---
LV Name /dev/vg02/lv01
VG Name /dev/vg02
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule striped
LV Size (Mbytes) 1618304
Current LE 12643
Allocated PE 12643
Stripes 47
Stripe Size (Kbytes) 64
Bad block on
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c18t7d2 269 269
/dev/dsk/c18t7d3 269 269
/dev/dsk/c18t7d4 269 269
/dev/dsk/c18t7d5 269 269

etc.... - through 47 disks

--- Logical extents ---
LE PV1 PE1 Status 1
00000 /dev/dsk/c18t7d2 00000 current
00001 /dev/dsk/c18t7d3 00000 current
00002 /dev/dsk/c18t7d4 00000 current
00003 /dev/dsk/c18t7d5 00000 current
00004 /dev/dsk/c18t7d6 00000 current
00005 /dev/dsk/c18t7d7 00000 current
00006 /dev/dsk/c18t8d0 00000 current
00007 /dev/dsk/c18t8d1 00000 current

etc.....

12629 /dev/dsk/c18t11d3 00268 current
12630 /dev/dsk/c18t11d4 00268 current
12631 /dev/dsk/c18t11d5 00268 current
12632 /dev/dsk/c18t11d6 00268 current
12633 /dev/dsk/c18t11d7 00268 current
12634 /dev/dsk/c18t12d0 00268 current
12635 /dev/dsk/c18t12d1 00268 current
12636 /dev/dsk/c18t12d2 00268 current
12637 /dev/dsk/c18t12d3 00268 current
12638 /dev/dsk/c18t12d4 00268 current
12639 /dev/dsk/c18t12d5 00268 current
12640 /dev/dsk/c18t12d6 00268 current
12641 /dev/dsk/c18t12d7 00268 current
12642 /dev/dsk/c18t13d0 00268 current
EOF
------- snip--------

I cannot rebuild the VG as there is an existing database resident on this one lv.

I thought maybe I would reduce the vg by the 10 LUN's I had added and start adding them again. I understand about vgextending over the alternate paths to the controllers; I already did that.

What I need to know is:

1. Do I need to vgextend the VG differently than #vgextend vg02 /dev/dsk/c16t2d0 /dev/dsk/c18t2d0?

2. Do I need to lvextend a different way than # lvextend -L /dev/vg02/lvol1.

The man pages on lvextend give flags that I have never used - the "-m" for mirror copies and the "-l" for logical extent number. Do I need to use these flags?

I hate being an ignoramus... but I really don't get it.

;-)

[ConnieK]

Please - can anyone out there answer this?

[Patrick Wallek]

When you try to extend this LV, it will try to evenly extend it over the # of PVs specified in the 'stripes' parameter, which in this case is 47.

You are NOT going to be able to extend this VG without adding 47 more PVs. Just adding 10 PVs is NOT going to work.

[RajuD]

Hi ConnieK ,

It is impossible to extend the lv unless you allocated 47luns.

Possible solution...
Take complete filesystem backup of LV persent in the vg.

Destroy all the luns

Recreate the vg

Restore the backp.

[shanmuhanandam]

Hi,
As per the output you given, your existing lv is striped accross 47 luns. for extending the same lv you need more 47 luns.
Or else take the entire lv backup, remove it, then add 10 new luns, create the new lv, then restore the data.

Thanks,
Shan,.

[ConnieK]

Now I understand! I appreciate all the input and the clarifications at the bottom.

I did not build the original vg or lv, so I had no idea what was done. And still have no idea WHY it was done this way - striping over 47 luns!

I intend to clarify/verify with the customer that this is exactly what is wanted meaning actual striping of data over 57 LUNs (about 1824 GB) - to back up the data for restore, to destroy the existing lv, add in the additional 10 LUNs, and recreate the lv, striping across 57 LUNs, and then restore the data.

Also, I will need to check to ensure I actually can create one lv that big (1,867,776 MB).

[RajuD]

HI ConnieK,

sorry for the late reply...

I hope it could be a complex task to take complete data backup; since the data present in the volume group is huge in size.

I belive this vg has been created purely to improve performace, the lv presented in the volume might provide you output of read performance for the user...

Suggestion: Kindly co-ordinate with your application team / Customer get the exact requirement; Check with them if they able to utilize a new volume group by allocating 10 if you create...

[chris huys_4]

Hi,

You seem to have figured it out, but for future reference. ;)

When in doubt, design it out, so in my best ascii-art ;)

This is the current situation.

column1 column2 column47
lun 1 lun 2 ... lun 47
c18t7d2 c18t7d3 c18t13d0
+------+ +-----+ +------+
LE0|PE0 |LE1 |PE0 | ... LE46|PE0 |
+------+ +-----+ +------+
LE47|PE1 |LE49 |PE1 | ... LE93|PE1 |
+------+ +-----+ +------+
. . .
. . .
. . .
LE |PE268 |LE |PE268| ...LE |PE268 |
12596| |12596| | 12642| |
+------+ +-----+ +------+

The "first" stripe exists out of logical extent 0, LE0, till, logical extent 46, LE46, i.e.
all "physical extent 0", PE0, of all 47 luns.

NOTE: "stripes are the rows, luns the columns".

The "second" stripe exists out of LE47 till LE93, i.e. all "physical extents 1"
of all 47 luns.

The "269th" stripe, exists out of LE12596 till LE12642, i.e. all PE268 of all 47 luns.

NOTE: We "say its a 47 stripes logical volume", but what we "mean" with that,
in fact, is that 1 stripe, consists out of 47 luns, not that the volume
contains 47 stripes, as you can see in the above volume, which has a total of 269 stripes.

Every lun until now had 269 physical extents, PE0 till PE269, and each
extent is 128Mbyte in size, PE Size (Mbytes) 128.

Thus every of the 47 existing luns, until now, was 128Mbyte*269PE=34432 Mbyte
in size, or around 33Gbyte in size.

Now by adding 10 extra luns of around 33 Gbyte, what was done in fact, ..

column1 column2 column47
lun 1 lun 2 ... lun 47
c18t7d2 c18t7d3 c18t13d0
+-------+ +------+ +------+
LE0|PE0 |LE1 |PE0 | ... LE46|PE0 |
+-------+ +------+ +------+
LE47|PE1 |LE48 |PE1 | ... LE93|PE1 |
+-------+ +------+ +------+
. . .
. . .
. . .
LE |PE268 |LE |PE268 | ...LE |PE268 |
12596| |12596| | . 12642| |
+-------+ +------+ . +------+
|lun48 | |lun49 | . |lun57 |.
|c16t2d0| |cxtydz| |cx9ty9dz9|
+-------+ +------+ +--------+
LE |PE269 |LE |PE269 |.LE |PE269 |
12643| |12644| |.12652| |
+-------+ +------+ +-------+
LE |PE270 | |PE270 | |PE270 |
12653| | | | | |

.. was that while, "column1", "is extended", with lun48, the first of the 10 newly added luns,
and this up to "column10", which is extended with lun57, the 10th of the 10 newly added luns,
after this "the columns11 till columns 47" "couldnt extent" with a "newly added lun" anymore..
And thus no new stripes of 47 luns could be created anymore and thus the lvextend failed.

Destroying the logical volume and creating a 57 striped logical volume and then restoring the
data will work, but will also make that in the future, when the logical volume would need again
to be extended, the same problem will exist, now not with 47 luns, but instead with 57 luns.


Personally, I would just destroy the logical volume and just create a 'normal', i.e. not striped,
logical volume of now 57 luns and restore the data to such a logical volume, or at least go to
a 8 striped volume or some such.

Yes, you might loose some performance, but todays diskarrays cache/logic, should alleviate most
of it, and extending the logical volume in the future, will not get you into the same kind of problems.

NOTE: LVM is just not flexible enough to consider striping, especially large amount of luns per stripe,
(payable) VxVM is must better for this purpose as its allows for online changing of the number of stripes.

Greetz,
Chris

[ConnieK]

Closed