Same capacity disks, but different PE in VG

[Dimych]

Hi All.

I created VG and LV (/dev/disk/disk1246 - EVA8400-1, /dev/disk/disk1111 - EVA8400-2):

#vgcreate -V 2.1 -s 32 -S 2t /dev/vg02 /dev/disk/disk1246

#lvcreate /dev/vg02

#lvextend - l 32799 /dev/vg02/lvol1

extend this VG:

vgextend /dev/vg02 /dev/disk/disk1111

and....:

# vgdisplay -v vg02
--- Volume groups ---
VG Name /dev/vg02
VG Write Access read/write
VG Status available
Max LV 2047
Cur LV 1
Open LV 1
Max PV 2048
Cur PV 2
Act PV 2
Max PE per PV 65536
VGDA 4
PE Size (Mbytes) 32
Total PE 65536
Alloc PE 32799
Free PE 32737
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0
VG Version 2.1
VG Max Size 2t
VG Max Extents 65536

--- Logical volumes ---
LV Name /dev/vgu33/lvol1
LV Status available/syncd
LV Size (Mbytes) 1049568
Current LE 32799
Allocated PE 32799
Used PV 1

--- Physical volumes ---
PV Name /dev/disk/disk1246
PV Status available
Total PE 32799
Free PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk1111
PV Status available
Total PE 32737
Free PE 32737
Autoswitch On
Proactive Polling On

but:

# diskinfo /dev/rdisk/disk1246
SCSI describe of /dev/rdisk/disk1246:
vendor: HP
product id: HSV450
type: direct access
size: 1074790400 Kbytes
bytes per sector: 512

# diskinfo /dev/rdisk/disk1111
SCSI describe of /dev/rdisk/disk1111:
vendor: HP
product id: HSV450
type: direct access
size: 1074790400 Kbytes
bytes per sector: 512

why two identical disks have different PE?

my brain is crashed (((

Thanks in advance!

[Matti_Kurkela · Accepted solution]

When you created a VG, you set a maximum size limit for it:

#vgcreate -V 2.1 -s 32 -S 2t /dev/vg02 /dev/disk/disk1246

When 2 TiB is divided into 32 MiB-sized extents, the result is exactly 65536 extents.

So the vgcreate command set it as the maximum number of extents for this VG:

# vgdisplay -v vg02

[...]
VG Max Size 2t
VG Max Extents 65536

[...]

The first disk added to the VG was disk1246. Because its size was just slightly more than 1 TiB (in fact, exactly 1025 GiB), all of its capacity could be used, as expected. But when adding disk1111, the VG size reached the 2 TiB limit you set at the VG creation time. So the system did as it was told: it only allocated just enough extents to disk1111 to make the VG reach its maximum size of exactly 2 TiB, and no more.

As a result, some of the space of disk1111 is not mapped into extents, and will be unusable until the VG max size/max extents limits are extended. If the VG is not yet in use, it might be easier to remove it and re-create it with a much larger maximum size limit. If the VG is already in use, read the man page of the vgmodify command:

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c02925766/c02925766.pdf

Lesson learned: when creating a VG, don't set the VG maximum size limit to what you want the VG to be now. Instead, try and make an estimate of the maximum size this VG could possibly need to grow. If unsure, err on the high side. When the VG size reaches the configured maximum size, you will need extra steps (vgmodify) to extend the VG any further. In some cases, those extra steps will require unmounting the filesystem - i.e. application downtime.

In LVM 1.0, the VG maximum size parameters had a direct effect to the size of the VG configuration data (apparently one configuration data record was allocated for each physical extent). If you had a standard-sized root LV and several huge VGs, the LVM configuration backup files could have caused the root filesystem to become 100% full. (That did happen to our 11.11 systems, during a SAN storage migration.)

In LVM 2.x, the configuration data is stored in a different, much more compact form, so you should not need to worry about the size of the LVM configuration data.

[Torsten.]

agreed.

[Dimych]

Oohh, Matti_Kurkela, Thank You.

 I'm probably blinded ...
As I did not see? It's time to holiday.

# vgmodify -a -S 2099136m /dev/vg02

# vgdisplay -v vg02
--- Volume groups ---
VG Name /dev/vg02
VG Write Access read/write
VG Status available
Max LV 2047
Cur LV 1
Open LV 1
Max PV 2048
Cur PV 2
Act PV 2
Max PE per PV 65598
VGDA 4
PE Size (Mbytes) 32
Total PE 65598
Alloc PE 32799
Free PE 32799
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0
VG Version 2.1
VG Max Size 2099136m
VG Max Extents 65598

--- Logical volumes ---
LV Name /dev/vgu02/lvol1
LV Status available/syncd
LV Size (Mbytes) 1049568
Current LE 32799
Allocated PE 32799
Used PV 1

--- Physical volumes ---
PV Name /dev/disk/disk1246
PV Status available
Total PE 32799
Free PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk1111
PV Status available
Total PE 32799
Free PE 32799
Autoswitch On
Proactive Polling On

# lvextend -m 1 -s /dev/vg02/lvol1

# lvsync -T /dev/vg02/lvol1 &

And I got my mirror

Thanks again