strange LVM configuration, pb with mirrored LV into VG00

[Cedric Bruno]

Hello !

Here is a strange issue I encountered this morning about a LVM configuration.

First of all, PV /dev/dsk/c4t6d0 belonging to VG00 was changed due to many write errors, for safety as it appeared as "CLAIMED".

Before replacing this disk the following changes were made :
0) where c4t6d0 is used ?
pvdisplay -v /dev/dsk/c4t6d0 | head -50
--- Physical volumes ---
PV Name /dev/dsk/c4t6d0
VG Name /dev/vg00
PV Status available
Allocatable yes
VGDA 2
Cur LV 9
PE Size (Mbytes) 4
Total PE 1023
Free PE 28
Allocated PE 995
Stale PE 1
IO Timeout default
Autoswitch On

--- Distribution of physical volume ---
LV Name LE of LV PE for LV
/dev/vg00/lvol1 24 24
/dev/vg00/lvol2 128 128
/dev/vg00/lvol3 32 32
/dev/vg00/lvol4 64 64
/dev/vg00/lvol6 128 128
/dev/vg00/lvol7 160 160
/dev/vg00/lvol8 171 171
/dev/vg00/lvol9 224 224
/dev/vg00/lvol10 64 64
1) lvreduce -m 0 /dev/vg00/lvol1 /dev/dsk/c4t6d0 ...
lvreduce -m 0 /dev/vg00/lvol10 /dev/dsk/c4t6d0
2) vgreduce /dev/vg00 /dev/dsk/c4t6d0
3) disk was replaced
4) recreating mirror structure
pvcreate -B /dev/rdsk/c4t6d0
mkboot /dev/rdsk/c4t6d0
mkboot -a "hpux -lq" /dev/rdsk/c4t6d0
cd /usr/sbin/diag/lif
mkboot -b updatediaglif -p ISL -p HPUX -p AUTO -p LABEL /dev/rdsk/c4t6d0
vgextend /dev/vg00 /dev/dsk/c4t6d0
lvextend -m 1 /dev/vg00/lvol1 /dev/dsk/c4t6d0
lvextend -m 1 /dev/vg00/lvol2 /dev/dsk/c4t6d0
lvextend -m 1 /dev/vg00/lvol3 /dev/dsk/c4t6d0
lvextend -m 1 /dev/vg00/lvol4 /dev/dsk/c4t6d0
lvextend -m 1 /dev/vg00/lvol6 /dev/dsk/c4t6d0
lvextend -m 1 /dev/vg00/lvol7 /dev/dsk/c4t6d0
lvextend -m 1 /dev/vg00/lvol8 /dev/dsk/c4t6d0
lvextend -m 1 /dev/vg00/lvol9 /dev/dsk/c4t6d0
lvextend -m 1 /dev/vg00/lvol10 /dev/dsk/c4t6d0

but for LVOL9 we had the following message :
ogical volume "/dev/vg00/lvol9" could not be extended.
Failure possibly caused by strict allocation policy.

Investigating, I found some funny clues :

It seems that LVOL6 is not mirrored as it should be :

--- Logical volumes ---
LV Name /dev/vg00/lvol6
VG Name /dev/vg00
LV Permission read/write
LV Status available/syncd
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 512
Current LE 128
Allocated PE 256
Stripes 0
Stripe Size (Kbytes) 0
Bad block off
Allocation strict/contiguous
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c7t6d0 128 128
/dev/dsk/c4t6d0 128 128

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c7t6d0 00360 current /dev/dsk/c4t6d0 00248 current
....
00127 /dev/dsk/c7t6d0 00487 current /dev/dsk/c4t6d0 00375 current

Therefore, LVOL7 and LVOL8 use 3 PV instead of 2 !!!

LV Name /dev/vg00/lvol7
LV Status available/syncd
LV Size (Mbytes) 896
Current LE 224
Allocated PE 448
Used PV 3

LV Name /dev/vg00/lvol8
LV Status available/syncd
LV Size (Mbytes) 1068
Current LE 267
Allocated PE 534
Used PV 3


--- Logical volumes ---
LV Name /dev/vg00/lvol7
VG Name /dev/vg00
LV Permission read/write
LV Status available/syncd
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 896
Current LE 224
Allocated PE 448
Stripes 0
Stripe Size (Kbytes) 0
Bad block off
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c4t4d0 64 64
/dev/dsk/c7t6d0 160 160
/dev/dsk/c4t6d0 224 224

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c7t6d0 00488 current /dev/dsk/c4t6d0 00376 current
.....
00222 /dev/dsk/c4t4d0 00062 current /dev/dsk/c4t6d0 00598 current
00223 /dev/dsk/c4t4d0 00063 current /dev/dsk/c4t6d0 00599 current

--- Logical volumes ---
LV Name /dev/vg00/lvol7
VG Name /dev/vg00
LV Permission read/write
LV Status available/syncd
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 896
Current LE 224
Allocated PE 448
Stripes 0
Stripe Size (Kbytes) 0
Bad block off
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c4t4d0 64 64
/dev/dsk/c7t6d0 160 160
/dev/dsk/c4t6d0 224 224

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c7t6d0 00488 current /dev/dsk/c4t6d0 00376 current
......
00222 /dev/dsk/c4t4d0 00062 current /dev/dsk/c4t6d0 00598 current
00223 /dev/dsk/c4t4d0 00063 current /dev/dsk/c4t6d0 00599 current

Could someone help me with this issue ?
Thanks for any reply.

Best regards,

Cedric

[John Guster]

after the new disk is in the system as claimed. The following the steps need be executed:
pvcreate -B /dev/rdsk/c4t6d0
vgextend vg00 /dev/dsk/c4t6d0
mkboot /dev/rdsk/c4t6d0
mkboot -s "hpux lq /stand/vmunix" /dev/rdsk/c4t6d0
lvextend -m 1 /dev/vg00/lvol1 /dev/dsk/c4t6d0
do lvol2, 3, 4,...10
lvlnboot -vR
lvlnboot -v vg00 to make sure it is correct.

In your case, is seems there are 3 disks in vg00. Can you check what size of other 2 disks? seems none of other 2 disks can hold all lvols in vg00.
can you post vgdisplay -v vg00 and lvlnboot -v vg00?
Cheers.

[John Guster]

typo, sorry. mkboot -a not -s as option.

[John Guster]

typo again..hpux -lq, missing "-"
not a good day, isn't it?

[Cedric Bruno]

Hi John, thanks to your reply.

VGDISPLAY VG00
--- Volume groups ---
VG Name /dev/vg00
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 18
Open LV 18
Max PV 16
Cur PV 8
Act PV 8
Max PE per PV 4608
VGDA 16
PE Size (Mbytes) 4
Total PE 8184
Alloc PE 7904
Free PE 280
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg00/lvol1
LV Status available/syncd
LV Size (Mbytes) 96
Current LE 24
Allocated PE 48
Used PV 2

LV Name /dev/vg00/lvol2
LV Status available/syncd
LV Size (Mbytes) 512
Current LE 128
Allocated PE 256
Used PV 2

LV Name /dev/vg00/lvol3
LV Status available/syncd
LV Size (Mbytes) 128
Current LE 32
Allocated PE 64
Used PV 2

LV Name /dev/vg00/lvol4
LV Status available/syncd
LV Size (Mbytes) 256
Current LE 64
Allocated PE 128
Used PV 2

LV Name /dev/vg00/lvol5
LV Status available/syncd
LV Size (Mbytes) 3196
Current LE 799
Allocated PE 1598
Used PV 2

LV Name /dev/vg00/lvol6
LV Status available/syncd
LV Size (Mbytes) 512
Current LE 128
Allocated PE 256
Used PV 2

LV Name /dev/vg00/lvol7
LV Status available/syncd
LV Size (Mbytes) 896
Current LE 224
Allocated PE 448
Used PV 3

LV Name /dev/vg00/lvol8
LV Status available/syncd
LV Size (Mbytes) 1068
Current LE 267
Allocated PE 534
Used PV 3

LV Name /dev/vg00/lvol9
LV Status available/syncd
LV Size (Mbytes) 896
Current LE 224
Allocated PE 224
Used PV 1

LV Name /dev/vg00/lvol10
LV Status available/syncd
LV Size (Mbytes) 256
Current LE 64
Allocated PE 128
Used PV 2

LV Name /dev/vg00/lvol11
LV Status available/syncd
LV Size (Mbytes) 2560
Current LE 640
Allocated PE 1280
Used PV 2

LV Name /dev/vg00/lvol12
LV Status available/syncd
LV Size (Mbytes) 2048
Current LE 512
Allocated PE 512
Used PV 2

LV Name /dev/vg00/lvol13
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 256
Allocated PE 512
Used PV 2

LV Name /dev/vg00/lvol14
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 256
Allocated PE 512
Used PV 2

LV Name /dev/vg00/lvol15
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 256
Allocated PE 512
Used PV 2

LV Name /dev/vg00/lvol16
LV Status available/syncd
LV Size (Mbytes) 764
Current LE 191
Allocated PE 382
Used PV 2

LV Name /dev/vg00/lvol17
LV Status available/syncd
LV Size (Mbytes) 508
Current LE 127
Allocated PE 254
Used PV 2

LV Name /dev/vg00/lvol18
LV Status available/syncd
LV Size (Mbytes) 512
Current LE 128
Allocated PE 256
Used PV 2


--- Physical volumes ---
PV Name /dev/dsk/c4t5d0
PV Status available
Total PE 1023
Free PE 0
Autoswitch On

PV Name /dev/dsk/c4t4d0
PV Status available
Total PE 1023
Free PE 0
Autoswitch On

PV Name /dev/dsk/c7t5d0
PV Status available
Total PE 1023
Free PE 0
Autoswitch On

PV Name /dev/dsk/c7t4d0
PV Status available
Total PE 1023
Free PE 64
Autoswitch On

PV Name /dev/dsk/c7t6d0
PV Status available
Total PE 1023
Free PE 28
Autoswitch On

PV Name /dev/dsk/c7t3d0
PV Status available
Total PE 1023
Free PE 0
Autoswitch On

PV Name /dev/dsk/c4t3d0
PV Status available
Total PE 1023
Free PE 96
Autoswitch On

PV Name /dev/dsk/c4t6d0
PV Status available
Total PE 1023
Free PE 92
Autoswitch On

LVDISPLAY LVOL7
--- Logical volumes ---
LV Name /dev/vg00/lvol7
VG Name /dev/vg00
LV Permission read/write
LV Status available/syncd
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 896
Current LE 224
Allocated PE 448
Stripes 0
Stripe Size (Kbytes) 0
Bad block off
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c4t4d0 64 64
/dev/dsk/c7t6d0 160 160
/dev/dsk/c4t6d0 224 224

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c7t6d0 00488 current /dev/dsk/c4t6d0 00376 current
...............
00223 /dev/dsk/c4t4d0 00063 current /dev/dsk/c4t6d0 00599 current

LVDISPLAY LVOL8
--- Logical volumes ---
LV Name /dev/vg00/lvol8
VG Name /dev/vg00
LV Permission read/write
LV Status available/syncd
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 1068
Current LE 267
Allocated PE 534
Stripes 0
Stripe Size (Kbytes) 0
Bad block off
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c7t6d0 171 171
/dev/dsk/c7t3d0 96 96
/dev/dsk/c4t6d0 267 267

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c7t6d0 00648 current /dev/dsk/c4t6d0 00600 current
....
00266 /dev/dsk/c7t6d0 00994 current /dev/dsk/c4t6d0 00866 current

The three disks have the same physical space.

LVNBOOT VG00
Boot Logical Volume configured
Boot Definitions for Volume Group /dev/vg00:
Physical Volumes belonging in Root Volume Group:
/dev/dsk/c4t5d0 (10/0.5.0)
/dev/dsk/c4t4d0 (10/0.4.0)
/dev/dsk/c7t5d0 (10/8.5.0)
/dev/dsk/c7t4d0 (10/8.4.0)
/dev/dsk/c7t6d0 (10/8.6.0) -- Boot Disk
/dev/dsk/c7t3d0 (10/8.3.0)
/dev/dsk/c4t3d0 (10/0.3.0)
/dev/dsk/c4t6d0 (10/0.6.0) -- Boot Disk
Root: lvol3 on: /dev/dsk/c7t6d0
/dev/dsk/c4t6d0
Swap: lvol2 on: /dev/dsk/c7t6d0
/dev/dsk/c4t6d0
Dump: lvol2 on: /dev/dsk/c7t6d0, 0
Dump: lvol11 on: /dev/dsk/c4t5d0, 1
Dump: lvol13 on: /dev/dsk/c4t4d0, 2
Dump: lvol15 on: /dev/dsk/c4t4d0, 3
Dump: lvol17 on: /dev/dsk/c4t5d0, 4

[John Guster]

lovl6 :
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c7t6d0 00360 current /dev/dsk/c4t6d0 00248 current
....
00127 /dev/dsk/c7t6d0 00487 current /dev/dsk/c4t6d0 00375 current
note c7t6d0 PE360-PE487, but c4t6d0 PE248-PE375. If it is one to one mirror, you will see PE## is the same between 2 disks.do lvdisplay -v /dev/vg00/lvol1...lvol10 to record the PE info to create a complete mapping between PE of each disk to lvol1-lvol10, then you start mirror the new disk. very likely your lvols are not created as the # indicates, that is lvol5 may be created 3rd not 5th. Cheers.

[Cedric Bruno]

Hi again John,

We finally managed to solve this issue.

We had to do some pvmove, lvextend then lvreduce in order to set up the good symetric for this lvol.
It's fine now, the order is restored ;-)

Thanks and have a nice day.

Regards,

Cedric

[Cedric Bruno]

Mirror was recreated.