LVM mirror with PVG Strict Distributed by MC/SG

[Thomas Stuetzer]

Hello,
we have two VA7400 and server and and one Broadcade Switch (test environment). On both server is MC/SG installed. The MC/SG packages has a VG (vgtest) included.
For the primary node (sapr3man) the mirror goes from one PVG VA_RZ over PVG VA_ARZ. So in case of failure on VA7400 , I have no date lost.
When we switch the MC/SG packages to the alternate node (server sandb) the mirror is distributed over the PVG's. So I could lost data in case of failure if one VA7400.

I have no idea what is the cause for this behavior.
Any ideas?


more /etc/lvmpvg
VG /dev/vgtest
PVG VA_RZ
/dev/dsk/c4t1d2
/dev/dsk/c4t1d3
PVG VA_ARZ
/dev/dsk/c6t1d2
/dev/dsk/c6t1d3


server:sapr3man
lvdisplay -v /dev/vgtest/lvb1
Allocation PVG-strict/distributed
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c4t1d2 8 8
/dev/dsk/c4t1d3 7 7
/dev/dsk/c6t1d2 8 8
/dev/dsk/c6t1d3 7 7

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c6t1d2 00008 current /dev/dsk/c4t1d2 00005 current
00001 /dev/dsk/c4t1d3 00010 current /dev/dsk/c6t1d3 00007 current
00002 /dev/dsk/c6t1d2 00009 current /dev/dsk/c4t1d2 00006 current
00003 /dev/dsk/c4t1d3 00011 current /dev/dsk/c6t1d3 00008 current
00004 /dev/dsk/c6t1d2 00010 current /dev/dsk/c4t1d2 00007 current


server: sandb
lvdisplay -v /dev/vgtest/lvb1

Allocation PVG-strict/distributed
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c4t1d2 8 8
/dev/dsk/c4t1d3 7 7
/dev/dsk/c6t1d2 8 8
/dev/dsk/c6t1d3 7 7

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c6t1d2 00008 current /dev/dsk/c4t1d2 00005 current
00001 /dev/dsk/c4t1d3 00010 current /dev/dsk/c6t1d3 00007 current
00002 /dev/dsk/c6t1d2 00009 current /dev/dsk/c4t1d2 00006 current
00003 /dev/dsk/c4t1d3 00011 current /dev/dsk/c6t1d3 00008 current
00004 /dev/dsk/c6t1d2 00010 current /dev/dsk/c4t1d2 00007 current
00005 /dev/dsk/c4t1d3 00012 current /dev/dsk/c6t1d3 00009 current
00006 /dev/dsk/c6t1d2 00011 current /dev/dsk/c4t1d2 00008 current
00007 /dev/dsk/c4t1d3 00013 current /dev/dsk/c6t1d3 00010 current


ioscan
ext_bus 4 0/4/0/0.2.16.39.0 fcparray CLAIMED INTERFACE FCP Array
Interface
target 8 0/4/0/0.2.16.39.0.0 tgt CLAIMED DEVICE
disk 3 0/4/0/0.2.16.39.0.0.0 sdisk CLAIMED DEVICE HP A
6189A
/dev/dsk/c4t0d0 /dev/rdsk/c4t0d0
target 9 0/4/0/0.2.16.39.0.1 tgt CLAIMED DEVICE
disk 8 0/4/0/0.2.16.39.0.1.2 sdisk CLAIMED DEVICE HP A
6189A
/dev/dsk/c4t1d2 /dev/rdsk/c4t1d2
disk 9 0/4/0/0.2.16.39.0.1.3 sdisk CLAIMED DEVICE HP A
6189A
/dev/dsk/c4t1d3 /dev/rdsk/c4t1d3

ext_bus 6 0/4/0/0.2.17.39.0 fcparray CLAIMED INTERFACE FCP Array
Interface
target 12 0/4/0/0.2.17.39.0.0 tgt CLAIMED DEVICE
disk 14 0/4/0/0.2.17.39.0.0.0 sdisk CLAIMED DEVICE HP A
6189A
/dev/dsk/c6t0d0 /dev/rdsk/c6t0d0
target 13 0/4/0/0.2.17.39.0.1 tgt CLAIMED DEVICE
disk 6 0/4/0/0.2.17.39.0.1.2 sdisk CLAIMED DEVICE HP A
6189A
/dev/dsk/c6t1d2 /dev/rdsk/c6t1d2
disk 7 0/4/0/0.2.17.39.0.1.3 sdisk CLAIMED DEVICE HP A
6189A
/dev/dsk/c6t1d3 /dev/rdsk/c6t1d3

[Steven E. Protter]

Shalom,

run this:

lvdisplay -v | more

There are probably stale units on one of the mirror copies.

SEP

[Torsten.]

Thomas,

I cannot see a problem so far.

This looks like "EXTENT-BASED MIRRORED STRIPES" (have a look at "man lvcreate").

http://docs.hp.com/en/B2355-60105/lvcreate.1M.html

Do you experience problems?

You provided a lot of information, but not all.

It looks to me like all c6t1d2 are mirrored to c4t1d2 and c6t1d3 to c4t1d3. This is correct.

[Thomas Stuetzer]

Hello Steven and Torsten,

Sorry - this was the wrong output from server sapr3man.
I try again
So first I reduce the mirror copies by
lvreduce -m 0 /dev/vgtest/lvb1 /dev/dsk/c6t1d3 /dev/dsk/c6t1d2
lvdisplay -v |head -30
--- Logical volumes ---
LV Name /dev/vgtest/lvb1
VG Name /dev/vgtest
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 60
Current LE 15
Allocated PE 15
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation PVG-strict/distributed
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c4t1d2 8 8
/dev/dsk/c4t1d3 7 7


--- Logical extents ---
LE PV1 PE1 Status 1
00000 /dev/dsk/c4t1d2 00005 current
00001 /dev/dsk/c4t1d3 00010 current
00002 /dev/dsk/c4t1d2 00006 current
00003 /dev/dsk/c4t1d3 00011 current
00004 /dev/dsk/c4t1d2 00007 current
00005 /dev/dsk/c4t1d3 00012 current


Create the mirror again
lvextend -m 1 /dev/vgtest/lvb1 VA_RZ VA_ARZ
lvdisplay -v |head -30
--- Logical volumes ---
LV Name /dev/vgtest/lvb1
VG Name /dev/vgtest
LV Permission read/write
LV Status available/syncd
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 60
Current LE 15
Allocated PE 30
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation PVG-strict/distributed
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c4t1d2 8 8
/dev/dsk/c4t1d3 7 7
/dev/dsk/c6t1d2 8 8
/dev/dsk/c6t1d3 7 7

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c4t1d2 00005 current /dev/dsk/c6t1d2 00008 current
00001 /dev/dsk/c4t1d3 00010 current /dev/dsk/c6t1d3 00007 current
00002 /dev/dsk/c4t1d2 00006 current /dev/dsk/c6t1d2 00009 current
00003 /dev/dsk/c4t1d3 00011 current /dev/dsk/c6t1d3 00008 current


So far this looks nice. The original is on PVG VA_RZ and the mirror is on PVG VA_ARZ.
Now I switch the packages to a other node and back (the cluser packages contain die VG vgtest. The VG vgtest is avtivated by vgchange -a e)

after the packages switch the output has been changed.

-- Logical volumes ---
LV Name /dev/vgtest/lvb1
VG Name /dev/vgtest
LV Permission read/write
LV Status available/syncd
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 60
Current LE 15
Allocated PE 30
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation PVG-strict/distributed
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c4t1d2 8 8
/dev/dsk/c4t1d3 7 7
/dev/dsk/c6t1d2 8 8
/dev/dsk/c6t1d3 7 7
--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c6t1d2 00008 current /dev/dsk/c4t1d2 00005 current
00001 /dev/dsk/c4t1d3 00010 current /dev/dsk/c6t1d3 00007 current
00002 /dev/dsk/c6t1d2 00009 current /dev/dsk/c4t1d2 00006 current
00003 /dev/dsk/c4t1d3 00011 current /dev/dsk/c6t1d3 00008 current

That means the mirror is on PVG VA_RZ and VA_ARZ! Why is this changed?
Thomas

[Torsten.]

Did you read about EXTENT-BASED MIRRORED STRIPES?


IMHO, the system does exactly what it is configured for, because the policy is "distributed".

[Deoncia Grayson_1]

when you did your failover, you did a manual failover?

also I would be curious as to how your /etc/lvmpvg file is set up on your secondary node... could you post your output from that server here as well?

[Thomas Stuetzer]

Hello,

I did the switch over manual.
the /etc/lvmpvg is on both node the identical.
sandb:/#more /etc/lvmpvg
VG /dev/vgtest
PVG VA_RZ
/dev/dsk/c4t1d2
/dev/dsk/c4t1d3
PVG VA_ARZ
/dev/dsk/c6t1d2
/dev/dsk/c6t1d3

Torsten,
I read read about EXTENT-BASED MIRRORED STRIPES.
I undestand this so:
with the PVG strict policy (Allocation PVG-strict/distributed) mirrors of LE can not share the same PVG. Is this right?
But this happen after
the VG is deactivated and activated again.
Thomas

[Torsten.]

Not really, but if you see a "c6" device together with another "c6" device on the same line, that would mean the mirrors are on the same device.

In case you see a "c6" and a "c4" device on the same line, mirroring is OK.

The "PVG-strict" setting will ensure the copy is not on the same disk or in the same PVG.

Please provide a "vgdisplay -v vgtest" too.

[Torsten. · Accepted solution]

If you look at this lines:

00000 /dev/dsk/c6t1d2 00008 current /dev/dsk/c4t1d2 00005 current

00001 /dev/dsk/c4t1d3 00010 current /dev/dsk/c6t1d3 00007 current


you will see the first logical extend (00000) has a physical extend on c6t1d2 and a copy on c4t1d2 - this is what we expect.

The second LE has a PE on c4t1d3 and a mirror copy on c6t1d3.

The same for all other lines and extends.

[Deoncia Grayson_1]

Your mirrors is fine and it is not residing on the same disk, if you look at your output more closely, notice on Server A your first line, reads:

00000 /dev/dsk/c4t1d2 00005 current /dev/dsk/c6t1d2 00008 current

so essentially disk c4t1d2 is mirrored to c6t1d2

now on server B it reads:
00000 /dev/dsk/c6t1d2 00008 current /dev/dsk/c4t1d2 00005 current

essentially the same mirroring is done: c6t1d2 is mirrored to c4t1d2, nothing has changed except the display looks different because it list c6t1d2 first instead of c4t1d2...

second line: Server A
00001 /dev/dsk/c4t1d3 00010 current /dev/dsk/c6t1d3 00007 current

second line: Server B
00001 /dev/dsk/c4t1d3 00010 current /dev/dsk/c6t1d3 00007 current

third line: Server A
00002 /dev/dsk/c4t1d2 00006 current /dev/dsk/c6t1d2 00009 current

third line: Server B
00002 /dev/dsk/c6t1d2 00009 current /dev/dsk/c4t1d2 00006 current

fourth line: Server A
00003 /dev/dsk/c4t1d3 00011 current /dev/dsk/c6t1d3 00008 current

fourth line: Server B
00003 /dev/dsk/c4t1d3 00011 current /dev/dsk/c6t1d3 00008 current


Basically in short, your mirroring is fine and it does not reside on the same disk.

[Thomas Stuetzer]

Hello,

thank you for the explanation. Now I have understood the lvm mirror is ok( I should read the mirror in line not in column!).
My aim is to set up a mirror from from storage system A to storage system B. The hardware pfad from device c4 in PVG VA_RZ match with storage system A and and hardware pfad from device c6 in PVG VA_ARZ match with storage system B.
So want to setup a mirror like this:
--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c4t1d2 00005 current /dev/dsk/c6t1d2 00008 current
00001 /dev/dsk/c4t1d3 00010 current /dev/dsk/c6t1d3 00007 current
00002 /dev/dsk/c4t1d2 00006 current /dev/dsk/c6t1d2 00009 current
00003 /dev/dsk/c4t1d3 00011 current /dev/dsk/c6t1d3 00008 current

This is done with
lvreduce -m 0 /dev/vgtest/lvb1 /dev/dsk/c6t1d3 /dev/dsk/c6t1d2
lvextend -m 1 /dev/vgtest/lvb1 VA_RZ VA_ARZ

After I deactivate the volume group and activate the volume group the picture is like this:
--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c6t1d2 00008 current /dev/dsk/c4t1d2 00005 current
00001 /dev/dsk/c4t1d3 00010 current /dev/dsk/c6t1d3 00007 current
00002 /dev/dsk/c6t1d2 00009 current /dev/dsk/c4t1d2 00006 current
What is the reason for this change?

Here are the volume group
--- Volume groups ---
VG Name /dev/vgtest
VG Write Access read/write
VG Status available, exclusive
Max LV 255
Cur LV 3
Open LV 3
Max PV 16
Cur PV 4
Act PV 4
Max PE per PV 1016
VGDA 8
PE Size (Mbytes) 4
Total PE 508
Alloc PE 60
Free PE 448
Total PVG 2
Total Spare PVs 0
Total Spare PVs in use 0
--- Logical volumes ---
LV Name /dev/vgtest/lvd1
LV Status available/syncd
LV Size (Mbytes) 40
Current LE 10
Allocated PE 20
Used PV 4

LV Name /dev/vgtest/lva1
LV Status available/syncd
LV Size (Mbytes) 20
Current LE 5
Allocated PE 10
Used PV 4
LV Name /dev/vgtest/lvb1
LV Status available/syncd
LV Size (Mbytes) 60
Current LE 15
Allocated PE 30
Used PV 4


--- Physical volumes ---
PV Name /dev/dsk/c4t1d2
PV Status available
Total PE 127
Free PE 111
Autoswitch On

PV Name /dev/dsk/c4t1d3
PV Status available
Total PE 127
Free PE 113
Autoswitch On
PV Name /dev/dsk/c6t1d2
PV Status available
Total PE 127
Free PE 111
Autoswitch On

PV Name /dev/dsk/c6t1d3
PV Status available
Total PE 127
Free PE 113
Autoswitch On


--- Physical volume groups ---
PVG Name VA_RZ
PV Name /dev/dsk/c4t1d2
PV Name /dev/dsk/c4t1d3

PVG Name VA_ARZ
PV Name /dev/dsk/c6t1d2
PV Name /dev/dsk/c6t1d3

[Patrice Le Guyader]

Hello Thomas,

Did you still have the order of disks in vgcreate command when you've made you vg ?

Can you post an extract of ioscan with disks?

kenavo
Pat

[Torsten.]

The reason could be your command

lvextend -m 1 /dev/vgtest/lvb1 VA_RZ VA_ARZ

instead of only

lvextend -m 1 /dev/vgtest/lvb1

Perhaps you can test this.

Anyway, the result looks good as it is and will give high performance.

[Thomas Stuetzer]

Hello Pat,
no, I haven't the order of disk for the vgcreate command.
the ioscan show this:

sapr3man:/etc/cmcluster/pkgtest1#ioscan -fnkH0/4/0/0.2.16.39.0.1
Class I H/W Path Driver S/W State H/W Type Description
=========================================================================
target 9 0/4/0/0.2.16.39.0.1 tgt CLAIMED DEVICE
disk 8 0/4/0/0.2.16.39.0.1.2 sdisk CLAIMED DEVICE HP A6189A
/dev/dsk/c4t1d2 /dev/rdsk/c4t1d2
disk 9 0/4/0/0.2.16.39.0.1.3 sdisk CLAIMED DEVICE HP A6189A
/dev/dsk/c4t1d3 /dev/rdsk/c4t1d3
sapr3man:/etc/cmcluster/pkgtest1#ioscan -fnkH0/4/0/0.2.17.39.0.1
Class I H/W Path Driver S/W State H/W Type Description
=========================================================================
target 13 0/4/0/0.2.17.39.0.1 tgt CLAIMED DEVICE
disk 6 0/4/0/0.2.17.39.0.1.2 sdisk CLAIMED DEVICE HP A6189A
/dev/dsk/c6t1d2 /dev/rdsk/c6t1d2
disk 7 0/4/0/0.2.17.39.0.1.3 sdisk CLAIMED DEVICE HP A6189A
/dev/dsk/c6t1d3 /dev/rdsk/c6t1d3

[Thomas Stuetzer]

The mirror is ok. It was may fault to compare the left side with the right side of the output from lvdisplay -v section
--- Logical extents ---

the command lvextend -m 1 /dev/vgtest/lvb1
give the same result as before. The output of lvdisplay is changed after deactivated and activated the VG again.