LVM mirror with PVG Strict Distributed by MC/SG

[Thomas Stuetzer]

Hello,
we have two VA7400 and server and and one Broadcade Switch (test environment). On both server is MC/SG installed. The MC/SG packages has a VG (vgtest) included.
For the primary node (sapr3man) the mirror goes from one PVG VA_RZ over PVG VA_ARZ. So in case of failure on VA7400 , I have no date lost.
When we switch the MC/SG packages to the alternate node (server sandb) the mirror is distributed over the PVG's. So I could lost data in case of failure if one VA7400.

I have no idea what is the cause for this behavior.
Any ideas?


more /etc/lvmpvg
VG /dev/vgtest
PVG VA_RZ
/dev/dsk/c4t1d2
/dev/dsk/c4t1d3
PVG VA_ARZ
/dev/dsk/c6t1d2
/dev/dsk/c6t1d3


server:sapr3man
lvdisplay -v /dev/vgtest/lvb1
Allocation PVG-strict/distributed
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c4t1d2 8 8
/dev/dsk/c4t1d3 7 7
/dev/dsk/c6t1d2 8 8
/dev/dsk/c6t1d3 7 7

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c6t1d2 00008 current /dev/dsk/c4t1d2 00005 current
00001 /dev/dsk/c4t1d3 00010 current /dev/dsk/c6t1d3 00007 current
00002 /dev/dsk/c6t1d2 00009 current /dev/dsk/c4t1d2 00006 current
00003 /dev/dsk/c4t1d3 00011 current /dev/dsk/c6t1d3 00008 current
00004 /dev/dsk/c6t1d2 00010 current /dev/dsk/c4t1d2 00007 current


server: sandb
lvdisplay -v /dev/vgtest/lvb1

Allocation PVG-strict/distributed
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c4t1d2 8 8
/dev/dsk/c4t1d3 7 7
/dev/dsk/c6t1d2 8 8
/dev/dsk/c6t1d3 7 7

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c6t1d2 00008 current /dev/dsk/c4t1d2 00005 current
00001 /dev/dsk/c4t1d3 00010 current /dev/dsk/c6t1d3 00007 current
00002 /dev/dsk/c6t1d2 00009 current /dev/dsk/c4t1d2 00006 current
00003 /dev/dsk/c4t1d3 00011 current /dev/dsk/c6t1d3 00008 current
00004 /dev/dsk/c6t1d2 00010 current /dev/dsk/c4t1d2 00007 current
00005 /dev/dsk/c4t1d3 00012 current /dev/dsk/c6t1d3 00009 current
00006 /dev/dsk/c6t1d2 00011 current /dev/dsk/c4t1d2 00008 current
00007 /dev/dsk/c4t1d3 00013 current /dev/dsk/c6t1d3 00010 current


ioscan
ext_bus 4 0/4/0/0.2.16.39.0 fcparray CLAIMED INTERFACE FCP Array
Interface
target 8 0/4/0/0.2.16.39.0.0 tgt CLAIMED DEVICE
disk 3 0/4/0/0.2.16.39.0.0.0 sdisk CLAIMED DEVICE HP A
6189A
/dev/dsk/c4t0d0 /dev/rdsk/c4t0d0
target 9 0/4/0/0.2.16.39.0.1 tgt CLAIMED DEVICE
disk 8 0/4/0/0.2.16.39.0.1.2 sdisk CLAIMED DEVICE HP A
6189A
/dev/dsk/c4t1d2 /dev/rdsk/c4t1d2
disk 9 0/4/0/0.2.16.39.0.1.3 sdisk CLAIMED DEVICE HP A
6189A
/dev/dsk/c4t1d3 /dev/rdsk/c4t1d3

ext_bus 6 0/4/0/0.2.17.39.0 fcparray CLAIMED INTERFACE FCP Array
Interface
target 12 0/4/0/0.2.17.39.0.0 tgt CLAIMED DEVICE
disk 14 0/4/0/0.2.17.39.0.0.0 sdisk CLAIMED DEVICE HP A
6189A
/dev/dsk/c6t0d0 /dev/rdsk/c6t0d0
target 13 0/4/0/0.2.17.39.0.1 tgt CLAIMED DEVICE
disk 6 0/4/0/0.2.17.39.0.1.2 sdisk CLAIMED DEVICE HP A
6189A
/dev/dsk/c6t1d2 /dev/rdsk/c6t1d2
disk 7 0/4/0/0.2.17.39.0.1.3 sdisk CLAIMED DEVICE HP A
6189A
/dev/dsk/c6t1d3 /dev/rdsk/c6t1d3

[Steven E. Protter]

Shalom,

run this:

lvdisplay -v | more

There are probably stale units on one of the mirror copies.

SEP

[Torsten.]

Thomas,

I cannot see a problem so far.

This looks like "EXTENT-BASED MIRRORED STRIPES" (have a look at "man lvcreate").

http://docs.hp.com/en/B2355-60105/lvcreate.1M.html

Do you experience problems?

You provided a lot of information, but not all.

It looks to me like all c6t1d2 are mirrored to c4t1d2 and c6t1d3 to c4t1d3. This is correct.

[Thomas Stuetzer]

Hello Steven and Torsten,

Sorry - this was the wrong output from server sapr3man.
I try again
So first I reduce the mirror copies by
lvreduce -m 0 /dev/vgtest/lvb1 /dev/dsk/c6t1d3 /dev/dsk/c6t1d2
lvdisplay -v |head -30
--- Logical volumes ---
LV Name /dev/vgtest/lvb1
VG Name /dev/vgtest
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 60
Current LE 15
Allocated PE 15
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation PVG-strict/distributed
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c4t1d2 8 8
/dev/dsk/c4t1d3 7 7


--- Logical extents ---
LE PV1 PE1 Status 1
00000 /dev/dsk/c4t1d2 00005 current
00001 /dev/dsk/c4t1d3 00010 current
00002 /dev/dsk/c4t1d2 00006 current
00003 /dev/dsk/c4t1d3 00011 current
00004 /dev/dsk/c4t1d2 00007 current
00005 /dev/dsk/c4t1d3 00012 current


Create the mirror again
lvextend -m 1 /dev/vgtest/lvb1 VA_RZ VA_ARZ
lvdisplay -v |head -30
--- Logical volumes ---
LV Name /dev/vgtest/lvb1
VG Name /dev/vgtest
LV Permission read/write
LV Status available/syncd
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 60
Current LE 15
Allocated PE 30
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation PVG-strict/distributed
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c4t1d2 8 8
/dev/dsk/c4t1d3 7 7
/dev/dsk/c6t1d2 8 8
/dev/dsk/c6t1d3 7 7

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c4t1d2 00005 current /dev/dsk/c6t1d2 00008 current
00001 /dev/dsk/c4t1d3 00010 current /dev/dsk/c6t1d3 00007 current
00002 /dev/dsk/c4t1d2 00006 current /dev/dsk/c6t1d2 00009 current
00003 /dev/dsk/c4t1d3 00011 current /dev/dsk/c6t1d3 00008 current


So far this looks nice. The original is on PVG VA_RZ and the mirror is on PVG VA_ARZ.
Now I switch the packages to a other node and back (the cluser packages contain die VG vgtest. The VG vgtest is avtivated by vgchange -a e)

after the packages switch the output has been changed.

-- Logical volumes ---
LV Name /dev/vgtest/lvb1
VG Name /dev/vgtest
LV Permission read/write
LV Status available/syncd
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 60
Current LE 15
Allocated PE 30
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation PVG-strict/distributed
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c4t1d2 8 8
/dev/dsk/c4t1d3 7 7
/dev/dsk/c6t1d2 8 8
/dev/dsk/c6t1d3 7 7
--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c6t1d2 00008 current /dev/dsk/c4t1d2 00005 current
00001 /dev/dsk/c4t1d3 00010 current /dev/dsk/c6t1d3 00007 current
00002 /dev/dsk/c6t1d2 00009 current /dev/dsk/c4t1d2 00006 current
00003 /dev/dsk/c4t1d3 00011 current /dev/dsk/c6t1d3 00008 current

That means the mirror is on PVG VA_RZ and VA_ARZ! Why is this changed?
Thomas

[Torsten.]

Did you read about EXTENT-BASED MIRRORED STRIPES?


IMHO, the system does exactly what it is configured for, because the policy is "distributed".

[Deoncia Grayson_1]

when you did your failover, you did a manual failover?

also I would be curious as to how your /etc/lvmpvg file is set up on your secondary node... could you post your output from that server here as well?

[Thomas Stuetzer]

Hello,

I did the switch over manual.
the /etc/lvmpvg is on both node the identical.
sandb:/#more /etc/lvmpvg
VG /dev/vgtest
PVG VA_RZ
/dev/dsk/c4t1d2
/dev/dsk/c4t1d3
PVG VA_ARZ
/dev/dsk/c6t1d2
/dev/dsk/c6t1d3

Torsten,
I read read about EXTENT-BASED MIRRORED STRIPES.
I undestand this so:
with the PVG strict policy (Allocation PVG-strict/distributed) mirrors of LE can not share the same PVG. Is this right?
But this happen after
the VG is deactivated and activated again.
Thomas

[Torsten.]

Not really, but if you see a "c6" device together with another "c6" device on the same line, that would mean the mirrors are on the same device.

In case you see a "c6" and a "c4" device on the same line, mirroring is OK.

The "PVG-strict" setting will ensure the copy is not on the same disk or in the same PVG.

Please provide a "vgdisplay -v vgtest" too.

[Torsten. · Accepted solution]

If you look at this lines:

00000 /dev/dsk/c6t1d2 00008 current /dev/dsk/c4t1d2 00005 current

00001 /dev/dsk/c4t1d3 00010 current /dev/dsk/c6t1d3 00007 current


you will see the first logical extend (00000) has a physical extend on c6t1d2 and a copy on c4t1d2 - this is what we expect.

The second LE has a PE on c4t1d3 and a mirror copy on c6t1d3.

The same for all other lines and extends.

[Deoncia Grayson_1]

Your mirrors is fine and it is not residing on the same disk, if you look at your output more closely, notice on Server A your first line, reads:

00000 /dev/dsk/c4t1d2 00005 current /dev/dsk/c6t1d2 00008 current

so essentially disk c4t1d2 is mirrored to c6t1d2

now on server B it reads:
00000 /dev/dsk/c6t1d2 00008 current /dev/dsk/c4t1d2 00005 current

essentially the same mirroring is done: c6t1d2 is mirrored to c4t1d2, nothing has changed except the display looks different because it list c6t1d2 first instead of c4t1d2...

second line: Server A
00001 /dev/dsk/c4t1d3 00010 current /dev/dsk/c6t1d3 00007 current

second line: Server B
00001 /dev/dsk/c4t1d3 00010 current /dev/dsk/c6t1d3 00007 current

third line: Server A
00002 /dev/dsk/c4t1d2 00006 current /dev/dsk/c6t1d2 00009 current

third line: Server B
00002 /dev/dsk/c6t1d2 00009 current /dev/dsk/c4t1d2 00006 current

fourth line: Server A
00003 /dev/dsk/c4t1d3 00011 current /dev/dsk/c6t1d3 00008 current

fourth line: Server B
00003 /dev/dsk/c4t1d3 00011 current /dev/dsk/c6t1d3 00008 current


Basically in short, your mirroring is fine and it does not reside on the same disk.