LVM and VxVM

LVM mirror with PVG Strict Distributed by MC/SG

 
SOLVED
Go to solution
Thomas Stuetzer
Advisor

LVM mirror with PVG Strict Distributed by MC/SG

Hello,
we have two VA7400 and server and and one Broadcade Switch (test environment). On both server is MC/SG installed. The MC/SG packages has a VG (vgtest) included.
For the primary node (sapr3man) the mirror goes from one PVG VA_RZ over PVG VA_ARZ. So in case of failure on VA7400 , I have no date lost.
When we switch the MC/SG packages to the alternate node (server sandb) the mirror is distributed over the PVG's. So I could lost data in case of failure if one VA7400.

I have no idea what is the cause for this behavior.
Any ideas?


more /etc/lvmpvg
VG /dev/vgtest
PVG VA_RZ
/dev/dsk/c4t1d2
/dev/dsk/c4t1d3
PVG VA_ARZ
/dev/dsk/c6t1d2
/dev/dsk/c6t1d3


server:sapr3man
lvdisplay -v /dev/vgtest/lvb1
Allocation PVG-strict/distributed
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c4t1d2 8 8
/dev/dsk/c4t1d3 7 7
/dev/dsk/c6t1d2 8 8
/dev/dsk/c6t1d3 7 7

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c6t1d2 00008 current /dev/dsk/c4t1d2 00005 current
00001 /dev/dsk/c4t1d3 00010 current /dev/dsk/c6t1d3 00007 current
00002 /dev/dsk/c6t1d2 00009 current /dev/dsk/c4t1d2 00006 current
00003 /dev/dsk/c4t1d3 00011 current /dev/dsk/c6t1d3 00008 current
00004 /dev/dsk/c6t1d2 00010 current /dev/dsk/c4t1d2 00007 current


server: sandb
lvdisplay -v /dev/vgtest/lvb1

Allocation PVG-strict/distributed
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c4t1d2 8 8
/dev/dsk/c4t1d3 7 7
/dev/dsk/c6t1d2 8 8
/dev/dsk/c6t1d3 7 7

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c6t1d2 00008 current /dev/dsk/c4t1d2 00005 current
00001 /dev/dsk/c4t1d3 00010 current /dev/dsk/c6t1d3 00007 current
00002 /dev/dsk/c6t1d2 00009 current /dev/dsk/c4t1d2 00006 current
00003 /dev/dsk/c4t1d3 00011 current /dev/dsk/c6t1d3 00008 current
00004 /dev/dsk/c6t1d2 00010 current /dev/dsk/c4t1d2 00007 current
00005 /dev/dsk/c4t1d3 00012 current /dev/dsk/c6t1d3 00009 current
00006 /dev/dsk/c6t1d2 00011 current /dev/dsk/c4t1d2 00008 current
00007 /dev/dsk/c4t1d3 00013 current /dev/dsk/c6t1d3 00010 current


ioscan
ext_bus 4 0/4/0/0.2.16.39.0 fcparray CLAIMED INTERFACE FCP Array
Interface
target 8 0/4/0/0.2.16.39.0.0 tgt CLAIMED DEVICE
disk 3 0/4/0/0.2.16.39.0.0.0 sdisk CLAIMED DEVICE HP A
6189A
/dev/dsk/c4t0d0 /dev/rdsk/c4t0d0
target 9 0/4/0/0.2.16.39.0.1 tgt CLAIMED DEVICE
disk 8 0/4/0/0.2.16.39.0.1.2 sdisk CLAIMED DEVICE HP A
6189A
/dev/dsk/c4t1d2 /dev/rdsk/c4t1d2
disk 9 0/4/0/0.2.16.39.0.1.3 sdisk CLAIMED DEVICE HP A
6189A
/dev/dsk/c4t1d3 /dev/rdsk/c4t1d3

ext_bus 6 0/4/0/0.2.17.39.0 fcparray CLAIMED INTERFACE FCP Array
Interface
target 12 0/4/0/0.2.17.39.0.0 tgt CLAIMED DEVICE
disk 14 0/4/0/0.2.17.39.0.0.0 sdisk CLAIMED DEVICE HP A
6189A
/dev/dsk/c6t0d0 /dev/rdsk/c6t0d0
target 13 0/4/0/0.2.17.39.0.1 tgt CLAIMED DEVICE
disk 6 0/4/0/0.2.17.39.0.1.2 sdisk CLAIMED DEVICE HP A
6189A
/dev/dsk/c6t1d2 /dev/rdsk/c6t1d2
disk 7 0/4/0/0.2.17.39.0.1.3 sdisk CLAIMED DEVICE HP A
6189A
/dev/dsk/c6t1d3 /dev/rdsk/c6t1d3
14 REPLIES 14
Steven E. Protter
Exalted Contributor

Re: LVM mirror with PVG Strict Distributed by MC/SG

Shalom,

run this:

lvdisplay -v | more

There are probably stale units on one of the mirror copies.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Torsten.
Acclaimed Contributor

Re: LVM mirror with PVG Strict Distributed by MC/SG

Thomas,

I cannot see a problem so far.

This looks like "EXTENT-BASED MIRRORED STRIPES" (have a look at "man lvcreate").

http://docs.hp.com/en/B2355-60105/lvcreate.1M.html

Do you experience problems?

You provided a lot of information, but not all.

It looks to me like all c6t1d2 are mirrored to c4t1d2 and c6t1d3 to c4t1d3. This is correct.

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Thomas Stuetzer
Advisor

Re: LVM mirror with PVG Strict Distributed by MC/SG

Hello Steven and Torsten,

Sorry - this was the wrong output from server sapr3man.
I try again
So first I reduce the mirror copies by
lvreduce -m 0 /dev/vgtest/lvb1 /dev/dsk/c6t1d3 /dev/dsk/c6t1d2
lvdisplay -v |head -30
--- Logical volumes ---
LV Name /dev/vgtest/lvb1
VG Name /dev/vgtest
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 60
Current LE 15
Allocated PE 15
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation PVG-strict/distributed
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c4t1d2 8 8
/dev/dsk/c4t1d3 7 7


--- Logical extents ---
LE PV1 PE1 Status 1
00000 /dev/dsk/c4t1d2 00005 current
00001 /dev/dsk/c4t1d3 00010 current
00002 /dev/dsk/c4t1d2 00006 current
00003 /dev/dsk/c4t1d3 00011 current
00004 /dev/dsk/c4t1d2 00007 current
00005 /dev/dsk/c4t1d3 00012 current


Create the mirror again
lvextend -m 1 /dev/vgtest/lvb1 VA_RZ VA_ARZ
lvdisplay -v |head -30
--- Logical volumes ---
LV Name /dev/vgtest/lvb1
VG Name /dev/vgtest
LV Permission read/write
LV Status available/syncd
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 60
Current LE 15
Allocated PE 30
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation PVG-strict/distributed
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c4t1d2 8 8
/dev/dsk/c4t1d3 7 7
/dev/dsk/c6t1d2 8 8
/dev/dsk/c6t1d3 7 7

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c4t1d2 00005 current /dev/dsk/c6t1d2 00008 current
00001 /dev/dsk/c4t1d3 00010 current /dev/dsk/c6t1d3 00007 current
00002 /dev/dsk/c4t1d2 00006 current /dev/dsk/c6t1d2 00009 current
00003 /dev/dsk/c4t1d3 00011 current /dev/dsk/c6t1d3 00008 current


So far this looks nice. The original is on PVG VA_RZ and the mirror is on PVG VA_ARZ.
Now I switch the packages to a other node and back (the cluser packages contain die VG vgtest. The VG vgtest is avtivated by vgchange -a e)

after the packages switch the output has been changed.

-- Logical volumes ---
LV Name /dev/vgtest/lvb1
VG Name /dev/vgtest
LV Permission read/write
LV Status available/syncd
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 60
Current LE 15
Allocated PE 30
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation PVG-strict/distributed
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c4t1d2 8 8
/dev/dsk/c4t1d3 7 7
/dev/dsk/c6t1d2 8 8
/dev/dsk/c6t1d3 7 7
--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c6t1d2 00008 current /dev/dsk/c4t1d2 00005 current
00001 /dev/dsk/c4t1d3 00010 current /dev/dsk/c6t1d3 00007 current
00002 /dev/dsk/c6t1d2 00009 current /dev/dsk/c4t1d2 00006 current
00003 /dev/dsk/c4t1d3 00011 current /dev/dsk/c6t1d3 00008 current

That means the mirror is on PVG VA_RZ and VA_ARZ! Why is this changed?
Thomas
Torsten.
Acclaimed Contributor

Re: LVM mirror with PVG Strict Distributed by MC/SG

Did you read about EXTENT-BASED MIRRORED STRIPES?


IMHO, the system does exactly what it is configured for, because the policy is "distributed".

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Deoncia Grayson_1
Honored Contributor

Re: LVM mirror with PVG Strict Distributed by MC/SG

when you did your failover, you did a manual failover?

also I would be curious as to how your /etc/lvmpvg file is set up on your secondary node... could you post your output from that server here as well?
If no one ever took risks, Michelangelo would have painted the Sistine floor. -Neil Simon
Thomas Stuetzer
Advisor

Re: LVM mirror with PVG Strict Distributed by MC/SG

Hello,

I did the switch over manual.
the /etc/lvmpvg is on both node the identical.
sandb:/#more /etc/lvmpvg
VG /dev/vgtest
PVG VA_RZ
/dev/dsk/c4t1d2
/dev/dsk/c4t1d3
PVG VA_ARZ
/dev/dsk/c6t1d2
/dev/dsk/c6t1d3

Torsten,
I read read about EXTENT-BASED MIRRORED STRIPES.
I undestand this so:
with the PVG strict policy (Allocation PVG-strict/distributed) mirrors of LE can not share the same PVG. Is this right?
But this happen after
the VG is deactivated and activated again.
Thomas

Torsten.
Acclaimed Contributor

Re: LVM mirror with PVG Strict Distributed by MC/SG

Not really, but if you see a "c6" device together with another "c6" device on the same line, that would mean the mirrors are on the same device.

In case you see a "c6" and a "c4" device on the same line, mirroring is OK.

The "PVG-strict" setting will ensure the copy is not on the same disk or in the same PVG.

Please provide a "vgdisplay -v vgtest" too.

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Torsten.
Acclaimed Contributor
Solution

Re: LVM mirror with PVG Strict Distributed by MC/SG

If you look at this lines:

00000 /dev/dsk/c6t1d2 00008 current /dev/dsk/c4t1d2 00005 current

00001 /dev/dsk/c4t1d3 00010 current /dev/dsk/c6t1d3 00007 current


you will see the first logical extend (00000) has a physical extend on c6t1d2 and a copy on c4t1d2 - this is what we expect.

The second LE has a PE on c4t1d3 and a mirror copy on c6t1d3.

The same for all other lines and extends.

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Deoncia Grayson_1
Honored Contributor

Re: LVM mirror with PVG Strict Distributed by MC/SG

Your mirrors is fine and it is not residing on the same disk, if you look at your output more closely, notice on Server A your first line, reads:

00000 /dev/dsk/c4t1d2 00005 current /dev/dsk/c6t1d2 00008 current

so essentially disk c4t1d2 is mirrored to c6t1d2

now on server B it reads:
00000 /dev/dsk/c6t1d2 00008 current /dev/dsk/c4t1d2 00005 current

essentially the same mirroring is done: c6t1d2 is mirrored to c4t1d2, nothing has changed except the display looks different because it list c6t1d2 first instead of c4t1d2...

second line: Server A
00001 /dev/dsk/c4t1d3 00010 current /dev/dsk/c6t1d3 00007 current

second line: Server B
00001 /dev/dsk/c4t1d3 00010 current /dev/dsk/c6t1d3 00007 current

third line: Server A
00002 /dev/dsk/c4t1d2 00006 current /dev/dsk/c6t1d2 00009 current

third line: Server B
00002 /dev/dsk/c6t1d2 00009 current /dev/dsk/c4t1d2 00006 current

fourth line: Server A
00003 /dev/dsk/c4t1d3 00011 current /dev/dsk/c6t1d3 00008 current

fourth line: Server B
00003 /dev/dsk/c4t1d3 00011 current /dev/dsk/c6t1d3 00008 current


Basically in short, your mirroring is fine and it does not reside on the same disk.

If no one ever took risks, Michelangelo would have painted the Sistine floor. -Neil Simon