Mirroring an entire volume group

[David Owens_1]

I have a volume group (vg02) consisting of 4 disks on a single SCSI channel. There are only 2 logical volumes in the volume group.

I have 4 available disks on a seperate SCSI channel that I would like to use as a mirror copy of vg02.

Can I just create another VG and mirror that way or do I need to create PVGs for both sets of disks in the same VG?

Also, the HP's on-line docs recommend using SAM whenever possible to create mirrors. Is this even possible with SAM?

Yes, I do have Mirror Disk/UX installed and running.


Thanks,
David Owens
Ingersoll-Rand

[Michael Steele_2 · Accepted solution]

When you mirror its always from within the save volume group. In this case only 'vg02' can mirror by logical volume, from one disk to another.

Mirroring from one SCSI bus to another, (* c1 to c2 *) and from one target on the bus to another, (* c1t3 to c2t3 *) is important because of SCSI bus priorities.

For example mirroring from one disk to another on the same SCSI bus would not only create a single point of failure at the controller but also a I/O performance problem since two different targets with two different bus priorities would be in use, c1t3 to c1t4 for example is not the way to go, nor is c1t3 to c2t4. The targets t3 and t4 have different priorities.

So vgextend in all of those other disks to vg02 and then use lvextend. Use 'pvdisplay' for this.

vgextend /dev/vg## /dev/dsk/cXtYdZ

pvdisplay -v /dev/dsk/cXtYdZ
| more (* review lvols residing on pv *)

lvextend -m 1 /dev/vg##/lvol# /dev/dsk/cXtYdZ

[Eugeny Brychkov]

David,
MirrorDisk/UX software allows mirroring of logical volumes within volume group ONLY. Thus you need to include these spare disks into VG (vgextend) and then using command line ('lvextend -m' option, do not use SAM) mirror existing or new logical volumes between disks within volume group
Eugeny

[curt larson_1]

Can I just create another VG and mirror that way ?

no, you'll need to add all 8 drives to the same vg.

do I need to create PVGs for both sets of disks in the same VG?

no, you don't need to create pvgs. Using pvgs can make some administration tasks easier to manage but reduces your flexibility. With only 8 discs and 2 logical volumes, I don't think it'll matter if you use them or not.

but if you do stripe your lvs across the 4 disks on one controller. I'd put the set of disk on one controller in one pvg and the set of disks from the other controller in a different pvg.

Also, the HP's on-line docs recommend using SAM whenever possible to create mirrors. Is this even possible with SAM?

I don't know, I've never used SAM to create mirrors.

[S.K. Chan]

If I were you I would set up PVG for both set of disks because I do not want to manually track/or set how the LVs are mirrorred. If I set up PVG and turn mirroring as PVG-strict than I can rest my mind knowing that my mirrored LVs always resides on 2 separate PVGs (2 separate SCSI channels). This is how you can do it ..
Existing 4 disks in vg01 (say A,B,C,D) on SCSI channel A. The other 4 disks (A2,B2,C2,D2) on SCSI channel B. Now create /etc/lvmpvg ... (see man pages for lvmpvg if not sure of the format of that file). I think it should look like this ..
VG /dev/vg01
PVG pvg1


....
PVG pvg2


....
Before that you would include A2->D2 PVs into you vg01 by vgextending it.
# pvcreate
# vgextend /dev/vg01
and so on ...
Once everything are in place, run vgdisplay to make sure the PVGs are setup. Next thing to do is for you existing LVs (say lvol1 and lvol2) you got to change its behavior to PVG-strict-allocation.
# lvchange -s g /dev/vg01/lvol1
==> repeat for lvol2
Now extend the mirror copy..
# lvextend -m 1 /dev/vg01/lvol1
==> repeat for lvol2
an by doing that the mirrored copies will be placed in pvg2 automatically. Check it by running "pvdisplay" on the disk.

Believe me .. you do not want to use SAM for LVM .. :)

[Steven E. Protter]

Mirroring is one logical volume within a volume group at a time.

SAM is useful for getting the commands, if you don't know them.

In the end, its quite cumbersome if you have lots of logical volumes to mirror.

lvextend -m 1 /dev/vg02/lvname /dev/dsk/c1t6d0 /dev/dsk/c-t6do

Thats all you need to do.

I do recommend you have a plan before you start, or you will learn like I did, by building and breaking a ton of mirrors before you're happy with what you get.

Note: These guys above me deserve some points, they gave great detailed answers. My advice is hook them up!

Good Luck.

SEP

[twang]

The general steps for creating mirror boot disk:

/dev/dsk/cxtyd0 will be the disk you want to
create a mirror copy for the existing root disk inside the
same vg, vg00
if cxtyd0 already belongs to vg00, pls do a
vgreduce to free this disk first

# vgreduce /dev/vg00 /dev/dsk/cxtyd0

1) pvcreate -B /dev/rdsk/cxtyd0
2) mkboot -l /dev/rdsk/cxtyd0
3) mkboot -a "hpux -lq (;0)/stand/vmunix" /dev/rdsk/cxtyd0
4) vgextend /dev/vg00 /dev/dsk/cxtyd0
5) lvextend -m 1 /dev/vg00/lvol1 /dev/dsk/cxtyd0
lvextend -m 1 /dev/vg00/lvol2 /dev/dsk/cxtyd0
lvextend -m 1 .... lvol3 ..
...
...
6) pls check for if:
lvol1 on /stand (this is boot lv)
lvol3 on / (this is root lv)
lvol2 is swap lv

7) lvlnboot -r /dev/vg00/lvol3 (root on /)
8) lvlnboot -s /dev/vg00/lvol2 (swap)
9) lvlnboot -R /dev/vg00
10)lvlnboot -v (to confirm all correct)
11) vgcfgbackup /dev/vg00

[Sunil Sharma_1]

Hi,

In HPUX LVM you can not mirror volume group, unit of mirroring is Logical volume so the stapes you have to follow is following.

1. pvcreate /dev/rdsk/cxtydz (all four disks)
2. vgextend /dev/vg02 /dev/dsk/cxtydz /dev/dsk/cx... (all 4 disks)

3. lvextend -m 1 /dev/vg02/lvol1
4. lvexpend -m 1 /dev/vg02/lvol2

I assume your logical volume name is lvol1 and lvol2.

Sunil

[David Owens_1]

Thanks everyine for your help. I really appreciate your help. I still have few question before moving forward.

1. I my lvmtab output for this VG looks like:

/dev/vg02
/dev/dsk/c2t0d0
/dev/dsk/c2t1d0
/dev/dsk/c2t2d0
/dev/dsk/c2t3d0

Since all disks are 9 GB and one of my logical volumes is 26 GB, am I correct in assumming that my disks are striped? If so, shouldn't I create a PVG? I don't see any other way to mirror c2t0d0 to c1t0d0 and so forth.


Thanks,
David

[S.K. Chan]

Check your lvols to see if they are stripe or not ..(example ..)
# lvdisplay /dev/vg02/lvol2|grep -i stripe
If you see ..
Stripes 0
Stripe Size (Kbytes) 0
It means the lvol is not stripe and if that's the case the 26GB LV you're talking about is distributed to probably 3 of your PVs. That's not stripes. You can proceed with the PVG creation I mentioned above.

[Helen French]

For the last question:

From your /etc/lvmttab output it is NOT necessary that the disks are striped. Even without striping you can have a 26GB LV in vg02 which might be spanned across 3 (or even 4) disks. If you want to get more information on that lv:

# lvdisplay -v /dev/vg02/lvol_name

If you suspect H/W stripe, then check the RAID status from SAM or the utility.

In either case, I would suggest you to use PVGs for mirroring the LVs in this case. You can create 2 different PVGs, one with the original and the other with new disks.

[James R. Ferguson]

Hi David:

An 'lvdisplay' (e.g. lvdisplay /dev/vgNN/lvolX) will reveal whether you are truly striped or using distributed allocation.

If the 'lvdisplay' returnes zero (0) for the "stripes" field, then true striping is being used and you *cannot* mirror.

The 'lvdisplay' "allocation" field will show various values. You need to have "PVG-strict" allocation. You can 'lvchange' to "PVG-strict" if not otherwise set.

See the aan pages for 'lvcreate', 'lvdisplay' and 'lvchange' for more information.

Regards!

...JRF...

[David Owens_1]

Okay, my LVs are not striped but it seems that for multiple disks PVGs are the way to go. Anyway, I still going to assume that all VGs do not need to be included in the /etc/lvmpvg file. I actually have 3 VGs but only want to mirror one for now. Listed below are the steps I intend to take. Someone please tell me if I am missing something.

1. Extend vg02 to include the 4 disk from another SCSI channel.
# pvcreate -f /dev/rdsk/c1t0d0
# pvcreate -f /dev/rdsk/c1t1d0
# pvcreate -f /dev/rdsk/c1t2d0
# pvcreate -f /dev/rdsk/c1t3d0
# vgextend /dev/vg02 /dev/dsk/c1t0d0
# vgextend /dev/vg02 /dev/dsk/c1t1d0
# vgextend /dev/vg02 /dev/dsk/c1t2d0
# vgextend /dev/vg02 /dev/dsk/c1t3d0

2. Create a new ASCII text file for /etc/lvmpvg.
VG /dev/vg02
PVG PVG0
/dev/dsk/c2t0d0
/dev/dsk/c2t1d0
/dev/dsk/c2t2d0
/dev/dsk/c2t3d0
PVG PVG1
/dev/dsk/c1t0d0
/dev/dsk/c1t1d0
/dev/dsk/c1t2d0
/dev/dsk/c1t3d0

3. Run
# vgdisplay -v /dev/vg02
to insure PVGs are correct.

4. Set logical volumes to strict.
# lvchange -s g /dev/vg02/iman_vol1
# lvchange -s g /dev/vg02/iman_vol2

5. Mirror the logical volumes.
# lvextend -m 1 /dev/vg02/iman_vol1
# lvextend -m 1 /dev/vg02/iman_vol2

6. Check physical volumes.
# pvdisplay -v /dev/dsk/c1t0d0
# pvdisplay -v /dev/dsk/c1t1d0
# pvdisplay -v /dev/dsk/c1t2d0
# pvdisplay -v /dev/dsk/c1t3d0
# pvdisplay -v /dev/dsk/c2t0d0
# pvdisplay -v /dev/dsk/c2t1d0
# pvdisplay -v /dev/dsk/c2t2d0
# pvdisplay -v /dev/dsk/c2t3d0


Thanks,
David

[S.K. Chan]

Looks good .. add a few more checks along the ways just to be sure ..
==> After step4
# lvdisplay /dev/vg02/iman_vol1|grep -i allocation
and you should see "pvg-strict" is set. Repeat for "iman_vol2".
==> After step5
# lvdisplay -v /dev/vg02/iman_vol1|more
and make sure the extents are now mirrored to the set of disks in PVG1.

[Michael Steele_2]

First, that you have 26 GB in a logical volume and only 9 GB disks is not an indication of PVG's. It only indicates that one 9gb disk filled to capacity and the 'lvcreate' operation continued onto other 9gb disks until complete.

PVG's:

pvcreate -f /dev/rdsk/cXtYdZ
vgcreate -g pvg0 /dev/vg## /dev/dsk/cXtYdZ

One disk currently resides in pvg0. To add a 2nd disk:

vgextend -g pvg0 /dev/vg01 /dev/dsk/cXtYdZ

Or, edit /etc/lvmpvg:

vi /etc/lvmpvg
vg01
pvg0
c1t9d0
c3t8d0
c4t7d0

Note: The advantages gained with striping is via increased I/O performance. Note the above example has 3 different controllers c1, c3 and c4. For you to use one controller c1t9d0, c1t8do, d4t7d0, would probably create an I/O bottleneck. Use 'sar -d 5 5' and observe 'avwait', 'avserv' and '%busy' to detect this.

Finally, creating logical volumes. Use 32kb or 64kb stripe size and ignore adding the disk during the 'lvcreate' operation.

lvcreate -D y -I 64 -s g -r n /dev/vg##

One final comment, I am recommending stripe and mirror because if you don't mirror, I repeat, if you don't mirror and a disk crashes in this configuration then you restore from tape.

[James R. Ferguson]

Hi Michael:

The 'lvcreate' you suggest is illegal.

# lvcreate -D y -I 64 -s g -r n /dev/vg##

...specifies *distributed* (-D y) allocation which requires PVG-strict (-s g) which is fine. *However* you added true striping (-I 64) which requires the addition of '-i' *but* which is *disallowed* with distributed allocaton.

The distributed allocation policy is incompatible with the striped scheduling policy ( -i stripes ). When doing striped scheduling, '-i' and '-I' must be specified together.

I would also not turn off LVM block relocation ('-r n') unless this logical volume where housed in a disk array like an EMC one.

Regards!

...JRF...

[Michael Steele_2]

Oops, -I 1024.

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x6b8019434a69d711abdc0090277a778c,00.html

[Helen French]

I would also check the lvdisplay output at Step #6:

# lvdisplay -v /dev/vg02/lvol_name

You need to make sure that the mirroring is finished and the extents are not in "stale" status. I would also update my recovery tape with new LVM settings.

[David Owens_1]

Thanks everyone for your help. I mirrored these LVs and seems to be working great. Now that I have it up and running, I have a few more questions.

1. When running lvdisplay -v on each logical volume, I noticed that by the third disk the PEs for both PVGs were not matching exactly. They were one off. Does that really matter?

2. How will I know when a disk fails?

3. What is the procedure for replacing a bad disk and re-synching?

4. What happens if a SCSI channel fails?


Thanks,
David

[Michael Steele_2]

1. - Probably the LVM header with is one PE.

2. - dd if=/dev/dsk/cXtYdZ of=/dev/null count=1000000

(* if dd commands hangs then bad disk *)

3. - replace disk

- vgcfgrestore -n /dev/vg## /dev/rdsk/cXtYdZ

- vgchange -a y /dev/vg##

- vgsync /dev/vg##

4. - I assume you mean the SCSI controller. If bad then it will show up in ioscan as 'NO_HW'.

ioscan -fnkC ext_bus

My experience with these failing SCSI controllers is they display intermittent errors not a complete failure of the device. Also use the 'sea / search' command from BCH since SCSI device will report in at the firmware/BCH level. If they don't report in then its bad.

[S.K. Chan]

1) Not sure what you meant but as long as the extents count and the "mirrored" extents are on ANY disk in PVG1 then you're fine. Perhaps you can provide the lvdisplay output.

2) You'll get all kinds of SCSI error messages in /var/adm/syslog/syslog.log, in /etc/dmesg output and perhaps EMS. Depending on what monitoring script you have in place, just make sure you monitor disk failure. We have an LVM monitoring script actively monitors changes in LVM configuration so we will know when a PV is "unavailable" (ie possible disk failure). The LV that resides on the failed disk most likely will show "stale" extents.

3) Assuming /etc/lvmconf/vg02.conf is good. You would first shutdown and replace the failed disk. Make sure the scsi id remains the same on the new disk. Then boot the system up in single user mode ..
ISL> hpux -is (;0)/stand/vmunix
# pvcreate -f /dev/rdsk/cXtXd0
# vgcfgrestore -n /dev/vg02 /dev/rdsk/cXtXd0
# vgchange -a y /dev/vg02
# vgsync /dev/vg02
# shutdown -r 0

4) Same as 2) I would think. This will definately be detected by EMS if you have it running at the first place and root will get the error messages from EMS. All the PVs in PVG1 will be "unavailable" (from the vgdisplay -v output). All LVs will show "stale".

[David Owens_1]

Here is output from lvdisplay showing the PEs are not matching. Look near bottom of list. Maybe I am too paranoid and this is not a problem.

Also, running a daily script to search for stale connections is simple enough. I was just checking to see if mirror disk/ux or HP-UX displayed some type of notification.


Thanks,
David

[S.K. Chan]

Mmm .. I did not see the whole list .. the last 3 lines show ..
3477 /dev/dsk/c2t1d0 1307 current /dev/dsk/c1t1d0 1307 current
3478 /dev/dsk/c2t1d0 1308 current /dev/dsk/c1t1d0 1308 current
3479 /dev/dsk/c2t1d0 1309 current /dev/dsk/c1t1d0 1309 current
In the distribution of LVs section the total PE/LE for c2t0d0+c2t1d0+c2t2d0 and c1t0d0+c1t1d0+c1t2d0 matches. So at least this is consistent. Can you re-post you output but only show the "output" that you're talking about.

[David Owens_1]

It looks something like this and is consistent for the remainder of the listing.

4337 /dev/dsk/c2t1d0 2167 current /dev/dsk/c1t1d0 2167 current
4338 /dev/dsk/c2t1d0 2168 current /dev/dsk/c1t1d0 2168 current
4339 /dev/dsk/c2t2d0 0000 current /dev/dsk/c1t1d0 2169 current
4340 /dev/dsk/c2t2d0 0001 current /dev/dsk/c1t2d0 0000 current
4341 /dev/dsk/c2t2d0 0002 current /dev/dsk/c1t2d0 0001 current
4342 /dev/dsk/c2t2d0 0003 current /dev/dsk/c1t2d0 0002 current
4343 /dev/dsk/c2t2d0 0004 current /dev/dsk/c1t2d0 0003 current
4344 /dev/dsk/c2t2d0 0005 current /dev/dsk/c1t2d0 0004 current

[Michael Steele_2]

Regarding: "...4339 /dev/dsk/c2t2d0 0000 current /dev/dsk/c1t1d0 2169 current

4340 /dev/dsk/c2t2d0 0001 current /dev/dsk/c1t2d0 0000 current..."

I'm fairly certain this is because of the LVM header, which is stored in one PE. I don't believe you need an LVM header on each disk so one is all that is necessary.