Replace vg00 disk with ignite

[hydrocct]

I need to replace the two mirrored vg00 disks by bigger ones. This is a two node cluster with two packages and two shared vgs. The new disks do not apply to the shared vgs. The plan is as follows:
1) make_tape_recovery of both nodes (vg00 only) while packages are mounted and running on each node (this is a production system)
2) Switch all packages to node A
3) halt node B and replace vg00 disks
4) boot node B from ignite tape, modify size of vg00 logical volumes in interactive mode, and let ignite reinstall the system.
5) repeat steps 2-5 in reverse order

I have reached step 4, but after hitting the GO! button, ignite complained that the contents of the shared vgs would be erased, so I interrupted the process.

Can this be done without affecting the shared vgs? How can I have ignite not see the shared vgs, while keeping all config files related to those vgs on vg00?

[RAC_1]

Could not quite get it.

How many vgs you have? Where SG resides? On which VG?

If I understand your question correctly, you want to replace the disks in vg00? VG00 is mirrored? right? If yes then you can just switch the package to another node, do cmhaltnode, break the mirror, put big disk, mirror again, then boot from big disk, replace another boot disk in vg and mirror again. Done with one node. Same to the other.

[Sridhar Bhaskarla]

Hi,

I would suggest remove the cables to the storage and keep even only the bigger disks. This way you are sure that it is not going to touch your data.

You can very easily import the volume groups back by either preview exporting the volume groups with -s or -f option.

-Sri

[hydrocct]

On each node, there are 3 VGs: VG00, VGP1, and VGP2. The last two are shared and each contain a cluster package. I understand your solution, which does not use Ignite-UX, but I think problems with result from bad VGIDs on the vg00 disks. It all depends on how you add the new larger disks to vg00. Also, my vg00 was built with a limit on disk size at 18 GB. The new disks are 36 GB. Ignite is the safest way of rebuilding vg00.

[hydrocct]

For Sri,

Could you be more specific as to what cables you want disconnected and which commands are you refering to? The vg00 disks are regulare SCSI mounted on the L2000 server. The shared VG disks are in fibre channel cabinet.

[RAC_1]

You can take ignite backup so that you can use if require. The method I told is fast, but does not take care of large size of the disk. You can take care of it as follows.

1. Swtich all packages to another node.
2. Package is active on another node, so shared vgs are active on another node. On another node, do vgexport -pv -s -m /tmp/map.files vgname. Repeat it for all shared vgs.
3. On first node (from where you switched the package)break the mirror, boot and replace disk.
4. Prepare the static copy of root disk. Procedure is at the bottom. (This will take care of large size of disk.)
5. Boot. Now boot from this large disk. Replace another disk and mirror again.
6. Copy all map files for all shared vgs from another node. Do vgimport.
7. Switch package to first node.
8. Follow procedure for another node.

Procedure for preparing staic copy.

Note: This will create an identical copy of the current vg00. The

new volume group needs to as big as vg00. This will also be a static

version of the primary boot disk which could be use in case of

problem.

Note: The following example is using the disk c1t6d0 and the

volume group vg01

1) Initialize the disk and make it bootable

pvcreate -B /dev/rdsk/c1t6d0

Note: the -B parameter tells pvcreate that this will be a bootable

disk.

mkboot /dev/dsk/c1t6d0

mkboot -a "hpux" /dev/rdsk/c1t6d0

2) Create the volume group

mkdir /dev/vg01

mknod /dev/vg01/group c 64 0x010000

vgcreate /dev/vg01 /dev/dsk/c1t6d0
(Use -e option in vgcreate to take care of large size of replaced disk)

3) Find the size of each logical volume in vg00

vgdisplay -v /dev/vg00 | more

look at LV Size (Mbytes) for each logical volume and note it.

Note: this example will use these value:

lvol1 84M

lvol2 256M

lvol3 140M

lvol4 500M

lvol5 64M

lvol6 20M

lvol7 500M

lvol8 500M

Note: The size of the new logical volumes needs to be exactly the

same as the size of the logical volumes on the primary root disk.

4) Create the first 3 logical volumes contiguous (needed by the system)

lvol1:

lvcreate -L 84 -C y -r n /dev/vg01

lvol2:

lvcreate -L 256 -C y -r n /dev/vg01

lvol3:

lvcreate -L 140 -C y -r n /dev/vg01

5) Now create the other logical volumes

lvol4:

lvcreate -L 500 /dev/vg01

lvol5:

lvcreate -L 64 /dev/vg01

lvol6:

lvcreate -L 20 /dev/vg01

lvol7:

lvcreate -L 500 /dev/vg01

lvol8:

lvcreate -L 500 /dev/vg01

6) Copy each logical volume except the swap which is usually lvol2.

dd if=/dev/vg00/rlvol1 of=/dev/vg01/rlvol1 bs=1024k

dd if=/dev/vg00/rlvol3 of=/dev/vg01/rlvol3 bs=1024k

dd if=/dev/vg00/rlvol4 of=/dev/vg01/rlvol4 bs=1024k

dd if=/dev/vg00/rlvol5 of=/dev/vg01/rlvol5 bs=1024k

dd if=/dev/vg00/rlvol6 of=/dev/vg01/rlvol6 bs=1024k

dd if=/dev/vg00/rlvol7 of=/dev/vg01/rlvol7 bs=1024k

dd if=/dev/vg00/rlvol8 of=/dev/vg01/rlvol8 bs=1024k

7) Verify the integrity of all the new volume except swap.

Note: The following lines are base on a system with vxfs

filesystems except for /stand (lvol1) which needs to be hfs.

fsck -F hfs /dev/vg01/rlvol1

fsck -F vxfs /dev/vg01/rlvol3

fsck -F vxfs /dev/vg01/rlvol4

fsck -F vxfs /dev/vg01/rlvol5

fsck -F vxfs /dev/vg01/rlvol6

fsck -F vxfs /dev/vg01/rlvol7

fsck -F vxfs /dev/vg01/rlvol8

8) Now configure the Boot Data Reserved Area (BDRA)

Note: The following commands assume that /stand is lvol1,

swap is lvol2 and / is lvol3

lvlnboot -b /dev/vg01/lvol1 /dev/vg01

lvlnboot -r /dev/vg01/lvol3 /dev/vg01

lvlnboot -s /dev/vg01/lvol2 /dev/vg01

lvlnboot -d /dev/vg01/lvol2 /dev/vg01

9) Modify the fstab file on the new disk.

a) If /tmp_mnt doesn't exist create it

mkdir /tmp_mnt

b) Mount the new root filesystem on /tmp_mnt

mount /dev/vg01/lvol3 /tmp_mnt

c) change to etc directory on the new disk.

cd /tmp/etc

d) Modify all occurence of vg00 in the fstab for vg01

sed "s/vg00/vg01/" fstab > fstab.out

mv fstab fstab.BAK

mv fstab.out fstab

e) Unmount the new root filesystem

cd /

umount /tmp_mnt

[Sridhar Bhaskarla]

I meant to remove the cables that are going to fiber channel storage. That way you are sure that your data will be in tact. Since the packages are going to be on the other server, you can generate map files there and import them back on the restored server.

#vgexport -p -s -v -m /tmp/vgxx.s.map vgxx
#vgexport -p -v -m /tmp/vgxx.map -f /tmp/vgxx.disk vgxx

The second export command will capture all the disks into vgxx.disk file. That is only if the first option doesn't work or takes a long time.

DO it on all the VGs and copy them back to the restored server. Create directories and group files and import the VGs.

#mkdir /dev/vgxx
#mknod /dev/vgxx/0x0?0000
(get the minor number from the seondary node)
#vgimport -v -s -m /tmp/vgxx.s.map vgxx

-Sri

[hydrocct]

RAC,

Thanks for such a detailed procedure. Essentially, what you are doing is:
1)booting on half vg00 with the new disk connected,
2) creating vg01 as a copy of vg00 (one may increase the size of each logical volume)
3) erase all references to vg00 in vg01
4) halt, remove vg00 and insert the second new disk
5) boot from vg01 and mirrior onto the second new disk
Now the system uses vg01 as its boot vg.

The process is lengthy and prone to human error. I am looking for an ignite-based solution which would be more automated and reliable

[hydrocct]

I have a third server on which I can transfer production, giving complete access to the cluster for maintenance. This 3rd server is not part of the cluster. If I boot each cluster member with the cluster disabled (hence using vg00 only), I can do a make_tape_recovry of vg00 only. When booting and installing from the ignite tape, should it again complain about the shared vgs, even with these not having been mounted during the archive creation?

[Sridhar Bhaskarla]

Hi,

Ignite is not supposed to touch anything other than vg00 if the tape is created with -x inc_entire=vg00 option. But to be on the safer side, you can disconnect the cables to your external storage and recover the image.

1. Create make_tape_recoveries for both the servers.
2. Move the package to other node and halt node1.
2. Disconnect cables to the external storage. Only disks you have in the system will be bigger disks. This way you are sure your shared VGs as well your original disks are not touched.
3. Once the system is up, reconnect the cables, do 'ioscan' followed by 'insf' and vgimport the VGs. If the device files are changed then you will have to change your lock disk definition in your cluster configuration file and re-apply the configuration. But it doesn't hurt as you will have to take downtime anyway to move the package.

You may be unnecessarily complicating the procedure. Backout is real easy. Put back the old disks and boot.

-Sri