HP-UX 11.23 failed internal mirrored vg00 disk

[ManuelRC]

Hi.

I've got an old HP-UX 11.23 server. It has 2 internal disks for vg00, mirrored (except one filesystem).

One of the disks has failed, and we need to replace it.

Before replacing the failed disk with the new one, I'd like to check the steps we'll need to do.

I've seen this post (http://h30499.www3.hp.com/t5/LVM-and-VxVM/Replace-Mirror-Disk-Boot-HPUX-11-23/td-p/4951953#.Uv4Z2OKDNMt) and the "When Good Disks Go Bad.pdf".

In the post the steps are:

vgcfgrestore

mkboot

mkboot -a

vgchange

lvlnboot

vgsync

In the pdf, there are a lot more steps, first try to reduce the mirrors, with lvreduce, then removing the failed disk from the vg with vgreduce, disable LVM access, etc...

Before taking any action, I'd like to check the steps we should do.

I've attached some information about the failed disk.

Thanks

[likid0]

Disabling failed disk
#pvchange -a N /dev/dsk/c1t1d0s2   ---> in your case, it's not present

Physically Replace the disk, and Verify the new disk 

#ioscan -fC disk

Cleaning old EFI configuration
#idisk -Rw /dev/rdsk/c1t1d0

Using VI Create disk partitioning description file
#vi /tmp/partitionfile
3
EFI 500MB
HPUX 100%
HPSP 400MB

Partitioning of new disk
#idisk -f /tmp/partitionfile -w /dev/rdsk/c1t1d0

Partitioning verification
#idisk /dev/rdsk/c1t1d0

Creating DSF
#insf -e -H $(lssf /dev/dsk/c1t1d0 | awk ' { print $(NF-1) } ')

Initializing FAT File system for the EFI partition
#efi_fsinit -d /dev/rdsk/c1t1d0s1

Creating LVM bootable disk
#pvcreate -Bf /dev/rdsk/c1t1d0s2

Copy software boot (with EFI)
#mkboot -e -l /dev/rdsk/c1t1d0

Modification of AUTO file for the boot quorum
#mkboot -a boot vmunix -lq /dev/rdsk/c1t1d0

Verifying« AUTO » file
# efi_cp -d /dev/rdsk/c1t1d0s1 -u /EFI/HPUX/AUTO /tmp/AUTO
#cat /tmp/AUTO

Checking the contents of the LIF directory
#efi_ls -d /dev/rdsk/c1t1d0s1

Restoring LVM configuration on the new disk
#vgcfgrestore -n vg00 /dev/rdsk/c1t1d0s2

Enabling disk:
#pvchange -a y /dev/dsk/c1t1d0s2 
#vgchange -a y vg00
#lvlnboot -R /dev/vg00

Resynchronization LV only if vchange does not start automatically the resync
# vgsync /dev/vg00

Check sync,once it is in the state available/syncd it has finished the sync
#vgdisplay -v vg00 | grep -E '(LV Name|LV Status)'
check how many stale PE's are left on the lvol change LVOLNAME with name of lvol
#lvdisplay -v LVOLNAME | grep stale | wc -l

[ManuelRC]

Thanks a lot.

I'll try it and I will post the results.

[ManuelRC]

I will follow these steps this afternoon, but I've got one last question.

pvdisplay and ioscan does not show info about c1t1d0s2, but the device files are still present in /dev/dsk

#ls -l /dev/dsk/ | grep c1t1d0
brw-r-----   1 bin        sys         31 0x011000 Jun 11  2007 c1t1d0
brw-r-----   1 bin        sys         31 0x011001 Jun 11  2007 c1t1d0s1
brw-r-----   1 bin        sys         31 0x011002 Jun 11  2007 c1t1d0s2
brw-r-----   1 bin        sys         31 0x011003 Jun 11  2007 c1t1d0s3

Should I delete these files before replacing the disk?

[ManuelRC]

Well, yesterday I tried to follow these steps, but I had some minor problems, and a big problem.

First of all, the new disk is not c1t1d0, instead it is c1t2d0.

#ioscan -funC disk

...

disk        87  0/4/1/0.0.0.2.0      sdisk          CLAIMED     DEVICE       HP      DG0146FAMWL
                               /dev/dsk/c1t2d0     /dev/dsk/c1t2d0s2   /dev/rdsk/c1t2d0    /dev/rdsk/c1t2d0s2
                               /dev/dsk/c1t2d0s1   /dev/dsk/c1t2d0s3   /dev/rdsk/c1t2d0s1  /dev/rdsk/c1t2d0s3

...

In a second place, the second mkboot command failed:

#mkboot -a boot vmunix -lq /dev/rdsk/c1t2d0 ==> Failed

Taking a look at the other post and at the "When good disks...pdf" I finally executed this other command, that ended OK:

#mkboot -a "hpux -lq (;0)/vmunix" /dev/rdsk/c1t2d0

Everything went well until the vgcfgrestore command.

Since c1t2d0s2 was not in vg00 before, I cannot restore the vg00 conf.

So now I suppose I've got two options:

 - try to reduce the vg00, to remove the old c1t1d0s2 disk. Then extend vg00 with the new c1t2d0s2, and recreate the mirrors.

 - try to remove and re-add the new disk, and see if it can get the original c1t1d0 name, instead of c1t2d0. Then continue with the procedure.

I don't like the first option, because right now vg00 is not OK, and I don't know how it will behave.

I've attached some information about vg00.

As you can see, all the OS lvols are mirrored and stale (except lvFAULT, that was stripped), so I think I need to remove the failed copy of the mirrors.

What scares me is that lvlnboot -v only show an error, it doesn't show any more information. It should show some information about the remaining disk.

#lvlnboot -v
lvlnboot: Couldn't query physical volume "/dev/dsk/c1t1d0s2":
The specified path does not correspond to physical volume attached to
this volume group

Yesterday I tried to move /etc/lvmtab and recreate it with vgscan, but this deleted vg00 from the system:

#vgscan -v

Creating "/etc/lvmtab".

vgscan: Couldn't access the list of physical volumes for volume group "/dev/vg00".

vgscan: Couldn't access the list of physical volumes for volume group "/dev/SAPVG".

[The same error for all my VGs] 

Physical Volume "/dev/dsk/c0t0d0" contains no LVM information

Physical Volume "/dev/dsk/c1t2d0s2" is not part of a Volume Group

vgscan:  has no correspoding valid raw device file under /dev/rdsk.

Verification of unique LVM disk id on each disk in the volume group /dev/vg00 failed.

/dev/SAPVG

/dev/dsk/c10t0d1

/dev/dsk/c10t0d5

[...]

#vgdisplay -v /dev/vg00

vgdisplay: Volume group "/dev/vg00" does not exist in the "/etc/lvmtab" file.
vgdisplay: Cannot display volume group "/dev/vg00".

I recovered the original /etc/lvmtab, but now I'm not sure what should I do.

I now that this system is not OK, and that I should solve all the errors, but right now I need to recover a mirrored vg00, and then I will focus on the other problems.

At this point I'd like to remove the new c1t2d0 disk, and try to re-add it with the old c1t1d0 name, to see if I'm able to recover vg00.

What should I do to do this?

Thanks.

[likid0]

Hi,

If the disk was inserted in the same slot it doesn't have to change de scsi id, did you replace the disk in the same slot?.

What model is your server?.

#model

#uname -a

Can you also run:

#lvlnboot -v /dev/vg00

We can remove all the stale info in vg00 from the failed disk, and add the new c1t2d0 to the mirror, but first you need to have the boot LABLE working for the "good" disk c1t0d0s2.

[ManuelRC]

I've been told (the server is hundreds km away from my office) that the disk was inserted in the same slot. Removed the old failed disk, and inserted the new one in the same slot.

I suspect that when I did ioscan, the /dev/(r)dsk/c1t1d0* device files still existed in the server, so this may have forced the new disk to get a different name. Is this possible?

#model
ia64 hp server rx6600

#uname -a
HP-UX node01 B.11.23 U ia64 2654504742 unlimited-user license

#lvlnboot -v /dev/vg00
lvlnboot: Couldn't query physical volume "/dev/dsk/c1t1d0s2":
The specified path does not correspond to physical volume attached to
this volume group

Thanks a lot.

[likid0]

Hi,

No the problem is not with not removing the device files, that step is not needed.

The problem is that rx6600 has a sas controller and you need some additional steps to get the old device ID in use.

Here you have a guide that has the steps to follow, the online disk replace is around page 66:

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c02017117/c02017117.pdf

[ManuelRC]

Thanks a lot.

So now I suppose I need to do a replace target:

# sasmgr replace_tgt -D /dev/sasd0 -q old_tgt=/dev/dsk/c1t1d0 -q new_tgt_hwpath=0/4/1/0.0.0.2.0

But first of all, I think I should delete all the new conf I've already done on the new c1t2d0(s*).

How can erase and delete the conf that has been done?

Thanks.

[ManuelRC]

OK, already donde with idisk -R, but ioscan still showed the c1t2d0sX devices.

Deleted them with rmsf, and now it seems ok.

I will go on with sasmgr replace_tgt, and I will update this.

Thanks.