Mirrored disc has crashed

[MarkSyder]

Hi everybody.

Why does this happen just before Christmas?

A disc in a Surestore SC10 has crashed. The disc is mirrored, so the users are still working without any problems, but I have made arrangements to have a replacement disc sent to my house (too late for it to be sent to work). When it arrives I am going to come into work to install it. I don't expect problems, but if I do have problems there may not be any people on the forum to help me, so I'll ask about the potential pitfalls in advance.

I've read all sorts of things on here about breaking mirrors - I haven't broken the mirror on this system because the system seems to be working and I don't see the point in changing things that are working. Do I need to do this before I install the replacement disc?

When I install the new disc, do I simply run ioscan and wait for the two mirrors to sync with each other or is there more to it than that?

All attempts to help will be rewarded with points, but please be patient - you may have to wait till January so I can assess exactly how much help each answer has been.

Mark Syder (like the drink but spelt different)

[Luk Vandenbussche]

Hi Mark,

If the mirroring is done well, you can recreate is with

pvcreate -f /dev/rdsk/cxtydz
vgcfgrestore /dev/vgxx
vgsync /dev/vgxx

[Patrick Wallek]

I assume it's a non-vg00 disk. So:

# vgcfgrestore -n vg?? /dev/rdsk/c?t?d?
# vgchange -a y vg??
# vgsync vg??

[James R. Ferguson]

Hi Mark:

If you are sure about which mirror is failing, I'd break each logical volume's mirror:

# lvreduce -m 0 /dev/vgNN/lvolN /dev/dsk/cXtYdZ'

...and then 'vgreduce' the physical volume from the volume group.

When you, replace the disk, 'vgextend' the new physical volume (after a 'pvcreate') and 'lvextend' each logical volue to remirror.

BTW, a new, execllent whitepaper on disk replacement is :

http://docs.hp.com/en/5991-1236/When_Good_Disks_Go_Bad.pdf

Be sure that you install the appropriate LVM OLR (onlines-replacement) patches mentioned therein for optimal help.

Regards!

...JRF...
Regards!

...JRF...

[Bill Hassell]

Since the disk has crashed (or otherwise not working), you are already unmirrored (stale extents in the terminology of lvdisplay -v). So with the SC10 (hot pluggble), you just pull out the bad disk, then put in the new one, make it available as an LVM disk with pvcreate, restore the LVM header information to the front of the disk with vgcfgrestore. Syncing won't occur automatically (except on a reboot) so you can manually sync each lvol with lvsync, or do it with one command: vgsync vgWhatever.

[Victor BERRIDGE]

Hi Mark
just in case, there is the DocId KBAN00000347
LVM: Procedure for replacing an LVM disk in HP-UX 10.x and 11.x DocId: KBAN00000347


Excerpt:
CHAPTER 2 : Replacing a NON-Boot disk WITH Mirroring
----------------------------------------------------------------------

Mirroring introduces an interesting twist to the recovery process.
Because LVM keeps a map of 'stale' extents on each disk, it is only
aware of individual extents which are in need of update, and it does
not map this to entire disks. This makes for quick mirror recovery in
the case that a disk has temporarily lost connection with the host, or
has lost power. In addition, it can greatly speed up the recovery time
in the instance of a failed disk.

Example configuration:

Volume group /dev/vg00 contains the three disks, with the logical
volume configuration as shown:

|---------| |---------| |---------|
| lvol 1 | | lvol 4 | | lvol 4 |
|---------| | | | mirror |
| lvol 2 | |---------| |---------|
|---------| | lvol 5 | | lvol 5 |
| lvol 3 | | | | mirror |
|---------| |---------| |---------|

hardware address 52.6.0 52.5.0 52.4.0
device file (/dev/dsk/) c0t6d0 c0t5d0 c0t4d0

This involves a head crash on the disk at address 52.4.0.

[Step 2.1]

Have the customer engineer replace the faulty disk, and then boot the system.
You can boot the system into multi-user mode. For logical volumes that
didn't have a mirror, and that resided on the replaced disk, you will
have to deny access to the filesystem. Do this by unmounting the relevant
filesystems. (However, it's possible that they will not have been mounted as
part of the boot-up sequence anyway.)

Note: If the disk being replaced is Hot-Pluggable (or Hot-Swappable) a
reboot may not be necessary. Please inquire your customer engineer to
determine if a reboot is required.


[Step 2.2]

Restore the LVM configuration/headers onto the new disk from your
backup of the LVM configuration:

# vgcfgrestore -n [volume group name] /dev/rdsk/cXtYdZ
where X is the 'card instance number' of the scsi bus attached to
that card. Y is the 'SCSI ID' of the disk (or array controller, in
the case of an array), and Z is the 'LUN number' (typically 0 for a
non-array type disk). Note that if an HP Customer Engineer replaces
the disk at the same address, the device file name will not change.
In that case the name will be what it was prior to the replacement.
For our example:

# vgcfgrestore -n /dev/vg00 /dev/rdsk/c0t4d0

[Step 2.3]

Reactivate the volume group so that the new disk can be attached,
since it wasn't configured in at boot time. This should also resync
any mirrors that resided on the faulty disk.

# vgchange -a y [volume group name]

For our example, the volume group vg00 will already be activated, but
it will not know of the replaced disk; therefore, this step is still
required so that LVM will now know that the disk is again available
and the resync will occur:

# vgchange -a y /dev/vg00

Now check that the mirrors are syncronized, run this command for each
logical volume that resides on the replaced disk:

# lvdisplay -v /dev/vg00/lvol4 | more
# lvdisplay -v /dev/vg00/lvol5 | more

If any extents show up as being stale it is recommended to run a vgsync
or lvsync to force the syncronization of the mirrored extents using the
following commands:

# vgsync /dev/vg00

or

# lvsync /dev/vg00/lvol4
# lvsync /dev/vg00/lvol5

[Step 2.4]

For any file systems on the faulty disk that didn't have mirror copies,
you will have to rebuild the file systems and restore the data. Follow
the steps 1.4 and 1.5 in chapter 1 for guidance.

Step 2.5]

If you booted your system into single-user mode in step 2.1, reboot
your system now and allow it to boot into multiuser mode. If you were
already in multi-user mode, then no further action is required.

At this stage, your system should be fully functioning.

.....


All the best
Victor

[erics_1]

I agree will Bill on this. I would add that after running vgcfgrestore, a vgchange -a y normally will kick off a vgsync automatically. If not, certainly run vgsync. If for some reason the disk is still 'claimed' in ioscan, I would break the mirror and reduce out the disk before replacing.

Hope this helps.

Eric

[OldSchool]

Mark,

If you haven't already, I'd look at
http://docs.hp.com/en/5991-1236/When_Good_Disks_Go_Bad.pdf

Deals w/ disk failures and LVM (from LVM team).

I'd go ahead and break the mirror and remove the disk from the volume group, as recommended section 5.

just my 2cents

[MarkSyder]

Hi everybody.

pvcreate fails because the disc is already in /etc/lvmtab. I follow the advice in the document and get the following result from pvdisplay /dev/dsk/c0t1d0:

pvdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t1d0":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query all of the physical volumes.
pvdisplay: Couldn't retrieve the names of the physical volumes
belonging to volume group "/dev/vg03".
pvdisplay: Cannot display physical volume "/dev/dsk/c0t1d0".

Guidance appreciated. I may be going home shortly - I've come in specially to do this - so I may not be able to act on your advice till tomorrow.

Mark

[Eric Antunes]

Hi Mark,

I don't use to do this (that's HP job) but the process the HP Engineer runs its very similar to Bill's Hassel post.

Best Regards,

Eric Antunes

[Shameer.V.A]

Hi Mark,
Procedure to replace a root disk is as follows

ISL> hpux -lq
# pvcreate -f -B /dev/rdsk/cXtXd0
pvcreate: The physical volume "/dev/dsk/c4t6d0" is already recorded in
the "/etc/lvmtab" file.

# vgcfgrestore -n /dev/vg00 /dev/rdsk/cXtXd0
vgcfgrestore: Couldn't query physical volume "/dev/dsk/c4t6d0":

# mkboot /dev/rdsk/cXtXdX

# mkboot -a "hpux -lq" /dev/rdsk/cXtXdX

# vgchange -a y /dev/vg00
Volume group "/dev/vg00" has been successfully changed.

#vgsync /dev/vg00

To verify offline diagnostics are installed on root and mirror disks:
# lifls -il /dev/rdsk/cXtXd0

To download offline diagnostics:
# cd /usr/sbin/diag/lif

if server is a K-Class:
# mkboot -b updatediaglif -p ISL -p HPUX -p LABEL -p AUTO /dev/rdsk/cXtXd0

if server is a L/N/V/RP Class:
# mkboot -b updatediaglif2 -p ISL -p HPUX -p LABEL -p AUTO /dev/rdsk/cXtXd0

# lvlnboot -Rv /dev/vg00


Please find the attached documents also for reference..

Hope this will help ..

Shameer

[Bill Hassell]

You wrote:

"pvcreate fails because the disc is already in /etc/lvmtab. I follow the advice in the document and get the following result from pvdisplay /dev/dsk/c0t1d0:

pvdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t1d0":
...

pvcreate fails if the replacement disk has some LVM information. You have to force the LVM information with pvcreate -f. Then run vgcfgrestore to put back the correct information. The information above says pvdisplay "couldn't query" really means that this disk doesn't belong (yet) to the volume group. Once vgcfgrestore is complete, run vgchange to tie all the disks together and make sure vgsync is running.

[Cem Tugrul]

hi Mark,

how about these steps;
mv /etc/lvmtab /etc/lvmtab.old
vgscan -av

[Torsten.]

Don't do a pvcreate, because all information about the disk config is stored. Restore the config by

# vgcfgrestore -n VGxx /dev/rdsk/cXtXdX

and activate the vg:

vgchange -a y vgxx

now sync the vg:

vgsync vgxx

That's all, except your disk is a boot device. In this case don't forget to make it bootable by restoring the boot area:

# mkboot -l /dev/rdsk/cXtXdX

# lvlnboot -r /dev//lvol3
# lvlnboot -b /dev//lvol1
# lvlnboot -s /dev//lvol2
# lvlnboot -d /dev//lvol2

check:

# lvlnboot -v

[Sivakumar TS]

Hi Mark,

vgchange -a n

vgcfgrestore /dev/

vgchnage -a y

if required,

vgsync /dev/

this should serve the purpose.

With Regards,

Siva.

[MarkSyder]

Hi everybody and happy new year to you all.

I raised a call with HP and eventually found that the problem was caused by an unused logical volume on the old disc. The system could see this lvol (I suspect as a result of the vgscan) and this was causing the error messages. Once I removed the lvol the pvcreate etc. went smoothly.

Mark

[Paul Coffey_1]

Mark,

What steps did you take to find the unused logical volume? I'm having the same problem at the moment.

[MarkSyder]

Hi Paul.

I logged a call with HP and they emailed me a script to run on the server. They then talked me through removing the incorrect lvol.

I checked my notes yesterday because someone else had started a similar thread, but I can't find the command I used to delete it.

Mark

[MarkSyder]

Paul,

I forgot to say: you can borrow my forum signature if you want.

Mark Syder (like the drink but spelt different)