country
hp home products & services support solutions how to buy
spacer
hp logo - invent
corner IT resource center corner
go search
go online help
go contact hp
go IT resource center home
go my profile
go logout
go maintenance and support
technical knowledge base
go support info by product
go knowledge trees
go individual patches
go more...
go forums
go training and education
go planning, design, and implementation
corner corner

Welcome, James R. Ferguson!
  (CA114842)
spacer
Procedure for replacing an LVM disk in HP-UX 10.x and 11.x
Current Path Home
Score : 0
Document Type : MD
Date : 2000 May 15
Description : Procedure for replacing an LVM disk in HP-UX 10.x and 11.x
Document Id : KBAN00000347
Search String : KBAN00000347

You may provide feedback on this document

View the printer friendly version of this document



Procedure for replacing an LVM disk in HP-UX 10.x and 11.x
DocId:KBAN00000347  Updated:20011214

DOCUMENT

Problem Description

What is the procedure for replacing a disk under LVM?  How do I
prepare for an LVM disk crash?  What information do I need to
have backed up/printed out to help in the recovery?

Configuration Info

Operating System - HP-UX
Version - 10.x and 11.x
Hardware System - HP9000
Series - 700/800

Solution

Note: The information in this document applies to HP-UX versions
      10.x and later.

This document replaces document UNX1001086

Despite all the effort various HP disk divisions put into making
highly reliable disk drives, occasionally a disk mechanism goes bad
and needs to be replaced.  After the old disk is replaced with a new
one (retaining the hardware address of the original to avoid
confusion), the data needs to be restored to that disk from a backup.
Before LVM, in the simple world of hard sectioning, this meant
checking your notes for the original sectioning scheme, then
recreating file systems as necessary on those sections, and restoring
the data from the backup.

With LVM, the picture is slightly more complicated, since one disk
could have various pieces of several logical volumes on it.  The layout
of those logical volumes (LVs or lvols) must first be restored, and the
data for each of those LVs restored from backup.

This document endeavors to provide a step-by-step guide to replacing a
faulty LVM disk, and outlines the general sequence of commands required
to perform the task.  It is divided into four chapters and one appendix:

  Chapter 1:      Replacing a NON-Boot Disk WITHOUT LVM Mirroring
  Chapter 2:      Replacing a NON-boot Disk WITH LVM Mirroring
  Chapter 3:      Replacing a Root Disk WITHOUT LVM Mirroring
  Chapter 4:      Replacing a Boot Disk WITH LVM Mirroring

  Appendix A:     Preparing for the Recovery of an LVM System

At the start of each of the chapters there is an example system
configuration, which is referred to in the examples for each of the
steps.  Use these example systems to understand the process, but
please be careful to issue the required commands relevant to the system
actually being recovered.

Of particular importance is Appendix A, which outlines the steps that
must be performed PRIOR to the disk failing.  Please make sure that you
read this section carefully, and implement the required procedures as
soon as possible.  Your system recovery may rely on these steps.  It is
recommended that you familiarize yourself with the procedures outlined in
this document prior to the time of ever needing them so that you
understand fully the steps involved.

If you have any questions about the recovery process, please contact your
local Hewlett-Packard Customer Response Center for assistance.

-------------------------------------------------------------------------
CHAPTER 1 :     Replacing a NON-boot Disk WITHOUT Mirroring
-------------------------------------------------------------------------

Example configuration:

  Volume group /dev/vg00 contains the three disks, with the logical
  volume configuration as shown:

                          |---------|     |---------|     |---------|
                          | lvol 1  |     | lvol 3  |     | lvol 5  |
                          |---------|     |---------|     |---------|
                          | lvol 2  |     | lvol 4  |     | lvol 6  |
                          |---------|     |---------|     |         |
                          | lvol 3  |     | lvol 5  |     |         |
                          |---------|     |---------|     |---------|

  hardware address          52.6.0          52.5.0          52.4.0
  device file (/dev/dsk/)   c0t6d0          c0t5d0          c0t4d0

  The scenario for this chapter is that the disk at hardware address
  52.4.0 has a head crash, and as a result it is unusable.  The steps
  below outline a method that can be used to recover from this state.

[Step 1.1]

Have the engineer replace the faulty disk, and then boot the system in
single user mode.  This will ensure that when we are working with the
system, there will be a minimum of processes running on the system.
To boot into single-user mode, boot from the primary boot path, and
interact with IPL.  At the ISL> prompt enter this string:

    ISL> hpux -is

Note: The above string should work for most systems.  If your system
fails to boot using this string follow the steps below.

At the ISL> prompt, determine the original boot string:

  ISL> lsautofl

This command will return a string such as:

  hpux (52.6.0;0)/stand/vmunix

or

  hpux (;0)/stand/vmunix

The output here is dependent on the type of system you have.  Once you
have this information, simply add the string -is after the hpux, and
this will boot the system into single user mode.  For our example:

  ISL> hpux -is (52.6.0;0)/stand/vmunix

[Step 1.2]

Restore the LVM configuration/headers onto the new disk from your backup
of the LVM configuration:

  # vgcfgrestore -n [volume group name] /dev/rdsk/cXtYdZ

Where X is the 'card instance number' of the SCSI bus attached to
that card.  Y is the 'SCSI ID' of the disk (or array controller, in
the case of an array), and Z is the 'LUN number' (typically 0 for a
non-array type disk). Note that if the HP Customer Engineer replaces
the disk at the same address, the device file name will not change.
In that case the name will be what it was prior to the replacement.
For our example:

  # vgcfgrestore -n /dev/vg00 /dev/rdsk/c0t4d0

[Step 1.3]

Reactivate the volume group (VG) so that the new disk can be attached,
since it wasn't configured in at boot time:

  # vgchange -a y [volume group name]

For our example, the volume group vg00 will already be activated, but
it will not know of the replaced disk; therefore, this step is still
required so that LVM will now know that the disk is again available:

  # vgchange -a y /dev/vg00

The vgchange command will activate each specified volume group and all
associated physical and logical volumes for read-write access.  In the case of
vg00,it would initially have been activated with c0t4d0 in an unknown state.
vgchange tells vg00 to look again at c0t4d0, which is now in a known state. It
is important to remember that even though lvol5 and lvol6 are now active they
are void of data.

[Step 1.4]

Determine which logical volumes spanned onto that disk.  You only need
to recreate and restore data for the volumes that actually touched that
disk.  Other LVs in the volume group are still OK.

  # pvdisplay -v /dev/dsk/c0tXd0

will show a listing of all the extents on disk lu X, and to what logical
volume they belong.  This listing is fairly long, so you might want to
pipe it to more or send it to a file.  For our example:

  # pvdisplay -v /dev/dsk/c0t4d0 | more
  .....
  .....
   --- Distribution of physical volume ---
   LV Name            LE of LV  PE for LV
   /dev/vg00/lvol5    50        50
   /dev/vg00/lvol6    245       245
   .....
   .....

From this we can see that logical volumes /dev/vg00/lvol5 and
/dev/vg00/lvol6 have physical extents on this disk, but /dev/vg00/lvol1
through /dev/vg00/lvol4 don't, so we will need to recreate and restore
lvol5 and lvol6 only.

Note: Even though lvol5 was also in part on another disk drive, it
      will need to be treated as if the entire lvol was lost, not just
      the part on c0t4d0.

[Step 1.5]

Restore the data from your backup onto the replacement disk for
the logical volumes identified in step 1.4.  For raw volumes, you can
simply restore the full raw volume using the utility that was used to
create your backup.  For file systems, you will need to recreate the
file systems first.  For our example:

For HFS:

  # newfs -F hfs /dev/vg00/rlvol5
  # newfs -F hfs /dev/vg00/rlvol6

For JFS:

  # newfs -F vxfs /dev/vg00/rlvol5
  # newfs -F vxfs /dev/vg00/rlvol6

Note that we use the raw logical volume device file for the newfs
command.  For file systems that had non-default configurations, please
consult the man page of newfs for the correct options.

After a file system has been created on the logical volume mount the
file system under the mount point that it previously occupied.  Take whatever
steps are necessary to prevent your applications or users from accessing the
filesystem until the data has been recovered.  Now that the filesystem has been
created simply restore the data for that file system from backups.

Note: You will need to have recorded how your file systems were
      originally created in order to perform this step.  The only
      critical feature of this step is that the file system be at
      least as large as before the disk failure.  You can change
      other file system parameters, such as those used to tune the
      file system's performance.

For the file system case, there is no need to worry about data on the
disk (c0t5d0) that was newer then the data on the tape.  The newfs
wiped out all data on the lvol5.  For the raw volume access, you may
have to specify your restore utilities over-write option to guarantee
bringing the volume back to a known state.

[Step 1.6]

The final step in the recovery process is to reboot the system.  When
the system restarts, the recovery process will be complete.

  # cd /
  # shutdown -r

If you have any questions or problems with the above steps, please
contact your local Hewlett Packard Customer Response Center.


----------------------------------------------------------------------
CHAPTER 2 :     Replacing a NON-Boot disk WITH Mirroring
----------------------------------------------------------------------

Mirroring introduces an interesting twist to the recovery process.
Because LVM keeps a map of 'stale' extents on each disk, it is only
aware of individual extents which are in need of update, and it does
not map this to entire disks.  This makes for quick mirror recovery in
the case that a disk has temporarily lost connection with the host, or
has lost power.  In addition, it can greatly speed up the recovery time
in the instance of a failed disk.

Example configuration:

  Volume group /dev/vg00 contains the three disks, with the logical
  volume configuration as shown:

                          |---------|     |---------|     |---------|
                          | lvol 1  |     | lvol 4  |     | lvol 4  |
                          |---------|     |         |     | mirror  |
                          | lvol 2  |     |---------|     |---------|
                          |---------|     | lvol 5  |     | lvol 5  |
                          | lvol 3  |     |         |     | mirror  |
                          |---------|     |---------|     |---------|

  hardware address          52.6.0          52.5.0          52.4.0
  device file (/dev/dsk/)   c0t6d0          c0t5d0          c0t4d0

This involves a head crash on the disk at address 52.4.0.

[Step 2.1]

Have the customer engineer replace the faulty disk, and then boot the system.
You can boot the system into multi-user mode.  For logical volumes that
didn't have a mirror, and that resided on the replaced disk, you will
have to deny access to the filesystem.  Do this by unmounting the relevant
filesystems. (However, it's possible that they will not have been mounted as
part of the boot-up sequence anyway.)

Note: If the disk being replaced is Hot-Pluggable (or Hot-Swappable) a
reboot may not be necessary.  Please inquire your customer engineer to
determine if a reboot is required.


[Step 2.2]

Restore the LVM configuration/headers onto the new disk from your
backup of the LVM configuration:

  # vgcfgrestore -n [volume group name] /dev/rdsk/cXtYdZ

where X is the 'card instance number' of the scsi bus attached to
that card.  Y is the 'SCSI ID' of the disk (or array controller, in
the case of an array), and Z is the 'LUN number' (typically 0 for a
non-array type disk). Note that if an HP Customer Engineer replaces
the disk at the same address, the device file name will not change.
In that case the name will be what it was prior to the replacement.
For our example:

  # vgcfgrestore -n /dev/vg00 /dev/rdsk/c0t4d0

[Step 2.3]

Reactivate the volume group so that the new disk can be attached,
since it wasn't configured in at boot time.  This should also resync
any mirrors that resided on the faulty disk.

  # vgchange -a y [volume group name]

For our example, the volume group vg00 will already be activated, but
it will not know of the replaced disk; therefore, this step is still
required so that LVM will now know that the disk is again available
and the resync will occur:

  # vgchange -a y /dev/vg00

Now check that the mirrors are syncronized, run this command for each
logical volume that resides on the replaced disk:

    # lvdisplay -v /dev/vg00/lvol4 | more
    # lvdisplay -v /dev/vg00/lvol5 | more

If any extents show up as being stale it is recommended to run a vgsync
or lvsync to force the syncronization of the mirrored extents using the
following commands:

    # vgsync /dev/vg00

or

    # lvsync /dev/vg00/lvol4
    # lvsync /dev/vg00/lvol5

[Step 2.4]

For any file systems on the faulty disk that didn't have mirror copies,
you will have to rebuild the file systems and restore the data.  Follow
the steps 1.4 and 1.5 in chapter 1 for guidance.

[Step 2.5]

If you booted your system into single-user mode in step 2.1, reboot
your system now and allow it to boot into multiuser mode.  If you were
already in multi-user mode, then no further action is required.

At this stage, your system should be fully functioning.

--------------------------------------------------------------------------------
CHAPTER 3 :     Replacing a Root Disk WITHOUT Mirroring
--------------------------------------------------------------------------------

With the failure of the boot disk the information on disk that is required to
boot the system is lost. The best method for recovery is to reload the boot disk
from a make_recovery tape.  The make_recovery command is supplied with the
Ignite-UX product available for free at www.software.hp.com.

If a make_recovery tape is unavailable then the system will need to be installed
with the Core OS/Install cd.  The original kernel and operating system
can be recovered from backups after that.  ITRC document rcfaxrestore001 has
complete instructions for recovering a system.

----------------------------------------------------------------------
CHAPTER 4 :     Replacing a boot disk WITH mirroring
----------------------------------------------------------------------

As with chapter 2, mirroring adds a new twist to the recovery of a boot
disk, and makes the recovery much simpler than without mirroring.

Example configuration:

  Volume group /dev/vg00 contains the three disks, with the logical
  volume configuration as shown:


                          |---------|     |---------|     |---------|
                          | lvol 1  |     | lvol 1  |     | lvol 4  |
                          |         |     | mirror  |     |         |
                          |---------|     |---------|     |---------|
                          | lvol 2  |     | lvol 2  |     | lvol 5  |
                          |         |     | mirror  |     |         |
                          |---------|     |---------|     |         |
                          | lvol 3  |     | lvol 3  |     |         |
                          |         |     | mirror  |     |         |
                          |---------|     |---------|     |---------|

  hardware address          52.6.0          52.5.0          52.4.0
  device file (/dev/dsk/)   c0t6d0          c0t5d0          c0t4d0

This scenario involves a head crash of the disk at hardware address
52.6.0.

[Step 4.1]

Shutdown the system, have the customer engineer replace the faulty
disk, and then boot the system in single-user mode from the alternate
boot disk.  If you only have two disks in the root volume group, then
you will need to override quorum as you boot.  Note that you don't have
to boot in single-user mode if all of the disk data is mirrored
elsewhere (but you may still have to override quorum).

Note: If the disk being replaced is Hot-Pluggable (or Hot-Swappable) a
reboot may not be necessary.  Please inquire your customer engineer to
determine if a reboot is required.  If a reboot is not required replace the
disk and go on to step 4.2.

For our example:

  ISL> hpux -is (52.5.0;0)/stand/vmunix

OR

  ISL> hpux -is -lq (52.5.0;0)/stand/vmunix
  (if you need to override quorum.)

[Step 4.2]

Restore the LVM configuration/headers onto the replaced disk from
your backup of the LVM configuration, activate the volume group
with the knowledge of the replaced disk, use mkboot to add boot
information, and syncronize the BDRA with these commands:

    # vgcfgrestore -n /dev/vg00 /dev/rdsk/c0t6d0
    # vgchange -a y /dev/vg00
    # mount /usr
    # mkboot /dev/rdsk/c0t6d0
    # mkboot -a "hpux (;0)/stand/vmunix" /dev/rdsk/c0t6d0
    # lvlnboot -R

Note: If /usr doesn't reside in /dev/vg00 the volume group that does
contain /usr's logical volume will also have to be activated.

At this point it is a good idea to verify that the BDRA information is
accurate.  This can be done with the lvlnboot comman using the -v
option.  Here is an example of the output:

    # lvlnboot -v
    Boot Definitions for Volume Group /dev/vg00:
    Physical Volumes belonging in Root Volume Group:
            /dev/dsk/c0t6d0 (52.6.0) -- Boot Disk
            /dev/dsk/c0t5d0 (52.5.0) -- Boot Disk
            /dev/dsk/c0t4d0 (52.4.0)
    Boot: lvol1     on:     /dev/dsk/c0t6d0
                            /dev/dsk/c0t5d0
    Root: lvol3     on:     /dev/dsk/c0t6d0
                            /dev/dsk/c0t5d0
    Swap: lvol2     on:     /dev/dsk/c0t6d0
                            /dev/dsk/c0t5d0
    Dump: lvol2     on:     /dev/dsk/c0t6d0, 0


Note: You must have performed the command vgcfgbackup to save off
      the headers prior to the disk failure (refer to Appendix A.).

[Step 4.3]

Now, synchronize mirrored logical volumes:

  # vgsync /dev/vg00

[Step 4.4]

If you have any logical volumes that resided on the faulty disk that
were not mirrored, you will need to recreate them as per steps 1.4
and 1.5 from chapter 1.

[Step 4.5]

The final step in the recovery process is to reboot the system.  When
the system restarts, the recovery process will be complete.

  # cd /
  # shutdown -r


At this stage, your system should be fully recovered.  It is a good idea to
verify the system can be booted from the newly replaced drive.


-----------------------------------------------------------------------
APPENDIX  A:    Preparing for the Recovery of an LVM System
-----------------------------------------------------------------------

These are procedures to ensure that the system's data and
configuration are recoverable in the event of a system failure.

1. Load any patches for LVM.

2. Regularly back up your entire system.

        Without a valid backup, you run a real chance of losing some or
        all of your data.  Ensure that you back up ALL of your important
        data, including the operating system directories such as:

                 /
                 /usr
                 /dev
                 /etc
                ...and so on.

        In addition, regularly test that your backups are working by
        restoring a test file randomly from your backups.  It is risky
        to assume that your backup is working because it is not logging
        any errors.  Many backup utilities have the capability to do
        some sort of validation of the backup media.  For example,
        fbackup has the -N option that can allow you to check for
        discrepancies between backup indices and what is actually on
        the tape.  Refer to the fbackup(1M) man page for more information.

        Use make_recovery to create a bootable recovery media.

3. Back up the important files separately.

        Take an extra copy of the very important files, preferably to
        another system as well as to another tape.  This will speed up
        recovery in the event of a system crash.  The files that should
        be backed up are:

                /etc/passwd
                /etc/group
                /etc/lvmtab
                /etc/lvmconf/*
                /etc/fstab

        There are many other important files on your system that you
        may wish to back up separately.  The files listed above are
        required to ensure a smooth system recovery.

4. Regularly print out the configuration of your system.

        The configuration details stored on the system may not be
        accessible during a recovery.  A printed copy is an invaluable
        reference.  We recommend printing the configuration details
        once a week and every time a change is made.  One thing to note
        is that some of the commands outlined below create large amounts
        of output.  An alternative to printing them is to output the
        information to a file and then storing the file off to tape.
        This allows quick recovery of the information when needed.  You
        could include this configuration file with the backup in step 3.

        The easiest way to save the configuration is to set up a cron
        job to run regularly, so that even if you don't remember to do
        it, the system will.

        The output from the following commands is recommended:

        /usr/sbin/ioscan  -fk

        /usr/sbin/vgdisplay  -v

        /usr/sbin/lvlnboot  -v

        /usr/sbin/lvdisplay -v /dev/vgXX/lvYY  (for every logical volume)

        /usr/sbin/pvdisplay  -v  /dev/dsk/c#t#d0   (for every LVM disk)

        lp /etc/fstab

        As an alternative, an intelligent script can be written that
        will detect any changes in the configuration and only print out
        those changes.  An example script is included at the end of this
        appendix.

5. Update the boot structures after every change to the root volume
   group.

        This task is only required if you are using LVM on your boot
        disk.  Whenever you make changes to the root volume group,
        which is usually named /dev/vg00, you must update the Boot
        Disk Reserved Area (BDRA) on the boot disk.  To do this issue
        the following command:

        /etc/lvlnboot -R

----------------------------------------------------------------

Example script for LVM configuration recording
**********************************************



#!/usr/bin/ksh

WORKDIR=/lvmbackup   # directory is regularly backed up, of course
LOG=$WORKDIR/log
SYSADM=root

if [ -f "$LOG" ]
then
  rm -f "$LOG"
fi

if [ ! -d "$WORKDIR" ]
then
  echo "missing directory $WORKDIR"
  exit 1
fi

cd $WORKDIR

/usr/sbin/vgdisplay -v > vgdisplay.new

LVMVGS=`grep "VG Name" vgdisplay.new | awk '{ print $3 }'`
LVMPVOLS=`grep "PV Name" vgdisplay.new | sort -u | awk '{ print $3 }'`
LVMLVOLS=`grep "LV Name" vgdisplay.new | sort -u | awk '{ print $3 }'`

/usr/sbin/pvdisplay -v $LVMPVOLS > pvdisplay.new

/usr/sbin/lvdisplay -v $LVMLVOLS > lvdisplay.new

/usr/sbin/lvlnboot -v > lvlnboot.new 2> /dev/null

/usr/sbin/ioscan -fk > ioscan.new

cp /etc/fstab fstab.new

for CURRENT in *new
do
  ORIG=${CURRENT%.new}

  if diff $CURRENT $ORIG > /dev/null
  then
    # files are the same....do nothing
    rm $CURRENT
  else
    # files differ...make the new file the current file, move old
    # one to file.old

    echo `date` "The config for $ORIG has
    changed." >> $LOG
    echo "Copy of the new $ORIG config has
    been printed" >> $LOG

  lp $CURRENT
  mv $ORIG ${ORIG}old
  mv $CURRENT $ORIG
  fi
done

if [ -s "$LOG" ]
then
  mailx -s "LVM configs have changed" $SYSADM < $LOG
fi

exit 0



To help us improve our content, please provide feedback and any additional comments below. If you have a problem or a question that needs immediate attention, please submit a call or contact your HP Response Center instead.

Relevance to my question
           Provided the information I needed.
           Helped me and/or pointed me to the information.
           Did not pertain to my question/problem.
Content Quality
           Is good, please provide more like it.
           Is hard to understand and/or needs more detail.
           Has incorrect technical information.
           Took the recommended action, but it did not solve my problem.
Access to resources
           Cannot access the Web site listed.
           Cannot find the resource listed.


Comments:
As much as possible, please identify the sections of the document to which your comments relate. When you have finished, press the Submit button to mail your evaluation and comments.


reset

printing instructions
go printing instructions
Privacy Statement Legal Notices © 1994-2001 Hewlett-Packard Company