problems with disk

emmett_2 — Fri, 07 Oct 2005 13:44:57 GMT

We had a failed drive...the CE pulled out the disk and put in a new one without any intervention before pulling out the failed disk.
now the disk is showing as unavailable with stale partitions.....

this was a mirrored disk.....not rootvg....

neither lvm software or olr software is installed on this server

Re: problems with disk

Luk Vandenbussche — Fri, 07 Oct 2005 14:14:05 GMT

Hi,
here you have a lot of info on disk mirroring

LVM: Procedure for replacing an LVM disk in HP-UX 10.x and 11.x DocId: KBAN00000347 Updated: 20040602

DOCUMENT
Problem Description

What is the procedure for replacing a disk under LVM? How do I
prepare for an LVM disk crash? What information do I need to
have backed up/printed out to help in the recovery?

Configuration Info

Operating System - HP-UX
Version - 10.x and 11.x
Hardware System - HP9000
Series - 700/800

Solution

Note: The information in this document applies to HP-UX versions
10.x and later.

This document replaces document UNX1001086

Despite all the effort various HP disk divisions put into making
highly reliable disk drives, occasionally a disk mechanism goes bad
and needs to be replaced. After the old disk is replaced with a new
one (retaining the hardware address of the original to avoid
confusion), the data needs to be restored to that disk from a backup.
Before LVM, in the simple world of hard sectioning, this meant
checking your notes for the original sectioning scheme, then
recreating file systems as necessary on those sections, and restoring
the data from the backup.

With LVM, the picture is slightly more complicated, since one disk
could have various pieces of several logical volumes on it. The layout
of those logical volumes (LVs or lvols) must first be restored, and the
data for each of those LVs restored from backup.

This document endeavors to provide a step-by-step guide to replacing a
faulty LVM disk, and outlines the general sequence of commands required
to perform the task. It is divided into four chapters and one appendix:

Chapter 1: Replacing a NON-Boot Disk WITHOUT LVM Mirroring
Chapter 2: Replacing a NON-boot Disk WITH LVM Mirroring
Chapter 3: Replacing a Root Disk WITHOUT LVM Mirroring
Chapter 4: Replacing a Boot Disk WITH LVM Mirroring

Appendix A: Preparing for the Recovery of an LVM System

At the start of each of the chapters there is an example system
configuration, which is referred to in the examples for each of the
steps. Use these example systems to understand the process, but
please be careful to issue the required commands relevant to the system
actually being recovered.

Of particular importance is Appendix A, which outlines the steps that
must be performed PRIOR to the disk failing. Please make sure that you
read this section carefully, and implement the required procedures as
soon as possible. Your system recovery may rely on these steps. It is
recommended that you familiarize yourself with the procedures outlined in
this document prior to the time of ever needing them so that you
understand fully the steps involved.

If you have any questions about the recovery process, please contact your
local Hewlett-Packard Customer Response Center for assistance.

-------------------------------------------------------------------------
CHAPTER 1 : Replacing a NON-boot Disk WITHOUT
Mirroring
-------------------------------------------------------------------------

Example configuration:

Volume group /dev/vg00 contains the three disks, with the logical
volume configuration as shown:

|---------| |---------| |---------|
| lvol 1 | | lvol 3 | | lvol 5 |
|---------| |---------| |---------|
| lvol 2 | | lvol 4 | | lvol 6 |
|---------| |---------| | |
| lvol 3 | | lvol 5 | | |
|---------| |---------| |---------|

hardware address 52.6.0 52.5.0 52.4.0
device file (/dev/dsk/) c0t6d0 c0t5d0 c0t4d0

The scenario for this chapter is that the disk at hardware address
52.4.0 has a head crash, and as a result it is unusable. The steps
below outline a method that can be used to recover from this state.

[Step 1.1]

Have the engineer replace the faulty disk, and then boot the system in
single user mode. This will ensure that when we are working with the
system, there will be a minimum of processes running on the system.
To boot into single-user mode, boot from the primary boot path, and
interact with IPL. At the ISL> prompt enter this string:

ISL> hpux -is

Note: The above string should work for most systems. If your system
fails to boot using this string follow the steps below.

At the ISL> prompt, determine the original boot string:

ISL> lsautofl

This command will return a string such as:

hpux (52.6.0;0)/stand/vmunix

or

hpux (;0)/stand/vmunix

The output here is dependent on the type of system you have. Once you
have this information, simply add the string -is after the hpux, and
this will boot the system into single user mode. For our example:

ISL> hpux -is (52.6.0;0)/stand/vmunix

[Step 1.2]

Restore the LVM configuration/headers onto the new disk from your backup
of the LVM configuration:

# vgcfgrestore -n [volume group name] /dev/rdsk/cXtYdZ

Where X is the 'card instance number' of the SCSI bus attached to
that card. Y is the 'SCSI ID' of the disk (or array controller, in
the case of an array), and Z is the 'LUN number' (typically 0 for a
non-array type disk). Note that if the HP Customer Engineer replaces
the disk at the same address, the device file name will not change.
In that case the name will be what it was prior to the replacement.
For our example:

# vgcfgrestore -n /dev/vg00 /dev/rdsk/c0t4d0

[Step 1.3]

Reactivate the volume group (VG) so that the new disk can be attached,
since it wasn't configured in at boot time:

# vgchange -a y [volume group name]

For our example, the volume group vg00 will already be activated, but
it will not know of the replaced disk; therefore, this step is still
required so that LVM will now know that the disk is again available:

# vgchange -a y /dev/vg00

The vgchange command will activate each specified volume group and all
associated physical and logical volumes for read-write access. In the case of
vg00,it would initially have been activated with c0t4d0 in an unknown state.
vgchange tells vg00 to look again at c0t4d0, which is now in a known state. It
is important to remember that even though lvol5 and lvol6 are now active they
are void of data.

[Step 1.4]

Determine which logical volumes spanned onto that disk. You only need
to recreate and restore data for the volumes that actually touched that
disk. Other LVs in the volume group are still OK.

# pvdisplay -v /dev/dsk/c0tXd0

will show a listing of all the extents on disk lu X, and to what logical
volume they belong. This listing is fairly long, so you might want to
pipe it to more or send it to a file. For our example:

# pvdisplay -v /dev/dsk/c0t4d0 | more
.....
.....
--- Distribution of physical volume ---
LV Name LE of LV PE for LV
/dev/vg00/lvol5 50 50
/dev/vg00/lvol6 245 245
.....
.....

From this we can see that logical volumes /dev/vg00/lvol5 and
/dev/vg00/lvol6 have physical extents on this disk, but /dev/vg00/lvol1
through /dev/vg00/lvol4 don't, so we will need to recreate and restore
lvol5 and lvol6 only.

Note: Even though lvol5 was also in part on another disk drive, it
will need to be treated as if the entire lvol was lost, not just
the part on c0t4d0.

[Step 1.5]

Restore the data from your backup onto the replacement disk for
the logical volumes identified in step 1.4. For raw volumes, you can
simply restore the full raw volume using the utility that was used to
create your backup. For file systems, you will need to recreate the
file systems first. For our example:

For HFS:

# newfs -F hfs /dev/vg00/rlvol5
# newfs -F hfs /dev/vg00/rlvol6

For JFS:

# newfs -F vxfs /dev/vg00/rlvol5
# newfs -F vxfs /dev/vg00/rlvol6

Note that we use the raw logical volume device file for the newfs
command. For file systems that had non-default configurations, please
consult the man page of newfs for the correct options.

After a file system has been created on the logical volume mount the
file system under the mount point that it previously occupied. Take whatever
steps are necessary to prevent your applications or users from accessing the
filesystem until the data has been recovered. Now that the filesystem has been
created simply restore the data for that file system from backups.

Note: You will need to have recorded how your file systems were
originally created in order to perform this step. The only
critical feature of this step is that the file system be at
least as large as before the disk failure. You can change
other file system parameters, such as those used to tune the
file system's performance.

For the file system case, there is no need to worry about data on the
disk (c0t5d0) that was newer then the data on the tape. The newfs
wiped out all data on the lvol5. For the raw volume access, you may
have to specify your restore utilities over-write option to guarantee
bringing the volume back to a known state.

[Step 1.6]

The final step in the recovery process is to reboot the system. When
the system restarts, the recovery process will be complete.

# cd /
# shutdown -r

If you have any questions or problems with the above steps, please
contact your local Hewlett Packard Customer Response Center.

----------------------------------------------------------------------
CHAPTER 2 : Replacing a NON-Boot disk WITH Mirroring
----------------------------------------------------------------------

Mirroring introduces an interesting twist to the recovery process.
Because LVM keeps a map of 'stale' extents on each disk, it is only
aware of individual extents which are in need of update, and it does
not map this to entire disks. This makes for quick mirror recovery in
the case that a disk has temporarily lost connection with the host, or
has lost power. In addition, it can greatly speed up the recovery time
in the instance of a failed disk.

Example configuration:

Volume group /dev/vg00 contains the three disks, with the logical
volume configuration as shown:

|---------| |---------| |---------|
| lvol 1 | | lvol 4 | | lvol 4 |
|---------| | | | mirror |
| lvol 2 | |---------| |---------|
|---------| | lvol 5 | | lvol 5 |
| lvol 3 | | | | mirror |
|---------| |---------| |---------|

hardware address 52.6.0 52.5.0 52.4.0
device file (/dev/dsk/) c0t6d0 c0t5d0 c0t4d0

This involves a head crash on the disk at address 52.4.0.

[Step 2.1]

Have the customer engineer replace the faulty disk, and then boot the system.
You can boot the system into multi-user mode. For logical volumes that
didn't have a mirror, and that resided on the replaced disk, you will
have to deny access to the filesystem. Do this by unmounting the relevant
filesystems. (However, it's possible that they will not have been mounted as
part of the boot-up sequence anyway.)

Note: If the disk being replaced is Hot-Pluggable (or Hot-Swappable) a
reboot may not be necessary. Please inquire your customer engineer to
determine if a reboot is required.

[Step 2.2]

Restore the LVM configuration/headers onto the new disk from your
backup of the LVM configuration:

# vgcfgrestore -n [volume group name] /dev/rdsk/cXtYdZ

where X is the 'card instance number' of the scsi bus attached to
that card. Y is the 'SCSI ID' of the disk (or array controller, in
the case of an array), and Z is the 'LUN number' (typically 0 for a
non-array type disk). Note that if an HP Customer Engineer replaces
the disk at the same address, the device file name will not change.
In that case the name will be what it was prior to the replacement.
For our example:

# vgcfgrestore -n /dev/vg00 /dev/rdsk/c0t4d0

[Step 2.3]

Reactivate the volume group so that the new disk can be attached,
since it wasn't configured in at boot time. This should also resync
any mirrors that resided on the faulty disk.

# vgchange -a y [volume group name]

For our example, the volume group vg00 will already be activated, but
it will not know of the replaced disk; therefore, this step is still
required so that LVM will now know that the disk is again available
and the resync will occur:

# vgchange -a y /dev/vg00

Now check that the mirrors are syncronized, run this command for each
logical volume that resides on the replaced disk:

# lvdisplay -v /dev/vg00/lvol4 | more
# lvdisplay -v /dev/vg00/lvol5 | more

If any extents show up as being stale it is recommended to run a vgsync
or lvsync to force the syncronization of the mirrored extents using the
following commands:

# vgsync /dev/vg00

or

# lvsync /dev/vg00/lvol4
# lvsync /dev/vg00/lvol5

[Step 2.4]

For any file systems on the faulty disk that didn't have mirror copies,
you will have to rebuild the file systems and restore the data. Follow
the steps 1.4 and 1.5 in chapter 1 for guidance.

[Step 2.5]

If you booted your system into single-user mode in step 2.1, reboot
your system now and allow it to boot into multiuser mode. If you were
already in multi-user mode, then no further action is required.

At this stage, your system should be fully functioning.

-------------------------------------------------------------------------------
-
CHAPTER 3 : Replacing a Root Disk WITHOUT Mirroring
-------------------------------------------------------------------------------
-

With the failure of the boot disk the information on disk that is required to
boot the system is lost. The best method for recovery is to reload the boot
disk
from a make_recovery tape. The make_recovery command is supplied with the
Ignite-UX product available for free at www.software.hp.com.

If a make_recovery tape is unavailable then the system will need to be
installed
with the Core OS/Install cd. The original kernel and operating system
can be recovered from backups after that. ITRC document rcfaxrestore001 has
complete instructions for recovering a system.

----------------------------------------------------------------------
CHAPTER 4 : Replacing a boot disk WITH mirroring
----------------------------------------------------------------------

As with chapter 2, mirroring adds a new twist to the recovery of a boot
disk, and makes the recovery much simpler than without mirroring.

Example configuration:

Volume group /dev/vg00 contains the three disks, with the logical
volume configuration as shown:

|---------| |---------| |---------|
| lvol 1 | | lvol 1 | | lvol 4 |
| | | mirror | | |
|---------| |---------| |---------|
| lvol 2 | | lvol 2 | | lvol 5 |
| | | mirror | | |
|---------| |---------| | |
| lvol 3 | | lvol 3 | | |
| | | mirror | | |
|---------| |---------| |---------|

hardware address 52.6.0 52.5.0 52.4.0
device file (/dev/dsk/) c0t6d0 c0t5d0 c0t4d0

This scenario involves a head crash of the disk at hardware address
52.6.0.

[Step 4.1]

Shutdown the system, have the customer engineer replace the faulty
disk, and then boot the system in single-user mode from the alternate
boot disk. If you only have two disks in the root volume group, then
you will need to override quorum as you boot. Note that you don't have
to boot in single-user mode if all of the disk data is mirrored
elsewhere (but you may still have to override quorum).

Note: If the disk being replaced is Hot-Pluggable (or Hot-Swappable) a
reboot may not be necessary. Please inquire your customer engineer to
determine if a reboot is required. If a reboot is not required replace the
disk and go on to step 4.2.

For our example:

ISL> hpux -is (52.5.0;0)/stand/vmunix

OR

ISL> hpux -is -lq (52.5.0;0)/stand/vmunix
(if you need to override quorum.)

[Step 4.2]

Restore the LVM configuration/headers onto the replaced disk from
your backup of the LVM configuration, activate the volume group
with the knowledge of the replaced disk, use mkboot to add boot
information, and syncronize the BDRA with these commands:

# vgcfgrestore -n /dev/vg00 /dev/rdsk/c0t6d0
# vgchange -a y /dev/vg00
# mount /usr
# mkboot /dev/rdsk/c0t6d0
# mkboot -a "hpux (;0)/stand/vmunix" /dev/rdsk/c0t6d0
# lvlnboot -R

Note: If /usr doesn't reside in /dev/vg00 the volume group that does
contain /usr's logical volume will also have to be activated.

At this point it is a good idea to verify that the BDRA information is
accurate. This can be done with the lvlnboot comman using the -v
option. Here is an example of the output:

# lvlnboot -v
Boot Definitions for Volume Group /dev/vg00:
Physical Volumes belonging in Root Volume Group:
/dev/dsk/c0t6d0 (52.6.0) -- Boot Disk
/dev/dsk/c0t5d0 (52.5.0) -- Boot Disk
/dev/dsk/c0t4d0 (52.4.0)
Boot: lvol1 on: /dev/dsk/c0t6d0
/dev/dsk/c0t5d0
Root: lvol3 on: /dev/dsk/c0t6d0
/dev/dsk/c0t5d0
Swap: lvol2 on: /dev/dsk/c0t6d0
/dev/dsk/c0t5d0
Dump: lvol2 on: /dev/dsk/c0t6d0, 0

Note: You must have performed the command vgcfgbackup to save off
the headers prior to the disk failure (refer to Appendix A.).

[Step 4.3]

Now, synchronize mirrored logical volumes:

# vgsync /dev/vg00

[Step 4.4]

If you have any logical volumes that resided on the faulty disk that
were not mirrored, you will need to recreate them as per steps 1.4
and 1.5 from chapter 1.

[Step 4.5]

The final step in the recovery process is to reboot the system. When
the system restarts, the recovery process will be complete.

# cd /
# shutdown -r

At this stage, your system should be fully recovered. It is a good idea to
verify the system can be booted from the newly replaced drive.

-----------------------------------------------------------------------
APPENDIX A: Preparing for the Recovery of an LVM
System
-----------------------------------------------------------------------

These are procedures to ensure that the system's data and
configuration are recoverable in the event of a system failure.

1. Load any patches for LVM.

2. Regularly back up your entire system.

Without a valid backup, you run a real chance of losing some or
all of your data. Ensure that you back up ALL of your important
data, including the operating system directories such as:

/
/usr
/dev
/etc
...and so on.

In addition, regularly test that your backups are working by
restoring a test file randomly from your backups. It is risky
to assume that your backup is working because it is not logging
any errors. Many backup utilities have the capability to do
some sort of validation of the backup media. For example,
fbackup has the -N option that can allow you to check for
discrepancies between backup indices and what is actually on
the tape. Refer to the fbackup(1M) man page for more information.

Use make_recovery to create a bootable recovery media.

3. Back up the important files separately.

Take an extra copy of the very important files, preferably to
another system as well as to another tape. This will speed up
recovery in the event of a system crash. The files that should
be backed up are:

/etc/passwd
/etc/group
/etc/lvmtab
/etc/lvmconf/*
/etc/fstab

There are many other important files on your system that you
may wish to back up separately. The files listed above are
required to ensure a smooth system recovery.

4. Regularly print out the configuration of your system.

The configuration details stored on the system may not be
accessible during a recovery. A printed copy is an invaluable
reference. We recommend printing the configuration details
once a week and every time a change is made. One thing to note
is that some of the commands outlined below create large amounts
of output. An alternative to printing them is to output the
information to a file and then storing the file off to tape.
This allows quick recovery of the information when needed. You
could include this configuration file with the backup in step 3.

The easiest way to save the configuration is to set up a cron
job to run regularly, so that even if you don't remember to do
it, the system will.

The output from the following commands is recommended:

/usr/sbin/ioscan -fk

/usr/sbin/vgdisplay -v

/usr/sbin/lvlnboot -v

/usr/sbin/lvdisplay -v /dev/vgXX/lvYY (for every logical volume)

/usr/sbin/pvdisplay -v /dev/dsk/c#t#d0 (for every LVM disk)

lp /etc/fstab

As an alternative, an intelligent script can be written that
will detect any changes in the configuration and only print out
those changes. An example script is included at the end of this
appendix.

5. Update the boot structures after every change to the root volume
group.

This task is only required if you are using LVM on your boot
disk. Whenever you make changes to the root volume group,
which is usually named /dev/vg00, you must update the Boot
Disk Reserved Area (BDRA) on the boot disk. To do this issue
the following command:

/etc/lvlnboot -R

----------------------------------------------------------------

Example script for LVM configuration recording
**********************************************
NOTE: In some cases, this script may give an error when it runs the
first time. If so, just rerun the script and it should run fine.

#!/usr/bin/ksh

WORKDIR=/lvmbackup # directory is regularly backed up, of course
LOG=$WORKDIR/log
SYSADM=root

if [ -f "$LOG" ]
then
rm -f "$LOG"
fi

if [ ! -d "$WORKDIR" ]
then
echo "missing directory $WORKDIR"
exit 1
fi

cd $WORKDIR

/usr/sbin/vgdisplay -v > vgdisplay.new

LVMVGS=`grep "VG Name" vgdisplay.new | awk '{ print $3 }'`
LVMPVOLS=`grep "PV Name" vgdisplay.new | sort -u | awk '{ print $3 }'`
LVMLVOLS=`grep "LV Name" vgdisplay.new | sort -u | awk '{ print $3 }'`

/usr/sbin/pvdisplay -v $LVMPVOLS > pvdisplay.new

/usr/sbin/lvdisplay -v $LVMLVOLS > lvdisplay.new

/usr/sbin/lvlnboot -v > lvlnboot.new 2> /dev/null

/usr/sbin/ioscan -fk > ioscan.new

cp /etc/fstab fstab.new

for CURRENT in *new
do
ORIG=${CURRENT%.new}

if diff $CURRENT $ORIG > /dev/null
then
# files are the same....do nothing
rm $CURRENT
else
# files differ...make the new file the current file, move old
# one to file.old

echo `date` "The config for $ORIG has
changed." >> $LOG
echo "Copy of the new $ORIG config has
been printed" >> $LOG

lp $CURRENT
mv $ORIG ${ORIG}old
mv $CURRENT $ORIG
fi
done

if [ -s "$LOG" ]
then
mailx -s "LVM configs have changed" $SYSADM < $LOG
fi

exit 0

topic Re: problems with disk in Operating System - HP-UX

problems with disk

Re: problems with disk

Re: problems with disk

Re: problems with disk

Re: problems with disk

Re: problems with disk

Re: problems with disk

Re: problems with disk

Re: problems with disk

Re: problems with disk