MIRROR ???

[Guido Herrera]

HI
IM HAVE A HP UX 11.0 AND A REPALCE DISK FAILED BUT WHEN REMOVE DISK FROM LOGICAL VOLUME PRESENTE THE NEXT.
vgcfgbackup: /etc/lvmtab is out of date with the running kernel:Kernel indicates 3 disks for "/dev/vg04"; /etc/lvmtab has 2 lv³ disks. Cannot proceed with backup.
WHEN IM MAKE LVDISPLAY SEE THE FOLLOWING.


--- Logical volumes ---
LV Name /dev/vg04/lvol1
VG Name /dev/vg04
LV Permission read/write
LV Status available/stale
Mirror copies 2
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 20000
Current LE 5000
Allocated PE 15000
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c6t11d0 5000 5000
/dev/dsk/c4t11d0 5000 5000
LE PV1 PE1 Status 1 PV2 PE2 Status 2 PV3
PE3 Status 3
0000 ??? 0000 stale /dev/dsk/c6t11d0 0000 current /dev/d
sk/c4t11d0 0000 current
0001 ??? 0001 stale /dev/dsk/c6t11d0 0001 current /dev/d
sk/c4t11d0 0001 current
0002 ??? 0002 stale /dev/dsk/c6t11d0 0002 current /dev/d
sk/c4t11d0 0002 current
0003 ??? 0003 stale /dev/dsk/c6t11d0 0003 current /dev/d
sk/c4t11d0 0003 current
0004 ??? 0004 stale /dev/dsk/c6t11d0 0004 current /dev/d
sk/c4t11d0 0004 current
0005 ??? 0005 stale /dev/dsk/c6t11d0 0005 current /dev/d
sk/c4t11d0 0005 current
0006 ??? 0006 stale /dev/dsk/c6t11d0 0006 current /dev/d
sk/c4t11d0 0006 current
0007 ??? 0007 stale /dev/dsk/c6t11d0 0007 current /dev/d
sk/c4t11d0 0007 current
0008 ??? 0008 stale /dev/dsk/c6t11d0 0008 current /dev/d
sk/c4t11d0 0008 current
0009 ??? 0009 stale /dev/dsk/c6t11d0 0009 current /dev/d
sk/c4t11d0 0009 current

how i can eliminate this "???"

Please help me, thank you

[Steven E. Protter]

Shalom,

You need to reintegrate this disk.

mv /etc/lvmtab /etc/lvmtab.old

vgscan -a

pvcreate the volume.

You will need to rebuild your mirrors as well.

Here is a general guide.

---guide---
pvcreate -B /dev/rdsk/c1t0d0 #use real disk

mkboot -l /dev/rdsk/c1t0d0
mkboot -a "hpux -lq (;0)/stand/vmunix" /dev/rdsk/c1t0d0 # use real disk


# mkboot -b /usr/sbin/diag/lif/updatediaglif -p ISL -p AUTO -p HPUX -p PAD -p LABEL /dev/rdsk/c?t?d?

If you are running 64-bit OS:

# mkboot -b /usr/sbin/diag/lif/updatediaglif2 -p ISL -p AUTO -p HPUX -p PAD -p LABEL /dev/rdsk/c?t?d?


vgextend /dev/vg00 /dev/dsk/c1t0d0 # same thing
lvextend -m 1 /dev/vg00/lvol1 /dev/dsk/c1t0d0

# real disk. repeat for other lvols

lvlnboot -r /dev/vg00/lvol3 # root fs /
lvlnboot -s /dev/vg00/lvol2 #swap
lvlnboot -d /dev/vg00/lvol2 #swap/dump
lvlnboot -b /dev/vg00/lvol1
lvlnboot -R
lvlnboot -v
setboot
setboot -a 52.1.0 # second disk
---end guid---

SEP

[Torsten.]

What steps did you?

Is the new disk already in place?

The message

"Kernel indicates 3 disks for "/dev/vg04"; /etc/lvmtab has 2 disks"

looks like you removed the old disk, inserted the new and vgextend it instead of just vgcfgrestore the configuration to it. Is this true?

In this case "vgreduce -f vg04" will help you.

But please confirm first and in case of doubt post again.

[Steven E. Protter]

More details would indeed be helfpul.

Following my advice will probably work but is risky without a broader understanding of your situation.

SEP

[nanan]

You might do such as Torsten's guess.
My thought is that you need to rebuild the mirror VG.
you may not be able to remove the ghost device.

It seems to be related with lvm cumulative Patch.

good luck!

[Prashanth.D.S]

Hi Guido,

Please follow the below mentioned lenghty action plan to resolve this issue.... :-)

The above error indicates a serious problem with the volume group. No changes
should be made to the volume group configuration prior to repairing the volume
group.

This error indicates that vgdisplay(1m) information Cur PV and Act PV
disagree. Cur PV and Act PV should always agree for the volume group. This
error also indicates that the /etc/lvmtab file, which is used to match which
physical volumes belong to a volume group, is out of date with the LVM data
structures in memory and on disk. vgcfgbackup(1m) cannot complete
successfully whenever the number of current physical volumes disagrees with the
number of active physical volumes. Modifying the volume group while in this
state could cause the vgcfgbackup(1m) backup of the volume group to be
inconsistent with the volume group itself and resulting in a more difficult
repair/recovery process.

Each physical volume of each a volume group has a counter indicating the number
of physical volumes currently within the volume group. This information is
contained within the disk's volume group reserve area (VGRA). The above error
indicates the information within the VGRA shows a different number of physical
volumes than the system currently sees attached to this volume group. At
volume
group activation time the /etc/lvmtab file is used by the system to know what
physical volumes belong to each volume group.

This document will explain what to look at and how to repair this situation.
Use the following steps to isolate and repair the problem:

1. Try to locate missing disk device(s).

Isolating what happened to the volume group to get it into this state can
be very difficult. Here are some suggestions:

a. Use the command strings /etc/lvmtab or
vgdisplay -v /dev/vg_name to see what disk devices are
currently attached to the volume group.

b. Check the date of and physical volumes contained in the last good
vgcfgbackup.

Use: ll /etc/lvmconf/VG_NAME.conf to see the date of the last
good backup.

NOTE: If the volume group has been modified since the time of the
last good vgcfgbackup, then there is the potential that the backup file
is out of date with the LVM data structures on the disk(s) attached to
this volume group. If this is the case then vgcfgrestore(1m)
may no longer work for this volume group.

Use: vgcfgrestore -n /dev/VG_NAME -l to see the list of
physical volumes contained within the last good backup.

Use the list from the above vgcfgrestore(1m) command to compare
to the list from step 1a to see if there are differences. If there is
a physical volume listed in the backup listing that is not in the
/etc/lvmtab file, you may be able to vgcfgrestore(1m) to that
physical volume. Make sure the physical volume is unused before
overwriting with vgcfgrestore. See step 2 for details.

c. Check the system for old copies of /etc/lvmtab file.

A common reason for the system to be in this state is that the lvmtab
has been recreated while unable to communicate with one or more of the
physical volumes belonging the the volume group. If the lvmtab is
recreated while the system is unable to query a physical volume, that
physical volume will not be added to the lvmtab file. One can see how
this can cause the lvmtab to mismatched with kernel memory.

To check for old copies of /etc/lvmtab using the following:
ll /etc/lvmtab* or ll /tmp/lvmt*

Use strings(1)on the backup lvmtab file to try to determine
which disk device(s) in the volume group differ in comparison
with the current lvmtab file.


If the missing physical volume cannot be determined or the missing physical
volume cannot be readded to the problem volume group then skip to Step
3.
Reasons the physical volume could not be readded might be it has been added
to another volume group and cannot be removed or is no longer physically
connected to this system.

2. If possible, restore the missing physical volume into the volume group.

a. Verify the missing physical volume and it's alternate path(s), if
necessary, are not in use.

Use strings /etc/lvmtab to verify that the physical volume and
any of it's alternate paths do not belong to any other volume groups or
are not mounted or in use by applications on the system. A common
error that can lead to this type of problem is when an alternate path
is added to a different volume group than the primary path. Consult
your disk device's manuals to determine if it is capable of using
pvlinks or alternate paths. The system will not allow an alternate
link to a different volume group unless pvcreate -f
is executed. pvcreate(1m) is not needed to add an alternate
path to a volume group and should be only run on the primary path.

b. If missing physical volume(s) and alternate path(s) are not in use then
use vgcfgrestore(1m) to restore the physical volume(s) to the
volume group.

Example: vgcfgrestore -n /dev/vg03 /dev/rdsk/c2t7d2

c. If needed restore or recreate /etc/lvmtab.

If the /etc/lvmtab does not contain the physical volume(s) that were
vgcfgrestored to, then this file must be updated. If the lvmtab shows
the correct physical volumes then skip this step.

If the /etc/lvmtab does not show the correct physical volumes and you
were able to find an old /etc/lvmtab file in the previous step then save
the current version of /etc/lvmtab and copy the old lvmtab backup file
into place. Use the strings(1) command to insure all volume
groups show the correct physical volume before changing the lvmtab file.

If the /etc/lvmtab file is not correct and there is no old copy of
/etc/lvmtab that is correct use vgexport(1m) and vgimport(1m)
to correct the lvmtab.

NOTE: For the root volume group, typically vg00, you must first
boot into lvm maintenance mode. See below for details


Overview:
1. vgchange -a n /dev/vg_name
2. vgexport -m /tmp/mapfile /dev/vg_name
3. mkdir /dev/vg_name
4. mknod /dev/vg_name/group c 64 0x0X0000

NOTE: The minor number (0x0X0000) must be unique for each volume
group. Substitute X for a number not in use on the system.
Use: ll /dev/*/group to see existing group files on the
system.

Example: mknod /dev/vg01/group c 64 0x010000

5. vgimport -m /tmp/mapfile /dev/vg_name /dev/dsk/pv_name

NOTE: The above commmand requires each of each physical volume in
volume group to be specified at the end of the command. This
allows the lvmtab to be correctly rebuilt with all physical
volumes belonging to the volume group.

d. Activate the volume group.

Use: vgchange -a y /dev/vg_name to activate the volume group
after the restoring any missing physical volumes. If all was completed
correctly then vgdisplay /dev/vg_name should show Cur PV and Act
PV now agree.

e. Get a backup of the volume group.

Use vgcfgbackup /dev/vg_name to insure there is a good
volume group backup now that Cur PV and Act PV agree.

3. If you cannot locate the missing disk device or cannot restore that
device back into the volume group, then use vgreduce(1m) to forcibly
reduce out the missing physical volume.

NOTE: vgreduce -f should be used as a last resort. If
vgcfgrestore(1m) cannot be used to make the Cur PV and Act PV agree,
then vgreduce -f may be required. Here are the steps to successfully
use vgreduce -f:

a. Get a list of logical volumes belonging to the volume group.

Use: vgdisplay -v /dev/vg_name to get a list of logical volumes
for the volume group.

b. Find out which logical volume(s) reside on the disk device(s) to be
forcibly reduced.

Use lvdisplay -v /dev/vg_name/lv_name | more to see if any
of the logical volumes extents show ??? in the PV section. Page
through every logical extent for each logical volume in the volume
group. ??? indicate that the extents shown reside on a physical
volume that the system is unable to query. Any logical volume with
??? will have to be removed using lvremove(1m) in order for
vgreduce -f to complete successfully.

c. Remove logical volumes with ??? in their lvdisplay(1m) output.

Since logical volumes that show ??? have missing or unavailable data
they will have to removed. In order for vgreduce -f to succeed
all logical volumes with extents on the physical volume to be reduced
must first be removed. Once the volume group is in the correct state,
Cur PV = Act PV, the logical volumes can be recreated and any lost data
restored.

Use: lvremove /dev/vg_name/lvol_name

d. Forcibly reduce out the physical volume.

Use: vgreduce -f /dev/vg_name

NOTE: The above command does not require a physical volume argument. It
must be run on a active volume group.


e. If the vgreduce -f command does not work or does not give any
error and vgdisplay(1m) still shows that Cur PV and Act PV
disagree then use the following steps to vgexport and vgimport the
volume group prior to trying Step 3d again.

This procedure can be used when vgreduce fails to reduce a physical
volume that can no longer be queried by the system. If executing the
following procedure on the root volume group, usually vg00, you must
first boot into LVM maintenance mode (** For steps see below).

1. Get the /dev/vg_name/group minor number and physical volumes
belonging to the volume group.

Use: ll /dev/vg00/group to get 0x###### minor number.
vgdisplay -v /dev/vg_name to get physical volumes.

2. vgchange -a n /dev/vg_name
NOTE: Skip this step if booting maintanence mode for root volume
group.

3. vgexport -m /mapfile /dev/vg_name

4. mkdir /dev/vg_name

5. mknod /dev/vg_name/group c 64 0x0#0000
Re-use minor number obtained from step 1.

6. vgimport -m /mapfile /dev/vg_name pv_name [pv_name ...]

NOTE: Specify all the physical volumes obtained from step 1. Do not
include the physical volume that you are trying to remove or
that couldn't be queried.


** Steps to boot into maintenance mode and active. :
1. shutdown -hy now
2. interrupt boot sequence
3. boot from primary boot path and interact with ISL

NOTE: Procedure used for steps 2 and 3 may very slightly depending
on machine model.

4. enter the following at the IPL> prompt:
IPL> hpux -lm (;0)/stand/vmunix


f. Retry the vgreduce -f command specified in step 3d.

This time the vgreduce should succeed and give you a message
similar to: "PV with key # sucessfully deleted from vg /dev/vg_name".
It should also display:

Repair done, please do the following steps.....:
1. save /etc/lvmtab to another file
2. remove /etc/lvmtab
3. use vgscan(1m) -v to recreate /etc/lvmtab
4. NOW use vgcfgbackup(1m) to save the LVM setup

Follow the above 4 steps.

vgdisplay /dev/vg_name should now show Cur PV and Act PV
agree.

Best Regards,
Prashanth

[Mridul Shrivastava]

Hi,

Since you are not getting the disk details in pvdisplay I would recommend you to get key value for that disk and then use lvreduce using that key value and at last do vgreduce. The procedure for doing the same is given below:

# lvdispay -kv /dev/vg04/

This will show you some numeric value ( 0,1 or 2) for the first disk showing as ???. Make a note of that then execute the following command.

# lvreduce -k -m 1 /dev/vg04/

this will remove the "???" information from lv.Then execute the following command:

#vgreduce -f /dev/vg04

This will remove all the pv's which are not listed in lvmtab for this vg. After doing this you need to recreate the mirror copies.
Someone have destroyes the header information and which has resulted in this issue.