Unable to backup VG (vgcfgbackup fails)

[Ra Jose]

I have a VG that has 2 physical disks. I can see these from vgdisplay -v output. But it is showing Cur PV = 3 and Act PV = 2. This shows that the VG had 3 PVs but only 2 are onlined.

This vg has 2 LVs. One LV is good. Problem is with other LV. This second LV shows some of its initial extents on PV as ??? and then shows all other extents correctly. Looks like the initial extents that this lvol is looking is the third PV that the vgdisplay showed in Cur PV but has only 2 disks. May be the initial extents of this second lvol maps tho this 3rd disk. I am unable to find which this 3rd disk is.

When I do, vgcfgbackup, I get
vgcfgbackup: /etc/lvmtab is out of date with the running kernel:Kernel indicates 5 disks for "/dev/vgPTRTB"; /etc/lvmtab has 4 disks.

This vg has 2 physical disks (+ 2 alt paths), so totally 4 disks as per vgcfgbackup. I am wondering how to find this 5th disk.

I read all the threads on this (somewhat sounds like ghost device or phantom device), tried all possible ways vgscan -v, but cannot get it to vgcfgbackup.

This is HP-UX 11iv1 (B.11.11) on rp7420.

Thank you all for your replies.

Rajos

[Sandman!]

vgreduce the non-existent disk from/dev/ vgPTRTB; save the /etc/lvmtab file and generate a fresh one with "vgscan -av".

[James R. Ferguson]

Hi:

You certainly don't want to take a backup of an LVM configuration that isn't consistent, and that's exactly what 'vgcfgbackup' isn't letting you do.

It is probable that you have a failed disk or failed connection to a disk. An 'ioscan -fnC disk' would be helpful. Look for a "NO HW" as a start.

Regards!

...JRF...

[Mridul Shrivastava · Accepted solution]

Execute the following commands for the affected LV:

lvdisplay -v /dev/vgxx/
lvdisplay -v -k /dev/vgxx/

Then compare the outputs.. you will get a key value of the disk for which it is showing ???.. then you can use the following command to remove the disk from the lv

lvreduce -k -m 0 /dev/vgxx/

disk is showing as ??? because it is failed or missing so this the only way out to remove the disk from lv/vg... once this is done execute the following command.

vgreduce -f

This will remove all the disk which are not present in /etc/lvmtab for this Vg.. once lv is reduced..

-k This option should be used only in the
special instance when you want to reduce a
mirrored logical volume on a physical volume
that is missing or has failed.

The vgreduce command with -f option removes all missing physical volume from the volume group.

[Hareesh Kumar]

Hi Rjos,

Please go through the HP knowledge base document KBRC00000632. This document can help you in resolving the issue.

** edited to comply with forum guidelines **

[Ra Jose]

Thank you all. All the options you suggested, I have tried. Please note that the LVs in this VG is not mirrored. So if I use lvdisplay -k -v, this shows me the keys that have ??? in PV column. But I cannot use lvreduce -m 0 -k option as this is not mirrored. The cmd gave error message that the lv is already mirror=0.

ioscan -fnC disk shows no NO_HW status. I am wondering how to find this third PV that this VG is looking for.

Just to see, if I can search all my disks using vgid, I did a vgexport using -s option.
This is a stand-alone server and does not use MCSG, but just for heck, I tried with vgexport -s option. Then removed the vg from lvmtab file using vgexport -v option. Re-created the VG, group file and then did vgimport with -s option, to force lvm to scan all disks for vgid. This exercise did not work.

I either need to remove that offending device from the kernel (i don't know how), or find the device (which one I don't know) and add it to the VG.

I rebooted, it did not help.

Rajos

[Torsten.]

How about vgreduce -f?

See

http://docs.hp.com/en/B2355-60105/vgreduce.1M.html

[Ra Jose]

I tried
vgreduce -f also. It asked me to try lvreduce with -m option and lvreduce. The lvreduce -m opton did not work.

I have given below o/p of vgreduce -f (the o/p truncated to save space, actual VG not given due to security).

# vgreduce -f volume_group
skip alternate link /dev/dsk/c14t9d5
skip alternate link /dev/dsk/c14t9d7
Not all extents are free. i.e. Out of 538 PEs, only 0 are free.
You must free all PEs using lvreduce/lvremove before the PV can be removed.
Example: lvreduce -A n -m 0 /dev/vg01/lvol1.
lvremove -A n /dev/vg01/lvol1
Here's the map of used PEs

--- Logical extents ---
LE LV PE Status 1
0000 lvol1 0000 ???
0001 lvol1 0001 ???
0002 lvol1 0002 ???
0003 lvol1 0003 ???
0004 lvol1 0004 ???

The lvreduce o/p

# lvreduce -A n -m 0 /dev/volume_group/lvol1
lvreduce: "MirrorCopies" parameter "0" is not smaller than existing number "0";
therefore no mirrors are removed.

Rajos

[Hareesh Kumar]

Hi Rajos,

Keep a copy of /etc/lvmtab file and then procedd with the commands

1. Get the /dev/vg_name/group minor number and physical volumes belonging to the volume group.
Use: ll /dev/vg00/group to get 0x###### minor number. vgdisplay -v /dev/vg_name to get physical volumes.
2. vgchange -a n /dev/vgPTRTB
3. vgexport -m /tmp/mapfile /dev/vgPTRTB
4. mkdir /dev/vgPTRTB
5. mknod /dev/vgPTRTB/group c 64

Example: mknod /dev/vg01/group c 64 0x010000

6. vgimport -m /tmp/mapfile /dev/vgPTRTB /dev/dsk/pv1 /dev/dsk/pv2

NOTE: The above commmand allows the lvmtab to be correctly rebuilt with all physical volumes belonging to the volume group. Specify all the physical volumes obtained from step 1. Do not include the physical volume that you are trying to remove or that couldn't be queried.

7. vgchange -a y /dev/vgPTRTB
8. vgcfgbackup /dev/vgPTRTB

If you still have issues ..!!!

a. Get a list of logical volumes belonging to the volume group.

Use: vgdisplay -v /dev/vgPTRTB to get a list of logical volumes for the volume group.

b. Find out which logical volume(s) reside on the disk device(s) to be forcibly reduced.

Use lvdisplay -v /dev/vgPTRTB/lv_name | more to see if any of the logical volumes extents show ??? in the PV section. Page through every logical extent for each logical volume in the volume group. ??? indicate that the extents shown reside on a physical volume that the system is unable to query. Any logical volume with ??? will have to be removed using lvremove in order for
vgreduce -f to complete successfully.

c. Remove logical volumes with ??? in their lvdisplay output.

Since logical volumes that show ??? have missing or unavailable data they will have to removed. In order for vgreduce -f to succeed all logical volumes with extents on the physical volume to be reduced must first be removed. Once the volume group is in the correct state, Cur PV = Act PV, the logical volumes can be recreated and any lost data restored.

Use: lvremove /dev/vgPTRTB/lvol_name

d. Forcibly reduce out the physical volume.

Use: vgreduce -f /dev/vgPTRTB

NOTE: The above command does not require a physical volume argument. It must be run on a active volume group.

e. If the vgreduce -f command does not work or does not give any error and vgdisplay still shows that Cur PV and Act PV
disagree then use the following steps to vgexport and vgimport the volume group prior to trying complete step again.

[Ra Jose]

Hareesh,

I did all these steps you mentioned. The one problem I have is lvremove
This lv which has ??? is comprised of more disks. If I do lvdisplay -v lvol
It shows me first 538 logical extennts on ??? and then from 539 onwards until end (upto 1076 logical extents which this lvol has),logical extents are good and show correct physical device.

The problem is with the first 538 logical extents showing up on some physical device which I can't find.

If I do a lvremove, it is going to wipe out all of this.

Rajos

[Ra Jose]

This is what I learnt after going thru all the methods of repairing this one logical volume that has some extents showing pv status as ??? and other extents showing correct pv device.

There was no option but to remove the lvol and recreate from scratch. We were able to restore the data in this lvol from tape (although little old).

Thanks to many who suggested vgreduce -f,
but this works only if lvols that have ??? are removed from the VG. In this case, I had a hybrid lvol (some ??? and others good). Nobody knows on which physical devices those ??? existed.

Thanks to all of you for your suggestions.
Rajos