How to track a missing vg

[Nancy Calderon]

Hello team,
Yesterday I was trying to extend a fs, with no good results, today I can't see that fs present on the OS, not even the vg.

Is there any way to track if somebody delete it?

Nancy

Yesterday:
#vgdisplay -v /dev/vgapp06

--- Volume groups ---
VG Name /dev/vgapp06
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 3
Open LV 3
Max PV 16
Cur PV 5
Act PV 5
Max PE per PV 6475
VGDA 10
PE Size (Mbytes) 32
Total PE 2965
Alloc PE 2783
Free PE 182
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vgapp06/lvol1
LV Status available/syncd
LV Size (Mbytes) 57024
Current LE 1782
Allocated PE 1782
Used PV 3

# lvdisplay -v /dev/vgapp06/lvol3| more
--- Logical volumes ---
LV Name /dev/vgapp06/lvol3
VG Name /dev/vgapp06
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 2016
Current LE 63
Allocated PE 63
Stripes 0
Stripe Size (Kbytes) 0
Bad block NONE
Allocation PVG-strict/distributed
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c105t3d5 32 32
/dev/dsk/c105t3d6 31 31



Today:
# bdf /apps/pedi
bdf: /apps/pedi: No such file or directory

# lvdisplay /dev/vgapps06/lvol3
lvdisplay: "/dev/vgapps06/lvol3": No such file or directory
Please find attached more vg , lv and disks info

[Torsten. · Accepted solution]

Hi Nancy,

the allocation policy was distributed, so you need space on all 3 physical volumes in order to extend it.

Some of your PVs have no free extends.

vgapp06 is not listed in /etc/lvmtab.

Do you still see the disk device files?
Any result if you run "vgscan -v -p"?

[James R. Ferguson]

Hi:

Do you have a volume group named 'vgapps06'? Is it activated?

Does:

# ls -l /dev/vgapps06

...show device files?

Did someone 'vgexport' the volume group?

Regards!

...JRF...

[Torsten.]

Search the syslog file or (if rebooted last night) the OLDsyslog for related messages.

LVM related messages are stored there.

[Eric SAUBIGNAC]

Bonjour Nancy,

The LV you try to extend is PVG-strict/distributed. It means that extends are distributed over 2 or more disks (sometimes done for better performances). So you must have enough free space on all disks that compose the LV.

Some solutions :

- change pvg strict distributed allocation to non distributed with lvchange
- you have 5 disks in your VG. For exemple, maybe you have 3 disks with enough space to extend the LV and the two other ones are almost full. In this case modify (with vi) /etc/lvmpvg to restrict the PVG to the 3 disks.
- take a look at how the PVGs are constructed (/etc/lvmpvg), take a look at free space on the disks that compose the PVGs then restrict size of extension to the lowest free capacity x number of disk in PVG

Eric

[Nancy Calderon]

Tks for your quick response:

# vgchange -a y /dev/vgapp06
vgchange: Volume group "/dev/vgapp06" does not exist in the "/etc/lvmtab" file.
# ioscan -fnCdisk | grep /dev/dsk/c105t3d5
# ioscan -fnCdisk | grep /dev/dsk/c105t3d6
# ioscan -fnCdisk | grep /dev/dsk/c105t7d3

No track of the disks

[Nancy Calderon]

# vgscan -v -p
vgscan: The physical volume "/dev/dsk/c0t6d0" is already recorded in the "/etc/l
vmtab" file.
vgscan: The physical volume "/dev/dsk/c2t6d0" is already recorded in the "/etc/l
vmtab" file.
vgscan: The physical volume "/dev/dsk/c32t0d4" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t0d5" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t0d6" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t0d7" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t1d0" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t1d1" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t1d2" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t1d3" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t1d4" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t1d5" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t1d6" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t1d7" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t2d0" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t2d1" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t2d2" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t2d3" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t2d4" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t2d5" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t2d6" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t2d7" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t3d0" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t3d1" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t3d2" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t3d3" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c32t3d4" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t0d4" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t0d5" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t0d6" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t0d7" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t1d0" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t1d1" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t1d2" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t1d3" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t1d4" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t1d5" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t1d6" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t1d7" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t2d0" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t2d1" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t2d2" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t2d3" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t2d4" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t2d5" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t2d6" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t2d7" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t3d0" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t3d1" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t3d2" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t3d3" is already recorded in the "/etc/
lvmtab" file.
vgscan: The physical volume "/dev/dsk/c35t3d4" is already recorded in the "/etc/
lvmtab" file.
Current path "/dev/dsk/c35t0d4" is an alternate link, skip.
Current path "/dev/dsk/c35t0d5" is an alternate link, skip.
Current path "/dev/dsk/c35t0d6" is an alternate link, skip.
Current path "/dev/dsk/c35t0d7" is an alternate link, skip.
Current path "/dev/dsk/c35t1d0" is an alternate link, skip.
Current path "/dev/dsk/c35t1d1" is an alternate link, skip.
Current path "/dev/dsk/c35t1d2" is an alternate link, skip.
Current path "/dev/dsk/c35t1d3" is an alternate link, skip.
Current path "/dev/dsk/c35t1d4" is an alternate link, skip.
Current path "/dev/dsk/c35t1d5" is an alternate link, skip.
Current path "/dev/dsk/c35t1d6" is an alternate link, skip.
Current path "/dev/dsk/c35t1d7" is an alternate link, skip.
Current path "/dev/dsk/c35t2d0" is an alternate link, skip.
Current path "/dev/dsk/c35t2d1" is an alternate link, skip.
Current path "/dev/dsk/c35t2d2" is an alternate link, skip.
Current path "/dev/dsk/c35t2d3" is an alternate link, skip.
Current path "/dev/dsk/c35t2d4" is an alternate link, skip.
Current path "/dev/dsk/c35t2d5" is an alternate link, skip.
Current path "/dev/dsk/c35t2d6" is an alternate link, skip.
Current path "/dev/dsk/c35t2d7" is an alternate link, skip.
Current path "/dev/dsk/c35t3d0" is an alternate link, skip.
Current path "/dev/dsk/c35t3d1" is an alternate link, skip.
Current path "/dev/dsk/c35t3d2" is an alternate link, skip.
Current path "/dev/dsk/c35t3d3" is an alternate link, skip.
Current path "/dev/dsk/c35t3d4" is an alternate link, skip.
Couldn't stat physical volume "/dev/dsk/c1t2d0":
Invalid argument
Physical Volume "/dev/dsk/c32t0d1" contains no LVM information
Physical Volume "/dev/dsk/c32t0d2" contains no LVM information
Physical Volume "/dev/dsk/c32t0d3" contains no LVM information
Physical Volume "/dev/dsk/c35t0d1" contains no LVM information
Physical Volume "/dev/dsk/c35t0d2" contains no LVM information
Physical Volume "/dev/dsk/c35t0d3" contains no LVM information


/dev/vg00
/dev/dsk/c0t6d0
/dev/dsk/c2t6d0



/dev/vgappsedi01
/dev/dsk/c32t0d4
/dev/dsk/c35t0d4



/dev/vg01
/dev/dsk/c32t0d5
/dev/dsk/c32t0d6
/dev/dsk/c32t0d7
/dev/dsk/c32t1d0
/dev/dsk/c32t1d1
/dev/dsk/c32t3d3
/dev/dsk/c32t3d4
/dev/dsk/c35t0d5
/dev/dsk/c35t0d6
/dev/dsk/c35t0d7
/dev/dsk/c35t1d0
/dev/dsk/c35t1d1
/dev/dsk/c35t3d3
/dev/dsk/c35t3d4



/dev/vg02
/dev/dsk/c32t1d2
/dev/dsk/c32t1d3
/dev/dsk/c32t1d4
/dev/dsk/c32t1d5
/dev/dsk/c32t1d6
/dev/dsk/c32t1d7
/dev/dsk/c32t2d0
/dev/dsk/c32t2d1
/dev/dsk/c32t2d2
/dev/dsk/c32t2d3
/dev/dsk/c35t1d2
/dev/dsk/c35t1d3
/dev/dsk/c35t1d4
/dev/dsk/c35t1d5
/dev/dsk/c35t1d6
/dev/dsk/c35t1d7
/dev/dsk/c35t2d0
/dev/dsk/c35t2d1
/dev/dsk/c35t2d2
/dev/dsk/c35t2d3



/dev/vg03
/dev/dsk/c32t2d4
/dev/dsk/c32t2d5
/dev/dsk/c32t2d6
/dev/dsk/c32t2d7
/dev/dsk/c32t3d0
/dev/dsk/c32t3d1
/dev/dsk/c32t3d2
/dev/dsk/c35t2d4
/dev/dsk/c35t2d5
/dev/dsk/c35t2d6
/dev/dsk/c35t2d7
/dev/dsk/c35t3d0
/dev/dsk/c35t3d1
/dev/dsk/c35t3d2


#

[Torsten.]

But you are on the right server?

;-))

Even the mount point is away.

Someone was probably very accurate while removing it.

Even yesterday it was not in lvmtab!

[Nancy Calderon]

I also thought if it was the right server and yes it is :(

[SivaOX]

Hi Nancy,

Please check your typo,
# lvdisplay /dev/vgapps06/lvol3

Your LVM resides on vgapp06, not vgapps06.

[Torsten.]

It cannot be vanished without a trace.

Yesterday it was not in lvmtab - this cannot be. What are the timestamps of fstab and lvmtab files? Are the disk device files still there? There must be something in the syslog file.

[Nancy Calderon]

Hi all,
# vgdisplay vgapp06
vgdisplay: Volume group "/dev/vgapp06" does not exist in the "/etc/lvmtab" file.

vgdisplay: Cannot display volume group "vgapp06".
# vgdisplay /dev/vgapp06
vgdisplay: Volume group "/dev/vgapp06" does not exist in the "/etc/lvmtab" file.

vgdisplay: Cannot display volume group "/dev/vgapp06".

[Eric SAUBIGNAC]

Oups ... my previous answer was based only on the first part of your attached document. Did not understood that this VG no more exists :-(

By chance, does /.sh_history exists ? If yes keep it in a corner, and analyze it.

[Avinash20]

> How many System admin do u have who know the root password

> Also check in syslog/ (if rebooted OLDsyslog)
Path: /var/adm/syslog/syslog.log
/var/adm/syslog/OLDsyslog.log

for LVM

# cat /var/adm/syslog/syslog.log | grep LVM
# cat /var/adm/syslog/syslog.log | grep vgexport

[smatador]

Hi,
If the vg disapear that means somebody do an umount of you /apps/pedi, deactivate the vg and vgexport it. lots of command.
So you could have some log on syslog.log
Check the time of the vgexport [ if it occur by night it could be a script or batch, strange behaviour but in your case, all could be possible]. If you don't see the vgexport command, look at the .sh_history.

For recovering it, look at the /etc/lvmconf.
check for you vgapps06.conf and mapfile and try to vgimport it.

[Nancy Calderon]

Hi guys,
There are some vgexports but not from vgapp06

/dev/vg01
Mar 3 17:30:01 pan LVM[25719]: vgexport -p -m /sys_tools/DRVGMAPS.d/vg02.pan -s
/dev/vg02
Mar 3 17:30:02 pan LVM[25720]: vgexport -p -m /sys_tools/DRVGMAPS.d/vg03.pan -s
/dev/vg03
Mar 3 17:30:02 pan LVM[25736]: vgexport -p -m /sys_tools/DRVGMAPS.d/vgappsedi01
.pan -s /dev/vgappsedi01
Mar 4 05:32:52 pan LVM[17178]: vgexport -p -s -m vg00.maps.temp vg00
Mar 4 05:32:56 pan LVM[17247]: vgexport -p -s -m vg01.maps.temp vg01
Mar 4 05:33:00 pan LVM[17322]: vgexport -p -s -m vg02.maps.temp vg02
Mar 4 05:33:03 pan LVM[17394]: vgexport -p -s -m vg03.maps.temp vg03
Mar 4 05:33:05 pan LVM[17463]: vgexport -p -s -m vgappsedi01.maps.temp vgappsed
i01
Mar 4 06:30:01 pan LVM[6546]: vgexport -p -m /sys_tools/DRVGMAPS.d/vg01.pan -s
/dev/vg01
Mar 4 06:30:01 pan LVM[6559]: vgexport -p -m /sys_tools/DRVGMAPS.d/vg02.pan -s
/dev/vg02
Mar 4 06:30:01 pan LVM[6563]: vgexport -p -m /sys_tools/DRVGMAPS.d/vg03.pan -s
/dev/vg03
Mar 4 06:30:01 pan LVM[6565]: vgexport -p -m /sys_tools/DRVGMAPS.d/vgappsedi01.
pan -s /dev/vgappsedi01
Mar 4 17:30:03 pan LVM[26564]: vgexport -p -m /sys_tools/DRVGMAPS.d/vg01.pan -s
/dev/vg01
Mar 4 17:30:03 pan LVM[26638]: vgexport -p -m /sys_tools/DRVGMAPS.d/vg02.pan -s
/dev/vg02
Mar 4 17:30:03 pan LVM[26642]: vgexport -p -m /sys_tools/DRVGMAPS.d/vg03.pan -s
/dev/vg03
Mar 4 17:30:03 pan LVM[26662]: vgexport -p -m /sys_tools/DRVGMAPS.d/vgappsedi01
.pan -s /dev/vgappsedi01
Mar 5 05:32:37 pan LVM[13427]: vgexport -p -s -m vg00.maps.temp vg00
Mar 5 05:32:42 pan LVM[13499]: vgexport -p -s -m vg01.maps.temp vg01
Mar 5 05:32:46 pan LVM[13581]: vgexport -p -s -m vg02.maps.temp vg02
Mar 5 05:32:49 pan LVM[13651]: vgexport -p -s -m vg03.maps.temp vg03
Mar 5 05:32:51 pan LVM[13723]: vgexport -p -s -m vgappsedi01.maps.temp vgappsed
i01
Mar 5 06:30:00 pan LVM[2735]: vgexport -p -m /sys_tools/DRVGMAPS.d/vg01.pan -s
/dev/vg01
Mar 5 06:30:01 pan LVM[2738]: vgexport -p -m /sys_tools/DRVGMAPS.d/vg02.pan -s
/dev/vg02
Mar 5 06:30:01 pan LVM[2741]: vgexport -p -m /sys_tools/DRVGMAPS.d/vg03.pan -s
/dev/vg03
Mar 5 06:30:01 pan LVM[2742]: vgexport -p -m /sys_tools/DRVGMAPS.d/vgappsedi01.
pan -s /dev/vgappsedi01

[smatador]

Hi,
For all Gurus, Nancy vgscan -pv does not see the pv of the "vgapps06" normally
vgscan -pv examine the VGID of every disk on the system. I'm wrong ?
But the output does not see the pv of vgapps06. That means some pvcreate could be also do on this 3 pv?

[smatador]

Hi,
Just to be sure, could you do a pvdisplay of each pv of the disapear vgapps06?

[Nancy Calderon]

Hello guys,
I know now what happened, some other sysadmin just told me he removed the vg.

Thanks a lot for your responses.

[Nancy Calderon]

...

[Avinash20]

Hmmm,, What you could do is to compare the output of

# ioscan -fnC disk

and

# strings /etc/lvmtab

## Check for some extra disk which are available.
## If you get that, check for pvdisplay on those disk

[James R. Ferguson]

Hi:

> I know now what happened, some other sysadmin just told me he removed the vg.

Well now, isn't _communication_ a good thing!?!

...JRF...