topic Re: Lost Volume Groups in Operating System - HP-UX

Lost Volume Groups

Nobody's Hero — Sat, 20 Aug 2005 13:59:15 GMT

Hi All,

Recently I have been doing Nclass Upgrades from 11.0 > 11.11 running Oracle and Peoplesoft. Very good luck so far with doing upgrades v.s cold install. Here is what has happened this time.

1) I shutdown and installed 2 new larger internal disks for the OS. (all went well)

2) Booted off of the ignite tape to resize filesystes. (Went well)

3) Let the system rebuild and reboot. (all went well)

when the system came up, I am missing a few volume groups for my oracle data. All of the root vg00 is fine.

I am missing: /dev/vgfinap and /dev/vgfindb

I dont see the entry in /etc/lvmtab anymore for these 2 volume groups. My /etc/fstab looks OK. The disks in these volume groups are a bunch of EMC fibre attached disks. Each one is populated on ? 18 Gbyte disk, so there are probably somewhere around 100 disks. Any idea how I can start to get my disks back. I am sure my data on the disks are still OK.

I proceeded to continue with the upgrade to 11.11. That seems to be going OK so far. I just need to get my disks and vg's back.

EMERGENCY: Production Oracle Upgrade In Progress.

Re: Lost Volume Groups

Slawomir Gora — Sat, 20 Aug 2005 14:22:27 GMT

Hi,
maby vgscan will help you

Re: Lost Volume Groups

Nobody's Hero — Sat, 20 Aug 2005 14:23:40 GMT

Already tried that..no go....

vgimport maybe ????

Re: Lost Volume Groups

Raj D. — Sat, 20 Aug 2005 14:31:30 GMT

Hi Robert,
Do you know which disks were for that two Volume group , i.e vgfinap and vgfindb.

If you know the disks device number , u can check as follows:

1. iocan -fnC disk [ check for that disks are visible by the system].

2. Check if the vg directory is present in the dev directory , with the group file , else need to create, with mknod.

3. use # vgimport /dev/vgfinap /dev/dsk/cxtxdx /dev/dsk/cytydy .. (Total no of disks contains for that vg)

4. Activate the vg.

5. Add entry in /etc/fstab. and mount the Filesystems. [ You need to have details of previous bdf output, to match lvol-> to fs.]
Hope this will help you to analyse what caused the problem and to get back the vg.

Cheers,
RajD.

Re: Lost Volume Groups

Sameer_Nirmal — Sat, 20 Aug 2005 14:41:59 GMT

Run vgcfgrestore -l -n for those missing vgs. If the Vg's lvm conf file is up to date , then it should show the disks under VG.
Check all those EMC luns using pvdisplay

It may be posible that LVM header has been overwritten or corrupted on those disks.
Just we do while adding new disks, you can use vgcfgrestore -n /dev/rdsk/c?t?d?

Re: Lost Volume Groups

Sameer_Nirmal — Sat, 20 Aug 2005 15:12:26 GMT

Just want to make few points.

You would see vg entry in /etc/lvmtab only when there is corresponding vg conf file in /etc/lvmconf . During system startup vgscan is run to read these conf files for each vgs and it also queries the pvs whcih are supposed to be under those vgs. Then it update the conf file if required.

As said by Raj , we have to make sure all the required things for those VG are in the system like physical disks ( EMC disks ), vg group file etc..

It is very rare that the LVM header from those disks ( 100 !!! ) are overwritten .
Maybe something is wrong at EMC side if you are not getting any single LUN from EMC!!!Hence you have to verify them first using ioscan then pvdisplay . Check the vg config files in /etc/lvmconf

You have to be very careful and look around to make sure all before doing anything.

Let us know if you have doubts or confusion.

Re: Lost Volume Groups

Alex Lavrov. — Sat, 20 Aug 2005 15:38:09 GMT

Try to issue this command on all the disks:

echo 0x2010?2X|adb /dev/dsk/cXtYdZ|expand|tr -d " "|sed "s/2010:/VGID /"

You should see the VGID of each disk and it might be helpfull to see which disks belong toget her to which VG. The info is taken from the kernel itself, by supplying the debugger specific offset.

Alex.

Re: Lost Volume Groups

Bill Hassell — Sat, 20 Aug 2005 17:17:26 GMT

Take a look at syslog and also /etc/rc.log to see if volume groups failed to activate because of missing disks. When the system boots up, each volume group is activated but a failure there will not remove the /dev/directories. Something else has happened either during the backup or the recovery steps. You need to look carefully at the Ignite log when you created the tape. Hopefully, you used the -v option and saved a copy of this file on another system. Ignite keeps a log of the last make_tape_recovery in the file:

/var/opt/ignite/recovery/latest/recovery.log

However, to recover the missing volume groups, you'll need import the missing volumes and for that, you need a vgdisplay of the lvol names (unless they are all default names lvol1, lvol2, lvol3, ...). Start by creating the missing directories and re-creating the group device file. You'll need to choose the group minor number either from previous documentation or by finding a unique number from the command:

ll /dev/*/group

Then use the vgimport command, but check the /etc/lvmconf directory for the last mapfiles.

Re: Lost Volume Groups

Devender Khatana — Mon, 22 Aug 2005 00:09:54 GMT

Hi,

You can plug in the old OS disks if still available to find out these details related to these VG's. Then you can use the vgimport command to import these VG's. Are these EMC disks visible to system in ioscan.

Data indeed should be intact and importing back will solve the issue.

HTH,
Devender

Re: Lost Volume Groups

Mark Nieuwboer — Mon, 22 Aug 2005 00:27:20 GMT

Hi,

Normely after a upgrade SAN device must be propegated again over the system.
Try insf -e to make the disk visible. If not check at the EMC side if the node is viseble.

grtz. Mark

Re: Lost Volume Groups

morganelan — Mon, 22 Aug 2005 00:41:07 GMT

Hi,
Try to use this commands:
#symcfg discover
#powermt check
#powermt config
#powermt display
#powermt save