all vg information lost

[Ravinder Kaul]

Hi

On one of the servers all the volume group information seems to be lost including vg00. Is there any way to recover it. None of the LVM commands like vgscan vgimport vgcfgrestore works. It comes out without doing anything. SAM shows the status of the disks as follows. Any hope?:(
TIA
RK
0/0/1/1.0.0 1 Unused -- 17366 IBM DMVS18D ^ x
xx 0/0/1/1.2.0 1 Unused -- 8683 SEAGATE ST39103LC x
xx 0/0/2/0.0.0 1 Unused -- 17366 IBM DMVS18D x
xx 0/0/2/1.2.0 1 Unused -- 0 HP DVD-ROM 6x x
xx 0/3/0/0.0.0 1 Unused -- 70007 HP 73.4GATLAS10K3_ x
xx 0/3/0/0.1.0 1 Unused -- 70007 HP 73.4GATLAS10K3_ x
xx 0/3/0/0.10.0 1 Unused -- 70007 HP 73.4GATLAS10K3_ x
xx 0/3/0/0.11.0 1 Unused -- 70007 HP 73.4GATLAS10K3_ x
xx 0/3/0/0.12.0 1 Unused -- 70007 HP 73.4GATLAS10K3_ x
xx 0/3/0/0.13.0 1 Unused -- 70007 HP 73.4GATLAS10K3_

[Joseph Loo]

hi,

have you lost or corrupted the files, e.g. vg00.conf under /etc/lvmconf?

what does dmesg say?

have you try,
# vgcfgbackup /dev/vgXX

what is the output return?

regards.

[Mark Grant]

Looks to me like someone broke your system. If vgscan can't find the volume groups and vgcfgrestore can't re-create the volume information for you, I think you are in the realm of ignite and restores from backup.

However, before panicing, it might be nice to know if you get any errors from the lvm commands. Also, an ioscan output from your disks might be useful too.

[Robert-Jan Goossens]

Hi,

Try this,

# mv /etc/lvmtab /etc/lvmtab.old
# vgscan -a -v
# vgchange -a y /dev/vg00
# vgdisplay -v vg00

Robert-Jan

[Cheryl Griffin]

Take SAM out of the picture. What does command line say for the system?
# lvlnboot -v
# vgdisplay -v /dev/vg00

Maybe it's a SAM problem reading the disks or the configuration or potentially a SAM patch issue.

What os?

[Ravinder Kaul]

Thanks everyone for their replies. None of the lvm command gives either any output or error. Simply comes out. That is why ran SAM shows which error "no volume groups defined on the system". All the conf files r avl. under /etc/lvmconf
ioscan shows all disks as claimed. /dev/vgxx also present.It seems that the system is not able to sense the disks as even being vg enabled .

[Robert-Jan Goossens]

Hi,

could you check the state of your installed patches ?

# swlist -l fileset -a state -a revision

Robert-Jan

[Suresh Patoria]

Hi,

if u have the system files backup then restore the files from the backup and then u try pls vgimport, vgcfgrestore etc.

if u don't have the system files backup then u can restore the files from Ignite backup

Thanx

[Bharat Katkar]

Ravindra,
Seems very unusual. In this situation backup seems to play important role.
I suspect this as an OS corruption. I think if you have backups intact, one thing u can try is boot the core os CD and run recovery mode. Let it reinstall OS again keeping all other things as they are.
But this should be last option you can try.
Before proceeding it will better if you can collect map files if you have stored anywhere in the system, VG structure and collect as much as info you can about your PV,VG and Filesystems.

See if this could help.

All the best.

[Bharat Katkar]

Forgot to tell you something. If you have latest Ignite Backup of VG00 that should be an better option. Again make sure you have Every details of you Other VG's and Filesystems.

ps. I was in UWB pune, you are the one you use to come there and if so where is this problem exactly.

[generic_1]

Are those San disks? If so I was just curious if some other server saw them and wiped them because someone that they had free disks to use.

[Ravinder Kaul]

bharat: nice to see you on the forum.no this is not in uwb pune.
jeff: there is a shared disk array connected to the server. But if you notice even the root disks are shown as being unused where they should be showing as vg00.
Let me explain the scenario in some detail:
There are 2 servers with some vgs on the disk array under MCSG env. This server rebooted by itself abruptly. I had read somewhere that under certain conditions of MCSG env one of the server can reboot itself.After that one of the disks under mcsg showed cross device link on both the servers but other vgs were avl. I ran vgcfgrestore on the disk on this server with -f option giving the name of a backup file but it came out with error that -f not found. I want to know whether this has something to do with all vg information getting wiped out. I rebooted this server and after that it lost all the vg information I can vgimport the vg information from the map files of this server but it is not giving any o/p to any LVM command. It shows all the disk as being unused and no vgs activated . I guess I will have to restore the OS from the ignite tape

[Douglas D. Denney]

Have you eliminated the possiblity of some sort of HW failure? With glitchy problems like that, I would suspect HW.

[KapilRaj]

A wild guess !

Do you have the lvm device driver loaded into the kernel currently ?. Let me apologise for this , I do not know how to do that !!.
But logically if u do not have the driver in the kernel any LVM disk would be shown as empty / unused (Again a wild guess !!).

Also check whether ur /usr filesystem is corrupted, Once I was trying to unmount a filesystem for days and atlast came to know that /usr/sbin/umount command is "ZERO" bytes long !!

Nice to see u here ravi ... i hope u remember me ....

Kaps (Hint ... the one with a half bottle hard liquor caught red-handed at office :) )

[Mohanasundaram_1]

Hi Ravi,

Since you say the system was in a serviceguard setup and you are having this LVM problem in ONE of the server, I would presume that the other Server in the cluster is able to see the disks properly and use it as well.

In that case, the problem will be isolated to one server. Therefore I would ignite that server and then import the map files of shared volume groups from the working server.

But if the shared VGs are not accessible by both the clustered nodes, then you need to look at those vgcfgrestore on all those volumes.
I guess your problem of LVM is isolated to one server and hence easy to recover from the map files of the working node.

[James Lynch · Accepted solution]

There are two versions of every LVM related command, one dynamically linked and one statically linked. The dynamic ones are located in /usr/sbin and the static ones are in /sbin. By default, root's PATH should be set to find the ones on /usr/sbin first. Check your PATH and try to use the ones in /sbin or vice versa.

You should also know that most of LVM commands are hard links to one main executable (on 11.11 there are 26 files all linked together). What that means is, if someone accidentlly overwrote and one of the hard linked files, then all of the remianing LVM commands would also be overwritten. This is what I suspect happened to your system. If that is the case then you still should be able to use the ones in /sbin.

Lastly, if your VG00 is truely gone, and you know the exact sizes, PV locatations of all of you VG00 lvols, and all of the lvols are contingous, you can rebuild your VG00 from maintenance mode. I have done this sucessfully on several occassions, but the three previous conditions must be true. This a complicated procedure and is only a last resort if you don't have a valid make_tape_recovery image.

JL

[Florian Heigl (new acc)]

Hi,

please take care to work slowly and carefully in this kind of situation.

I'm missing some information - You said vgscan etc. don't work - is the system still up and running?

if yes:
move /etc/lvmtab to /etc/lvmtab.sav, then retry a vgscan -p -v (-p doesn't actually write information)

if this doesn't work check Your vg* binaries, most LVM commands are one and the same binary, maybe someone simply wiped it out.

try to take a backup of the actual *user data*

also, going to lvm mainteance mode might be to consider.

but personally I'd start the vgscan, and if it doesn't work, launch a backup and open a priority++ call with hp the very same moment.
rather keep the box up as long as it still works :)

[Ravinder Kaul]

Kapil:good to hear from u. How can I forget??
Well I had the same experience . Found out that the size of all lv commands in /usr/sbin is zero. James also had it right on . lv commands in /sbin are intact. the hard link seems to have broken. Thanks Jeff,Douglas,Mohan,Florian with chippng in with ur i/ps.What can be the reason for hard links to break. Should now only /usr/be restored to restore the links

Thanks

[Bill Hassell]

Hard links don't break. They are simply additional names for the same inode. If all the filenames show a zero length, then someone logged in as root zeroed the length of one of the commands (ie, cat /dev/null > /usr/sbin/lvcreate). That destroys the contents of the inode and thus all files will show the same size because it is a single file mwith multiple names. As mentioned, you have two copies, /sbin and /usr/sbin. Do NOT reboot but get a backup tape and restore just one of the commands in each directory (ie, /sbin/lvcreate /usr/sbin/lvcreate). This will restore all the commands (the links).

[Bill Hassell]

Oh, and change the root password immediately foillowed by a scan through root's .sh_history file. The system will continue to function without the LVM commands since the drivers know where the volume groups and lvols are located (in RAM). But if you reboot, the system will simply crash as there will be no way to activate any volume groups.