topic Re: Invalid Superblock in Operating System - HP-UX

Invalid Superblock

Lai Nee Shyang_1 — Tue, 23 Oct 2007 02:17:03 GMT

Hi There

I need help to recover a "badly damage" VG file system, which keeps prompting me cannot find super-block. Example:
#> fsck -F vxfs -y -o full,nolog /dev/vgtest/rlvol25
vxfs fsck: not a valid vxfs file system
invalid super-block
search for auxiliary super-block? (ynq)y
alternate super-block not found
vxfs fsck: cannot initialize aggregate
file system check failure, aborting ...

Well... what happened was, this VG is actually VG00 of another system. A SAN boot disk was added to the vg00 disk group but freaky accident happened to the boot disk, and what was left was the SAN disk. I'm task to "swing" this SAN disk to another system to try and recover some of the file system content.

Appreciate and advises....

cheers

Lai

Re: Invalid Superblock

Torsten. — Tue, 23 Oct 2007 02:22:54 GMT

"this VG is actually VG00 of another system"

What result gives "vgdisplay -v" for this VG?
Consider this is really not a vxfs, but hfs or swap.

Can you boot the other system into single user mode from this disk?

Re: Invalid Superblock

Lai Nee Shyang_1 — Tue, 23 Oct 2007 02:44:35 GMT

Hi There

Please bear with me while I try to explain how it leads to this mess.

The VG00 was introduce with a SAN disk via vgextend and lvextend (-m 2). The sole purpose, was to create a split environment for some testings, therefore after the sAN disk completed mirror, a lvsplit command was issue and bootconf/lvlnboot was modify to point to the SAN disk whenever the test environment is required.
However, something happend at the VG00 disks and a make_tape_recovery was done on the VG00. To "preserve" the content of the SAN disk, the make_tape_recovery only recover to the 2 internal boot disks, thus the new VG00 nolonger has the SAN disk in the VG. I'm task to recover one filesystem in the SAN disks so that the testing can resume somewhere else.

I've imported the VG onto a dev system, it complains about the 2 missing disks in VG00 but the vgimport was successful, and I've vgreduce the 2 missing disks from the VG. There were many LVs some belong to the original VG00 and some to the SAN disk. I've manage to lvremove those without a valid PVs leaving only those with valid PVs (the SAN disk). But now my last hurdle is I cannot clean the FS for mounting.

sorry if I cause more confusion then giving a clearer picture.

Lai

Re: Invalid Superblock

Lai Nee Shyang_1 — Tue, 23 Oct 2007 02:47:53 GMT

Hi Torsten

vgdisplay shows :VG Name /dev/vgtest
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 14
Open LV 14
Max PV 16
Cur PV 3
Act PV 1
Max PE per PV 4553
VGDA 2
PE Size (Mbytes) 16
Total PE 4543
Alloc PE 4349
Free PE 194
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vgtest/lvol13
LV Status available/syncd
LV Size (Mbytes) 352
Current LE 22
Allocated PE 22
Used PV 1

LV Name /dev/vgtest/lvol16
LV Status available/syncd
LV Size (Mbytes) 4096

.........

--- Physical volumes ---
PV Name /dev/dsk/c15t1d1
PV Status available
Total PE 4543
Free PE 194
Autoswitch On

I do not need to boot from this disk again, I only require to recover some data from one of the filesystem. So far the fsck doesn't work on any of the LVs listed.

Thanks and Cheers

Lai

Re: Invalid Superblock

Matti_Kurkela — Tue, 23 Oct 2007 03:36:38 GMT

Can you describe us how the old, damaged vg00 was laid out?

The fact that your example shows /dev/vgtest/rlvol25 suggests that the damaged system had a very non-standard vg00 layout: lvol numbers go at least up to 25 instead of default 8.

If I understand your situation correctly, you've had to force the volume group activation as all the required PVs are not present. But did you have to do anything else?

It might be that the beginning of this particular filesystem was on the damaged disk and only the "tail end" was on the SAN disk.
If the beginning of the filesystem is missing, fsck needs some help in locating an alternate superblock. A superblock is a key filesystem structure, and multiple copies of it are located in various locations in the filesystem. If none of them are available, the situation is very bad.

To understand the problem better, it would be important to see the output of the following commands:

vgdisplay -v vgtest
lvdisplay -v /dev/vgtest/lvol25

(The latter will produce a long list of extents. It will point out if some parts of the LV are missing.)

MK

Re: Invalid Superblock

Lai Nee Shyang_1 — Tue, 23 Oct 2007 03:46:34 GMT

Hi MK

thanks for your prompt reply.

The VG00 was normal, eg. lvol1, lvol2 etc. A SAN disk was introduced, lvextend -m 2 , and lvsplited, thus creating lvol1, lvol2,.. lvol1_X, lvol2_X (_X are the splited volumes). After VG activation, lvol1, 2... are all invalid because only the SAN disk was recoverd and open to the dev system. As such, lvol1-lvol12 are all invalid, and I lvremoved them.

I've checked the lvdisplay -v /dev/vgtest/lvol25, all PEs are current and pointing to the PV in vgtest

Below is the vgdisplay output, the lvdisplay output is sampled, I've verify they are correct. cheers

Lai

___________________________________________
#> vgdisplay -v vgtest
--- Volume groups ---
VG Name /dev/vgtest
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 14
Open LV 14
Max PV 16
Cur PV 3
Act PV 1
Max PE per PV 4553
VGDA 2
PE Size (Mbytes) 16
Total PE 4543
Alloc PE 4349
Free PE 194
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vgtest/lvol13
LV Status available/syncd
LV Size (Mbytes) 352
Current LE 22
Allocated PE 22
Used PV 1

LV Name /dev/vgtest/lvol16
LV Status available/syncd
LV Size (Mbytes) 4096
Current LE 256
Allocated PE 256
Used PV 1

LV Name /dev/vgtest/lvol17
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 64
Allocated PE 64
Used PV 1

LV Name /dev/vgtest/lvol18
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 64
Allocated PE 64
Used PV 1

LV Name /dev/vgtest/lvol19
LV Status available/syncd
LV Size (Mbytes) 5008
Current LE 313
Allocated PE 313
Used PV 1

LV Name /dev/vgtest/lvol20
LV Status available/syncd
LV Size (Mbytes) 5120
Current LE 320
Allocated PE 320
Used PV 1

LV Name /dev/vgtest/lvol21
LV Status available/syncd
LV Size (Mbytes) 5120
Current LE 320
Allocated PE 320
Used PV 1

LV Name /dev/vgtest/lvol22
LV Status available/syncd
LV Size (Mbytes) 8000
Current LE 500
Allocated PE 500
Used PV 1

LV Name /dev/vgtest/lvol23
LV Status available/syncd
LV Size (Mbytes) 6160
Current LE 385
Allocated PE 385
Used PV 1

LV Name /dev/vgtest/lvol24
LV Status available/syncd
LV Size (Mbytes) 4096
Current LE 256
Allocated PE 256
Used PV 1

LV Name /dev/vgtest/lvol25
LV Status available/syncd
LV Size (Mbytes) 4000
Current LE 250
Allocated PE 250
Used PV 1

LV Name /dev/vgtest/lvol26
LV Status available/syncd
LV Size (Mbytes) 23552
Current LE 1472
Allocated PE 1472
Used PV 1

LV Name /dev/vgtest/lvol27
LV Status available/syncd
LV Size (Mbytes) 1008
Current LE 63
Allocated PE 63
Used PV 1

LV Name /dev/vgtest/lvol28
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 64
Allocated PE 64
Used PV 1

--- Physical volumes ---
PV Name /dev/dsk/c15t1d1
PV Status available
Total PE 4543
Free PE 194
Autoswitch On

--- Logical volumes ---
LV Name /dev/vgtest/lvol25
VG Name /dev/vgtest
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 4000
Current LE 250
Allocated PE 250
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c15t1d1 250 250

--- Logical extents ---
LE PV1 PE1 Status 1
00000 /dev/dsk/c15t1d1 02500 current
00001 /dev/dsk/c15t1d1 02501 current
00002 /dev/dsk/c15t1d1 02502 current
00003 /dev/dsk/c15t1d1 02503 current
00004 /dev/dsk/c15t1d1 02504 current
00005 /dev/dsk/c15t1d1 02505 current
00006 /dev/dsk/c15t1d1 02506 current
00007 /dev/dsk/c15t1d1 02507 current

Re: Invalid Superblock

Matti_Kurkela — Tue, 23 Oct 2007 12:21:01 GMT

OK, now that's solid information.

If all the PE:s of lvol25 are there, the LV should be usable, but apparently the fsck is not finding anything recognizable as a vxfs filesystem.

But... are you *sure* these LVs are supposed to use the vxfs filesystem? As the vxfs file system check does not seem to recognize it at all, it might actually be a HFS filesystem.

Run the command

fstyp -v /dev/vgtest/lvol25

and tell us what it says.
Check out all the other LVs with fstyp to find out what kind of filesystems they contain, if any.

Another thing... which version of HP-UX was in use on the damaged system, and which version is on this "rescue" system?
If the damaged system had a newer HP-UX version than the rescue system, the damaged system might use a newer version of vxfs, which the rescue system cannot read without some upgrades.

In this case, you might need to install a more advanced version of vxfs to the rescue system to be able to read this disk.

Or maybe the disk was used as "raw" LVs for some sort of database?

In that case, there will be no filesystem at all, and you will need to install the correct version of the database software on the rescue system. Someone with the appropriate DBA skills must then handle the rest of the rescue job.

As there are so many LVs waiting for rescue, it would be extremely beneficial to get a copy of the /etc/fstab of the damaged system, if at all possible. Even an old or incomplete copy might offer useful clues.

MK

Re: Invalid Superblock

Sandman! — Tue, 23 Oct 2007 12:51:49 GMT

I agree with Matti and maybe you should try your fsck command again only this time specifying the HFS argument to the "-F" option. And remove the "-o" cmd line option along with its arguments viz.,

# fsck -F hfs -y /dev/vgtest/rlvol25

Re: Invalid Superblock

Lai Nee Shyang_1 — Tue, 23 Oct 2007 20:58:59 GMT

*kao tao* <--- bow

U guys are brilliant... The orginating system was a V2, the test system is a V1.. I was having a wild goose chase all along.. sigh.

I have "swing" the disks to a V2 test system and "wala" every fs can be fsck properly.

Ok.. one strange thing to note, I'm able to fsck and mount the filesystems (eg. /var, /usr etc) But one of the filesystem recovered only have 1 directory and 1 file in it. According to the user, it should be fully populated with files/directores.

Could it be damage when I tried to fsck it on a V1 systems. But strangely, if it is damange why only 1 file and 1 directory is able to survive.

Thanks guys.

Lai