EXT2/EXT3 filesystem recovery

[Stuart Browne]

I don't suppose anybody has any experience with restoring an ext[23] filesystem that has had it's boot sector corrupted (badly) ?

One partition of one of my HDD's using ext3 did this. File system is completely inaccessable.

I've dumped a backup (19GB) to another machine so I can play with it, and have tried restoring the filesystem parameters manually, but this was mostly unsuccessful (inode table after an e2fsck was still mostly fried).

I can't tell how bad the corruption is as I have nothing to compare it too.

I guess I'm asking if anybody knows how much filesystem header there is before the inode table data.

[Vernon Brown_4 · Accepted solution]

You might find something here; docs on ext2; not sure how close to ext3 it is.

http://www.nongnu.org/ext2-doc/ext2.html

Good luck !!

Vern

[Steven E. Protter]

This happened to me back in the red hat 6.2 ext2 days.

My recovery method was to rebuild the system from scratch with empty filesystms and then restore files as needed.

To bad Ignite hasn't been ported to Linux. That would make things easy for you.

Good Luck.

SEP

[Colin Topliss]

FYI, ext3 provides journaling. You can mount an ext3 filesystem as ext2 (ie journaling disabled). Its just a change to the mount options.

What error do you get when you try and mount this filesystem? I guess maybe that if your attempts at restoring have failed so far, there might not be much hope. What steps did you try, and what were the results?

Col.

[Colin Topliss]

Oh, there was something else I just remembered.

You don't actually say what version of Linux you're running. There were issues with 6.0 and 7.2 where, if you had a partition that required an fsck, there were occascions where you had to fsck several times before all errors were corrected. I had one system that ran through the fsck OK, marked the filesystem as clean, then went on to produce multiple I/O errors! Manually unmounting and fsck'ing several times finally cured the problem.

This happened on an ext2 filesystem.

Col.

[Stuart Browne]

Things that are known:

The first 2-10k of the filesystem is destroyed.

'file -s /dev/hda5' returns just 'data'.

After some mucking around..

e2retrieve: failed to do anything useful, inode tables too corrupt for it.
e2recover: had a good go at it. was able to find superblocks and some of the inode data, but was unable to list any of the details.

With the image dump I took as a starting point (yay 20gb file!), I tried re-making the fielsystem over the top to get the correct filesystem header, restoring the filesystem, then incrementally writing the 'good' filesystem headers back.

This was close to successful, however it appears as if the inode table was too destroyed for it to make much sense of the data, or I was a few byts off in my restoring information for 'fsck' to make sense of it.

Vernon, with the info you provided, I'll try this attack again, as I can work out where the segment boundaries are. Hopefully with a bit of patience and time, I can reconstruct it.

For the moment however, I've been manually extracting configuration files etc. with the help of 150+ split-files, and 'grep -ab'. Painful, and very slow, but it's gotten most of the system into working shape again.

Just a few monitoring databases missing now.

I'll keep posting back..

[dirk dierickx]

the first kb's of the fs are destroyed, i think when you run fsck you should use an alternative superblock (fsck normally takes the first one).

also, if your boot fs is damaged, is that /boot or /. if you can still access /etc i guess it should be possible to boot from cd and rerun lilo.

if you have a dump image, why don't you just reformat the fs and put the dump back?

[Roberto Polli]

did you try to replace the superblock with a right one?

fsck -b GOOD_BLOCK /dev/hda5

did you check /dev/hda5 has correct major,minor number?

To get a list of good sb you can use numbers resulting from fsck'ing another hd.


Peace, R>

[Stuart Browne]

Dirk, the image dump is after the corruption.

Roberto, yes, major and minor are fine, it was a real corruption of the filesystem.

As for both, yes. I tried using an alternate superblock, this still failed.

After a few hours of analysis, and wading through the first 872kb of the filesystem here's what I found.

The boot record + superblock (0) are destroyed.
The group descriptors (0) table is destroyed.
The first roughly 241664 bytes of the first inode table are also destroyed.

So, *wheee!*

This leaves me doing some rather nasty stuff in an attempt to recover what I can..

Piece together the first 1024 bytes from the re-made dummy filsystem. Pull the 2nd superblock and group descriptors, blank out the block bitmap/inode bitmp and inode table in the first section, and run an fsck, and see what can be recovered!

Yowser! Talk about ugly!

Anyway, off I go to attempt this. Go 'Manual Filesystem Recovery'! :)

If worst comes to worse, perhaps I'll be able to use one of the other rescue tools after this.

[Stuart Browne]

Attempt number 1: failed.

Incorrectly made up header :P

'dd' work is confusing. Combine that with not being 100% sure of some of these values (for confusion's sake), and you've got alot of fun!

Confusion which caused failure:

Root partition meta-data layout, offset in filesystem 'blocks' (not disk sectors or standard 512 byte blocks):

(taken from http://www.nongnu.org/ext2-doc/ext2.html).

offset # of blocks description
-------- ----------- -----------
0 1 boot record
-- block group 0 --
(1024 bytes) 1 superblock
2 1 group descriptors
3 1 block bitmap
4 1 inode bitmap
5 214 inode table
219 7974 data blocks

Ok, fine.

-- block group 1 --
8193 1 superblock backup
8194 1 group descriptors backup
8195 1 block bitmap
8196 1 inode bitmap
8197 214 inode table
8408 7974 data blocks


Ahh..

Now, this states that it allocates an entire 'block for the superblock backup. Whilst it might do this, it also fills the remainder of the block with 'garbage'. This garbage doesn't like living in the root subset of data.

So, with my file system, 4096 byte blocks, it turns out like this:

bytes
0-1024 boot_record (from dummy fs)
1025-2048 superblock (1, from bad fs)
2049-8192 chr 0's (/dev/zero)
8193-12288 group_descriptors (1, from bad fs)
12289-884736 from dummy fs (zero'd inodes basically).

I was thinking of trying to grab the good-half of the root inode table, but that's just pushing it.

Currently restoring a backup of the bad fs image, and will re-try. 'conv=notrunc' is my friend now *nod* :)

Will report back later.

[Stuart Browne]

blech.. numbering off-by-one :P