Advanced file system - File set info missing

[Rakesh Jha_1]

After carsh of one GS140 running Tru64 V5.1B-3, file systems related to a database failed to mount.
I treid salvage and fixfdmn utilities. When I run fixfdmns I get messages -
# showfsets wbt_back5
showfsets: domain 'wbt_back5' has no filesets
# /sbin/advfs/fixfdmn wbt_back5
fixfdmn: Checking the RBMT.
fixfdmn: Clearing the log on volume /dev/disk/dsk133c.
fixfdmn: Checking the BMT mcell data.
fixfdmn: Checking the deferred delete list.
fixfdmn: Checking the root tag file.
fixfdmn: Unable to continue, no fileset information was found.

fixfdmn is not able to continue, no changes made to domain, exiting.

Is there a way to restore the file set info?

[Kapil Jha]

Hello Rakesh,
Can you plese let us know what all steps you followed after crash.what was the cause of the crash?
And how you are running fixfdmn?
have u booted through CD or tried to umount filesystem(which m sure would not have been possiblem).
Need more info.
BR,
Kapil

[Rakesh Jha_1]

Hello Kapil,

The cuase of server crash is unknown, I am ruuning a separate thred for that as well. THe server boots up. Actually we boot server from SAN and it runs several DBs. All DBs came up ok except one which has five volumes.

When I mounted these volume they complained about fileset info missing. Also when I saw disklabel there was no partition used whereas it should have shown 'c' partition with AdvFs.

I tried doing salvage and fixfdmn as shown, in my first mail. I know fileset, if enter the fileset using mkfset, file system mounts ok with no data on it.

[Kapil Jha]

I think as you have mounted filesets , you will have to restore data from backup.
Its strange how disk configuration gone.

Guru's I would like you to shed some light on this.

Rakesh,
Do you have any analysis from HP side on hardware problems.

BR,
Kapil

[Venkatesh BL]

Can you try running 'fixfdmn' in verbose mode and tells us if you find anything more? Are you sure you have all the exact disk that made up the 'wbt_back5' domain before crash?

[Jack M. Estes II]

Rakesh:

This is a longshot but Holmesian logic dictates I ask:

Any chance your database was using raw volumes? If so, there may not be any filesets to recover. That would really be unusual; I see that kind of thing done more often on AIX, but a quick look at your /etc/fstab would probably settle that question.

You mention the O/S boots from the SAN and IIRC from the other thread, this is a single node cluster, right? OK, so what kind of SAN is this attached to? I ask so I can suggest commands to run on the SAN itself to try to figure out what's going wrong. What does verify -f on the domain yield?

From the output you posted, TRU64 knows about the domain but doesn't see any filesets inside that domain (you mention there should be 5, right?). What about an ls -la on /etc/fdmns? Do the links point to the right disk device for your missing filesets?

Oh man, you don't mention whether LSM is involved or not, but I'm assuming you're letting the SAN handle mirroring/RAID stuff. If LSM IS involved, you might have a problem with your fstab since this is a single node cluster.

What does /etc/fstab look like for the filesets you want? Are the fstype entries for these filesets AdvFS or LSMnopriv? If the latter, change it to "AdvFS" and see if verify -f and fixfdmn will work their magic. Obviously make sure you're looking at the right fstab (there should only be one since it's a single node cluster, but if it used to be multinode you might have extra CDSLs still floating around.

Your fixfdmn command ran because there were no mounted filesets (duh!), but if your domain has been extended over time as the database grew, you might have a multipage RBMT file with a full last page; a condition that fixfdmns can't handle out of the box. HP issued a ERP for that (your version of TRU64 is suseptible) which is included in B-4. You can probably still find just that patch here: http://www12.itrc.hp.com/service/patch/patchDetail.do?patchid=T64KIT1000660-V51BB26-E-20060526&admit=109447627+1214977552912+28353475

If /etc/fdmns looks right (domains and fileset device links are all there and correct) then you might be screwed. Did the wbt_back5 domain panic with metadata errors? Well, that stuff should be in the binary.errlog which I can search while looking at the problems from your other thread.

Is there anything interesting in the /var/adm/ logs? When you send me your binary.errlog, consider running syscheck -escalate and gimme that tarball to review too.

Jack