Corrupted filesystem after power outage

[Rui Vilao]

Greetings,

After a power outage, we got a filesystem corruption on /opt.

When I run fsck on single user mode, I get the following error:

# fsck -F vxfs -o full /dev/vg00/lvol6
vxfs fsck: file system had I/O error(s) on meta-data.
vxfs fsck: file system had I/O error(s) on meta-data.
pass0 - checking structural files
pass1 - checking inode sanity and blocks
vxfs fsck: fsck read failure bno = 188896, off = 0, len = 8192

However, the filesystem still can be mounted...

Any help/suggestion is apreciated,

Kind Regards,

Rui.

[Uday_S_Ankolekar]

Hi,

Check if you have any stale on this LV

-USA..

[Rui Vilao]

More info:

On syslog I get repeatedly:
...
Mar 20 17:40:05 prcc80 vmunix: vxfs: mesg 016: vx_ilisterr - /opt file system error reading inode 4512
Mar 20 17:40:05 prcc80 vmunix: vxfs: mesg 016: vx_ilisterr - /opt file system error reading inode 4513
Mar 20 17:40:05 prcc80 vmunix: vxfs: mesg 016: vx_ilisterr - /opt file system error reading inode 4514
Mar 20 17:40:05 prcc80 vmunix: vxfs: mesg 016: vx_ilisterr - /opt file system error reading inode 4515
Mar 20 17:40:05 prcc80 vmunix: vxfs: mesg 016: vx_ilisterr - /opt file system error reading inode 4516

...



The problem seems to be located on
/opt/java/bin


# ls -li /opt/java/bin
/opt/java/bin/javac_g not found
/opt/java/bin/javaverify not found
/opt/java/bin/javaverify_g not found
/opt/java/bin/jre not found
/opt/java/bin/jre_g not found
total 8
4494 -r-xr-xr-x 1 bin bin 3229 Apr 22 1999 .java_wrapper
4495 dr-xr-xr-x 3 bin bin 96 Nov 6 2000 PA_RISC
4508 lrwxr-xr-x 1 root sys 13 Nov 6 2000 jar -> .java_wrapper
4509 lrwxr-xr-x 1 root sys 13 Nov 6 2000 java -> .java_wrapper
4510 lrwxr-xr-x 1 root sys 13 Nov 6 2000 java_g -> .java_wrapper
4511 lrwxr-xr-x 1 root sys 13 Nov 6 2000 javac -> .java_wrapper

[Sandip Ghosh]

I would suggest if you have latest ignite backup, restore it. And check for the error again by doing fsck. If the problem still exist call HP for the replacement of the disk.

And if you don't have a ignite backup then try to take a ignite backup in this condition. otherwise you may face some more problem.

Sandip

[Rui Vilao]

No stale...

# lvdisplay -v /dev/vg00/lvol6
--- Logical volumes ---
LV Name /dev/vg00/lvol6
VG Name /dev/vg00
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 1024
Current LE 64
Allocated PE 64
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c0t6d0 64 64

--- Logical extents ---
LE PV1 PE1 Status 1
0000 /dev/dsk/c0t6d0 0219 current
0001 /dev/dsk/c0t6d0 0220 current
0002 /dev/dsk/c0t6d0 0221 current
0003 /dev/dsk/c0t6d0 0222 current
0004 /dev/dsk/c0t6d0 0223 current
0005 /dev/dsk/c0t6d0 0224 current
0006 /dev/dsk/c0t6d0 0225 current
0007 /dev/dsk/c0t6d0 0226 current
0008 /dev/dsk/c0t6d0 0227 current
0009 /dev/dsk/c0t6d0 0228 current
0010 /dev/dsk/c0t6d0 0229 current
0011 /dev/dsk/c0t6d0 0230 current
0012 /dev/dsk/c0t6d0 0231 current
0013 /dev/dsk/c0t6d0 0232 current
0014 /dev/dsk/c0t6d0 0233 current
0015 /dev/dsk/c0t6d0 0234 current
0016 /dev/dsk/c0t6d0 0235 current
0017 /dev/dsk/c0t6d0 0236 current
0018 /dev/dsk/c0t6d0 0237 current
0019 /dev/dsk/c0t6d0 0238 current
0020 /dev/dsk/c0t6d0 0239 current
0021 /dev/dsk/c0t6d0 0240 current
0022 /dev/dsk/c0t6d0 0241 current
0023 /dev/dsk/c0t6d0 0242 current
0024 /dev/dsk/c0t6d0 0243 current
0025 /dev/dsk/c0t6d0 0244 current
0026 /dev/dsk/c0t6d0 0245 current
0027 /dev/dsk/c0t6d0 0246 current
0028 /dev/dsk/c0t6d0 0247 current
0029 /dev/dsk/c0t6d0 0248 current
0030 /dev/dsk/c0t6d0 0249 current
0031 /dev/dsk/c0t6d0 0250 current
0032 /dev/dsk/c0t6d0 0251 current
0033 /dev/dsk/c0t6d0 0252 current
0034 /dev/dsk/c0t6d0 0253 current
0035 /dev/dsk/c0t6d0 0254 current
0036 /dev/dsk/c0t6d0 0255 current
0037 /dev/dsk/c0t6d0 0256 current
0038 /dev/dsk/c0t6d0 0257 current
0039 /dev/dsk/c0t6d0 0258 current
0040 /dev/dsk/c0t6d0 0259 current
0041 /dev/dsk/c0t6d0 0260 current
0042 /dev/dsk/c0t6d0 0261 current
0043 /dev/dsk/c0t6d0 0262 current
0044 /dev/dsk/c0t6d0 0263 current
0045 /dev/dsk/c0t6d0 0264 current
0046 /dev/dsk/c0t6d0 0265 current
0047 /dev/dsk/c0t6d0 0266 current
0048 /dev/dsk/c0t6d0 0267 current
0049 /dev/dsk/c0t6d0 0268 current
0050 /dev/dsk/c0t6d0 0269 current
0051 /dev/dsk/c0t6d0 0270 current
0052 /dev/dsk/c0t6d0 0271 current
0053 /dev/dsk/c0t6d0 0272 current
0054 /dev/dsk/c0t6d0 0273 current
0055 /dev/dsk/c0t6d0 0274 current
0056 /dev/dsk/c0t6d0 0275 current
0057 /dev/dsk/c0t6d0 0276 current
0058 /dev/dsk/c0t6d0 0277 current
0059 /dev/dsk/c0t6d0 0278 current
0060 /dev/dsk/c0t6d0 0279 current
0061 /dev/dsk/c0t6d0 0280 current
0062 /dev/dsk/c0t6d0 0281 current
0063 /dev/dsk/c0t6d0 0282 current

[Rui Vilao]

Thanks...
But, sorry, we have no ignite backup...

[pap]

Hi,

It seems that there are some bad blocks created on disk due to power outage. It will be a great thing for you if you can keep
1.full backup of the system.
2. FUll ignite image of the system on tape using :
make_tape_recovery -v -I a /dev/rmt/0mn inc_entire=vg00 inc_entire=vg01

Take one full ignite backup of system.

3. You can restore your /opr file system if you have recent backup available.

4.Run some diagnostics on your /opt disks using stm, xstm or offline diagnostics environment to find out what went bad on your disks.

5. Try to replace the disk as soon as possible as the number of bad blocks always try to increase and will create major problem later on.

Hope this helps.

-pap

[John Payne_2]

You might be in trouble at this point. This is problibly a thing you want to get HP in on.

John

[Rui Vilao]

I thank you all for your suggestions.

However two things are still not clear for me.

1. Is this a HW or a SW failure?

2. If this is a SW failure there must be a way to correct/recover this problem.
Apparently only 5 inodes are missing/corrupted (cf. syslog). I couldn't find the files
to which they belong with a find /opt/ -inum -print. Moreover there seems to
be a problem with 5 softlinks on /opt/java... One solution might be to re-install /opt/java
Another solution could be to re-create a filesystem, copy the content of /opt to it and
rename it to /opt.

What do you think about this?

Any further suggestion is appreciated,

Kind Regards,

Rui.

[John Carr_2]

Hi

the answer is both.

the power outage caused the system to fail which resulted in the corruption of the s/w on the disk.

John.

[John Payne_2]

You should be able to get away with one of those.

First try getting rid of /opt/java and running fsck. (This is the least invasive and least time consuming way.) If the fsck clears, reinstall java, and you are on your way. This will problibly work for you if java is the only corruption.

If not. Then it is really pointing to something else, and you could try making a new mount point and moving your best loved stuff, but if you have a backup of the system, it would be better to recreate aand move the essentials, then restore form tape...

Good Luck

John

[binil]

what is the size of that file system?. If there is free space in any of the other filesystem do a copy (as it is mountable)to there (using cpio with proper options),then format that partion and copy back.
If Disk is having Physical problem,then better to take a IgniteUX backup and restore after replacing the disk.

[Sanjay_6]

Hi,

Take whatevery backups you can. Recreate the filesystem. Restore from backups. Restore from whatever old backup you have.

Hope this helps.

Regds

[Sandip Ghosh]

If the missing files are related to /opt/java only, then swremove the java stuff. Then reinstall the Java. It should be fine. But after completing everything you should run a fsck on /opt to see if the disk has created any bad block or not. And this time I think you should not forget to take an ignite backup of the system.

Sandip

[Frank Slootweg]

> Is this a HW or a SW failure?

As others have mentioned, it probably is both, i.e. a hardware error resulting in a corrupt or/and unreadable filesystem.

To check for a hardware error, do a read test of the raw LV:

dd if=/dev/vg00/rlvol6 of=/dev/null bs=64k
(note the "r" before "lvol6")

Also do a read test of the raw disk/PV, in case other LVs are affected as well:

dd if=/dev/rdsk/c0t6d0 of=/dev/null bs=64k
(again note the "r")

If these give errors, then have the drive checked/fixed/replaced.

[R D SUNDAR RAJAN]

Hello,
It seems like the filesystem was resized . The size of the filesystem is 1024 MB . what is the fstyp -v /dev/vg00/lvol6 show as the size of the filesystem.
This type of error happens if the filesystem is larger than the volume. try increasing the volume and doing an fsck.

[Peter Kloetgen]

Hi Rui,

check out for a hardware error as described by the others using the dd- command with raw device files of logical volume and disk. For me it seems like you have a corrupted superblock. During the power failure perhaps these few files where in use and now they are corrupted. Easiest way seems to be to delete them and restore them from backup. You could also try to use a copy of your superblock, using /etc/sbtab. After using fsck- command with option -b your system will ask you for a reboot -n probably. Try this, if that doesn't work, use a backup.

Allways stay on the bright side of life!

Peter

[Mike_305]

Hi,

Have you try booting system in single user and doing full fsck on the filesystems or umount this one file system and running fsck.

fsck -F vxfs -o full /dev/vgXX/lvolY

Hope this helps.

Mike