topic Re: VXFS errors on mirrored root fs in Operating System - HP-UX

VXFS errors on mirrored root fs

Gordon Brown_5 — Thu, 12 Feb 2004 06:10:16 GMT

We had a bit of an accident in which /usr/lib/libc.2 was mv'ed into a sub directory.

This obviously upset everything. To fix the problem I booted from the recovery CD and mv'ed the file back to its correct resting place.

Since then I have noticed in syslog.log messages such as :-
Feb 12 10:06:31 cliff vmunix: msgcnt 1 vxfs: mesg 010: vx_ialloc - / file system
inode 113773 not free
Feb 12 10:06:31 cliff vmunix: msgcnt 2 vxfs: mesg 016: vx_ilisterr - / file syst
em error reading inode 113773

About 300 odd messages like this appeared in the log until this morning the system complained again that libc.2 was missing.

A reboot (via GSP + RS ) and automatic fsck cleared the problem and the system is working again; but the messages are appearing again.

Using find (-inum) to identify the file associated with the inode listed in the message above actually pointed to an inode on another file system.

It looks to me that my mirrored root file system, although physically mirrored, is not logically correct.

Is it possible that one half of the has mirror /usr/lib/libc.2 and the other /usr/lib/safe/libc.2 - if the system decides to access the file system through the second half of the mirror it cannot see libc.2.

Any ideas ? My intention is to
1. Split the mirror, shutdown the machine.
2. Restart with the recovery CD
3. Run a full structural check on / and ensure libc.2 is where it should be.
4. Restart
5. Re-mirror the root FS.

Hope to hear from you shortly ( Response Centre hasn't been much help).

Re: VXFS errors on mirrored root fs

RAC_1 — Thu, 12 Feb 2004 06:15:16 GMT

Boot into single user mode and fsck all file systems. This should correct it.

Re: VXFS errors on mirrored root fs

Mark Grant — Thu, 12 Feb 2004 06:16:00 GMT

This looks like filesystem corruption to me. First thing you do, and you do it now is a full backup of everything on every other filesystem and another one on the filesystem as it is now.

I don't think you can have a mirror that looks different in any way to the primary unless you have stale extents and that isn't going to put a file in a different place.

Do a full fsck on the root filesystem.

I personally would be tempted to re-install this machine if you can.

Re: VXFS errors on mirrored root fs

Gordon Brown_5 — Thu, 12 Feb 2004 06:17:38 GMT

Yes but surely the crash via reset and th eresulting bcheckrc fsck would have done that ? Th esystem was up about an hour when it started reporting vxfs errors again....

Re: VXFS errors on mirrored root fs

Paula J Frazer-Campbell — Thu, 12 Feb 2004 06:22:46 GMT

Hi

Try splitting the mirror, mounting the split and comparing the structure for libc.2.

If different on the mouinted mirror split rm it, unmount and then join it back to the root mirror.

BEFORE DOING THIS.

Full system backup
Make_tape_recovery
single user mode.

HTH

Paula

Re: VXFS errors on mirrored root fs

Gordon Brown_5 — Fri, 13 Feb 2004 05:01:46 GMT

Dear All,

Just to close the loop - I had a lengthy discussion with th eresponse centre and we agreed that if you mount a mirrored fs from th erecovery CD you are only working with one half of the mirror.

Because the mv command didn't physically move the file - it only affected the directory entries I ended up with a corrupted mirror; although the status flags indicated all was well. If the O/S attempted to read libc.2 via the second channel it failed to find it but access via the primary channel succeeded.

The cure was as follows.

lvreduce -m 0 /dev/vg00/lvol3 /dev/dsk/onsecondpath
reboot server to single user.(I used recovery CD but probably didn't need to )
fsck -y
reboot to multiuser.

Job done.

Thanks for your input and enjoy the points.

G