topic Re: File system error in Operating System - HP-UX

File system error

Helen French — Tue, 06 Aug 2002 17:36:21 GMT

Hi everybody!

I have this message in the syslog.log:
"vmunix: vxfs: mesg 016: vx_ilisterr - /data1 file system error reading inode 120294"

I found this file system having issues while reading and unmounted it. Did a full fsck and fsck returned the error:
"vxfs fsck: file system had I/O error(s) on meta-data."

I checked all disks with stm and it didn't report any errors! lvdisplay and diskinfo are in good condition. I have total 9 disks in this LV and all seems good. What do you think the best way - a newfs or check the disks with hardware support and replace ? Or any other suggestions ?

OS verion is HP-UX 10.20 (vxfs file system)
Thanks!
Shiju

Re: File system error

Pete Randall — Tue, 06 Aug 2002 17:39:38 GMT

Shiju,

See if you can isolate the problem to one disk by reading them with dd:
dd if=/dev/dsk/cxtxtx of=/dev/null bs=1024k

Then you can work with hardware support to get the bad one replaced.

Pete

Re: File system error

Hai Nguyen_1 — Tue, 06 Aug 2002 17:42:58 GMT

Shiju,

You may want to run fsck against the /data1 filesystem two, three times. I once got a disk error and the first fsck could not fix everything but the second one did.

Hai

Re: File system error

Helen French — Tue, 06 Aug 2002 17:59:57 GMT

Thanks for the replies:

Pete, I was doing dd and not yet finished.
Hai, I tried fsck several times with different options.

I was just curious about any patches or anyother work-arounds as the stm didn't gave any hardware errors.

Thanks!
Shiju

Re: File system error

Jeff Schussele — Tue, 06 Aug 2002 18:01:43 GMT

Hi Shiju,

Not sure how helpful it would be to you, but I sure would want to know what file the inode points to - do

ls -i /data1 | grep 120294

HTH,
Jeff

Re: File system error

Helen French — Tue, 06 Aug 2002 18:05:54 GMT

Jeff:

Thanks for the reply. Actually I had inode errors for lot of inodes. I 've posted only one for simplicity. However, I wish I could check your option, but I can't becasue the file system is not mountable now! ..hehe.. sorry :((

any other suggestion ?

Thanks!
Shiju

Re: File system error

Hai Nguyen_1 — Tue, 06 Aug 2002 18:13:56 GMT

Shiju,

Have you tried fsck with "-o full" yet? It is used to FORCE fsck to make a full check of the metadata instead of just replaying the intent log.

Hai

Re: File system error

MANOJ SRIVASTAVA — Tue, 06 Aug 2002 18:15:23 GMT

Hi Shiju

This can be possible if all the PV's were not switched on when the machien was getting witched on like you said the VG ahs 6 PV's so if some are connected via a Jamica Box and are not swithecd on or some such issue this can be piossible . The possible soultions are ( incase vgdispaly -v /dev/vgxx give the nos of PV and Current PV's as same )

1. Reboot the server but frist the external disks.

2. Still the same error then u need tor ecrete the lvol where this error is coming.

Manoj Srivastava

Re: File system error

Helen French — Tue, 06 Aug 2002 18:40:33 GMT

Thanks for all your replies:

Hai, As mentioned in my first post, I did a FULL check on the file system.

Thanks Manoj: You were very close to the problem. The system was working good and the problem came inbetween. All the disks were ON. But as you mentioned, I saw a 'POWERFAIL' error in syslog about one disk and that was actually the disk failure itself. I did the dd test and got error on that disk now and checking all other disks now. I will post the results.

Thanks!
Shiju

Re: File system error

S.K. Chan — Tue, 06 Aug 2002 18:51:36 GMT

This may/may-not be of any help. Take a look at patch PHKL_16751.
http://support2.itrc.hp.com/service/patch/patchDetail.do?patchid=PHKL_16751&context=hpux:800:10:20
It solves a few vx_ilisterr error of reading inode. Check the description of the patch.

Re: File system error

Helen French — Tue, 06 Aug 2002 19:05:36 GMT

Hi S.K:

Thanks for pointing out! I already have that patch installed.

Shiju

Re: File system error

Pete Randall — Wed, 07 Aug 2002 09:15:23 GMT

So, Shiju, did you find the offensive drive?

Just Curious,
Pete

Re: File system error

Stefan Farrelly — Wed, 07 Aug 2002 09:33:41 GMT

Weve seen these errors before, theyre almost always caused by a bad block developing on the drive/filesystem while it is in use, causing some data to get corrupted. Only way to fix it is copy off what you can, newfs it, reload your data.

Normally we run dd on it, as youve done, to prove there is a bad block. Almost always we successfully run mediainit to repair the bad block, then newfs it, and reload, and its fine.

Re: File system error

Frank Slootweg — Wed, 07 Aug 2002 10:57:25 GMT

Only use mediainit as a *last* resort, i.e. *after* all other approaches.

On a modern disk, and with LVM bad block relocation enabled, mediainit should not be neccessary and may make things worse (without you knowing that it got worse).

For details, see for example Bill Hassell's response in this thread:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x4b34eea29889d611abdb0090277a778c,00.html

Re: File system error

Stefan Farrelly — Wed, 07 Aug 2002 11:30:18 GMT

Frank,

the link you provided about Bills response - in it Bill Hassel says absolutely nothing about mediainit !!

Re: File system error

Frank Slootweg — Wed, 07 Aug 2002 11:38:33 GMT

Stefan,

*I* talk about mediainit, its (non-)use, possible bad effects, etc., *and* talk about LVM bad block relocation.

*Bill's* response gives more detail about LVM bad block relocation. That is why I gave that pointer.

Re: File system error

Helen French — Wed, 07 Aug 2002 15:05:09 GMT

Hello everybody!

Thanks again for all the replies.

Pete, I found the disk which is bad and replaced.

Stefan and Frank, thanks for your inputs. It was a HDD error and I replaced it, did vgcfgrestore, newfs and restoring data now.

Thanks!
Shiju