File system error

[Helen French]

Hi everybody!

I have this message in the syslog.log:
"vmunix: vxfs: mesg 016: vx_ilisterr - /data1 file system error reading inode 120294"

I found this file system having issues while reading and unmounted it. Did a full fsck and fsck returned the error:
"vxfs fsck: file system had I/O error(s) on meta-data."

I checked all disks with stm and it didn't report any errors! lvdisplay and diskinfo are in good condition. I have total 9 disks in this LV and all seems good. What do you think the best way - a newfs or check the disks with hardware support and replace ? Or any other suggestions ?

OS verion is HP-UX 10.20 (vxfs file system)
Thanks!
Shiju

[Pete Randall]

Shiju,

See if you can isolate the problem to one disk by reading them with dd:
dd if=/dev/dsk/cxtxtx of=/dev/null bs=1024k

Then you can work with hardware support to get the bad one replaced.

Pete

[Hai Nguyen_1]

Shiju,

You may want to run fsck against the /data1 filesystem two, three times. I once got a disk error and the first fsck could not fix everything but the second one did.

Hai

[Helen French]

Thanks for the replies:

Pete, I was doing dd and not yet finished.
Hai, I tried fsck several times with different options.

I was just curious about any patches or anyother work-arounds as the stm didn't gave any hardware errors.

Thanks!
Shiju

[Jeff Schussele]

Hi Shiju,

Not sure how helpful it would be to you, but I sure would want to know what file the inode points to - do

ls -i /data1 | grep 120294

HTH,
Jeff

[Helen French]

Jeff:

Thanks for the reply. Actually I had inode errors for lot of inodes. I 've posted only one for simplicity. However, I wish I could check your option, but I can't becasue the file system is not mountable now! ..hehe.. sorry :((

any other suggestion ?

Thanks!
Shiju

[Hai Nguyen_1]

Shiju,

Have you tried fsck with "-o full" yet? It is used to FORCE fsck to make a full check of the metadata instead of just replaying the intent log.

Hai

[MANOJ SRIVASTAVA]

Hi Shiju


This can be possible if all the PV's were not switched on when the machien was getting witched on like you said the VG ahs 6 PV's so if some are connected via a Jamica Box and are not swithecd on or some such issue this can be piossible . The possible soultions are ( incase vgdispaly -v /dev/vgxx give the nos of PV and Current PV's as same )


1. Reboot the server but frist the external disks.

2. Still the same error then u need tor ecrete the lvol where this error is coming.



Manoj Srivastava

[Helen French]

Thanks for all your replies:

Hai, As mentioned in my first post, I did a FULL check on the file system.

Thanks Manoj: You were very close to the problem. The system was working good and the problem came inbetween. All the disks were ON. But as you mentioned, I saw a 'POWERFAIL' error in syslog about one disk and that was actually the disk failure itself. I did the dd test and got error on that disk now and checking all other disks now. I will post the results.

Thanks!
Shiju

[S.K. Chan]

This may/may-not be of any help. Take a look at patch PHKL_16751.
http://support2.itrc.hp.com/service/patch/patchDetail.do?patchid=PHKL_16751&context=hpux:800:10:20
It solves a few vx_ilisterr error of reading inode. Check the description of the patch.

[Helen French]

Hi S.K:

Thanks for pointing out! I already have that patch installed.

Shiju

[Pete Randall · Accepted solution]

So, Shiju, did you find the offensive drive?

Just Curious,
Pete

[Stefan Farrelly]

Weve seen these errors before, theyre almost always caused by a bad block developing on the drive/filesystem while it is in use, causing some data to get corrupted. Only way to fix it is copy off what you can, newfs it, reload your data.

Normally we run dd on it, as youve done, to prove there is a bad block. Almost always we successfully run mediainit to repair the bad block, then newfs it, and reload, and its fine.

[Frank Slootweg]

Only use mediainit as a *last* resort, i.e. *after* all other approaches.

On a modern disk, and with LVM bad block relocation enabled, mediainit should not be neccessary and may make things worse (without you knowing that it got worse).

For details, see for example Bill Hassell's response in this thread:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x4b34eea29889d611abdb0090277a778c,00.html

[Stefan Farrelly]

Frank,

the link you provided about Bills response - in it Bill Hassel says absolutely nothing about mediainit !!

[Frank Slootweg]

Stefan,

*I* talk about mediainit, its (non-)use, possible bad effects, etc., *and* talk about LVM bad block relocation.

*Bill's* response gives more detail about LVM bad block relocation. That is why I gave that pointer.

[Helen French]

Hello everybody!

Thanks again for all the replies.

Pete, I found the disk which is bad and replaced.

Stefan and Frank, thanks for your inputs. It was a HDD error and I replaced it, did vgcfgrestore, newfs and restoring data now.

Thanks!
Shiju