1826043 Members
3471 Online
109690 Solutions
New Discussion

Re: can not fsck

 
SOLVED
Go to solution
Michael_33
Regular Advisor

can not fsck

Hi all,

when I do a fsck -F vxfs -y /dev/vg00/lvol7
it said " had I/O error on meta data", I go to single user mode, fsck again, and got the same
error.

any idea?
19 REPLIES 19
Stefan Farrelly
Honored Contributor

Re: can not fsck


Add on the -o full option and retry. If you still have the problem you may need to check you have the latest VXFS/JFS patches installed.

Im from Palmerston North, New Zealand, but somehow ended up in London...
steven Burgess_2
Honored Contributor

Re: can not fsck

Hi Micheal

This can be down to bad blocks on the area of disk your lvol resides. I had a similar issue the other day and found the only way to clear it was to

Backup the data in that area.

Luckily for me it was only /tmp

I then had to run a newfs on that lvol

In your case

newfs -F vxfs /dev/vg00/rlvol7

(Note the newfs is performed on the raw volume)

HTH

Steve
take your time and think things through
steven Burgess_2
Honored Contributor

Re: can not fsck

Hi again

Try Stefan's option first - I did run -o full though and it still didn't cure it

Regards

Steve
take your time and think things through
Rainer von Bongartz
Honored Contributor

Re: can not fsck


I suspect you got a faulty disk

Run cstm to confirm if disk is faulty
# cstm
cstm>map
cstm>sel dev
cstm>info
cstm>infolog
He's a real UNIX Man, sitting in his UNIX LAN making all his UNIX plans for nobody ...
steven Burgess_2
Honored Contributor

Re: can not fsck

Micheal

Have you got any errors messages in the syslog or from dmesg. Somthing with regard to

inode number , vxfs , /dev/vg00/lvol7

HTH

Steve
take your time and think things through
Steven Sim Kok Leong
Honored Contributor

Re: can not fsck

Hi,

To check whether the disk is faulty, you can also use dd such as:

# dd if=/dev/vg00/lvol7 of=/dev/null bs=8192

or

# dd if=/dev/dsk/cXtYdZ of=/dev/null bs=8192

If the dd is unsuccessful, you have a faulty disk which cannot be read properly.

Hope this helps. Regards.

Steven Sim Kok Leong
Michael_33
Regular Advisor

Re: can not fsck

thanks all,

1. -o still the same error

2. #vgdisplay -v vg00
it shown lvol7 belongs to /dev/dsk/c0t6d0

3.ctsm shows error N/A

#diskinfo -v /dev/rdsk/c0t6d0
gives this message:

additional inquiry bytes: (32)31 (33)30 (34)36
...... (139)0

any idea?
Michael_33
Regular Advisor

Re: can not fsck

hi Steven,

how long did dd if/of take?
so far I run 5mins, no messages return.
steven Burgess_2
Honored Contributor

Re: can not fsck

Micheal

I think we can rule out a problem with the whole disk as you would have errors elsewhere

A newfs will almost certainly clear your problem. A little harsh some may add. What data have you got in that lvol ? Back it up first > single user and perform action

HTH

Steve
take your time and think things through
Michael_33
Regular Advisor

Re: can not fsck

grear steven,

there are many error message in syslog.log
like this:

error reading inode 79625
steven Burgess_2
Honored Contributor

Re: can not fsck

Micheal

You can ctrl c to stop the dd, you will get somthing like

in
out

If the 2 figures are the same then you are ok

If you get I/O ERROR then you have a problem with the disk. In which case you have nothing to lose running the newfs

HTH

Steve
take your time and think things through
steven Burgess_2
Honored Contributor

Re: can not fsck

Hi mate

exactly the same symptoms I had with /tmp

This also caused issues with users who were trying to collect data from that area. I could see it from the terminal as root. They were using ftp applications which couldn't read the data in that area. Plus other things such as my backups were failing

Steve
take your time and think things through
Michael_33
Regular Advisor

Re: can not fsck

to steven:

are you sure of this:
newfs -F vxfs /dev/vg00/rlvol7

btw,I have the backup.
steven Burgess_2
Honored Contributor

Re: can not fsck

Hi Micheal

go to single user
shutdown -y 0
filesystem will be unmounted
perform the newfs
reboot
restore your data

Regards

Steve

take your time and think things through
Michael_33
Regular Advisor

Re: can not fsck

hi steven,

ctrl -c
I got this:

1269500 in
1269500 out

it seems the harddisk is ok, right?
steven Burgess_2
Honored Contributor

Re: can not fsck

Micheal

Yes, your input and output to the disk is ok. Therefore it's just an area on the disk which is bad. I know it seems a radical step having to wipe the data off an lvol then restoring it. I have been a little worried in the past following actions that HP have given me.

If you want official advise on how to clear this and have a support contract then I would get an engineer to confirm the actions. That way you having something to fall back on if anything should go wrong.

For sound peace of mind

Steve
take your time and think things through
George Abraham_1
Regular Advisor
Solution

Re: can not fsck

hai

I am pretty sure that your disk has some bad areas.. Better to get it replaced before it can give you worse problems...

Use stm/ cstm to do a through checking on the disk...


Cheero
George
keep smiling
Bill Hassell
Honored Contributor

Re: can not fsck

fsck can *never* fix I/O errors. I/O errors are hardware failures, and fsck is a tool to fix logical and structural errors in the directories. The syslog error are very important to be checked and acted on ASAP. Consider writing a cron job that looks for problems every hour or two each day and mails the results to your sysadmin team. Something like this:

grep -i -e crit -e warn -e err -e fail -e problem /var/adm/syslog/syslog.log

Now, LVM is smart enough to fix this *IF* bad block relocation has been turned on. Use lvchange -r y to turn on the feature for each logical volume (note: some big diskarrays require this to be off and use diskarray tools to fix the error).

Once BadBlockRelocation is on, the next attepmt to write to the bad spot will cause the relocation to occur and all will be well again. Since you don't have any control over whether directory blocks and metadata is updated, newfs will handle this for you. Don't try to just restore the data without newfs. The directory metadata is corrupted and even though it may eventually get rewritten, some of it must be intact to know where to put the new data.

Note that a bad spot will always stay bad (can't be read) and will get worse with age. If this disk is not mirrored, get the Mirror/UX product, mirror your disks and plan for a replacement. If these are old (and likely very small) disks, look at a complete upgrade with mirroring.


Bill Hassell, sysadmin
steven Burgess_2
Honored Contributor

Re: can not fsck

Hi Michael

The advice from Bill is a sound as we are ever going to hear. From the prophets mouth I would say.

So - we don't have IO errors, so physical connectivity is ok - timeouts, termination etc

We do have a bad spot - which , as Bill advises a newfs will clear then a restore of the data in that area. But , for a disk to develop bad spots is a sign that it is on its way out so a replacement should be organised.

I'm off for a game of football now

Good luck

Post how you got on with the fix

Regards

Steve
take your time and think things through