Read IO Error while copying a file

[shirish_3]

OS Version HP UX 11.0
We have a dbf file in an Logical volume. While backing up that file with Data Protector we received Read IO error. While copying the file also I am getting same error. I tried dd as well as cpio but got same error. But strange thing is I am able to tar that file. while doing tar i get following message
#tar cvf ../1nov04.tar ./impdata.dbf
a ./impdata.dbf 2088976 blocks
./impdata.dbf: file changed size

When I restore this file on other machine, database is not coming up this file.

There are not disk errors. Application as well as database is running without any error.
Only one file is having problem among all the file systems.

In syslog.log following error is noticed
Oct 5 01:10:12 hp15 vmunix: msgcnt 1 vxfs: mesg 038: vx_dataioerr - /dev/vgfn/lvdata03 file system file data read error

I could not run fsck as could not get outage.
I am planning to move PEs of the logical volume to different PV ONLINE. Does anyone have other solution/suggestion...Please Suggest

Many Thanks

[RAC_1 · Accepted solution]

Chances are that, the disk that holds this lvol has problems.

diskinfo /dev/dsk/cxtxdx
Does it returns OK??? Any erros for the disk in EMS, syslog?? dmesg??

You may try as follows.

mount -F vxfs -o ro /dev/vgxx/lvolx /xxx
then try copying file somewhere else.

Anil

[shirish_3]

diskinfo is showing proper output.

I can not unmount the file system...No Outage

[Bill Hassell]

These are definitely disk errors. Your database and application simply have not accessed the area with the error. You will probably see the database error when you run a full export. The fact that tar, cpio, dd and DataProtector plus the disk error in syslog indicates that you have an unrecoverable error on your disk.

Since you have no outage available, you must be running a production system 24x7 and it sounds like you do not have disk mirroring. This is a serious situation (no disk mirroring) and you'll need to plan on how to salvage what you have so far, and how to get around the inevitable database error when you try to export or otherwise save /dev/vgfn/lvdata03. Note that you WILL get the same error when you try to move the PE's to another volume so that will not fix anything.

[Steven E. Protter]

Must be a bad disk.

At this point, I'd fire up a tool light cstm mstm or xstm and run excercize on every disk present.

One of them will come up red as in dead and fail the exercize. That disk will need to be replaced. The tablespace will have to be restored from backup.

SEP

[shirish_3]

I ran full database export, it is failing indicating read error.

Is there any way I can save my data?

[shirish_3]

Can I mirror the LV on Operating System Level
using lvextend.

Once mirroring is done, I will break the mirror and run fsck on mirrored copy.

Can any one comment on this?

Regards
Shirish

[Bill Hassell]

Mirroring will fail during lvsync. In order to mirror, the source must be readable. Mirroring must be done before the failure. fsck cannot fix bad spots on the disk. It is only used to straighten out the directory structure. In your case, the data area has an unreadable spot and there is no way to recover this filesystem except to create a new lvol on another disk and restore the data from your most recent backup.

Once you are back online with the restored lvol, you'll need to backup every lvol in /dev/vgfn. I would make a couple of backups because you are going to remove the VG, replace the disk, then recreate the VG and lvols, and restore them.

[Steven E. Protter]

The data sitting on the disk is bad. No amount of slight of hand will make it readable. Not fsck, not mirror/ux. Its useless. It has to be restored from some form of backup.

SEP

[Kent Ostby]

You can also use this to test out the disk:

dd if=/dev/rdsk/ of=/dev/null bs=64k

More then likely it will come back with a message that says somethign like:

X number of records in
X number of records out
I/O Error

Best regards,

Kent M. Ostby