topic Re: I/O error in Operating System - HP-UX

I/O error

shankar_3 — Fri, 08 Apr 2005 07:47:57 GMT

Hi Experts,

I need a help on following things..

Daily the server DB is going down.. Following are error..

KCF: write/open error block=0x3d2a online=1
file=159 /oradata/u005/ACCDW/ACCDW_gcldm_idx_04.dbf
error=27072 txt: 'HP-UX Error: 14: Bad address
Additional information: 15658'
Fri Apr 8 14:21:13 2005
Errors in file /oracle/admin/ACCDW/bdump/dbw0_8850_accdw.trc:
ORA-01242: data file suffered media failure: database in NOARCHIVELOG mode
ORA-01114: IO error writing block to file 159 (block # 15658)
ORA-01110: data file 159: '/oradata/u005/ACCDW/ACCDW_gcldm_idx_04.dbf'
ORA-27072: skgfdisp: I/O error
HP-UX Error: 14: Bad address
Additional information: 15658
DBW0: terminating instance due to error 1242

And oracle corresponding dsikd are not giving any I/O error while checking with dd command.

Help me !

With Regards,
Shankar

Re: I/O error

shankar_3 — Fri, 08 Apr 2005 07:50:54 GMT

Additionaly..

For all disks which is releated to oracle giving the following outputs in dd

dd if=/dev/rdsk/c0t2d0 of=/dev/null bs=2048 count=1000
1000+0 records in
1000+0 records out

Shankar

Re: I/O error

Robert-Jan Goossens_1 — Fri, 08 Apr 2005 07:55:13 GMT

Hi Shankar,

Could it be permission problem on the mountpoints or a largefiles problem on the filesystems?

Regards,
Robert-Jan

Re: I/O error

Peter Godron — Fri, 08 Apr 2005 07:55:35 GMT

Shankar,
the problem is proably no the reading, but the writing!

Is there any additional info in the trc file?
I would say you have a fault block/disk.
See metalink for rescuing/moving the good data.
May be worthwhile looking at a new disk!
Regards

Re: I/O error

Devender Khatana — Fri, 08 Apr 2005 08:07:19 GMT

Hi,

It appears to be a disk problem. That is why it is giving you a write error. Which disks are u using? starage or what? Are alternate paths available to the disk?
Moreover your dd is not checking the complete disk for reading.

HTH,
Devender

Re: I/O error

shankar_3 — Fri, 08 Apr 2005 23:10:04 GMT

let me check and confirm

But i need further more info on this..

Shankar

Re: I/O error

Bill Hassell — Sat, 09 Apr 2005 19:20:37 GMT

Your dd test is invalid--it's only testing a small part of the disk. Rerun the dd commands without a count and with a much larger blocksize:

dd if=/dev/rdsk/c0t2d0 of=/dev/null bs=128k

And look in syslog (/var/adm/syslog/syslog.log). The errors will be timestamped so look at syslog at Fri Apr 8 14:21:13 2005.

Note that the bad address error may be due to using lvreduce. lvreduce knows nothing about filesystems and without the very latest version of HP-UX, the very latest LVM and VxFS pathes and the priced-product Online JFS, you cannot reduce a filesystem with destroying the data. Using lvreduce on a filesystem means that the directory will provide an address to the VxFS filesystem that no longer exists.

Re: I/O error

Indira Aramandla — Sun, 10 Apr 2005 20:38:11 GMT

Hi Shankar,

In case of KCF: write/open errors, error code and additional information is the most important part. Check and verify to see if it is a Disk problem first. If the Harddisk is fine, then it could be the oracle data file have problme while writing to it. The File suffered media failure and there is some I/O error in writing to the datafile as seen in the aler.log.

This could had been caused because the file was locked by some backup process, and so the database opened failed. Make sure that the backup window does not exceed and does not clash with the DB open time. Online backup is recommended, to avoid these problems

If the datafile has become unusable then it either needs to be recovered or dropped. It depends on the type of data that is on the tablespace (data file).

If the datafile is dropped all data in it will be lost. But if you have a logical backup, then one of the solutions will be :- You have to turn the tablespace offline in order to issue an alter database open and then you should be able to drop the tablespace (datafile), recreate the tablespace with new datafile sizes, recover the data from my logic backup (via imp utility)

The other solution will be to recover in point in time before the problem occurred from a good backup

In general If the Media recovery is required then
restore the old backup of the datafile
recover the datafile/tablespace

For error â 'HP-UX Error: 14: Bad addressâ , there is a BUG. BUG 1533290 â Aynch read/write failedâ This is if the RDBMS release is Oracle Server - Enterprise Edition 8.1.7.0 and OS is HP-UX 11.0The fix or a workaround for this bug is to set the parameter DISK_ASYNCH_IO to FALSE and install HP-UX patch PHKL_22126

Re: I/O error

shankar_3 — Mon, 11 Apr 2005 08:22:00 GMT

hey folks,

I'm getting some log in syslog..

Is releted to DB ?

Apr 8 21:25:30 suboz032 vmunix: WARNING: ioctl: DLKM is not initialized
Apr 8 21:25:30 suboz032 vmunix: WARNING: ioctl: DLKM is not initialized

Mean while i am checking all disks releted to DB

Shankar

Re: I/O error

RAC_1 — Mon, 11 Apr 2005 09:03:49 GMT

Your DLKM error is about kernel module DLKM as a whole. Looks that your last kernel compile was not successful.

What does, following say??

echo "mod_initialized/D" | adb =k /stand/vmunix /dev/kmem

If it says 0, you will have to recompile the kernel again.

Anil

Re: I/O error

TwoProc — Mon, 11 Apr 2005 10:28:00 GMT

See if your file is bad from a disk perspective...

dd \
if=/oradata/u005/ACCDW/ACCDW_gcldm_idx_04.dbf \
of=/dev/null bs=64K

(that's one command above).

If the above fails complaining about a read error - then its a disk error.

Otherwise you may have bad data in your database blocks. Have you any recent RMAN backups (hot or cold)? If you have bad data blocks it would probably have been detected in the backup.

Re: I/O error

Yogeeraj_1 — Mon, 11 Apr 2005 23:09:56 GMT

hi,

Make sure your backups are all ok. You will have to recover your database. Hopefully, if you have your full backup and all the archived redologs you can get close to your last transactions...

good luck

regards
yogeeraj

Re: I/O error

shankar_3 — Tue, 19 Apr 2005 11:05:26 GMT

Hi Folks,

Thanks for valuable points ..

Customer did the hardware upgrade.

According to that swap increased.

Kernel reconfigured.

DB recreated..

Going well.

No hard disk problem found..

Bye

Re: I/O error

shankar_3 — Tue, 19 Apr 2005 11:06:44 GMT

I am closing my thread