cancel
Showing results for 
Search instead for 
Did you mean: 

I/O error

shankar_3
Advisor

I/O error

Hi Experts,

I need a help on following things..

Daily the server DB is going down.. Following are error..

KCF: write/open error block=0x3d2a online=1
file=159 /oradata/u005/ACCDW/ACCDW_gcldm_idx_04.dbf
error=27072 txt: 'HP-UX Error: 14: Bad address
Additional information: 15658'
Fri Apr 8 14:21:13 2005
Errors in file /oracle/admin/ACCDW/bdump/dbw0_8850_accdw.trc:
ORA-01242: data file suffered media failure: database in NOARCHIVELOG mode
ORA-01114: IO error writing block to file 159 (block # 15658)
ORA-01110: data file 159: '/oradata/u005/ACCDW/ACCDW_gcldm_idx_04.dbf'
ORA-27072: skgfdisp: I/O error
HP-UX Error: 14: Bad address
Additional information: 15658
DBW0: terminating instance due to error 1242

And oracle corresponding dsikd are not giving any I/O error while checking with dd command.

Help me !

With Regards,
Shankar
13 REPLIES
shankar_3
Advisor

Re: I/O error

Additionaly..

For all disks which is releated to oracle giving the following outputs in dd

dd if=/dev/rdsk/c0t2d0 of=/dev/null bs=2048 count=1000
1000+0 records in
1000+0 records out

Shankar
Robert-Jan Goossens_1
Honored Contributor

Re: I/O error

Hi Shankar,

Could it be permission problem on the mountpoints or a largefiles problem on the filesystems?

Regards,
Robert-Jan
Peter Godron
Honored Contributor

Re: I/O error

Shankar,
the problem is proably no the reading, but the writing!

Is there any additional info in the trc file?
I would say you have a fault block/disk.
See metalink for rescuing/moving the good data.
May be worthwhile looking at a new disk!
Regards
Devender Khatana
Honored Contributor

Re: I/O error

Hi,

It appears to be a disk problem. That is why it is giving you a write error. Which disks are u using? starage or what? Are alternate paths available to the disk?
Moreover your dd is not checking the complete disk for reading.

HTH,
Devender
Impossible itself mentions "I m possible"
shankar_3
Advisor

Re: I/O error

let me check and confirm

But i need further more info on this..

Shankar
Bill Hassell
Honored Contributor

Re: I/O error

Your dd test is invalid--it's only testing a small part of the disk. Rerun the dd commands without a count and with a much larger blocksize:

dd if=/dev/rdsk/c0t2d0 of=/dev/null bs=128k

And look in syslog (/var/adm/syslog/syslog.log). The errors will be timestamped so look at syslog at Fri Apr 8 14:21:13 2005.

Note that the bad address error may be due to using lvreduce. lvreduce knows nothing about filesystems and without the very latest version of HP-UX, the very latest LVM and VxFS pathes and the priced-product Online JFS, you cannot reduce a filesystem with destroying the data. Using lvreduce on a filesystem means that the directory will provide an address to the VxFS filesystem that no longer exists.


Bill Hassell, sysadmin
Indira Aramandla
Honored Contributor

Re: I/O error

Hi Shankar,

In case of KCF: write/open errors, error code and additional information is the most important part. Check and verify to see if it is a Disk problem first. If the Harddisk is fine, then it could be the oracle data file have problme while writing to it. The File suffered media failure and there is some I/O error in writing to the datafile as seen in the aler.log.

This could had been caused because the file was locked by some backup process, and so the database opened failed. Make sure that the backup window does not exceed and does not clash with the DB open time. Online backup is recommended, to avoid these problems

If the datafile has become unusable then it either needs to be recovered or dropped. It depends on the type of data that is on the tablespace (data file).

If the datafile is dropped all data in it will be lost. But if you have a logical backup, then one of the solutions will be :- You have to turn the tablespace offline in order to issue an alter database open and then you should be able to drop the tablespace (datafile), recreate the tablespace with new datafile sizes, recover the data from my logic backup (via imp utility)

The other solution will be to recover in point in time before the problem occurred from a good backup

In general If the Media recovery is required then
restore the old backup of the datafile
recover the datafile/tablespace

For error â 'HP-UX Error: 14: Bad addressâ , there is a BUG. BUG 1533290 â Aynch read/write failedâ This is if the RDBMS release is Oracle Server - Enterprise Edition 8.1.7.0 and OS is HP-UX 11.0The fix or a workaround for this bug is to set the parameter DISK_ASYNCH_IO to FALSE and install HP-UX patch PHKL_22126
Never give up, Keep Trying
shankar_3
Advisor

Re: I/O error

hey folks,

I'm getting some log in syslog..

Is releted to DB ?

Apr 8 21:25:30 suboz032 vmunix: WARNING: ioctl: DLKM is not initialized
Apr 8 21:25:30 suboz032 vmunix: WARNING: ioctl: DLKM is not initialized

Mean while i am checking all disks releted to DB

Shankar


RAC_1
Honored Contributor

Re: I/O error

Your DLKM error is about kernel module DLKM as a whole. Looks that your last kernel compile was not successful.

What does, following say??

echo "mod_initialized/D" | adb =k /stand/vmunix /dev/kmem

If it says 0, you will have to recompile the kernel again.

Anil
There is no substitute to HARDWORK
TwoProc
Honored Contributor

Re: I/O error

See if your file is bad from a disk perspective...

dd \
if=/oradata/u005/ACCDW/ACCDW_gcldm_idx_04.dbf \
of=/dev/null bs=64K

(that's one command above).

If the above fails complaining about a read error - then its a disk error.

Otherwise you may have bad data in your database blocks. Have you any recent RMAN backups (hot or cold)? If you have bad data blocks it would probably have been detected in the backup.


We are the people our parents warned us about --Jimmy Buffett
Yogeeraj_1
Honored Contributor

Re: I/O error

hi,

Make sure your backups are all ok. You will have to recover your database. Hopefully, if you have your full backup and all the archived redologs you can get close to your last transactions...

good luck

regards
yogeeraj
No person was ever honoured for what he received. Honour has been the reward for what he gave (clavin coolidge)
shankar_3
Advisor

Re: I/O error

Hi Folks,

Thanks for valuable points ..

Customer did the hardware upgrade.

According to that swap increased.

Kernel reconfigured.

DB recreated..

Going well.

No hard disk problem found..

Bye
shankar_3
Advisor

Re: I/O error

I am closing my thread