Operating System - HP-UX
1748086 Members
5187 Online
108758 Solutions
New Discussion юеВ

ORA-01578: ORACLE data block corrupted

 
SOLVED
Go to solution
Ralph Grothe
Honored Contributor

ORA-01578: ORACLE data block corrupted

Hello,

DBAs catch the Oracle error given in the subject line when performing certain tasks.
They insist that this error relates to some HW defect.
However, I cannot identify such.
As they obtain another Oracle error which generously points to the assumed corrupt file,
(viz. ORA-01110: data file 1: '/export/oracle/dbs1/inwb01/sys/sys01inwb01.dbf')
I checked the volume for stale extends,
but everything looks ok.

Neither do stale LEs appear

# lvdisplay -v $(vgdisplay -v vg02|awk '/LV Name/{print$NF}')|grep -ci stale
0

nor stale PEs

# pvdisplay -v $(vgdisplay -v vg02|awk '/PV Name/{print$3}')|grep -i stale
Stale PE 0
Stale PE 0
Stale PE 0
Stale PE 0


The disks come from a A6218A HP RAID Box.
Unfortunately, I'm not too familiar with the plethora of admin commands for this thing.
From "man armdsp" I discovered that it supplies an -a switch which shows the state of fairly all components and ingredients.

When I perform this statement

armdsp -a $(armdsp -i $(uname -n)|awk '/nique/{print$NF}')

I get what I attached as an upload (sorry, for the waste of bandwidth)
I'm not knowledgable enough to understand all of that, but the gist looks pretty ok to me (especially the seperate per disk lines)

Is there anything else I could check to dismiss a HW problem?

Is a filesystem check sensible (in view of my other evidence), and can it only be performed on an unmounted filesystem, although this box is MCOE (i.e. contains stuff like OnlineJFS)?

Rgds.
Ralph
Madness, thy name is system administration
19 REPLIES 19
Sanjay_6
Honored Contributor

Re: ORA-01578: ORACLE data block corrupted

Hi Ralph,

If this is different from your other post, then check the syslog and the EMS log to see if any hardware error is logged over there.

If this is on the same system for which you have another post, that problem could be the cause of this error.

Hope this helps.

regds
Simon Hargrave
Honored Contributor

Re: ORA-01578: ORACLE data block corrupted

Do you have EMS running? If so this should have highlighted any hardware issues. Check /var/opt/resmon/log/event.log. There should also be ESM messages in syslog.log.

Check also dmesg output for SCSI errors, etc.

You can also run stm and get an Information on the disk in question. This should show you if there were any disk errors recorded. (select the disk in question, tools menu, information, run information). You'll see read errors/ write errors etc here.

If not, then try (at a quiet time) this simple test: -

dd if=/export/oracle/dbs1/inwb01/sys/sys01inwb01.dbf of=/dev/null

this will read through the whole file, and bin the results. The point is, if this succeeds, the disk itself isn't faulty and the problem lies with the data itself.
Ralph Grothe
Honored Contributor

Re: ORA-01578: ORACLE data block corrupted

EMS agents are running.
I checked with the (c) option in
/etc/opt/resmon/lbin/monconfig

The /var/opt/resmon/logs/event.log
has as its latest record an entry from Mar 7 2004.

No, this is totally unrelated to my other thread.
In fact these boxes are fairly recent rp7410 clustered irons, well equipped, opposed to the K-class case in the other thread.

Yes, I usually read from the disk device to /dev/null to verify that at least can be read from a suspicious disk.
Madness, thy name is system administration
Volker Borowski
Honored Contributor

Re: ORA-01578: ORACLE data block corrupted

Hi Ralph,

if the databse runs archivelog mode, I'd first recommend to not shut down this database until this one is fixed.
Stay with online backups!
Stop overwriting tapes which contain old redologs and backups at once !

Second: is this DB a copy of another one ?
If yes, did you do an online copy or an offline copy from a shutdown aborted source database ?
In this case there is a good chance that the 1578 is related to some recovery problems for segments that have been build with the nologging clause -> Details @ Metalink.

Let the DBA do a dbverify check on the file in charge. It should give you some additional info about the error.
If it does not find any error, we need to check how this error is reproduced.

Restore the datafile in charge from a backup to a seperate location and do a dbverify check on the restored file.
If the error is not in the restored file, and you have all redologs since then, you are able to recover this.
If the corruption is even in the oldest backed up file from where you have redologs until now you are in trouble.

Next step would be to try to export the database (if not too big).

What Oracle release are you on ?

Regards
Volker
Steven E. Protter
Exalted Contributor

Re: ORA-01578: ORACLE data block corrupted

Hi Ralph,

Nice to see you, sorry you are having trouble.

Unfortuneately, you may still have had a hardware problem.

Fire up xstm csm or mstm and run disk excersize on all involved disks. Even it its a LUN presented from a disk array, it needsd to test out properly.

We've had some glitches with our disk array itself, that corruped that data in oracle. We were forced to restore. Ours involved a firmware upgrade disrupting disk operation. The disks tested out fine after the disruption but the data on them was totally useless.

If a momentary disruption of your disk array occurred, there may be no record of it. dmesg or /var/adm/syslog/syslog.log may or may not show something. The disk array log might be a good place to look.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Volker Borowski
Honored Contributor

Re: ORA-01578: ORACLE data block corrupted

... forgot: The "bible" note for blockcorruption in Oracle is Metalink-Note:28814.1

V.
Ralph Grothe
Honored Contributor

Re: ORA-01578: ORACLE data block corrupted

Things are getting worse.
Clients now complain about bad performance.
Having looked with glance at it I can see that the servers are instead even underloaded.
I.e. no CPU usage, very little disk I/O, low network packet rates.
When I look at the wait stats of certain Oracle procs I can see that they are blocked on STRMS.
Looks like the Oracle ICP is impaired.
All four listeners on both cluster nodes are running.
If however, the bad data block that Oracle reports, refers to a table in the data directory that is responsible for keeping persistance of Oracle/SQLNet IPC it sounds plausible to me.
I also called for help from network and firewall admins to look at their hops if something crashed, resetted, overflew etc.
because the NIC settings (modes link speeds etc.) and routing look ok.

Volker,

how can I query the instance if it is running in archive logging mode?
I cannot identify the current parameter file.
There are too many, looks DBAs have done quite a bit of parameter testing.
Albeit, I can connect to an instance (n.b. sqlplus tells it's release 9.2.0.4.0) /as sysdba.
But I don't know the view from the data directory that I have to query.
Could you tell me (e.g. like v$instance)?
Btw. if I connect locally to an instance does Oracle use a Unix socket for IPC (i.e. do I require the listener to be running?)?
I will pass your valuable suggestions to our DBA, but I cannot reach her today anymore.
So please stay tuned to this thread.
DB backup/recovery seems a binary business, as your final statement suggests.
Either got everything, or nothing.

SEP,

first thing I tried was to look at the disks through stm.
But in the map there I cannot see the single disks of the RAID box.
Doesn't look like stm supports it.
Apart from that I can only run informational requests, since I lack knowledge of passwords for execrcising or other tasks.


Madness, thy name is system administration
Ralph Grothe
Honored Contributor

Re: ORA-01578: ORACLE data block corrupted

Volker,

what is a Metalink in Oracle lingo?
Anything comparable to a Unix (hard)link?
Madness, thy name is system administration
Sanjay_6
Honored Contributor

Re: ORA-01578: ORACLE data block corrupted

Hi Ralph,

Metalink for oracle is like ITRC is for hp.

Hope this helps.

Regds