Oracle instances goes to down after using HP data Protector 6.0

[Amar Ujala]

Hi,

I am using data protector 6for backup.Backup is scheduled and running well on daily basis since 2 months.Tomorrow when I was taking a online backup of oracle10g manually, one of the instance of database goes down due to one datafile goes to offline.My DBA told us that there is some issues with HP data protector.

But Data protector behave normally and taking backup very well.


Thanks
Amar Ujala

[Aashique]

Hi,
I dont think so for DP the datafile goes offline.its not possible.

Thanks & Regards
Aashique

[Asif Sharif]

Salam Amar,

Check DP logs for more details, As it doesn't happen normally.

Regards,
Asif Sharif

[Amar Ujala]

Here is log generated by DP....

Normal] From: BSM@dialimb2 "ONLINE_DB1_WPSDB" Time: 5/27/2008 2:25:01 PM
OB2BAR application on "dialdb1" successfully started.

[Normal] From: ob2rman@dialdb1 "WPSDB" Time: 05/27/08 14:25:26
Starting backup of target database.

Net service name: WPSdb.
Instance status: OPEN.
Instance name: WPSDB1.
Database DBID = 1798825845.
Database control file type: CURRENT.
Database log mode: ARCHIVELOG.


[Normal] From: ob2rman@dialdb1 "WPSDB" Time: 05/27/08 14:25:31
Starting Oracle Recovery Manager.


Recovery Manager: Release 10.2.0.1.0 - Production on Tue May 27 14:25:32 2008

Copyright (c) 1982, 2005, Oracle. All rights reserved.

RMAN> CONNECT TARGET *
2> run {
3> allocate channel 'dev_0' type 'sbt_tape'
4> parms 'ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=WPSDB,OB2BARLIST=ONLINE_DB1_WPSDB,OB2BARHOSTNAME=dialdb1)';
5> backup incremental level 0
6> format 'ONLINE_DB1_WPSDB.dbf'
7> database;
8> backup
9> format 'ONLINE_DB1_WPSDB.dbf'
10> archivelog all;
11> backup
12> format 'ONLINE_DB1_WPSDB.dbf'
13> current controlfile;
14> }
15> EXIT
connected to target database: WPSDB (DBID=1798825845)
using target database control file instead of recovery catalog

allocated channel: dev_0
channel dev_0: sid=279 instance=WPSDB1 devtype=SBT_TAPE
channel dev_0: Data Protector A.06.00/PHSS_35908/PHSS_35909/DPSOL_00261/DPLNX_

Starting backup at 27-MAY-08
channel dev_0: starting incremental level 0 datafile backupset
channel dev_0: specifying datafile(s) in backupset
input datafile fno=00003 name=/u02/data/WPSDB/sysaux01.dbf
input datafile fno=00004 name=/u02/data/WPSDB/users01.dbf
input datafile fno=00015 name=/u02/data/WPSDB/undotbs03.dbf
input datafile fno=00001 name=/u02/data/WPSDB/system01.dbf
input datafile fno=00009 name=/u02/data/WPSDB/INSTANCE.dbf
input datafile fno=00008 name=/u02/data/WPSDB/INDEXTS.dbf
input datafile fno=00010 name=/u02/data/WPSDB/LOBTS.dbf
input datafile fno=00002 name=/u02/data/WPSDB/undotbs01.dbf
input datafile fno=00006 name=/u02/data/WPSDB/AUDITLOG.dbf
input datafile fno=00012 name=/u02/data/WPSDB/TEMPLATE.dbf
input datafile fno=00013 name=/u02/data/WPSDB/WORKITEM.dbf
input datafile fno=00007 name=/u02/data/WPSDB/COMP.dbf
input datafile fno=00011 name=/u02/data/WPSDB/STAFFQRY.dbf
input datafile fno=00014 name=/u02/data/WPSDB/SCHEDTS.dbf
channel dev_0: starting piece 1 at 27-MAY-08
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on dev_0 channel at 05/27/2008 14:25:43
RMAN-10038: database session for channel dev_0 terminated unexpectedly
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
ORA-01023: Cursor context not found (Invalid cursor number)
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on dev_0 channel at 05/27/2008 14:25:43
RMAN-10038: database session for channel dev_0 terminated unexpectedly
RMAN-06900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT row
RMAN-06901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rows
ORACLE error from target database:
ORA-01023: Cursor context not found (Invalid cursor number)

[Major] From: ob2rman@dialdb1 "WPSDB" Time: 05/27/08 14:25:44
The database reported error while performing requested operation.

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on dev_0 channel at 05/27/2008 14:25:43
RMAN-10038: database session for channel dev_0 terminated unexpectedly
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
ORA-01023: Cursor context not found (Invalid cursor number)
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on dev_0 channel at 05/27/2008 14:25:43
RMAN-10038: database session for channel dev_0 terminated unexpectedly
RMAN-06900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT row
RMAN-06901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rows
ORACLE error from target database:
ORA-01023: Cursor context not found (Invalid cursor number)


[Major] From: ob2rman@dialdb1 "WPSDB" Time: 05/27/08 14:25:45
Oracle Recovery Manager completed with errors.

[Major] From: ob2rman@dialdb1 "WPSDB" Time: 05/27/08 14:25:45
Backup of target database failed.

[Normal] From: BSM@dialimb2 "ONLINE_DB1_WPSDB" Time: 5/27/2008 2:25:23 PM
OB2BAR application on "dialdb1" disconnected.

[Normal] From: BSM@dialimb2 "ONLINE_DB1_WPSDB" Time: 5/27/2008 2:25:54 PM

Backup Statistics:

Session Queuing Time (hours) 0.00
----------------------------------------
Completed Disk Agents ........ 0
Failed Disk Agents ........... 0
Aborted Disk Agents .......... 0
----------------------------------------
Disk Agents Total ........... 0
========================================
Completed Media Agents ....... 0
Failed Media Agents .......... 0
Aborted Media Agents ......... 0
----------------------------------------
Media Agents Total .......... 0
========================================
Mbytes Total ................. 0 MB
Used Media Total ............. 0
Disk Agent Errors Total ...... 0

[Asif Sharif]

http://forums12.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1211950767717+28353475&threadId=893404
http://forums12.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1211950747161+28353475&threadId=884493

Regards,
Asif Sharif

[Yogeeraj_1]

hi Amar,

Questions:
- Did you install any patches recently?

- What does your alert_.log show?

revert!
kind regards
yogeeraj

[Amar Ujala]

Hi,

There is no any patch update recently...

I am attaching the alert_SID.log.


Thanks
Amar Ujala

[Yogeeraj_1]

hi Amar,

If you look at the log file, it seems more a problem at the Oracle Database level:


...
Tue May 27 14:25:35 2008
KCF: write/open error block=0x7352 online=1
file=4 /u02/data/WPSDB/users01.dbf
error=17500 txt: 'ODM ERROR V-41-4-3-261-12 Not enough space'
Automatic datafile offline due to write error on
file 4: /u02/data/WPSDB/users01.dbf
Tue May 27 14:25:36 2008
Errors in file /u08/oracle/admin/WPSDB1/bdump/wpsdb1_dbw0_15988.trc:
ORA-00600: internal error code, arguments: [KSFD_DECAIOPC], [0xC00000007C036CC8], [], [], [], [], [], []
Tue May 27 14:25:37 2008
Errors in file /u08/oracle/admin/WPSDB1/bdump/wpsdb1_dbw0_15988.trc:
ORA-00600: internal error code, arguments: [KSFD_DECAIOPC], [0xC00000007C036CC8], [], [], [], [], [], []
DBW0: terminating instance due to error 471
Tue May 27 14:25:37 2008
Trace dumping is performing id=[cdmp_20080527142537]
Tue May 27 14:25:40 2008
Dump system state for local instance only
System State dumped to trace file /u08/oracle/admin/WPSDB1/bdump/wpsdb1_diag_15947.trc
Tue May 27 14:25:42 2008
Instance terminated by DBW0, pid = 15988
Tue May 27 14:26:39 2008
...


The instance is crashing because of a problem with datafile "/u02/data/WPSDB/users01.dbf".

I would suggest that you contact Oracle Support for further troubleshooting and corrective action. (metalink.oracle.com)

Good luck!
kind regards
yogeeraj

[Amar Ujala]

Is there any requirement of update the patch i.e phkl_35334.It is related with ODM.