Oracle error after HP UX patch updation

[Mikej_2]

Do you have space left in /backup/archives ?
What gives :
df -k /backup/archives
and :
bdf /backup/archives

?

[Syed Salman Mansoor]

I have checked it earlier also, there is no issue of free space. We have more than 10 GB of free space (I think :))

[T G Manikandan]

did you check the syslog.log and the oracle log files for additional information please.

[Syed Salman Mansoor]

My syslog.log show

Mar 17 23:32:56 pibaspr vmunix:
Mar 17 23:32:56 pibaspr vmunix: SCSI: Write error -- dev: b 31 0x010100, errno: 22, resid: 0,
Mar 17 23:32:56 pibaspr vmunix: blkno: 142735392, sectno: 4848128, offset: 146161041408, bcount: 1048576.
Mar 17 23:32:56 pibaspr vmunix: blkno: 142736416, sectno: 4843354, offset: 146162089984, bcount: 1015808.
Mar 17 23:32:56 pibaspr vmunix: blkno: 142751776, sectno: 98304, offset: 146177818624, bcount: 1048576.
Mar 17 23:32:56 pibaspr vmunix: blkno: 142752800, sectno: 82402, offset: 146178867200, bcount: 1048576.
Mar 17 23:32:56 pibaspr vmunix:
Mar 17 23:35:00 pibaspr above message repeats 3 times
Mar 17 23:32:56 pibaspr vmunix: SCSI: Write error -- dev: b 31 0x010100, errno: 22, resid: 0,
Mar 17 23:35:00 pibaspr above message repeats 3 times

Oracle Log: I have already sent.

[T G Manikandan · Accepted solution]

grep 22 /usr/include/sys/errno.h points to **INVALID ARGUMENT**

Did you manually remount any of the mount points after the server restart.

I suspect if you had mount options like async for the file system.

Please let me know.

[Syed Salman Mansoor]

No I didn't mounted any filesystem after reboot. All were mounted automatically.

My fstab is

"/etc/fstab" 11 lines, 495 characters
/dev/vg00/lvol3 / vxfs delaylog 0 1
/dev/vg00/lvol1 /stand vxfs tranflush 0 1
/dev/vg00/lvol4 /tmp vxfs delaylog 0 2
/dev/vg00/lvol5 /home vxfs delaylog 0 2
/dev/vg00/lvol6 /opt vxfs delaylog 0 2
/dev/vg00/lvol7 /usr vxfs delaylog 0 2
/dev/vg00/lvol8 /var vxfs delaylog 0 2
/dev/vg00/oracle /oracle vxfs rw,suid,largefiles,delaylog,datainlog 0 2
/dev/dsk/c1t0d1 /oracledb vxfs rw,suid,delaylog,datainlog,largefiles 0 1
/dev/dsk/c1t0d2 /backup vxfs defaults 0 2
/dev/vg00/lv13 ... swap pri=1 0 2

Actually I forgot to look into fstab before applying patches but

# ls -lt /etc/fstab
-rw-r--r-- 1 root sys 495 Oct 12 00:40 /etc/fstab

Meaning it is not disturbed.

[Syed Salman Mansoor]

Also wanted to add that my disks are local scsi disks. I dont have SAN attached.

[T G Manikandan]

can you please attach the oracle alert log file please.

Thanks

[Syed Salman Mansoor]

Tue Mar 17 23:30:13 2009
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
SCN scheme 3
LICENSE_MAX_USERS = 0
SYS auditing is enabled
Starting up ORACLE RDBMS Version: 9.2.0.7.0.
System parameters with non-default values:
processes = 4000
sessions = 4405
timed_statistics = TRUE
resource_limit = TRUE
shared_pool_size = 637534208
sga_max_size = 4296498144
large_pool_size = 16777216
java_pool_size = 0
control_files = /oracle/home/oradata/pibassrv/control01.ctl, /oracle/home/oradata/pibassrv/control02.ctl, /oracle/home/oradata/pibassrv/control03.ctl
db_block_size = 8192
db_cache_size = 3254779904
compatible = 9.2.0.0.0
log_archive_start = FALSE
log_archive_dest = /backup/archives
log_archive_dest_1 =
log_archive_dest_2 =
standby_archive_dest = /oracledb/pibasdb/archive
log_archive_format = %t_%s.arc
db_file_multiblock_read_count= 16
fast_start_mttr_target = 300
undo_management = AUTO
undo_tablespace = UNDOTBS2
undo_retention = 10800
remote_os_roles = TRUE
remote_login_passwordfile= EXCLUSIVE
audit_sys_operations = TRUE
db_domain =
commit_point_strength = 200
instance_name = pibassrv
job_queue_processes = 10
hash_join_enabled = TRUE
background_dump_dest = /oracle/home/admin/pibassrv/bdump
user_dump_dest = /oracle/home/admin/pibassrv/udump
core_dump_dest = /oracle/home/admin/pibassrv/cdump
open_links = 200
open_links_per_instance = 200
audit_trail = DB
sort_area_size = 524288
db_name = pibassrv
open_cursors = 3000
star_transformation_enabled= FALSE
_no_or_expansion = TRUE
query_rewrite_enabled = FALSE
pga_aggregate_target = 209715200
PMON started with pid=2
DBW0 started with pid=3
LGWR started with pid=4
CKPT started with pid=5
SMON started with pid=6
RECO started with pid=7
CJQ0 started with pid=8
Tue Mar 17 23:30:15 2009
ALTER DATABASE MOUNT
Tue Mar 17 23:30:19 2009
Successful mount of redo thread 1, with mount id 1896047879
Tue Mar 17 23:30:19 2009
Database mounted in Exclusive Mode.
Completed: ALTER DATABASE MOUNT
Tue Mar 17 23:30:26 2009
alter database archivelog
Completed: alter database archivelog
Tue Mar 17 23:30:33 2009
alter database open
Tue Mar 17 23:30:33 2009
LGWR: Primary database is in CLUSTER CONSISTENT mode
Thread 1 opened at log sequence 1271
Current log# 4 seq# 1271 mem# 0: /oracle/home/oradata/pibassrv/log4.ora
Successful open of redo thread 1
Tue Mar 17 23:30:33 2009
SMON: enabling cache recovery
Tue Mar 17 23:30:33 2009
Successfully onlined Undo Tablespace 11.
Tue Mar 17 23:30:33 2009
SMON: enabling tx recovery
Tue Mar 17 23:30:33 2009
Database Characterset is WE8ISO8859P1
replication_dependency_tracking turned off (no async multimaster replication found)
Completed: alter database open
Tue Mar 17 23:31:18 2009
Errors in file /oracle/home/admin/pibassrv/bdump/pibassrv_reco_1997.trc:
ORA-02019: connection description for remote database not found
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
Tue Mar 17 23:31:18 2009
Errors in file /oracle/home/admin/pibassrv/bdump/pibassrv_reco_1997.trc:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
Tue Mar 17 23:31:18 2009
Errors in file /oracle/home/admin/pibassrv/bdump/pibassrv_reco_1997.trc:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
Tue Mar 17 23:31:18 2009
Errors in file /oracle/home/admin/pibassrv/bdump/pibassrv_reco_1997.trc:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
Tue Mar 17 23:31:18 2009
Errors in file /oracle/home/admin/pibassrv/bdump/pibassrv_reco_1997.trc:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
Tue Mar 17 23:31:18 2009
Errors in file /oracle/home/admin/pibassrv/bdump/pibassrv_reco_1997.trc:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325

..................

Tue Mar 17 23:31:40 2009
Thread 1 advanced to log sequence 1272
Current log# 1 seq# 1272 mem# 0: /oracle/home/oradata/pibassrv/log1.ora
Tue Mar 17 23:31:40 2009
ARCH: Evaluating archive log 4 thread 1 sequence 1271
ARCH: Beginning to archive log 4 thread 1 sequence 1271
Creating archive destination LOG_ARCHIVE_DEST_1: '/backup/archives/1_1271.arc'
ARCH: Completed archiving log 4 thread 1 sequence 1271
Tue Mar 17 23:31:51 2009
Errors in file /oracle/home/admin/pibassrv/bdump/pibassrv_reco_1997.trc:
ORA-02019: connection description for remote database not found
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
Tue Mar 17 23:31:51 2009
Errors in file /oracle/home/admin/pibassrv/bdump/pibassrv_reco_1997.trc:

................................
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
Tue Mar 17 23:32:56 2009
create tablespace abc datafile '/oracledb/pibasdb/datafiles/vicky.dbf' size 50M logging extent management local segment space management auto
Tue Mar 17 23:32:56 2009
ORA-19502 signalled during: create tablespace abc datafile '/oracledb/pibasdb/...
Tue Mar 17 23:33:52 2009
Errors in file /oracle/home/admin/pibassrv/bdump/pibassrv_reco_1997.trc:
ORA-02019: connection description for remote database not found
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
Tue Mar 17 23:33:52 2009
Errors in file /oracle/home/admin/pibassrv/bdump/pibassrv_reco_1997.trc:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found



===================================================================================



/oracle/home/admin/pibassrv/bdump/pibassrv_reco_1997.trc
Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.7.0 - Production
ORACLE_HOME = /oracle/home/Ora9i
System name: HP-UX
Node name: pibaspr
Release: B.11.23
Version: U
Machine: ia64
Instance name: pibassrv
Redo thread mounted by this instance: 1
Oracle process number: 7
Unix process pid: 1997, image: oracle@pibaspr (RECO)

*** SESSION ID:(6.1) 2009-03-17 23:31:18.430
*** 2009-03-17 23:31:18.430
ERROR, tran=3.13.177816, session#=1, ose=0:
ORA-02019: connection description for remote database not found
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
*** 2009-03-17 23:31:18.433
ERROR, tran=5.31.204647, session#=1, ose=0:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
*** 2009-03-17 23:31:18.434
ERROR, tran=6.15.169919, session#=1, ose=0:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
*** 2009-03-17 23:31:18.435
ERROR, tran=7.16.172083, session#=1, ose=0:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
*** 2009-03-17 23:31:18.435
ERROR, tran=3.43.178069, session#=1, ose=0:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
*** 2009-03-17 23:31:18.436
ERROR, tran=3.24.178078, session#=1, ose=0:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
*** 2009-03-17 23:31:18.437
ERROR, tran=7.28.172198, session#=1, ose=0:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
*** 2009-03-17 23:31:18.439
ERROR, tran=1.18.295041, session#=1, ose=0:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
*** 2009-03-17 23:31:18.440
ERROR, tran=10.9.316890, session#=1, ose=0:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
*** 2009-03-17 23:31:18.452
ERROR, tran=11.29.52287, session#=1, ose=0:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325

[T G Manikandan]

This looks like a HPUX issue, the patch database points to PHKL_38574 for 11.31.

hmm you are running HPUX 11.23,checking for patch for 11.23