topic Re: Oracle error after HP UX patch updation in Operating System - HP-UX

Oracle error after HP UX patch updation

Syed Salman Mansoor — Tue, 17 Mar 2009 04:37:27 GMT

I recently updated my HP UX 11.23 to latest available patches. Everything seems to be working fine but on Oracle 9i following error occured while creating archives.

Errors in file /oracle/home/admin/pibassrv/bdump/pibassrv_arc1_557.trc:

ORA-19502: write error on file "/backup/archives/1_85.arc", blockno 1 (blocksize=1024)

ORA-27072: skgfdisp: I/O error

HPUX-ia64 Error: 22: Invalid argument

Can anybody have any clue?

Re: Oracle error after HP UX patch updation

T G Manikandan — Tue, 17 Mar 2009 08:29:12 GMT

You need to check the disk for the mountpoint /backup, check whether its online.

Also check syslog.log to see additional information for this error.

Re: Oracle error after HP UX patch updation

Jozef_Novak — Tue, 17 Mar 2009 08:46:09 GMT

Hello,

I did a quick search on this topic, all results suggest to check the file that is unable to be written to. This might include incorrect file permissions, filesystem full or a disk HW error.

Jozef

Re: Oracle error after HP UX patch updation

Syed Salman Mansoor — Tue, 17 Mar 2009 09:05:35 GMT

/backup filesystem is mounted. Space is also available and permission is 644 with owner is oracle.

Re: Oracle error after HP UX patch updation

T G Manikandan — Tue, 17 Mar 2009 09:11:20 GMT

ls -ld /backup
ls -ld /backup/archives

it should be 755 and not 644, please check.

Re: Oracle error after HP UX patch updation

Steven E. Protter — Tue, 17 Mar 2009 10:39:16 GMT

Shalom,

Try relinking oracle.

That usually solvesproblems like this.

SEP

Re: Oracle error after HP UX patch updation

T G Manikandan — Tue, 17 Mar 2009 11:30:33 GMT

Steven is right, relinking oracle should solve most of these issues unless you did not explicitly change permissions for these directories.

Re: Oracle error after HP UX patch updation

Syed Salman Mansoor — Tue, 17 Mar 2009 11:35:50 GMT

Yes. /backup have 755 and /backup/archive have 777.

Re: Oracle error after HP UX patch updation

rariasn — Tue, 17 Mar 2009 11:44:43 GMT

Hi,

Stop all Oracle process, and

$ relink all

rgs.

Re: Oracle error after HP UX patch updation

Syed Salman Mansoor — Tue, 17 Mar 2009 18:03:31 GMT

I relinked the oracle binaries but still facing same issue. If I make datafiles and archives < 4 MB it works fine. But when it exceeds this size again same issue occurs. Please advice as this is my production database.

Re: Oracle error after HP UX patch updation

Mikej_2 — Tue, 17 Mar 2009 22:33:26 GMT

Do you have space left in /backup/archives ?
What gives :
df -k /backup/archives
and :
bdf /backup/archives

?

Re: Oracle error after HP UX patch updation

Syed Salman Mansoor — Wed, 18 Mar 2009 03:54:04 GMT

I have checked it earlier also, there is no issue of free space. We have more than 10 GB of free space (I think :))

Re: Oracle error after HP UX patch updation

T G Manikandan — Wed, 18 Mar 2009 08:59:01 GMT

did you check the syslog.log and the oracle log files for additional information please.

Re: Oracle error after HP UX patch updation

Syed Salman Mansoor — Wed, 18 Mar 2009 15:24:50 GMT

My syslog.log show

Mar 17 23:32:56 pibaspr vmunix:
Mar 17 23:32:56 pibaspr vmunix: SCSI: Write error -- dev: b 31 0x010100, errno: 22, resid: 0,
Mar 17 23:32:56 pibaspr vmunix: blkno: 142735392, sectno: 4848128, offset: 146161041408, bcount: 1048576.
Mar 17 23:32:56 pibaspr vmunix: blkno: 142736416, sectno: 4843354, offset: 146162089984, bcount: 1015808.
Mar 17 23:32:56 pibaspr vmunix: blkno: 142751776, sectno: 98304, offset: 146177818624, bcount: 1048576.
Mar 17 23:32:56 pibaspr vmunix: blkno: 142752800, sectno: 82402, offset: 146178867200, bcount: 1048576.
Mar 17 23:32:56 pibaspr vmunix:
Mar 17 23:35:00 pibaspr above message repeats 3 times
Mar 17 23:32:56 pibaspr vmunix: SCSI: Write error -- dev: b 31 0x010100, errno: 22, resid: 0,
Mar 17 23:35:00 pibaspr above message repeats 3 times

Oracle Log: I have already sent.

Re: Oracle error after HP UX patch updation

T G Manikandan — Wed, 18 Mar 2009 15:45:51 GMT

grep 22 /usr/include/sys/errno.h points to **INVALID ARGUMENT**

Did you manually remount any of the mount points after the server restart.

I suspect if you had mount options like async for the file system.

Please let me know.

Re: Oracle error after HP UX patch updation

Syed Salman Mansoor — Wed, 18 Mar 2009 16:02:39 GMT

No I didn't mounted any filesystem after reboot. All were mounted automatically.

My fstab is

"/etc/fstab" 11 lines, 495 characters
/dev/vg00/lvol3 / vxfs delaylog 0 1
/dev/vg00/lvol1 /stand vxfs tranflush 0 1
/dev/vg00/lvol4 /tmp vxfs delaylog 0 2
/dev/vg00/lvol5 /home vxfs delaylog 0 2
/dev/vg00/lvol6 /opt vxfs delaylog 0 2
/dev/vg00/lvol7 /usr vxfs delaylog 0 2
/dev/vg00/lvol8 /var vxfs delaylog 0 2
/dev/vg00/oracle /oracle vxfs rw,suid,largefiles,delaylog,datainlog 0 2
/dev/dsk/c1t0d1 /oracledb vxfs rw,suid,delaylog,datainlog,largefiles 0 1
/dev/dsk/c1t0d2 /backup vxfs defaults 0 2
/dev/vg00/lv13 ... swap pri=1 0 2

Actually I forgot to look into fstab before applying patches but

# ls -lt /etc/fstab
-rw-r--r-- 1 root sys 495 Oct 12 00:40 /etc/fstab

Meaning it is not disturbed.

Re: Oracle error after HP UX patch updation

Syed Salman Mansoor — Wed, 18 Mar 2009 16:04:57 GMT

Also wanted to add that my disks are local scsi disks. I dont have SAN attached.

Re: Oracle error after HP UX patch updation

T G Manikandan — Wed, 18 Mar 2009 16:07:12 GMT

can you please attach the oracle alert log file please.

Thanks

Re: Oracle error after HP UX patch updation

Syed Salman Mansoor — Wed, 18 Mar 2009 16:22:05 GMT

Tue Mar 17 23:30:13 2009
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
SCN scheme 3
LICENSE_MAX_USERS = 0
SYS auditing is enabled
Starting up ORACLE RDBMS Version: 9.2.0.7.0.
System parameters with non-default values:
processes = 4000
sessions = 4405
timed_statistics = TRUE
resource_limit = TRUE
shared_pool_size = 637534208
sga_max_size = 4296498144
large_pool_size = 16777216
java_pool_size = 0
control_files = /oracle/home/oradata/pibassrv/control01.ctl, /oracle/home/oradata/pibassrv/control02.ctl, /oracle/home/oradata/pibassrv/control03.ctl
db_block_size = 8192
db_cache_size = 3254779904
compatible = 9.2.0.0.0
log_archive_start = FALSE
log_archive_dest = /backup/archives
log_archive_dest_1 =
log_archive_dest_2 =
standby_archive_dest = /oracledb/pibasdb/archive
log_archive_format = %t_%s.arc
db_file_multiblock_read_count= 16
fast_start_mttr_target = 300
undo_management = AUTO
undo_tablespace = UNDOTBS2
undo_retention = 10800
remote_os_roles = TRUE
remote_login_passwordfile= EXCLUSIVE
audit_sys_operations = TRUE
db_domain =
commit_point_strength = 200
instance_name = pibassrv
job_queue_processes = 10
hash_join_enabled = TRUE
background_dump_dest = /oracle/home/admin/pibassrv/bdump
user_dump_dest = /oracle/home/admin/pibassrv/udump
core_dump_dest = /oracle/home/admin/pibassrv/cdump
open_links = 200
open_links_per_instance = 200
audit_trail = DB
sort_area_size = 524288
db_name = pibassrv
open_cursors = 3000
star_transformation_enabled= FALSE
_no_or_expansion = TRUE
query_rewrite_enabled = FALSE
pga_aggregate_target = 209715200
PMON started with pid=2
DBW0 started with pid=3
LGWR started with pid=4
CKPT started with pid=5
SMON started with pid=6
RECO started with pid=7
CJQ0 started with pid=8
Tue Mar 17 23:30:15 2009
ALTER DATABASE MOUNT
Tue Mar 17 23:30:19 2009
Successful mount of redo thread 1, with mount id 1896047879
Tue Mar 17 23:30:19 2009
Database mounted in Exclusive Mode.
Completed: ALTER DATABASE MOUNT
Tue Mar 17 23:30:26 2009
alter database archivelog
Completed: alter database archivelog
Tue Mar 17 23:30:33 2009
alter database open
Tue Mar 17 23:30:33 2009
LGWR: Primary database is in CLUSTER CONSISTENT mode
Thread 1 opened at log sequence 1271
Current log# 4 seq# 1271 mem# 0: /oracle/home/oradata/pibassrv/log4.ora
Successful open of redo thread 1
Tue Mar 17 23:30:33 2009
SMON: enabling cache recovery
Tue Mar 17 23:30:33 2009
Successfully onlined Undo Tablespace 11.
Tue Mar 17 23:30:33 2009
SMON: enabling tx recovery
Tue Mar 17 23:30:33 2009
Database Characterset is WE8ISO8859P1
replication_dependency_tracking turned off (no async multimaster replication found)
Completed: alter database open
Tue Mar 17 23:31:18 2009
Errors in file /oracle/home/admin/pibassrv/bdump/pibassrv_reco_1997.trc:
ORA-02019: connection description for remote database not found
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
Tue Mar 17 23:31:18 2009
Errors in file /oracle/home/admin/pibassrv/bdump/pibassrv_reco_1997.trc:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
Tue Mar 17 23:31:18 2009
Errors in file /oracle/home/admin/pibassrv/bdump/pibassrv_reco_1997.trc:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
Tue Mar 17 23:31:18 2009
Errors in file /oracle/home/admin/pibassrv/bdump/pibassrv_reco_1997.trc:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
Tue Mar 17 23:31:18 2009
Errors in file /oracle/home/admin/pibassrv/bdump/pibassrv_reco_1997.trc:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
Tue Mar 17 23:31:18 2009
Errors in file /oracle/home/admin/pibassrv/bdump/pibassrv_reco_1997.trc:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325

..................

Tue Mar 17 23:31:40 2009
Thread 1 advanced to log sequence 1272
Current log# 1 seq# 1272 mem# 0: /oracle/home/oradata/pibassrv/log1.ora
Tue Mar 17 23:31:40 2009
ARCH: Evaluating archive log 4 thread 1 sequence 1271
ARCH: Beginning to archive log 4 thread 1 sequence 1271
Creating archive destination LOG_ARCHIVE_DEST_1: '/backup/archives/1_1271.arc'
ARCH: Completed archiving log 4 thread 1 sequence 1271
Tue Mar 17 23:31:51 2009
Errors in file /oracle/home/admin/pibassrv/bdump/pibassrv_reco_1997.trc:
ORA-02019: connection description for remote database not found
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
Tue Mar 17 23:31:51 2009
Errors in file /oracle/home/admin/pibassrv/bdump/pibassrv_reco_1997.trc:

................................
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
Tue Mar 17 23:32:56 2009
create tablespace abc datafile '/oracledb/pibasdb/datafiles/vicky.dbf' size 50M logging extent management local segment space management auto
Tue Mar 17 23:32:56 2009
ORA-19502 signalled during: create tablespace abc datafile '/oracledb/pibasdb/...
Tue Mar 17 23:33:52 2009
Errors in file /oracle/home/admin/pibassrv/bdump/pibassrv_reco_1997.trc:
ORA-02019: connection description for remote database not found
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
Tue Mar 17 23:33:52 2009
Errors in file /oracle/home/admin/pibassrv/bdump/pibassrv_reco_1997.trc:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found

===================================================================================

/oracle/home/admin/pibassrv/bdump/pibassrv_reco_1997.trc
Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.7.0 - Production
ORACLE_HOME = /oracle/home/Ora9i
System name: HP-UX
Node name: pibaspr
Release: B.11.23
Version: U
Machine: ia64
Instance name: pibassrv
Redo thread mounted by this instance: 1
Oracle process number: 7
Unix process pid: 1997, image: oracle@pibaspr (RECO)

*** SESSION ID:(6.1) 2009-03-17 23:31:18.430
*** 2009-03-17 23:31:18.430
ERROR, tran=3.13.177816, session#=1, ose=0:
ORA-02019: connection description for remote database not found
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
*** 2009-03-17 23:31:18.433
ERROR, tran=5.31.204647, session#=1, ose=0:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
*** 2009-03-17 23:31:18.434
ERROR, tran=6.15.169919, session#=1, ose=0:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
*** 2009-03-17 23:31:18.435
ERROR, tran=7.16.172083, session#=1, ose=0:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
*** 2009-03-17 23:31:18.435
ERROR, tran=3.43.178069, session#=1, ose=0:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
*** 2009-03-17 23:31:18.436
ERROR, tran=3.24.178078, session#=1, ose=0:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
*** 2009-03-17 23:31:18.437
ERROR, tran=7.28.172198, session#=1, ose=0:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
*** 2009-03-17 23:31:18.439
ERROR, tran=1.18.295041, session#=1, ose=0:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
*** 2009-03-17 23:31:18.440
ERROR, tran=10.9.316890, session#=1, ose=0:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325
ORA-02019: connection description for remote database not found
*** 2009-03-17 23:31:18.452
ERROR, tran=11.29.52287, session#=1, ose=0:
ORA-02062: distributed recovery received DBID c63e2355, expected 25efd325

Re: Oracle error after HP UX patch updation

T G Manikandan — Fri, 20 Mar 2009 16:15:58 GMT

This looks like a HPUX issue, the patch database points to PHKL_38574 for 11.31.

hmm you are running HPUX 11.23,checking for patch for 11.23