ora_pmon and crsd.bin

[Robin T. Slotten]

I forgot to respond to your crash question.

/etc/rc.config.d/savecrash

# SAVECRASH: Set to 0 to disable saving system crash dumps.
SAVECRASH=1

# SAVECRASH_DIR:Directory name for system crash dumps. Note: the filesystem
# in which this directory is located should have as much free
# space as your system has RAM.
# Default directory:
# SAVECRASH_DIR=/var/adm/crash

--------
If you have less space than RAM, you can sometimes pull some information out of the dump anyway.

You can do some self analysis using Q4. Look in the what.out.
------
ITRC DOCUMENT ID: OZBEKBRC00000611

USING Q4 TO ANALYZE SYSTEM DUMP FILES
(For HPUX 10.10-11.23 systems)

http://www12.itrc.hp.com/service/cki/docDisplay.do?docLocale=en&docId=emr_na-c01021636-8

Rob...

[John Jimenez]

Thanks the info in Feb, Robin. I did not see it cime in last month. I will assign some points

Today we had oracle_pmon issues again. The Oracle Admin, said that some logs filled up. Scripts run durimg Data Protector backups that clean them up. But There has been lots of changes today, so it looks like some logs filled up.

[Yogeeraj_1]

> But There has been lots of changes today, so it looks like some logs filled up.

If you want to find out the interval during which this occurred, you can run the following query:
=========================================
prompt*****************************************************
prompt*** Redolog Switch Rate by Date and Hour ****
prompt*****************************************************
set heading on;
column day format a3
col Total for 99G990;
col h00 for 999;
col h01 for 999;
col h02 for 999;
col h03 for 999;
col h04 for 999;
col h05 for 999;
col h06 for 999;
col h07 for 999;
col h08 for 999;
col h09 for 999;
col h10 for 999;
col h11 for 999;
col h12 for 999;
col h13 for 999;
col h14 for 999;
col h15 for 999;
col h16 for 999;
col h17 for 999;
col h18 for 999;
col h19 for 999;
col h20 for 999;
col h21 for 999;
col h22 for 999;
col h23 for 999;
col h24 for 999;


break on report
compute max of "Total" on report
compute max of "h00" on report
compute max of "h01" on report
compute max of "h02" on report
compute max of "h03" on report
compute max of "h04" on report
compute max of "h05" on report
compute max of "h06" on report
compute max of "h07" on report
compute max of "h08" on report
compute max of "h09" on report
compute max of "h10" on report
compute max of "h11" on report
compute max of "h12" on report
compute max of "h13" on report
compute max of "h14" on report
compute max of "h15" on report
compute max of "h16" on report
compute max of "h17" on report
compute max of "h18" on report
compute max of "h19" on report
compute max of "h20" on report
compute max of "h21" on report
compute max of "h22" on report
compute max of "h23" on report


SELECT trunc(first_time) "Date",
to_char(first_time, 'Dy') "Day",
count(1) as "Total",
SUM(decode(to_char(first_time, 'hh24'),'00',1,0)) as "h00",
SUM(decode(to_char(first_time, 'hh24'),'01',1,0)) as "h01",
SUM(decode(to_char(first_time, 'hh24'),'02',1,0)) as "h02",
SUM(decode(to_char(first_time, 'hh24'),'03',1,0)) as "h03",
SUM(decode(to_char(first_time, 'hh24'),'04',1,0)) as "h04",
SUM(decode(to_char(first_time, 'hh24'),'05',1,0)) as "h05",
SUM(decode(to_char(first_time, 'hh24'),'06',1,0)) as "h06",
SUM(decode(to_char(first_time, 'hh24'),'07',1,0)) as "h07",
SUM(decode(to_char(first_time, 'hh24'),'08',1,0)) as "h08",
SUM(decode(to_char(first_time, 'hh24'),'09',1,0)) as "h09",
SUM(decode(to_char(first_time, 'hh24'),'10',1,0)) as "h10",
SUM(decode(to_char(first_time, 'hh24'),'11',1,0)) as "h11",
SUM(decode(to_char(first_time, 'hh24'),'12',1,0)) as "h12",
SUM(decode(to_char(first_time, 'hh24'),'13',1,0)) as "h13",
SUM(decode(to_char(first_time, 'hh24'),'14',1,0)) as "h14",
SUM(decode(to_char(first_time, 'hh24'),'15',1,0)) as "h15",
SUM(decode(to_char(first_time, 'hh24'),'16',1,0)) as "h16",
SUM(decode(to_char(first_time, 'hh24'),'17',1,0)) as "h17",
SUM(decode(to_char(first_time, 'hh24'),'18',1,0)) as "h18",
SUM(decode(to_char(first_time, 'hh24'),'19',1,0)) as "h19",
SUM(decode(to_char(first_time, 'hh24'),'20',1,0)) as "h20",
SUM(decode(to_char(first_time, 'hh24'),'21',1,0)) as "h21",
SUM(decode(to_char(first_time, 'hh24'),'22',1,0)) as "h22",
SUM(decode(to_char(first_time, 'hh24'),'23',1,0)) as "h23"
FROM V$log_history
group by trunc(first_time), to_char(first_time, 'Dy')
Order by 1;

clear breaks

set heading off;

=========================================
And from there you can investigate further.

hope this helps!

kind regards
yogeeraj

[Robin T. Slotten]

UPDATE on our issues. We added another CPU to the 2 nodes and the load decreased from about 80% ambient load spiking at 100% for 15-20 minutes at a time, down to about 30-40%. Our crashes have mostly stopped. My DBA found an issue about accumulating too many logs in /u10/app/oracle/product/10.2/crs/log//client
Apparently as the number of logs increases, Oracle scans through them for some reason I didn't grasp. At some point crsd.bin cannot keep up and the load increases drastically and eventually will panic one of the nodes.
Rob...

[John Jimenez]

Thanks for the script Yogeeraj. I guess the interface the Oracle Admin uses for monitering, stopped working a week ago, and he has not figured out how to make it work yet. that is why it filled up. I will see if I can find time today to run this script.
Robin,
Thank you for this information. We have 4 CPU's on both of our RP7420's. CPU is never an issue (usually 20%-25% at most, of course unless these oracle processes go crazy. But it sounds like cleaning up logs has been one of the issues oracle admin. has encountered.