Operating System - HP-UX
1748169 Members
4437 Online
108758 Solutions
New Discussion юеВ

Re: ora_pmon and crsd.bin

 
John Jimenez
Super Advisor

ora_pmon and crsd.bin

early this morning users started having issues. On the system side the only thing that I could see abnormal was that ora_pmon was using 99% of one cpu on server#1

oracle 4115 1 237 Jan 19 ? 3722:48 ora_pmon_MXPROD2

The Oracle admin fixed it. I think he restarted the listener. I was still concerned about ora_pmon, but we have never seen this go high, and thought we would deal with this off hours instead of causing more issues on hours

but now an hour later users are getting errors again and crsd.bin are hogging all 4 cpu's on server#2. Also Zombies are increasing.

Has anyone ever seed this happen? do you think these two processes are related?
Hustle Makes things happen
14 REPLIES 14
John Jimenez
Super Advisor

Re: ora_pmon and crsd.bin

I am rebooting server1 which still had ora_mon using 100%. Once it comes up I am going to rboot server2, which crsd.bin is taking 350% (we have 4 cpu's). both of these servers are running hpux 11.23.
Hustle Makes things happen
John Jimenez
Super Advisor

Re: ora_pmon and crsd.bin

Something else new that I have not seen before. When I rebooted I got 3 swap reservation failures.

System shutdown time has arrived
reboot: CAUTION: some process(es) wouldn't die
Deferred swap reservation failure pid: 27556
Deferred swap reservation failure pid: 27556
Deferred swap reservation failure pid: 28522

sync'ing disks (0 buffers to flush):
0 buffers not flushed
0 buffers still dirty
Hustle Makes things happen
Robin T. Slotten
Trusted Contributor

Re: ora_pmon and crsd.bin

We have a similar issue
We have been battling very high CPU loads caused by crs_stat.bin. crsd.bin is usually near the top of the list of high CPU utilization as well. We have been incurring system panics at an increasing rate for several months. Last week we installed an additional processor in each of our systems. That relieved the system panics (Caused by Oracle calling for a TOC)to a degree. We believe the problem is load related, but have not been able to determine the problem so far. The DBA has opened a number of TAR's with Oracle with out any results. I under stand we will have an Oracle consultant on site within a week to see if (s)he can determine the cause of the system crashes. While I cannot offer any solutions, I can tell you that we have experienced very high loading by crs processes.

I'll update if I find anything else.
Rob...
IF you do it more than twice, write a script.
skt_skt
Honored Contributor

Re: ora_pmon and crsd.bin


we are using oracle 10g(10.2.0.2)/CRS cluster on HP-UX 11.11. But taking this much CPU by those oracle processes are not normal in our enviroment..

Also oracle had provided TOC patch(PHKL) which will help to create crash dump in case of TOC.

Robin T. Slotten
Trusted Contributor

Re: ora_pmon and crsd.bin

We are running 10G(10.2.0.2)/crs on 11.23. You don't happen to have the PHKL-number that I can cross-ref do you?

Rob... ( I'd give you point for that if I could.)
IF you do it more than twice, write a script.
John Jimenez
Super Advisor

Re: ora_pmon and crsd.bin

Yes if you have that patch number it would be apprciated. And if I have the patch, where do I look for the crash dump?
In past years I have only ran flat file databases on HP/UX. Almost Two years ago we got a new app on Oracle 10g with RAC on two RP7420's each running hpux (11.23) with 16 gig ram 8 gig of swap. patch bundles are of 6-2007. We seem to have this crsd.bin issue off and on. We were okay for about 6 months, but have gotten this problem for the 2nd time in two months.
Hustle Makes things happen
skt_skt
Honored Contributor

Re: ora_pmon and crsd.bin

TOC patch PHKL_36700 for hp-ux 11.11
Robin T. Slotten
Trusted Contributor

Re: ora_pmon and crsd.bin


The equivalent patch for 11.23 is PHKL_34941. The bad news for us is we already have that patch on. But thanks.
Rob...
IF you do it more than twice, write a script.
John Jimenez
Super Advisor

Re: ora_pmon and crsd.bin

I have that one too. This morning everyone was logged into server1 no one into server2 and crsd.bin went crazy on server1. I had to reboot both of them again this morning.

PHKL_34941 Oracle Clusterware, Setboot for PCI Express
Hustle Makes things happen