registrar service consuming 90 % of CPU resources

[user1221]

Hi guyz,
We have a two node RAC cluster with Oracle 10gR2 running on HP_UX B.11.31,on the second node the registrar, cpe_em and the midaemon is consuming more than 90% CPU resources. We had a downtime on jun 18, since then the CPU usage y these services are like this.

Here is the Output of "TOP"
System: hisdbs02 Wed Aug 5 12:55:41 2009
Load averages: 1.02, 1.05, 1.11
500 processes: 438 sleeping, 61 running, 1 zombie
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.78 12.9% 0.0% 20.8% 66.3% 0.0% 0.0% 0.0% 0.0%
1 1.02 13.9% 0.0% 7.9% 78.2% 0.0% 0.0% 0.0% 0.0%
2 1.05 12.9% 0.0% 81.2% 5.9% 0.0% 0.0% 0.0% 0.0%
3 1.22 34.7% 0.0% 54.5% 10.9% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 1.02 18.8% 0.0% 41.6% 39.6% 0.0% 0.0% 0.0% 0.0%

System Page Size: 4Kbytes
Memory: 12440112K (5179416K) real, 14405228K (6542032K) virtual, 3670644K free Page#
1/12

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
2 ? 18933 root 241 20 4920K 672K run 14720:31 93.53 93.36 registrar
3 ? 10961 root 241 20 10440K 1424K run 26946:47 93.46 93.30 cpe_em
2 ? 1785 root -16 20 286M 262M run 3592:12 20.11 20.07 midaemon
0 ? 11452 oracle 154 20 17048M 8960K sleep 0:30 3.51 3.51 oracleorcl2
3 ? 29394 oracle 154 20 17048M 8368K sleep 0:07 2.60 2.59 oracleorcl2
2 ? 122 root 191 20 6264K 5568K run 1073:18 2.50 2.49 vxfsd
1 ? 19279 oracle 154 20 17046M 8328K sleep 238:54 2.31 2.31 ora_pz99_orcl2

.and the SAR
HP-UX hisdbs02 B.11.31 U ia64 08/05/09

12:56:06 %usr %sys %wio %idle
12:56:11 20 48 0 32
12:56:16 22 51 1 26
12:56:21 18 55 0 27
12:56:26 19 52 2 27
12:56:31 22 48 0 29

Average 21 51 1 28

I dont have any idea about the registrar, cpe_em and midaemon service. PLease suggest what to do, can I restart those services?

[Jaime Bolanos Rojas.]

Hi Tagi!

Update to your latest version of STM/EMS found at software.hp.com. That should take care of the issue.

Regards,

Jaime.

[Jineesh]

Hi,

there are known issues with EMS regards to the above.You can restart the EMS services in the server by monconfig command.But the issue may reoccur.Update the EMS to its latest.This will solve your issue

[Roopesh Francis_1 · Accepted solution]

Hi Tagi,

you can restart the registrar process for reducing cpu utilization temporally however i would recommend to update it to latest software. below are few threads which discussed same kind of issues.
http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1219639
http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1359171
http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=398927


Thanks

[Jeeshan]

This a problem related to EMS/STM.

Stop EMS:

/etc/opt/resmon/lbin/monconfig and stop EMS

Stop the p_client process:

Edit the /etc/initab and comment out the line "ems4:3456:respawn:/etc/opt/resmon/lbin/p_client" and then run "init q" to have the system re-read the inittab.



Stop Diagnostics:

/sbin/init.d/diagnostic stop



Kill the following processes if they are running (a kill -9 may have to be used):

registrar

diagmond

p_client

cclogd

memlogd

psmctd

diaglogd

then check the registrar process is spawning or not. if not then

Restart diagnostics:

/sbin/init.d/diagnostic start

Edit the /etc/inittab and uncomment "ems4:3456:respawn:/etc/opt/resmon/lbin/p_client" and re-read the inittab by running "init q" . This will cause the p_client process to spawn.

Start EMS:

/etc/opt/resmon/lbin/monconfig and start EMS using "E" option.

Then check the result.

[user1221]

I upgraded EventMonitoring system to latest version 04.20.31.04 and it resolved the issue. The "Registrar" "CPE_EM" and "MIDEAMON" processes CPU utilization is normal now, Alhamdulillah.

[user1221]

Upgrade the Event monitoring system to 04.20.31.04.