System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

More number of registrar process running, System Hangs

pandora_1
Frequent Advisor

More number of registrar process running, System Hangs

Hi All,
OS VErsion : B.11.23 U 9000/800
Service Guard : A.11.17.00 With RAC

More number of registrar process running on one Server which is Clustered environment. This had happened suddenly and the CPU is fully utilised by registrar ,which has broght Oracle RAC to a "hit" ?
ANyone help why these many process running ?
Currently this process is stopped ?
Could this be any hardware issue ? how could I check it ?
Output of top is pasted

System: abcds, Thu Apr 3 02:49:23 2008
Load averages: 12.76, 12.58, 12.60
277 processes: 194 sleeping, 83 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 9.90 9.9% 0.0% 90.1% 0.0% 0.0% 0.0% 0.0% 0.0%
1 16.17 10.9% 0.0% 89.1% 0.0% 0.0% 0.0% 0.0% 0.0%
2 12.31 7.1% 0.0% 92.9% 0.0% 0.0% 0.0% 0.0% 0.0%
3 12.66 6.5% 0.0% 93.5% 0.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 12.76 8.6% 0.0% 91.4% 0.0% 0.0% 0.0% 0.0% 0.0%

Memory: 678304K (611680K) real, 1735736K (1487220K) virtual, 5661648K free Page# 1/18

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
2 ? 12662 root 241 20 2496K 492K run 70:12 11.17 11.15 registrar
0 ? 14371 root 241 20 2496K 492K run 51:08 10.95 10.93 registrar
3 ? 12084 root 241 20 2496K 492K run 54:01 10.84 10.82 registrar
2 ? 10968 root 241 20 2496K 492K run 63:10 10.65 10.63 registrar
2 ? 28863 root 241 20 2496K 492K run 57:34 10.27 10.25 registrar
2 ? 26509 root 241 20 2496K 492K run 72:08 10.04 10.02 registrar
2 ? 7042 root 240 20 2496K 492K run 84:47 10.01 9.99 registrar
2 ? 20152 root 241 20 2496K 492K run 57:15 9.78 9.77 registrar
2 ? 22733 root 138 20 2496K 492K run 56:31 9.74 9.72 registrar
3 ? 19440 root 241 20 2496K 492K run 93:37 9.25 9.23 registrar
3 ? 24067 root 240 20 2496K 492K run 65:45 9.07 9.05 registrar
3 ? 6568 root 240 20 2496K 492K run 53:06 9.00 8.99 registrar
2 ? 1441 root 241 20 2496K 492K run 54:04 8.95 8.94 registrar
2 ? 22044 root 241 20 2496K 492K run 60:35 8.86 8.85 registrar
3 ? 15765 root 138 20 2496K 492K run 82:11 8.58 8.56 registrar
0 ? 2640 root 241 20 2496K 492K run 64:29 8.50 8.48 registrar

Awaiting a early solution

Thanks
Azad
10 REPLIES
Jeeshan
Honored Contributor

Re: More number of registrar process running, System Hangs

Salam Azad

registrar is actually an EMS. please identify how many processes is running

#ps -ef|grep registrar
a warrior never quits
pandora_1
Frequent Advisor

Re: More number of registrar process running, System Hangs

Wasalam ahsan,
Its stopped at the moment. At the time of issue it was 48 Processes running
I have attached the log file of registrar processes running
Jeeshan
Honored Contributor

Re: More number of registrar process running, System Hangs

First identify what is the version of your STM.

If its A.29 its need the latest patch. Additionally you can do the followings.
the registrar is definitely needed for EMS, without this process ISEE will not work correctly. For the version A.29 a patch is needed for EMS (PHSS_28843), it might help to install this. In addition you should restart EMS using the following procedure:
1) Stop the registar if neccesary by commenting out the 1 entries in /etc/inetd.conf and re-reading the inetd configuration with inetd -c

2) Issue ps -ef | grep registrar to check if any registrar processes are remaining still, kill them if necessary

3) Stop monitoring
# /etc/opt/resmon/lbin/monconfig
Select (K)ill (disable) monitoring

ATTENTION! Within a Serviceguard Cluster, package dependencies might be configured, which could cause a package to switch,
in case a resource would go to a DOWN state.
In case EMS is stopped, this resource would be considered to be DOWN and the package would switch.
If you are unsure, follow the instructions in the document before you go ahead!

4) Issue ps -ef | grep monitor to check if any monitors are still running, kill them if necessary


5) Stop p_client
Comment out the p_client entry in /etc/inittab :
# vi /etc/inittab

6) Change the following line
ems4:3456:respawn:/etc/opt/resmon/lbin/p_client
to
#ems4:3456:respawn:/etc/opt/resmon/lbin/p_client

7) Reread /etc/inittab :
# init q


8) Issue ps -ef | grep p_client to check if p_client is still running, if yes kill the process


9) Delete the persistence files and pipes
# cd /etc/opt/resmon/persistence
# rm p*
# rm m*
# cd /etc/opt/resmon/pipe
# rm *


10) Have diagnostics remap the system hardware
#cstm
cstm>remap
cstm>quit

Allow enough time for remap to finish before restarting the monitors.
When the remap is in progress you get the message ***Remap in progress*** when issuing map at the cstm prompt;
when the remap has finished the map is displayed when issuing the command map at the cstm prompt.
It might take up to a few minutes depending on the amount of hardware connected.


11) Clean up monitor flags in the diagnostics directory
# cd /var/stm/data/tools/monitor
# rm *.hwa


12) Re-enable registrar in /etc/inetd.conf and run inetd -c


13) Restart p_client
# vi /etc/inittab
Change the following line
#ems4:3456:respawn:/etc/opt/resmon/lbin/p_client
back to
ems4:3456:respawn:/etc/opt/resmon/lbin/p_client

14) Re-read /etc/inittab :
# init q


15) Restart the EMS monitors
# /etc/opt/resmon/lbin/monconfig
Select (E)nable Monitoring

It might be a good idea to update the diagnostics. Be careful if this is a service guard cluster when you try to uninstall the support tools. If the above procesure help resolving the registrar problem, you can update the support toolssimply by doing a swinstall for the new version without an previous uninstall.
a warrior never quits
Jeeshan
Honored Contributor

Re: More number of registrar process running, System Hangs

before doing the EMS restart must ensure the patch is applied, if not than contact with HP while applying then do th steps.

Be careful!!!
a warrior never quits
Jeeshan
Honored Contributor

Re: More number of registrar process running, System Hangs

Steven E. Protter
Exalted Contributor

Re: More number of registrar process running, System Hangs

Shalom Azad,

One thing is certain, it has nothing to do with Oracle.

As noted above your system is saturated with incomplete EMS processes.

I'd get a good backup, apply those patches and restart the system.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
pandora_1
Frequent Advisor

Re: More number of registrar process running, System Hangs

Hi All,

Thanks for the immediete response, Well I have checked for PHSS_35807 patch which is applied already.
swlist -l patch | grep PHSS_35807

PHSS_35807.STM-CATALOGS 1.0 HPUX Support Tools Manager Catalogs applied
PHSS_35807.STM-MAN 1.0 HPUX Support Tools Manager Manual Pages applied
PHSS_35807.STM-SHLIBS 1.0 HPUX Support Tools Manager Share Libraries applied
PHSS_35807.STM-UUT-RUN 1.0 HPUX Support Tools Manager Unit Under Test Runtime applied


Do I need to reapply to try out
pandora_1
Frequent Advisor

Re: More number of registrar process running, System Hangs

Hi,

Could any hardware problem also cause more registrar process to spawn .
Madenzo
Occasional Visitor

Re: More number of registrar process running, System Hangs

Hi,

How do I check the CSTM version of the HP11.23 server
Ivan Krastev
Honored Contributor

Re: More number of registrar process running, System Hangs

Just start cstm and you will see the version.

regards,
ivan