System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

Glance display too many died process !!!

HP-UX_Ali
Regular Advisor

Glance display too many died process !!!

Hello all,

Pls check and advice, how to remove the died process from the memory, as consumption of memory is 100%.

cmclconfd 10424 root 0.0 1 1.3 320kb 1.0mb died
cmclconfd 10490 root 0.0 1 1.3 316kb 1.0mb died
cmclconfd 10439 root 0.0 1 1.3 316kb 1.0mb died
cmclconfd 10500 root 0.0 1 1.3 316kb 1.0mb died
cmclconfd 10435 root 0.0 1 1.3 320kb 1.0mb died
cmclconfd 10426 root 0.0 1 1.3 316kb 1.0mb died
cmclconfd 10496 root 0.0 1 1.3 320kb 1.0mb died
cmclconfd 10457 root 0.0 1 1.3 316kb 1.0mb died
cmclconfd 10447 root 0.0 1 1.3 316kb 1.0mb died
cmclconfd 10379 root 0.0 1 1.3 320kb 1.0mb died
cmclconfd 10498 root 0.0 1 1.3 316kb 1.0mb died
cmclconfd 10281 root 0.0 1 1.3 320kb 1.0mb died
cmclconfd 10275 root 0.0 1 1.3 316kb 1.0mb died
cmclconfd 10314 root 0.0 1 1.3 304kb 1.0mb died
cmclconfd 10271 root 0.0 1 1.3 320kb 1.0mb died
cmclconfd 10264 root 0.0 1 1.3 320kb 1.0mb died
cmclconfd 10246 root 0.0 1 1.3 320kb 1.0mb died
cmclconfd 10240 root 0.0 1 0.0 316kb 1.0mb died
cmclconfd 10265 root 0.0 1 1.3 320kb 1.0mb died
cmclconfd 10248 root 0.0 1 1.3 316kb 1.0mb died
cmclconfd 10261 root 0.0 1 1.3 320kb 1.0mb died
cmclconfd 10298 root 0.3 1 1.3 592kb 1.0mb died
cmclconfd 10273 root 0.0 1 1.3 312kb 1.0mb died
cmclconfd 10295 root 0.0 1 1.3 320kb 1.0mb died
cmclconfd 10316 root 0.0 1 1.3 320kb 1.0mb died
cmclconfd 10341 root 0.0 1 1.3 316kb 1.0mb died
cmclconfd 10302 root 0.0 1 1.3 300kb 1.0mb died
cmclconfd 10309 root 0.0 1 1.3 320kb 1.0mb died
cmclconfd 10510 root 0.0 1 1.3 320kb 1.0mb died
cmclconfd 10328 root 0.0 1 1.3 316kb 1.0mb died
cmclconfd 10306 root 0.0 1 1.3 320kb 1.0mb died
cmclconfd 10527 root 0.0 1 1.3 312kb 1.0mb new
cmclconfd 10284 root 0.0 1 1.3 320kb 1.0mb died
cmclconfd 10525 root 0.0 1 1.3 320kb 1.0mb new
cmclconfd 10414 root 0.0 1 1.3 316kb 1.0mb died
cmclconfd 10250 root 0.0 1 1.3 316kb 1.0mb died
cmclconfd 10369 root 0.0 1 1.3 320kb 1.0mb died
cmclconfd 10376 root 0.0 1 1.3 320kb 1.0mb died
cmclconfd 10401 root 0.0 1 1.3 312kb 1.0mb died
cmclconfd 10411 root 0.0 1 1.3 316kb 1.0mb died
cmclconfd 10366 root 0.0 1 1.3 320kb 1.0mb died
cmclconfd 10409 root 0.0 1 1.3 316kb 1.0mb died
cmclconfd 10389 root 0.0 1 1.3 316kb 1.0mb died
cmclconfd 10394 root 0.0 1 1.3 320kb 1.0mb died
cmclconfd 10507 root 0.0 1 1.3 320kb 1.0mb died
cmclconfd 10364 root 0.0 1 1.3 316kb 1.0mb died
cmclconfd 10399 root 0.0 1 1.3 316kb 1.0mb died
cmclconfd 10349 root 0.1 1 1.4 304kb 1.0mb died
cmclconfd 10373 root 0.0 1 1.3 316kb 1.0mb died



Regards
Ali
13 REPLIES
Dennis Handly
Acclaimed Contributor

Re: Glance display too many died process !!!

Why do you think these are still in memory?
What does this show:
ps -f -p 10373 -p 10349 -p 10399
And:
UNIX95=EXTENDED_PS ps -H -f -C cmclconfd
HP-UX_Ali
Regular Advisor

Re: Glance display too many died process !!!

Thanks Denies

belwo is the output..

Then why memory is 100% in glance output with so many died process. and few oracle process. this is the second node where the first node has only 60 % memory utilization.

ps -f -p 10373 -p 10349 -p 10399
UID PID PPID C STIME TTY TIME COMMAND
root>UNIX95=EXTENDED_PS ps -H -f -C cmclconfd
UID PID PPID C STIME TTY TIME CMD
root 11743 1931 0 Dec 19 ? 00:00:01 cmclconfd -c
root 1945 1931 0 Dec 19 ? 03:43:59 cmclconfd -p
root 11805 1931 0 Dec 19 ? 00:00:01 cmclconfd -c
root 11818 1931 0 Dec 19 ? 00:00:01 cmclconfd -c
root 11721 1931 0 Dec 19 ? 00:00:01 cmclconfd -c
root 6009 1931 1 10:21:04 ? 00:00:00 cmclconfd -c
Dennis Handly
Acclaimed Contributor

Re: Glance display too many died process !!!

>Then why memory is 100% in glance output with so many died process.

What HP-UX version are you running?

There must be some other processes using up the memory? Or used for the file cache?

>ps -f -p 10373 -p 10349 -p 10399

So those PIDs did die.

root 11743 1931 Dec 19 cmclconfd
root 01945 1931 Dec 19 cmclconfd
root 11805 1931 Dec 19 cmclconfd
root 11818 1931 Dec 19 cmclconfd
root 11721 1931 Dec 19 cmclconfd
root 06009 1931 10:21:04 cmclconfd

And these PIDs may be using .3 Mb each.
HP-UX_Ali
Regular Advisor

Re: Glance display too many died process !!!

Hi Dennis

its B.11.31, i cannot see other process up eating much memory..

So is there any way to remove the dead process from glance too.


Regards,
Dennis Handly
Acclaimed Contributor

Re: Glance display too many died process !!!

>is there any way to remove the dead process from glance too.

glance will automatically remove them after they die.
HP-UX_Ali
Regular Advisor

Re: Glance display too many died process !!!

Hello Dennis,

Thanks for the reply..but still i can see the died process in glance output. since it is 2 days glance didnt removed the died process :). Still the memory utilizing is 100%. As the shock is it is not even reducing 1%. and even i cannot see other process eating much memory.

As i said it is second node. and even in 1st node the utilization is only 65%.

Pls advise.... to reslove issue. i suspect still the died process eating up the memory. Even the swap process is 60% used.

Thanks Ali
Dennis Handly
Acclaimed Contributor

Re: Glance display too many died process !!!

>but still I can see the died process in glance output. since it is 2 days glance didn't removed the died process :).

The same ones? What does ps(1) show for them? When you tried a few above, they weren't there.

>Still the memory utilizing is 100%.

This is a good thing, you want it 100%. If not used by a process you want it used for the file cache.

>I suspect still the died process eating up the memory. Even the swap process is 60% used.

I don't see how? Dead is dead.
Bill Hassell
Honored Contributor

Re: Glance display too many died process !!!

Memory usage in HP-UX is not simple. Programs use memory, but some programs also reserve shared memory, and the kernel has a large buffer cache which will shrink in size when more programs need additional memory.

Since you have glance, just type the letter o (not zero) and press Enter to select process selection by usage. Select the Sort Key rss (resident set size) and set that as your default. Now the general screen will show your processes sorted by RSS size.

You can also see the largest processes sorted by size (KBytes) from the command line:

UNIX95=1 ps -e -o vsz,pid,ruser,args | sort -rn | head -20


Bill Hassell, sysadmin
HP-UX_Ali
Regular Advisor

Re: Glance display too many died process !!!

Thanks Bill...


But pls check the attachement.

Sorry dennies i didnt got your point on this..


Regards
Ali
Bill Hassell
Honored Contributor

Re: Glance display too many died process !!!

You have massive runaway processes and they appear to be in an endless loop. The processes marked as DIED will not appear again (the process ID keeps changing). The processes start to run and get killed by some other process. Your system is very sick and unless you know the process(es) that are causing this condition, you'll have to reboot into single user mode and stop any scripts or programs that were added before this problem started occurring. It appears that this may be part of a cluster so if possible, failover this node to the other and troubleshoot the problem.


Bill Hassell, sysadmin
HP-UX_Ali
Regular Advisor

Re: Glance display too many died process !!!

Thanks Bills.. appreciate your quick response and advice...

You are right this is env is 4 node metro-rac cluster and the issue arises in one of the DR site (2nd node) which is not in much use but i wonder why this is happening and after some troubleshoot, i noticed their is no such script or programms which is killing any process but one thing i suspect does it due to crs process ?... , i want to troubleshoot this without reboot....

Once again thanks for your support and time.

Thanks & Regards
Ali....


Steven E. Protter
Exalted Contributor

Re: Glance display too many died process !!!

Shalom,

This is a problem. These processes are hung and have not released their memory.

They are related to serviceguard.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Dennis Handly
Acclaimed Contributor

Re: Glance display too many died process !!!

>dennis I didn't got your point on this.

As Bill said, you seem to have continual cmclcond processes that just keep coming back.
And more kill & sleep.

And of course, while they are alive, they take up memory and CPU resources.

>I want to troubleshoot this without reboot.

I'm not sure how successful you'll be.
You might start with a hierarchical process tree:
UNIX95=EXTENDED_PS ps -ef -H