topic Re: Hidden PIDs in Operating System - Linux

Hidden PIDs

Danesh Qureshi — Fri, 28 Sep 2007 06:23:02 GMT

When I run the top command to check the processor activity it displays PIDs and other useful information on my screen. Now when type the ps command I notice that some of the PIDs displayed by top do not appear in ps.

I have attached output of top. The output of top displays PIDs 19533,21416,1594,1595 and 1595.

For some reason ps is not displaying the above PIDs.

Can anyone shed some light on this?

Re: Hidden PIDs

Steven E. Protter — Fri, 28 Sep 2007 06:44:54 GMT

Shalom,

java is a well known resource hog and its using a ton of resources on your system.

java launches individual processes in response to certain application requests and the discrepancy may be due to the fact that the process you are monitoring is closed by the time you get to the ps command.

I suppose the process table is corrupted but your system would not last long in that condition. I don't see anything on the process list that sticks out other than java using a lot of resources and dominating your whole system.

Perhaps you can attach your ps command and output.

Could be you formulated the ps command incorrectly.

SEP

Re: Hidden PIDs

Stuart Browne — Fri, 28 Sep 2007 07:10:07 GMT

If you're really worried about hidden processes (which incidentally should not be possible to hide form a super user), you might want to verify your ps binary.

If it's a redhat based system, an 'rpm -V procps' should do that for you, assuming that a 'which ps' returns '/bin/ps'.

Re: Hidden PIDs

Danesh Qureshi — Fri, 28 Sep 2007 09:01:24 GMT

Hi,

The HP DL380 G4 server is running RedHat Linux As 3 update 4.

The output of rpm -V procps give me nothing
[root#]

The output which ps see below
[root]# which ps
/bin/ps
[root]#

I have attached a file containing the output of top and the ps -ef.

You will notice the java processes show up in top but not is ps -ef.

I was wondering if this is expected behaviour of RedHat Linux As 3 that processes are hidden from ps but not top.

Re: Hidden PIDs

Danesh Qureshi — Fri, 28 Sep 2007 09:02:22 GMT

Sorry forgot to attach the file. here it is.

Re: Hidden PIDs

Ivan Ferreira — Fri, 28 Sep 2007 10:00:23 GMT

Maybe the process starts and die very fast, and from the output of ps, you have to check the full path to the java command, not just "java", for example:

root 1593 1561 3 Sep27 ? 01:06:14 /local/bea92/jdk150_08/bin/java -ms1536m -mx1536m -XX:MaxPermSize=192m -XX:NewSize=128m -XX:Ma

You can also try:

ps auxw |grep java

Re: Hidden PIDs

Danesh Qureshi — Fri, 28 Sep 2007 10:22:24 GMT

have attached another file containing the output of the top command and you will notice those java PIDs are still running.

See output of ps auwx | grep -i java

[root]# ps auxw | grep -i java
root 1593 3.5 52.3 2496840 2109584 ? S Sep27 70:14 /local/bea92/jdk150_08/bin/java -ms1536m -mx1536m -XX:a
root 10252 0.1 5.4 1529816 218108 ? S Sep27 3:20 /local/bea92/jdk150_08/bin/java -ms128m -mx1024m -XX:N.
root 20427 0.0 0.0 3700 668 pts/3 S 16:14 0:00 grep -i java

Re: Hidden PIDs

Ivan Ferreira — Fri, 28 Sep 2007 10:41:36 GMT

I think that is interesting. What is your procps version?

rpm -qi procps

Checking the procps changelog here:

http://procps.sourceforge.net/changelog.html

In Version 3.1.5
serious hidden-process problem (3.1.3+) fixed

Maybe you need to update your procps package.

Also, check if you can find in /proc a directory that corresponds with the hidden processes.

Re: Hidden PIDs

Danesh Qureshi — Fri, 28 Sep 2007 11:36:35 GMT

I have attached a file containing output of
rpm -qi procps

I have cd to /proc directory

The interesting thing is when I use ls -l I cannot see those PIDs but when I ls -la I can those PIDs which are also in top.

I have appended the output
of ls -l and ls -la in the attached file to see what I am seeing.

What are your thoughts now?

Re: Hidden PIDs

Ivan Ferreira — Fri, 28 Sep 2007 15:00:42 GMT

Investigating a little further, it seems like these "dot directories" in /proc, are threads, and you can display with p-m option of ps, for example, ps auxm

Re: Hidden PIDs

Ivan Ferreira — Fri, 28 Sep 2007 15:09:50 GMT

Please see this thread:

http://uwsg.indiana.edu/hypermail/linux/kernel/0612.1/1851.html

Re: Hidden PIDs

Danesh Qureshi — Mon, 01 Oct 2007 09:10:21 GMT

Hi Ivan,

When I type "ps auxm" it lists a lot of output.

See attached file which contains output of "ps auxm".

So what are these threads in Linux and do I need to worry about them. I have noticed that these hidden processes take up a lot of memory. See output of hidden PIDs using large amount of memory. Why is that?

Re: Hidden PIDs

Ivan Ferreira — Mon, 01 Oct 2007 15:54:29 GMT

>>> So what are these threads in Linux and do I need to worry about them.

Threads are a way for a program to fork (or split) itself into two or more simultaneously (or pseudo-simultaneously) running tasks. I don't know what is, maybe your developers? What does the Java application on your system? They key thing is to identify if the application should fork too many threads, or the threads are not ending.

Until you understand the function of these programs, you should be worried.

I have noticed that these hidden processes take up a lot of memory. See output of hidden PIDs using large amount of memory. Why is that?

Java programs are know to use a lot of memmory. But Linux handles the memory so most part of the memory used by process are shared, and a small part "private".

Re: Hidden PIDs

Heironimus — Tue, 02 Oct 2007 10:08:33 GMT

Threads (aka lightweight processes) don't have their own resources, they all share the resources of the process that owns them. If you look through the list, whole groups of threads have the exact same memory use. The entire process is using that much memory, all those threads are just sharing it. If any one thread increases the memory use then the parent process actually grows and every thread it owns will show the increase.

Should you worry about the threads? Are you the application developer? Are you the application admin? Are you getting complaints about performance or seeing resource issues on the system? If not, then you probably don't need worry. Threads are a normal part of a running system, especially if you're hosting Java apps.

Re: Hidden PIDs

Danesh Qureshi — Wed, 03 Oct 2007 11:41:51 GMT

The DBA has logged a call with WebCT who develop the application and are awaiting feedback from them to let us know how to resolve the performance issue with their WebCT application. I will let you know the outcome of the issue. In the mean time thank you all for your help and advice.