topic Re: Process hangs with 100% SYS in Operating System - HP-UX

Process hangs with 100% SYS

Fredrik Lännergren — Thu, 16 Feb 2006 05:22:55 GMT

Hi,

I have a problem that is confusing me...

Sometimes (i.e., not always) one of the the processes (program developed by us) that I start hangs, consuming 100% system usage. gdb can't attach to it and truss is refused entry. When two of these hang simultaneously, I can't even log on to the machine (it being a 2 CPU one)...

How can I figure out what is happening?
It's HP-UX 11.23.

Thanks in advance,

Re: Process hangs with 100% SYS

Muthukumar_5 — Thu, 16 Feb 2006 05:32:32 GMT

See process usage as,

# vmstat
# UNIX95= ps -ef -o cpu,pcpu,pid,comm

It will show the highest cpu% using process.

# top

will also show the process with cpu% rating. (Not accurate always).

--
Muthu

Re: Process hangs with 100% SYS

Arunvijai_4 — Thu, 16 Feb 2006 05:36:34 GMT

Hello,

A better way to debug the problem with logging enabled in your application and try running it.

Also, did you change any kernel parameters ?

-Arun

Re: Process hangs with 100% SYS

Matti_Kurkela — Thu, 16 Feb 2006 05:44:54 GMT

Sounds like the process might be doing something like an infinite loop, calling some system function repeatedly or with insane parameters that make the system function take ages to complete. Because most of the time is spent inside the system function, you get 100% SYS usage.

You could freeze the process for investigation with "kill -STOP ". A stopped process will not respond to most signals, but you can use tools like lsof to get more information on what the process was doing. You can then unfreeze the process with "kill -CONT ", and terminate it as necessary.

With "kill -ABRT " or "kill -QUIT " you should be able to force the process to exit creating a core dump file. You might then be able to get more information by analyzing the core dump.

Of course you must have enabled the core dump creation (if "ulimit -c" is set to zero, core file creation is disabled) before the process is started.

Re: Process hangs with 100% SYS

Fredrik Lännergren — Thu, 16 Feb 2006 06:25:08 GMT

Thanks for the answers, but:

Muthukumar: I already know what process is doing this.

Arunvijai: This is during startup of the process, and the source code is a few hundred kloc. It would take awhile to do that, so I would like to find some shortcut first...

Matti Kurkela: The process doesn't respond to _any_ kills, including kill -9, kill -STOP and kill -ABRT. It just continues happily eating 100% system time.

Mind you, this problem only happens sometimes, so it's some kind of timing issue; inserting debug code may change the behavior of the program.

Any other ideas? *looks hopeful*

Re: Process hangs with 100% SYS

Fredrik Lännergren — Thu, 16 Feb 2006 07:59:55 GMT

I just found out that I can't kill the process because it has pri of 134... in fact, _all_ my processes have a PRI above 128....but what could possibly make that happen?

Appreciate any hint... faulty kernel parameters, maybe?

Re: Process hangs with 100% SYS

Victor BERRIDGE — Thu, 16 Feb 2006 08:25:10 GMT

Hi Fredrik,

I dont see what PRI has to do with not being able to do a kill...
Under which UID is the program running?
If root why?
If not root did you try as root to kill -9 PI and PPID at the same time?
Now as root did you use glance/gpm to monitor the systems activity before then during your program execution? you should be able to follow and see when it is consuming resources what it is doing or why it is waiting...

It is difficult to give you some advice since you didnt explain what the program does and so we dont have much clues...

All the best
Victor

Re: Process hangs with 100% SYS

Hein van den Heuvel — Thu, 16 Feb 2006 08:36:31 GMT

What is the (virtual) memory situtation.
If you are low-on / out-off free pages, and teh swapper kicks in aggresivelly then you may see behaviour similar to this: lots of sys time, unresponsive, ...

fwiw,
Hein.

Re: Process hangs with 100% SYS

Fredrik Lännergren — Thu, 16 Feb 2006 09:09:21 GMT

Hi Victor,

I saw a message at http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=939681
saying that kernel prio processes can't be signalled. A message in thread http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=190861
says that PRI 128-153 are unkillable. Attempting to kill it along with its PPID, there's no change.

The process is owned by me, not root. I'm not allowed root access. The process cannot be attached to with a debugger, nor trussed. GlancePlus didn't give much... Not a single system call is made, only four files are open, there are 0% wait state for disk io, streams, semaphores, network or anything else. Disk IO rate is 0. When I attempt to look at the process memory regions in GlancePlus, it hangs totally and can't be killed.

As for what the program does... well, given that is encompasses a few hundred thousand lines of code, that would take some explaining. Thing is, I don't even know in what state of startup it is, so I don't know what's going on either. Lots of file I/O is usually involved at startup, though. All I know is that it's early on; its mirror processes running the same binary use 53 MB, whereas this copy only got to 1.3 MB RAM before it "stuck" at 100% system, and it has stuck for the last six hours.

I'm not sure what you would like to know?

Re: Process hangs with 100% SYS

Kent Ostby — Thu, 16 Feb 2006 09:14:51 GMT

Fredrik --

If you have q4 on your system, you might be able to get a stack trace by running:

q4 -p /stand/vmunix /dev/kmem

then at the q4 prompt:

trace processor 0
trace processor 1

One of those will be your q4 process probably, but the other should be your hung process.

Another way would be to get a TOC of your machine when this is happening and then use
the document OZBEKBRC00000611 to pull stack traces of all the processes (the part using analyze.pl and the Analyze command).

http://www1.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=OZBEKBRC00000611

Re: Process hangs with 100% SYS

Victor BERRIDGE — Thu, 16 Feb 2006 09:22:44 GMT

Just a little one...

What is the parent PID ?

Courage!
Victor

Re: Process hangs with 100% SYS

Fredrik Lännergren — Thu, 16 Feb 2006 09:25:43 GMT

Hi Hein,

These numbers are from gpm memory report:
-----------------------------------------
Phys Mem: 4.0 gb
Sys Mem: 694 mb
Buf Cache: 1.2 gb
User Mem: 1.2 gb
Free Mem: 940 mb

Total VM: 1.6 gb
Active VM: 1.4 gb
------------------------------------------
On the other hand, the CPU report says that it spends an average of over 50% on V (page) faults? How can this add up?

Re: Process hangs with 100% SYS

Fredrik Lännergren — Thu, 16 Feb 2006 09:29:00 GMT

Hi Victor,
PPID is 3919, and its PPID in turn is 1, which it should be since that process, our monitor so to speak, supposed to run as a daemon.

BR,
Fredrik

Re: Process hangs with 100% SYS

Fredrik Lännergren — Thu, 16 Feb 2006 09:30:49 GMT

Hi Kent,

I couldn't find q4 on the machine, unfortunately. Besides, it sounds to me like you have to be root to run it? I don't have root access....

Re: Process hangs with 100% SYS

Victor BERRIDGE — Thu, 16 Feb 2006 09:40:51 GMT

q4 should be in /usr/contrib/bin, but wait and follow Kents advice for he is a q4 guru...

All the best
Victor

Re: Process hangs with 100% SYS

Fredrik Lännergren — Thu, 16 Feb 2006 09:43:53 GMT

Hi Victor,

well, in that case, I definitely don't have q4. :)

BR,
Fredrik

Re: Process hangs with 100% SYS

Victor BERRIDGE — Thu, 16 Feb 2006 10:03:07 GMT

Are you on an itanium box?
when you type model, what do you get?

Normally on (well PA-risc anyway) it is with the core-os...

All the best
Victor

Re: Process hangs with 100% SYS

Fredrik Lännergren — Thu, 16 Feb 2006 10:05:44 GMT

Hi Victor,

no, actually, it's not an Itanium. Unless someone has fooled me greatly....

"model" gives: 9000/800/A500-7X

BR,
Fredrik

Re: Process hangs with 100% SYS

Victor BERRIDGE — Thu, 16 Feb 2006 10:09:54 GMT

What about :
ant:/usr/contrib/Q4/bin $ ll
total 12394
-r-xr-xr-x 1 bin bin 250792 Nov 5 2003 getasm
-r-xr-xr-x 1 bin bin 460837 Nov 5 2003 kmeminfo
-r-xr-xr-x 1 bin bin 495616 Nov 5 2003 nm.elf
-r-xr-xr-x 1 bin bin 932 Nov 5 2003 nm.q4.sw
-r-xr-xr-x 1 bin bin 32768 Nov 5 2003 nm.som
-r-xr-xr-x 1 bin bin 1546916 Nov 5 2003 perl
-r-xr-xr-x 1 bin bin 61 Nov 5 2003 q4
-r-xr-xr-x 1 bin bin 139615 Nov 5 2003 q4.pxdb
-r-xr-xr-x 1 bin bin 3119736 Nov 5 2003 q4exe
-r-xr-xr-x 1 bin bin 2292 Nov 5 2003 q4pxdb
-r-xr-xr-x 1 bin bin 290816 Nov 5 2003 q4pxdb64
-r-xr-xr-x 1 bin bin 559 Nov 5 2003 set_env
ant:/usr/contrib/Q4/bin $

All the best
Victor

Re: Process hangs with 100% SYS

Fredrik Lännergren — Thu, 16 Feb 2006 10:17:05 GMT

Hi Victor,

ah, _there_ it was. I now tried running what Kent wrote, but all I got was:

@(#) q4 $Revision: 11.X B.11.23l Wed Jun 23 18:05:11 PDT 2004$ 0
q4: (error) failed to open kmem, errno = d

I guess that's q4 lingo for "DENIED" ?

BR,
Fredrik