topic Re: Process consuming 100% CPU in Operating System - HP-UX

Process consuming 100% CPU

hpuxsa — Wed, 22 Dec 2004 02:11:16 GMT

We have Oracle CRM application 11.5.9 running on Oracle DB 9.2.5 and HPUX11.11.

When an user connects to the application and run some forms it starts a process f60webmx. We have noticed that one of the 'f60webmx' process was consuming 100% cpu. We have later found that it happens only when the connection that started the process goes into a 'CLOSE_WAIT' state (the user had terminated the session abruptly).

Following are the setting for
ndd -get /dev/tcp tcp_keepalive_interval
7200000
# ndd -get /dev/tcp tcp_ip_abort_interval
600000

Currently we fix this problem by killing the process everytime. Is there anything we can do from the OS side to fix this.

Thanks,
Franklin.

Re: Process consuming 100% CPU

KapilRaj — Wed, 22 Dec 2004 03:36:28 GMT

We faced a similier one .. It is an application bug and must be taken with the developers ;).

The root cause being the application who does not call the "close" function to release the socket

Kaps

Re: Process consuming 100% CPU

Ralph Grothe — Wed, 22 Dec 2004 04:23:19 GMT

Maybe your developers of f60webmx could implement a signalhandler to act on receiving a SIGIO (see man sigaction(2), signal(5), select(5), fcntl(2)),
either by attempting a clean close of the socket or a graceful shutdown?

Re: Process consuming 100% CPU

Eric Antunes — Wed, 22 Dec 2004 04:27:00 GMT

Hi,

Ask programers if they had set 'Interaction Mode' property of the Form to 'Non-Blocking'? If so, see the details for bug 1081777.

Best Regards,

Eric Antunes

Re: Process consuming 100% CPU

TwoProc — Wed, 22 Dec 2004 18:03:46 GMT

Well, that's funny - I've got the SAME exact configuration as you and I see that same problem. FYI - this problem has been around since at least Oracle Apps 10.7 (which was the first use of the forms server that I was exposed to, and I think, the first usage of forms server (f60webmx) that was used by Oracle. This problem also exists on Linux as well as HPUX - so I'm going to conclude it's on all system variants.

Anyway, I don't have a direct fix - except I run the following script on our Linux Middle Tiers (used to be HPUX - backend is still HPUX). Script is attached to this response. Be aware that it may take a bit of tweaking to run in HPUX - but not much. It is written for bash, but I don't think I've got anything in there that's not ksh compliant (my brain writes scripts in ksh anyways by default).

You can play with the script to see how it works for you. What it does is runs top to see if there are processes that meet certain requirements that are set via variables in the script, then it knocks them off. It scrolls lots of output to tell you what it's doing. It has a switch to make it actually kill processes "-k", if you don't pass the kill switch to it - it will just run and tell you what it would do if you let it. From that you can observe whether or not it fits the bill for you (I believe it's exactly what you need until Oracle fixes the problem - and don't hold your breath - I've been waiting for six years).

The script currently looks for f60webmx's taking over 60% cpu for over 60 minutes of runtime. I do this for starters - you can ratchet down the numbers as you see are able. Just make sure and be conservative with these numbers for a while. Keep in mind also, that the number for how much cpu time used by a forms server could be much less (and therefore much better able to detect) in the morning, than in the afternoon after it's more difficult to tell if a process has 40 minutes of process time "honestly" or due to a runaway process.

Anyways, put this script in a while loop on a terminal (like below) or you can cron job it. I prefer to leave it running on an old spare terminal that I have so I can see it when I care to...

while /bin/true
do
date
find_runaway_f60 -k # Leave -k off if you just want to watch it ...
sleep 300 #wait 5 min
done