Lot of sh children process (Oracle user) use CPU

[MR VILLOT MR MONTAGNE]

Hi,

I have a host with oracle applications and one instance. When i launch top command, i get several children process using CPU :
CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
1 ? 7089 app1159 152 20 532M 152M run 342:10 92.83 92.67 java
5 pts/0 27091 app1159 255 39 2288K 576K run 7633:17 44.54 44.46 sh
4 pts/9 25448 app1159 255 39 2288K 576K run 6193:32 44.36 44.28 sh
7 pts/4 19012 app1159 255 39 2288K 576K run 10306:25 44.10 44.02 sh
7 pts/6 16509 app1159 255 39 2288K 584K run 5488:15 43.82 43.75 sh
5 pts/5 3637 app1159 255 39 2288K 576K run 6533:23 43.28 43.20 sh
4 pts/tc 13957 app1159 255 39 2208K 508K run 5315:31 43.20 43.12 sh

If I do, ps -ef | grep 25448, I get for example :
appora 25448 1 255 Feb 5 pts/9 6194:22 /usr/bin/sh
I don't know how to identify what it is used by Oracle (to kill process).

Thanks for help
Laurent

[Sridhar Bhaskarla]

Hi Laurent,

25448 is the pid of this shell process. If it spawned child processes, then you would see 25448 as the ppid (parent process id) for other processes. Your ps -ef would have gotten those processes too.

Since it is not showing any child processes, it's most likely that these are scripts run by the user 'appora'.

But before killing these sessions, I would run tusc on the processes to make sure. You can also run 'glance' and see the process status for each of these processes.

-Sri

[Steven E. Protter]

ps -ef | grep app

This will get you a process list

I'm attaching a pattern match process killer.

It will wipe out oracle if you are not careful.

SEP

[Bill Hassell · Accepted solution]

As Steven says, be careful about the processes that you try to kill. ps | grep is a really bad way to track down processes. ps has it's own set of grep-tools which are 100x better. grep cannot match on a field, it just blindly looks for a string so looking for sh will find ksh, csh, bash, unhasdaemon and even user ID's like sherry and josh.

So let's start by finding all the sh processes and sorting by CPU time:

UNIX95= ps -C sh -o pcpu,pid,ppid,ruser,args | sort -rn | more -e

That will list the biggest CPU hogs at the top (the column headings will be at the bottom of the list). Now, sh shells that consume huge amounts of CPU time (more than a fraction of 1%) are usually broken connections. If you have a lot of PC users, these may be crashed sessions that have not been setup correctly in /etc/profile. The most common problem is that someone removed the last line in /etc/profile that reads:

trap 1 2 3

What this does is to restore 'normal' processing to the shell whenever it is asked to terminate (including a crashed telnet session). The beginning of /etc/profile has: trap "" 1 2 3 which turns OFF trap processing including SIGHUP (hangup). Then when the session is trashed by the PC user, the shell goes into an infinite loop trying to talk to a non-existent terminal.

Once you fix the trap processing (verify that traps are normal by typing trap at the shell prompt. There should be nothing returned. If instead you see:

$ trap
trap -- '' QUIT
trap -- '' INT
trap -- '' HUP

then trap restoration in /etc/profile or .profile has been broken.

As for the existing shell processes that are consuming massive CPU time, look for the children of these shell sessions. Probably the fastest is to search for the PID (in this case, grep is OK since we want PID and PPID values). If the process ID is 1234 then:

ps -ef | grep 1234

If these processes have no child processes where the parent (PPID) is the shell's ID, then use kill -15 to terminate the shells. If this does not work, you'll have to use kill -9 and then go to work looking for the messed up trap setting and get it fixed. This often happens when a newbie shell writer turns off traps to prevent escapes by the user but also eliminates the HUP trap.

[MR VILLOT MR MONTAGNE]

Thanks for help,

I copy /etc/profile on /myhome but must I let the first line trap "" 1 2 3 in .profile?
It is set to cancel signals during loading of my profile.
Now, if I type trap, i get :
trap -- '' QUIT
trap -- '' INT
trap -- '' HUP
trap -- 'echo '\''logout'\' EXIT

The .profile for user is get from /etc/skel/.profile (without settings about trap)

Thanks
Laurent

[Bill Hassell]

/etc/profile must NOT be copied to your $HOME directory. /etc/profile is globale and every login will run /etc/profile. The trap "" 1 2 3 should be at the top of /etc/profile and trap 1 2 3 MUST be at the bottom of /etc/profile. If not, someone with root privileges has messed around with the script. A copy (unmodified version) of /etc/profile is located in /usr/newconfig/etc/profile, but don't overwrite /etc/profile without comparing the two. Your current problem is that /etc/profile is missing trap 1 2 3 (notice: no "" on the second trap command) and that is causing all your shells to consume CPU time.