User default shell impact on program performance

[Prasad Pillutla]

Hi
We are running HPUX 11.0 Class N machine. We are using Oracle 734 with associated tool like Developer 2000 tools and pro*c for batch.
Recently, we have noticed a significant performance degrade on an annual batch job when run in production environment. But the same job on development environment with lower resources runs 10 times faster.
After dredful investigations, it pointed to the user environment who is submitting the job.
In the development environment the user's default shell is /usr/bin/ksh.
In production environment the user's default shell is /usr/bin/sh.
We proved the above theory by modifying development default shell to /usr/bin/sh and it run slower when compared to the default shell of /usr/bin/ksh.
This job is submitted throught a third party tool in the background.
Any thoughts on this, like job scheduling in the two shells, priority etc.
Any help is much appreciated.

[Patrick Wallek]

That is odd.

The only thing I can think of right now is that you may have issues with various 'ulimit' values. You might check that for both shells.

[Bill Hassell]

Unless the shell (ksh or sh) have massive tasks to perform (which will show up as huge CPU consumption), there is likely another reason. ksh and sh are both POSIX shells and have similar code bases. Normally, shell scripts complete so quickly that it is not possible to measure the time they use.

Since this is a batch job, the only way to trace what components are causing the dalays is to start Measureware and configure the process groups carefully to separate the shell and Oracle processes. I would be concerned about using Oracle 734...it is really old, especially for use on HP-UX 64bit 11.0 or later.

[Prasad Pillutla]

Thanks .
While the job is running, the average cpu load is around1.5 to 2. This machine has 6cpus and 12GB mem.

[T G Manikandan]

I would like to add some points here

1.Do you have the similar configuration of Oracle instance on both the machines or you are pointing the
application to look at a Oracle instance remotely from there.
The point is that is your application looking for some files outside the system that is on the network.

2.Do you have a log file for your application so that you could trace which batch is taking long time.
Also you could use GlancePlus to find the process taking too much resources.


3.Are your kernel parameters tuned as recommended by the application.


Assuming that you have investigated a lot about the problem,I dont think the shell of the user should matter.

Thanks

[Timo Ruiter]

Assuming it really is the shell that is causing the performance degradation of your job, you should concentrate your investigation on the statements in your shell script that are actually processed by the shell, e.g. testing for existance of files, expanding wildcards, etcetera.

I found that testing a simple thing like the value of a variable in the posix shell (/usr/bin/sh) using

typeset -i i=0
while test $i -lt 10000
do
i=$((i + 1))
done

is not very efficient.
Using

typeset -i i=0
while [[ $i -lt 10000 ]]
do
i=$((i + 1))
done

runs two times as fast.

I'm not sure, but I think that the 'test' statement in the Korn shell is a builtin command, which is not the case with the posix shell, which uses /bin/test to test things.

Maybe this helps

Timo

[Chris Wilshaw]

Prasad,

We have also recently had upgrade performance issues of a similar nature.

A solution that we found was that the system performance using /usr/bin/sh was significantly reduced when the EDITOR variable was set to vi (no such problem existed when using /usr/bin/ksh).

In our case, we noticed that when we scanned barcode data into the system, using ksh, the entire code was displayed on screen immediately, whereas when using sh, with EDITOR=vi set in the users .profile a distinct pause was present part way through the display of the code.

You may wish to check if this is the case on your system too.

Chris

[harry d brown jr]

Can you do a

what /usr/bin/sh

Also, what's the last patch bundle installed?

live free or die
harry

[Tom Maloy]

So just changing the shell from ksh to sh slowed the job down a lot.

You should check the startup files and differences between shell built-in commands.

The ksh startup files (/etc/profile, $HOME/.profile, $ENV) may be setting the PATH differently or setting other variables that have an impact.

ksh has many more built-ins than sh does. So sh may have to spawn a sub-shell to do an echo while ksh uses it's built-in equivalent. sh may be spawning many sub-shells during this process.

Tom

[David Totsch]

I have seldom seen the shell get in the way, so I, too, would have been blaming differences in how Oracle is configured in the two nodes. However, I *have* seen the shell get in the way. Pet Peve #847: what are these "not a teletype" messages in my output? Answer: why are you setting an interactive environment for a batch job? EDITOR is a known suspect. ENV is in the top-10 wanted list. ENV is great for making sure that an environment specific to ksh or POSIX is set appropriately, but, exporting ENV can turn a demure shell into a performance hog (it causes the environment file pointed to by ENV to be sourced *every* time a shell script is called -- many of the commands in any UNIX are shell scripts). If your script is calling other scripts repeatedly, the problem compounds. Plus, shells are not very efficient (they are interpreted code), fortunately, their impact is _usually_ negligible. Want further proof? If the script is short, port it back to Bourne (/usr/old/bin/sh) and see what happens.

-dlt-

[Bill Hassell]

And to answer pet peeve #847, all profiles (including /etc/profile, .profile, .kshrc, bashrc, etc) must explicitly test for the lack of a controlling tty device. The "not a teletype" messages seen in logfiles are due to commands like ttytype, tabs, tset, etc which are always executed. Instead, bypass all terminal or tty commands like this:

[[ -o interactive ]] && INTERACTIVE=/usr/bin/true || INTERACTIVE=/usr/bin/false

if $INTERACTIVE
then
eval $(/usr/bin/ttytype -s)
/usr/bin/tabs
fi

Now you won't see the "typewriter" message.