1820880 Members
3620 Online
109628 Solutions
New Discussion юеВ

Strange "ksh" problem

 
SOLVED
Go to solution
yyghp
Super Advisor

Strange "ksh" problem

I met a strange problem when I run a shell script with Pro*C program, which has parallel threads.
There's a loop in the shell script(attachment is the whole script):

# Run the precostcalc program in parallel each processing a supplier
# This line is commented & next line added by Indranil ------cat $SUPPLIERS | while read supplier
while read supplier
do
if [ `jobs | wc -l` -lt $SLOTS ]
then
(
precostcalc $CONNECT $supplier $ADDL_OPTIONS || touch $failed;
) &
else
# Loop until a thread becomes available
while [ `jobs | wc -l` -ge $SLOTS ]
do
: # Null command
done
(
precostcalc $CONNECT $supplier $ADDL_OPTIONS || touch $failed;
) &
fi


I found that such loop never stop (endless loop !) until I pressed "Ctrl+C", which resulted:
Pid 20983 received a SIGSEGV for stack growth failure.
Possible causes: insufficient memory or swap space,
or stack size exceeded maxssiz.
Memory fault(coredump)

because `jobs | wc -l` always 1 and $SLOTS = 1, that's why the loop cannot stop.

But if I change the 1st line of the script from "#! /usr/bin/ksh" to "#! /usr/bin/sh", that is, to use the "sh" instead of "ksh", the vaule of `jobs | wc -l` can change to 0 after several circles which stops the loop !

What difference of "ksh" and "sh" makes this happened ? Do they have different way to handle multi-thread ? or something else ?

I have applied the latest patch for "ksh": PHCO_30316, but the loop still cannot be stopped.

Any suggestion for me?
Thanks a lot !
15 REPLIES 15
yyghp
Super Advisor

Re: Strange "ksh" problem

Up...
Could anyone reply my ticket ?
Problem with "ksh" and "sh"...
Thanks !
harry d brown jr
Honored Contributor

Re: Strange "ksh" problem

In your script, add:

set -x

near the top but not at the top.


Also, you should NEVER use ksh to do looping like that without sleeps because you are hammering your machine if all of the "slots are full" with senseless executions/forks/pipes. Add a sleep or change your precostcalc program to initate multiple instances.

live free or die
harry d brown jr
Live Free or Die
hein coulier
Frequent Advisor

Re: Strange "ksh" problem

I'm not sure it's an alternative to you, but maybe you can reduce the looping by

execute_a_command &
pid=$!

wait ${pid}
john korterman
Honored Contributor

Re: Strange "ksh" problem

Hi,
it might have to do with the use of "||", as e.g. described in Document id: KBRC00005490

http://www5.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000062907544

regards,
John K.
it would be nice if you always got a second chance
yyghp
Super Advisor

Re: Strange "ksh" problem

Hi harry,

Yes, you are right, adding "sleep" into the loop can solve the problem, and yesterday I found that even one or two "echo" into the loop can help, I think it just made millisecond "sleep", incredible !
Why??
Why such sleep makes that difference ? Why the "sh"(bsh) doesn't have this issue but "ksh" does !?

Thanks again !
yyghp
Super Advisor

Re: Strange "ksh" problem

Hi John,

Thanks for your reply !
I read "ksh: handles $? and || command separator differently to sh" from your link that you mentioned, but I don't understand the solution for my case. I don't use "$?" in my script. Could you please tell me details about what I should change? Thanks!



# Run the precostcalc program in parallel each processing a supplier
# This line is commented & next line added by Indranil ------cat $SUPPLIERS | while read supplier
while read supplier
do
if [ `jobs | wc -l` -lt $SLOTS ]
then
(
precostcalc $CONNECT $supplier $ADDL_OPTIONS || touch $failed;
) &
else
# Loop until a thread becomes available
while [ `jobs | wc -l` -ge $SLOTS ]
do
: # Null command
done
(
precostcalc $CONNECT $supplier $ADDL_OPTIONS || touch $failed;
) &
fi


Jeroen Peereboom
Honored Contributor

Re: Strange "ksh" problem

I don't know the exact differences between sh and ksh.
But I have 2 remarks:
1. add a sleep, as already suggested by Harry.

2. Assuming $SLOTS is more or less static, you can remove the then part of the if-construction (and so the whole if-construction) yielding:

while read supplier
do
# Loop until a thread becomes available
while [ `jobs | wc -l` -ge $SLOTS ]
do
sleep 1
done
(
precostcalc $CONNECT $supplier $ADDL_OPTIONS || touch $failed;
) &

By the way, jobs is a shell built in command. Maybe that's something else to consider.

JP
yyghp
Super Advisor

Re: Strange "ksh" problem

Thanks Jeroen !

I am still hoping that someone can tell me what difference between "ksh" and "sh" made this happen. Because if I don't have any problem to run it with "sh", but I get endless loop when I run it with "ksh"...

I hope I don't need to change the source code because it's another company wrote that code and they said the other clients don't have problem to run the script with "ksh"...

Yes, Jeroen, I suspected "jobs" as well because when I used "set -x" on the top, I could trace the value of "`jobs | wc -l`", and they had different numbers between "ksh" and "sh"...

Thanks again !
john korterman
Honored Contributor

Re: Strange "ksh" problem

Hi,
sorry about my first posting; I did not see that you had posted the whole script - and after reading that it I am painfully aware, that the problem has nothing to do with "||".
I think that the following doc describes your situation concerning putting processes in a subshell in the background, but unfortunately, does not suggest any solution.
Document id: A1456085
http://www5.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000006352554

regards,
John K.
it would be nice if you always got a second chance
yyghp
Super Advisor

Re: Strange "ksh" problem

Thanks John !

Anyone else can help me to find out the reason for this case ?
Why does the "ksh" work fine if I put some "echo"s or a "sleep" inside the loop, it can work like running as "sh" ?

Thanks again !
yyghp
Super Advisor

Re: Strange "ksh" problem

Hi Harry,

You said "you should NEVER use ksh to do looping like that without sleeps because you are hammering your machine if all of the "slots are full" with senseless executions/forks/pipes." in your post, I thought you might know the "substaintial" reason for this, and some differences between "ksh" and "sh"...
I am waiting for your response... Thanks !
Fred Ruffet
Honored Contributor

Re: Strange "ksh" problem

I imagine Harry is refering to this line in beggining of ksh man page :
"ksh is a command programming language"

In fact Ksh is a bit more complex, offers more feature, but is a little bit more heavy. So it takes more time to launch and initiate, and uses more memory.

Regards,

Fred
--

"Reality is just a point of view." (P. K. D.)
yyghp
Super Advisor

Re: Strange "ksh" problem

Fred, I don't think it's just a memory issue... if so, "sleep" won't make difference...
Fred Ruffet
Honored Contributor

Re: Strange "ksh" problem

Begining of the sentence : "it takes more time to launch and initiate". :)

Regards,

Fred
--

"Reality is just a point of view." (P. K. D.)
john korterman
Honored Contributor
Solution

Re: Strange "ksh" problem

Hi again,

I think that the reason why the loop works in ksh when inserting a sleep command is that the sleep command simply prevents the script from using 100% of the cpu resources; without the sleep command inserted, the script may *hammer etc.* and prevent any other process from getting any resources and thus prevent them from finishing. My argument is then that the sleep command makes it possible for other processes to finish, therefore enabling the script itself eventually to finish.
However, I still think that the jobs command in ksh cannot *see* the jobs, as these are launched in a sub-shell. So what I'm saying is actually that it is rather accidental that the script at all stops when using ksh.
(And the reason why it works in sh without sleep is that sh can see the jobs).
But perhaps other members can set me right.

Regards,
John K.
it would be nice if you always got a second chance