Scripting question

[Kris Spander]

Hi,

I have a script that does four large sort operations that take about 25-35 minutes each to complete. None of the sorts depends on the others so I am able to start each one from a master script as a background job. It is important that all of them finish before the master script continues. Right know I have a sleep 2400 after the fourth sort operation is started but sometimes that delay is not long enough. I am thinking about using ps to check the status of all the sorts. Does anybody have an example using ps to look for different processes?

Thanks,
Kris

[A. Clay Stephenson · Accepted solution]

We don't need no stinkin' ps there is already a shell built-in to wait for background processes to complete. Let's see; it's called "wait".

echo "Starting sort(s) in background"
mysort1 &
mysort2 &
mysort3 &
mysort4 &
echo "Waiting for all sorts to finish"
wait
echo "Now safe to proceed".

--------------------------------

That's all there is to it! Man sh-posix for details and look for "wait".

[A. Clay Stephenson]

I suppose that I should add that wait comes in 2 forms:

wait # waits for all background processes to finish

wait ${BKGND_PID} # wait until a particular background process is finished. Noting the ${!} is the PID of the last background task started makes it possible to capture that PID and store it in a variable.

sort1 &
SORT1_PID=${!}
...
...
wait ${SORT1_PID}

You would probably gain a bit more insight by reading the man 2 wait pages because there the underlying mechanisms (the system calls) of the shell's wait are described.

[Dennis Handly]

If you truly have 4 large sorts you may want to measure the time it takes to do it with Clay's suggestion verses just doing them one at a time without putting them in the background.

If you have gigabytes of data or don't have 4 CPUs, it may not be helpful doing them all at once?

Of course if you find it takes lots less than 4 * 25 minutes, I guess you don't have to do my experiment.

[Marvin Strong]

as was pointed out, wait is what you should use. replace your sleep with wait and you should get the desired results.

[James R. Ferguson]

Hi Kris:

Along the lines of performance and *testing* the most expedient methodology as Dennis suggests, might include choosing different filesystems for sort's stringing and merging phases by setting the 'TMPDIR' environmental variable differently for any concurrent operations. See the 'sort(1)' manpages for more information.

More importantly, if your sort is making extensive transformations during its key comparisons (e.g. performing substring extractions to form sort keys) you should consider pre-creating those keys in a Schwartzian transform to reduce CPU overhead.

Optimizing the *way* you perform your sorts may offer as much (or more) performance gain as trying to run them in parallel.

Regards!

...JRF...

[A. Clay Stephenson]

I absolutely agree that choosing the correct sorting algorithms and methodologies is wise. You may already be doing that. "Wait" is the answer to your question but you may not be asking the right question.

I can tell you a story that is absolutely true. Many decades ago, I was part of a 3-person company that decided to tackle the DEC, Data General, and IBM mini-computers of the world with clusters of Apple II's connected to shared, very large (20MiB) hard drives. (Each of these 20MiB drives cost $6750.) This was before the IBM PC was introduced. One of the primary tasks that had to be done was large sorts. At that time (and still is) my only formal Computer Science training was 6 hours of FORTRAN and the only sort I had ever done (or knew about) was the bubble sort. My first implementation on an inversely ordered file took about 3 minutes to complete and this was a small data set. Well, I knew that dog wouldn't hunt on the real data so I sat down and starting thinking about sorting. I realized that if I could somehow divide the data into values smaller than some "fulcrum" value and larger than some "fulcrum" value and then divide those smaller chunks the same way and continue until the chunks were 3 or less then I would have a sorted data set and because the exchanges were occurring over large extents rather than just between adjacent elements of the bubble sort, the sort would be many orders of magnitude faster. Literally the first time I ran this Pascal program, it finished in 2 seconds where the bubble sort took 3 minutes! I knew there had to be a bug because it ran so fast but then I realized that for this sort the inversely ordered data set was almost as trivial as the ordered data set because the sort would be completed in a single pass. I later learned (when I actually found out that there were books about algorithms) that the sort I just described was Hoare's quicksort. I could have saved a lot of hard thinking by going to the library first but, in physics, one tends to think in terms of first principles.

This is the long way of saying that you really need to decide if you are asking the right question.

Food for thought,
Clay

[Kris Spander]

Hi guys,

Thanks for all the ideas! We are running this on a 6 CPU RP-7400 and the sorts do complete sooner when executing in parallel. The 25-35 minute times are actually faster than our earlier sorts that took over an hour to finish. We use Syncsort. We are already dividing the work across separate file systems.

"Wait" was really the answer I needed. Duh!
I welcome any other ideas to make this operation faster. At least now thanks to you guys the master script waits properly.

Thanks,
Kris

[James R. Ferguson]

Hi Kris:

"...other ideas...?" How about my original [unscored] reply [above]? :-)

Regards!

...JRF...

[Kris Spander]

Hi JRF,

I'm so sorry. I didn't realize that I had missed assigning you points. To make up for it, I assigned 10 points for each of your replies.

Thanks,
Kris