Scripting question

[Kris Spander]

Hi,

I have a script that does four large sort operations that take about 25-35 minutes each to complete. None of the sorts depends on the others so I am able to start each one from a master script as a background job. It is important that all of them finish before the master script continues. Right know I have a sleep 2400 after the fourth sort operation is started but sometimes that delay is not long enough. I am thinking about using ps to check the status of all the sorts. Does anybody have an example using ps to look for different processes?

Thanks,
Kris

[A. Clay Stephenson · Accepted solution]

We don't need no stinkin' ps there is already a shell built-in to wait for background processes to complete. Let's see; it's called "wait".

echo "Starting sort(s) in background"
mysort1 &
mysort2 &
mysort3 &
mysort4 &
echo "Waiting for all sorts to finish"
wait
echo "Now safe to proceed".

--------------------------------

That's all there is to it! Man sh-posix for details and look for "wait".

[A. Clay Stephenson]

I suppose that I should add that wait comes in 2 forms:

wait # waits for all background processes to finish

wait ${BKGND_PID} # wait until a particular background process is finished. Noting the ${!} is the PID of the last background task started makes it possible to capture that PID and store it in a variable.

sort1 &
SORT1_PID=${!}
...
...
wait ${SORT1_PID}

You would probably gain a bit more insight by reading the man 2 wait pages because there the underlying mechanisms (the system calls) of the shell's wait are described.

[Dennis Handly]

If you truly have 4 large sorts you may want to measure the time it takes to do it with Clay's suggestion verses just doing them one at a time without putting them in the background.

If you have gigabytes of data or don't have 4 CPUs, it may not be helpful doing them all at once?

Of course if you find it takes lots less than 4 * 25 minutes, I guess you don't have to do my experiment.

[Marvin Strong]

as was pointed out, wait is what you should use. replace your sleep with wait and you should get the desired results.

[James R. Ferguson]

Hi Kris:

Along the lines of performance and *testing* the most expedient methodology as Dennis suggests, might include choosing different filesystems for sort's stringing and merging phases by setting the 'TMPDIR' environmental variable differently for any concurrent operations. See the 'sort(1)' manpages for more information.

More importantly, if your sort is making extensive transformations during its key comparisons (e.g. performing substring extractions to form sort keys) you should consider pre-creating those keys in a Schwartzian transform to reduce CPU overhead.

Optimizing the *way* you perform your sorts may offer as much (or more) performance gain as trying to run them in parallel.

Regards!

...JRF...

[A. Clay Stephenson]

I absolutely agree that choosing the correct sorting algorithms and methodologies is wise. You may already be doing that. "Wait" is the answer to your question but you may not be asking the right question.

I can tell you a story that is absolutely true. Many decades ago, I was part of a 3-person company that decided to tackle the DEC, Data General, and IBM mini-computers of the world with clusters of Apple II's connected to shared, very large (20MiB) hard drives. (Each of these 20MiB drives cost $6750.) This was before the IBM PC was introduced. One of the primary tasks that had to be done was large sorts. At that time (and still is) my only formal Computer Science training was 6 hours of FORTRAN and the only sort I had ever done (or knew about) was the bubble sort. My first implementation on an inversely ordered file took about 3 minutes to complete and this was a small data set. Well, I knew that dog wouldn't hunt on the real data so I sat down and starting thinking about sorting. I realized that if I could somehow divide the data into values smaller than some "fulcrum" value and larger than some "fulcrum" value and then divide those smaller chunks the same way and continue until the chunks were 3 or less then I would have a sorted data set and because the exchanges were occurring over large extents rather than just between adjacent elements of the bubble sort, the sort would be many orders of magnitude faster. Literally the first time I ran this Pascal program, it finished in 2 seconds where the bubble sort took 3 minutes! I knew there had to be a bug because it ran so fast but then I realized that for this sort the inversely ordered data set was almost as trivial as the ordered data set because the sort would be completed in a single pass. I later learned (when I actually found out that there were books about algorithms) that the sort I just described was Hoare's quicksort. I could have saved a lot of hard thinking by going to the library first but, in physics, one tends to think in terms of first principles.

This is the long way of saying that you really need to decide if you are asking the right question.

Food for thought,
Clay

[Kris Spander]

Hi guys,

Thanks for all the ideas! We are running this on a 6 CPU RP-7400 and the sorts do complete sooner when executing in parallel. The 25-35 minute times are actually faster than our earlier sorts that took over an hour to finish. We use Syncsort. We are already dividing the work across separate file systems.

"Wait" was really the answer I needed. Duh!
I welcome any other ideas to make this operation faster. At least now thanks to you guys the master script waits properly.

Thanks,
Kris

[James R. Ferguson]

Hi Kris:

"...other ideas...?" How about my original [unscored] reply [above]? :-)

Regards!

...JRF...

[Kris Spander]

Hi JRF,

I'm so sorry. I didn't realize that I had missed assigning you points. To make up for it, I assigned 10 points for each of your replies.

Thanks,
Kris

[James R. Ferguson]

Hi Kris:

/* NO points for this post please! */

Thanks, Kris. Ten would have been enough. It's been one of those days :-))

I look forward to contributing .

/* NO ponits for this post please! */

Regards!

...JRF...

[Dennis Handly]

>Clay: At that time (and still is) my only formal Computer Science training was 6 hours of FORTRAN and the only sort I had ever done (or knew about) was the bubble sort.

Something similar happened to me with Fortran and graduate school. Except it ran over my 2 minute time allotment and aborted. I must have read about shell sort in my CS courses and it finished well under my limit.

[A. Clay Stephenson]

Well Dennis, to make the sort worse this was Pascal that looked like FORTRAN and had plenty of goto's just like good FORTRAN code is supposed to --- but, it did run perfectly the first time. A little later, I actually "discovered" recursion and my sort became much cleaner because I didn't have to use arrays to keep up with all the counters and my Pascal began to look like Pascal. I was quite surprised that when I did find Wirth's book about a year later that I had independently devised a rather famous sorting algorithm simply from thinking from first principles about how sorting should be done. The hardest thing for me to come to grips with was choosing what I called the "fulcrum" (ie, the hingepoint in Hoare speak) -- the value that the elements should be less than or greater than (or equal to). Finally, I punted and decided that I would take the first element, the middle element, and the last element of this partition and sort them and the middle value would become the fulcrum for the entire partition.

All of this was written in UCSD Pascal which was an emulation of an imaginary machine (the p-engine). The only machine specific part of this OS was the p-engine. The Pascal (or FORTRAN) compiled to this p-code and this meant that you could literally take code from an Apple and execute it on a Z-80, a 6809, a 68000, an 8086, ... or other box unchanged -- as long as it had a machine specific p-engine.

The rest of the story is that the p-system was almost the OS of the IBM pc. If the company that owned the rights to the OS (Softech Microsystems) had been able to see the big picture and cut the per unit price to IBM, the world would probably be quite different today.

------------------------------------

Kris, no points please for all of my rambling down memory lane.