Error with paralell sorts

[Chartier Jerome]

Hi all,

I made a script that is merging, sorting and then splitting files in 500000 files.
The merge is ok, but with the sort command, I've got this error message:

sort: A write error occurred while sorting

The sort extract is as below:

${SORT} -o ${OUTDIR}/${SDAY}/INTEC_${SDAY}sorted.txt ${OUTDIR}/${SDAY}/INTEC_${SDAY}.txt
\rm -fr ${OUTDIR}/${SDAY}/INTEC_${SDAY}.txt

The files are quite big, and I paralellise it for 30 processes ($SDAY is each days of the month).
It is working fine with one, two,three paralells, but with four, I've got the error message mentionned.

ulimit is :
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 1048576
stack(kbytes) 392192
memory(kbytes) unlimited
coredump(blocks) 4194303

The mount point is 12% full, /tmp is not full.

Has someone already met this thing, or have an idea?

I am thinking on some kernel parameters or sort limitations????
Version
HP-UX Release 11i: November 2000

Thanks in advance for all your help

Regards

Jerome C

[A. Clay Stephenson · Accepted solution]

My best guess is that either a filesystem is filling up (/var/tmp or whatever your TMPDIR is defined to be) or your are attempting to write a 2GB or larger file on a filesystem without largefiles enabled. You need to do continuous df or bdf while the sort is running. Also, add something like this:
STAT=${?}
if [[ ${STAT} -ne 0 ]]
then
echo "Sort failed; status ${STAT}" >&2
fi
immediately after the sort command. The errno may be very telling about which limit you are hitting.

[Chartier Jerome]

Hi all,

Thanks for your reply.
During some bdf during the script execution, I've seen /var going up to 100%, then crashing sort with the error.


Any more idea?

Thanks in advance

Jerome

[A. Clay Stephenson]

Well that is exactly what I expected. What you need to do is enlarge /var/tmp (possibly by creating a separate filesystem for it) or
setting and exporting TMPDIR before lauching sort.

e.g.
export TMPDIR=/xxx/mybigfilesystem
sort ...