topic Re: The Sort Utility and available diskspace in Operating System - OpenVMS

The Sort Utility and available diskspace

Dominic Olivastro — Mon, 22 Aug 2005 11:18:37 GMT

When I sort a very big file, the sort utility uses the SortworkN disks but always uses the smallest disk first. This means I always get the following messages:

%SORT-E-WRITEERR, error writing _HSD10$DIA140:[PATENTS]SORTWORK.TMP;
-SYSTEM-F-EXDISKQUOTA, disk quota exceeded

We're of two minds here. Some of us say that this "E" error for sort and the "F" error from system are handled by the Sort Utility, so that it just goes to the next SortworkN disk. Proof of this is that Sort does not just die, but continues on and reports its statistics correctly.

Others say it is an "E" and "F" error, and therefore the results of the sort are not correct.

Any ideas?

Dom

Re: The Sort Utility and available diskspace

Hein van den Heuvel — Mon, 22 Aug 2005 11:46:57 GMT

>> Some of us say that this "E" error for sort and the "F" error from system are handled by the Sort Utility.

Agreed. This can happen with diskspace and also virtual memory (where sort switches to disk earlier then it intended).

>> ;
-SYSTEM-F-EXDISKQUOTA, disk quota exceeded

Quota exceeded is kind of a silly reason to run into trouble isn't it? Or is this just meant as an example and are you mostly concerned about really running out of actual disk space.

Hein.

Re: The Sort Utility and available diskspace

Antoniov. — Mon, 22 Aug 2005 11:53:11 GMT

Dominic,
HELP /MES WRITEERR WRITEERR
WRITEERR, error writing 'file-spec'

Facility: Shared by several facilities

Explanation: During an OpenVMS RMS file system operation, an error is
encountered while writing a file.

If this error occurred during a macro assembly operation,
the assembler encountered an I/O error when writing to the
output object module or listing file; 'file-spec' is the file
specification of the file being written.

User Action: Determine that the file is open and that you have write
access. If this is a macro assembly error, retry the assembly.
If the error is reproducible, notify your system manager.

I think your sorted file is not correct. However I don't risk lost data :-!

Antonio Vigliotti

Re: The Sort Utility and available diskspace

Dominic Olivastro — Mon, 22 Aug 2005 12:32:53 GMT

Hein -- Just an example. My major concern is this: Can I trust the output of the sort. I think that the error messages are being handled within the sort, and it corrects the error just by moving to the next sortworkn disk. I base this on the fact that SORT continues from this error, and gives me a $Status showing success.

Antoniov -- I'm aware of the explanation, but the question remains, "Is sort handling this error? If not, why does it not just crash?"

Thanks to both of you guys.

Dom

Re: The Sort Utility and available diskspace

Hein van den Heuvel — Mon, 22 Aug 2005 13:35:36 GMT

Sort is handling the eror.

Use /STAT to confirm that there were as many input records as output records.

I'll include a most exquisite answer from an orginal sort engineer from our archives:

Hien.

----------------------------------------
Note 24.10 Strange SORT behaviour
CLT::GILBERT "Multiple inheritence happens" 4-OCT-1988
----------------------------------------

> We will consider down grading the error message to warning.

Thanks. I'm guilty of rampant conservatism for making it E-level in the first place. The code that diagnoses this problem *does* continue gracefully, and *does* produce correct results; so an I-level diagnostic seems most appropriate -- strangely enough!

%SORT-I-WRITEERR, Error writing to your scratch file
-SYSTEM-F-HEADCRASH, Head crash on disk
-SYSTEM-F-SMITHEREENS, Disk exploded and caught on fire
%SORT-S-TIMEX, But VAX Sort/Merge takes a licking and keeps on ticking!

Re: The Sort Utility and available diskspace

Dominic Olivastro — Mon, 22 Aug 2005 13:54:12 GMT

Thanks so much, Hein. This is a huge relief for me. For a long time, people have been arguing that the results of the sort are invalid.

And I always include a "/Stat" qualifier, and I always check the $Status symbol.

Thanks again,
Dom

Re: The Sort Utility and available diskspace

John Gillings — Mon, 22 Aug 2005 17:30:30 GMT

Dom,

As Hein has pointed out, it takes a seriously bad event to break SORT. Just check the stats.

There are techniques for creating sort work areas that permit sortwork files to bypass disk quotas.

Re: The Sort Utility and available diskspace

Dominic Olivastro — Mon, 22 Aug 2005 17:51:07 GMT

" There are techniques for creating sort work areas that permit sortwork files to bypass disk quotas."

That's interesting. Can you give me some hints?

Dom

Re: The Sort Utility and available diskspace

John Gillings — Mon, 22 Aug 2005 22:51:22 GMT

>Can you give me some hints?

Sure, In the Guide to OpenVMS Security manual you'll find a cookbook example of a "project directory". This is a directory owned by a resource identifier, with access control lists and identifier's granted to allow authrized users to create files in that directory. The files will be owned by the identifier, and therefore subject to the identifier's quota, rather than the quota of the creator.

One thing to be a bit wary of is to make sure each user specifies unique file names for their SORTWORK files (especially concurrent users). Otherwise theycan tread on each others toes. If you're running a late enough version use F$UNIQUE()

$ DEFINE SORTWORK1 PROJDIR1:'F$UNIQUE()
$ DEFINE SORTWORK2 PROJDIR2:'F$UNIQUE()

etc...

The only issue here is users can create other files in the same directory, and consume any extended quota. Simple way around that is to regularly delete anything not currently open.

A possible alternative which might be more bullet proof (but I've never tested), you could install SORT32 as a protected subsystem image. Work areas could be setup to only be writeable from the SORT32 image.

See the Guide to OpenVMS System Security for details on protected subsystems - they're an incredibly powerful, but little used feature of OpenVMS. In effect they allow you to bind a set of objects to a specific set of images. For example, some data files that can only be read or written from certain applications.

Re: The Sort Utility and available diskspace

Antoniov. — Tue, 23 Aug 2005 02:15:44 GMT

Dominic,
as other thread I report same link
http://h71000.www7.hp.com/doc/731FINAL/6489/6489pro_023.html
where you can see some hints about sort.

Antonio Vigliotti