Operating System - OpenVMS
1748010 Members
4393 Online
108757 Solutions
New Discussion юеВ

SORTMERGE.EXE / SORTSHR.EXE behaviour [OpenVMS/VAX v6.2 (VAX 4000-106A under CharonVAX)]

 
Mark_Corcoran
Frequent Advisor

SORTMERGE.EXE / SORTSHR.EXE behaviour [OpenVMS/VAX v6.2 (VAX 4000-106A under CharonVAX)]

I'd made a stab at some code to routinely renumber down NETSERVER.LOG on a couple of accounts that are heavy FAL users, and found today that it hadn't been working as per original testing.

The code was doing a DIRECTORY NETSERVER.LOG /NOHEAD /NOTRAIL /NOSIZ /NODATE /OUT=file1 then doing a SORT file1 file2 /KEY=(POS:1,SIZ:80,ASCENDING)

Starting with an internal index/intended version number of 1, It would then post-process file2, checking the version number of the files (now theoretically in ascending version number order), and if the file version number wasn't the same as the index, it would renumber it to that version, bump the index number, then loop around reading other records in the file, performing the same work until it got an EOF.

The problem is that SORT isn't doing what I expected, e.g. if I manually create A.A with the following:

USER$DISK:[TEST]NETSERVER.LOG;263
USER$DISK:[TEST]NETSERVER.LOG;262
USER$DISK:[TEST]NETSERVER.LOG;261
USER$DISK:[TEST]NETSERVER.LOG;260
USER$DISK:[TEST]NETSERVER.LOG;259
USER$DISK:[TEST]NETSERVER.LOG;179
USER$DISK:[TEST]NETSERVER.LOG;3
USER$DISK:[TEST]NETSERVER.LOG;2
USER$DISK:[TEST]NETSERVER.LOG;1

Then I do SORT A.A TT: /KEY=(POS:1,SIZ:80,ASC) I get the following:
USER$DISK:[TEST]NETSERVER.LOG;1
USER$DISK:[TEST]NETSERVER.LOG;179
USER$DISK:[TEST]NETSERVER.LOG;2
USER$DISK:[TEST]NETSERVER.LOG;259
USER$DISK:[TEST]NETSERVER.LOG;260
USER$DISK:[TEST]NETSERVER.LOG;261
USER$DISK:[TEST]NETSERVER.LOG;262
USER$DISK:[TEST]NETSERVER.LOG;263
USER$DISK:[TEST]NETSERVER.LOG;3

[Since file ;1 has a version number of 1 it doesn't then need to be renamed, but the next file has a version number of ;173 and renaming it to the bumped index number of ;2 fails because version ;2 already exists]

All of this set me thinking about the old days where an errant process that created 1000s of files in one directory then caused performance issues for subequent creation/deletion of files in the same directory because of having to shuffle entries up/down in the .DIR file and rewrite blocks back to disk.

If you didn't have / couldn't / weren't allowed to install DFU, you had to create your own reverse-deletion script to do pretty similar to what I've demonstrated above (and I've found one I wrote years ago).

However, the experience of SORT above makes me wonder how reverse deletion (using DIR, SORT & DELETE in a command procedure) ever worked for files which had combinations of version numbers of 1, 2, 3, 4, and 5 digits in length.

Am I misremembering it working, or is there something else going on here?

[I've pretty much only used plain-vanilla qualifiers on SORT, and never used specification files or collation sequences, but from my brief reading this afternoon, I don't see that they would help with the what I'm seeing]

 

Mark

[Formerly appearing as woeisme]
9 REPLIES 9
Steven Schweda
Honored Contributor

Re: SORTMERGE.EXE / SORTSHR.EXE behaviour [OpenVMS/VAX v6.2 (VAX 4000-106A under CharonVAX)]

> Am I misremembering it working, [...]

   That'd be my guess.  If you're sorting the whole file spec
alphabetically, then "173" comes before "2".  If you convert the version
numbers to a leading-zero format (";00002", ";00173", ...), then you can
get the desired result from an alphabetical sort.

   If you can tell SORT to treat the version-number field as a number,
instead of alphabetically, then it's ok with me.  For a single file
spec (less version), where the ";" is in a known position, that might be
possible/practical.

Craigers01
Advisor

Re: SORTMERGE.EXE / SORTSHR.EXE behaviour [OpenVMS/VAX v6.2 (VAX 4000-106A under CharonVAX)]

I dug up an old DCL file that does a sort/delete. It DOES seem to have the
issue you bring up.

Maybe it is not TOO bad since it works downward. One of ten times or so it
would make a slightly inefficient delete, followed by nine or so ideal
deletes? It seems like the code likey adds value, but it is flawed. An
ideal approach may be to append trailing ZZZZs or something, sort, then
ignore them during delete.

Anyway, I thought I would share that, at least in my case, this was a
problem.
Steven Schweda
Honored Contributor

Re: SORTMERGE.EXE / SORTSHR.EXE behaviour [OpenVMS/VAX v6.2 (VAX 4000-106A under CharonVAX)]

> [...] An ideal approach may be to append trailing ZZZZs or something,
> sort, then ignore them during delete.

   Appending characters doesn't change the problem.  "173ZZZZZ" comes
before "2ZZZZZ", because "1" comes before "2".

ITS $ write sys$output "173ZZZZZ" .lts. "2ZZZZZ"
1

   But, even if it worked, I wouldn't call any such scheme "ideal", if a
two-key sort is available.

   On the bright side, leading zeros work:

ITS $ write sys$output "00173" .lts. "00002"
0

   And are relatively harmless:

its $ dire /nohead /notrail t.c;00002
ITS$DKA0:[SMS]t.c;2

   Unless you get excessive:

its $ dire /nohead /notrail t.c;000002
%DIRECT-E-OPENIN, error opening t.c;000002 as input
-RMS-F-VER, error in version number

abrsvc
Respected Contributor

Re: SORTMERGE.EXE / SORTSHR.EXE behaviour [OpenVMS/VAX v6.2 (VAX 4000-106A under CharonVAX)]

What you have observed is the expected behavior.  The way it was explained to me many years ago (No, I won't say how many...), is to view the field like a fractional decimal value.  To use your version number set (1, 179, 2, 259), think of these as having a decimal point on the left.  This makes them respectively:  .1, .179, .2, .259)   If you place these in order, they would appear as listed.  Another way is to add Zeros  to the end to end up with the same number of digits and look at them like integers:  100, 179,  200, 259.  Again, the order remains the same although the "value" here changes.  A 1 becomes 100 for example.  For alpha sorting, the characters are viewed and compared by position.  This means that 1 and 179 from a character comparison are the same for position 1 and the second is a "larger" value because it has more characters and thus is placed later.

 

Make sense?

 

Dan

Steven Schweda
Honored Contributor

Re: SORTMERGE.EXE / SORTSHR.EXE behaviour [OpenVMS/VAX v6.2 (VAX 4000-106A under CharonVAX)]

> [...] The way it was explained to me [...]

   Seems pretty long-winded, compared with, say, "CHARACTER is the
default data type."  It's an alphabetical sort, not a numerical sort.
And, as the existence of the two different types should suggest, they're
not equivalent.

abrsvc
Respected Contributor

Re: SORTMERGE.EXE / SORTSHR.EXE behaviour [OpenVMS/VAX v6.2 (VAX 4000-106A under CharonVAX)]

Steven, would it be possible to provide something constructive for a change?  While Iknow what it means for an alphabetic sort vs. a numeric one, perhaps there are others that do not.  An explanation of what that means or how to view it should be helpful to those that do not.  I try to view these postings as an opportunity to teach those that may not have the knowledge.  I haveto believe that at one time you too did not know the difference.  Rather than critiize an attempt to behelpful to those with less knowledge, perhaps you can spend an equivalent amount of energy being helpful.

Mark_Corcoran
Frequent Advisor

Re: SORTMERGE.EXE / SORTSHR.EXE behaviour [OpenVMS/VAX v6.2 (VAX 4000-106A under CharonVAX)]

Thanks folks for the replies...

>> Am I misremembering it working, [...]
>   That'd be my guess.

TBH, it does mostly work (in that a (mostly) reverse delete does occur - though largely depending on the extant version numbers of files) - it's just not the perfect panacea because it isn't entirely 100% in reverse order, as Craig notes:

>Maybe it is not TOO bad since it works downward. One of ten times or so it
>would make a slightly inefficient delete, followed by nine or so ideal
>deletes? It seems like the code likey adds value, but it is flawed.

I've not looked at the maths (or math for left-ponders) to determine how often it gets it right, and whilst it will depend on the extant versions, I think it would likely get it right more often than not.

 

>If you convert the version numbers to a leading-zero format (";00002", ";00173", ...), then you can get the desired result from an alphabetical sort.
Which is what I was really looking to avoid doing in DCL, as this is part of the LGICMD and is running in network mode & batch for one account, and in interactive for another.

I had been hoping that I could find some way of coercing or cajoling SORT into treating the "field" as a number, but spaces are treated the same regardless of whether DECIMAL is specified.


>If you can tell SORT to treat the version-number field as a number,
>instead of alphabetically, then it's ok with me.  For a single file
>spec (less version), where the ";" is in a known position, that might be
>possible/practical.
In this case, the .COM is only being used for NETSERVER.LOG, so the device/directory/file/type are always the same length and therefore the position will always be known, but there doesn't appear to be any obvious way of getting SORT on OpenVMS/VAX to treat the field (spaces included) as a number.


>What you have observed is the expected behavior. The way it was explained to me many years ago (No, I won't say how many...),
 
Interestingly Dan, an old reverse-delete .COM of mine is no more recent than ~MAR-2008, though I would likely occasionally have been knocking-up one-offs to do the same thing at my first employer that I left 9 years previously.


>For alpha sorting, the characters are viewed and compared by position.
The first few times I read that, I thought you meant that SORT on OpenVMS/AXP behaved differently

I assume that HYPERSORT on AXP is no different in its behaviour (we have some AXP boxen here, but they are supported by another team).

 

I ended up post-processing the temporary file created by DIR /OUTPUT to create another temporary file with the records from the first file prefixed by a 5-digit (max ver 32767 after all) zero-led line number, then SORTed on that to create a third temporary file.

It's a bit of a faff, but there's less bureaucracy involved than creating an excutable that I can call from DCL (and besides, I have application executable bug fixes and features that have a higher priority than manually having to renumber .LOG files down once every few weeks).


When attempting non-standard use of commands/code where help/TFM is less than generous with examples, I'll invariably search [000000...]*.COM for instances of command usage, or SYS$EXAMPLES for instances of code.

In doing so yesterday, I found one of the original application developers bemoaning the same problem I am having.

He used one of the application's compiled code modules (not an executable image in itself) to add numbering to lines, sort it, then more compiled code to remove the numbering (the problem is that the code requires shareable images to be installed and library code modules to have been loaded by the application;  that's fine if the application happens to be running;  in the case of LGICMD execution, I can't be sure that it is, so I had to resort to DCL).

 

One other unrelated "new" thing that I found with VMS yesterday was when I made a typo in DCL using DIRECTORY - it still worked though quite how/why it was defaulting, I don't know...

3 disks mounted - DKA0, DKA100, DKA200

I'd done DIR DKA:[000000...]*.COM and it listed .COM files from DKA0 (but reported the device as DKA) - my current default at the time was on DKA100, so I assume it happened to be listing files on the first mounted DK device on controller A.

TBH, I'd have expected it to whine about omission of the unit number - it's not behaviour I've seen before because I've never made that typo before, and it's not the kind of (to me) unusual default behaviour I might inentionally seek out or rely upon (e.g. one story I heard of where code (possibly on a Ferranti Argus) was written to work with a particular HDD type/model, such that by reading certain things in a certain order, the heads would be in a convenient position on the next read to avoid waiting for another revolution or something like that, and was thus much quicker ("poor man's cache") - though needless to say - not documented in the code by the long-departed programmer (comments?  on ┬г┬г┬г/$$$ disks? are you crazy?!?!)

 

Mark

[Formerly appearing as woeisme]
Hein_vdHeuvel_d
Advisor

Re: SORTMERGE.EXE / SORTSHR.EXE behaviour [OpenVMS/VAX v6.2 (VAX 4000-106A under CharonVAX)]


Simple solution without sorting using LAN$ACP as example.
Note - The semicolon at the end of the commands is critical

$ purge/keep=12 sys$manager:lan$acp.log
$
$ ! step 1 - rename reversing version to name or directory starting at version 1
$ rena/log sys$manager:lan$acp.log;* sys$manager:lan$acp.tmp;
%RENAME-I-RENAMED, LAN$ACP.LOG;37 renamed to LAN$ACP.TMP;1
%RENAME-I-RENAMED, LAN$ACP.LOG;36 renamed to LAN$ACP.TMP;2
:
%RENAME-I-RENAMED, LAN$ACP.LOG;27 renamed to LAN$ACP.TMP;11
%RENAME-I-RENAMED, LAN$ACP.LOG;26 renamed to LAN$ACP.TMP;12

$ ! step 2 - rename back to original reversing version at version 1
$
$ rena/log sys$manager:lan$acp.tmp;* sys$manager:lan$acp.log;
%RENAME-I-RENAMED, LAN$ACP.TMP;12 renamed to LAN$ACP.LOG;1
%RENAME-I-RENAMED, LAN$ACP.TMP;11 renamed to LAN$ACP.LOG;2
:
%RENAME-I-RENAMED, LAN$ACP.TMP;2 renamed to LAN$ACP.LOG;11
%RENAME-I-RENAMED, LAN$ACP.TMP;1 renamed to LAN$ACP.LOG;12


>>> Which is what I was really looking to avoid doing in DCL, as this is part of the LGICMD

This is your main problem.
The 'code' should run only once every 100 files or so.
So start with a F$SEARCH/F$PARSE test, for example whether the highest netserver.log version is > 100.

Next simple solution if you insist on a sorting algoritm (I would not know why though)
Don't start at version 1 but start at version 10000 or 1000 !
Now everything sorts as text as long as you stay under 10,000 (resp 1000) versions


Next simple solution is to use a better hammer.

$ perl -le "for (sort {$va=$a; $vb=$b; $va=~s/.*;//; $vb=~s/.*;//; $va <=> $vb }
glob q(sys$manager:LAN$ACP.LOG;*)) { $o=$_; $v+; s/;.*/;$v/; rename $o, $_}

- loop over output array from sorted glob function
- sort using unnamed subroutine
- in subroutine copy implicit sort variables $a and $b with file names from glob to 'version' a and b
- strip all up to the version numbers from the version strings
- numeric sort ( <=> ) using the version strings ( Perl as a 'CMP' operator for string compares)
- out of the sort rourint, in the main for block now:
- copy implicit sort output $_ to 'old' name.
- increment counter with default start at 0 becoming 1.
- change version part of $_ string to get the counter
- rename old to new

Note - it is intriguing how this topic did not at all match the actual problem you tried to solve surely because of a
frustration/lack of understanding with an initial approach. It's good to try to understand why it did not work. It is possibly
better to step back and evaluate what you are really trying to accomplish.


Good luck!
Hein

*formerly appearing as Hein van den Heuvel  - I think. I activated and alternate account I guess*

Mark_Corcoran
Frequent Advisor

Re: SORTMERGE.EXE / SORTSHR.EXE behaviour [OpenVMS/VAX v6.2 (VAX 4000-106A under CharonVAX)]

>Note - it is intriguing how this topic did not at all match the actual problem you tried to solve surely because of a frustration/lack of understanding with an initial approach.
The query wasn't so much "how do I achieve this?", more (in hindsight) "does SORT behave differently in more recent versions of OpenVMS (and/or on different architectures)".

I think that in the past when I had created one-off .COMs to do this reverse deletion (before creating one that was generic rather than tied to a specific filename), it worked simply because there had been no gaps in the file versions (and certainly not having files across a mixture of ;n, ;nn, ;nnn, ;nnnn and ;nnnnn).

Most likely it was to delete file versions created by something that had gotten stuck in a loop, and which were not purged/deleted by any housekeeping command procedure.


>It is possibly better to step back and evaluate what you are really trying to accomplish.
An easy/ier life - including not having to manually renumber versions down every couple of months.

Most files which have new versions created, are created by command procedures, and the procedures themselves tidy up their own mess.

The "uncontrollable" issue is NETSERVER.LOG because the frequency with which FAL connections are made tends to mean that some of the netserver processes persist until the application is shut down.

[There are scheduled maintenance periods when this is done, but not frequently* enough to either prevent ;32767 being reached, or (more commonly) for the version number to exceed a reporting threshold.

Increasing the version number at which reporting occurs simply delays the inevitable.

I know that it is possible to alter NETSERVER$TIMEOUT, but then I'd really need to spend time (that I don't have) looking at the existing NETSERVER.LOG files to try and determine a compromise that wouldn't cause an impact from the overhead of process creation]


>$ ! step 1 - rename reversing version to name or directory starting at version 1
>$ rena/log sys$manager:lan$acp.log;* sys$manager:lan$acp.tmp;
...
>$ ! step 2 - rename back to original reversing version at version 1
>$ rena/log sys$manager:lan$acp.tmp;* sys$manager:lan$acp.log;
That's a far simpler solution, though I'd need to do some testing of the behaviour if it happens to run as a new FAL process is created (with corresponding NETSERVER.LOG creation), and whether NETSERVER.LOG (in that particular user directory) is tidied by housekeeping jobs (pulling the rug out from underneath RENAME).

It's not a use I had seen before (as regards the ;* for P1 and ; for P2), and I see that it is only in the penultimate paragraph of HELP for RENAME that it alludes to this behaviour (an Example entry in HELP wouldn't have gone amiss).

But then, it's not the first time (and it won't be the last) that an old dog learns new tricks - I only recently encountered the /OUTPUT positional qualifier for @ ...

I had never considered @ as a "command", so it never occurred to me that it might have qualifiers; it was only when I was investigating why attempts to invoke application commands from the DCL prompt might crash with a traceback depending on what SYS$OUTPUT translated to, that I discovered it.

[DEFINE/ASSIGN of SYS$OUTPUT to redirect to a file with an explicit/implicit /SUPERVISOR_MODE results in /TRANSLATION_ATTRIBUTES=(CONCEALED,TERMINAL) generating an equivalence name of _node$device which posed a problem for lower-level Macro-32 routines that try to extract a file name from the equivalence name - it required SYS$OUTPUT to be defined /USER_MODE

Of course, this behaviour is specific to SYS$OUTPUT - if you DEFINE WIBBLE DEVICE:[DIR]FILE.TYP then assuming you have plain-vanilla DEFINE, it will still be an implicit /SUPERVISOR_MODE, but the equivalence name will now be DEVICE:[DIR]FILE.TYP" rather than "_node$device:"]

I think that it would be safe to say that even the oldest of OpenVMS hands (including ex- DEC/CPQ/HPers, and the folks at VSI) don't know - from memory - everything about all aspects of OpenVMS.

Mark

[Formerly appearing as woeisme]