PURGE/KEEP=../BEFORE=..

[Jan van den Ende]

Hi all,

Today we got into a discussion of the behavior of the title subject.

Does it:

a. Purge down to /KEEP= versions, but refuse to delete any files after /BEFORE

or

b. Scan for any file /BEFORE, anf trim those down to /KEEP.

We did some testing.

$ DIR/DAT X.X
X.X;7 22-JAN-2007 15:08:10;00
X.X;6 22-JAN-2007 15:07:50.00
X.X;5 22-JAN-2007 15:06:40.00
X.X;4 17-JAN-2007 12:10:00.00
X.X;3 17-JAN-2007 12:08:00.00
X.X;2 17-JAN-2007 11:55:00.00
X.X;1 16-JAN-2007 16:00:00.00
Total of 7 files

(well, not exactly, but this is the gist of it)

$ PURGE /BEFORE=19-JAN/KEEP=6
(a. should leave 7 files, b. six)

$ DIR/TOT X
Total of 7 files.
Seems to point to a.

$ PURGE /BEFORE=19-JAN/KEEP=3
Now we expect 1 file gone, but:
$ DIR/TOT X
Total of 5 files. !!!! ;1 & ;2 gone
Only TWO files /BEFORE left

Repeat: No change

$ PURGE /BEFORE=19-JAN/KEEP=2
version ;3 also gone!
Only ONE file /BEFORE left !!!

$ PURGE/BEFORE="15:07"/KEEP=2
AGAIN, only ONE before left!


Anybody seen this before?
Anybody any explanation?

We tentatively concluded this was a little bug in the algoritm, but we await your vision.

Proost.

Have one on me.

jpe

[Jan van den Ende]

Sorry, I forgot:

This is on 7.3-2

Proost.

Have one on me.

jpe

[Hoff]

Odd. I'd tend to simplify the command -- this would not be the first odd behavior from PURGE. A workaround, of course. (That, or file or directory version limits.)

Lob a bug report at HP -- whomever within OpenVMS engineering owns DCL also owns this decision, and they own the vision around how this was intended to work, and the decision around whether the code or the documentation will change. If nothing else, the existing documentation could be clarified around the ordering and the intent.

I'd half wonder if the file selection order might (also) be dependent on the order of the qualifiers. There have been weirder things implemented.

[Jan van den Ende]

Hoff,

we also wondered about the order of qualifiers.
As far as we investigated that, there is no difference.

But it still seems strange, that the actual numeber of kept versions is one lees than the specified (for b.)
And it definitely seems we can discard a.

Proost.

Have one on me.

jpe

[Doug Phillips]

Odd indeed. Doesn't look like any expected result.

It seems reasonable to me that (b.) is the way it should work; consider only those files meeting the /before=date criteria (exclude all other files from consideration) and operate only on those files. I think /before /since should always be the primary selectors.

(a.) doesn't make sense to me.

Compare this to PURGE/SINCE=date/KEEP=n for expected behavior. You would expect anything older that SINCE=date to be ignored. So with your example, PURGE/SINCE=22-JAN/KEEP=2 we should see only X.X;5 deleted, and that's the way it works.

I recall not too long ago, a copy /since /before bug that caused no files to be selected regardless of the dates, even though a dir with the same dates would show the files.

[Jon Pinkley]

Jan,

In your example, it seems that algorithms a. and b. are reversed, i.e. wouldn't a. leave 6 and b. leave 7?

I submitted a problem report in January 2002 for VMS 7.2-2, because I did not feel that purge/keep=(n>1)/before=date did the correct thing.

Call Reference #: C020125-17

I don't think I am allowed to post their entire response, but here is part of it:

"The /BEFORE selects only those files dated prior to the specified date. This sets the starting point for the Purge command and then follows the /KEEP qualifier. This behavior has been this way since before the V6.1 time frame and the entire logic of the facility would need to be rearranged to implement what is eventually determined to be the correct behavior. The PURGE command itself is used so often that the possibility of introducing a situation where files get deleted that were not intended to be deleted is too big a risk for the limited gain."

We ended up using a command procedure to do what we wanted, which was equivalent to skipping the n highest versions of the file, then deleting all other versions of the file if and only if they matched the /before=date.

In any event, what Purge does when /keep is not specified, should be identical to what it does when /keep=1 is specified. Also, /keep=1 should be consistent with /keep=2, and that does not seem to be the case when /before is used.

[Robert Gezelter]

Jan,

I don't have an opportunity to check this out throughly at this instant (a quick break from dealing with re-caulking a shower faucet to stop a leak), however a quick comment about the behavior.

Whatever the behavior is, it should be consistent. In this case, I am referring to how /BEFORE and /AFTER behave, singly and in concert. I would re-run your test cases, and see if the behavior differs for any range.

If it does differ, then it should really not be considered a documentation problem, but a real issue.

- Bob Gezelter, http://www.rlgsc.com

[Martin Hughes]

Tested this on V5.5-2H4, the results were more like you would expect;

$ dire/date x.x

X.X;34 6-DEC-2006 10:49:23.00
X.X;33 1-DEC-2006 09:57:55.00
X.X;32 1-DEC-2006 09:57:29.00
X.X;31 1-DEC-2006 09:56:18.00
X.X;30 1-DEC-2006 09:55:31.00
X.X;29 16-NOV-2006 16:27:29.00
X.X;28 16-NOV-2006 16:25:59.00
X.X;27 16-NOV-2006 16:19:07.00
X.X;3 15-DEC-2006 13:02:25.38
X.X;2 17-AUG-2005 16:22:06.04
X.X;1 13-APR-2005 12:28:16.78
Total of 11 files.

$ purge/before=1-dec-2006/keep=5 x.x
$ dire/date x.x

X.X;34 6-DEC-2006 10:49:23.00
X.X;33 1-DEC-2006 09:57:55.00
X.X;32 1-DEC-2006 09:57:29.00
X.X;31 1-DEC-2006 09:56:18.00
X.X;30 1-DEC-2006 09:55:31.00
X.X;29 16-NOV-2006 16:27:29.00
X.X;28 16-NOV-2006 16:25:59.00
X.X;27 16-NOV-2006 16:19:07.00
X.X;3 15-DEC-2006 13:02:25.38
X.X;2 17-AUG-2005 16:22:06.04
Total of 10 files.

This logic is as per "a." in the original post.

Seems like a bug with later versions. However the logic is supposed to work, the bottom line is files should never be deleted unintentionally.

[Doug Phillips]

Martin's example and Jon's Call response both describe the (b) logic;

>>
b. Scan for any file /BEFORE, and trim those down to /KEEP.
<<

which is how the /before and /since qualifiers work with all other commands.

i.e.

$ PURGE /KEEP=n /BEFORE=time

The command should act as if the "/BEFORE=time" files did not exist.

Jan's test results don't seem to all comply with that logic, though, and I think that was what he was questioning.

[Jan van den Ende]

Jon,

well spotted. Upon re-reading the original statement does look a bit awkward.
So, it HAS been noted before!

Bob,

agreed, it DOES seem like a real issue!

Martin,
So it DID work as expected once!
And I coudn't agree more with your bottom line!

Doug,
you got the point.
Software should behave like what is reasonable to expect, but ANY ambiguity should be decided upon by the ocumentation, which MUST be as correct as the code should be!

Proost.

Have one on me.

jpe

[Doug Phillips]

Ooops.

What I meant to say was:

The command should act as if *only* the "/BEFORE=time" files existed.

Sorry.

The best explaination maybe:

In Martin's example, if first you do

$ dir/before=1-dec-2006/date x.x

you will only see the files that

$ purge/before=1-dec-2006/keep=5 x.x

should consider.

[Jon Pinkley]

The behavior described in response to C020125-17, the v7.2-2 or v5.41H2 implementation of purge/before/keep=(n>1) wasn't the behavior I expected.

I think it should work as original a. logic, i.e. it should always leave the n highest versions, then delete all other versions of the file as long as the before test is true.

In other words, if more than n versions of the file exist after purge it should be because they didn't meet the date criteria. (assuming delete privilege, files not locked, etc.)

To me, that is the most consistent with the description of the command in help.

That is not the way it works for /keep=n when n>1 and /before is specified.

VMS normally works with the following behavior guidelines:

1. Consistency
2. Least surprise

The description of the /before clause is ambiguous, otherwise we wouldn't be having this discussion. However, let's look at the possible uses of purge/keep=../before=..

Originally, I complained about the behavior because I didn't think purge was deleting files that it should have. We have a weekly full backup and daily incremental/norecord backups (these are usually referred to as differential backups in PC land). We have a development executables directory and we wanted to keep the highest two versions, so we would always be able to fall back to the previous version. We wanted to make sure that no versions less than a week old were ever deleted, so they would be on a weekly full backup that had a longer retention period than the incremental backup tapes. I used 8 days to be sure they would be on the weekly tape.

So in our daily cleanup job, I had

$ purge/keep=2/before=-8-0 dev_exe:*.* ! purge to 2 versions unless recent

This didn't delete as many files as I expected, and so I opened a call.

Can someone provide an example of when the behavior exhibited in the v5.41H2 implementation would be more useful than that described above (skip highest n versions, delete all others that are before the specified time)?

In summary I think that any correct implementation must meet the following criteria.

1. Highest n versions of the file are preserved when /keep=n is specified.
2. No file [/created|/modified][/before|/since]=date should be deleted.

The behavior I expected would meet those criteria. The v5.41H2 probably does, if it didn't there would be many more complaints than there are.

In addition to the correctness requirements, the implementation should also be consistent.

I claim that the behavior exhibited by v5.41H2, v7.2-2 and probably all recent versions is not consistent.

For the case n=1 it appears to follow rules I propose. For n>1 it uses a different set of rules (at least v5.41H2 appears consistent with response to C020125-17).

Martin, can you test this? What happens if you use /keep=1? Does it leave one file dated 16-Nov-2006? If it does not, then it is not consistent with /keep=5, where 5 files dated before 1-DEC-2006 00:00:00 were left. What happens if /keep is not specified? It should do the same thing it does when you specify /keep=1.

The lack of consistency is what causes files to be accidentally deleted.

In my opinion, the simple /keep=n highest versions delete all other versions unless they fail some other criteria (like date) is the correct way it should have been implemented.

Whether it should be changed now is a different issue.

Remember purge is primarily version number based (highest n versions), not latest (most recent) n instances of the file. Those are two very different things that happen to usually coincide.

A correct implementation of purge /keep=n must never delete the highest n versions of the file, even if those versions have the oldest time stamps.

If date based (keep the 5 newest) behavior is wanted, it should use a different switch like /by_date.

I wish purge had a switch that would list the files that would be deleted. I know about /confirm, that isn't what I want, since it requires confirmation for each file that would be deleted. An easy to implement solution would be purge /confirm=no where no would be provided for each file confirmation question, but the question would still be asked.

[Doug Phillips]

Jon,

The key word is:

> 1. Consistency

All other DCL commands with /BEFORE and /SINCE use that filter first, not last.

Our expectation should be that if we do:

$ dir/before=time x.x

then any other command's /before=time should select those same files.

If files that should be excluded by any /BEFORE or /SINCE date selection were operated upon, that would be completely inconsistent with other commands and with the documentation and help.

PURGE does seem to be broken in that it doesn't always delete the expected files. Maybe we should discuss whether or not fixing it would hurt anything --- whether it ever get fixed or not is another question.

[Jan van den Ende]

Jon,

well worded.

Doug,
indeed:
The key word is:
Consistency

I think we all agree that the current behavior is not what might be expected.
The behavior should be either a. or b.; whichever would be the actual implementation should be VERY clearly documented.
The current behavior I would call just plain broken; and, at odds with the documentation.

fwiw,

Proost.

Have one on me.

jpe

[Doug Phillips]

Jan,

Sorry to keep this bar-tab running;-), but, your (a) option can't be considered as correct.

The qualifiers don't qualify each other, they qualify the "purge files". Any qualifier which modifies another, say so:

i.e.
/MODIFIED /CREATED /BACKUP

Modifies the time value specified with the
/BEFORE or the /SINCE qualifier.


Jon,

you said:
>>
1. Highest n versions of the file are preserved when /keep=n is specified.
<<
Right. Help /keep says: "Specifies the maximum number of versions of the __specified_ _files__ to be retained in the directory."
(_emphasis mine)

>>
2. No file [/created|/modified]
[/before|/since]=date should be deleted
<<
No, this is opposite from the actual meaning.

Help /before says:
"Selects only those files dated prior to the specified time."

The /before does not modify the /keep, it qualifies the files being selected; i.e. defines the _specified_files_.

Remember what the qualifiers are "qualifying", then it's clear how it should work.

So, applying the explanations from help:

$purge /keep=2 /before=31-dec-2006 [filespec]

Says:
Keep a maximum of 2 versions of any [filespec] files dated prior to 31-dec-2006.

The documentation is clear. PURGE just doesn't always work that way, and *that* is what is causing the confusion, and, Jan, that's why you originally posted, no?

[Jan van den Ende]

Doug,

>>>
Sorry to keep this bar-tab running;-),
<<<

Well, don't be sorry. you are just demonstrating that this is an issue worth discussing.

>>>
The documentation is clear. PURGE just doesn't always work that way, and *that* is what is causing the confusion, and, Jan, that's why you originally posted, no?
<<<
That is the entire point. If you specify /KEEP=n then (assuming n or more egigeble files present) it should ... KEEP n (specified) versions, and NOT (n - 1 ), as is obviously what is happening.

Proost.

Have one on me.

jpe

[Jon Pinkley · Accepted solution]

Doug, I agree with you; my wording of condition 2 was not correct... I should have stated only files... (matching date qualifiers) should be deleted. If you read the other parts of my reply you will see that is what I meant.

I was expecting the behavior to work like delete/exclude=(n highest version numbers)/before=specified_date. And that is the way it works when n=1. Perhaps that is why I had that expectation.

This discussion has convinced me that the behavior should be as you describe, since the "selected files" should be the set of files selected by the filespec and the qualifiers. The bug in purge is not limited to /before as is demonstrated by the following:

$ show system/noproc ! not portable ...
OpenVMS V8.3 on node SIGMA 23-FEB-2007 22:23:30.28 Uptime 1 00:56:43
$ analyze/image/select=(id,link) sys$system:delete.exe; ! not portable
SYS$COMMON:[SYSEXE]DELETE.EXE;1
"X-05"
29-JUN-2006 18:18:42.45
$ directory/own/date=(cre,mod) pt.tmp;*

Directory SYS$SYSROOT:[SYSMGR]

PT.TMP;5 1-JAN-2000 01:00:31.11 1-JAN-2000 01:01:31.11 [SYSTEM]
PT.TMP;4 2-JAN-2000 01:00:30.99 2-JAN-2000 01:01:30.99 [1,1]
PT.TMP;3 3-JAN-2000 01:00:30.88 3-JAN-2000 01:01:30.88 [1,1]
PT.TMP;2 4-JAN-2000 01:00:30.78 4-JAN-2000 01:01:30.78 [SYSTEM]
PT.TMP;1 5-JAN-2000 01:00:30.64 5-JAN-2000 01:01:30.64 [SYSTEM]

Total of 5 files.
$ purge/keep=2/by_own=[1,1]/log pt.tmp
%PURGE-I-FILPURG, SYS$SYSROOT:[SYSMGR]PT.TMP;3 deleted (0 blocks)
$ directory/own/date=(cre,mod) pt.tmp;*

Directory SYS$SYSROOT:[SYSMGR]

PT.TMP;5 1-JAN-2000 01:00:31.11 1-JAN-2000 01:01:31.11 [SYSTEM]
PT.TMP;4 2-JAN-2000 01:00:30.99 2-JAN-2000 01:01:30.99 [1,1]
PT.TMP;2 4-JAN-2000 01:00:30.78 4-JAN-2000 01:01:30.78 [SYSTEM]
PT.TMP;1 5-JAN-2000 01:00:30.64 5-JAN-2000 01:01:30.64 [SYSTEM]

Total of 4 files.

I have tested this on 7.2-2, 7.3-2 and 8.3, they all show this behavior. Including reproducer as attachment.

I think I have narrowed the conditions necessary to exhibit bug.

Some definitions:

Selection set: The set of files that meet the filespec and qualifiers.

Statement of problem:

If the highest version of a specific device:[directory]file.type is not a member of the selection set, and /keep versions > 1, then one less file is kept than specified by the /keep qualifier.

Expected cause: from source listings: [V732.DELETE.LIS]PURGE.LIS (which was more handy than 8.3)

In routine purge_ods2_files, at line 931 there is a check for a change in device, directory, name or type. If there is change, then in line 936 the versions (matching purge selection) is unconditionally set to 1, i.e. no check is made to see if this meets the requirements of the common qualifiers. If it isn't the highest version, then the routine purge_this_file is called. purge_this_file calls DEL$MATCH_FILENAME which checks if the file matches the selection criteria, and if so increments the versions seen, and if gtr than the specified /keep versions, deletes the file.

I believe the following would fix the problem:

In PURGE.B32, replace line 936 " versions = 1; ! Reset the version count" with

versions = 0; ! Reset the versions matching purge selection
status = DEL$MATCH_FILENAME (FAB); ! Does highest version match purge selection?
IF .status
THEN versions = .versions + 1; ! If so, increment the versions matching purge selection

NB: This would change the behavior for the case

$ purge /before=(date earlier than date of highest version)

as one file would be kept before that time. But it would be consistent.

[Jan van den Ende]

jon,

Thank you!

I think this is just about the full explanation. Definitely a bug I would say.

Now it is up to Engeneering to fix it. In my view even a modification to the Docs to reflect actual behavior is not enough!

Anyone know how to bring this more directly to their attention, now that Advocacy is gone?

Proost.

Have one on me.

jpe

[Doug Phillips]

Jon,

Excellent work! (Jan, that sure looks like a 10 ponter to me!)

Thinking about your original need, it would be nice to have modifiers for the /KEEP qualifier. I, too, have needed to purge older files but keep at least n versions regardless of age. Something like /KEEP=([MAX=]4,MIN=2), maybe even allow before & since /KEEP modifiers.

Probably not worth the effort, though, because it isn't that hard to do using lexicals, and the need has been rare.