Operating System - OpenVMS
1748182 Members
3461 Online
108759 Solutions
New Discussion юеВ

Re: PURGE/KEEP=../BEFORE=..

 
SOLVED
Go to solution
Doug Phillips
Trusted Contributor

Re: PURGE/KEEP=../BEFORE=..

Ooops.

What I meant to say was:

The command should act as if *only* the "/BEFORE=time" files existed.

Sorry.

The best explaination maybe:

In Martin's example, if first you do

$ dir/before=1-dec-2006/date x.x

you will only see the files that

$ purge/before=1-dec-2006/keep=5 x.x

should consider.

Jon Pinkley
Honored Contributor

Re: PURGE/KEEP=../BEFORE=..

The behavior described in response to C020125-17, the v7.2-2 or v5.41H2 implementation of purge/before/keep=(n>1) wasn't the behavior I expected.

I think it should work as original a. logic, i.e. it should always leave the n highest versions, then delete all other versions of the file as long as the before test is true.

In other words, if more than n versions of the file exist after purge it should be because they didn't meet the date criteria. (assuming delete privilege, files not locked, etc.)

To me, that is the most consistent with the description of the command in help.

That is not the way it works for /keep=n when n>1 and /before is specified.

VMS normally works with the following behavior guidelines:

1. Consistency
2. Least surprise

The description of the /before clause is ambiguous, otherwise we wouldn't be having this discussion. However, let's look at the possible uses of purge/keep=../before=..

Originally, I complained about the behavior because I didn't think purge was deleting files that it should have. We have a weekly full backup and daily incremental/norecord backups (these are usually referred to as differential backups in PC land). We have a development executables directory and we wanted to keep the highest two versions, so we would always be able to fall back to the previous version. We wanted to make sure that no versions less than a week old were ever deleted, so they would be on a weekly full backup that had a longer retention period than the incremental backup tapes. I used 8 days to be sure they would be on the weekly tape.

So in our daily cleanup job, I had

$ purge/keep=2/before=-8-0 dev_exe:*.* ! purge to 2 versions unless recent

This didn't delete as many files as I expected, and so I opened a call.

Can someone provide an example of when the behavior exhibited in the v5.41H2 implementation would be more useful than that described above (skip highest n versions, delete all others that are before the specified time)?

In summary I think that any correct implementation must meet the following criteria.

1. Highest n versions of the file are preserved when /keep=n is specified.
2. No file [/created|/modified][/before|/since]=date should be deleted.

The behavior I expected would meet those criteria. The v5.41H2 probably does, if it didn't there would be many more complaints than there are.

In addition to the correctness requirements, the implementation should also be consistent.

I claim that the behavior exhibited by v5.41H2, v7.2-2 and probably all recent versions is not consistent.

For the case n=1 it appears to follow rules I propose. For n>1 it uses a different set of rules (at least v5.41H2 appears consistent with response to C020125-17).

Martin, can you test this? What happens if you use /keep=1? Does it leave one file dated 16-Nov-2006? If it does not, then it is not consistent with /keep=5, where 5 files dated before 1-DEC-2006 00:00:00 were left. What happens if /keep is not specified? It should do the same thing it does when you specify /keep=1.

The lack of consistency is what causes files to be accidentally deleted.

In my opinion, the simple /keep=n highest versions delete all other versions unless they fail some other criteria (like date) is the correct way it should have been implemented.

Whether it should be changed now is a different issue.

Remember purge is primarily version number based (highest n versions), not latest (most recent) n instances of the file. Those are two very different things that happen to usually coincide.

A correct implementation of purge /keep=n must never delete the highest n versions of the file, even if those versions have the oldest time stamps.

If date based (keep the 5 newest) behavior is wanted, it should use a different switch like /by_date.

I wish purge had a switch that would list the files that would be deleted. I know about /confirm, that isn't what I want, since it requires confirmation for each file that would be deleted. An easy to implement solution would be purge /confirm=no where no would be provided for each file confirmation question, but the question would still be asked.
it depends
Doug Phillips
Trusted Contributor

Re: PURGE/KEEP=../BEFORE=..

Jon,

The key word is:

> 1. Consistency

All other DCL commands with /BEFORE and /SINCE use that filter first, not last.

Our expectation should be that if we do:

$ dir/before=time x.x

then any other command's /before=time should select those same files.

If files that should be excluded by any /BEFORE or /SINCE date selection were operated upon, that would be completely inconsistent with other commands and with the documentation and help.

PURGE does seem to be broken in that it doesn't always delete the expected files. Maybe we should discuss whether or not fixing it would hurt anything --- whether it ever get fixed or not is another question.
Jan van den Ende
Honored Contributor

Re: PURGE/KEEP=../BEFORE=..

Jon,

well worded.

Doug,
indeed:
The key word is:
Consistency

I think we all agree that the current behavior is not what might be expected.
The behavior should be either a. or b.; whichever would be the actual implementation should be VERY clearly documented.
The current behavior I would call just plain broken; and, at odds with the documentation.

fwiw,

Proost.

Have one on me.

jpe
Don't rust yours pelled jacker to fine doll missed aches.
Doug Phillips
Trusted Contributor

Re: PURGE/KEEP=../BEFORE=..

Jan,

Sorry to keep this bar-tab running;-), but, your (a) option can't be considered as correct.

The qualifiers don't qualify each other, they qualify the "purge files". Any qualifier which modifies another, say so:

i.e.
/MODIFIED /CREATED /BACKUP

Modifies the time value specified with the
/BEFORE or the /SINCE qualifier.


Jon,

you said:
>>
1. Highest n versions of the file are preserved when /keep=n is specified.
<<
Right. Help /keep says: "Specifies the maximum number of versions of the __specified_ _files__ to be retained in the directory."
(_emphasis mine)

>>
2. No file [/created|/modified]
[/before|/since]=date should be deleted
<<
No, this is opposite from the actual meaning.

Help /before says:
"Selects only those files dated prior to the specified time."

The /before does not modify the /keep, it qualifies the files being selected; i.e. defines the _specified_files_.

Remember what the qualifiers are "qualifying", then it's clear how it should work.

So, applying the explanations from help:

$purge /keep=2 /before=31-dec-2006 [filespec]

Says:
Keep a maximum of 2 versions of any [filespec] files dated prior to 31-dec-2006.

The documentation is clear. PURGE just doesn't always work that way, and *that* is what is causing the confusion, and, Jan, that's why you originally posted, no?
Jan van den Ende
Honored Contributor

Re: PURGE/KEEP=../BEFORE=..

Doug,

>>>
Sorry to keep this bar-tab running;-),
<<<

Well, don't be sorry. you are just demonstrating that this is an issue worth discussing.

>>>
The documentation is clear. PURGE just doesn't always work that way, and *that* is what is causing the confusion, and, Jan, that's why you originally posted, no?
<<<
That is the entire point. If you specify /KEEP=n then (assuming n or more egigeble files present) it should ... KEEP n (specified) versions, and NOT (n - 1 ), as is obviously what is happening.

Proost.

Have one on me.

jpe
Don't rust yours pelled jacker to fine doll missed aches.
Jon Pinkley
Honored Contributor
Solution

Re: PURGE/KEEP=../BEFORE=..



Doug, I agree with you; my wording of condition 2 was not correct... I should have stated only files... (matching date qualifiers) should be deleted. If you read the other parts of my reply you will see that is what I meant.

I was expecting the behavior to work like delete/exclude=(n highest version numbers)/before=specified_date. And that is the way it works when n=1. Perhaps that is why I had that expectation.

This discussion has convinced me that the behavior should be as you describe, since the "selected files" should be the set of files selected by the filespec and the qualifiers. The bug in purge is not limited to /before as is demonstrated by the following:

$ show system/noproc ! not portable ...
OpenVMS V8.3 on node SIGMA 23-FEB-2007 22:23:30.28 Uptime 1 00:56:43
$ analyze/image/select=(id,link) sys$system:delete.exe; ! not portable
SYS$COMMON:[SYSEXE]DELETE.EXE;1
"X-05"
29-JUN-2006 18:18:42.45
$ directory/own/date=(cre,mod) pt.tmp;*

Directory SYS$SYSROOT:[SYSMGR]

PT.TMP;5 1-JAN-2000 01:00:31.11 1-JAN-2000 01:01:31.11 [SYSTEM]
PT.TMP;4 2-JAN-2000 01:00:30.99 2-JAN-2000 01:01:30.99 [1,1]
PT.TMP;3 3-JAN-2000 01:00:30.88 3-JAN-2000 01:01:30.88 [1,1]
PT.TMP;2 4-JAN-2000 01:00:30.78 4-JAN-2000 01:01:30.78 [SYSTEM]
PT.TMP;1 5-JAN-2000 01:00:30.64 5-JAN-2000 01:01:30.64 [SYSTEM]

Total of 5 files.
$ purge/keep=2/by_own=[1,1]/log pt.tmp
%PURGE-I-FILPURG, SYS$SYSROOT:[SYSMGR]PT.TMP;3 deleted (0 blocks)
$ directory/own/date=(cre,mod) pt.tmp;*

Directory SYS$SYSROOT:[SYSMGR]

PT.TMP;5 1-JAN-2000 01:00:31.11 1-JAN-2000 01:01:31.11 [SYSTEM]
PT.TMP;4 2-JAN-2000 01:00:30.99 2-JAN-2000 01:01:30.99 [1,1]
PT.TMP;2 4-JAN-2000 01:00:30.78 4-JAN-2000 01:01:30.78 [SYSTEM]
PT.TMP;1 5-JAN-2000 01:00:30.64 5-JAN-2000 01:01:30.64 [SYSTEM]

Total of 4 files.

I have tested this on 7.2-2, 7.3-2 and 8.3, they all show this behavior. Including reproducer as attachment.

I think I have narrowed the conditions necessary to exhibit bug.

Some definitions:

Selection set: The set of files that meet the filespec and qualifiers.

Statement of problem:

If the highest version of a specific device:[directory]file.type is not a member of the selection set, and /keep versions > 1, then one less file is kept than specified by the /keep qualifier.

Expected cause: from source listings: [V732.DELETE.LIS]PURGE.LIS (which was more handy than 8.3)

In routine purge_ods2_files, at line 931 there is a check for a change in device, directory, name or type. If there is change, then in line 936 the versions (matching purge selection) is unconditionally set to 1, i.e. no check is made to see if this meets the requirements of the common qualifiers. If it isn't the highest version, then the routine purge_this_file is called. purge_this_file calls DEL$MATCH_FILENAME which checks if the file matches the selection criteria, and if so increments the versions seen, and if gtr than the specified /keep versions, deletes the file.

I believe the following would fix the problem:

In PURGE.B32, replace line 936 " versions = 1; ! Reset the version count" with

versions = 0; ! Reset the versions matching purge selection
status = DEL$MATCH_FILENAME (FAB); ! Does highest version match purge selection?
IF .status
THEN versions = .versions + 1; ! If so, increment the versions matching purge selection

NB: This would change the behavior for the case

$ purge /before=(date earlier than date of highest version)

as one file would be kept before that time. But it would be consistent.
it depends
Jan van den Ende
Honored Contributor

Re: PURGE/KEEP=../BEFORE=..

jon,

Thank you!

I think this is just about the full explanation. Definitely a bug I would say.

Now it is up to Engeneering to fix it. In my view even a modification to the Docs to reflect actual behavior is not enough!

Anyone know how to bring this more directly to their attention, now that Advocacy is gone?

Proost.

Have one on me.

jpe
Don't rust yours pelled jacker to fine doll missed aches.
Doug Phillips
Trusted Contributor

Re: PURGE/KEEP=../BEFORE=..

Jon,

Excellent work! (Jan, that sure looks like a 10 ponter to me!)

Thinking about your original need, it would be nice to have modifiers for the /KEEP qualifier. I, too, have needed to purge older files but keep at least n versions regardless of age. Something like /KEEP=([MAX=]4,MIN=2), maybe even allow before & since /KEEP modifiers.

Probably not worth the effort, though, because it isn't that hard to do using lexicals, and the need has been rare.