topic Re: Mark a DIRECTORY as NOBACKUP? in Operating System - OpenVMS

Mark a DIRECTORY as NOBACKUP?

Jan van den Ende — Thu, 26 Jan 2006 14:03:43 GMT

Situation:

VMS 7.3-1
Occasionally BACKUP does not fit in one tape, and next morning an operator has to supply a continuation tape and resume BACKUP.
This is a 24 hour shop, most of activity during office hours, but non-negligeable activity at night also.

The consequences: Far less consistent backups as would have otherwise been possible, and severe performance degradation in the early office hours.

Investigation showed the following:
Also during the night a batch job collects all kind of mutations and formats them in "reports" in a format acceptable to another (non-VMS) system.
Those reports are collected into one dedicated directory, and when reporting is done, they get tranferred to the other system. If that is successfull, they are deleted.
Unvariably the tape-full situation happened when this temporary directory had much data in the backup.
And the value of having it in the backup is nill....

BACKUP without /IMAGE but with /EXCLUDE would:
- make the backup procedure much more complicated (now it just processes all whole disks one at a time)
- take much more time

The simple solution looks to be set the files /NOBACKUP.

I do not know the exact programs that make the extractions, they are from a 3rd party.

We tried setting the .DIR file to /NOBACKUP, which is happily accepted, but BACKUP still backups the files (which would be a minor nuisance) AND the data, which is what we try to avoid.

Any ideas anyone?

Proost.

Have one on me.

jpe

Re: Mark a DIRECTORY as NOBACKUP?

Uwe Zessin — Thu, 26 Jan 2006 14:29:03 GMT

Can you do a dumb:
$ set file/nobackup ...

just before the BACKUP starts? It won't hit open files I guess, but should turn down the amount at least a little bit.

Re: Mark a DIRECTORY as NOBACKUP?

Jim_McKinney — Thu, 26 Jan 2006 14:30:37 GMT

> setting the .DIR file to /NOBACKUP

If the goal is to exclude the data in the files that are "pointed to" by the directory then you'd need to set the NOBACKUP attribute on each of those file individually.

Re: Mark a DIRECTORY as NOBACKUP?

Jan van den Ende — Thu, 26 Jan 2006 14:41:09 GMT

Uwe,
Trouble is, there ARE no files yet when BACKUP starts.
And they may already be gone when the BACKUP of THAT drive starts (which is the situation when things go well).

We HAVE already questioned the correctness of the whole setup, since the "moment" of backup of related files is streched for hours, so integrety of a restored system is questionable anyhow.
But the applications are being transfered "ASAP", so any investment in the system will be overruled.
I am rather pessimistic about the duration of ASAP, but it is "politically incorrect" to say so. :-(

Thanks anyway Uwe, but no solution.

Proost.

Have one on me.

jpe

Re: Mark a DIRECTORY as NOBACKUP?

Walter Miller_1 — Thu, 26 Jan 2006 14:59:09 GMT

I thought the first thing backup did was read indexf.sys and any files created after that scan would not be saved during an image backup. This would make Uwe's suggestion seem appropriate.

Re: Mark a DIRECTORY as NOBACKUP?

Jim_McKinney — Thu, 26 Jan 2006 14:59:45 GMT

> during the night a batch job collects all kind of mutations and formats them in "reports"

Perhaps you could modify the batch job code to set the nobackup attribute on each file as it moves them into your target directory?

Re: Mark a DIRECTORY as NOBACKUP?

Jan van den Ende — Thu, 26 Jan 2006 15:24:27 GMT

Jim,

If the goal is to exclude the data in the files that are "pointed to" by the directory then you'd need to set the NOBACKUP attribute on each of those file individually.

Exactly that was the intention: hoping it would be ingeretid. Not so. :-(
Ane ACE to trigger that would also be welcome, but I know not of it.

Second idea:
something along those lines is the ugly, hard way to get (near) there. It might well be where this will lead to, if this collective has no more elegant solution.

Walter:
but those scans are performed per disk, therefor, rather spaced in time.

Proost.

Have one on me.

jpe

Re: Mark a DIRECTORY as NOBACKUP?

David B Sneddon — Thu, 26 Jan 2006 19:08:17 GMT

Jan,

Is it possible to set up a "scratch" disk for
these files and not backup that disk? You could
also use it for other junk.
Just a thought...

Dave

Re: Mark a DIRECTORY as NOBACKUP?

John Gillings — Thu, 26 Jan 2006 19:56:12 GMT

Jan,

Directory files are ALWAYS backed up, regardless of backup attributes. This is required to ensure incremental restores are done correctly.

At the moment there is no way to specify that files created in a particular directory should have the NOBACKUP attribute by default. It has been requested in the past. If you log a case and lodge a formal request for an enhancement, you will increase the weight of the request. Since NOBACKUP is ignored on directory files, the attribute could be used to specify a default for files within the directory.

In the mean time, you could place an ACE on the directory like:

(IDENTIFIER=BACKUP_PROCESS,OPTIONS=DEFAULT,ACCESS=NONE)

and grant the identifier BACKUP_PROCESS to the process that does your backup. Make sure you remove BYPASS privilege. This should prevent the backup process from reading the files. You'll get error messages which will look ugly, but it should achieve what you want. To reduce the number of errors you could block access to the directory from the backup process itself perhaps?

Other than that, you'll need a /EXCLUDE list.

Re: Mark a DIRECTORY as NOBACKUP?

Steven Schweda — Thu, 26 Jan 2006 23:03:26 GMT

> Since NOBACKUP is ignored on directory
> files, the attribute could be used to
> specify a default for files within the
> directory.

I've wished for this many times, but I'm not
a profit center.

Re: Mark a DIRECTORY as NOBACKUP?

Volker Halle — Fri, 27 Jan 2006 01:14:51 GMT

Jan,

would it be possible to create and mount a LD container file (set /NOBACKUP) and have the temporary files be created in a directory on that 'disk' ?

Volker.

Re: Mark a DIRECTORY as NOBACKUP?

Jan van den Ende — Fri, 27 Jan 2006 07:09:46 GMT

David,
that would require investing in an extra disk. Because the current dusk connectivity is exhausted, that would require adding an extra controller (which I do not even know if a slot is available). See above, no way!

John,
request coming up.
Setting NO access to the DIR file will not help much. It is not only BYPASS, which the account does not have, but also READALL, which it DOES have, and VERY probably needs.
I don't think I even DARE try without.

Volker,
THAT might be an idea! I will pass it on, and if they do it and it works, I definitely will report!

Proost.

Have one on me.

jpe

Re: Mark a DIRECTORY as NOBACKUP?

Chris Davis_9 — Fri, 27 Jan 2006 13:00:00 GMT

Hi Jan,

You say that a batch job produces the report files in this directory.

Would it not be possible to either:

1. Modify the batch job to set the /NOBACKUP flag on the files after creation

2. Have another job run (batch or detached) which sets the /NOBACKUP flag on any files in

the directory. This could be set to start at a specific time, run until a specific time, checking every 30 seconds, 1 minute or whatever.

In our setup we have some detached jobs running every 30 seconds or so to check if there is any work for them to do.

Some check specific database files for header records showing that a batch of work is

complete, allowing us to separate printing tasks from data entry programs.

Some check specific pathworks directories for files of a certain specification. This

allows us to automatically process files submitted by PC users.

a simple

$LOOP:
$ set file /nobackup [dir.subdir]*.*;*
$ wait 00:01:00
$ goto loop

...either running constantly as a detached job or submitted to start at a specific time
and with a time check in it should do the job.

Regards

Chris

Re: Mark a DIRECTORY as NOBACKUP?

Robert_Boyd — Mon, 30 Jan 2006 10:27:56 GMT

If you really wanted to get carried away with a process that set the files to /NOBACKUP status as each one was put in place and can't modify the batch script that creates them:

1. Create an identifier like NO_BACKUP_DOORBELL

2. set appropriate access control entries on the directory to trigger an alarm for successful writes

3. Create a detached process to listen for security alarms for the NO_BACKUP_DOORBELL identifier. When an alarm comes in, set the proper file(s) to /NOBACKUP.

It would obviously be much simpler to be able to alter the create so that the script does this directly rather than having to write something fancy to clean up after the script.

Robert

Re: Mark a DIRECTORY as NOBACKUP?

Jan van den Ende — Mon, 30 Jan 2006 13:03:22 GMT

All,

I entered an enhancement request in Advocacy.

As soon as it gets a number I will report here again, asking for your "I agree"'s.

In the meantime, the "watching procedure" road was chosen.
Ugly, but workable.

Proost.

Have one on me.

jpe

Re: Mark a DIRECTORY as NOBACKUP?

Klaes-Göran Carlsson — Tue, 31 Jan 2006 04:56:47 GMT

Hi all

Could a backup/before be a solution?

Re: Mark a DIRECTORY as NOBACKUP?

Jan van den Ende — Mon, 13 Feb 2006 13:52:53 GMT

Hi all,

if you think you might have some future use for this:

http://www.hpuseradvocacy.org/node/1987

Add your "I agree" and report here, please.

Proost.

Have one on me.

jpe

Re: Mark a DIRECTORY as NOBACKUP?

Jan van den Ende — Tue, 14 Feb 2006 04:18:44 GMT

ping

Proost.

Have one on me.

jpe

Re: Mark a DIRECTORY as NOBACKUP?

Duncan Morris — Tue, 14 Feb 2006 04:24:11 GMT

I have added my support on the advocacy site.

Re: Mark a DIRECTORY as NOBACKUP?

Ian Miller. — Tue, 14 Feb 2006 04:32:34 GMT

been there, done that :-)