Backup volume to subdirectory

[Norm Hansen]

Can the backup command copy an entire volume, preserving directory structure, to another volume in a subdirectory?

The context is disk to disk backups on an outside (NFS mounted) device. Each backup should be kept for n days, so shoving it into a date-named subdirectory seems reasonable, but the backup command won't cooperate for me.

$ backup/log/ignore=inter dsa1:[000000...] dnfs3:[0514...]
does not preserve directory tree. It creates directories, but not subdirectories, so dsa1:[live.data]file.dat;1 ends up turning into
dnfs3:[0514.live]
dnfs3:[0514.data]file.dat;1

$ backup/log/ignore=inter dsa1:[000000...] dnfs3:[*...]
preserves the directory tree, but it isn't in a subdirectory, so the next backup will have a bunch of name conflicts.

Don't want to do an /image, since it won't write to a directory.

I'm sure I could throw together some DCL to get the top level directories and do
$ backup/ignore=inter dsa1:[dir1...] dnfs3:[0514.dir1...] /log
$ backup/ignore=inter dsa1:[dir2...] dnfs3:[0514.dir2...] /log
etc.
but it seems that there must be a way to do it that I'm just missing.

Anyone got an idea?

[Robert Gezelter]

Norm,

I would suggest writing the Save Set to the NFS volume. The Save Set encapsulates the entire volume and directory structure.

An NFS device cannot accommodate ACLs and other structures, so the best way is to write a Save Set and possibly ZIP'ing it to save space.

- Bob Gezelter, http://www.rlgsc.com

[Steven Schweda]

> [...] disk to disk backups [...]

Do you want individual files on the remote
disk, or BACKUP save sets? (Save sets might
be easier.)

[Norm Hansen]

In relation to save sets...

I was planning to avoid them, for a couple of reasons; perhaps you can correct my thinking:

1) The device is involved in a de-duplication scheme to reduce eventual size of the backup. The backup vendor recommends against compression, since that introduces randomization that the de-duplication cannot see through. Not sure whether a save set would yield similar problems. Haven't gotten far enough along to test the proposition one way or the other.

2) The backup scheme is two-tiered. The files are being backed up to a staging location that the (VMS-unaware) backup system can see and then make its own backup. If they are all separate files, the backup administrator would at least be able to see the particular files I need restored instead of having to pull the whole save set back to the staging location (meaning, slightly quicker).


I am aware, and concerned, about losing file attributes in the process. Would not the same concern apply to the save set itself, though?

[RBrown_1]

As I recall, you need to create a concealed logical name for your backup destination.

$ define dnfs3:[0514] dest/tran=conc
$ backup/log dsa1:[*...] dest:[*...]

[Robert Gezelter]

Norm,

First, the contents of the BACKUP Save Set (uncompressed/unencrypted) would probably not affect the de-duplication scheme). There is a CRC (optional) on every block, but I would not consider that an issue.

The loss of attributes, ACLs, and other elements is a significant shortcoming. Getting these correct for arbitrary files on a restore can be a very tedious and error prone activity.

Fixing the attributes of the Save Set is far easier, since the procedures that created them are well known (when compared to arbitrary assemblages of files).

Whether de-duplication has a higher yield than compression is a good question. YMMV (your mileage may vary) applies here. I have seen data streams that would benefit from one and not the other, and vice versa. On a technical level, there is no guaranteed winner.

On the other question (directory structure, file attributes, and access control), the save set approach is, IMHO, a far superior alternative.

If staging bandwidth and performance is a concern, then the answer may lie in creating a larger number of less embracive save sets.

- Bob Gezelter, http://www.rlgsc.com

[John Gillings]

Norm,

The trick with getting BACKUP to replicate a directory structure is to keep the wildcards identical on input and output specifications. If you're moving whole trees up or down, define rooted devices to make them appear to be at the same level.

So

BACKUP source:[000000...]*.*;* dest:[000000...]*.*;*

I also find it useful to at least start with BACKUP/LOG, check that the files are going where you want, then ^Y, delete the destination files and rerun with /NOLOG.

As you've noticed, [*...] is not the same as [000000...]

[P Muralidhar Kini]

Hi Norm,

I remember having a similar requirement sometime back and faced some problem while using the BACKUP command.

Check the attachement for the commands that i have tried.

* Backup Command 1
BACKUP $3$DKA300:[000000.MURALI...]*.*;* $3$DKA500:[000000.TEMP...]*.*;* /log

Copies the source to destination but does not restore the directory structure.
Looks like directory within a directory structure is not maintained but
file within a directory structure is maintained

* Backup Command 2
backup $3$DKA300:[000000.MURALI*...]*.*;* $3$DKA500:[000000.TEMP...]*.*;* /log

This tricks the backup to copy the source to destination and preserve the
directory structure.
Note that the source spec has "MURALI*", so if there are any directories in
the source with say "MURALI1" or "MURALI2" then they also might get selected
and copied to destination.

I have tried this on V73-2.

Regards,
Murali

[P Muralidhar Kini]

Hi Norm,

>> $ backup/log/ignore=inter dsa1:[000000...] dnfs3:[0514...]
>> does not preserve directory tree. It creates directories, but not
>> subdirectories,
Having a wildcard in the source specification is what might trick backup in
giving you what you want.

Try the following command and let me know whether it solves your problem
$ backup/log/ignore=inter dsa1:[*...]*.*;* dnfs3:[0514...]*.*;*

This should preserve the directory structure.

Regards,
Murali

[RBrown_1 · Accepted solution]

Earlier, I said:

$ define dnfs3:[0514] dest/tran=conc

This should have been:

$ define dest dnfs3:[0514.] /tran=conc

(For those keeping score at home, I had the logical and equivalence names reversed, AND I left out the dot "." in the equivalence name. :-( )

Once this logical name is defined, then this BACKUP command will do what you have asked for:

$ backup dsa1:[*...] dest:[*...]

(Add other backup qualifiers as desired.)

Sorry for the inconvenience.

[Shriniketan Bhagwat]

Hi,

[*...] only selects the directories which contains files and subdirectories and not the empty directories, where as [000000...]*.*;* selects all the files in the volume. You can feel the difference just issuing the $ dir on both the specifications. This is how the RMS wildcard operation works and hence BACKUP should also work in the same lines. As Murali demonstrated, in this case [*...] and [000000...] is preserving the entire directory structure may be a trick with wildcard to use. What version of VMS are you using?


Regards,
Ketan

[Jan van den Ende]

Hohn, Murali:

>>>

BACKUP source:[000000...]*.*;* dest:[000000...]*.*;*
<<<

I strongly advise against that!

As long as dest is a physical device, it is OK, but when dest is defined as DEV:[DEST.] (a conceadled device derived from a directory, you will now create an actual DEV:[DEST]000000.DIR, after which all bets are off.
I have had to deal with several of such occurrences, and repaires are time-consuming at best. (and if it happens to occur on SYSTEM DISK volumes, consider yourself lucky if repair can be done without a full re-init and restore!)

I strongly recommand to always use
$ BACKUP concdev1:[*...] concdev2:[*...]
or for one dir out of concdec1:
$ BACKUP concdev1:[dir...] concdev2:[*...]
I never yet met a situation where it was not possible to easily define (maybe temporarily) concdev1 & concdev2 suitably.

If needs arises, to rename the top dir, output may be chosen as concdev2:[dir2...]

Note that BACKUP/BY_OWN=ORIGINAL maintains all ownerships, protections, and ACLs for you.

Just my EUR 0.02

Proost.

Have one on me.

jpe

[P Muralidhar Kini]

Hi Jpe,

>> BACKUP source:[000000...]*.*;* dest:[000000...]*.*;*
>> I strongly advise against that!

I agree with you.
I would rather prefer -
$ backup source:[*...] dest:[*...]

This command may work find for "source:[*...]".
But in case you want to backup contents of a particular directory in the disk
(i.e. source:[test...]*.*;*) then it will again give problems.
Check my attachment in my previous reply.

There was this time when i had to move some set of files from one disk to
another. The source directory had a lot of files and directory in it.
I ran in to problems using "[000000...]*.*;*" in the source and destination
of the BACKUP command.

I ended up writing my own command procedure, which would take as input the
"DIR [...]" of the source directory. It would then parse it and create directory at
the destination followed by copy of files from source to destination.
So much work to just copy files around!

Regards,
Murali

[John McL]

Norm, you mention "$ backup/log/ignore=inter dsa1:[000000...] dnfs3:[0514...] does not preserve directory tree. It creates directories, but not subdirectories"

I can't say that I'm surprised. The presence of the top level directory [000000] is often ignored when you are dealing with a directory just below it (e.g. [000000.HANSEN] will be processed as if it was [HANSEN]). My guess is that BACKUP establishes whether it's a directory tree or not from the first file that it processes and it doesn't repeat the test every time.

I wonder if [000000.*...] would do what you want? I've thought of this while I typed the above, but maybe someone else has already suggested it; unfortunately I can see all the postings while I'm replying.

[Norm Hansen]

Thanks to everyone for the input.

RBrown answered the original stated question with something that works. I had tried something similar, but had not been able to get it quite right.

Several people have spoken to advantages of save sets, which would be answers to the followup question. I'll try it out with save sets and see whether the de-dup will work in real life with an aggregated file.

[Norm Hansen]

Top points to RBrown for first submitting the concealed logical solution.

High points to P Muralidhar Kini for first pointing out wildcards on both sides. I had expected that this would proceed from current subdirectory (instead of root) on source volume, but it seems to work as advertised.

Also points to those who discussed save sets and contributed pros/cons on backup syntax.