topic Re: Copy a file with same format in Operating System - OpenVMS

Copy a file with same format

rbhagwat — Mon, 14 Mar 2011 13:53:12 GMT

Hi,

I am learning the RMS/FDL APIs right now. One
program I am trying to write is to make an exact
copy of a file in terms of it's organization
and data.

Any pointers on how to do this using RMS APIs ?
Do we copy in FAB to out FAB and then copy records one by one ?

Re: Copy a file with same format

Hein van den Heuvel — Mon, 14 Mar 2011 14:34:57 GMT

In order to allow us to help you better, it may help to indicate that is the ultimate goal.

- Just to learn RMS?
- Simple (sequential) files or indexed as well?
- Just to copy a FILE
- Become comfortable enough to tackle 'the real problem'

RMS - You may want to start out with the COPYFILE example program in : http://h71000.www7.hp.com/doc/731final/4523/4523pro_002.html

To meet your current specification you would to copy fields like RAT, RFM from infab to outfab before $CREATE and check out details for XABFHC

The example copies a record at a time.
It is also possible to use $CREATE to get a file with the right attributes and then use SYS$READ + SYS$WRITE to move the raw blocks.
That is essentially what COPY does.

INDEXED -
You can use FDL$GENERATE to help setup FAB, RAB, XABS. Generate from input, adapt to become output.

Or you can roll your own, by hooking up KEY and AREA XABS. See show_roots.c example in:
http://h71000.www7.hp.com/doc/731final/4523/4523pro_003.html

COPY - A simple API for copy is sorely missing. Some use CONV$CONVERT as alternative. This can give the exact same attributes and records, but the bits might be different. Others use callable BACKUP. I suspect that mostly folks resort to call SPAWN to execute a COPY command. While there is some overhead here, and error handling gets less precise, typically such spawn would be reasonable notably for significant sized files. For frequent (more than 1/sec copies?) for smallish files you may want to roll your own.

Also look around in SYS$HELP. For example:
SYS$COMMON:[SYSHLP.EXAMPLES.CDSA.AES]AES.C;1

Hope this helps to get you going,
Regards,
Hein

Re: Copy a file with same format

rbhagwat — Mon, 14 Mar 2011 14:50:10 GMT

Hi,

Thanks for the help. The final goal is to
come up with a prototype that can backup
VMS files on another machine. This will be
backed and stored at common places/s where
UNIX files are also backed up.
So we need to design a kind of format of save set that we store and later use to restore the file/s. Thus, this has to be
done for all kinds of VMS files (seq/rel/idx)

I am going through RMS APIs (and FDL and other 2-3 related manuals) and it is huge
information to say the least. I am breaking
down this activity further. Copying files to
another file (keeping format intact) is a way
forward in that (hopefully). If that works,
next step is somehow dump all the FAB/RAB
information alongwith data in a backup set
(I was thinking of using FDL files instead
of dumping the structures as a first step).

Is this the right direction ?

Thanks,
Rahul

Re: Copy a file with same format

Hoff — Mon, 14 Mar 2011 14:52:59 GMT

The "Is there a callable copy?" question has been asked before.

As for resources for answering your own questions and learning how to research technical questions on OpenVMS...

The OpenVMS FAQ is here:

http://labs.hoffmanlabs.com/node/1

Which includes the section:

>>>
How do I do a file copy from a program?
There are several options available for copying files from within a program. Obvious choices include using lib$spawn( ), system( ), sys$sndjbc( ) or sys$creprc( ) to invoke a DCL COPY command. Other common alternatives include using the callable convert routines and the BACKUP application programming interface (V7.1 and later).
<<<

There is a massive repository of VMS information in the comp.os.vms newsgroup, and going back for a couple of decades of VMS-related questions, here:

https://groups.google.com/forum/?fromgroups#forum/comp.os.vms

The HP documentation can be searched with the addition of the following Google keyword onto your search:

site:h71000.www7.hp.com

The Ask The Wizard area has a zip archive available where you can download the entire archive of questions and answers (and search it) here:

http://www.hp.com/go/openvms/wizard

There is an RMS library for C programs available here:

http://labs.hoffmanlabs.com/node/1260

Re: Copy a file with same format

Hein van den Heuvel — Mon, 14 Mar 2011 14:55:05 GMT

Oh wait, it's you again.
I should have looked.
Get yourself professional, expert advice.

Best regards,
Hein

Re: Copy a file with same format

Hoff — Mon, 14 Mar 2011 15:02:48 GMT

Since you're clearly intent on spending more time and effort on a project that appears very likely to eventually fail, have a look at how zip and unzip preserve the metadata.

This is not a simple problem (and I know how to do this) and you're adding a whole layer of complexity with this off-platform restoration requirement, and (if you want to do a bootable VMS restoration) you've got a yet larger project.

I'd suggest that you look to provide better and more direct input your manager's schedule time and your company's budget. Tell your manager that you are struggling with this.

The further you drag this out, the worse the issues here will look. If you drag this out too far, and though there will also be clear management-level errors here, it'll be you that gets blamed for this.

That's not intended as a reflection on you, it's a reflection of the complexity of the task involved, and with the complete lack of technical familiarity with the target platform.

Re: Copy a file with same format

Hoff — Mon, 14 Mar 2011 15:10:44 GMT

Hey Hein, how many years to get BACKUP working reliably?

BACKUP was hitting nasty errors as recently as V6-ish with image backlink errors, and there have been BACKUP errors in subsequent VMS releases including problems with GPT restoration and related baggage in the V8-ish range.

I'm not entirely certain that the "new" file links stuff has been particularly documented, either. But I digress.

And that was with the folks working on BACKUP being far more familiar with RMS and the XQP structures than is the OP, and with direct access to the engineers working on MOUNT and XQP and RMS.

Re: Copy a file with same format

rbhagwat — Mon, 14 Mar 2011 15:16:17 GMT

Hi Hein, Hoff,

One of the reasons I am doing this is there
is lack of VMS developers around and we do
need (eventually) internal experience on this.

Please let me know options available to
get external expertise and advise on this.

I appreciate all the help that has been
offered till now. Having started totally new
to VMS, I moved on like this with this help
1. Using GNV to straight compile things as
porting strategy to realize that this
is not going to get anywhere as to even
compile a single file (with various
scripting issues due to our complex make
environment)
2. Compiling individual files to realize that
there are lot of things not supported by
CRTL.
3. Using RMS to develop small programs and
working way up to have a new code base
(backup/restore portion) totally.

Thanks,
Rahul

Re: Copy a file with same format

Hein van den Heuvel — Mon, 14 Mar 2011 15:30:44 GMT

Try to send Email to some of us who tried to help already. Or make an Email address for yourself somehow available (in your profile).

It is not appropriate for for use to suggest a specific name in this name or number in a reply this forum.
I wouldn't want to slight someone.
Several options involving folks I'd like to stay friendly with. :-)

It should be obvious that any resource who has worked on tools like OpenVMS Backup or related software, or on tools like ZIP should be able to provide you with valuable help.
We have hinted to such names already.

Regards,
Hein

Re: Copy a file with same format

rbhagwat — Mon, 14 Mar 2011 18:04:13 GMT

Thanks. I will reach out once it is approved.

Re: Copy a file with same format

Andy Bustamante — Mon, 14 Mar 2011 21:53:56 GMT

I don't want to cast any doubt on the qualifications and ability of anyone who's already posted a reply, but there is a network backup solution available for OpenVMS. Symantec NetBackup may provide the functionality you need. The application allows you to save and restore OpenVMS files. There is no "bare metal" restore option, you'll either need to maintain operating system tapes or be prepared to install a recovery operating system. You could attempt to deploy your own solution of course.

Re: Copy a file with same format

Robert Gezelter — Mon, 14 Mar 2011 23:21:58 GMT

rbhaqwat,

I agree with Hein and Hoff, among others. This is a complex project, and is definitely not a starting point.

As has been noted, BACKUP took quite a bit of care to get right, and had its share of issues along the way. Starting the climb to this level of functionality from nothing is definitely a challenge.

I would concur with getting some professional assistance from someone with significant experience in these areas.

- Bob Gezelter, http://www.rlgsc.com

Re: Copy a file with same format

Steven Schweda — Tue, 15 Mar 2011 01:16:37 GMT

> [...] Symantec NetBackup may provide the
> functionality you need. [...]

Do you think that they'll sell him the source
code? He wants to _sell_ some software, not
to _buy_ some software.

> [...] So we need to design a kind of format
> [...]

Because you've looked at the existing
Info-ZIP stuff, and it's all unsuitable?

> Is this the right direction ?

I doubt it, but if that's what you want to
do, ...

> [...] have a look at how zip and unzip
> preserve the metadata.

I could be biased, but it's just possible
that if you get the same piece of advice from
everyone who knows anything, then perhaps you
should spend a little time investigating that
suggestion. Just a thought. I don't want to
tell you how to do your job.

Re: Copy a file with same format

Shriniketan Bhagwat — Tue, 15 Mar 2011 02:39:16 GMT

Hi,

I agree with Hoff that BACKUP has taken these many years to be at this stage. It took quit some time for us to fix the backlink errors between VMS$COMMON.DIR and SYSCOMMON.DIR (BACKUP-I-BTCROUT) and also while adding /GPT qualifier. While designing the applications like BACKUP, it is required to know the disk structures, file system, RMS APIs, system services etc. Ultimately it is the question of data integrity. You may archive some critical data today and you do not want to see the restore errors after a year when you restore it. We had few such instances in the past. Luckily they did not lead to data loss.

Regards,
Ketan

Re: Copy a file with same format

Jeremy Begg — Thu, 24 Mar 2011 03:47:06 GMT

There is an alternative to processing the FAB and XAB structures yourself. You might find a solution using the FDL routines (see the "OpenVMS Utility Routines" manual).

Specifically, you can use FDL$GENERATE to create a textual representation of the file's RMS structure. You could then store this meta-data on the backup server, along with the file contents (read using block I/O rather than record I/O).

To restore the file you could use FDL$CREATE or FDL$PARSE and then write the file (again using block I/O).

Note that I have not tried this and there may be significant problems with this approach that have not occurred to me in the 2 minutes it took to type this response. But you might find it an instructive exercise nonetheless.

Regards,
Jeremy Begg

Re: Copy a file with same format

rbhagwat — Thu, 24 Mar 2011 14:56:35 GMT

Hi,

Thanks. I thought about it and saw the FDLs
generated. It did not generate all the
information that we wanted to backup and
restore.

Thanks,
Rahul

PS - This C RTL is not really useful at all.
I am moving towards using system services
almost for everything. (So this is like a new
code instead of porting). Specifically,
exec and vfork cannot simulate what fork() and
exec() does on UNIX fully.

Re: Copy a file with same format

Hoff — Thu, 24 Mar 2011 15:25:19 GMT

Most C applications don't use the full copy on write virtual memory management capabilities of the fork/exec processing; only a select few do. And VMS lacks copy on write.

There are hooks in the CRTL to allow some of the (other) capabilities of fork/exec to be replicated on VMS. See the decc$set_child_* calls for maintaining (some of) the context.

VMS is lacking the clone() call that you'll find in Linux.

Some low-level file system details are available here:

http://labs.hoffmanlabs.com/node/209
http://labs.hoffmanlabs.com/node/112
http://labs.hoffmanlabs.com/node/28

There are pointers to some of the specs from there.

You'll also be working in the XQP layer for some of what you need; that's where the metadata is most directly accessible. And for a full-disk bootable restoration, you will need to use logical I/O calls for some of the restoration processing.

There's an RMS-level C library here:

http://labs.hoffmanlabs.com/node/1260

That library will be nowhere near what you will need to have here, given what you're up to, too.