SCP or SFTP to copy binary files?

[Thomas A. Williams]

I had considered backup savesets or zipfiles already. We have a job on each of our systems (about 30 of them) that copies various files to different directories on a central server using DECnet. The way it's written, it would be a major rewrite to zip up the files and then unzip them and move them to the right place. Ditto for using backup savesets. Long story. If SCP kept the file attributes like DECnet does, we wouldn't have this headache.

[Jim_McKinney]

Using the createFDL/SFTP/convertFDL approach will fail. CONVERT will read the file using the current file attributes to determine how to interpret what it finds. Since these do not match what is really inside those reads will not find record boundaries. CONVERT just reads and writes records so if it cannot determine the record boundaries correctly then what it outputs won't be what you want. The usual strategy for procedures such as this involve either encapsulating the transfered files in a container file that will behave (ZIP and BACKUP are two formats) or just resetting the file meta-data once it arrives at the destination. Does SET FILE/ATTR=(LRL:8192,MRS:8192,RFM:FIX,RAT:NONE) not produce a file that is usable?

[Thomas A. Williams]

>Does SET FILE/ATTR=(LRL:8192,MRS:8192,RFM:FIX,RAT:NONE) not produce a file that is usable?

Yes, it actually does.

Before:

File Organization: sequential
Record Format: stream-LF
Record Attributes: carriage-return
Maximum Record Size: 0
Longest Record: 32767


After:

File Organization: sequential
Record Format: fixed
Record Attributes:
Maximum Record Size: 8192
Longest Record: 8192




Thanks for the syntax help.

Question: It would be difficult, though, to make this dynamic enough to handle every file organization. I guess I could store all the properties prior to copying the file, but that adds a higher level of complexity...

What say you?

Let me mull this over and report back here...

[Jim_McKinney]

From the source system a script something like the following may work executed after the SFTP...

$! p1 is the filename both local and remote
$ lrl = f$file(p1,"lrl")
$ mrs = f$file(p1,"mrs")
$ rfm = f$file(p1,"rfm")
$ rat = f$file(p1,"rat")
$ ssh host set file/attr=(lrl:'lrl',mrs:'mrs',rfm:'rfm','rat:'rat') 'p1'

[Steven Schweda]

> [...] auditors are making us convert to SSH
> and we will have to disable DECnet at some
> point.

I don't know if anything ever came of it,
but there was some discussion (comp.os.vms?)
of using Stunnel (or something similar) to
provide an encrypted (secure?) IP tunnel
through which DECnet-over-IP could be run.
If this could be made to work, it would seem
to be a way to provide (good old) DECnet
functionality while restricting the external
network to IP, and keeping all the raw bits
disguised at the same time.

Perhaps someone here remembers if a
satisfactory conclusion was ever reached on
making this concept work.

http://www.stunnel.org/

[Ian Miller.]

A prebuilt version of stunnel for VMS is available at
http://h71000.www7.hp.com/opensource/opensource.html#stunnel

along with some documention. I have not seen anyone say they have used stunnel with DECnet/IP but its an interesting idea.

[Thomas A. Williams]

I'd love to be able to use stunnel - but they won't allow it because it's not suported by HP.

HP: Please start supporting stunnel. Please.

[Art Wiens]

I may have missed something along the way but your thread starts by saying VMS patches as having problems once they're at the destination. I'm assuming this is an example of the failure you're having.

What type of files are you really sending with your 30 nodes copying to a central site? I imagine PCSI files might be considered "special" containing both text and executables.

What's in your files?

We don't use HP TCPIP, but using TCPware sending to HPUX, I am able to SCP and/or SFTP ASCII files successfully (even though it says it's sending them in "binary").

Cheers,
Art

[Thomas A. Williams]

Actually, we're two people here working on similar tasks, both using my forum account.

I send the following:

OPERATOR.LOG files
ACCOUNTNG.DAT files
PSDC*.CPD files (Performance advisor files)
ERRLOG.SYS files
SECURITY.AUDIT$JOURNAL files
some other text and third-party files.

[Richard Whalen]

The SFTP protocol comes from the Unix world, where all files are just streams of bytes, and file characteristics consist of owner, protection and dates. Revisions to the protocol have added named attributes with a private format and text transfers, but few implementations use these features.

Process Software's SSH for OpenVMS implements the text transfer operation and will communicate VMS attributes via the private format to the remote SFTP server when it recognizes that the remote SFTP server is capable of accepting them.

[Thomas A. Williams]

>What type of files are you really sending with your 30 nodes copying to a central site? I imagine PCSI files might be considered "special" containing both text and executables.

>What's in your files?

Just to elaborate - Some day soon all other protocols will be shut off and all users (developers, DBA's, admins etc) will have to exclusively use SSL. I'm anticipating the user community to start screaming "I can't copy such and such a file with SCP/SFTP successfully". I want to have a solution in place to be able to handle that complaint. So it really is any sort of binary file that is non-standard from an SSL/SSH perspective, which is many VMS/RMS binary files.

[Dave Laurier]

Hi Thomas,

In that case I would like to mention the Oracle Rdb backup files. If you are using the OpenSSH port from HP then you will certainly have issues with these files as it only supports fixed length 512 byte for binary files.

In my case the *.RBF files are fixed length 32256 byte records. A workaround may be to archive the files prior to transfer using ZIP.

Best regards,

Dave Laurier

[Dave Laurier]

>Normally the PCSI files are in fixed length 512 bytes format so this should not give any problems.

>> They're fixed, but 8192 byte length. I guess thats why it doesn't work.

I see now that there are indeed different formats, at least at my test site:

DEC-AXPVMS-DFU-V0301--1.PCSI = Fixed length 512 byte records

DEC-AXPVMS-VMS732_UPDATE-V0600--4.PCSI = Fixed length 8192 byte records

So probably there is no guarantee that the PCSI files will always have the same format.

Best regards,

Dave Laurier