topic Re: RCP with big binary files in Operating System - OpenVMS

RCP with big binary files

Wim Van den Wyngaert — Wed, 31 May 2006 03:49:58 GMT

VMS 7.3, TCP 5.3 eco 2, username SYSMGR_WVW.

$ ucx show prox
VMS User_name Type User_ID Group_ID Host_name
SYSMGR_WVW C SYSMGR_WVW *

$ rcp salpv1:login.com wim.lis/log
%RCP-S-COPIED, DSA1:[OPERATIONS.SYSMGR.][WVW]WIM.LIS; copied (8 blocks)
%RCP-S-NEWFILES, 1 file(s) created

$ rcp salpv1:php.zip wim.lis/log
%RCP-S-COPIED, DSA1:[OPERATIONS.SYSMGR.][WVW]WIM.LIS; copied (119 blocks)
%RCP-S-NEWFILES, 1 file(s) created

$ rcp salpv1:"srv_falbck:[dsmmgr.database]arbvol1.gls" ":srv_arbbck:[dsmmgr.database]arbvol1.gls"/log
%RCP-E-LOSTCONN, lost connection

There is no error mesasge in the tcpip$rcp log file. The tcpip$rsh log file shows
%CONV-F-READERR, error reading !AS

Dir of the file with problems :
Record format: Variable length, maximum 255 bytes, longest 1 byte
Record attributes: Carriage return carriage control

What is wrong ?

Wim

BTW : don't want copy/ftp or ftp.

Re: RCP with big binary files

Steven Schweda — Wed, 31 May 2006 05:07:51 GMT

> big binary files

How big is "big"? Bigger than 2GB?

> Dir of the file with problems :
> Record format: Variable length, maximum 255 bytes, longest 1 byte
> Record attributes: Carriage return carriage control

Why do you call that "binary"?

What's the OS of the destination system?

Re: RCP with big binary files

Wim Van den Wyngaert — Wed, 31 May 2006 05:11:03 GMT

It's VMS to VMS (same version, in fact same node for this test).
It's a file of 100 MB.
It's binary because the contents is a database (non-printable).

Whats wrong : %RCP-E-LOSTCONN, lost connection

Wim

Re: RCP with big binary files

Steven Schweda — Wed, 31 May 2006 06:12:24 GMT

It's not exactly your file (many copies of
the VMS FAQ, EVE+COPY), but:

XXX.XXX;8 217572 31-MAY-2006 05:33:25.33 (RWED,RWED,RE,)

Record format: Variable length, maximum 0 bytes, longest 110 bytes
Record attributes: Carriage return carriage control

alp $ rcp alp:"ALP$DKA0:[SMS]xxx.xxx" ":dka0:[SMS.test]rcp_x2.x" /log
%RCP-S-COPIED, DKA0:[SMS.TEST]RCP_X2.X; copied (210593 blocks)
%RCP-S-NEWFILES, 1 file(s) created

Of course, after RCP, the file attributes
have changed to:

Record format: Stream_LF, maximum 0 bytes, longest 0 bytes
Record attributes: Carriage return carriage control

Is _that_ why you insist on using RCP?

alp $ tcpip show version

HP TCP/IP Services for OpenVMS Alpha Version V5.4 - ECO 5
on a COMPAQ Professional Workstation XP1000 running OpenVMS V7.3-2

Or perhaps it's the (lack of) speed that you
like? On a smaller (~10000-block) test file,
RCP took about 34 seconds, COPY/FTP about 7s.

Re: RCP with big binary files

Wim Van den Wyngaert — Wed, 31 May 2006 06:24:29 GMT

Yep. Nice piece of software.

In the mean time I found that the destination is stream but the origin may not be variable record length format when the contents is not (eve says illegal record).

I want(ed) to use rcp because it uses proxies. I don't like putting passwords in scripts.

Wim

Re: RCP with big binary files

Steven Schweda — Wed, 31 May 2006 06:29:42 GMT

DECnet? Zip first? I assume that with
your software version(s), SCP would be even
worse than RCP, but it, too, can obviate the
password, and it might have different (less
troublesome?) problems.

Re: RCP with big binary files

Wim Van den Wyngaert — Wed, 31 May 2006 06:40:21 GMT

The goal is to move away from decnet. And copy/ftp doesn't use proxies.

And yes, speed is a (small) problem. Copying 128 MB took 4 minutes with rcp, 3 minutes with decnet and 3 minutes with ftp. I'm willing to accept that. But not the file type handling.

Wim

Re: RCP with big binary files

Steven Schweda — Wed, 31 May 2006 06:53:53 GMT

> The goal is to move away from decnet.

"Doctor, it hurts when I do this."

"Don't do that."

DECnet-over-IP? I don't know of a one-step,
non-DECnet procedure which will move a
file without a password, and preserve its
RMS attributes.

Is the application unable to use a
stream-format file?

Re: RCP with big binary files

Wim Van den Wyngaert — Wed, 31 May 2006 07:01:14 GMT

No.

Re: RCP with big binary files

Hein van den Heuvel — Wed, 31 May 2006 07:17:09 GMT

Wim wrote> What is wrong ?

Just about everythign it seems!
1) That you are using RCP?
2) The current implementation of RCP?
3) The files to be copied
4) The security aspects?

Anyway, that performance difference.. is that per chance all CPU time?

>>> There is no error mesasge in the tcpip$rcp log file. The tcpip$rsh log file shows
%CONV-F-READERR, error reading !AS

Yikes... they call convert as part of RCP?
How can that be needed for simple sequential files? That will change bytes around, but hopefully will not change data.

>>> In the mean time I found that the destination is stream but the origin may not be variable record length format when the contents is not (eve says illegal record).

Please explore the full details of this.
How did this file come to be? What is it really? (attach text file with dump/block=count=1 output if needed).

>>> Dir of the file with problems :
Record format: Variable length, maximum 255 bytes, longest 1 byte
Record attributes: Carriage return carriage control

If that is true, then that is a horrible file where each character requires a call to RMS and each character takes up 4 bytes:
2 bytes for a record length word, 1 byte data and 1 byte filler to re-align the next record length word.

Hope this helps some,
Hein.

Re: RCP with big binary files

Wim Van den Wyngaert — Wed, 31 May 2006 07:24:36 GMT

Hein,

If HP supports rcp, it should work properly. On VMS.

The file is variable record length, max 255, longest 1 but this is not true. The file is a DSM database file and contains simply binary data.

And no, it was (rounded) wall time (elapsed). The 5 was actualy 5.5.

I think I will abandon this rcp stuff. I think it transfers via sys$net (thus has no id of the file format) and converts it (just like my remote backup). But to stream. Good on Unix but not on VMS.

Wim

Re: RCP with big binary files

Wim Van den Wyngaert — Wed, 31 May 2006 08:02:57 GMT

Btw : got it to work after doing set file/at=(rfm:fix,lrl:512). But can't do that for the files I'm copying in real live.

Wim

Re: RCP with big binary files

Joseph Huber_1 — Wed, 31 May 2006 08:21:39 GMT

If only TCPIP/IP protocols possible, and both sides are VMS systems, may I suggest to use HGFTP client and server ?

It allows anonymous FTP with "CD ~user" to a user directory, and is using VMS file mode for binary transfers.

Re: RCP with big binary files

Wim Van den Wyngaert — Wed, 31 May 2006 08:24:56 GMT

Rcp also has anonymous ftp. But that's a no go since file protections don't allow that.

Why o why didn't they implement proxies for ftp ...

Wim

Re: RCP with big binary files

Steven Schweda — Wed, 31 May 2006 09:11:20 GMT

> Yikes... they call convert as part of RCP?
> How can that be needed for simple sequential files?

By UNIX standards, variable-length format
records are not "simple". It's reasonable
(if not always correct) to assume that files
like this are text, and convert
appropriately. RCP lacks a "binary" option.

And in this case, it appears that the
original RMS attributes make no sense.

> Btw : got it to work after doing set file/at=(rfm:fix,lrl:512).

So, is the application unable to use a
fixed-512 format file?

Re: RCP with big binary files

Wim Van den Wyngaert — Wed, 31 May 2006 09:15:02 GMT

> RCP lacks a "binary" option.
On VMS yes. One some platforms the -b exists.

> So, is the application unable to use a
fixed-512 format file?
I don't want to find out if it works. I'm not going to change each file I copy. Also, since it is a db, you never know for sure.

Wim

Re: RCP with big binary files

Hein van den Heuvel — Wed, 31 May 2006 10:21:22 GMT

>> If HP supports rcp, it should work properly. On VMS.

Absolutely. I 100% agree.
Raise a serious support issue through the proper channels.

>> The file is variable record length, max 255, longest 1 but this is not true. The file is a DSM database file and contains simply binary data.

Forget the support call with HP.
VMS can not and will not support files that are not what they pretend to be.

I encourage you to raise a serious support call to the DSM providers. They should have given their file attributes reflective of, or at least consistent with the data in them.

Now, after the fact, one should expect them the be able to tell you which alternative attribute is acceptable to them. There is no way that the false label is help them and/or is a requirement.

Furhermore, then DSM provider should be able to recommend an appropriate copy tool / solution to you which may, or might not, require a export tool of sorts. Native, binary, DB files are rarely transportable.

>> Btw : got it to work after doing set file/at=(rfm:fix,lrl:512). But can't do that for the files I'm copying in real live.

Sure you can. It is a better lie than the current one! But by all means confirm with your DSM provider. They may have given those false attribute to make it clear that you can not copy those files around! (playing devils advocate here).

Why blame VMS when the layered product is clearly not playing by the rules.

Steven wrote> By UNIX standards, variable-length format records are not "simple". It's reasonable
(if not always correct) to assume that files
like this are text, and convert
appropriately. RCP lacks a "binary" option.

Ya But... a call out to CONVERT is very VMS-ish, part of the added value of the port to VMS. If you go through that trouble, then just read the darn records!
(Or figure out EXCHANGE/NET). RMS will do the decoding (as it does underlying CONVERT.
And also, if you wanted to go block mode and roll your own, then it is not rocket sience either.

Here is a brute-force/silly perl script I use on Windoze to read VMS variable length files which are 'accidently' transfered binary:

binmode STDIN;
while (read STDIN,$length_word,2) {
# avoid using "S" or "V". just do the math.
($length,$null) = unpack ("CC",$length_word);
$length += $null*256;
last if ($length > 32767);
$read = read STDIN,$line,$length;
print "$line\n";
read STDIN,$null,1 if ($length & 1);
}

Wim> I don't want to find out if it works. I'm not going to change each file I copy. Also, since it is a db, you never know for sure.

Well, IMHO you should. Just fix those files and move on, hopefully after getting confirmation by the DB provider, but possibly without.
The 'games' with RCP/FTP/CONVERT/... are far more dangerous than changing the attributes. Changing attributes either works or it does not.

Misrepresenting them can actually lead to (silent) data corruption. For example, when RMS encounters a record size > EOF it will just not return any data, and it is free to play with the 'fill' byte.

$ crea tmp.txt
aap
noot
mies
$ set file/attr=(rfm:fix,lrl:50:mrs:50) tmp.tmp
$ open/read/write tmp tmp.txt
$ read tmp record
$ shwo symb record
RECORD = "..aap...noot..mies................................"
$ write sys$output f$cvsi(6*8,2*8,record)
4
$ record[6*8,2*8]=12345
$ write/update tmp record
$ close tmp
$ set file/attr=(rfm:var,lrl:0,mrs:0) tmp.txt
$ type tmp.txt
aap
$

- Silent corruption!

- Yes I have a symbol 'shwo'... some times it is easier to work around known problems then to truly fix them.

Cheers,
Hein.