SFTP -B problems but not consistent

[Hoff]

This forum software is bad at in-line quoting.   I fully expect the format of this message to be corrupted.   Ah, well.

Looks like the sftp command parsing has been broken for a while, around the extended parsing setting.  Ah, well.

-D 99 shows all debug data.   The -v is pretty limited at what it shows.

3.  I noticed that when manually connecting to user@solarisbox, both P1A and T1A get the SFTP2> prompt but:

On P1A, the ascii command responds with the message that it is now in ascii mode (and the ascii -s command shows me the current settings.)

On T1A the ascii command is accepted but produces no output.  The ascii -s command produces no output.

That reeks of a software or client or local configuration difference.

sftp is not FTP.    There's typically no ASCII command.  More than a few clients either lack that, or ignore that for compatibility with folks that think they're using FTP.   In terms of what the ASCII and BINARY do — or are supposed to do, per what FTP does — are to identify the sort of data and the record organization.   BINARY means "don't mess with this".   ASCII means "mess with this, if the server thinks it needs to".

I also tried the getext/setext commands - which work on P1A but not on T1A - yet the installation was the standard PCSI  command to PROD INST TCPIP (and later, of course, the TELNET patch).  Because I've been burned more than once on bad installations, I ran an MD5 checksum on the images - and they match on P1A and T1A.  So I know I have the right versions installed.

That's the images.   Check the [.ssh2] directories involved for differences that might be lurking there.

FWIW, MD5 is not effective against intentional corruptions, and generating intentional collisions is entirely feasible.   SHA1 is looking dicy.   Yes, either should spot accidental or innocuous corruptions.   But if you're at all concerned with security, don't use either of these.  Use SHA2 or better, as available.

5.  As far as the file format for the layer-2 stuff, I have tried that both ways, as direct output from DCL and as output from CONVERT/FDL of the DCL file to a STREAM_LF file.  Hoff, this is why I have layer 1 and layer 2.  Would you prefer that I called these the DCL batch layer and the SFTP2 batch layer?  Because that is why they are separate.  I use the "divide and conquer" method, so the two layers are the way they are because they are handled by different command processors.

In any case, I have noticed that on T1A, when I gave SFTP an unconverted script, it first converted the file to STREAM_LF and THEN it said "Can't read the batchfile." 

There's so much here that I'm having trouble shorting through the details.   If all the files are Stream_LF, then discussions of Layer1, Layer2 and such seem... not entirely relevant.  I don't care what the files are or contain or otherwise — and don't need to know.   Just something akin to that there's a test file with identical contents and format being used for all tests.

6.  I asked the sys admin of the SOLARIS box about whether he had any restrictions on multiple servers using the same keys, but he says that in fact many of our projects do that because this of the DoD regulation on copying audit logs.  The other  production, test, and development boxes at our site send their audit logs via PKI using one key per project.  He is not doing anything to block files from the various network enclaves we have set up.

Another reason not to mention which entity is involved — sharing private keys akin to what's described is analogous to sharing passwords, and sooner or later that gets somebody in trouble when some password or some key needs to be revoked.

7.  Yes, we have HPE support and I will try that if I reach the point of being unable to resolve this transfer problem on my own  or through the forum.  It is just that once I escalate this to HP support, I commit myself to a more time-consuming process because of having to "vette" all of the exhibits I have to send for analysis by getting our network security guys to bless the files as "suitable sanitized" for publication.

Point'm here? 

My next try will be to convert the file to be transferred to STREAM_LF before I put it on the other system, because UNIX-like text files are the same as OpenVMS STREAM_LF format - so ascii vs binary transfers would then make no difference.

Isn't that what the Layer1 and Layer2 stuff did?   Or are we testing with different files?

[Steven Schweda]

> In any case, I have noticed that on T1A, when I gave SFTP an unconverted
> script, it first converted the file to STREAM_LF and THEN it said "Can't
> read the batchfile."

   Note that I showed an example of that happening (give or take a
sloppy quotation):

alp $ sftp "-B" ALP$DKC0:[SMS.itrc]sbat.dat mba
Warning: Converting file alp$dkc0:[sms.itrc]sbat.dat to Stream_LF.
Warning: File alp$dkc0:[sms.itrc]sbat.dat converted
 successfully to Stream_LF.
Error: Could not read the batchfile.

   That is, using a VMS-style file spec for the batch file.  Of course,
when one of us provides an actual command with its actual output, and
the other of us provides a vague description of a command and its
output, it may be difficult to make a detailed comparison between those
commands.


> [...] I ran an MD5 checksum on the images - and they match on P1A and
> T1A.  So I know I have the right versions installed.

   Knowing exactly which "the images" you compared might permit others
to judge the value of the comparison.  Clearly, if you see different
behavior with respect to the "ascii" command, then there must be _some_
difference between the environments on the two systems.  That difference
could lie in the executables or the run-time libraries, or in the
configuration files (system-wide or user-specific), or, probably, in
still other places.

[The_Doc_Man · Accepted solution]

Some quick replies: 

Steven, you asked:     And the output is the same on both systems?

Yes - the version-finder commands report the same versions on both systems.  If I take it a couple of steps farther along, I also get the same version information for the component modules when I do an ANALYZE/IMAGE on the SSH or SFTP images.  (i.e. TCPIP$SSH_SFTP2.EXE and a couple of others.)

Both Hoff and Steven commented on the difference between the ASCII commands.

Hoff:   That reeks of a software or client or local configuration difference.

Steven:   Sounds to me like a suspicious difference.

I agree, but I have learned to be less excitable as I draw nearer to retirement.  I am not down-playing it, but there are some things I have had to learn to accept - for now - and just work around it.

Hoff - I know that MD5 isn't absolutely fool-proof - but as a quick scheme to see if two large files appear superficially to be the same, it is decent.

I'm going to declare this problem closed based on the following:

I have extensively applied CONVERT/FDL to take the scripts that I am building into Stream_LF record format.  The file to be transferred is also converted similarly.  The files are dropped into the same folder from which I will be working so I can use "local" file parsing rather than fiddling with OpenVMS or UNIX file specifications.  SFTP might be dumber than a box of rocks but it can still do a PUT just fine when the default directory contains the desired file.  The PKI handshake works OK and the PUT command no longer encounters - or causes - format differences.  The network security guys are happy with what I give them.

Having said that, if you folks want to just put this on the shelf and forget it, go ahead.  I will separately pursue the problem of differences in the behavior of the ASCII command on the two servers, but so far I have not found any obvious differences in the configuration that would affect what is essentially a presentation-layer command.

FWIW, the things I have tried/checked/verified:

1.  The same versions of TCPIP services are now running on both systems and HP has no later version of TCPIP Services to which I could upgrade.

2.  The O/S patches are to the same level - UPDATE 10, SYS 6, and the other major and minor patch kits.  The only current patch we DON'T have is the TZ patch for the Russian time zones.  We've got everything else that applies, and those patches are on both systems.

3.  The TCPIP$SSH default SSH2_CONFIG. file is identical line-for-line on both systems.  I've done what I call a "blink comparison" with two windows open and viewing the corresponding files, one in each window.  When I switch between the two windows with ALT-TAB, if there differences they would stand out.  But nothing shows up.

4.  These scripts run as SYSTEM.  There is no "local" SSH2_CONFIG. file in the [SYSMGR.SSH2] directory for either system.  There is a copy of the SOLARIS-originated RSA key in that directory for both systems.

5.  No differences exit in the logical names that might apply based on SHOW LOGICAL *SSH* and SHOW LOGICAL *FTP*.

6.  Per the SOLARIS administrator, there are no system-specific sub-configuration segments for the two servers in his SSHD2_CONFIG. equivalent file; nor does he have differences between the sub-nets at other levels in his networking configuration.  He believes that he is treating my P1A and T1A servers identically.

7.  The interface settings on my server hardware are the same on both systems, differing in IP addresses only showing up in the 3rd and 4th octets of our internal IvP4 addresses that run behind our firewalls.  Both OpenVMS systems talk to the SOLARIS server which is on a different subnet than either OpenVMS server.  In both cases, we have defined a local host name with $ TCPIP SET HOST /LOCAL commands.  The connections work by name interactively from both OpenVMS servers to the SOLARIS server.

8.  On the OpenVMS boxes, all files are owned either by SYSTEM or by account TCPIP$SSH as appropriate.  The files have the same permission schemes (i.e. not protected) - [RWED,RWED,RWED,RWED] except for directories, which of course don't include DELETE permission.  No ACLs are present.  SYSTEM, of course, runs with BYPASS, but the audit logs tell me that the scripts are not accessing the files with privilege - they are using the permission masks.

9.  Not that I think it would have this particular effect, but neither disk quotas nor disk capacity are issues here.  Nor are SYSGEN parameters.  These systems were configured so that if either one failed, the other one could take over full operation almost seamlessly.  It's not like we saturate 2 x Intel 9300-4c (8 cores total) with something that is almost continuously I/O-bound, so we purposely made them structurally as close to each other as possible in case we had to temporarily host two environments on the same box.  Therefore, finding a difference at that level between the two environments is as much a surprise to me as it was to those of you who commented on it.

I guess I was hoping that someone would see these symptoms and tell me where to look for discrepancies, but it looks like this one is a bit more opaque than some of the things I've encountered in the OpenVMS world.

In any case, contributors, thanks for the feedback.