SFTP -B problems but not consistent

[The_Doc_Man]

First, I haven't visited in a while so I'm a bit rusty in posting.

Down to business.  This is a U.S. Dept. of Defense site so I will obscure some things that need to be obscured.  Our goal, as mandated by network security officers, is to make copies of our AUDIT log files on another server so that, per D.o.D. regulation, the system administrator doesn't read his own audit logs.  (It's called "division of responsibility.")  Folks who have ever worked with SPLUNK will understand.  So here is the situation.

2 OpenVMS servers, let's call them P1A and T1A (production and test/development, respectively) that have to send files to a virtual SOLARIS server.  I can't tell you much about the SOLARIS server, but the narrative will show that the SOLARIS box isn't the problem.

The P1A and T1A boxes are identical hardware - each an rx2800 i2, each running OpenVMS 8.4 and each patched about once per quarter, last patch was Tuesday 10/13/15.  We have a smattering of the other patches, but the biggies are UPDATE 10 and SYS 6, which we have.  So we are up to date on patches.  Each running TCPIP Services for OpenVMS 5.7-13ECO5, which we needed because of the AESnnn-CTR ciphers.

I have  a script that runs at midnight to do a SET AUDIT "new_log" operation.  I capture that binary file, use the obvious ANALYZE/AUDIT command (with /FULL for format choice), get a file, and rename it to include node name and date tag.  I have a script that is supposed to transfer the file to the SOLARIS server.  It comes in two layers.  One layer sets up context and logical names and builds the second-layer script for SFTP2's -B option.  All logical names get translated and/or parsed by layer 1 so that layer 2 is about as plain-vanilla as it gets.  The second layer contains a few commands including the ASCII and PUT commands plus an LS -L option so I can later verify transfer.  The second layer batch file ends with an EXIT.  This first layer creates the 2nd layer and uses CONVERT/FDL to make it a STREAM_LF file, which I can later verify with DIR/FULL on said file.  The STREAM_LF file is the one actually named in the SFTP2 -B command.

From the P1A box, the script works.  It has worked now for several days.  Since we are testing the process, I waited to do the T1A box until P1A was working - which it now is.  So ....

When I copy the script from P1A to T1A and copy the SSH keys, I can MANUALLY log in as system on T1A, issue the SFTP2 command to the <user>@<solaris-box> address, and get the SFTP2> prompt without requiring a password - so PKI is working as it should.    When I run the script /USER=SYSTEM from T1A, (the same script that works on P1A correctly), I get the message that tells me it cannot read the batch file.

The log file verifies that the command fails with "Could not read the batchfile."  On P1A, no problem.  On T1A, no batch file.  Yet I can manually type the batch file.  I've checked directory permissions and file ownerships, but they match on both nodes.  One of the other posts with a similar problem says "Think UNIX" when using SFTP2, so I've tried both of these syntaxes:

SFTP2 "-BT1A$DKA1:[PUBLIC]XFR.COM" <user>@<solaris-box>

and

SFTP2 "-BT1A$DKA1/PUBLIC/XFR.COM" <user>@<solaris-box>

The ONLY difference on T1A between these two cases is one extra line logged in the first layer's log file:  The one where I used UNIX syntax says

Warning: Error trying to access file T1A$DKA1/PUBLIC/XFR.COM

Error: Could not reach the batchfile.

Where I used OpenVMS pathing syntax, the warning line is omitted.

It should also be noted that on the T1A logs, I can see the "welcome/warning" banner that is D.o.D. standard, so the connection works.  It is reading the batch file from SFTP that is the issue.

I've also tried "-B"T1A$DKA1:[PUBLIC]XFR.COM - which fails.  Can't do "-B"T1A$DKA1/PUBLIC/XFR.COM because DCL intervenes and says there is no such option as /PUBLIC.

It isn't permissions.  The job runs as SYSTEM and all files are [RWED,RWED,RWED,RWED], except any involved directories are [RWE,RWE,RWE,RWE].  The SFTP batch files would be deleted if the job succeeded, but since the job aborts they don't get deleted.  It is clear from the fact that the banner is in the log files that it is SFTP that can't read the batch file. 

I am using the system-default SSH2_CONFIG file on both systems and they are line-for-line identical.  They both specify "AnyStdCipher" for the Ciphers option.  Neither system has a "local" config file for SYS$SYSDEVICE:[SYSMGR.SSH2], which is where (I think) the SYSTEM config file would go if I had a private copy.  But I don't.

I checked the system logical names for templates *FTP* and *SSH* but there are no differences other than device names because of the SAN drives we use for some of our archive directories.  Nothing that would affect this situation.

Does anyone have suggestions as to where else I might look in order to discover why P1A works and T1A fails on identical batch jobs?  I'll freely admit that it is probably some silly little error on my part, but I'm starting to grasp at straws here.  If I still had hair, I'd be pulling it out right about now.

[KIP]

Interesting...

I would take steps to manually establish new keys from the T1A Node out to the target UNIX platform and make sure of the SSH connection that way first before relying on the batch process while testing.  This would exclude issues of multiple IP addresses using the same keys. 

Just my 2c.

Regards,

John K.

[The_Doc_Man]

KIP - thanks for your interest, but I don't think it is the connection that is the problem unless it is the daemon on the SOLARIS side that reads the file.

I have already manually verified interactively that the PKI SSH keys are set up correctly.  I can interactively just type

$ SFTP2 <user>@<solaris-box>

and get the SFTP2> prompt.  From there, the commands  PWD  and  LS -L work correctly.  It is not the handshake between the boxes.  It is that I cannot get T1A to read the batch file and I don't know why P1A works but T1A doesn't.

I added "-D 2" to the command line to see exactly where it failed.  (This is why I am certain that it isn't the server at the other end or the session itself.)  Here are some excerpts, which I will sanitize, and you'll know I sanitized because I'll use the angle-brackets to show what got redacted.

openvms_emulation/OPENVMS_EMULATION.C:2403: SYS$COMMAND is a disk or mailbox device; it has been redefined or process is running in batch mode.<CR>

 <redacted several lines>

SshFileCopy/SSHFILECOPY.C:1373: Connecting to remote host. (host = <user>@<solaris-box>, user = NULL, port = NULL)<CR>

argv[0] = /sys$system/tcpip$ssh_ssh2<CR>

argv[1] = -v<CR>

argv[2] = -x<CR>

argv[3] = -a<CR>

argv[4] = -o<CR>

argv[5] = passwordprompt %U@%H's password: <CR>

argv[6] = -o<CR>

argv[7] = authenticationnotify yes<CR>

argv[8] = -o<CR>

argv[9] = BatchMode yes<CR>

argv[10] = <user>@<solaris-box><CR>

argv[11] = -s<CR>

argv[12] = sftp<CR>

 <redacted several lines>

debug(15-OCT-2015 12:31:24.29): Ssh2Client/SSHCLIENT.C:1655: Creating transport protocol.<CR>

debug(15-OCT-2015 12:31:24.29): SshAuthMethodClient/SSHAUTHMETHODC.C:104: Added "publickey" to usable methods.<CR>

debug(15-OCT-2015 12:31:24.29): SshAuthMethodClient/SSHAUTHMETHODC.C:104: Added "keyboard-interactive" to usable methods.<CR>

debug(15-OCT-2015 12:31:24.29): SshAuthMethodClient/SSHAUTHMETHODC.C:104: Added "password" to usable methods.<CR>

debug(15-OCT-2015 12:31:24.29): Ssh2Client/SSHCLIENT.C:1696: Creating userauth protocol.<CR>

debug(15-OCT-2015 12:31:24.29): client supports 3 auth methods: 'publickey,keyboard-interactive,password'<CR>

debug(15-OCT-2015 12:31:24.29): SshUnixTcp/SSHUNIXTCP.C:1758: using local hostname <>t1a.<rest of fqdn><CR>

<redacted several lines>

debug(15-OCT-2015 12:31:24.30): Ssh2Transport/TRCOMMON.C:1935: Using Client order for common key exchange algorithms.<CR>

debug(15-OCT-2015 12:31:24.30): Ssh2Transport/TRCOMMON.C:1139: Sending packet with type 2 to connection<CR>

debug(15-OCT-2015 12:31:24.30): Ssh2Transport/TRCOMMON.C:1139: Sending packet with type 20 to connection<CR>

debug(15-OCT-2015 12:31:24.30): Ssh2Transport/TRCOMMON.C:1139: Sending packet with type 2 to connection<CR>

debug(15-OCT-2015 12:31:24.30): Ssh2Transport/TRCOMMON.C:1139: Sending packet with type 30 to connection<CR>

debug(15-OCT-2015 12:31:25.15): Ssh2Transport/TRCOMMON.C:2832: >TR packet_type=20<CR>

debug(15-OCT-2015 12:31:25.15): Ssh2Transport/TRCOMMON.C:2394: lang s to c: `de-DE,en-US,es-ES,fr-FR,it-IT,ja-JP,ko-KR,pt-BR,zh-CN,

debug(15-OCT-2015 12:31:25.15): Ssh2Transport/TRCOMMON.C:2459: c_to_s: cipher aes128-ctr, mac hmac-sha1, compression none<CR>

debug(15-OCT-2015 12:31:25.15): Ssh2Transport/TRCOMMON.C:2462: s_to_c: cipher aes128-ctr, mac hmac-sha1, compression none<CR>

debug(15-OCT-2015 12:31:25.15): Ssh2Transport/TRCOMMON.C:1139: Sending packet with type 2 to connection<CR>

debug(15-OCT-2015 12:31:25.15): Ssh2Transport/TRCOMMON.C:1139: Sending packet with type 30 to connection<CR>

debug(15-OCT-2015 12:31:25.16): Ssh2Transport/TRCOMMON.C:2832: >TR packet_type=31<CR>

debug(15-OCT-2015 12:31:25.17): Remote host key found from database.<CR>

debug(15-OCT-2015 12:31:25.18): Ssh2Transport/TRCOMMON.C:1139: Sending packet with type 2 to connection<CR>

<some lines redacted>

debug(15-OCT-2015 12:31:25.31): server offers auth methods 'publickey,password,keyboard-interactive'.<CR>

debug(15-OCT-2015 12:31:25.31): Ssh2AuthPubKeyClient/AUTHC-PUBKEY.C:1677: adding keyfile "/SYS$SYSROOT/SYSMGR/ssh2/<key names server so redacted>" to

debug(15-OCT-2015 12:31:25.31): Ssh2Transport/TRCOMMON.C:1139: Sending packet with type 2 to connection<CR>

debug(15-OCT-2015 12:31:25.31): Ssh2Transport/TRCOMMON.C:1139: Sending packet with type 50 to connection<CR>

debug(15-OCT-2015 12:31:25.31): Ssh2Transport/TRCOMMON.C:2832: >TR packet_type=60<CR>

debug(15-OCT-2015 12:31:25.31): Constructing and sending signature in publickey authentication.<CR>

debug(15-OCT-2015 12:31:25.31): Ssh2AuthPubKeyClient/AUTHC-PUBKEY.C:869: ssh_client_auth_pubkey_send_signature: reading /SYS$SYSROO

debug(15-OCT-2015 12:31:25.33): Ssh2Transport/TRCOMMON.C:1139: Sending packet with type 2 to connection<CR>

debug(15-OCT-2015 12:31:25.33): Ssh2Transport/TRCOMMON.C:1139: Sending packet with type 50 to connection<CR>

debug(15-OCT-2015 12:31:25.43): Ssh2Transport/TRCOMMON.C:2832: >TR packet_type=52<CR>

debug(15-OCT-2015 12:31:25.43): Ssh2AuthPubKeyClient/AUTHC-PUBKEY.C:1915: Public key authentication was successful.<CR>

debug(15-OCT-2015 12:31:25.43): Ssh2Common/SSHCOMMON.C:310: Received SSH_CROSS_AUTHENTICATED packet from connection protocol.<CR>

debug(15-OCT-2015 12:31:25.43): Ssh2Common/SSHCOMMON.C:852: num_channels now 1<CR>

<long sequence redacted>

Sftp2/SFTP2.C:4520: buffer: 'AUTHENTICATED YES

<more redaction>

debug(15-OCT-2015 12:31:26.30): Ssh2Transport/TRCOMMON.C:1139: Sending packet with type 2 to connection<CR>

debug(15-OCT-2015 12:31:26.30): Ssh2Transport/TRCOMMON.C:1139: Sending packet with type 94 to connection<CR>

debug(15-OCT-2015 12:31:26.32): Ssh2Transport/TRCOMMON.C:2832: >TR packet_type=94<CR>

debug(15-OCT-2015 12:31:26.32): SshConnection/SSHCONN.C:2312: >CXN packet_type=94<CR>

SshFileCopy/SSHFILECOPY.C:1301: Connection to remote host '<user>@<solaris-box>', ready to serve requests.<CR>

Error: Could not read the batchfile.<CR>

SshUnixPipeStream/SSHUNIXPIPESTREAM.C:481: ssh_pipe_stream_destroy<CR>

%TCPIP-E-SSH_FC_ERROR, error in ssh file transfer operation

SYSTEM job terminated at 15-OCT-2015 12:31:26.34

The log file clearly shows I have an autheticated (via PKI) connection using AES128-CTR and SHA1, but it is the batch file operation that fails miserably.  The batch file gets processed from P1A but not from T1A.

[KIP]

Are P1A and T1A nodes in the same cluster or separate (non-clustered servers)?  I'm thinking it must be on the UNIX side...

[Hoff]

ssh and sftp are sensitive to the protections on the key files, and particularly the private keys.  Check that those are properly secured.

Some versions of TCP/IP Services have expected various files to be Stream_LF, and some other releases do not.  That seems to vary by release and by specific applications, unfortunately.  In short, see if switching to Stream_LF works.

I'll assume T1A is the SCS host name, but DKA1 is an unusual name for a SCSI disk.

Since you're on current releases, you very likely have access to HP support, so call HP support and ask for their assistance. 

I'd expect the command to work with traditional parsing:

sftp  "-B"  batchfile user@example.com

Using RMS extended parsing reduces the need to quote stuff to avoid the C RTL downcasing switches and such, and I'd generally recommend that approach.   SET PROCESS /PARSE=EXTENDED, etc.

OpenVMS filename syntax should work, here.   If you have a logical name for the device — or a logical name that redirects the device — check for logical name translations both interactively and when invoked in batch. 

There's also a whole lot of LAYER1 and LAYER2 and some THIS-n-THAT and we-can't-tell-you-about-Solaris and they're Audit Files and it involves DoD and a whole lot of other stuff clouding the question.   A whole bunch of details that are irrelevant to the question, AFAICT.   Please expunge all that stuff, and distill this down to the command or commands used, the debug log generated, and the error messages. 

When you post commands and logs here, I'd also ask that you consistently replace your sensitive information with example.com, example.org and/or example.net throughout, and user1 and user2 or such for the usernames, and that you particularly do NOT add < or > or other characters as that makes it impossible to determine what syntax was used for the various commands, or where there are differences.   Please make these edits consistently throughout all the commands and all the data, as well.  Use global search and replace.  That'll help us spot syntax errors, without you having revealed some sensitive host name or another.

Some related reading: http://labs.hoffmanlabs.com/node/1118

Your reference to "think Unix" implies you've read this thread: http://h30499.www3.hp.com/t5/Networking/Using-SFTP-in-Batch-Mode-on-OpenVMS/td-p/5021612 — and if you're thinking Unix, then you need a / at the front of the absolute path.   Otherwise you get a relative path, and — unless you're in the exact right spot in the file system — an error.   Something akin to "/T1A$DKA1/PUBLIC/XFR.COM"

[Steven Schweda]

> [...] OpenVMS 8.4 [...] TCPIP Services for OpenVMS
> 5.7-13ECO5 [...]

   Newer than mine, so I know nothing, but...

alp $ tcpip show version

  HP TCP/IP Services for OpenVMS Alpha Version V5.6 - ECO 5
  on a COMPAQ Professional Workstation XP1000 running OpenVMS V8.3

alp $ sftp -"V"
alp$dkc0:[sys0.syscommon.][sysexe]tcpip$ssh_sftp2.exe: SSH Secure Shell OpenVMS
(V5.5) 3.2.0 on COMPAQ Professional Workstation  - VMS V8.3


> Using RMS extended parsing reduces the need to quote stuff
> to avoid the C RTL downcasing switches and such [...]

   You'd like to think so, but, around here:

alp $ show proc /parse
[...]
Parse Style: Extended

alp $ sftp -B sbat.dat mba
-b requires an argument greater than zero.

> [...] Something akin to "/T1A$DKA1/PUBLIC/XFR.COM"

   Right.  For example:

alp $ dire sbat.dat

Directory ALP$DKC0:[SMS.itrc]

sbat.dat;1          

Total of 1 file.

alp $ type sbat.dat
pwd

exit
alp $

alp $ sftp "-B" ALP$DKC0:[SMS.itrc]sbat.dat mba
Warning: Converting file alp$dkc0:[sms.itrc]sbat.dat to Stream_LF.
Warning: File alp$dkc0:[sms.itrc]sbat.dat converted
 successfully to Stream_LF.
Error: Could not read the batchfile.

%TCPIP-E-SSH_FC_ERROR, error in ssh file transfer operation

alp $ dire sbat.dat

Directory ALP$DKC0:[SMS.itrc]

sbat.dat;2          sbat.dat;1          

Total of 2 files.

   So, although it couldn't find the result, the conversion
apparently worked, so we shouldn't see that Stream_LF
complaint again.

alp $ sftp "-B" ALP$DKC0/SMS/itrc/sbat.dat mba
Warning: Error trying to access file alp$dkc0/sms/itrc/sbat.dat.
Error: Could not read the batchfile.

%TCPIP-E-SSH_FC_ERROR, error in ssh file transfer operation

  But, with the leading slash:

alp $ sftp "-B" /ALP$DKC0/SMS/itrc/sbat.dat mba
sftp> pwd
/Users/sms
sftp> exit

alp $

[The_Doc_Man]

Hoff - I'll try to reduce problem statement size.  Not an excuse, but on another forum where I contribute, folks get gigged for not having ENOUGH details.  So I was trying to strike a balance.  (Yeah, I know... what did that balance ever do to me to deserve such treatment...?)

OK, some answers and some more questions:

1.  T1A and P1A are NOT cluster members.   

They are the logical successors of a configuration that was on Alphas as a cluster as recently as 2008 or 2009.  By the time we finally migrated off the Alphas to the Integrity servers, they were stand-alone already but retained some vestiges of that cluster environment.  I.e. the logical naming scheme on both systems includes both production-side and test-side references.  But they are no longer talking to each other.  Since the boot-up schemes assume that you have a cluster-like setup (i.e. they look at SYS0 paths even on a non-cluster system), we still have multi-valued logical names that go through SYS0 and VMS$COMMON folders.

2.  I will have to try extended parsing.  That's a good idea.

3.  I noticed that when manually connecting to user@solarisbox, both P1A and T1A get the SFTP2> prompt but:

On P1A, the ascii command responds with the message that it is now in ascii mode (and the ascii -s command shows me the current settings.)

On T1A the ascii command is accepted but produces no output.  The ascii -s command produces no output.

4.  I'm guessing that the discussion of transfers using relative directories might be relevant because I adjusted the layer-1 script to COPY the audit file to my SYS$PUBLIC folder and then SET DEFAULT to SYS$PUBLIC. 

When I then build the layer-2 script, I no longer have to do an lcd command because the file I want is now in my default directory - so I just supply the file name and type when doing the put - and that makes the put command work on T1A.

However, because SFTP2 does not appear to have honored the ascii command (see #3), the file was transferred in binary format rather than honoring the ascii translation handshake that is supposed to go through the file system to split the file into separate lines.

I also tried the getext/setext commands - which work on P1A but not on T1A - yet the installation was the standard PCSI  command to PROD INST TCPIP (and later, of course, the TELNET patch).  Because I've been burned more than once on bad installations, I ran an MD5 checksum on the images - and they match on P1A and T1A.  So I know I have the right versions installed.

5.  As far as the file format for the layer-2 stuff, I have tried that both ways, as direct output from DCL and as output from CONVERT/FDL of the DCL file to a STREAM_LF file.  Hoff, this is why I have layer 1 and layer 2.  Would you prefer that I called these the DCL batch layer and the SFTP2 batch layer?  Because that is why they are separate.  I use the "divide and conquer" method, so the two layers are the way they are because they are handled by different command processors.

In any case, I have noticed that on T1A, when I gave SFTP an unconverted script, it first converted the file to STREAM_LF and THEN it said "Can't read the batchfile." 

6.  I asked the sys admin of the SOLARIS box about whether he had any restrictions on multiple servers using the same keys, but he says that in fact many of our projects do that because this of the DoD regulation on copying audit logs.  The other  production, test, and development boxes at our site send their audit logs via PKI using one key per project.  He is not doing anything to block files from the various network enclaves we have set up.

7.  Yes, we have HPE support and I will try that if I reach the point of being unable to resolve this transfer problem on my own  or through the forum.  It is just that once I escalate this to HP support, I commit myself to a more time-consuming process because of having to "vette" all of the exhibits I have to send for analysis by getting our network security guys to bless the files as "suitable sanitized" for publication.

My next try will be to convert the file to be transferred to STREAM_LF before I put it on the other system, because UNIX-like text files are the same as OpenVMS STREAM_LF format - so ascii vs binary transfers would then make no difference.

[The_Doc_Man]

As to versions:

$ tcpip show version

HP TCP/IP Services for OpenVMS Industry Standard 64 Version V5.7 - ECO 5

on an HP Integrity rx2800 i2 (1.60GHz/5.0MB) running OpenVMS V8.4

$ sftp2 "-V"

T1A$DKA0:[SYS0.][SYSEXE]TCPIP$SSH_SFTP2.EXE: SSH

Secure Shell OpenVMS (V5.5) 3.2.0 on HP Integrity rx2800 i2 (1.60GHz - VMS V8.4

$ prod show prod

------------------------------------ ----------- ---------

PRODUCT KIT TYPE STATE

------------------------------------ ----------- ---------

HP I64VMS AVAIL_MAN_BASE V8.4 Full LP Installed  <===============

HP I64VMS C V7.3-20 Full LP Installed

HP I64VMS CDSA V2.4-322 Full LP Installed

HP I64VMS CSWB V1.1-12 Full LP Installed

HP I64VMS CSWS V2.2 Full LP Installed

HP I64VMS DCE V3.2 Full LP Installed

HP I64VMS DCPS V2.7 Full LP Installed

HP I64VMS DWMOTIF V1.7 Full LP Installed

HP I64VMS DWMOTIF_SUPPORT V8.4 Full LP Installed

HP I64VMS HPBINARYCHECKER V1.1 Full LP Installed

HP I64VMS KERBEROS V3.2-159 Full LP Installed

HP I64VMS OPENVMS V8.4 Platform Installed        <===================

HP I64VMS RTR V5.2-441 Full LP Installed

HP I64VMS SSL V1.4-502 Full LP Installed

HP I64VMS TCPIP V5.7-13ECO5 Full LP Installed    <===================

HP I64VMS TDC_RT V2.3-20 Full LP Installed

HP I64VMS VMS V8.4 Oper System Installed         <===================

HP I64VMS VMSI18N V8.4 Full LP Installed

HP I64VMS WBEMCIM V2.99-A100614 Full LP Installed

HP I64VMS WBEMPROVIDERS V2.0-4 Full LP Installed

HP VMS AVAIL_MAN_COL V3.1 Full LP Installed

LGTO I64VMS SNCLNT V8.0-0 Full LP Installed

SMARTSTAR VMS SMARTSTAR V7.2-6 Full LP Installed

------------------------------------ ----------- ---------

23 items found

For some reason my screen capture didn't honor the spacing correctly because the above looks OK on my Reflection screen.

Anyway, I added arrows to help you find the versions of VMS and TCPIP in question.

We also have some things that came with the pre-loaded stand-alone backup installation kit (like the WBEM, RTR, and KERBEROS kits) but we don't configure them and it is easier to just let those sleeping dogs lie.

LGTO is, of course, the Legato backup software except that it is really EMC Networker.  SmartStar is a 3rd-party middleware package related to our use of ORACLE back-end databases.  The rest of that, you should probably be able to recognize.

[The_Doc_Man]

The question also came up about device names.

Don't ask me, I just use the names that show up after doing

$ SYSMAN IO AUTOCONFIGURE ALL

What we see is that our HP410i SmartControllers with SCSI-2 disks call themselves

nodename$DKAn

(And BTW, the Alphas did this too.)

The fiber-connected disks go through FGA0/FGC0 (dual-pathed) and are $1$DGAnnn

Since I use logical names for most of this, physical names really don't matter that much.  At one point I used  F$PARSE when building the layer-2 script that contained the PUT command, so I got all-physical device names and the paths did not include logical names (because none of them were /TRANSLATION=TERMINAL).  But on P1A, that didn't make a difference that I could see.  Actually, it didn't make that much of a difference on T1A either.  P1A worked correctly both ways, T1A didn't work correctly both ways - so I take that to mean that logical vs. physical naming was not a contributor to the problem.

[Steven Schweda]

> $ tcpip show version
> [...]
> $ sftp2 "-V"
> [...]

   And the output is the same on both systems?

> On P1A, the ascii command responds with the message that it is now in
> ascii mode (and the ascii -s command shows me the current settings.)

   Sounds to me like a suspicious difference.  (Around here, talking to
a Mac, "ascii [-s]" says nothing.)

[Hoff]

This forum software is bad at in-line quoting.   I fully expect the format of this message to be corrupted.   Ah, well.

Looks like the sftp command parsing has been broken for a while, around the extended parsing setting.  Ah, well.

-D 99 shows all debug data.   The -v is pretty limited at what it shows.

3.  I noticed that when manually connecting to user@solarisbox, both P1A and T1A get the SFTP2> prompt but:

On P1A, the ascii command responds with the message that it is now in ascii mode (and the ascii -s command shows me the current settings.)

On T1A the ascii command is accepted but produces no output.  The ascii -s command produces no output.

That reeks of a software or client or local configuration difference.

sftp is not FTP.    There's typically no ASCII command.  More than a few clients either lack that, or ignore that for compatibility with folks that think they're using FTP.   In terms of what the ASCII and BINARY do — or are supposed to do, per what FTP does — are to identify the sort of data and the record organization.   BINARY means "don't mess with this".   ASCII means "mess with this, if the server thinks it needs to".

I also tried the getext/setext commands - which work on P1A but not on T1A - yet the installation was the standard PCSI  command to PROD INST TCPIP (and later, of course, the TELNET patch).  Because I've been burned more than once on bad installations, I ran an MD5 checksum on the images - and they match on P1A and T1A.  So I know I have the right versions installed.

That's the images.   Check the [.ssh2] directories involved for differences that might be lurking there.

FWIW, MD5 is not effective against intentional corruptions, and generating intentional collisions is entirely feasible.   SHA1 is looking dicy.   Yes, either should spot accidental or innocuous corruptions.   But if you're at all concerned with security, don't use either of these.  Use SHA2 or better, as available.

5.  As far as the file format for the layer-2 stuff, I have tried that both ways, as direct output from DCL and as output from CONVERT/FDL of the DCL file to a STREAM_LF file.  Hoff, this is why I have layer 1 and layer 2.  Would you prefer that I called these the DCL batch layer and the SFTP2 batch layer?  Because that is why they are separate.  I use the "divide and conquer" method, so the two layers are the way they are because they are handled by different command processors.

In any case, I have noticed that on T1A, when I gave SFTP an unconverted script, it first converted the file to STREAM_LF and THEN it said "Can't read the batchfile." 

There's so much here that I'm having trouble shorting through the details.   If all the files are Stream_LF, then discussions of Layer1, Layer2 and such seem... not entirely relevant.  I don't care what the files are or contain or otherwise — and don't need to know.   Just something akin to that there's a test file with identical contents and format being used for all tests.

6.  I asked the sys admin of the SOLARIS box about whether he had any restrictions on multiple servers using the same keys, but he says that in fact many of our projects do that because this of the DoD regulation on copying audit logs.  The other  production, test, and development boxes at our site send their audit logs via PKI using one key per project.  He is not doing anything to block files from the various network enclaves we have set up.

Another reason not to mention which entity is involved — sharing private keys akin to what's described is analogous to sharing passwords, and sooner or later that gets somebody in trouble when some password or some key needs to be revoked.

7.  Yes, we have HPE support and I will try that if I reach the point of being unable to resolve this transfer problem on my own  or through the forum.  It is just that once I escalate this to HP support, I commit myself to a more time-consuming process because of having to "vette" all of the exhibits I have to send for analysis by getting our network security guys to bless the files as "suitable sanitized" for publication.

Point'm here? 

My next try will be to convert the file to be transferred to STREAM_LF before I put it on the other system, because UNIX-like text files are the same as OpenVMS STREAM_LF format - so ascii vs binary transfers would then make no difference.

Isn't that what the Layer1 and Layer2 stuff did?   Or are we testing with different files?

[Steven Schweda]

> In any case, I have noticed that on T1A, when I gave SFTP an unconverted
> script, it first converted the file to STREAM_LF and THEN it said "Can't
> read the batchfile."

   Note that I showed an example of that happening (give or take a
sloppy quotation):

alp $ sftp "-B" ALP$DKC0:[SMS.itrc]sbat.dat mba
Warning: Converting file alp$dkc0:[sms.itrc]sbat.dat to Stream_LF.
Warning: File alp$dkc0:[sms.itrc]sbat.dat converted
 successfully to Stream_LF.
Error: Could not read the batchfile.

   That is, using a VMS-style file spec for the batch file.  Of course,
when one of us provides an actual command with its actual output, and
the other of us provides a vague description of a command and its
output, it may be difficult to make a detailed comparison between those
commands.


> [...] I ran an MD5 checksum on the images - and they match on P1A and
> T1A.  So I know I have the right versions installed.

   Knowing exactly which "the images" you compared might permit others
to judge the value of the comparison.  Clearly, if you see different
behavior with respect to the "ascii" command, then there must be _some_
difference between the environments on the two systems.  That difference
could lie in the executables or the run-time libraries, or in the
configuration files (system-wide or user-specific), or, probably, in
still other places.

[The_Doc_Man · Accepted solution]

Some quick replies: 

Steven, you asked:     And the output is the same on both systems?

Yes - the version-finder commands report the same versions on both systems.  If I take it a couple of steps farther along, I also get the same version information for the component modules when I do an ANALYZE/IMAGE on the SSH or SFTP images.  (i.e. TCPIP$SSH_SFTP2.EXE and a couple of others.)

Both Hoff and Steven commented on the difference between the ASCII commands.

Hoff:   That reeks of a software or client or local configuration difference.

Steven:   Sounds to me like a suspicious difference.

I agree, but I have learned to be less excitable as I draw nearer to retirement.  I am not down-playing it, but there are some things I have had to learn to accept - for now - and just work around it.

Hoff - I know that MD5 isn't absolutely fool-proof - but as a quick scheme to see if two large files appear superficially to be the same, it is decent.

I'm going to declare this problem closed based on the following:

I have extensively applied CONVERT/FDL to take the scripts that I am building into Stream_LF record format.  The file to be transferred is also converted similarly.  The files are dropped into the same folder from which I will be working so I can use "local" file parsing rather than fiddling with OpenVMS or UNIX file specifications.  SFTP might be dumber than a box of rocks but it can still do a PUT just fine when the default directory contains the desired file.  The PKI handshake works OK and the PUT command no longer encounters - or causes - format differences.  The network security guys are happy with what I give them.

Having said that, if you folks want to just put this on the shelf and forget it, go ahead.  I will separately pursue the problem of differences in the behavior of the ASCII command on the two servers, but so far I have not found any obvious differences in the configuration that would affect what is essentially a presentation-layer command.

FWIW, the things I have tried/checked/verified:

1.  The same versions of TCPIP services are now running on both systems and HP has no later version of TCPIP Services to which I could upgrade.

2.  The O/S patches are to the same level - UPDATE 10, SYS 6, and the other major and minor patch kits.  The only current patch we DON'T have is the TZ patch for the Russian time zones.  We've got everything else that applies, and those patches are on both systems.

3.  The TCPIP$SSH default SSH2_CONFIG. file is identical line-for-line on both systems.  I've done what I call a "blink comparison" with two windows open and viewing the corresponding files, one in each window.  When I switch between the two windows with ALT-TAB, if there differences they would stand out.  But nothing shows up.

4.  These scripts run as SYSTEM.  There is no "local" SSH2_CONFIG. file in the [SYSMGR.SSH2] directory for either system.  There is a copy of the SOLARIS-originated RSA key in that directory for both systems.

5.  No differences exit in the logical names that might apply based on SHOW LOGICAL *SSH* and SHOW LOGICAL *FTP*.

6.  Per the SOLARIS administrator, there are no system-specific sub-configuration segments for the two servers in his SSHD2_CONFIG. equivalent file; nor does he have differences between the sub-nets at other levels in his networking configuration.  He believes that he is treating my P1A and T1A servers identically.

7.  The interface settings on my server hardware are the same on both systems, differing in IP addresses only showing up in the 3rd and 4th octets of our internal IvP4 addresses that run behind our firewalls.  Both OpenVMS systems talk to the SOLARIS server which is on a different subnet than either OpenVMS server.  In both cases, we have defined a local host name with $ TCPIP SET HOST /LOCAL commands.  The connections work by name interactively from both OpenVMS servers to the SOLARIS server.

8.  On the OpenVMS boxes, all files are owned either by SYSTEM or by account TCPIP$SSH as appropriate.  The files have the same permission schemes (i.e. not protected) - [RWED,RWED,RWED,RWED] except for directories, which of course don't include DELETE permission.  No ACLs are present.  SYSTEM, of course, runs with BYPASS, but the audit logs tell me that the scripts are not accessing the files with privilege - they are using the permission masks.

9.  Not that I think it would have this particular effect, but neither disk quotas nor disk capacity are issues here.  Nor are SYSGEN parameters.  These systems were configured so that if either one failed, the other one could take over full operation almost seamlessly.  It's not like we saturate 2 x Intel 9300-4c (8 cores total) with something that is almost continuously I/O-bound, so we purposely made them structurally as close to each other as possible in case we had to temporarily host two environments on the same box.  Therefore, finding a difference at that level between the two environments is as much a surprise to me as it was to those of you who commented on it.

I guess I was hoping that someone would see these symptoms and tell me where to look for discrepancies, but it looks like this one is a bit more opaque than some of the things I've encountered in the OpenVMS world.

In any case, contributors, thanks for the feedback.