topic Re: Using SFTP in a batch job in Operating System - OpenVMS

Using SFTP in a batch job

David Froble — Thu, 14 Apr 2011 20:28:25 GMT

I'm trying to use SFTP in a batch command file.

$ SFTP "-D 99" -o "batchmode no" fsrbi@209.172.179.22
fayave8A
ls /TEST/TO/CURRENT
exit

Looks simple enough to me. Doesn't work. So I set the SFTP debug mode to 99, and got lots of diagnostics. It appears that even though I specified the batchmode to be "no", and I had also tried it as "yes", SSH is determining that it is a batch mode, and is therefore surpressing the password prompt.

Short excerpt from the log file:

debug: server offers auth methods 'gssapi-with-mic,password,publickey,keyboard-i
nteractive'.

debug: Ssh2AuthPasswdClient/AUTHC-PASSWD.C:261: In Batchmode, so we're not askin
g the user for password.

debug: Ssh2AuthClient/SSHAUTHC.C:385: Method 'password' disabled.

debug: server offers auth methods 'gssapi-with-mic,password,publickey,keyboard-i
nteractive'.

debug: Ssh2Common/SSHCOMMON.C:185: DISCONNECT received: No further authenticatio
n methods available.

Re: Using SFTP in a batch job

Hoff — Thu, 14 Apr 2011 21:02:05 GMT

You will usually want to have batch-mode enabled (via the "-B" switch), and to set up the matching public certificate on the target box.

The sftp and scp commands aren't (per my experience and per the documentation for at least some of the TCP/IP Services releases) particularly stable around trying to embed a password, if they even permit the password at all. (I know TCP/IP Services V5.5 was pretty wonky here, though there might be (is?) a patch for that.)

The certificate path works rather better here, in general.

Using a straight scp command (with a public key on the far end) might be easier than having that sequence of input, too; you can likely specify the target file path in a quoted string. With the certificates, that likely removes the need for that sequence of embedded commands, too.

Re: Using SFTP in a batch job

Steven Schweda — Thu, 14 Apr 2011 21:21:56 GMT

> fayave8A

That's supposed to be the password?

In general, SSH, SCP, SFTP, and friends were
designed not to encourage users to store
passwords in files. (And you can probably
replace "encourage" with "permit". "S"
stands for "Secure", not "Sloppy".)

> [...] SSH is determining that it is a batch
> mode, [...]

More generally, that it has no controlling
terminal, therefore no (real) user
interaction is possible.

> [...],keyboard-interactive'.

They don't call it "keyboard-interactive" for
no reason.

> The certificate path works rather better
> here, in general.

Well, public keys, but yes. If you could get
this to work as you apparently wish to, then
I'd expect someone to fix it eventually, so
that it wouldn't work.

Re: Using SFTP in a batch job

Richard Whalen — Fri, 15 Apr 2011 11:38:54 GMT

The two above answers are spot on - SFTP won't do password or keyboard-interactive authentication when running as a batch job. It doesn't matter whether or not you tell it to use batch mode as it will figure out that it doesn't have a controlling terminal.

So, you'll have to set up for public key or host based authentication. See http://h71000.www7.hp.com/doc/83final/ba548_90007/ch03s08.html or http://h71000.www7.hp.com/doc/83final/ba548_90007/ch03s07.html
I recommend public key. Do NOT use a pass-phrase when creating your public key - this is the same as supplying a password and won't help for batch mode. (Though I have heard of some users being able to do strange things with pipes, I've always told them "don't tell me if it breaks because I won't try to fix it".)

When you put your public key on the remote system you need to know the specifics of how the implementation of SSH on that system uses to find the public key and what format it is in. The directions for TCP/IP Services work for TCP/IP Services (and MultiNet and TCPware), but won't work for OpenSSH based implementations, which are quite common. Also TCP/IP Services generates keys that follow the RFC and OpenSSH wants a different format. The SSHKEYGEN program on OpenSSH systems can convert between the two formats.

Re: Using SFTP in a batch job

Steven Schweda — Fri, 15 Apr 2011 12:40:34 GMT

> When you put your public key on the remote
> system you need to know [...]

A Forum search for keywords like, say,
ssh key format
should find some relevant old threads. If
you do need help, then actual info about the
systems involved (TCPIP SHOW VERSION,
'SSH "-V"', "uname -a", "ssh -V", ...) might
be useful.

> [...] The SSHKEYGEN program on OpenSSH
> systems [...]

Normally known as "ssh-keygen". "man
ssh-keygen" often works on UNIX(-like)
systems.

> fayave8A

Don't forget to change this password, if that
was an actual password. (We also have a user
name and an IP address.)

Re: Using SFTP in a batch job

David Froble — Fri, 15 Apr 2011 16:12:22 GMT

Yes, that's the IP, user, and password. Just testing, and password will be changed, so I didn't mind posting it if it would help anyone get the procedure working. If anybody wants to hack some lawn mower parts, be my guest.

What I'm reading is that what I need to do just won't work. The trading partner isn't playing with a full deck, and apparently doesn't care if my customer has to manually perform the transfers every day. That just isn't acceptable.

It appears that there will not be any cooperation with the trading partner as far as setting up certificates. Already tried that, but I guess I can grovel some more. Don't think it will help.

So, let me reframe my inquiry. Got a trading partner, wants to use SFTP, will not do more than issue IP, user, and password. Any ideas on how to automate the transfer of some files?

Re: Using SFTP in a batch job

Steven Schweda — Fri, 15 Apr 2011 18:01:39 GMT

> [...] What I'm reading is that what I need
> to do just won't work. [...]

It certainly won't be as easy as one might
hope. It might be possible to use Kermit
scripting to fake an interactive (-looking)
session with the SFTP client

> Any ideas on how to automate the transfer
> of some files?

It's not entirely clear to me which client
needs to talk to whose server, and I've never
tried this, but if you can puzzle out the
style of the server's SSH software, then it
might be possible to construct your own key
files, and then use SFTP manually, once, to
install your public key file (and any other
related stuff) on that server. You'd need to
get the file ownership (easy?) and
permissions (perhaps less easy) right on the
server, and you would need write access to
your home directory there. It's all easier
if the server owner cooperates, or if you
have full SSH access instead of only SFTP,
but, if you use public keys, then you really
do want to generate your own key files, so
that your private key remains your private
key.

Re: Using SFTP in a batch job

Hoff — Fri, 15 Apr 2011 18:01:44 GMT

If they're connecting into your servers, then they'll have to ship you a public key, because that's just the way sftp and scp works here.

If they're not technically inclined, then generate and give them a private key and tell them that it's their secret password (and it is), and tell them where to locate it in their file structure (you'll need to know the type of system and the ssh/sftp/scp tool), and drop the matching public key onto your own server. (I'd probably then delete the private key from the local servers, but that's your call.)

If you're connecting into them, then drop your own public key into the login directory and add the reference into whatever passes for AUTHORIZATION on the remote box, and Bob's your uncle.

Re: Using SFTP in a batch job

Steven Schweda — Fri, 15 Apr 2011 23:32:30 GMT

> [...] If anybody wants to hack some lawn
> mower parts, be my guest. [...]

You could be in luck.

> [...] Remote version: SSH-2.0-5.3.6.18 SSH Tectia Server

Interestingly, the Tectia SSH software seems
to use the same SSH2-style key file format as
the TCPIP SSH software. Better yet, Tectia
seems still to offer HTML documentation for
their products (unlike some vendors I could
name):

http://www.tectia.com/manuals/server-admin/53/index.html
http://www.tectia.com/manuals/server-admin/53/userauth-pk-authfile.html

According to which, all one should need to do
is drop a suitable public key file into the
".ssh2" directory on the server. Which I
did. Then, any old (Stream_LF, ideally, but
fear not) SFTP batch file can be used without
user interaction. For example:

alp $ type ls-lR.bat
pwd
ls -lR
quit

alp $ sftp "-B" ls-lR.bat fsrbi@209.172.179.22
sftp> pwd
/
sftp> ls -lR
.
TO
TO/ARCHIVE
TO/CURRENT
FROM
FROM/ARCHIVE
FROM/CURRENT
TEST
TEST/TO
TEST/TO/ARCHIVE
TEST/TO/ARCHIVE/NRB02081351084.TXT
TEST/TO/ARCHIVE/NRB02161537200.TXT
TEST/TO/ARCHIVE/NRB11231731097.TXT
TEST/TO/ARCHIVE/NRB02161314095.TXT
TEST/TO/CURRENT
TEST/FROM
TEST/FROM/ARCHIVE
TEST/FROM/CURRENT
.ssh2
.ssh2/authorized_keys
.ssh2/authorized_keys/sms_npp_id_dsa_1024_a.pub
RBI.TXT
sftp> quit

alp $

(Look! It's Uncle Bob!)

Feel free to delete my public key file
whenever it's convenient. (What do I care?
It's a _public_ key file.)

Fine print:

You do need to generate a key pair, if you
don't already have one lying around.

I didn't actually try this in a
non-interactive mode, but if no one asks for
or expects a password, then what could go
wrong?

Re: Using SFTP in a batch job

Steven Schweda — Sat, 16 Apr 2011 02:37:00 GMT

> [...] what could go wrong?

Apparently nothing. (Nothing obvious,
anyway.) It worked in a batch job, too, as
expected.

alp $ type ls-lR_bat.com
$ proc_dev_dir = f$parse( "A.;", f$environment( "procedure"))- "A.;"
$ set default 'proc_dev_dir'
$ sftp "-B" ls-lR.bat fsrbi@209.172.179.22

alp $ sub ls-lR_bat.com /log = ALP$DKC0:[SMS.ITRC.froble]ls-lR_bat.log
Job ls-lR_bat (queue SYS$BATCH_ALP, entry 763) started on SYS$BATCH_ALP

Job ls-lR_bat (queue SYS$BATCH_ALP, entry 763) completed

alp $ type ls-lR_bat.log
$ Set NoOn
$ VERIFY = F$VERIFY(F$TRNLNM("SYLOGIN_VERIFY"))
sftp> pwd
/
sftp> ls -lR
.
TO
TO/ARCHIVE
TO/CURRENT
FROM
FROM/ARCHIVE
FROM/CURRENT
TEST
TEST/TO
TEST/TO/ARCHIVE
TEST/TO/ARCHIVE/NRB02081351084.TXT
TEST/TO/ARCHIVE/NRB02161537200.TXT
TEST/TO/ARCHIVE/NRB11231731097.TXT
TEST/TO/ARCHIVE/NRB02161314095.TXT
TEST/TO/CURRENT
TEST/FROM
TEST/FROM/ARCHIVE
TEST/FROM/CURRENT
.ssh2
.ssh2/authorized_keys
.ssh2/authorized_keys/sms_npp_id_dsa_1024_a.pub
RBI.TXT
sftp> quit
SMS job terminated at 15-APR-2011 22:19:38.36

Accounting information:
Buffered I/O count: 1186 Peak working set size: 11760
Direct I/O count: 110 Peak virtual size: 186560
Page faults: 930 Mounted volumes: 0
Charged CPU time: 0 00:00:00.22 Elapsed time: 0 00:00:10.75
alp $

(The batch log file suffers from extra CR
characters on line endings of the directory
listing sent from the server, but I've seen
worse pollution.)

Re: Using SFTP in a batch job

David Froble — Mon, 25 Apr 2011 22:43:44 GMT

Some specifics:

My VMS system is the client, and I do not know the OS and such of the server.

I've produced private and public keys, and moved a copy of the public key to the remote server, using the directories Steven had already created in his testing.

I can now perform a SFTP connection without using password, and I can perform a SFTP connection using a batch file.

MANY THANKS TO STEVEN SCHWEDA FOR THE HELP AND EXAMPLES.

Working from my own user account, with it's default directory on the system disk, I get the following when executing the command file in a batch job.

-----------
C.COM
$ set verify
$
$ SFTP "-B" C.BAT fsrbi@209.172.179.22
------------
C.BAT
pwd
ls
ls "-R"
quit
------------
C.LOG
$ Set NoOn
$
$ VERIFY = F$VERIFY(F$TRNLNM("SYLOGIN_VERIFY"))
Performing SYS$MANAGER:SYLOGIN.COM ...
Performing [DFE]LOGIN.COM ....
$
$ SFTP "-B" C.BAT fsrbi@209.172.179.22
sftp> pwd
/
sftp> ls
Warning: Failed to set terminal mode to PASTHRU in sftp.
sftp> ls "-R"
Warning: Failed to set terminal mode to PASTHRU in sftp.
sftp> quit
DFE job terminated at 25-APR-2011 19:31:58.73
-----------

What I'm using:

BRYCE> tcpip show version

HP TCP/IP Services for OpenVMS Alpha Version V5.6
on an AlphaStation 200 4/233 running OpenVMS V8.3

BRYCE> sftp -v
Sftp2/SFTP2.C:4543: CRTL version (SYS$SHARE:DECC$SHARE ident) is: V8.3-00

SshFileCopy/SSHFILECOPY.C:1080: Making local connection.
Ssh2SftpServer/SSHFILEXFERS.C:1754: Received SSH_FXP_INIT
Ssh2SftpServer/SSHFILEXFERS.C:1799: version is 3
SshFileCopy/SSHFILECOPY.C:1019: Connection to local, ready to serve requests.
Sftp2/SFTP2.C:728: Connection ready.
SshReadLine/SSHREADLINE.C:3652: Initializing ReadLine...

So, first question, am I running some old stuff that doesn't work so well, and I need to get the latest TCPIP software?

Then the second issue. When I attempt to use the exact same files from a user account that has it's default directory on a disk other than the system disk, I can not perform an online SFTP, and of course not a batch job SFTP either.

$ SFTP "-B" C.BAT fsrbi@209.172.179.22
%SYSTEM-F-ACCVIO, access violation, reason mask=00, virtual address=000000000000
0000, PC=00000000000EA240, PS=0000001B

Improperly handled condition, image exit forced.
Signal arguments: Number = 0000000000000005
Name = 000000000000000C
0000000000000000
0000000000000000
00000000000EA240
000000000000001B

Register dump:
R0 = 0000000000000000 R1 = 0000000000000014 R2 = 0000000000015E80
R3 = 0000000000000000 R4 = 0000000000874750 R5 = 000000000005AA40
R6 = 0000000000000001 R7 = 000000000005AA40 R8 = 0000000000000001
R9 = 0000000000000000 R10 = 0000000000000001 R11 = 0000000000000000
R12 = 0000000000000001 R13 = 0000000000507840 R14 = 0000000000877B80
R15 = 0000000000877D28 R16 = 0000000000000000 R17 = 00000000000615E0
R18 = 0000000000060C70 R19 = 0000000000000CF4 R20 = 0000000000015C78
R21 = 0000000000000000 R22 = 0000000000877E6A R23 = 0000000000000019
R24 = 0000000000021999 R25 = 0000000000000001 R26 = 00000000000FD918
R27 = 0000000000013EA0 R28 = 0000000000000000 R29 = 000000007AE56D50
Warning: child process (/sys$system/tcpip$ssh_ssh2) exited with code -10.B

%TCPIP-E-SSH_FC_ERROR, error in ssh file transfer operation

I've got no idea what is causing the access violation. I'm really hoping that SFTP can be used from a user account with it's default directory not on the system disk.

Re: Using SFTP in a batch job

Steven Schweda — Tue, 26 Apr 2011 01:04:17 GMT

> [...] I do not know the OS and such of the
> server.

It's in the "-v" diagnostic messages. (I
didn't make that stuff up.)

> [...] the directories Steven had already
> created in his testing.

The directories were already there. All I
added was the public key file.

> [...] the command file in a batch job.

But the SFTP batch file worked well enough
interactively? (It's a start.) As you can
see, I had no trouble both ways.

> BRYCE> tcpip show version
>
> HP TCP/IP Services for OpenVMS Alpha Version V5.6
> on an AlphaStation 200 4/233 running OpenVMS V8.3

> [...] am I running some old stuff

Yes. (Hardware, too, it seems.)

> that doesn't work so well, and I need to
> get the latest TCPIP software?

Probably. (Have you ever seen a bug-free
TCPIP kit?) Around here, for example:

ALP $ tcpip show version

HP TCP/IP Services for OpenVMS Alpha Version V5.6 - ECO 5
on a COMPAQ Professional Workstation XP1000 running OpenVMS V8.3

> BRYCE> sftp -v
> [...]

I did suggest 'SSH "-V"', didn't I?

ALP $ ssh "-V"
alp$dkc0:[sys0.syscommon.][sysexe]tcpip$ssh_ssh2.exe: SSH Secure Shell OpenVMS (
V5.5) 3.2.0 on COMPAQ Professional Workstation - VMS V8.3

With hobbyist patch access having been
extinguished, I can't say what's current, but
five (or more) ECOs behind is generally not a
happy place to be.

> I've got no idea what is causing the access
> violation.

Usually, it's bad software, and yours is old
(so probably bad).

> [...] I'm really hoping that SFTP can be
> used from a user account with it's default
> directory not on the system disk.

My examples showing the prompt "alp $ " were
run from my personal account. (My SYS$LOGIN
is on the system disk, but wouldn't expect
that to matter.)

And that's "its default directory".