Running a sftp batch file from Windows/Cygwin to a remote OpenVMS System

[Wayne Johnson_3]

I have a process where I build a set of programs on several platforms using a Windows Server 2008 and Cygwin as a build master, and submitting jobs to various remote platforms via tools like scp.

I'm trying to add OpenVMS into our build process. I can manually build the programs by logging in with Putty, cd to the proper directory and then run the command:

make "VMS=8.3"

I have ssh working on the OpenVMS system and (finally) can log in without a password. To simulate Unix I have set up several item in login.com:

CD    == "@" + HOME - "]" + ".COMMANDS]CD.COM;"

MAKE == "$DKA100:[000000.KITS]GMAKE_IA64.EXE"

I first tried to use ssh to do this with:

ssh wajohnso@michae 'cd build/B_OpenVMSBuild/eadev; make "VMS=8."'

But apparently OpenVMS does not like the ;.  I tried the PIPE command, but it still has issues.

I tried the same with sftp batch command:

D:\dev\B_OpenVMSBuild>sftp -b script wajohnso@michae

 Welcome to HP OpenVMS Industry Standard 64 Operating System, Version V8.4
sftp> cd build
sftp> cd B_OpenVMSBuild
sftp> cd eadev
sftp> make "VMS=8.3"
Invalid command.

What sort of environment is in effect for ssh and sftp?  I assume it does not run login.com.  Any way to inject the login.com DCL in the sftp batch script? 

Anyone have any other suggestions?

[H.Becker]

>>> What sort of environment is in effect for ssh and sftp? I assume it does not run login.com. Any way to inject the login.com DCL in the sftp batch script?

You can easily find out with something like:

ssh wajohnso@michae 'write sys$output f$mode(),f$fao("!/")'

f$mode() will very likely show OTHER (the f$fao() makes sure the local prompt doesn't overwrite that output).

I guess, that in your (or the sytem) login command procedure there is a check for OTHER and then symbols like make aren't set. I also guess that you can fix that in your login.com, by inserting the shown assignment.

Next you want to do is a

ssh wajohnso@michae 'write sys$output make,f$fao("!/")'

to be sure it works as expected (that is, you shouldn't get a "%DCL-W-UNDSYM, undefined symbol ..." )

I guess you can do the same - or something similar for sftp.

[Steven Schweda]

> Anyone have any other suggestions?

   A useful problem report might be a good start.  For example:

      tcpip show version
      ssh "-V"

> [...] I tried the PIPE command, but it still has issues.

   Are these "issues" anything like _problems_?  As usual, showing
actual commands with their actual output can be more helpful then vague
descriptions or interpretations.

   PIPE seems to work for me, by the way.

> sftp> make "VMS=8.3"
> Invalid command.

   Do you think that SFTP is intended to provide a general remote
command mechanism?  I'd stick with SSH, unless you actually want SFTP
for transferring files.

[Hoff]

So basically, you're looking for help with Cygwin on Windows Server 2008, and you've asked a bunch of folks that use OpenVMS just because that platform is more or less located on the same network as the Cygwin box.  OK.  We can work with that, and — as OpenVMS users — we are of course experts at operating Cygwin on Windows Server 2008, or so we'll tell you.  ;-)

Slightly more seriously — we are experts, after all — see if there's a version of GNU expect around for Cygwin, and use that to script the remote access.   (Yep, looks like expect has been around for a while, some more and yet more.)

Now if you're serious about implementing build automation across heterogeneous systems and remote coordination and control beyond what's typical with ssh and expect and similar hackery — error handling for hand-scripted stuff gets hairy — then I'd likely concentrate on SaltStack, possibly Puppet or maybe distcc some other tool that handles these and other tasks, and try to avoid rolling my own solution here.  

I don't know if these or similar build automation tools have been ported to VMS, though SaltStack is Python, and there's a Python port for VMS available. 

There are also some existing open source projects around that manage distributed builds for themselves, and some of which do deal with OpenVMS, and what they're using is worth a look.  (Why re-invent the wheel, after all?)

Now if I were rolling my own solution and reinventing the wheel here, I'd probably use HTTP or HTTPS into a web server here (Apache or maybe WASD on OpenVMS), and CGI.  That gets me out of dealing with the odd-ball states inherent in trying to use expect or, um, trying to not use expect to drive an inherently-not-stateful serial line connection, and toward something that's more predictable and maintainable.  curl or wget commands can be used to trigger the remote scripts which then trigger the VMS makefiles or procedures or whatever, and that then gather the responses, as can various other tools.  Might go as far as using AJAX and JSON, for full buzz-phrase compliance.

It wouldn't surprise me that there are extensions or plug-ins or add-ons to Visual Studio or Eclipse or whatever you're using for your main development that can trigger remote, distributed builds across heterogeneous systems.  There are definitely distributed build tools around.  Whether one or more of those can be triggered from Cygwin and can access OpenVMS (somehow), I don't know, but I'd assume so.  Or use expect, which'll get you to your immediate goal, and with the various problems contending with variant returns and build errors that are to be, um, expected. 

The spell-checker here amusingly flags OpenVMS as a typo, but I digress. :-)

[Wayne Johnson_3]

Thanks for the response.  Sorry for the lack of information, I've been trying so many different variations I don't really have the info captured.  For the record, here's the system configuration:

$  tcpip show version

  HP TCP/IP Services for OpenVMS Industry Standard 64 Version V5.7 - ECO 2
  on an HP rx2620  (1.60GHz/3.0MB) running OpenVMS V8.4

$ ssh -V
debug(10-SEP-2014 13:56:10.90): Ssh2/SSH2.C:1896: CRTL version (SYS$SHARE:DECC$S
HR.EXE ident) isELF
  debug(10-SEP-2014 13:56:10.92): SshAppCommon/SSHAPPCOMMON.C:313: Allocating gl
obal SshRegex context.
debug(10-SEP-2014 13:56:10.95): SshConfig/SSHCONFIG.C:3461: Metaconfig parsing s
topped at line 4.
debug(10-SEP-2014 13:56:10.95): SshConfig/SSHCONFIG.C:887: Setting variable 'Ver
boseMode' to 'FALSE'.
debug(10-SEP-2014 13:56:10.96): SshConfig/SSHCONFIG.C:3369: Unable to open ssh2/
ssh2_config
  warning: You didn't specify a host name.
Type michae$dka0:[sys0.syscommon.][sysexe]tcpip$ssh_ssh2.exe -h for help.

After a little more research, I realized that the batch mode on sftp does not provide for running remote commands.  I had used a similar facility on z/OS and made a bit of a jump there.

I did manage to get the ssh and PIPE command to work after a bit more work.  For future reference, the commands look something like this:

    ssh wajohnso@michae 'PIPE cd build.$(BRANCH).eadev && make "VMS=8.3" && purge [...]'

Now the only issue I have left is coping back down the resulting programs.  When I use scp I get a funky error:

D:\dev\B_OpenVMSBuild>scp wajohnso@michae:"[.build.B_OpenVMSBuild.eadev.bin]*.*;*" .

 Welcome to HP OpenVMS Industry Standard 64 Operating System, Version V8.4
protocol error: unexpected <newline>

I've seen references to this error caused by a login script (.bashrc) outputing text when scp wasn't expecting it, much like the Welcome to HP... banner.  Any suggestions on how to remove or bypass that?

[Wayne Johnson_3]

Actually, I was asking the OpenVMS experts how to get Cygwin ssh to talk to OpenVMS ssh.

[Hoff]

If you do persist in this attempt to script a serial line (via ssh, in this case), then use GNU expect or similar tool.  

That'll allow you to script what happens over the connection.

That'll also have all the problems that scripting an indeterminate command sequence will encounter.

The path you're headed down (with expect or with a pipe) is gnarly, and there are already distributed build tools available that can deal with some of these issues.   

If you want to invest the time and effort in this project while improving the behavior across other environments you're working with, get one of the distributed build tools working in your environment, and get out of the business of maintaining your own bespoke environment where you can.

[Steven Schweda]

> ssh "-V"

> $ ssh -V
> [...]

   Those quotation marks were significant.  (HP's SSH program does not
naturally preserve command-line case.)

> D:\dev\B_OpenVMSBuild>scp wajohnso@michae:"[.build.B_OpenVMSBuild.eadev.bin]*.*;*" .

   I'm not a big scp user (especially with VMS), so I know nothing, but,
around here ("HP TCP/IP Services for OpenVMS Alpha Version V5.6 - ECO 5
on a COMPAQ Professional Workstation XP1000 running OpenVMS V8.3"), SCP
seems to have trouble with VMS-style file specs, and wildcards in
general.  You might get better (different?) results if you specified a

source like, say:

      "build/B_OpenVMSBuild/eadev/bin/*.*;*"

   Even then, I'd wonder what you expect to get from the ";*" part of

that on a non-VMS system. Also, depending on what's in your "[.bin]"
directory, I'd worry about losing valuable file attributes when copying
"binary" (non-text) files to a non-VMS system.

   If I wanted to move multiple files around, I'd probably create a
"tar" or Zip archive, and then move that.  SCP here does seem able to
copy one file to a simple destination.

> [...] the Welcome to HP... banner. Any suggestions on how to remove
> or bypass that?

   These aren't the droids you're looking for.  I wouldn't worry about
it.  Mine's uglier than yours, and it's not a problem:

alp $ scp alp-l:login.com .

@ SYS$MANAGER:ANNOUNCE.TXT
login.com | 9.9kB | 9.9 kB/s | TOC: 00:00:01 | 100%

   As usual, it can be more helpful to describe the actual task to be
performed, rather than asking how to solve particular problems in what
may be a poor solution to the actual task.

Your post has been changed because invalid HTML was found in the message body. The invalid HTML has been removed. Please review the message and submit the message when you are satisfied.

   I was satisfied before.  I don't know (or much care)  what was "removed".

[Wayne Johnson_3]

I understand that there are issues with this.  The system had worked fairly well for 12 years now. 

We are working on implementing Jenkins to manage the build environment, but with 19 different platforms and 3 million lines of code, the 4 small executables we will need to build on OpenVMS is unfortunatly not top on our priority list.

I ended up using sftp to download the files and it all appears to be working now (fingers crossed).

Thanks to everyone for their help.