topic Re: issue with sftp in Operating System - OpenVMS

issue with sftp

Kevin Carter_3 — Mon, 06 Oct 2008 16:25:26 GMT

A user is sending a file from a Unix platform with sftp. The file transfer uses public key authentication to log into an Alpha Station running OpenVMS V8.2. The Alpha Station is running HP TCP/IP V5.6 -ECO 2.

The file(s) transfer successfully for version one of the inbound file. When the Unix system sends another file with the same name; the VMS box does not receive a file. The logfile on the Unix system shows that the file was successfully transfered, but VMS does not create a new version of the file.

Doing a $ dir/full filename shows that 'No version limit' is set.

Any suggestions?

Kevin

Re: issue with sftp

Hoff — Mon, 06 Oct 2008 16:50:34 GMT

Several questions...

To confirm: when the second transfer completes, you find the old version on the system, and not the new? Or does the file now contain the new bits and not the old?

To confirm: when the second transfer specifies a second and different filename from any existing file in the target directory, you get the second file transfered correctly?

To confirm: the default directory is or is not a searchlist? That is, the operation is or is not involving the SYSTEM manager username or the SYS$SYSROOT: directories?

To confirm: the target LOGIN.COM and SYLOGIN.COM login procedures (for the purposes of testing the transfer) have been examined for and sanitized of errors and are otherwise known not interfering with the transfer.

To confirm: the enveloping directory itself does not also have a version limit set? What does the directory indicate for its version limit?

Also please post the identity of the Unix box sftp client and version, and please post the connection sequence including the put command used to transfer the file(s).

Also enable verification using sftp -vvv and try it again, and also enable the sftp daemon debugging per the manual. (IIRC, there's a debug logical name listed in the ssh manual for ssh (TCPIP$SSH_SERVER_DEBUG); haven't checked to see if there's one around for the sftp daemon, but I'd expect there's a way to turn on some logging either via logical name or via some sftp daemon command procedure or otherwise.)

Re: issue with sftp

Richard Whalen — Mon, 06 Oct 2008 16:51:54 GMT

Since SFTP was designed around the Unix model of version-less files, it's possible that second transfer either deleted the original file before transferring the new file, or truncated the original file and rewrote it with the new data.

Re: issue with sftp

Jan van den Ende — Mon, 06 Oct 2008 17:44:13 GMT

Kevin,

I know little about this stuff, but, with the quirks of TCPIP involved, I'd say there is one other thing to check (besides the points Hoff & Richard already made): maybe the second send is APPENDED to the first?

fwiw

Proost.

Have one on me.

jpe

Re: issue with sftp

Kevin Carter_3 — Mon, 06 Oct 2008 18:29:22 GMT

To confirm: when the second transfer completes, you find the old version on the system, and not the new?

>> yes the old version of the file is on the system

Or does the file now contain the new bits and not the old?

>> I will have to test this and get back to you, might take a day

To confirm: when the second transfer specifies a second and different filename from any existing file in the target directory, you get the second file transferred correctly

>> Yes, the file transfer was originally set up to transfer a file with an extension of the date concatenated and this worked fine; for example simdv6b.20081006 - todays file simdv6b.20081005 - yesterdays file. The problem started when I had the extension changed to simdv6b.txt for the production environment.

To confirm: the default directory is or is not a searchlist? That is, the operation is or is not involving the SYSTEM manager username or the SYS$SYSROOT: directories?

>> No the default directory is not a searchlist.

To confirm: the target LOGIN.COM and SYLOGIN.COM login procedures (for the purposes of testing the transfer) have been examined for and sanitized of errors and are otherwise known not interfering with the transfer.

>> the login.com contains only $ exit and the syslogin.com does not appear to be causing the issue. I can attach a copy if you like?

To confirm: the enveloping directory itself does not also have a version limit set? What does the directory indicate for its version limit?

>> a dir/full on the parent / only directory above the directory shows 'No default version limit'

Also please post the identity of the Unix box sftp client and version, and please post the connection sequence including the put command used to transfer the file(s).

>> will gather this info and post it, I do know that the transfer is being done in a script on the Unix box

Also enable verification using sftp -vvv and try it again, and also enable the sftp daemon debugging per the manual.

>> tried this, the log shows successfull transfer of the file - you may be correct that the second file is being appended to the first - again will know more after testing

(IIRC, there's a debug logical name listed in the ssh manual for ssh (TCPIP$SSH_SERVER_DEBUG); haven't checked to see if there's one around for the sftp daemon, but I'd expect there's a way to turn on some logging either via logical name or via some sftp daemon command procedure or otherwise.)

>> turned on verbose mode in ssh server - nothing unusual jumps out when compared to other logfiles for file transfer

Re: issue with sftp

Steven Schweda — Mon, 06 Oct 2008 18:34:45 GMT

Hmmm. Around here:

alp $ tcpip show version

HP TCP/IP Services for OpenVMS Alpha Version V5.4 - ECO 7
on a COMPAQ Professional Workstation XP1000 running OpenVMS V7.3-2

alp $ sftp "-V"
alp$dka0:[sys0.syscommon.][sysexe]tcpip$ssh_sftp2.exe: SSH Secure Shell OpenVMS
(V5.5) 3.2.0 on COMPAQ Professional Workstation - VMS V7.3-2

And after this:

alp $ show defa
ALP$DKA0:[SMS.test]

alp $ sftp sms@alp
sftp> put fred.txt
fred.txt | 798kB | 99.7 kB/s | TOC: 00:00:08 | 100%
sftp> put fred.txt
fred.txt | 798kB | 99.7 kB/s | TOC: 00:00:08 | 100%
sftp> quit

I see these:

alp $ dg [-]fred.txt

Directory ALP$DKA0:[SMS]

FRED.TXT;2 1596 6-OCT-2008 14:25:14.43 (RWED,RWED,RE,)
FRED.TXT;1 1596 6-OCT-2008 14:25:00.91 (RWED,RWED,RE,)

Total of 2 files, 3192 blocks.

I could fire up a UNIX system and try it from
there, but it seems unlikely that using a
UNIX SFTP client would make things worse.

Safe to assume that the UNIX user is not
specifying a VMS version number on the
destination? (Why should I need to assume
anything? Actual transcript(s)?)

Re: issue with sftp

Richard Whalen — Mon, 06 Oct 2008 19:43:57 GMT

The SSH File Transfer Protocol is really a file access protocol and contains operations such as open file, get/set file information, read/write data, close file, delete file. The SFTP user program uses these operations to transfer the data. SFTP user programs that run on Unix often do operations based upon Unix single version file system: if the file named already exists, then it will delete it or open and truncate it before transferring the new data.

Since the SSH File Transfer Protocol also allows for arbitrary extensions to the protocol there are some implementations that will request that an MD5 checksum be performed (over some portion of the file) and then only transfer the file if the MD5 checksum for the local file is different.

Re: issue with sftp

Hoff — Mon, 06 Oct 2008 19:49:07 GMT

Please show your work; post the verbose diagnostics and the commands used. The " nothing unusual jumps out" phrase can be a particularly deadly assumption when debugging stuff. That's arguably the definition of a good bug, after all. And another set of eyes can sometimes help spot those critters.

Re: issue with sftp

Kevin Carter_3 — Mon, 06 Oct 2008 21:16:16 GMT

Looks like I have some homework to do, thanks for the input so far. I will do as suggested and 'show your work'. This will probably take 24 hours as the person responsible for the sftp from the Unix side is not available today.

Kevin

Re: issue with sftp

Kevin Carter_3 — Mon, 06 Oct 2008 21:43:48 GMT

Mystery solved. The Unix sftp process overwrites the file on the VMS system. But it does not modify the date / time stamp on the file. So if a file simdv6b.dat is created on 5-oct-2008 and another file with the same name is sent on 6-oct-2008 I would not know unless the size of the file changed.

Re: issue with sftp

Kevin Carter_3 — Mon, 06 Oct 2008 21:45:01 GMT

By using trouble shooting suggestions from the discussion - was able to determine the cause of the problem.

Re: issue with sftp

Hoff — Tue, 07 Oct 2008 00:06:08 GMT

If you have a support contract, log a report against TCP/IP Services; you've found a bug somewhere. If this is a different file, it should have a different timestamp set, and it's the TCP/IP sftp server that's apparently skipping a step. (Auditors particularly like this class of bug, but that's another discussion.)

Re: issue with sftp

Steven Schweda — Tue, 07 Oct 2008 01:37:15 GMT

Using a UNIX (HP-UX 11.11) SSH client, with
the same VMS server as before, I got the same
result as shown before (two file versions).
Using a newer VMS system:

ALP2 $ tcpip show version

HP TCP/IP Services for OpenVMS Alpha Version V5.6 - ECO 2
on a COMPAQ Professional Workstation XP1000 running OpenVMS V8.3

I got the same result again, from both a VMS
client and an HP-UX client.

Whatever your problem is, I don't see it on
my systems.

> Mystery solved.

I doubt it.

> The Unix sftp process overwrites the file
> on the VMS system. [...]

The SFTP client process does _not_ write
anything on the SFTP server system. It
_asks_ the SFTP server to do things, and the
server process does the work. I know of no
way for the client to tell the server to do
anything in particular with VMS file
versions. Perhaps there's some way to
configure the SFTP _server_ to do something
exotic, but on my systems (with what must be
pretty close to the default configurations),
things work as I'd expect them to work, not
the way you say that they work for you.