issue with sftp

[Kevin Carter_3]

A user is sending a file from a Unix platform with sftp. The file transfer uses public key authentication to log into an Alpha Station running OpenVMS V8.2. The Alpha Station is running HP TCP/IP V5.6 -ECO 2.

The file(s) transfer successfully for version one of the inbound file. When the Unix system sends another file with the same name; the VMS box does not receive a file. The logfile on the Unix system shows that the file was successfully transfered, but VMS does not create a new version of the file.

Doing a $ dir/full filename shows that 'No version limit' is set.

Any suggestions?

Kevin

[Hoff · Accepted solution]

Several questions...

To confirm: when the second transfer completes, you find the old version on the system, and not the new? Or does the file now contain the new bits and not the old?

To confirm: when the second transfer specifies a second and different filename from any existing file in the target directory, you get the second file transfered correctly?

To confirm: the default directory is or is not a searchlist? That is, the operation is or is not involving the SYSTEM manager username or the SYS$SYSROOT: directories?

To confirm: the target LOGIN.COM and SYLOGIN.COM login procedures (for the purposes of testing the transfer) have been examined for and sanitized of errors and are otherwise known not interfering with the transfer.

To confirm: the enveloping directory itself does not also have a version limit set? What does the directory indicate for its version limit?

Also please post the identity of the Unix box sftp client and version, and please post the connection sequence including the put command used to transfer the file(s).

Also enable verification using sftp -vvv and try it again, and also enable the sftp daemon debugging per the manual. (IIRC, there's a debug logical name listed in the ssh manual for ssh (TCPIP$SSH_SERVER_DEBUG); haven't checked to see if there's one around for the sftp daemon, but I'd expect there's a way to turn on some logging either via logical name or via some sftp daemon command procedure or otherwise.)

[Richard Whalen]

Since SFTP was designed around the Unix model of version-less files, it's possible that second transfer either deleted the original file before transferring the new file, or truncated the original file and rewrote it with the new data.

[Jan van den Ende]

Kevin,

I know little about this stuff, but, with the quirks of TCPIP involved, I'd say there is one other thing to check (besides the points Hoff & Richard already made): maybe the second send is APPENDED to the first?

fwiw

Proost.

Have one on me.

jpe

[Kevin Carter_3]

To confirm: when the second transfer completes, you find the old version on the system, and not the new?

>> yes the old version of the file is on the system

Or does the file now contain the new bits and not the old?

>> I will have to test this and get back to you, might take a day

To confirm: when the second transfer specifies a second and different filename from any existing file in the target directory, you get the second file transferred correctly

>> Yes, the file transfer was originally set up to transfer a file with an extension of the date concatenated and this worked fine; for example simdv6b.20081006 - todays file simdv6b.20081005 - yesterdays file. The problem started when I had the extension changed to simdv6b.txt for the production environment.

To confirm: the default directory is or is not a searchlist? That is, the operation is or is not involving the SYSTEM manager username or the SYS$SYSROOT: directories?

>> No the default directory is not a searchlist.

To confirm: the target LOGIN.COM and SYLOGIN.COM login procedures (for the purposes of testing the transfer) have been examined for and sanitized of errors and are otherwise known not interfering with the transfer.

>> the login.com contains only $ exit and the syslogin.com does not appear to be causing the issue. I can attach a copy if you like?

To confirm: the enveloping directory itself does not also have a version limit set? What does the directory indicate for its version limit?

>> a dir/full on the parent / only directory above the directory shows 'No default version limit'

Also please post the identity of the Unix box sftp client and version, and please post the connection sequence including the put command used to transfer the file(s).

>> will gather this info and post it, I do know that the transfer is being done in a script on the Unix box

Also enable verification using sftp -vvv and try it again, and also enable the sftp daemon debugging per the manual.

>> tried this, the log shows successfull transfer of the file - you may be correct that the second file is being appended to the first - again will know more after testing

(IIRC, there's a debug logical name listed in the ssh manual for ssh (TCPIP$SSH_SERVER_DEBUG); haven't checked to see if there's one around for the sftp daemon, but I'd expect there's a way to turn on some logging either via logical name or via some sftp daemon command procedure or otherwise.)

>> turned on verbose mode in ssh server - nothing unusual jumps out when compared to other logfiles for file transfer

[Steven Schweda]

Hmmm. Around here:

alp $ tcpip show version

HP TCP/IP Services for OpenVMS Alpha Version V5.4 - ECO 7
on a COMPAQ Professional Workstation XP1000 running OpenVMS V7.3-2

alp $ sftp "-V"
alp$dka0:[sys0.syscommon.][sysexe]tcpip$ssh_sftp2.exe: SSH Secure Shell OpenVMS
(V5.5) 3.2.0 on COMPAQ Professional Workstation - VMS V7.3-2

And after this:

alp $ show defa
ALP$DKA0:[SMS.test]

alp $ sftp sms@alp
sftp> put fred.txt
fred.txt | 798kB | 99.7 kB/s | TOC: 00:00:08 | 100%
sftp> put fred.txt
fred.txt | 798kB | 99.7 kB/s | TOC: 00:00:08 | 100%
sftp> quit

I see these:

alp $ dg [-]fred.txt

Directory ALP$DKA0:[SMS]

FRED.TXT;2 1596 6-OCT-2008 14:25:14.43 (RWED,RWED,RE,)
FRED.TXT;1 1596 6-OCT-2008 14:25:00.91 (RWED,RWED,RE,)

Total of 2 files, 3192 blocks.


I could fire up a UNIX system and try it from
there, but it seems unlikely that using a
UNIX SFTP client would make things worse.

Safe to assume that the UNIX user is not
specifying a VMS version number on the
destination? (Why should I need to assume
anything? Actual transcript(s)?)

[Richard Whalen]

The SSH File Transfer Protocol is really a file access protocol and contains operations such as open file, get/set file information, read/write data, close file, delete file. The SFTP user program uses these operations to transfer the data. SFTP user programs that run on Unix often do operations based upon Unix single version file system: if the file named already exists, then it will delete it or open and truncate it before transferring the new data.

Since the SSH File Transfer Protocol also allows for arbitrary extensions to the protocol there are some implementations that will request that an MD5 checksum be performed (over some portion of the file) and then only transfer the file if the MD5 checksum for the local file is different.

[Hoff]

Please show your work; post the verbose diagnostics and the commands used. The " nothing unusual jumps out" phrase can be a particularly deadly assumption when debugging stuff. That's arguably the definition of a good bug, after all. And another set of eyes can sometimes help spot those critters.

[Kevin Carter_3]

Looks like I have some homework to do, thanks for the input so far. I will do as suggested and 'show your work'. This will probably take 24 hours as the person responsible for the sftp from the Unix side is not available today.

Kevin

[Kevin Carter_3]

Mystery solved. The Unix sftp process overwrites the file on the VMS system. But it does not modify the date / time stamp on the file. So if a file simdv6b.dat is created on 5-oct-2008 and another file with the same name is sent on 6-oct-2008 I would not know unless the size of the file changed.