data loss in the secure file transfer using java

[antony75]

Hi,

My java Appllication uses the sftp(secure file transfer protocol) for file transferring. when the load is high like 150 GB , data loss occuring for some files especially last chunk of data is missing.

Core file transfer code

while (true)

{

int read = bis.read(buffer); //Reading bytes into the byte array (buffer) from network stream (bis).

if (read==-1) {

break; //Break the loop, if there are no more bytes available in the InputStream.

}

bos.write(buffer,0,read); //Writing the same number of bytes what we read into buffer byte array into OutputStream

complete.update(buffer,0,read); //Giving the same number of bytes what we read above to the checksum calculation.

bytesSoFar += read; //Incrementing the bytes that we have read from the source.


//if the bytes transferred are equal to the source file size

If (bytesSoFar == fileInfo.getFileSize().intValue())

{

// Calculate the file checksum.

// Update DbAuditLog with checksum and no. of bytes read

}

else

{

// Update DbAuditLog with no. of bytes read

}

}

bos.flush();


Please help , why the data loss is happening when the big size files transfer

Note:

1.150 gb not a single file multiple files.

2.SFTP no java specific API, So connection is happening through java .io.sockect connection

Due to data loss, checksum also not correct.
This issue not occurring continuously.. some times whole transfer failing or success.

Is NFS mount plays any role when this occurrence happening .. OS is HP-UX ia64 itanium version.

OpenSSH_4.3p2-hpn, OpenSSL 0.9.7i 14 Oct 2005
HP-UX Secure Shell-A.04.30.007, HP-UX Secure Shell version

java version "1.5.0.05"


Now we are observing with glance tool which is provided by hp to analyze system activity

Application is running along with jconsole.

All the transaction are running under multithreaded way.. that means,most of the txns run multi threaded way , few txns run under singel thread. it depends on the threads available in the thread pool.

[Steven E. Protter]

Shalom,

Your ssh/sftp verion is very old.

That version is quite buggy I recall.

Make sure your code is solid and then look for OS and network causes.

SEP