topic SFTP Transfer Speed in Operating System - HP-UX

SFTP Transfer Speed

Bharathidasan P — Tue, 08 Aug 2006 04:46:42 GMT

Hi all,
I have to transfer a file (Approx. 500 kb) between two HP-UX Machines through SFTP.
The Script works in the following way.
1. Checks the Remote File Directory - if not creates the directory
2. Checks the Space - bdf
3. Transfers the File
4. Checks the File size
5. Verifies Checksum
Now it takes around 45-50 secs to complete the job...but its not an efficient transfer.

1. By increasing the buffer value, we can improve the speed - but more memory consumption
2. Any other Environment settings will help to improve the transfer Speed. If any thoughts, Please share With me..

cheers,
Bharathi..

Re: SFTP Transfer Speed

Peter Godron — Tue, 08 Aug 2006 04:52:35 GMT

Hi,
biggest impact on transfer speeds:
1. Network capacity
2. Disk speed

So unless you can change the network setup to a higher speed or use faster disks, I suspect you have improved transfer as much as you can.

Re: SFTP Transfer Speed

Steven E. Protter — Tue, 08 Aug 2006 04:55:01 GMT

Shalom,

The factors that most effect this transfer, the networking environment outside the HP-UX server are not being dealt with.

The best you can hope for by working on the HP-UX server is a marginal improvement.

ndd -h sup
ndd -h unsupported

You may be able to tweak things there, but this is hardly worth the effort. You are running strict checks for a good reason and executing some complex checks and this takes time.

You may find the checksum execution and such improves if you tune your kernel and have the strong random number generator installed.

SEP

Re: SFTP Transfer Speed

IT_2007 — Tue, 08 Aug 2006 14:03:59 GMT

Bharathi,

Check with network folks about switch settings which may not be matching to your server like 100Half duplex to 100 Full Duplex with auto negotiation off.

thanks,
Srini

Re: SFTP Transfer Speed

rick jones — Tue, 08 Aug 2006 19:50:39 GMT

Have you timed the individual steps to see which take the most time?

When you do the bdf is that against all the filesystems or just the remote file directory?

Step 5 makes step four redundant in the successful case. The successful case should be the most common case :) so it would be faster to just omit the filesize check and just do the checksum check.

One limit to the performance of a TCP transfer is the window size divided by the Round Trip Time. That is a "speed of light" sort of limit. You may need to consume the memory to get the higher transfer rate.

Those individual steps - are they individual remote operations, or are some combined ina single remote "call" as it were?

WRT duplex, unless you have link-level errors logged in lanadmin stats, don't go messing with duplex settings. Errors would be FCS or _late_ collisions, not just normal collisions.

Also, there is the matter of the crypto overhead.

Re: SFTP Transfer Speed

rick jones — Tue, 08 Aug 2006 19:53:13 GMT

BTW, doing a remote bdf on just the remote file directory would be a decent test of the existence of the remote directory and would then save a little time in the normal - aka successful - case.

# remsh sweb50 bdf /no/such/directory
bdf: /no/such/directory: No such file or directory

And so you could omit step one in the normal case, and make it a step 2b when the bdf fails with no such file...

Re: SFTP Transfer Speed

Matti_Kurkela — Wed, 09 Aug 2006 02:46:31 GMT

If you're using SFTP to transfer the file, it implies you might be using SSH to run the checks before and after the transfer. Is it so?

Establishing a SSH/SFTP connection requires a lot of good-quality random numbers (for session keys). If your servers don't have /dev/random and /dev/urandom, the SSH programs may need to use a slow way to generate the random numbers.

If you're using HP-UX 11.11, I would recommend installing the Strong Random Number Generator depot to all servers that have a lot of SSH/SFTP transfers. It is available for free in software.hp.com, "Security and manageability" group. A direct link is here:
http://h20293.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=KRNG11I

Re: SFTP Transfer Speed

Bharathidasan P — Wed, 09 Aug 2006 08:43:37 GMT

Hi all,

Thanks for your Kind Response.

I'm trying to improve the process by installing HP-UX Strong Random Number Generator,Yet to check the performance.

I'll update you guys,if i find the solution.

Once again Thanks for your Valubale Suggestions..

Cheers,
Bharathi..