Data Transfer rate "Fluctuation" using scp/sftp through Virtual Connect

[BE_UNIX_PROD]

Hello,

We are facing this issue and after trying for almost two weeks, no solution has been found yet. So any pointers in the right direction would be very highly appreciated. 

Architecture: Recently we have migrated two servers to the new integrity based infrastructure. There are two blades BL890C, in SG cluster, using Virtual Connect Flex-10. The cluster pkg is BC volume of Billing DB. They need to talk with DWH servers for fetching data for reports etc. DWH servers are located in another physical location and have a different vlan. Blades have HP-UX 11.31, while DWH servers have 11.23.

The Problem: While transfering data from Billing to DWH, data transfer rate fluctuates so much that its almost useless. It ranges from 10 MBps to 64 KBps, and its just random. i.e. it doesn't decrease/increase continuosly. In one instance its 8 MBps and next would be 500 KBps and so on. The problem is with one-sided data transfer, meaning that transfer from DWH to these servers is almost constant at 10 MBps. Also, there is no problem while transfering withing the same data center or same subnet. 

It would look like a basic case of network optimization or something to do with network infrastructure, but our network admins have ruled everything out from their end. They said everything is ok, although we have no details of their tests. The problem is so serious now that management is even thinking to role back the project (migrate back to old servers). Since no one seems to know too much about Virtual Connect, so all fingers are being pointed at it...!!! 

I'd provide any details necessary to make an analysis. We've even tried to tinker with TCP parameters, ssh parameters, turned off FlowControl in VC etc. But nothing has worked so far. 

Thanking in anticipation!

Hassan Gilani

Milan, Italy

[Dennis Handly]

Do you have a way to test it without the encryption of scp/sftp?  I.e. you may have CPU vs networking issues.

[BE_UNIX_PROD]

For the purpose of testing, I've shifted the BC pkg to the other node, so the server is currently doing nothing at all. I don't think it could be CPU problem in such case. Also I've observed the CPU utilization during the transfer and its absolutely normal. 

[BE_UNIX_PROD]

Gurus, please help me out in this issue... I've truly lost my way and am out of anymore options to think.