topic Re: SFTP Broken Pipe in Operating System - HP-UX

SFTP Broken Pipe

Aneesh Mohan — Wed, 02 Jun 2010 12:25:07 GMT

Hi,

We are facing Timeout error during SFTP file transfer between hpux systems .

SSH version :-

SFTP server (hpus49,11.31)--- 5.30
SFTP client (hpus29,11.11)---- 5.10

The script "DWH-ftp.sh" is using for SFTP file transfer. The script o/p has been attached.

hpus29^root:/Banktrade/appl/bin > cat DWH-ftp.sh
#! /bin/ksh
DAY=`date +%a`
#######################################
cd /Banktrade/appl/DWHDAILY/frombt
>Btrade_Ready
FLD="BTEXTN01.TXT BTEXTN02.TXT BTEXTN03.TXT BTEXTN04.TXT BTEXTN05.TXT BTEXTN06.T XT BTEXTN07.TXT"
for i in $FLD
do
if [ -f $i ] ; then
echo "\n\tFILE $i IS OK\n"
else
pwd
echo "\n\tDAILY FILE $i IS NOT OK, PLEASE CHECK...........\n"
exit 2
fi
done
sftp dwopr@dwhprod << AAA
cd /dwprod/staging/pr_btrade
pwd
rm BTEXTN??.???
put BTEXTN??.???
put Btrade_Ready
ls
bye
AAA
hpus29^root:/Banktrade/appl/bin >
==============================================================================

SCRIPT Captured OUPUT
===================
ATTACHED

==================================================================
hpus29^root:/Banktrade/appl/bin > nslookup dwhprod
Name Server: uxjedp03.alahli.com
Address: 10.80.1.26
Trying DNS
Name: hpus49.alahli.com
Address: 10.70.3.61
Aliases: dwhprod.alahli.com

=================================================================

hpus29^root:/etc/opt/ssh > cat ssh_config |grep -v "^#"|sort -u

StrictHostKeyChecking no
HashKnownHosts yes
Protocol 2,1
AddressFamily inet
BatchMode yes
Cipher blowfish
Ciphers blowfish-cbc,3des-cbc,aes128-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc
ConnectionAttempts 5
ControlMaster yes
RekeyLimit 2G
ServerAliveCountMax 10
ServerAliveInterval 5
hpus29^root:/etc/opt/ssh >
==================================================

hpus49^root:/etc/opt/ssh > cat ssh_config |grep -v "^#"|sort -u

StrictHostKeyChecking no
HashKnownHosts yes
AddressFamily inet
BatchMode yes
Cipher blowfish
Ciphers blowfish-cbc,3des-cbc,aes128-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc
ConnectionAttempts 5
ControlMaster yes
Protocol 2,1
RekeyLimit 2G
ServerAliveCountMax 10
ServerAliveInterval 5
hpus49^root:/etc/opt/ssh >
=========================================================

The client servr SSH version upgrade from 5.10 to 5.30 already scheduled.

Kindly please let me know if any of you have faced this kind of issues or any advise to me.

Thanks and Regards
Aneesh

Re: SFTP Broken Pipe

Steven E. Protter — Wed, 02 Jun 2010 13:28:48 GMT

Shalom Aneesh,

Have not seen these kind of results.

Not using your exact script and ssh settings either.

I would think that getting the upgrade done might help.

Or it might break more stuff.

SEP

Re: SFTP Broken Pipe

James R. Ferguson — Wed, 02 Jun 2010 15:02:22 GMT

Hi:

Things you don't offer:

(1) Did this ever work?
(2) If it has worked in the past, what's different now?
(3) Can you use your script to transfer *one*, *small* file successfully?
(4) Are these files very large? What size?
(5) If 'largefiles' are involved (>2GB) is 'largefiles' enabled for the receiving filesystem?

Regards!

...JRF...

Re: SFTP Broken Pipe

Aneesh Mohan — Sun, 06 Jun 2010 11:32:59 GMT

Hi James,

my apologies to the late reply...

(1) Did this ever work?
The script/simple command never returned with complete transfer of files .

hpus49^root:/var/adm/crash/testarea > ls -lrt
total 1582864
-rw------- 1 root sys 285835264 Jun 3 15:15 BTEXTN02.TXT.3
-rw------- 1 root sys 1550770 Jun 3 15:24 BTEXTN04.TXT
-rw------- 1 root sys 10990140 Jun 3 15:24 BTEXTN03.TXT
-rw------- 1 root sys 4044900 Jun 3 15:24 BTEXTN05.TXT
-rw------- 1 root sys 20955136 Jun 3 15:24 BTEXTN06.TXT
-rw------- 1 root sys 831870 Jun 3 15:36 BTEXTN01.TXT
-rw------- 1 root sys 831870 Jun 3 15:36 BTEXTN01.TXT.1
-rw------- 1 root sys 285835264 Jun 3 15:36 BTEXTN02.TXT
-rw------- 1 root sys 199524352 Jun 3 15:37 BTEXTN02.TXT.1
hpus49^root:/var/adm/crash/testarea >

(2) If it has worked in the past, what's different now?
It is a new deployment

(3) Can you use your script to transfer *one*, *small* file successfully?
Kinldy please find the below testing ..

from 10.70.3.61 #sftp 10.80.3.35

10.80.3.35 SFTP> mget BTEXTN02.TXT.1
Fetching /var/adm/crash/testarea/BTEXTN02.TXT.1 to BTEXTN02.TXT.1
/var/adm/crash/testarea/BTEXTN02.TX 100% 190MB 10.6MB/s 11.4MB/s 00:18

10.80.3.35 sftp> mput BTEXTN02.TXT.1
Uploading BTEXTN02.TXT.1 to /var/adm/crash/testarea/BTEXTN02.TXT.1
BTEXTN02.TXT.1 0% 256KB 1.0KB/s 0.0KB/s - stalled -Timeout, server not responding.

As per the above test mget from 10.80.3.35 is working and mput is dropping .

(4) Are these files very large? What size?
Even small files are also ..same effect.

sftp> mput BTEXTN01.TXT.1
Uploading BTEXTN01.TXT.1 to /var/adm/crash/testarea/BTEXTN01.TXT.1
BTEXTN01.TXT.1 47% 384KB 1.4KB/s 0.0KB/s - stalled -Timeout, server not responding.

It appear to me like firewall security issue , So I have escalated this problem to firewall secuirity team ,but not good response from them.

Also .. both source & destination are now with SSH 5.30 version,please do let me know is there any tunnings (like keepalive interval) can help to solve this behaviour .

Regards
Aneesh

Re: SFTP Broken Pipe

James R. Ferguson — Sun, 06 Jun 2010 12:03:46 GMT

Hi Aneesh:

You might try:

# ssh -o TCPKeepAlive=no -o ServerAliveInterval=20 hostname [...]

Regards!

...JRF...

Re: SFTP Broken Pipe

Aneesh Mohan — Sun, 06 Jun 2010 12:18:01 GMT

Thanks James,

Now the data transfer stopped immediately after 8 to 10 sec ...and the session is waiting for alive time expiry period.

hpus49^root:/var/adm/crash/testarea > sftp -o TCPKeepAlive=no -o ServerAliveInterval=20 hpus29
Connecting to hpus29...

sftp> lcd /var/adm/crash/testarea
sftp> cd /var/adm/crash/testarea
sftp> mput BTEXTN01.TXT.1
Uploading BTEXTN01.TXT.1 to /var/adm/crash/testarea/BTEXTN01.TXT.1
BTEXTN01.TXT.1 55% 448KB 0.0KB/s 0.0KB/s - stalled

The session is in "stalled" state now

Aneesh

Re: SFTP Broken Pipe

Aneesh Mohan — Sun, 06 Jun 2010 12:27:35 GMT

Please see ..

sftp> lcd /var/adm/crash/testarea
sftp> cd /var/adm/crash/testarea
sftp> mput BTEXTN01.TXT.1
Uploading BTEXTN01.TXT.1 to /var/adm/crash/testarea/BTEXTN01.TXT.1
BTEXTN01.TXT.1 55% 448KB 0.0KB/s 0.0KB/s - stalled -Timeout, server not responding.
hpus49^root:/var/adm/crash/testarea >

Aneesh

Re: SFTP Broken Pipe

James R. Ferguson — Sun, 06 Jun 2010 15:19:31 GMT

Hi (again):

See if this changes thing:

# ssh -o TCPKeepAlive=yes -o ServerAliveInterval=30 ServerAliveCountMax=5 hostname [...]

There are correcsponding 'ClientAlive*' parameters too.

Regards!

...JRF...

Re: SFTP Broken Pipe

Aneesh Mohan — Mon, 07 Jun 2010 07:30:15 GMT

Hi James,

It failed like above.

Aneesh

Re: SFTP Broken Pipe

Aneesh Mohan — Mon, 07 Jun 2010 07:38:01 GMT

source :10.70.3.61
destination:-10.80.3.35
In between firewalls :- 2 nos
command used :-
hpus49^root:/> sftp -o TCPKeepAlive=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=5 hpus29

cmd o/p:-
==========

sftp> mput BTEXTN01.TXT
Uploading BTEXTN01.TXT to /var/adm/crash/testarea/BTEXTN01.TXT
BTEXTN01.TXT 63% 512KB 0.0KB/s 0.0KB/s - stalled -Timeout, server not responding.

Source hpus49(10.70.3.35) details :-
======================================

hpus49^root:/ > traceroute 10.80.3.35
traceroute to 10.80.3.35 (10.80.3.35), 30 hops max, 40 byte packets
1 10.70.3.100 (10.70.3.100) 0.461 ms 0.324 ms 0.249 ms
2 10.90.90.2 (10.90.90.2) 1.096 ms 1.190 ms 0.885 ms
3 11.70.0.3 (11.70.0.3) 1.457 ms 1.411 ms 1.084 ms
4 10.80.90.99 (10.80.90.99) 1.009 ms 0.902 ms 1.075 ms
5 hpus29.alahli.com (10.80.3.35) 1.213 ms 0.988 ms 1.022 ms
hpus49^root:/ >

hpus49^root:/ > netstat -nr
Routing tables
Destination Gateway Flags Refs Interface Pmtu
127.0.0.1 127.0.0.1 UH 0 lo0 32808
10.70.3.61 10.70.3.61 UH 0 lan0 32808
10.70.3.0 10.70.3.61 U 2 lan0 1500
127.0.0.0 127.0.0.1 U 0 lo0 32808
default 10.70.3.100 UG 0 lan0 1500
hpus49^root:/ >

Destination hpus29(10.80.3.35) details:-
========================================
hpus29^root:/ > traceroute 10.70.3.61
traceroute to 10.70.3.61 (10.70.3.61), 30 hops max, 40 byte packets
1 10.80.3.100 (10.80.3.100) 0.517 ms 0.286 ms 0.255 ms
2 10.80.90.2 (10.80.90.2) 1.119 ms 1.089 ms 10.80.90.1 (10.80.90.1) 1.094 ms
3 11.70.0.2 (11.70.0.2) 1.270 ms 1.246 ms 1.252 ms
4 10.90.90.99 (10.90.90.99) 0.969 ms 1.035 ms 0.876 ms
5 hpus49.alahli.com (10.70.3.61) 1.053 ms 1.011 ms 1.067 ms
hpus29^root:/ > netstat -nr
Routing tables
Destination Gateway Flags Refs Interface Pmtu
127.0.0.1 127.0.0.1 UH 0 lo0 4136
10.80.1.21 10.80.1.21 UH 0 lan5 4136
10.80.41.21 10.80.41.21 UH 0 lan7 4136
10.80.3.35 10.80.3.35 UH 0 lan2 4136
10.80.1.0 10.80.1.21 U 2 lan5 1500
10.80.3.0 10.80.3.35 U 2 lan2 1500
10.80.41.0 10.80.41.21 U 2 lan7 1500
10.70.10.0 10.80.41.100 UG 0 lan7 1500
10.80.10.0 10.80.41.100 UG 0 lan7 1500
10.80.1.0 10.80.1.100 UG 0 lan5 1500
10.128.110.0 10.80.1.100 UG 0 lan5 1500
127.0.0.0 127.0.0.1 U 0 lo0 4136
default 10.80.3.100 UG 0 lan2 1500
hpus29^root:/ >

Re: SFTP Broken Pipe

Aneesh Mohan — Mon, 07 Jun 2010 07:44:09 GMT

Typo mistake in above info..

Source hpus49(10.70.3.35) details :-

correct ..

Source hpus49(10.70.3.61) details :-

aneesh