FTP and Netowrk issue

[Tapas Jha]

Hi,

We have an issue in FTP Server and network connectivity.
Please assit on the issue which i am depicting you now.

We have one restricted FTP Server connected in VLAN over a pix firewall.
One of the user is using this FTP Service using their VLAN.
Between FTP Server and Client machine there is one tunnel
established between their firewall and our firewall.

The Problem is users always complain about the FTP issue, either they
can not ftp the file, or they only can receive file properly but not
sending file properly, or they logged in FTP Server and after that they
disconnected by Server.
Now this case has been escalated. I have checked from server side
and didn't find any Server issue. But the user is always complains
that there is a problem in FTP Server.

Our and their network persons are telling that link is perfectly OK
and tunnel established properly, user coming to Server properly and
after that if it disconnected then there is a problem in Server side.

I am using HP-UX 11.00, L1000 Server.

What i checked so far is:
1) I checked xferlog and find that username , filename all are
coming. But there is no entry in xferlog on the time period which
they are mentioning.


In Syslog below entry :
Jun 30 12:22:42 FTPSERVER ftpd[28567]: FTP LOGIN FROM *.*.*.* [*.*.*.*], USER1
Jun 30 12:39:29 FTPSERVER ftpd[28567]:exiting on signal 14

But in xferlog there is no entry for this time and ip.
In Syslog below entry:
Jun 30 12:25:58 FTPSERVER ftpd[28634]: FTP LOGIN FROM *.*.*.* [*.*.*.*], bd
ftpusr
Jun 30 12:41:20 FTPSERVER ftpd[28634]: User USER1 timed out after 900 seconds at W
ed Jun 30 12:41:20 2004
Jun 30 12:41:20 FTPSERVER ftpd[28634]: FTP session closed

Entry on xferlog on that time period:
Wed Jun 30 12:26:10 2004 0.001070 *.*.*.* 6995 /ftpdir/export/work/179249.z
ip b _ o g USER1 ftp 0 * 1088578570 0.000000
Wed Jun 30 12:28:12 2004 0.000983 *.*.*.209 6995 /ftpdir/export/work/179249.z
ip b _ o g USER1 ftp 0 * 1088578692 0.000000
Wed Jun 30 12:28:25 2004 -1 X -1 X X X X X X ftp -1 X 1088578705 0.107422
Wed Jun 30 12:28:43 2004 -1 X -1 X X X X X X ftp -1 X 1088578723 0.158203
Wed Jun 30 12:29:19 2004 0.000875 *.*.*.* 16266 /ftpdir/export/work/179251.
zip b _ o g USER1 ftp 0 * 1088578759 0.000000
Wed Jun 30 12:30:08 2004 -1 X -1 X X X X X X ftp -1 X 1088578808 0.060547
Wed Jun 30 12:31:17 2004 -1 X -1 X X X X X X ftp -1 X 1088578877 0.185547
Wed Jun 30 12:33:22 2004 -1 X -1 X X X X X X ftp -1 X 1088579002 0.060547

BTW, CAN ANYONE TELL ME WHAT IS MEANING OF THE BELOW ENTRY IN xferlog:
Wed Jun 30 12:30:08 2004 -1 X -1 X X X X X X ftp -1 X 1088578808 0.060547
Wed Jun 30 12:31:17 2004 -1 X -1 X X X X X X ftp -1 X 1088578877 0.185547
Wed Jun 30 12:33:22 2004 -1 X -1 X X X X X X ftp -1 X 1088579002 0.060547

2) From network status i found (netstat -an ) that 7 are in CLOSE_WAIT state
since yesterday.

That may be due to unproper shutdown of connection.But how can i kill CLOSE_WAIT
state if it is not being killed after certain time.

2) I have checked nettl.LOG using netfmt command and found nothing wrong.

3) Server has very very less load average at all(0.04)
serve has two cpu's. Below is top result
#top
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.08 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
1 0.00 0.0% 0.0% 1.0% 99.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.04 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
Memory: 38024K (24412K) real, 67456K (28224K) virtual, 20072K free

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1000 26 974 3% 0 - 1 /dev/vg00/lvol2
dev 1000 25 975 2% 0 - 1 /dev/vg01/lvol11
reserve - 337 -337
memory 323 323 0 100%
total 2323 711 1612 31% - 0 -

With regards

Tapas

[Eric Antunes]

As every IT problem, first of all you must detect what is the origin of the issue.

Try to disable temporary the firewalls (one at a time or both at the same time) and see what happens...

[Eric Antunes]

One more thing. You have this after swapinfo command:

...
memory 323 323 0 100%
...

You may have a memmory issue...

[SAHA · Accepted solution]

It looks like firewall issue. Also check number of simultaneous conncetion allowed through tunnel.

thanks,

[rick jones]

Firewalls are notorious for breaking end-to-end principles and causing no end of trouble. You might check your netstat statistics to see about bad retransmission rates and the like.

CLOSE_WAIT means that the application has allegedly been told that the remote has issued a FIN, saying it will send no more data. At some point the application with the connection in CLOSE_WAIT should issue a close or shutdown socket call. If it does not, the implication is the application is broken (99 times out of ten).

So, the usual song and dance about making sure you are up on the latest FTP patches and/or the latest transport patches for your OS rev would seem to be applicable.

[Tapas Jha]

Hi guys,

Let me brief few more things regarding this issue. May be these clue can be helpful.
1)When User are connecting to one PC of the Server's ip range via dialup then they able to send and receive file thru ftp.
2)When tunnel established and trying to send files they faced problem. Though they are telling to receiving files using the tunnel have no problem.
(Seems when they are getting same ip range then no issue, when they are on different IP range then they are not able to send files).

Also, We have found when number of CLOSE_WAIT connection grows then system hangs, increasing load average(Normally load average is below 1 but when no. of CLOSE_WAIT is too high(average 2.46). Even we can't login using console also.

We have below ftp and transport patches installed.

# PHNE_21936 1.0 ftpd(1M) and ftp(1) patch
# PHNE_23949 1.0 ftpd(1M) and ftp(1) patch
# PHNE_29460 1.0 ftpd(1M) and ftp(1) patch

# PHNE_22397 1.0 cumulative ARPA Transport
patch
# PHNE_24715 1.0 cumulative ARPA Transport
patch
# PHNE_26771 1.0 cumulative ARPA Transport patch
# PHNE_29473 1.0 cumulative ARPA Transport patch

Rgds
Tapas

[Eric Antunes]

Hi,

Check also those 2 patches:

PHSS_27962 and PHSS_27964 as there have the keyword "firewall" included.

[rick jones]

My guess as to why the hang as CLOSE_WAITs increase would be consuming all the available file descriptors - eg nfile kernel setting.