ftp: connect: Can't assign requested address - possible tcp_conn_request_max tweak needed?

[Alzhy]

We've a process that spawns many zillions of FTP sessions to the same server it running (weird but that's how it was designed). The environment has been growing - with a new "region" added that we have to bump up nfiles, maxfiles_limit, etc. However, the TCP stack tunables are still defaults. For instance tcp_conn_request_max is at its default of 4096.

Any guidance if I am on the right track tweaking tcp_conn_request_max and possibly doubling or even trebling it? WHat tools do I need to call on to establish that there indeed is the need or am I totally pursuing the wrong path? Sorry but I still have a sketchy understanding of what the process does.

"netstat -a" linecount has been close to 10,000 with TCP ESTABLISHED count in the higher 3000's and a lot of TIMEWAITs.

netstat -p tail is as follows:

1326649 connection requests
1574589 connection accepts
2901238 connections established (including accepts)
3173715 connections closed (including 276135 drops)
26321 embryonic connections dropped
3774307177 segments updated rtt (of 3774307177 attempts)
185583 retransmit timeouts
18021 connections dropped by rexmit timeout
9956 persist timeouts
373752 keepalive timeouts
373530 keepalive probes sent
2 connections dropped by keepalive
0 connect requests dropped due to full queue
10254 connect requests dropped due to no listener


We have OVPA/Glance too.

[Steven E. Protter]

Shalom,

You need to up tcp_conn_request_max, in nddconf, you need to up nproc nfile and maxuprc at a minimum in the kernel.

SEP

[rick jones · Accepted solution]

I doubt that tcp_conn_request_max would do anything here. That setting only controls the maximum depth of a listen queue for a TCP endpoing in the LISTEN state.

Since you mention "zillions of FTP sessions" I will guess that you are churning through TCP connections quickly and running-out of "free" four-tuples of local/remote IP, local/remote port. There are greater details I can go into if you like.

IIRC the HP-UX FTP uses the anonymous port range for its connections - that is it does not call bind() with an IP or port number before calling connect(). So, since it would not be a good idea to reduce the length of time spent in TIME_WAIT, it would be good to increase the size of the anonymous port space. By default that runs from 49152 (tcp_smallest_anon_port) to 65535 (tcp_largest_anon_port - 65535 is the max since a port number is a 16 bit unsigned value).

You can probably drop tcp_smallest_anon_port to 32768 or even 16384. You should avoid dropping it below tcp_smallest_nonpriv_port.

There is the small matter of tcp_smallest_anon_port showing-up in the "unsupported" section of ndd -h on 11.11 - you should check with official support and ndd -h on your own system. Tweaking it should be relatively save - save some app using a hard-coded port above the value to which you set tcp_smallest_anon_port, and your FTP sessions getting there first.

The "don't raise the bridge, lower the river" answer would be to tweak your code to send more files per FTP session. If you are transferring only one file per session, you are burning through two TCP connections per file transferred. Either send more files per FTP session, or send files via something that only burns one connection per file - and even then, preferably multiple files per session.