HP-UX 11.11, connect() and read() fail with EADDRUNAVAIL

[troy Morrison]

I am trying to troubleshoot a problem on a remote system, where under minor
load the system appears to run out of some sort of resource that causes
connect() to begin returning -1 with errno == EADDRUNAVAIL.

This would of course suggest that the socket doing the connect()ing has
bound (with bind) to a source address that isn't configured on the machine.
However, in this case, there is no bind on the socket, as shown here in this trace (the host's local lan0 is configured at 147.128.44.88):

socket(AF_INET, SOCK_STREAM, 0) ............... = 4
fcntl(4, F_SETFD, 1) .......................... = 0
connect(4, 0x40031638, 16) .................... ERR#227 EADDRNOTAVAIL
sin_family: AF_INET
sin_port: 27009
sin_addr.s_addr: 147.128.44.88
shutdown(4, SHUT_RDWR) ........................ ERR#22 EINVAL
close(4) ...................................... = 0

This example is using the FLEXnet publisher lmstat tool, but when the system
gets into this state, it seems like *everything* starts to fail with this
error. Another example is telnet:

write(1, 0x4000c8a8, 10) ...................... = 10
T r y i n g . . . \n
socket(AF_INET, SOCK_STREAM, 0) ............... = 3
connect(3, 0x7a000980, 16) .................... ERR#227 EADDRNOTAVAIL
sin_family: AF_INET
sin_port: 23
sin_addr.s_addr: 127.0.0.1

I've tried to determine whether some kernel tuning value was out of whack or
what might otherwise cause this, but while I'm pretty experienced with UNIX,
I'm not the strongest on HP-UX.

Any ideas as to what might cause this?

Thanks,
Troy

[troy Morrison]

After weeks of looking at this off and on, of course I found the answer a few minutes after posting this.

# /usr/bin/ndd /dev/tcp tcp_smallest_anon_port tcp_largest_anon_port
45800

45900

Apparently on this system the ephemeral port range was only 100 ports, which get used very quickly. Resetting these values eliminates the problem.

Troy

[troy Morrison]

See previous post for the solution.

[rick jones]

Kudos for using tusc !-)

Certainly resetting the largest_anon_port to its proper value of 65535 is goodness. Better still would be to code the application such that it was not dependent on the system being configured "well" - by making explicit calls to bind() before calling connect() and selecting local port numbers in the range of 5000 to 65535.

That way, your application can churn through roughly 1000 connections per second before again tripping over trying to reuse a connection name (local/remote IP, local/remote port) for which there is still a TCP endpoint in TIME_WAIT.

[troy Morrison]

tusc is my savior :)

I thought about the wisdom of doing this, but decided it was sort of silly. I could potentially fix our software, but it would require identifying a ton of places that we create sockets and fixing all of them. Even then, not only could I not trust any binaries that might be called, I can't even trust library calls (gethostbyname might fail, for example, if the anonymous UDP port space was exhausted).

And I can't imagine the laughter I'd get reporting all those "bugs" to HP and other library vendors... :)

Troy