How to know when client socket is down ?

[Tim Howell]

What methodology should be used in the server application to know when the client socket is no longer working? I have a small app using tcp sockets that works fine until someone inevitably powers the remote unit down (it is a scale on the factory floor). When the client is powered back up, the server grants a new connection... but the program still waits for the data from the old connection (ignoring the new). The connection never times out or anything - netstat reports ESTABLISHED, even after days. The server is an rp5470 running HP-UX 11.11
I have tried 'select' for reading & writing to no avail. The server app thinks the socket is still up.
TIA

[Steven E. Protter]

Shalom,

You can use netstat -an to determine the connection state, which will change to WAIT CLOSE state, then use ndd to close the connection.

SEP

[Peter Godron]

Tim,
nettune (renamed in 11.0)
ndd

please have a look at:
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=99069

Things to also investigate:
tcp_keepstart - If connection idle for x start sending requests
tcp_keepstop - length of time to keep sending requests
tcp_keepalive_interval
tcp_keepstart
tcp_ip_abort_interval
tcp_keepstop
tcp_ip_abort_cinterval

[Tim Howell]

ndd looks like it affects all networking. I think what I'm looking for is something specific I can do within a program for one socket. Should this thread be moved to a development forum?
Thanks again;

[Ben Dehner]

What you can do at the application level depends on the language your app is written in.

In C, you want to use setsockopt() to set the SO_KEEPALIVE option on the socket.

In Perl, use the setsockopt() function as in C.

In Java, assuming java.net.Socket, use Socket.setSoTimeout() to set read timeouts.

[Tim Howell]

I was aware of setsockopt(), but according to docs, so_keepalive only comes into play after two hours of idle time. I need a way to know within miniutes.
Thanks again;

[rick jones]

You should define and implement an application-level heartbeat mechanism.

TCP can only guess that the remote is down if it tries to send something to the remote and receives no response after some number of tries and/or length of time. If your server application is just sitting there receiving, then the TCP is not trying to send something to the remote and so will never know the remote is down.

The heartbeat mechanism could take one of two forms:

1) The server sends an "are you there" message to which the client responds.

2) The server expects to receive an "i am here message" every N units of time from the client.

In either case, if the response is not forthcoming within the limits your application desires your server code can decide to call close() against that socket.

The SO_KEEPALIVE path is a dodge - it will accomplish what you want, but you will have to live within the _system-wide_ keepalive settings - two different applications on the same server might have different needs, so an application-specific mechanism is always best.

Another option, rather than having the keepalive/heartbeat all the time on the socket, is to notice that you've received a new connection from a client IP for which you have an old connection (the remote IP and port are available to you in the accept() call). You could then send the "are you there" on the old connection.

[Biswajit Tripathy]

Not sure if it's an option for you, but can you ask the operator powering down the remote unit to shut it down nicely and not pull the plug ? If the remote machine goes down because of reboot, the remote TCP/IP would close all the connections nicely before shutting down.

- Biswajit

[Tim Howell]

Rick,

I think your option - "notice that you've received a new connection from a client" - is a great idea. However, I am back to my original problem of not getting 'select' to work as expected in order to do it.
Is there something I'm missing etc, from the code snippets below? Select always returns 0, but netstat reports two established connections after I cycle the power on the client unit.
Thanks;
...
int listenfd;
fd_set readfds;
struct timeval timeout;
timeout.tv_sec = 1;
timeout.tv_usec = 0;
...

if ( (listenfd = socket(PF_INET,SOCK_STREAM, 0)) < 0) {
printf("sock error:");
...
}
...
/* bind,listen,accept, etc */
...
/* com loop taking place here */
...
FD_SET(listenfd, &readfds);
...
ret = select(listenfd, &readfds, NULL, NULL, &timeout);

if (ret > 0) {
printf("new connection ready!\n");
/* exit loop here */
}
else {
printf("select returns %d\n",ret);
...
}

/* end loop here */
...

[rick jones]

Try passing-in listenfd+1 as the nfds parameter. The fd space starts at zero, but the "count" starts at one. For example, fd 5 is actually the sixth entry in the mask...

[Tim Howell]

Thanks again - I did try that originally,

ret = select(listenfd+1, &readfds, NULL, NULL, &timeout);

but still no luck...

[rick jones · Accepted solution]

use tusc to make sure that the parameters to the select() call are what you think they are - try adding a -v to tusc.

[Biswajit Tripathy]

Tim,

May be I don't understand the problem
completely, but if the remote host is shutdown by
an operator "nicely", the remote TCP/IP MUST
send a FIN and terminate the connection (which
means, you will NOT have connections in
ESTABLISHED state in the server hanging
around for days).

The only 2 conditions under which the server
shows the ESTABLISHED state for connections
whose peers are down would be (1) if the peer
machine crashed, or (2) the route between client
and server goes down. In both the cases, you
could write a separate heartbeat mechanism
(may be a script) that would just ping the client
system and kill all the local applications that you
want killed when the remote system is not
ping'able. This would be much easy to do than
modifying the source code.

- Biswajit

[rick jones]

Biswajit - IIRC he said that the client is a scale on a factory floor that someone just powers-down. That is the classic sort of situation which would leave connections in ESTABLISHED on servers. "Back in the day" it was people doing that with their PCs - just hitting the power switch. These days that is less common since even PC OSes prefer to be shutdown gracefully for other reasons.

I could easily see someone just powering-down a scale.

Now...

It would be good to suggest to the scale manufacturer that they start thinking about a graceful shutdown of the device when someone hits the power switch. Afterall, their "scale" isn't just a scale any more, but a remote, weight-reporting computer :)

[Tim Howell]

Thanks to all - the issue is still open for suggestions for now while I am still persuing Rick's suggestion.

[Biswajit Tripathy]

Thanks, Rick, for explanation. I did not realise
the peculiar situation here.

Thinking aloud, I can probably see a _potential_
solution involving IPFilter system firewall here,
though I have never tried this. IPFilter system
firewall on HP-UX provides a kernel tunable
"fr_tcpidletimeout" that can be set to as small
as 5 minutes. This will remove connections in
ESTABLISHED state that are idle for more than
5 minutes. Interesting point is, it would close
only those connections that are tracked by
IPFilter (i.e that has an IPFilter rule set). For all
other connections, the system wide timeout
value should work. So, potentially, you could
set specific IPFilter rules for IP addresses of the
scales that is creating the mess and set the
"fr_tcpidletimeout" to 5 minutes that will
be applied to only those connections.

- Biswajit

[Tim Howell]

Thanks to all. Rick's solution solves my problem!

Rick,
great job - the tusc program was most helpful in troubleshooting!

Biswajit,
The IPFilter system firewall you describe is most interesting. While it doesn't solve my overall problem in this case, it is something worth studying for future needs...

Thanks again!