TCP Zerowindow"

[skt_skt]

HP-UX 11.11

I am trouble shooting network error and while lookig at the trace file we see lot of "TCP Zerowindow" (When the receiving host's buffer fills, the next acknowledgement contains a 0 in the window size, to stop transfer and allow the data in the buffer to be processed) and AFAIK this is some thing to do with "TCP/IP bufferes"


Any one know what could casue this and a fix?

[Mel Burslan]

Either receiving host needs a faster interface or sending host needs to throttle the data sent.

Usually second option, i.e., throttling is not option if you have too many senders and one target receiver. So, make sure you have a really fast interface listening for the traffic.

There might be an obscure ndd parameter but ndd is not my forte. I am sure Rick Jones on these forums, can provide an answer if there is one related to ndd.

[rick jones]

His name invoked, Rick Jones was compelled to appear :)

When a receiving TCP returns an ACK with a 0 in the window field, it means the receive socket buffer has filled. This means the receiving *application* has fallen-behind. If this is simply a transient sort of thing, you might have the application make a setsockopt() call to increase the size of SO_RCVBUF. If the application is brain-dead in that regard, you can probably use ndd to tweak tcp_recv_hiwater_def to a larger value.

However, if the application is *always* behind - that is the zero window ACKs are frequent the only thing to do is find some way to improve the performance of the application.

Zero window ACKs say nothing about the speed of the recieving NIC being too slow. When the receiving NIC (or its interrupt CPUs) are too slow you get recv errors reported in lanadmin statistics.

[skt_skt]

i dont see any RECV error. But here is the statisctis before and after we had the problem this morning. I am concerend about "persist timeouts"(When a receiver advertises a window size of 0, the sender stops sending data and starts the persist timer) and "full queue" and "due to no listener"

Also is "keepalive timeouts" a concern

Fri May 22 01:57:49 EDT 2009
2677614 embryonic connections dropped
3013248049 segments updated rtt (of 3013248049 attempts)
6171595 retransmit timeouts
481832 connections dropped by rexmit timeout
29608532 persist timeouts
2713014 keepalive timeouts
2699941 keepalive probes sent
969 connections dropped by keepalive
3305474 connect requests dropped due to full queue
2104496 connect requests dropped due to no listener

Fri May 22 02:22:49 EDT 2009

2677823 embryonic connections dropped
3014514533 segments updated rtt (of 3014514533 attempts)
6172290 retransmit timeouts
481882 connections dropped by rexmit timeout
29614986 persist timeouts
2713098 keepalive timeouts
2700024 keepalive probes sent
969 connections dropped by keepalive
3305952 connect requests dropped due to full queue
2104590 connect requests dropped due to no listener

[Mel Burslan]

Santosh,

as if you are describing the innards of a newly discovered denial of service attack. If your process is a webserver and it is infected somehow or designed to inflict pain to the visitors, this is exactly what you would do to make the connecting clients, spin their cycles waiting for your receive buffers to become available, even though your receive buffers are capable of receiving traffic.

I have listened to a computer security related a podcast few months ago, describing a DoS attack taking advantage of this very vulnerability of the tcp protocol communications. Is this listening application something you have written or somehow inherited from people you don't know and have no way of contacting. If it is the latter, I'd seriously suspect a malicious intent.

my 2 cents

[rick jones · Accepted solution]

Persist timeouts on the server mean it is the remote (clients) which are advertising a zero window.

Zero window ACKs from the server mean it is the server advertising a zero window.

Connections dropped due to full queue means that either tcp_conn_request_max is "too small" or the application(s) are making listen() calls with too small a backlog, or the application(s) are not calling accept() fast enough for the given backlog levels - which will be the minimum of what they pass in listen() and the value of tcp_conn_request_max.

Connections dropped due to no listener means just that - there was no listen endpoint for the destination port number in the arriving SYNchronize segment.

Whether this is or is not a Denial Of Service attack probably calls for further investigation.

[skt_skt]

"This means the receiving *application* has fallen-behind. If this is simply a transient sort of thing, you might have the application make a setsockopt() call to increase the size of SO_RCVBUF"


Rick, Can you explain me or get me doc/URL which explians setsockopt/SO_RCVBUF

[rick jones]

If there is a document on docs.hp.com for socket calls it would probably have "Socket" or "Sockets" in its title. Otherwise, one or more of the works of the late W. Richard Stevens - such as "Unix Network Programming" would be an excellent resource on socket calls.

BTW, earlier I think you said you saw now recv errors - indeed there would be none if the application is falling behind. The application's recv() (or read() etc) calls will still succeed, they just aren't being called "fast enough" by the application to keep-up with the rest of the stack and the network. There probably won't be any link-level recv errors either since TCP flow-control will kick-in when the window goes to zero bytes.