Server working normally but new clients can not connect

[Krister Valtonen]

We have a server application running at a customer that has a

hostname:#uname -a
HP-UX hostname B.11.23 U ia64 3804540709 unlimited-user license

4 CPU system. We also have a monitor program that regularily connects
to the server and restarts it if it can not get a successful
connect. The problem is that under high usage times the monitor
program can not connect to the server. The connect() call either
times out (1/3 of the times) or returns with ECONNREFUSED error (2/3
of the times). High usage means about 300-400 clients.

On the server side during these failed connects everything is normal
except that new clients can not connect. Existing clients are running
normally. I have monitored "netstat -p tcp" and seen an increse in the
number of "connect requests dropped due to full queue" during these
connect refuse occations. I do not think that the listen queue length
is the problem since we have

hostname:#ndd -get /dev/tcp tcp_conn_request_max
4096

and we are also passing SOMAXCONN = 4096 as backlog argument to our
listen() call.

We also see a sudden increse in the number of sockets in the
FIN_WAIT_2 state during these occations. Usually there are about 0-8
sockets in FIN_WAIT_2, but when new connections are refused the number
increases to 20-40.

I have also run gdb and run backtrace of all threads when the server
is refusing new connections. A very small number of the client threads
are doing some normal work, the vast majority are in
pthread_cond_wait. The main thread is sitting in

#0 0x60000000c0344450:0 in _select_sys+0x30 () from /usr/lib/hpux32/libc.so.1
#1 0x60000000c0358190:0 in select+0xd0 () from #/usr/lib/hpux32/libc.so.1

It seems that somehow client connect()s does not reach the server
select(). What can the problem be?

Krister Valtonen

[Arunvijai_4]

Hi Krister,

You can try with "tusc". It will provide where exactly it hangs.

http://hpux.connect.org.uk/hppd/hpux/Sysadmin/tusc-7.8/

-Arun

[Steven E. Protter]

Shalom Krister,

I'd do this:

tail -f /var/adm/syslog/syslog.log

I find it hard to believe this could be happening without a relavent error message in the log.

SEP

[Michael Steele_2]

Bon jour Krister:

I hate to say that it might not be a network problem. Have you checked for bottlenecks on the system with sar? Might be easier to id the disk, then the file system (pvdisplay) then the application.

(sar -u)
(sar -d)
(sar -d)
(sar -b)