too many sockets open on the system

[T S_1]

I have a Solaris 2.6 box with a web server and Oracle database running. It has a lot of RAM and disk space. One of the applications running on the system seems to be timing out and sometimes dying. The application is written in WebObjects and the developers are working on the proble.

I wanted to do a little investigation myself,I ran netstat and counted the number of sockets in each states. Here are two examples for high and low numbers of sockets in wait state. I found this really alarming, because I have other systems that are quite busy, but the number of sockets in the wait state is nearly nothing compare to this.

Here are the output:

4 (socketpair)
5 /tmp/.cgistub_7179
1 /var/opt/perf/datafiles/.perflbd.socket
25 ESTABLISHED
1 FIN_WAIT_2
2098 TIME_WAIT

4 (socketpair)
5 /tmp/.cgistub_7179
1 /var/opt/perf/datafiles/.perflbd.socket
23 ESTABLISHED
117 TIME_WAIT


Here is another output from running netstat, does this look normal, where the client and server are both the localhost 127.0.0.1 and the ports are listening on one another.


TCP
Local Address Remote Address Swind Send-Q Rwind Recv-Q State

127.0.0.1.32774 127.0.0.1.32773 32768 0 32768 0 ESTABLISHED
127.0.0.1.32773 127.0.0.1.32774 32768 0 32768 0 ESTABLISHED
47.61.0.123.32778 47.61.0.123.726 32768 0 32768 0 ESTABLISHED
47.61.0.123.726 47.61.0.123.32778 32768 0 32768 0 ESTABLISHED
127.0.0.1.64769 127.0.0.1.64768 32768 0 32768 0 ESTABLISHED
127.0.0.1.64768 127.0.0.1.64769 32768 0 32768 0 ESTABLISHED
127.0.0.1.54530 127.0.0.1.54529 32768 0 32768 0 ESTABLISHED
127.0.0.1.54529 127.0.0.1.54530 32768 0 32768 0 ESTABLISHED
127.0.0.1.54532 127.0.0.1.54531 32768 0 32768 0 ESTABLISHED
127.0.0.1.54531 127.0.0.1.54532 32768 0 32768 0 ESTABLISHED


Thank you

[Vincenzo Restuccia]

Check the process associated at the soccket with lsof.

[Brian Hackley]

Hello,

A Solaris 2.6 system with Oracle with many Oracle sockets in TIME_WAIT. Looking at TCP/IP Illustrated Volume 1, by Stevens, there is a nice definition of the various TCP States and TIME_WAIT means your connection is awaiting what is called the 2MSL timeout at connection close time. There are several cases and explainations offered in this book which you might want to investigate. On HP-UX 11.0 this is set to 1 minutes by default and can be tuned with ndd /dev/tcp tcp_time_wait_interval
On Solaris 2.6:
# ndd -get /dev/tcp tcp_close_wait_interval
240000 <-- 4 minutes

Hope this helps,

-> Brian Hackley

[T S_1]

Thank guys,

I was able to find more info on TCP/IP parameters for Solaris.

If you look a the first 2 lines from netstat, does it look unusual that the loopback address has 2 established connections, where it is acting as a server and a client at the same time, and on 2 different ports?

TCP
Local Address Remote Address Swind Send-Q Rwind Recv-Q State

127.0.0.1.32774 127.0.0.1.32773 32768 0 32768 0 ESTABLISHED
127.0.0.1.32773 127.0.0.1.32774 32768 0 32768 0 ESTABLISHED

E. Farah

[Kevin Wright]

Not really if your running a webserver, these could be daemons listening on the localhost, check your /etc/services file...ie, on our Cisco works box I see alot of the cscomd running on the localhost...
localhost.cscomd localhost.1100 ESBLISHED
localhost.1100 localhost.cscomd ESTABLISHED

As for what is really happenning with these, I am not sure, unless it is some type of GUI,