DB Informix hangs!!! errno 233

[Duda_1]

Hello my friends,
I have a problem with my Informix database, I think it's associate with the operating system (HPUX 11.00). Sometimes the application hangs (The established connection works fine but the new ones can´t connect) and in the DB online log we can find:
"listener-thread: err = -25573: Network driver cannot accept a connection on the port. System error = 233."
I look for information and I found that I should set this parameter: ndd -set /dev/tcp tcp_conn_request_max 4096
because by deafult it was set to 20.

One week later the error happens again, I notice that when happens the "connect requests dropped due to full queue" increase (netstat -p tcp).

I look for more information and I found that I shoul apply the patch PHNE_33395, but I'm not sure about it. Please if somebody can help me I'll appreciate it too much.

[RAC_1]

System error code 233 is as follows. So you could be out of buffer space/swap space etc.

# define ENOBUFS 233 /* No buffer space available */

Increasing tcp_Conn_request_max would not serve much. It is just applicable to max. no. of outstanding inbound connection request.

Look at total memeory utilization on your system, swap space. Also make sure that your are up to date on patches.

Anil

[Duda_1]

Wich parameters do you recomended to tune?

[RAC_1]

you should start by looking at what parameters to tune. First check what is causing the problem. How you have setup the buffer cache? static or dynamic.

glance -m will give those details.
then check how it is being used. again glance -m and sar -b 2 10 will help you with that.

swapinfo -mat will tell swap usage.

Depending upon the findings you may need to increase swap/buffer cahe settings.

Also, as said earlier, the patches.

[rick jones]

While the system may indeed be low on memory, it has been my experience that 99 times out of ten (yes You read that right - a saying I learned from a sports teacher long long ago :) an ENOBUFS on an accept() call simply means that by the time the local application got around to calling accept() the remote client gave-up and closed his connection (a TCP connection is actually "complete" before accept() is called)

That being the case, it suggests you either need clients with greater patience, or faster service in the DB.

Increasing tcp_conn_request_max is only indicated when you see connections dropped due to full queue in the output of netstat -p tcp, and even then, as already mentioned, it would only affect the depth of the listen (pending connection) queue (length being the minimum of tcp_conn_request_max and what the application uses in its listen() call)

[D Block 2]

hare u getting FIN_WAIT also ?

netstat -an |grep FIN

[rick jones]

Tom, I'm curious, what relationship do you see between TCP connections in FIN_WAIT_* states and ENOBUFS on accept()?

[Duda_1]

this is the output of: netstat -an | grep tcp| awk '{print $6}'| sort | uniq -c

315 CLOSE_WAIT
1770 ESTABLISHED
320 FIN_WAIT_2
65 LISTEN
49 TIME_WAIT

[Duda_1]

this is the output of: netstat -an | grep tcp| awk '{print $6}'| sort | uniq -c

315 CLOSE_WAIT
1770 ESTABLISHED
320 FIN_WAIT_2
65 LISTEN
49 TIME_WAIT

[D Block 2]

Duda,


my system w/ informix - running Your command at this moment shows no FIN WAITS...

netstat -an | grep tcp | awk '{ print $6 }' | sort | uniq -c
1292 ESTABLISHED
80 LISTEN
98 TIME_WAIT

You might want to study the client connections. And study how they close, this might be the issue. If the client uses a Browser and closes the window, you will get the FIN WAITS.

How about netstat ?

How many "oninit" is your system running ?
ps -efx |grep oninit

note, these processes have the same "open" socket connections. You can run LSOF on your system and Learn or study.

You will see that "oninit" processes have the same socket connections open for the TCP file numbers. So, there is many OINITS that have the same socket files open.

Question: talk with your DBA, and mention that there is a large increase workload, and that you suggest to lower the Engines on the oninit (or the number of CPUs).

This lower the # of engines based on the number actual system # CPUs, this might help your system performance.

[Steve Lewis]

As Rick said, your DB cannot respond to the connection requests quickly enough. Try the following Informix tuning recommendations:

1. Increase the number of soctcp threads in your DB.
2. reduce the number of connections each soctcp thread services e.g. change 2,100 to 4,50
3. Put your soctcp threads in a VP type of NET, not CPU.
4. Put local connections on a shared memory interface (type ipcshm) , to reduce the number of socket connections.
5. Get faster processors in your computer.

I like to have at least one NET VP poll thread per inbound network interface. They all listen on *.*, but somehow it seems to spread the load.

[Tim D Fulford]

off in a different direction... check kernel paramter, especially semmu (semaphore undo structures >128 and possibly ~512 from memory)... also check for other error messages in online.log I've see similar problems in the past.

Regads

Tim

[rick jones]

Duda's FIN_WAIT_2's are _probably_ nothing more than clients doing abortive TCP closes, or perhaps clients that are "sitting on" the connection close notification represented by the FIN they have received from the server. That number of connections is really nothing difficult for TCP to deal with.