inetd ::accept: No buffer space available

[Sunil.M.S]

Hi Guys,

We are having a rp 2470 server loaded with hpux 11.00 patch level sep 2002, hosting informix 7.31 database. This server is working as backend server for card based online transaction.One of the application binary is listening in the predefined port( configured in inetd.conf) for any card swipe hit and giving authorization to the transaction system.

Now a days we are facing the problem of some bad transactions ie 100 per day out of 16000 thousand transaction happening.Further investigations happened on this issue giving some clue as mentioned below.

1) I am getting the "inetd ::accept: No buffer space available" error in syslog file max 10 times a day at random times.As shown in itrc we suspected ndd parameters, Earlier the ndd parameter "tcp_conn_request_max" was default value 20 and we tried different higher values up to 16384 , and every time we executed command inetd -c after changing the parameter also.But still we are getting the same error in syslog and bad transaction continues.


2) The command netstat -sp tcp is giving the message "494 connect requests dropped due to full queue" .

I am not able to find out whether this problen due to any wrongly configured network parameter or application binary problem.

Thanks in advance

Regards
Sunil

[Muthukumar_5]

Refer this:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=894793

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=803600

Restart inetd as,

# /sbin/init.d/inetd stop
# /sbin/init.d/inetd start

inetd -c is used to reconfigure not for restarting.

It will solve the problem.

-Muthu

[Arunvijai_4]

You may need tune kernel/network parameters. Check the above threads given by Muthukumar for a good solution.

-Arun

[RAC_1]

This error could be on account of many things. Could be swap space, buffer cache, ndd tcp/ip settings. to start with look at following ndd tcp parameters.

tcp_recv_hiwater_def - Maximum receive window size
tcp_recv_hiwater_max - Upper bound on TCP receive buffer size
tcp_recv_hiwater_lfp - Maximum receive window size for fast links
tcp_recv_hiwater_lnp - Maximum receive window size for slow links

tcp_conn_request_max is outstanding max. requests and a value of 16384 is quite generous. for chaning above ndd parameters, do one chnage at a time, check logs and so on.

Also things to check.
Any other error message??
Buffer cache setting and use-glance -m, sar -b
swapinfo -mat

[AwadheshPandey]

link:
1)http://www.exim.org/pipermail/exim-users/Week-of-Mon-20000501/017767.html
http://archives.neohapsis.com/archives/postfix/2002-02/1952.html
2)
http://www.hp.com/products1/unix/java/infolibrary/tools/other_tools.ppt

Awadhesh

[Sunil.M.S]

Hi Thanks for your replys,

Muthu:Prior to put the query in itrc I had gone through the mentioned links and tried that.By mistake I thought inetd -c will take the changed configuration in to effect.Since the server is in production environment at present I won't be able to stop and start inetd services,I will do it later.

RAC:From the mentioned symptmes, I am not in a position to diagnose the root cause for the problem.I have checked the ndd parameters you mentioned in the replay and in my system the parameters are
tcp_recv_hiwater_def 32768
tcp_recv_hiwater_lfp 65536
tcp_recv_hiwater_lnp 8192

But i couldn't find the parameter tcp_recv_hiwater_max in the system.

I am not clear about the significance of these parameters and not sure about changing this value will help me out or not.

The command output of sar and swapinfo look ok in the system.I am not finding much constrain about those parameters.

Pls suggest

[RAC_1]

The values you have are the default values.
I have a felling that send and receive buffer sizes are not correctly set by your apps and hence a problem.

Increasing the values one by one and check if error returns. Change one parameter at a time to pinpoint the error.

If you serach the tecnical notes for tcp buffer settings, you will get a document that has all these details. (I think doc id is KBQA00000120)

You can get the help of each parameter as follows.

ndd -h "parameter"

[RAC_1]

Check this thread. It also talks about tcp_con_strategy parameter.

http://www.dutchworks.nl/htbin/hpsysadmin?h=3&dn=86591&q=connect%20requests%20dropped%20due%20to%20full%20queue&fh

[Steve Lewis]

I suggest you remove your database from the tcp stack by setting the connection mechanism to ipcshm in your sqlhosts file.

This will free up buffer space for real network connections as well as making the database go quicker.

This type of connection only works for processes local to the server.
Speak to your DBA about configuring this.

[Kent Ostby]

Also, check your nettl.LOG* files.

This can also be caused by a duplicate IP address.

[Sameer_Nirmal]

Well , in this case the issue with application couldn't be ruled out. The occurances of inetd error message could be ignored being transient if they are few. But in this case , this is a serious problems as the transactions are getting dropped and occurances are more which couldn't be ignored at all.
Moreover if the application is not taking care of killing the network processes which are no longer needed , such problems with TCP buffers are bound to happen.
I would refer the following docs and the KBQA00000120 as referred by RAC to judge the
application behaviour and adjust tcp parameters if required.

http://www1.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000079989991

http://www1.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000080361172

But I do believe , application tunning would be required since the application is listening all the time at the pre-defined tcp port.

[Jim Keeble]

See the knowledge article WTECKBRC00005581. The inetd error is a result of a change in accept() behaviour in 11.0 of hp-ux. The cause is that a client has reset it's connection to inetd before inetd has called accept(). In 10.20, the tcp stack dropped these "abandoned" connections silently, and the listening application never knew. Now, the default behaviour is to deliver the ENOBUFS to the application.

It is possible to emulate the 10.20 behaviour by setting the ndd parameter "tcp_early_conn_ind" to 0. The effect is that the tcp stack holds the clients connection until the complete 3 way tcp connection has completed, dropping those that are reset prematurely.

The bad transactions may truly be related to the tcp_conn_request_max parameter and the "dropped connection" stat, but it is a problem seperate from that of inetd. (Related possibly, but not directly.) I think Informix uses it's own listener process, and so wouldn't affected by inetd -c or stop/start of inetd. I think the informix listener would need to be stopped and restarted in order to take advantage of the tcp_conn_request_max change.

I hope this helps.