Samba error: no buffer space available

[enrico.nic]

Hi

We have an HP 9000 rp3410, HP-UX 11.11.

Recently I am finding this error in the samba log file:
[2005/07/12 16:36:51, 0] smbd/server.c:open_sockets_smbd(382)
open_sockets_smbd: accept: No buffer space available

When this error comes up, the file sharing through samba begins to be very slow.

In the forum there is indication about a kernel parameter to be increased, i.e. tcp_conn_max_size, but I haven't found it in the configurable parameters of the kernel.
Attached is the output of the "kmtune" command.

Can anybody please help me ?

Thank you

Enrico

[Alessandro Pilati]

Enrico,
take a look for the "tcp_conn_request_max" tunable parameter and see if it is reasonable high ( 4096 for example )
You cannot find it among the kernel tunable parameters because it is a network tunable parameter.

This is the command to get info about it:
ndd -get /dev/tcp tcp_conn_request_max

This is the command to increase the value:
ndd -set /dev/tcp tcp_conn_request_max 4096

Then take a look if you have the following patches installed:

PHNE_21767
PHNE_21217


Hope this can help,
Regards, Alessandro

[Ermin Borovac]

You may be hitting nfile (global number of files) limit. Do you see any errors in syslog.log about file table overflow or being full?

You can monitor your file table usage with glance or sar. If you use sar column to look at is file-sz. There are two numbers (as in x/y): x is current usage, y should correspond to nfile.

$ sar -v 5 5

[enrico.nic]

Hi Alessandro

The parameter is already set to 4096

# ndd -get /dev/tcp tcp_conn_request_max
4096

And the two indicated patches refer to HP-UX 11.0 - anyway, these are patches related to ARPA transport (the first) and LAN (the second): for both of these "products", the latest patches for HP-UX 11.11 have already been applied.

I am trying the second option (kernel param)

Thank you

[enrico.nic]

nfile in the kernel configurable parameters seems to be right (4096): the suggested sar command says that around 700 files are open of the 4107 available - and no error like "file table is full" is present in syslog.log

Any other suggestion is welcome

Thank you

Enrico

[rick jones]

Since someone already mentioned that tcp_conn_request_max is an ndd setting, I'll simply add that the _actual_ depth of the listen queue will be the minimium of that and what Samba passes-in on its listen() call(s).

Further, I suspect that tcp_conn_request_max is not related to the ENOBUFS from accept. 99 times out of 10 and ENOBUFS on accept means that by the time the application has gotten around to calling accept() the remote client has given-up on the connection.

Tuning tcp_conn_request_max and the listen() parameter _is_ indicated when netstat -p tcp shows connection requests dropped due to full queue.

If nfile were too small, the error from accept would be ENFILE. If maxfiles/maxfiles_lim were being hit it would be EMFILE.

[Ermin Borovac]

From what I can see by looking at samba source code the size of listen queue is hardcoded to 50 in one of the header files. So it looks like tuning tcp_conn_request_max won't help you.

The suggestion about nfile was taken from HP's knowledge base article (document ID 8606401569; SYSLOG Errors: open_sockets_smbd: accept: No buffer space available) which hints that hitting nfile limit might be one of the possible causes.

I suggest that you check performance of your server (especially cpu and memory utilisation) with 'glance' when the problem occurs. If your server is running out of resources, this situation is likely to occur.

Also it's a good idea to check ARPA patch you have installed and update to the latest one.

Finally, I suggest that you tune your kernel parameter dbc_max_pct from 50 to a value from 5 to 10. This parameter determines how much memory will be used for buffer cache and default of 50 (percent) is overkill for most situations.

[Duda_1]

I hace the same problem with my Informix Database:

listener-thread: err = -25573: Network driver cannot accept a connecti
on on the port. System error = 233.

And oserr 233 is about ENOBUFS: "No buffer space available".
Did you resolve your problem? How did you do?

[enrico.nic]

The error is not solved (yet ?).
Furthermore, there is a new error after changing some "socket options" in the samba configuration file:
Aug 1 12:31:40 gissi smbd[17168]: Failed to set socket option SO_KEEPALIVE (E
rror Invalid argument)
Aug 1 12:31:40 gissi smbd[17168]: Failed to set socket option IPTOS_LOWDELAY
(Error Invalid argument)
Aug 1 12:31:40 gissi smbd[17168]: Failed to set socket option TCP_NODELAY (Er
ror Invalid argument)

[Fabio Ettore]

Ciao Enrico,

that message into syslog.log

[2005/07/12 16:36:51, 0] smbd/server.c:open_sockets_smbd(382)
open_sockets_smbd: accept: No buffer space available]

means there are problems about system resources on your system. Usually the reason is a nfile kernel parameter too small.
As Ermin posted it can be checked by sar but you should verify when you obtain the error how many users are connected on CIFS/Samba folders.
A thought: did this message happen suddenly? Or do you know that in last period more users have permissions to use CIFS/Samba folders?

Try to increase nfile parameter.


About new message with socket options the message that you obtain with

Invalid argument

seems to be clear that the format of entries in smb.conf is not correct. I suggest to comment them and increase nfile.

That message surely refers to a system resource problem.

Hope this helps you.

Best regards,
Fabio

[Rita C Workman]

Enrico,

Ermin has pointed, in the last sentence, to the first parm to look at...
reduce dbc_max_% immediately. Depending on memory on your server set this value DOWN. You need about 300-400Mb for smaller memory boxes and 700-800Mb for larger memory. So set this % so it would calculate down to the correct amount of memory to give to the system. Probably anything from 20 down to 5.
But reduce this immediate !

Next, yes increase your nfile (set at 4097) to maybe 10,000 and observe this using sar -v for open file usage.
Then, maybe decrease (yes...decrease) the ninode parm from the strange number you have to 2048. You might be able to go even lower like 1024. Again use that sar -v command to monitor this. This parm creates a table that assigns the ninodes-and these values are re-usable, so the entries keep recycling.

Your value for 1Gb on shmmax may be right, or you may have enough resources to extend this as well.
Take a look at your semmaphore and message queue settings. Since you don't mention what your running, can't comment on these too much. But if your appl uses semmaphores, bet your running short on them with these settings ! Again use the sar command sar -mS to give you some heads up on this..

Hope this helps get you started,
Rgrds,
Rita

[Ying Li_1]

I'd like to know if the issue has been resolved on your side. The reason I'm asking is we experienced the same problem and hope to share your experiences.

Thanks.
-Ying