MSG send failed, queue potentially full errorno=11

[MAD_2]

The following messages are appearing on our server's console:

1: "Going to do a blocked sndmsg"
2: "MSG send failed, queue potentially full errorno=11"

These messages keep appearing on our console (An RP5470 running on HP-UX 11.0). The problem is that after some time, one of our telephony server application processes stops responding, and that affects our clients.

Any ideas of what may be the source of the problem? Thanks!

[Bill Hassell]

Sounds like your kernel parameters are way too low. errno=11 is:

from /usr/include/sys/errno.h:
#define EAGAIN 11 /* No more processes */

from man 2 errno:
[EAGAIN] Resource temporarily unavailable. This is likely a temporary condition, and later calls to the same routine may complete normally.

Look at the end of /var/adm/syslog/syslog.log and you'll probably see a bunch of additional messages. Start by running:

sar -v 1

Look at file-sz and proc-sz. The firsdt number is used and the second number is max. If either parameter is within 80% of the max, you need to regen the kernel to bump up nfile and nproc--double them to save time.

If these values are OK, then your application authors need to provide a better error message since the text is unique to the application.

[MAD_2]

Thanks for your response Bill. Will you please define the following:

proc-sz & file-sz. What do they represent?

Below is the output from a "sar -v 3 3". Since we have some application usage peak times, I need to watch what happens during those peak times, maybe then I will see those values reaching 80%.


14:38:20 text-sz ov proc-sz ov inod-sz ov file-sz ov
14:38:23 N/A N/A 192/8212 0 1444/20968 0 2325/18445 0
14:38:26 N/A N/A 192/8212 0 1442/20968 0 2326/18445 0
14:38:29 N/A N/A 192/8212 0 1438/20968 0 2326/18445 0

[MAD_2]

OK, here are even more details on these errors:

The messages appearing are:

1: "Going to do a blocked sndmsg"
2: "MSG send failed, queue 1603 potentially full errorno=11"

1: "Going to do a blocked sndmsg"
2: "MSG send failed, queue 854 potentially full errorno=11"

By using "ipcs" command I get:

# ipcs | grep aba
q 1603 0xaba00025 --rw-rw-rw- edge edge
q 854 0xaba00021 -Rrw-rw-rw- edge edge
m 9480 0xaba00010 --rw-rw-rw- edge edge

How can I adjust these parameters so that these queues are not filled so quickly?

[MAD_2]

And here is more information regarding largest-allowabable size for queue limits and current outstanding usage:

IPC status from /dev/kmem as of Sat May 3 15:18:49 2003
T ID KEY MODE OWNER GROUP CBYTES QNUM QBYTES
Message Queues:
q 0 0x3c1c0249 -Rrw--w--w- root root 0 0 16384
q 1 0x3e1c0249 --rw-r--r-- root root 0 0 264
q 2 0x6154000f --rw-rw---- edge edge 0 0 16384
q 1603 0xaba00025 --rw-rw-rw- edge edge 13948 25 16384
q 854 0xaba00021 -Rrw-rw-rw- edge edge 0 0 16384

[MAD_2]

OK, I believe I found the source of the problem... It does appear to be kernel parameters (specifically msgmnb):

Current Pending Associated
Name Value Value Type Module
????????????????????????????????????????????????????????????????????????????????????????????????????????????
msgmnb 16384 16384 Static N/A M
msgssz 8 8 Static N/A M


This, compared to how it was set to an old system of ours, it appears to be too low. On our previous production system it was set as follows:

Current Pending Associated
Name Value Value Type Module
????????????????????????????????????????????????????????????????????????????????????????????????????????????msgmnb 65536 65536 Static N/A M
msgssz 256 256 Static N/A M

I will be changing the values whenever we have some down time.

[Bill Hassell]

The numbers shown in sar -v are just fine:

nfile = 2325/18445

which means 2325 open files out of a maximum of 18445...plenty of extra file handles

nproc = 192/8212

means 192 processes running out of a maximum of 8212, again plenty of room.

The interprocess message queues appear to the the problem. Since the requirements to use message queues (or other interprocess methods) are very application dependent, the installation instructions should be very clear about kernel parameter requirements.

[Steven E. Protter]

Another case where shared memory requirements need to be met.

It can actually be frustrating, because a lot of application vendors don't bother providing instructions on what to do with shared memory resources on HP-UX.

I'd go through your system, inventory what you have running and then look at the kernel parameters mentioned above, plus shmmax shmseg etc and be extra generous in that area.

I'm running a bunch of systems with some very shared memory and ipc intensive applications. The application vendors commonly test HP-UX on clean systems without other products running. Many times I've run into issues where Oracle ate up resources needed by Software AG's adabas or the two products would not run at the same time and play nice together.

By following one or the other application vendors kernel recommendations, I got nowhere because I needed to add up the minimums from both manuals and go with more generous figures than that. Throw in apache which uses shared memory/ipc resources and the Systems Administrator really starts to earn his paycheck.

My rule, unscientific as it is, is be generous and then add 50% for growth.

SEP

[Terry Koob]

Adam,
How did changing those parameters work for you. I'm having the same problem.

Thanks