strange effect - processes hanging

[Armin Schneider]

Hello,

On two of our machines I face a strange phenomen:
we use a script calling startrfc (a SAP binary) to log on to sap and calling a function module inside SAP. Sometimes the script is "hanging" for several seconds (10 to 50). I traced the script using tusc/ truss

this is the output: (on both machines identical)

.....
08:11:01 writev(6, 0x800003ffff7f2890, 2) ........................................... = 513
iov_base[0]: 0x80000001000a66e1
iov_len[0]: 4
iov_base[1]: 0x8000000100138e48
iov_len[1]: 509
08:11:01 sigvec(SIGPIPE, 0x800003ffff7f25f0, 0x800003ffff7f2600) .................... = 0
vec.sv_handler: 0x1
vec.sv_mask: EMPTY SET
vec.sv_flags: SV_RESETHAND
ovec.sv_handler: 0x1
ovec.sv_mask: EMPTY SET
ovec.sv_flags: SV_RESETHAND
08:11:01 recv(6, 0x80000001000a66d5, 4, 0) .......................................... ERR#11 EAGAIN
08:11:03 select(7, 0x800003ffff7f2d90, NULL, NULL, NULL) ............................ [sleeping]
nfds: 7
readfds: 6
writefds: NULL
errorfds: NULL
timeout: NULL
08:11:13 select(7, 0x800003ffff7f2d90, NULL, NULL, NULL) ............................ = 1
nfds: 7
readfds: 6
writefds: NULL
errorfds: NULL
timeout: NULL
08:11:13 recv(6, "\0\0\n\t", 4, 0) .................................................. = 4
08:11:13 recv(6, "06cb02\0ffff\0\0\0\0\0\0\0\0\0\0".., 2569, 0) ..................... = 2569
08:11:13 gettimeofday(0x800003ffff7f18d8, 0x800003ffff7f18e8) ....................... = 0

......

Parameter nproc can't be the cause of the problem - it's monitored all time.
maybe parameter maxuprc ???

what do the experts think - where is the problem for "ERR#11 EAGAIN" ?

thanks

armin

[Armin Schneider]

additional information:

both machines are running HP-UX 11.11.
One is a L3000 the other one is a RP4440.

[RAC_1]

The error 11 is as follows.
#define EAGAIN 11 /* No more processes */


When you start this program, simutaneously, run sar -v 5 5 on other terminal and watch. Also it would be better, to do ps -u "user_you_use_to_start_program"|wc -l to check if it is hitting maxuprc or not.
Also from the shell you start the program, do ulimit -Sa and ulimit -Ha

Anil

[Steven E. Protter]

maxuprc is the maximum processes allowed for a user. The default is 75. That is not enough for ias and oracle. This could be the case here for you.

If user schmo tries to open more than 75 processes it stops working at proces 77. I don't know why there is slack.

My first idea here would be to set maxuprc at 200, which will require a boot. Then see if the error is still ocurring.

SEP

[Jeff Schussele · Accepted solution]

Hi Armin,

Yes - if you're not hitting the NPROC limit then I'd say it's probably maxuprc.
If this is a SAP system then maxuprc should be well into the thousands.

Rgds,
Jeff

[Armin Schneider]

ok - here is the output of sar -v 5 5

nproc is far away ....

15:06:15 text-sz ov proc-sz ov inod-sz ov file-sz ov
15:06:20 N/A N/A 526/8000 0 2246/18000 0 57550/90010 0
15:06:25 N/A N/A 527/8000 0 2248/18000 0 57568/90010 0
15:06:30 N/A N/A 526/8000 0 2248/18000 0 57574/90010 0
15:06:35 N/A N/A 526/8000 0 2247/18000 0 57591/90010 0
15:06:40 N/A N/A 526/8000 0 2246/18000 0 57601/90010 0

ok .... maxuprc is only 800 ....
lifting it up to 5000 should solve the problem ? what do you think ?

[Jeff Schussele]

Yes - I'd say 4096 would probably suffice.

Rgds,
Jeff

[Armin Schneider]

set maxuprc to 4096 (could be done dynamically)

but problem still persists

15:28:34 sigvec(SIGPIPE, 0x800003ffff7f25f0, 0x800003ffff7f2600) ......................................... = 0
vec.sv_handler: 0x1
vec.sv_mask: EMPTY SET
vec.sv_flags: SV_RESETHAND
ovec.sv_handler: 0x1
ovec.sv_mask: EMPTY SET
ovec.sv_flags: SV_RESETHAND
15:28:34 recv(6, 0x80000001000a66d5, 4, 0) ............................................................... ERR#11 EAGAIN
15:28:36 select(7, 0x800003ffff7f2d90, NULL, NULL, NULL) ................................................. [sleeping]
nfds: 7
readfds: 6
writefds: NULL
errorfds: NULL
timeout: NULL
15:28:43 select(7, 0x800003ffff7f2d90, NULL, NULL, NULL) ................................................. = 1
nfds: 7
readfds: 6
writefds: NULL
errorfds: NULL
timeout: NULL
15:28:43 recv(6, "\0\0\n\t", 4, 0) ....................................................................... = 4

[Armin Schneider]

I have to avoid this [sleeping] and wasting time ....
the problem is, there is a interface using the same way of communication and this interface is production critical ....

[Steven E. Protter]

You might want to contact SAP tech support.

kmtune | grep proc

Make sure everything is set logically.

Also, where is the box with regards to patching? There may be a patch for this problem.

SEP

[Armin Schneider]

vb_rcv_NEU:/tmp/NIPING/log#kmtune | grep proc
max_thread_proc 64 - 64
nproc 8000 - 8000


SAP Support is involved but they act like they don't want to help ....

[A. Clay Stephenson]

This has nothing to do with maxuprc. What you need to do is a man 2 recv and look at the reason the recv() system call can set errno to EAGAIN. The man page explains the reason(s) exactly

[Armin Schneider]

extract of man page

If recv(), recvfrom(), or recvmsg() fails, errno is set to one of the
following values.

[EAGAIN] Non-blocking I/O is enabled using O_NONBLOCK flag with fcntl() and the receive operation would block, or the socket has an error that was set asynchronously. An asynchronous error can be caused by a gateway failing to forward a datagram because the datagram exceeds the MTU of the next-hop network and the "Don't Fragment" (DF) bit in the datagram is set. (See SO_PMTU in getsockopt(2).)

so far so good ... but what can be done to solve the problem ?

[A. Clay Stephenson]

Now you look for any fcntl() system calls and see if the O_NONBLOCK bit is set. (O_NONBLOCK is defined in /usr/include/sys/fcntl.h) You will also find that most support organizations give you far more respect and better response when you can talk system calls and errno values with them because you have tremendously narrowed the scope of the problem.