topic Re: strange effect - processes hanging in Operating System - HP-UX

strange effect - processes hanging

Armin Schneider — Wed, 23 Feb 2005 08:48:36 GMT

Hello,

On two of our machines I face a strange phenomen:
we use a script calling startrfc (a SAP binary) to log on to sap and calling a function module inside SAP. Sometimes the script is "hanging" for several seconds (10 to 50). I traced the script using tusc/ truss

this is the output: (on both machines identical)

.....
08:11:01 writev(6, 0x800003ffff7f2890, 2) ........................................... = 513
iov_base[0]: 0x80000001000a66e1
iov_len[0]: 4
iov_base[1]: 0x8000000100138e48
iov_len[1]: 509
08:11:01 sigvec(SIGPIPE, 0x800003ffff7f25f0, 0x800003ffff7f2600) .................... = 0
vec.sv_handler: 0x1
vec.sv_mask: EMPTY SET
vec.sv_flags: SV_RESETHAND
ovec.sv_handler: 0x1
ovec.sv_mask: EMPTY SET
ovec.sv_flags: SV_RESETHAND
08:11:01 recv(6, 0x80000001000a66d5, 4, 0) .......................................... ERR#11 EAGAIN
08:11:03 select(7, 0x800003ffff7f2d90, NULL, NULL, NULL) ............................ [sleeping]
nfds: 7
readfds: 6
writefds: NULL
errorfds: NULL
timeout: NULL
08:11:13 select(7, 0x800003ffff7f2d90, NULL, NULL, NULL) ............................ = 1
nfds: 7
readfds: 6
writefds: NULL
errorfds: NULL
timeout: NULL
08:11:13 recv(6, "\0\0\n\t", 4, 0) .................................................. = 4
08:11:13 recv(6, "06cb02\0ffff\0\0\0\0\0\0\0\0\0\0".., 2569, 0) ..................... = 2569
08:11:13 gettimeofday(0x800003ffff7f18d8, 0x800003ffff7f18e8) ....................... = 0

......

Parameter nproc can't be the cause of the problem - it's monitored all time.
maybe parameter maxuprc ???

what do the experts think - where is the problem for "ERR#11 EAGAIN" ?

thanks

armin

Re: strange effect - processes hanging

Armin Schneider — Wed, 23 Feb 2005 08:54:00 GMT

additional information:

both machines are running HP-UX 11.11.
One is a L3000 the other one is a RP4440.

Re: strange effect - processes hanging

RAC_1 — Wed, 23 Feb 2005 09:02:44 GMT

The error 11 is as follows.
#define EAGAIN 11 /* No more processes */

When you start this program, simutaneously, run sar -v 5 5 on other terminal and watch. Also it would be better, to do ps -u "user_you_use_to_start_program"|wc -l to check if it is hitting maxuprc or not.
Also from the shell you start the program, do ulimit -Sa and ulimit -Ha

Anil

Re: strange effect - processes hanging

Steven E. Protter — Wed, 23 Feb 2005 09:03:46 GMT

maxuprc is the maximum processes allowed for a user. The default is 75. That is not enough for ias and oracle. This could be the case here for you.

If user schmo tries to open more than 75 processes it stops working at proces 77. I don't know why there is slack.

My first idea here would be to set maxuprc at 200, which will require a boot. Then see if the error is still ocurring.

SEP

Re: strange effect - processes hanging

Jeff Schussele — Wed, 23 Feb 2005 09:04:08 GMT

Hi Armin,

Yes - if you're not hitting the NPROC limit then I'd say it's probably maxuprc.
If this is a SAP system then maxuprc should be well into the thousands.

Rgds,
Jeff

Re: strange effect - processes hanging

Armin Schneider — Wed, 23 Feb 2005 09:09:11 GMT

ok - here is the output of sar -v 5 5

nproc is far away ....

15:06:15 text-sz ov proc-sz ov inod-sz ov file-sz ov
15:06:20 N/A N/A 526/8000 0 2246/18000 0 57550/90010 0
15:06:25 N/A N/A 527/8000 0 2248/18000 0 57568/90010 0
15:06:30 N/A N/A 526/8000 0 2248/18000 0 57574/90010 0
15:06:35 N/A N/A 526/8000 0 2247/18000 0 57591/90010 0
15:06:40 N/A N/A 526/8000 0 2246/18000 0 57601/90010 0

ok .... maxuprc is only 800 ....
lifting it up to 5000 should solve the problem ? what do you think ?

Re: strange effect - processes hanging

Jeff Schussele — Wed, 23 Feb 2005 09:12:56 GMT

Yes - I'd say 4096 would probably suffice.

Rgds,
Jeff

Re: strange effect - processes hanging

Armin Schneider — Wed, 23 Feb 2005 09:35:25 GMT

set maxuprc to 4096 (could be done dynamically)

but problem still persists

15:28:34 sigvec(SIGPIPE, 0x800003ffff7f25f0, 0x800003ffff7f2600) ......................................... = 0
vec.sv_handler: 0x1
vec.sv_mask: EMPTY SET
vec.sv_flags: SV_RESETHAND
ovec.sv_handler: 0x1
ovec.sv_mask: EMPTY SET
ovec.sv_flags: SV_RESETHAND
15:28:34 recv(6, 0x80000001000a66d5, 4, 0) ............................................................... ERR#11 EAGAIN
15:28:36 select(7, 0x800003ffff7f2d90, NULL, NULL, NULL) ................................................. [sleeping]
nfds: 7
readfds: 6
writefds: NULL
errorfds: NULL
timeout: NULL
15:28:43 select(7, 0x800003ffff7f2d90, NULL, NULL, NULL) ................................................. = 1
nfds: 7
readfds: 6
writefds: NULL
errorfds: NULL
timeout: NULL
15:28:43 recv(6, "\0\0\n\t", 4, 0) ....................................................................... = 4

Re: strange effect - processes hanging

Armin Schneider — Wed, 23 Feb 2005 09:36:48 GMT

I have to avoid this [sleeping] and wasting time ....
the problem is, there is a interface using the same way of communication and this interface is production critical ....

Re: strange effect - processes hanging

Steven E. Protter — Wed, 23 Feb 2005 09:39:25 GMT

You might want to contact SAP tech support.

kmtune | grep proc

Make sure everything is set logically.

Also, where is the box with regards to patching? There may be a patch for this problem.

SEP

Re: strange effect - processes hanging

Armin Schneider — Wed, 23 Feb 2005 09:47:07 GMT

vb_rcv_NEU:/tmp/NIPING/log#kmtune | grep proc
max_thread_proc 64 - 64
nproc 8000 - 8000

SAP Support is involved but they act like they don't want to help ....

Re: strange effect - processes hanging

A. Clay Stephenson — Wed, 23 Feb 2005 10:27:18 GMT

This has nothing to do with maxuprc. What you need to do is a man 2 recv and look at the reason the recv() system call can set errno to EAGAIN. The man page explains the reason(s) exactly

Re: strange effect - processes hanging

Armin Schneider — Wed, 23 Feb 2005 10:51:09 GMT

extract of man page

If recv(), recvfrom(), or recvmsg() fails, errno is set to one of the
following values.

[EAGAIN] Non-blocking I/O is enabled using O_NONBLOCK flag with fcntl() and the receive operation would block, or the socket has an error that was set asynchronously. An asynchronous error can be caused by a gateway failing to forward a datagram because the datagram exceeds the MTU of the next-hop network and the "Don't Fragment" (DF) bit in the datagram is set. (See SO_PMTU in getsockopt(2).)

so far so good ... but what can be done to solve the problem ?

Re: strange effect - processes hanging

A. Clay Stephenson — Wed, 23 Feb 2005 11:08:10 GMT

Now you look for any fcntl() system calls and see if the O_NONBLOCK bit is set. (O_NONBLOCK is defined in /usr/include/sys/fcntl.h) You will also find that most support organizations give you far more respect and better response when you can talk system calls and errno values with them because you have tremendously narrowed the scope of the problem.