Kernel Logging on HPUX 11i V1

[Mark_99_1]

On HPUX 11i V1.
Is it possible to log when a normal user reaches a kernel limit, ie.
if nproc is set to 200 and the user tries to run 201 processes ?

Glance is running on this server, but it only records the overall kernel usage and not for individual userid's. I have gone through the Glance metrics and cannot find anything that might help.

[Dennis Handly]

I'm not aware of any logging when you exceed the limit, other than fork will fail.

[Bill Hassell]

> Is it possible to log when a normal user reaches a kernel limit, ie.
if nproc is set to 200 and the user tries to run 201 processes ?

When *any* process tries to start when the proc table is at 200, there will be an error message like this in syslog (/var/adm/syslog/syslog.log)

proc: table is full

However, no one can read this file until the process count is reduced because no new processes can be started. nproc is a system-wide limit and not even root can login when the process table is full. nproc should be aggressively resized in order to prevent application failures and possible database corruption.

You can monitor how many processes are running and the nproc limit with:

sar -v 1 5

When the current proc count is about 60-70%, then it is time to double or triple nrpoc and rebuild the kernel. You could increase it by 10 or 20 but this just means you'll have to do it again (forcing a kernel rebuld and reboot). If it is really 200, make it 1000 as soon as possible. This will grow the kernel size by a few megs, but considering the consequences, this is a mandatory change.

[Mark_99_1]

Fork is the only message im aware of and doesn't get logged in the syslog.
Because its a normal user id and doesn't fill up all resourceses it doesn't get logged in the syslog.

There is an option in SAM to enable kernel logging but doesn't work. On further investigation it appears to be just a shell and requires coding behind it ( not sure why HP did this ?? ). I found a document saying i needed to email the WTEC group - which i did. The email came back as a mail failure. I logged a call with HP who advise the group was disbanded a few years ago.
HP also adivsed there is no way to log kernel usage for normal users.
I'll keep this question open for another 24 hours to see if some HP Guru can come up with a solution.

The reason for the question.
One of the application processes dies every now and then - no pattern. On doing an extract i can see nothing untoward happening.
There is no debug option in the application and the error message it produces is so vague its uselesss.
No error in any of the system logs.
This app has been runing fine for about 4 years. The only change made was that it was moved from being owned by root to a normal user.
The only thing i can think of is its reaching some kernel limit.
The kernel paramters are set to what the app requires but im thinking of increasing maxusers as this isn't mentioned in the kernel requirements for the application.

[nightwich]

Hi Mark


Try to use ulimit in the user.

Regards.

[Mark_99_1]

I didn't think ulimit was able to exceed kernel paramteters - please correct me if im wrong.

[nightwich]

yes you are right.


I do not think that is a kernel issue. I think is a user limit.


Have you run the command suggested

var -v 1 5


What is the messages that show up in syslog ??

Regards.

[Mark_99_1]

Hi,
$ ulimit -a
time(seconds) unlimited
file(blocks) 4194302
data(kbytes) 1048576
stack(kbytes) 262144
memory(kbytes) unlimited
coredump(blocks) 4194303
nofiles(descriptors) 2048

$ sar -v 1 5

HP-UX test1 B.11.11 U 9000/800 07/06/10

13:04:24 text-sz ov proc-sz ov inod-sz ov file-sz ov
13:04:25 N/A N/A 305/3968 0 2494/4656 0 7343/23993 0
13:04:26 N/A N/A 305/3968 0 2494/4656 0 7343/23993 0
13:04:27 N/A N/A 304/3968 0 2494/4656 0 7343/23993 0
13:04:28 N/A N/A 304/3968 0 2494/4656 0 7343/23993 0
13:04:29 N/A N/A 304/3968 0 2494/4656 0 7343/23993 0

There are no errors in syslog. Nothing to indicate a problem.
The glance extract shows no table limit breached overall.

[Bill Hassell · Accepted solution]

> $ sar -v 1 5
> 13:04:24 text-sz ov proc-sz ov inod-sz ov file-sz ov
> 13:04:25 N/A N/A 305/3968 0 2494/4656 0 7343/23993 0

305 used out of 3968 ... no problems there. Now given the lack of symptoms from the application, I would suspect maxuprc as a possible culprit. maxuprc is a kernel parameter that is used to prevent runaway programs from using all the entries in the process table. Your application will usually receive an errno 11 (EAGAIN) but nothing will be logged as this is not a process table oveflow -- it is a local user limit. Use this command to see the current value:

kmtune -q maxuprc

This is the maximum number of processes that a single user can own at the same time. It is common for a user like oracle or sybase to have forks fail due to maxuprc. Use this command to find the number of processes owned by appuser:

ps -f -u appuser

where appuser is the new owner for the application. root is not constrained by maxuprc. The good news is that maxuprc is dynamic and can be changed while the system is running. For example:

kmtune -u -s maxuprc=95

HP-UX 11.11 does not have man pages for kernel parameters but they were created for 11.23 and higher. Here is the maxuprc man page:

http://docs.hp.com/en/939/KCParms/KCparam.MaxUprc.html

For all the parameters in an interactive web page:

http://docs.hp.com/en/939/KCParms/KCparams.OverviewAll.html

[Mark_99_1]

I agree with the maxuprc and currently have a change request going through to double it.
It would just be nice to see this limit in a log so i can say "This is the problem" rather than a "i think it might be this..."