Kernel Logging on HPUX 11i V1

[Mark_99_1]

On HPUX 11i V1.
Is it possible to log when a normal user reaches a kernel limit, ie.
if nproc is set to 200 and the user tries to run 201 processes ?

Glance is running on this server, but it only records the overall kernel usage and not for individual userid's. I have gone through the Glance metrics and cannot find anything that might help.

[Dennis Handly]

I'm not aware of any logging when you exceed the limit, other than fork will fail.

[Bill Hassell]

> Is it possible to log when a normal user reaches a kernel limit, ie.
if nproc is set to 200 and the user tries to run 201 processes ?

When *any* process tries to start when the proc table is at 200, there will be an error message like this in syslog (/var/adm/syslog/syslog.log)

proc: table is full

However, no one can read this file until the process count is reduced because no new processes can be started. nproc is a system-wide limit and not even root can login when the process table is full. nproc should be aggressively resized in order to prevent application failures and possible database corruption.

You can monitor how many processes are running and the nproc limit with:

sar -v 1 5

When the current proc count is about 60-70%, then it is time to double or triple nrpoc and rebuild the kernel. You could increase it by 10 or 20 but this just means you'll have to do it again (forcing a kernel rebuld and reboot). If it is really 200, make it 1000 as soon as possible. This will grow the kernel size by a few megs, but considering the consequences, this is a mandatory change.

[Mark_99_1]

Fork is the only message im aware of and doesn't get logged in the syslog.
Because its a normal user id and doesn't fill up all resourceses it doesn't get logged in the syslog.

There is an option in SAM to enable kernel logging but doesn't work. On further investigation it appears to be just a shell and requires coding behind it ( not sure why HP did this ?? ). I found a document saying i needed to email the WTEC group - which i did. The email came back as a mail failure. I logged a call with HP who advise the group was disbanded a few years ago.
HP also adivsed there is no way to log kernel usage for normal users.
I'll keep this question open for another 24 hours to see if some HP Guru can come up with a solution.

The reason for the question.
One of the application processes dies every now and then - no pattern. On doing an extract i can see nothing untoward happening.
There is no debug option in the application and the error message it produces is so vague its uselesss.
No error in any of the system logs.
This app has been runing fine for about 4 years. The only change made was that it was moved from being owned by root to a normal user.
The only thing i can think of is its reaching some kernel limit.
The kernel paramters are set to what the app requires but im thinking of increasing maxusers as this isn't mentioned in the kernel requirements for the application.

[nightwich]

Hi Mark


Try to use ulimit in the user.

Regards.

[Mark_99_1]

I didn't think ulimit was able to exceed kernel paramteters - please correct me if im wrong.

[nightwich]

yes you are right.


I do not think that is a kernel issue. I think is a user limit.


Have you run the command suggested

var -v 1 5


What is the messages that show up in syslog ??

Regards.

[Mark_99_1]

Hi,
$ ulimit -a
time(seconds) unlimited
file(blocks) 4194302
data(kbytes) 1048576
stack(kbytes) 262144
memory(kbytes) unlimited
coredump(blocks) 4194303
nofiles(descriptors) 2048

$ sar -v 1 5

HP-UX test1 B.11.11 U 9000/800 07/06/10

13:04:24 text-sz ov proc-sz ov inod-sz ov file-sz ov
13:04:25 N/A N/A 305/3968 0 2494/4656 0 7343/23993 0
13:04:26 N/A N/A 305/3968 0 2494/4656 0 7343/23993 0
13:04:27 N/A N/A 304/3968 0 2494/4656 0 7343/23993 0
13:04:28 N/A N/A 304/3968 0 2494/4656 0 7343/23993 0
13:04:29 N/A N/A 304/3968 0 2494/4656 0 7343/23993 0

There are no errors in syslog. Nothing to indicate a problem.
The glance extract shows no table limit breached overall.

[Bill Hassell · Accepted solution]

> $ sar -v 1 5
> 13:04:24 text-sz ov proc-sz ov inod-sz ov file-sz ov
> 13:04:25 N/A N/A 305/3968 0 2494/4656 0 7343/23993 0

305 used out of 3968 ... no problems there. Now given the lack of symptoms from the application, I would suspect maxuprc as a possible culprit. maxuprc is a kernel parameter that is used to prevent runaway programs from using all the entries in the process table. Your application will usually receive an errno 11 (EAGAIN) but nothing will be logged as this is not a process table oveflow -- it is a local user limit. Use this command to see the current value:

kmtune -q maxuprc

This is the maximum number of processes that a single user can own at the same time. It is common for a user like oracle or sybase to have forks fail due to maxuprc. Use this command to find the number of processes owned by appuser:

ps -f -u appuser

where appuser is the new owner for the application. root is not constrained by maxuprc. The good news is that maxuprc is dynamic and can be changed while the system is running. For example:

kmtune -u -s maxuprc=95

HP-UX 11.11 does not have man pages for kernel parameters but they were created for 11.23 and higher. Here is the maxuprc man page:

http://docs.hp.com/en/939/KCParms/KCparam.MaxUprc.html

For all the parameters in an interactive web page:

http://docs.hp.com/en/939/KCParms/KCparams.OverviewAll.html

[Mark_99_1]

I agree with the maxuprc and currently have a change request going through to double it.
It would just be nice to see this limit in a log so i can say "This is the problem" rather than a "i think it might be this..."

[chris huys_4]

Hi,

> I agree with the maxuprc and currently have
> a change request going through to double it.
> It would just be nice to see this limit in a
> log so i can say "This is the problem"
> rather than a "i think it might be this..."

Shouldnt be to difficult to verify. Just put a ps -ef/lsof, checking every process, in cron, to be run, every f.e. 15 minutes.

When the application then "dies", check how many threads were forked, just before it died, and see if its near the amount of the maxuprc or any other related kernel parameter.

If the app dies a few times and the numbers are "near" and after the maxuprc has increased the "numbers" increase over this "number", you can be pretty sure that you found the problem.

Also Im not sure what you mean with "on doing the extract", but the application would surely leave a corefile, when it dies and so should give at least a reason, file core, why it died ?

Greetz,
Chris

[Mark_99_1]

The problem is capturing the problem and if there is anything that leads upto that problem or if it just happens.
From glance the tables are not near their limit but as the tables relate to the whole system, it dones't mean a user has reached a limit.
I was looking at running something like ps -ef|grep user|wc -l every minute.

Unfortunately, lsof is not something that comes standard with HPUX so i am not allowed to install it. This, to me, is one of the most useful commands ever invented - ranks alongside Dtrace.

extract = glance command.
The app doesn't leave a core file :-(

[Dennis Handly]

>the error message it produces is so vague it's uselesss.

Perhaps include it here?

>The app doesn't leave a core file

Any chance of modifying the application to log more info? Is it C or aC++?
Or call abort(3) to get a corefile? Or call U_STACK_TRACE to get a stack trace?

[Dennis Handly]

>lsof is not something that comes standard with HP-UX so I am not allowed to install it.

tusc is also a good tool to monitor system calls and find the last ones that fail.

Which is more important to your management, finding the solution to your problem or your security policies?

[Dennis Handly]

It looks like you need what Tru64 logs: :-)
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1438010

[chris huys_4]

> Unfortunately, lsof is not something that
> comes standard with HPUX

> The app doesn't leave a core file :-(

Then your management shouldnt have allowed a app that leaves no core file to be installed on a production system. ;)

Greetz,
Chris

[Mark_99_1]

Thanks for all your replies.
I agree with what you say about management. They want it fixed but won't give me the tools to do it!

[Mark_99_1]

Haven't been able to find the solution yet but see the only forward is to up the maxuprc limit and get debugging enabled on the application.