ncallout table overflow

[Vijayakrishna]

The system is getting rebooted often with the message "Panic : callout table overflow" We recently increased nfile parameter from 10000 to 40000 because of flie table overflow issues. Now we are getting this problem. Fllowing are the kernel parameters.
nfile 40000
nproc 2596
ncallout 2612
nkthread 5000
Please suggest me what could be the reason for callout table over flow and what we need to analyze to get root cause.

Thank you all.
VK

[Ashwani Kashyap]

What OS are you on . Is it 11.0 . Are you getting MUX errors in your log files or crash dumps . THen increase the ncallout parameter and also take a look at these patches or their superceded versions:
PHNE_25489
PHNE_24185
PHNE_24181

[John Poff]

Hello,

Here is the URL for the kernel parameters, and the description for ncallout:


http://www.docs.hp.com/hpux/onlinedocs/939/KCParms/KCparams.OverviewAll.html

ncallout specifies the maximum number of timeouts that can be scheduled by the kernel at any given time. Timeouts are used by:

alarm() system call,
setitimer() system call,
select() system call,
drivers,
uucp processes,
process scheduling.

When the system exceeds the timeout limit, it prints the following fatal error to the system console:


panic: timeout table overflow


I'm not a wizard about those system calls, but I'd try bumping up your ncallout parameter by a couple thousand and trying it again.

JP

[James R. Ferguson]

Hi:

'ncallout' specifies the maximum number of timeouts the kernel can concurrently schedule. These include 'alarm()', setitimer{}', and 'select()' system calls and process scheduling.

In 11.0, 'ncallout' is based on 'nproc'. In 11.11, 'ncallout' is based on 'nkthread'. For both version of 11.x, a safe value should be:

nkthread+64

Regards!

...JRF...

[Vijayakrishna]

OS is 11.0 and we donot have any MUX errors in syslog file. Is there any dependency for ncallout and how it will impact kernel memory. Also is there any way to findout total system calls waiting before the system crash.

Thank you for the information.

VK

[James George_1]

Hi

Look for any runaway process, dfunct process which is filling the tables...

# ps -ef | grep defunt ..if found kill them...

rgds / james

[doug hosking]

This could be due to a kernel configuration problem or a kernel bug. Your HP support folks could probably tell a lot about the specific cause by looking at the generated crash dump. Examples of known patches for 11.00 that fix some instances of callout table overflow panics are:

PHKL_24296:
( SR: 8606185626 CR: JAGad54828 )
When the beep duration is set to a very small value, there
is the possibility of getting a "callout table overflow"
panic if a lot of beeps are generated close together

PHKL_27024:
( SR:8606233087 CR:JAGae02311 )
System panics due to "callout table overflow" with the
following stack trace when continuous SCSI bus resets are
detected by the driver:
panic+0x14
settimeout_for_cpu+0x174
Ktimeout+0x3c
c8xx_reset_chip+0x129c
c8xx_isrRST+0x94
c8xx_isr+0x15cc
sapic_interrupt+0x2c

If you are running IDS/9000, a fix for another panic of this type is included in
IDS (J5083AA) V2.1.

[Ravi_8]

Hi, VK

look at this URL:
http://www.docs.hp.com/hpux/onlinedocs/939/KCParms/KCparam.Ncallout.html

[T G Manikandan]

http://www2.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000062903638

Thanks

[Vijayakrishna]

We are not using any MUX and also no errors related to MUX. Do we have any other issues related to callout without MUX?

Thank you

VK

[Sandip Ghosh]

"If the value of nproc is increased, ncallout should be increased proportionately. A general rule is that one callout per process should be allowed unless you have processes that use multiple callouts."

The above is taken from SAM help. Do you know the no. of callout used by your applications. May be your application is using multiple callouts from a single process. Increase that nacallout to 5000 and see the performance. I don't think that it will take much resource if you increase it.

Sandip

[Sandip Ghosh]

"If the value of nproc is increased, ncallout should be increased proportionately. A general rule is that one callout per process should be allowed unless you have processes that use multiple callouts."

The above is taken from SAM help. Do you know the no. of callout used by your applications. May be your application is using multiple callouts from a single process. Increase that nacallout to 5000 and see the performance. I don't think that it will take much resource if you increase it.


After changing the nfile to 40000, have you check how much the system is using? If the usage is not more than 50% then you can reduce the nfile to 30000 and look at the performance.

Sandip