Kernel panic after tuning kernel parameters

[Jan vd Vlugt]

For oracle we need to tune some tuneables:

kctune max_async_ports=27000
kctune max_thread_proc=3000
kctune maxdsiz=3221225472
kctune maxdsiz_64bit=274877906944
kctune maxfiles_lim=8192
kctune maxfiles=8192
kctune maxssiz=134217728

Here are the screenlogs

Vpux1:/ # kctune max_async_ports=27000
NOTE: The configuration being loaded contains the following change(s)
that cannot be applied immediately and which will be held for
the next boot:
-- The tunable max_async_ports cannot be changed in a dynamic
fashion.
WARNING: The automatic 'backup' configuration currently contains the
configuration that was in use before the last reboot of this
system.
==> Do you wish to update it to contain the current configuration
before making the requested change? y
* The automatic 'backup' configuration has been updated.
* The requested changes have been saved, and will take effect at
next boot.
Tunable Value Expression
max_async_ports (now) 50 Default
(next boot) 27000 27000


Vpux1:/ # kctune max_thread_proc=3000
* The automatic 'backup' configuration has been updated.
* The requested changes have been applied to the currently
running system.
Tunable Value Expression Changes
max_thread_proc (before) 256 Default Immed
(now) 3000 3000


Vpux1:/ # kctune maxdsiz=3221225472
* The automatic 'backup' configuration has been updated.
* The requested changes have been applied to the currently
running system.
Tunable Value Expression Changes
maxdsiz (before) 1073741824 Default Immed
(now) 3221225472 3221225472



Vpux1:/ # kctune maxdsiz=3221225472
* The automatic 'backup' configuration has been updated.
* The requested changes have been applied to the currently
running system.
Tunable Value Expression Changes
maxdsiz (before) 1073741824 Default Immed
(now) 3221225472 3221225472
Vpux1:/ # kctune maxdsiz_64bit=274877906944
* The automatic 'backup' configuration has been updated.
* The requested changes have been applied to the currently
running system.
Tunable Value Expression Changes
maxdsiz_64bit (before) 4294967296 Default Immed
(now) 274877906944 274877906944

Vpux1:/ # kctune maxfiles_lim=8192
* The automatic 'backup' configuration has been updated.
* The requested changes have been applied to the currently
running system.
Tunable Value Expression Changes
maxfiles_lim (before) 4096 Default Immed
(now) 8192 8192
Vpux1:/ # kctune maxfiles=8192
NOTE: The configuration being loaded contains the following change(s)
that cannot be applied immediately and which will be held for
the next boot:
-- The tunable maxfiles cannot be changed in a dynamic fashion.
* The automatic 'backup' configuration has been updated.
* The requested changes have been saved, and will take effect at
next boot.
Tunable Value Expression
max_async_ports (now) 50 Default
(next boot) 27000 27000
maxfiles (now) 2048 Default
(next boot) 8192 8192


Vpux1:/ # kctune maxssiz=134217728
* The automatic 'backup' configuration has been updated.
NOTE: No change to 'maxssiz' was needed.
Tunable Value Expression Changes
maxssiz 134217728 134217728 Immed

After the reboot:



Seconds left till autoboot - 3
[Read only - use Ctrl-Ecf for console write access.] 0
AUTOBOOTING...> System Memory = 20474 MB
loading section 0
.......................................................... (complete)
loading section 1
............... (complete)
loading symbol table
loading System Directory (boot.sys) to MFS
....
loading MFSFILES directory (bootfs) to MFS
...............
Launching /stand/vmunix
SIZE: Text:29449K + Data:7430K + BSS:5930K = Total:42810K
Console is on Serial COM1
Booting kernel...


Stored message buffer up to panic:
Found adjacent data tr. Growing size. 0x32f1000 -> 0x72f1000.
Pinned PDK malloc pool: base: 0xe000000100d0f000 size=117700K
Loaded ACPI revision 2.0 tables.

MFS is defined: base= 0xe000000100d0f000 size= 1444 KB
WARNING pdk_malloc: rmalloc returned 0.
Call was pdk_malloc (0x4e7c000, 0x11, 0x10000000).
pinned_pdk_malloc_avail_contig(): 0x6e1d480

System Panic:

panic: pdk_malloc: not enough free memory to allocate
Stack Trace:
IP Function Name
0xe000000000cbd0e0 pdk_malloc+0x280
0xe000000001c6d880 hdl_boottime_alloc+0x40
0xe0000000006f89f0 get_kmem+0x170
0xe00000000072db80 kmem_arena_xlarge_alloc+0x100
0xe00000000069adb0 kmem_arena_varalloc+0x230
0xe00000000154a630 ktune_alloc_memory+0x50
0xe000000000b96980 io_init_tunables+0x1a0
0xe000000000944e30 kc_dispatch+0x460
0xe00000000154a5a0 rm_tunable_init+0x60
0xe000000001604ff0 DoCalllist+0x3a0
End of Stack Trace

linkstamp: Thu Jun 25 14:11:04 METDST 2009
_release_version: @(#) $Revision: vmunix: B11.23_LR FLAVOR=perf Fri Aug 29 22:35:38 PDT 2003 $

*** A system crash has occurred. (See the above messages for details.)
*** The system is now preparing to dump physical memory to disk, for use
*** in debugging the crash.

ERROR: Your system crashed before I/O and dump configuration was complete.
This system does not support a crash dump under these circumstances.
Contact your HP support representative for assistance.

[Steven E. Protter]

Shalom,

Oracle has led you down the garden path.

I think this one is a problem:

kctune max_async_ports=27000

the system seemed stable with the other dynamically changed parameters.

To back this out, you will probably need to either boot your system off of install media and put the previous kernel into production. You an also boot single user mode.

cd /stand
mv vmunix.prev vmunix


Then boot. replace vmunix with whatever the backup file name was for the previous kernel.

SEP

[Torsten.]

Try to boot into tunable maintenance mode:

EFI> hpux -tm


from man page:

...
Boot the system in tunable maintenance mode, also known as "failsafe boot" mode. This option will disregard the tunable settings and module settings in the kernel configuration, and boot with known good settings instead.

...

[Jan vd Vlugt]

Thanks for replying!

Steven you are right!
it's the max_async_ports value! I'm recovering the system again.

Hp support has the following options:>>>>>

_init_tunables is allocating space for several things including max_async_ports, which is on the list of tunables that oracle recommended be bumped. Once I saw that I remembered seeing this problem before so I went hunting and found QXCR1000817755, which came about via wtec call 4000165477. That's the good news. The bad news, perhaps, is that that customer was planning an upgrade to 11.31 where this is not an issue, so an official patch was never cut. However, an UNOF was created as a stopgap until the upgrade and that UNOF is still accessible. From that
call:



System: lan.cup.hp.com
Patch: ~ftp/pub_tmp/asyncdsk_amdocs_11iv2/UNOF_AsyncMaxPortFix.depot



The patch can be installed as follows:



# swinstall -x autoreboot=true -x patch_match_target=true \
-s /tmp/UNOF_AsyncMaxPortFix.depot



If you read through the call, you'll see why no official patch was done for 11.23. So, the customer has 3 choices



1) Don't set max_async_ports so high (that really is a huge value!)
2) Install the UNOF
3) upgrade HPVM and run 11.31 guests (I suspect this one is not a viable
option)
<<<<<<
I don't like "UNOF's" and upgrading to 11.31 is not an option. So what's a nice "recommended value" for max_async_ports?

[VK2COT]

Hello,

max_async_ports represent the maximum
number of processes that can open /dev/async
concurrently.

As a result, the maximum is nproc, which is
the limit on the number of processes that
can run on your system at any given time.

Set this parameter to the sum of 'processes'
from init.ora + number of background processes.
I believe the background processes
started at instance startup will open
/dev/async twice.

If max_async_ports is reached, subsequent
processes will use synchronous I/O.

If lsof is installed you may check the current status using it on /dev/async.

# lsof /dev/async

Actually you may want to run the full check:

# lsof /dev/async | wc -l

# kctune max_async_ports

# kctune nproc

Some of the HP and Oracle documentation state that it should be at least 1024.
Albeit, I have also seen sites with values
up to 6000.

Maybe this helps you!

VK2COT