Application core dump in hp-ux system

[KYY]

Hi all. Need your advice on the issue where my customer will encounter application core dump when it reaches the high utilization level of 1GB. This is the system statistics when the application is not in use:

/home/super2 >/usr/sbin/swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
dev 4000 0 4000 0% 0 - 1 /dev/vg01/lvswap
reserve - 670 -670
memory 2046 751 1295 37%
total 10142 1421 8721 14% - 0 -
[mqs41qa]:/home/super2 >

[mqs41qa]:/home/super2 >vmstat
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 0 0 175392 181827 1380 353 0 0 0 0 0 1292 24170 1141 5 7 88

The attached file is the vmstat just before the application experiences a core dump. The OS is fine and has no problem when the application is dumping. This application is a new in-house application developed by my customer. I have tried tweaking the kernel parameters but to no avail.

:/home/super2 >/usr/sbin/sysdef
NAME VALUE BOOT MIN-MAX UNITS FLAGS
acctresume 4 - -100-100 -
acctsuspend 2 - -100-100 -
allocate_fs_swapmap 0 - - -
bufpages 52377 - 0- Pages -
create_fastlinks 0 - - -
dbc_max_pct 10 - - -
dbc_min_pct 5 - - -
default_disk_ir 0 - - -
dskless_node 0 - 0-1 -
eisa_io_estimate 768 - - -
eqmemsize 0 - - -
file_pad 10 - 0- -
fs_async 0 - 0-1 -
hpux_aes_override 0 - - -
maxdsiz 512000 - 0-655360 Pages -
maxdsiz_64bit 1048576 - 256-1048576 Pages -
maxfiles 2048 - 30-2048 -
maxfiles_lim 4096 - 30-2048 -
maxssiz 8192 - 0-655360 Pages -
maxssiz_64bit 65536 - 256-1048576 Pages -
maxswapchunks 0 - 1-16384 -
maxtsiz 24576 - 0-655360 Pages -
maxtsiz_64bit 262144 - 256-1048576 Pages -
maxuprc 256 - 3- -
maxvgs 10 - - -
msgmap 0 - 3- -
nbuf 27714 - 0- -
nfile 65536 - 14- -
ninode 4880 - 14- -
no_lvm_disks 0 - - -
nproc 4200 - 10- -
npty 60 - 1- -
nstrpty 60 - - -
nswapdev 10 - 1-25 -
nswapfs 10 - 1-25 -
public_shlibs 1 - - -
remote_nfs_swap 0 - - -
rtsched_numpri 32 - - -
sema 0 - 0-1 -
shmem 0 - 0-1 -
shmmni 1024 - 3-1024 -
streampipes 0 - 0- -
swapmem_on 1 - - -
swchunk 2048 - 2048-16384 kBytes -
timeslice 10 - -1-2147483648 Ticks -
unlockable_mem 0 - 0- Pages -
:/home/super2 >


The host is a rp3440 server, running on hp-ux 11.23. The physical memory size in the system is 2GB. Will try to patch the OS in a day or two. Any suggestion on what should I focus on to prevent the application from dumping?

[Venkatesh BL]

Did you investigate the core file to see whether the problem was due to memory?

If memory is a bottleneck, then, you could either increase the RAM size or increase the SWAP area or both!

[Sandeep_Chaudhary]

ur swapinfo and vmstat doesnt show any memory utilization or any sort of pageout.
U have to analyse the core dumo file.

[Dennis Handly]

Your maxdsiz is 2 Gb. Unless you have linked with -N or used chatr(1) magic options, you can only use 1 Gb.

I assume your core file is about 1 Gb?
If so, you have run out of address space and with either have to relink or use chatr. Or port to 64 bit.

As to what Sandeep suggested, you can download gdb and get a stack trace.

[SKR_1]

you can use tusc tool for analising application core dumps also.

Thanks
SKR

[Dennis Handly]

>SKR: you can use tusc tool for analyzing application core dumps also.

Did you mean that it would display the signal info? This is typically not as useful as using a debugger, unless you need to trace system call patterns.

[KYY]

Logged a call with HP Solution Centre and have the system hardware checked out. No problem with the hardware side. Unable to get the application dump from the system.

Will download GDB and try to get a stack trace. Have just patched the system 2 days ago, but did not install the Quick Pack for Jun 2008 due to space constraints. Awaiting the customer to run the application test. Any way to increase the "Heap size" in the system kernel? Thanks for all your advice.

[Dennis Handly]

>Any way to increase the "Heap size" in the system kernel?

As I mentioned, you either have to port to 64 bit mode, relink with -N or use "chatr +q3p enable".
You can of course increase maxdsiz to 4 Gb but most applications can't take advantage of it.

[KYY]

>As I mentioned, you either have to port to 64 bit mode, relink with -N or use "chatr +q3p enable".

How do I port the application to 64 bit mode and relink with -N? Is the "chatr" command to be used on the application executable file or something else? Can you give examples to the above? Thanks.

[Dennis Handly · Accepted solution]

>How do I port the application to 64 bit mode

This is a port it may take a long time. The first thing to do is compile and link with +DD64 and correct any errors and warnings from +M2.

>relink with -N?

Just add -N to your 32 bit link line, trivial.
cc main.o ... -N ...

>Is the "chatr" command to be used on the application executable file or something else?

Yes: chatr +q3p enable executable
Note: You can do both -N and chatr.

[KYY]

mqs41qa[/sgx/mqs-qst/bin]chatr +q4p enable otdsadapter
otdsadapter:
current values:
shared executable
shared library dynamic path search:
SHLIB_PATH enabled second
embedded path disabled first Not Defined
shared library list:
dynamic /opt/mqm/lib/libmqm.sl
dynamic /usr/local/lib/libstdc++.sl
dynamic /usr/lib/libm.2
dynamic /usr/lib/libpthread.1
dynamic /usr/lib/libc.2
shared library binding:
deferred
global hash table disabled
plabel caching disabled
global hash array size:1103
global hash array nbuckets:3
shared vtable support disabled
explicit unloading enabled
runtime checks disabled
static branch prediction disabled
executable from stack: D (default)
kernel assisted branch prediction enabled
lazy swap allocation disabled
text segment locking disabled
data segment locking disabled
third quadrant private data space disabled
fourth quadrant private data space disabled
third quadrant global data space disabled
data page size: D (default)
instruction page size: D (default)
nulptr references enabled
shared library private mapping disabled
shared library text merging disabled
chatr(error): cannot open file for writing otdsadapter
mqs41qa[/sgx/mqs-qst/bin]file otdsadapter
otdsadapter: PA-RISC2.0 shared executable dynamically linked -not stripped

Am still confused. Is there anything I should do before running the "chatr" command? Thanks.

[KYY]

mqs41qa[/sgx/mqs-qst2/bin]chatr +q4p enable otdsadapter
otdsadapter:
current values:
shared executable
shared library dynamic path search:
SHLIB_PATH enabled second
embedded path disabled first Not Defined
shared library list:
dynamic /opt/mqm/lib/libmqm.sl
dynamic /usr/local/lib/libstdc++.sl
dynamic /usr/lib/libm.2
dynamic /usr/lib/libpthread.1
dynamic /usr/lib/libc.2
shared library binding:
deferred
global hash table disabled
plabel caching disabled
global hash array size:1103
global hash array nbuckets:3
shared vtable support disabled
explicit unloading enabled
runtime checks disabled
static branch prediction disabled
executable from stack: D (default)
kernel assisted branch prediction enabled
lazy swap allocation disabled
text segment locking disabled
data segment locking disabled
third quadrant private data space enabled
fourth quadrant private data space disabled
third quadrant global data space disabled
data page size: D (default)
instruction page size: D (default)
nulptr references enabled
shared library private mapping disabled
shared library text merging disabled
new values:
shared executable
shared library dynamic path search:
SHLIB_PATH enabled second
embedded path disabled first Not Defined
shared library list:
dynamic /opt/mqm/lib/libmqm.sl
dynamic /usr/local/lib/libstdc++.sl
dynamic /usr/lib/libm.2
dynamic /usr/lib/libpthread.1
dynamic /usr/lib/libc.2
shared library binding:
deferred
global hash table disabled
plabel caching disabled
global hash array size:1103
global hash array nbuckets:3
shared vtable support disabled
explicit unloading enabled
runtime checks disabled
static branch prediction disabled
executable from stack: D (default)
kernel assisted branch prediction enabled
lazy swap allocation disabled
text segment locking disabled
data segment locking disabled
third quadrant private data space enabled
fourth quadrant private data space enabled
third quadrant global data space disabled
data page size: D (default)
instruction page size: D (default)
nulptr references enabled
shared library private mapping disabled
shared library text merging disabled

The funny thing is that when I run "chatr" on another executable file, it was successful.

[Dennis Handly]

>chatr +q4p enable otdsadapter

You should enable +q3p first and save +q4p if you need more.

>chatr(error): cannot open file for writing otdsadapter
>Is there anything I should do before running the "chatr" command?

You need to make it writable. Run chatr, then make it read only again.

[KYY]

Managed to execute "chatr" on the previous executeable application file. Customer has no issue running the application to 2GB level. Thanks Dennis.

[KYY]

Hopefully this would be my last query. Is the change executed on the executable application file by using "chatr" a permanent change or must the user execute it whenever after any system reboot? The customer feedback to me that he has to rerun the "chatr" command after they had rebooted the server.

[Dennis Handly]

>Is the change executed on the executable application file by using "chatr" a permanent change

It is permanent, a property of the executable, this does change the checksum.

[KYY]

Using "chatr" to modify the application executeable files so as to utilize the 3rd and 4th quadrant of the memory resources in the system. Application is able to go beyond the 50% mark of the system memory resource.