topic Re: why the program cause coredump? in Operating System - HP-UX

why the program cause coredump?

MA Qiang — Wed, 01 Jun 2005 04:47:48 GMT

#include
#include
#include
#include

#define FDATANUM 10000000
#define CDATANUM 500000
#define IDENNUM 50000
#define STR_LEN 256

main()
{
int ret;
char cpar[20][2048];
long lpar[100];
char cdata[CDATANUM][STR_LEN];
float fdata[FDATANUM];
long ciden[IDENNUM];
long fiden[IDENNUM];
char output[30];

strcpy(cpar[0],"A");

}

Our Server configuration:
HP 11iv1
memory:8GB
CPU:4
**********************************************
The error log:
> cc +DA2.0W +DS2.0 -DSS_64BIT_SERVER b.c

> ./a.out
Memory fault(coredump)

> file core
core: ELF-64 core file - PA-RISC 2.0 from 'a.out' - received SIGSEGV

> cc b.c

> ./a.out
Memory fault(coredump)

> file core
core: core file from 'a.out' - received SIGSEGV

**********************************************
Some Tunable Kernel Parameters:

aio_listio_max 256 - 256
aio_max_ops 2048 - 2048
aio_prio_delta_max 20 - 20
bcvmap_size_factor 2 - 2
dbc_max_pct 25 - 25
effective_maxpid - - ((NPROC<=30000)?30000:(NPROC*5/4))
eqmemsize 15 - 15
hfs_max_ra_blocks 8 - 8
hfs_max_revra_blocks 8 - 8
initmodmax 50 - 50
iomemsize - - 40000
ksi_alloc_max 20640 - (NPROC*8)
ksi_send_max 32 - 32
max_async_ports 50 - 50
max_fcp_reqs 512 - 512
max_mem_window 0 - 0
max_thread_proc 1000 - 1000
maxdsiz 0x10000000 - 0x10000000
maxdsiz_64bit 0x40000000 - 0X40000000
maxfiles 4096 - 4096
maxfiles_lim 4096 Y 4096
maxqueuetime - - 0
maxssiz 0x800000 - 0X800000
maxssiz_64bit 0x800000 - 0X800000
maxswapchunks 8192 - 8192
maxtsiz 0x4000000 Y 0X4000000
maxtsiz_64bit 0x40000000 Y 0X40000000
maxuprc 2322 Y ((NPROC*9)/10)
maxusers 320 - 320
maxvgs 10 - 10
modstrmax 500 - 500
msgmax 8192 Y 8192
ncsize 27808 - (NINODE+VX_NCSIZE)+(8*DNLC_HASH_LOCKS)
netmemmax - - 0
scsi_max_qdepth 8 Y 8
scsi_maxphys 1048576 - 1048576
sendfile_max 0 - 0
shmmax 4096000000 Y 4096000000
vol_dcm_replay_size 262144 - (256*1024)
vol_max_bchain 32 - 32
vol_max_nconfigs 20 - 20
vol_max_nlogs 20 - 20
vol_max_nmpool_sz 4194304 - (4*1024*1024)
vol_max_prm_dgs 1024 - 1024
vol_max_rdback_sz 4194304 - (4*1024*1024)
vol_max_vol 8388608 - (8*1024*1024)
vol_maxio 256 - 256
vol_maxioctl 32768 - 32768
vol_maxkiocount 2048 - 2048
vol_maxparallelio 256 - 256
vol_maxspecialio 256 - 256
vol_maxstablebufsize 256 - 256
vol_mvr_maxround 256 - 256
volcvm_cluster_size 16 - 16
voldrl_max_drtregs 2048 - 2048
voliomem_chunk_size 65536 - (64*1024)
voliomem_maxpool_sz 4194304 - (4*1024*1024)
voliot_iobuf_max 65536 - 65536
voliot_max_open 32 - 32
volraid_rsrtransmax 1 - 1
vps_pagesize 4 - 4
vx_maxlink 32767 - 32767
vx_ncsize 1024 - 1024
vxfs_max_ra_kbytes 1024 - 1024
vxtask_max_monitors 32 - 32

Thank you!

Re: why the program cause coredump?

Stephen Keane — Wed, 01 Jun 2005 05:45:16 GMT

It's probably your stack size is too small.

maxssiz 0x800000 - 0X800000
maxssiz_64bit 0x800000 - 0X800000

Re: why the program cause coredump?

harry d brown jr — Wed, 01 Jun 2005 06:44:22 GMT

I think Stephen is close.

When I commented out that ENORMOUS character array -> char cdata[CDATANUM][STR_LEN]; <- AKA => cdata[500000][256] and the next line -> float fdata[FDATANUM]; <- AKA => fdata[10 million], then the program ran fine. When allowed both back in, and changed CDATANUM from 500000 to 50000, it also ran fine.

I believe that changing maxssiz requires a reboot.

You are much BETTER off using malloc than creating such large static arrays.

live free or die
harry d brown jr

Re: why the program cause coredump?

Stephen Keane — Wed, 01 Jun 2005 07:40:00 GMT

Do the maths ...

char cdata[CDATANUM][STR_LEN];

CDATANUM x STR_LEN

= 500000 x 256 = 128,000,000 bytes or in hex 0x7A12000

You stack size is 0x800000 !

As Harry says, malloc would be preferable for such huge arrays, as it would allocate memory on the heap rather than the stack. But watch your maxdsize limit too!

Re: why the program cause coredump?

harry d brown jr — Wed, 01 Jun 2005 08:07:42 GMT

Here's some good reading on strings and malloc:

http://members.shaw.ca/ipatters/BeginC_9.html

live free or die
harry d brown jr

Re: why the program cause coredump?

MA Qiang — Wed, 01 Jun 2005 10:49:41 GMT

Thanks for your replies.

I want to know how to tune the Kernel Parameters to the suitable values according to our server's memory?

Regards.

Re: why the program cause coredump?

harry d brown jr — Wed, 01 Jun 2005 12:52:38 GMT

I changed my maxssiz to 0xb000000 (184,549,376) and it ran fine, but then again it's not doing much.

maxdsiz 0x10000000 - 0x10000000
maxdsiz_64bit 0x40000000 - 0X40000000
maxssiz 0xb000000 - 0XB000000
maxssiz_64bit 0xb000000 - 0XB000000
maxtsiz 0x4000000 Y 0X4000000
maxtsiz_64bit 0x40000000 Y 0X40000000

If you add more huge arrays like that, then you will need to increase the stack again (This really isn't recommended). You really should be using malloc.

To "tune" your machine we would need to know the memory available to use (max memory minus current consumption (without this program running)).

live free or die
harry d brown jr

Re: why the program cause coredump?

A. Clay Stephenson — Wed, 01 Jun 2005 14:46:53 GMT

I should tell you if I saw large statically allocated arrays like that, I would not hire you. It's very poor programming practice because it's not often that all of your arrays would need to be that large AND there is a hard-coded limit on all your data structures. Generally, any program that requires a stack greater than about 32MB is poorly written. You should really learn to code dynamic arrays (using calloc and realloc) which are allocated not from the stack segment but the data segment.

Re: why the program cause coredump?

MA Qiang — Thu, 02 Jun 2005 06:21:39 GMT

We can use dynamic arrays to achieve the goal.

I just don't know why HP-UX limits the Kernel Parameters so low by default. It causes inconvenience for the users. We often be asked to adjust the Kernel Parameters, but don't know which one is the key. We have not encountered this kind of problem on other platforms, even on old Compaq's ES40.

Anyway, thank everyone who answer my question.

Regards.

Re: why the program cause coredump?

harry d brown jr — Thu, 02 Jun 2005 06:39:06 GMT

The restrictions are in place to prevent an application from crippling a machine. Without these "controls" or "limitations" an application, like the one you posted, could cause the system to start paging.

The biggest issue with most people that write programs (Note I didn't say programmers, because anyone can write code but only programmers can program), and I did a lot of programming (in many languages) for twenty years, is that they typically have no concern or even knowledge of how to write efficient programs.

It's too bad that most people writing code don't have a systems background, if they did, they wouldn't be creating crap code.

live free or die
harry d brown jr

Re: why the program cause coredump?

MA Qiang — Thu, 02 Jun 2005 07:18:15 GMT

My opinion is HP should guide user to set the Kernel Parameters according user's computer configuration. The maxssiz's highest value is 401,604,608, it is not very large for a 8GB server because we have only 1 or 2 applications running on it. The default value of HP-UX's some Kernel Parameters is too low for most applications now.

Re: why the program cause coredump?

Steven E. Protter — Thu, 02 Jun 2005 07:22:49 GMT

The default values on the kernel get you to the point where the system is running well enough for you to install patchs, applications and modify and build a new kernel.

The default values have to be something, and its expected that they will need to be changed.

Newer Oracle installations check parameters and won't install until the kernel is changed, rebuilt and installed anew.

Other applications leave it up to you. When something starts to malfunction, checking kernel resources is almost always a good diagnostic step. It's really up to the application developers to run thorough tests and make sure they can give optimized recommendations for the kernel and other parameters.

SEP

Re: why the program cause coredump?

MA Qiang — Thu, 02 Jun 2005 07:33:58 GMT

If every application requires to adjust some Kernel Parameters, then we have to reboot our server again and again for every new app.

Re: why the program cause coredump?

harry d brown jr — Thu, 02 Jun 2005 07:43:11 GMT

You could max them (kernel parameters) out, which will leave your system vulnerable to failure. OR you can teach your code writers the correct programming practices. Sometimes we have to manage people like we do machines.

live free or die
harry d brown jr

Re: why the program cause coredump?

A. Clay Stephenson — Thu, 02 Jun 2005 09:29:17 GMT

Actually your current setting of ~400MB for maxssiz could be bad and much too large even if you have 8GB of physical memory. In the 32-bit data model both stack and data, by default, are allocated from the same 1GB quadrant so that by setting the stack at 400MB, you have now limited dynamic memory allocation to a maximum of 600MB no matter what maxdsiz is actually set to. This only applies to 32-bit application but you need to be aware of this feature. That's why limiting maxssiz to about 32MB is a very good and very generous size.

I've yet to see a UNIX/Linux box that was initially well-tuned regardless of the vendor. There are just too many variables. HP has a number of parameters that are typically too small and a few that are too large but the default values will run and then allow you to tune the kernel to suit. In practice, even with many different applications, it's not necessary to retune the kernel for each new application. With experience, you can get the values reasonable on the first attempt.

Re: why the program cause coredump?

MA Qiang — Thu, 02 Jun 2005 11:10:24 GMT

0x800000 is too low for our meteorological data processing.
There are about 15,000 observation stations around the world, and about 200 elements per observation.
In the extreme:
15000(stations)*200(elements)*24(times)=72000000(float)
72000000*4=288,000,000(bytes) It is only one day's observation data.

Should I increase the maxdsiz and maxdsiz_64bit for dynamic arrays? Especially the maxdsiz_64bit? I think there is almost no limit for 64bit.

Thank you!

Re: why the program cause coredump?

A. Clay Stephenson — Thu, 02 Jun 2005 11:21:36 GMT

If these arrays are dynamic then maxdsiz is the tunable that controls this (or maxdsiz_64bit if this is a 64-bit process) and in that case maxssiz can be much smaller.
You should make sure that the _64bit values are at least as large as their 32-bit counterparts. You are correct that in the 64-bit data model there are essentially no practical limits although it is still wise to limit maxdsiz_64bit to some reasonable value so that a coding error does not bring the system to its knees.

Generally the best way to handle your kind of data is dynamic arrays that expand in chunks of 100 or so elements so that the relatively expensive realloc() function does not have to be called very often.

Re: why the program cause coredump?

Stephen Keane — Thu, 02 Jun 2005 11:28:11 GMT

Also, do you really have to store all the data for all stations, for the entire day in memory at the same time? Can you not save data to a file as you go along? It is more expensive in terms of time (disk access is much slower than direct memory access), but unless you need to access all of the data at the same time, it's a lot easier on memory.