topic Re: memory problem in Operating System - HP-UX

memory problem

Ciaran Byrne — Tue, 15 Apr 2003 06:07:00 GMT

Hi,
I have just upgraded from 16gig to 28gig on a HP-UX 11i o/s. I am now seeing segmentation violation errors and application crashes. Other similar app's
with exactly the same kernel params/patches/code base are not experiencing this problem. Are there kernel params that need to change as a result?

Other points
* "Memory fault" message when using top or glance : happens occasionally and causes an exit.
* swap is fairly low PCT used (around 15%)
* shmmax is 1 gig
* maxdsiz is 1 gig
* 32 bit app
* Core dump is produced but does not show any relevant info.
* tusc logs
"exit(11) implicit] .................................................................................. WIFSIGNALED(SIGSEGV)|WCOREDUMP"

I dont think its the app as it was running without issue before the upgrade. And is experiencing the problem
under much less load than normal.

help!
Thanks,
CB

Re: memory problem

Michael Tully — Tue, 15 Apr 2003 06:27:59 GMT

Hi CB,

Seeing you have upgraded your RAM how are you utilising it. i.e. Have you started using memory windows, this being 32 bit application still utilises memory just like a 32 bit OS. Perhaps a look at the memory management paper in /usr/share/docs/mem_mgt.txt file will assist.

How much swap do have? Have you increased it?
Have you lowered your buffer cache % ? It should be around 300-500Mb in total.
You may need to increase your shared memory.

Some food for thought
Michael

Re: memory problem

Stefan Farrelly — Tue, 15 Apr 2003 06:33:16 GMT

Very interesting. Youve had a large increase in RAM but you dont say why you added 12Gb ? I presume because it was needed by some other app which is now using it ?

So, how much free RAM do you have now ? (vmstat)
What is the output from swapinfo -mt (dev usage should be ZERO to prove youre not running out of ram and it should be the same size as ram - 28Gb).

Re: memory problem

Ciaran Byrne — Tue, 15 Apr 2003 06:48:55 GMT

Hi Michael/Stefan,

thanks for the quick response.

SWAP
====
swapinfo -tm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1024 0 1024 0% 0 - 1 /dev/vg00/lvol2
dev 16000 0 16000 0% 0 - 1 /dev/vg01/lvswap
reserve - 4233 -4233
memory 22286 1332 20954 6%
total 39310 5565 33745 14% - 0 -

VMSTAT
======
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
2 0 0 1049506 5297766 79 42 0 0 0 0 0 1926 11282 2970 6 2 92

I have not increased the swap. The reason the memory was added was because the app was hitting 100% mem utilization.

Buffer Cache
============
Available : 1.92gb Requested : na
Used : 1.92gb
High : 1.92gb

Do you mean the shmmax value?

Thanks
CB

Re: memory problem

Ciaran Byrne — Tue, 15 Apr 2003 07:01:52 GMT

Hi,
one other piece of info. In the tusc logs I see tons of messages like
"[27571] brk(0x8cfc0000) ...................................................................................... ERR#12 ENOMEM"

Thanks,
CB

Re: memory problem

Stefan Farrelly — Tue, 15 Apr 2003 07:33:45 GMT

Youre only running 17Gb of swap (1+16) and with 28Gb of RAM you really should have at a minimum 28Gb of swap. Basically you are using RAM as extra swap which could be causing the memory errors/crashes.

Up swap to 28Gb total and Im sure your problems will go away. Easy to do on the fly.

Re: memory problem

Ciaran Byrne — Tue, 15 Apr 2003 18:21:55 GMT

Hi,
I changed the swap to 41 gig.
Still seeing the same problems. Now i am thinking its some way related to ulimit.

ulimit -a
==========
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 966852
stack(kbytes) 262144
memory(kbytes) unlimited
coredump(blocks) 4194303
nofiles(descriptors) 2048

It appears the Glance version needs to be upgraded because of the VPar installation.

Thanks,
Ciaran

Re: memory problem

Olivier ROBERT — Tue, 15 Apr 2003 19:33:24 GMT

Hi Ciaran,

That may be stupid, but do you have one swap volume, or is it fragmented across multiple volumes?

Another idea, maybe it would be interesting to check how it works if swap is managed the "lazy" way, i.e. without immediate reservation of the swap space. I don't know if it's possible on HP-UX, but that's what one can do on Tru64 UNIX by removing the "/sbin/swapdefault" symbolic link, and then rebooting.

If either solution works anyway, it would look like an HP-UX bug... and "lazy" swap management is not as good as immediate reservation, as far as performance is concerned.

Hope this helps,

Olivier

Re: memory problem

James Murtagh — Tue, 15 Apr 2003 19:48:29 GMT

Hi Ciaran,

The failing brk() call appears to indicate you are hitting the maxdsiz limit. See the "man brk" page for the three possible enomem errors returned.

I can see you have maxdsiz set to almost 1GB but what about maxdsiz_64bit? This may seem a funny question but if the process is first started as a 64bit process then exec'd it will take the lower limit of the two.

The first step is to ensure this is the cause. You can examine the processes memory regions in Glance, concentrate on the Data Segment field. If Glance is still causing you problems at the moment you can compile this program and run it with the pid as the argument. If the format is bad I will attach it instead.

#include
#include
#include
#include

main(int argc, char *argv[])
{

if(argc != 2)
{
printf("\n\tUsage: %s \n\n", argv[0]);
exit(1);
}

int i = atoi(argv[1]);

struct pst_status procmem;
pstat_getproc(&procmem, sizeof(struct pst_status), 0, i);

printf("\nProcess Data and stack sizes for process %d : \n\n", i);
printf("Process Data size is : %d KB\n", 4*(procmem.pst_dsize));
printf("Process Stack size is : %d KB\n", 4*(procmem.pst_ssize));
printf("Process Virtual Data size is : %d KB\n", 4*(procmem.pst_vdsize));
printf("Process Virtual Stack size is : %d KB\n", 4*(procmem.pst_vssize));

}

If these tests indicate you are reaching the maxdsiz limit you may be able to recompile the program and link the objests with the -N flag for EXEC MAGIC. This will almost double your data segment area for the 32bit process by allowing space in the text quadrant to be used too.

The big question though is why this would fail after the memory upgrade. Does anything in the code use the physical memory total to calculate variables or something similar? I would also run an swverify to ensure the installed OS software is configured properly.

Regards,

James.

Re: memory problem

James Murtagh — Tue, 15 Apr 2003 20:35:05 GMT

Hi again Ciaran,

I think the argument to brk() actually show us the problem:

"[27571] brk(0x8cfc0000).... ENOMEM"

That address is a third quadrant address for a 32bit process, hence beyond the end of the data quadrant. This is the list of ranges:

Q1 (quadrant 1): 0x00000000-0x3fffffff
Q2 (quadrant 2): 0x40000000-0x7fffffff
Q3 (quadrant 3): 0x80000000-0xbfffffff
Q4 (quadrant 4): 0xc0000000-0xffffffff

So it appears the application is asking for memory beyond its limit, hence ENOMEM. This may be also be due to fragmentation of the address space in Q2, if you are allocating and freeing chunks before this allocation request.

Regards,

James.

Re: memory problem

Ciaran Byrne — Tue, 15 Apr 2003 21:13:04 GMT

Hi,
thanks everyone for the responses so far. The application is a 32 bit app.

maxdsiz 990056448

maxdsiz_64bit 17179869184

The other thing to mention is that this is a VPar'd environment. Just combined two NPars from a SuperDome and split it into VPars.

I am about to look at the logproc to see what the process memory allocations were before it dies.

Thanks,
Ciaran

Re: memory problem

Ciaran Byrne — Tue, 15 Apr 2003 22:01:28 GMT

Re: memory problem

Ciaran Byrne — Tue, 15 Apr 2003 22:02:53 GMT

hi,
here is the mware report template

REPORT "Export MWA !DATE !COLLECTOR !SYSTEM_ID"
FORMAT ASCII
HEADINGS ON
SEPARATOR= "|"
SUMMARY=1

*********************************************************************
DATA TYPE PROCESS
* Record Identification Metrics
DATE
TIME

* Summary Metrics
PROC_PROC_NAME
PROC_USER_NAME
PROC_PROC_ID
PROC_CPU_TOTAL_UTIL
PROC_MEM_VIRT
PROC_MEM_RES
PROC_MINOR_FAULT
PROC_MAJOR_FAULT
PROC_INTEREST
PROC_STOP_REASON
PROC_PRI
PROC_THREAD_COUNT

Thanks,
Ciaran

Re: memory problem

James Murtagh — Tue, 15 Apr 2003 22:57:38 GMT

Hi Ciaran,

I don't think the output tells us too much I'm afraid (or either would my pstat code thinking of it). We can see the Resident size of the process just before it was killed but this doesn't say a lot. It could still then request a very large region that would hit virtual memory limits, which I think you are seeing. The tusc output would be of more interest if you wish to post this.

Regards,

James.

Re: memory problem

Ciaran Byrne — Tue, 15 Apr 2003 23:00:54 GMT

Re: memory problem

James Murtagh — Tue, 15 Apr 2003 23:11:49 GMT

Hi Ciaran,

I was just going to examine the memory allocation patterns for possible fragmentation issues. If you could provide the output of:

# egrep "brk|free"

that would help, or possibly attach the full tusc output using the attachment box at the bottom of the post screen.

Is this actually in-house code? Do you think it would be possible to recompile for EXEC_MAGIC to test it?

Regards,

James.

Re: memory problem

Ciaran Byrne — Tue, 15 Apr 2003 23:17:24 GMT

Re: memory problem

Ciaran Byrne — Tue, 15 Apr 2003 23:18:58 GMT

Hi,
to answer your other question. It is a mix or in-code and vendor code. It is working correctly on other similar servers and was working on this one before the VPar install and addition of memory.

Thanks,
Ciaran

Re: memory problem

James Murtagh — Tue, 15 Apr 2003 23:50:40 GMT

Hi Ciaran,

There are no free() calls from your list so there is probably no point pursuing this. I'm surprised there are so many ENOMEM errors reported though.

One thing that has caught my eye though - from your ulimit output the stack size is very large. As stack and data share Quadrant 2 this is imposing a limit on your data area as stack takes precedence. I would compare maxssiz to your other servers. You may also set the limit (using the posix shell) using:

# ulimit -s 8192

The 8192 is just an abritary figure. Then if you rerun the process and see how it goes.

Regards,

James.

Re: memory problem

Ciaran Byrne — Wed, 16 Apr 2003 00:19:25 GMT

Hi,
thanks James. We have set to ulimit -s 16384. And are re-running the test.

Also just got this message
/var/adm/sw # view swagent.log
"swagent.log" [Read only] 3168728 lines, 176327638 characters Warning: Out of m
emory saving lines for recovery - try using ed

Regards,
Ciaran