Virtual memory

[jpcast_real]

Hello,

I have a big problem that it is becoming me crazy . I have an application running over hp-ux 11i. Sometimes the application gets hung and I do not know the reason at all . I only know that when the process starts working wrong the operating system seems to be swaping. I have seen with the sar -d command figures of 30-40 in the avwait column.

I also see that the virtual memory queue in this moments grows and gobal_mem_page_request metric grows a lot also.

It seems that the application is requesting some virtual memory and can not get more, then it starts swapping , can it be possible ??

IT is a 32 bits application so I assume that has the limitation to reach a certain limit of memory.

Question, how can I know what it is really happening??? I know that the application uses several memory mapped files what I do not know exactly if this is a problem or not.

Any help will be rewarded.

[Bill Hassell]

32bit applications are extremely limited and the default fence (to prevent runaway programs from using all RAM) is as low as 64MB or 256MB. Change the kernel parameter MAXDSIZ to 2 or 3 billion so you can use up to that amount. Next, compile/link your program as an EXEC_MAGIC executable. This will allow more than the 960MB typically available to a standard executable. EXEC_MAGIC will allow up to about 1800MB local heap. If you need more than 1800, you can use chatr to change the the executable to use q3 or even q4. 32bit applications run in 4 quadrants of 1GB each and the data area is normally limited to q2 only. Note that enabling additional quadrants places restrictions on some program features.

Attached is a simple program that can obtain 960, 1800, 2800, 3800 MB as 32 bits and essentially unlimited RAM when compiled as a 64bit program.

[jpcast_real]

Hello Bill,

thanks for the answer. Some questions come to my mind:

- How can I know if my processes are reaching or not the memory limit.

- I have seen how when the process hang the stop reason change from "SYSTEM" to "PRI" and the priority increases from 153 to 230. Can this mean that any other process in the system is making this process to stop.

Thanks in advance.

[Dennis Handly]

>It seems that the application is requesting some virtual memory and can not get more, then it starts swapping, can it be possible??

Yes, if you can't get enough memory, then things start paging/swapping.

>how can I know what it is really happening???

Look at top(1) and see how much memory is being used. Or glance for more details.
Or use tusc to look at the system calls.
sar(1) will tell you if there are lots of page outs.

>I know that the application uses several memory mapped files

Are they mapped shared?

>How can I know if my processes are reaching or not the memory limit?

Your application should tell you when you get an error with mmap(2) or malloc(3).

>I have seen how when the process hang the stop reason change from "SYSTEM" to "PRI" and the priority increases from 153 to 230.

This means you are non-mutual and the OS is kicking you out because you are using too many resources. You need more CPUs and memory on your machine.
(Your priority is actually decreasing.)

>Bill: 32bit applications are extremely limited and the default fence

If this was the problem, a well written application should abort and tell you and not hang.

>compile/link your program as an EXEC_MAGIC executable.

For PA, you can use chatr(1) +q3p and +q3p without relinking with -N and get 3 quadrants.

>Note that enabling additional quadrants places restrictions on some program features.

Right, your memory mapped files would be limited. It would depend on whether you need more heap or more mmap(2).

[jpcast_real]

Hello,

I have found that the logical_io_rate is really high during the bad moments , mem_page_request increases a lot , and the disk usage of operating system disk grows a lot

I do not know how can I knwo if the memory mapped files are shared?


>This means you are non-mutual and the OS is kicking you out because you are using too many resources. You need more CPUs and memory on your machine.
(Your priority is actually decreasing.)

Yes but the general CPU behauviour seems to be ok, it seems that the O.S stops for doing another things!!!

Could you give me more information about EXEC_MAGIC???

[Dennis Handly]

>how can I know if the memory mapped files are shared?

By looking at the mmap(2) call. Do you have MAP_SHARED or MAP_PRIVATE?

>it seems that the OS stops for doing another things!!!

That's what it is suppose to do for non-mutual processes. Decrease their priority so other processes can do useful work. More memory would help, if you are doing lots of paging.

>Could you give me more information about EXEC_MAGIC?

What's to know? If you want more than 1 GB, you use it. If you don't you don't. Especially since the text isn't shared. See ld(1), -N:
http://docs.hp.com/en/B2355-60130/ld.1.html

[Ilkka Seittenranta]

Hi,

Here is one reason why application can stop work some short period.

If HP-UX uses lot of memory to pseudoswap, more than 60%.

After pseudoswap uses all free memory the system change to "idle-process"-state, the state is something "waiting until there is more recourses free". The nonsystem-process are that time to "halt"-state and that time load is something like 0.02

You must disable pseudoswap if you like find what kind performance problems you have. After pseudoswap uses all free memory you can't get any error messages in the syslog-file, there is not enough resources to open file handles!


Best Regards
Ilkka

[Laurent Menase]

I must go against the last affirmation about pseudoswap.
pseudoswap is just a trick which avoid to reserve real disk swap when there is enough memory available. It is only a counter!

- pseudoswap answers to the question why should I have 8G of swap when my application only take 4G, and I have 4G of memory.

So disable pseudoswap has no effect on that at all.
System will swap as soon as you use more memory than the physical memory.


you can first look at your memory usage use/kernel with glance.
Check also vm and vhand patch. There where a bug where a swapedout application could be stopped forever because ready and pagedout.

Now how to investigate why a process is getting more memory than expected?

take a core with a kill -core and look at the core file,
try to find pattern in the leak
instrument the code.........

[Ilkka Seittenranta]

The pseudoswap is a reason why "Sometimes the application gets hung and I do not know the reason at all". The glance not even OVO can give info when too much memory is used to pseudoswap. There is no tools to find out what happening there.

My quess is: The pseudoswap is a trick to hide one HP-UX characteristic "HP-UX swapping out all the time", because there is proactive system to handle process witch status is "sleep" too long time. The OS move them to swap area after some certain time. This is also the reason why HP-UX need bigger swap space than memory.

[Laurent Menase]

No at all.
If you want to avoid to swap you need to have enough memory.

hpux is only swapping when there is not enough free memory!

But hpux reserve swap space when virtual mem is allocated to be sure to be able to pageout the process if there is memory starvation.
just remember that virtual mem can be much larger than physical memory.

disabling pseudo swap never change when the system will swap.. pseudo swap is only a trick for swap reservation!
now you need to look at your memory
the amount in buffer cache, the number in kernel dynamic buffer, if kernel memory size is raising - except buffer cache- then you may be encountering a memory leak, and should contact hp support