Memory usage is only 14% still swapping is happening. why?

[ManojK_1]

Hi,

We are having a HP Integrity server with 64Gb physical Memory and 24Gb Swap.

The physical memory usage in the server is 14% (8Gb) still swapping is happening (page in and page out).

Recently our apllication failed with the following error
"java.lang.OutOfMemoryError: requested 536870920 bytes for Chunk::new. Out of swap space?
Possible causes:
- not enough swap space left, or
- kernel parameter MAXDSIZ is very small. "

We got some error in syslog also as below
"syslog: Java out of memory messages are marked with pid:10960 in /var/adm/syslog/syslog.log.
syslog: pid: 10960 4Kb 4Kb 4Kb PCT START/ 4Kb DEV NO ./
syslog: pid: 10960 TYPE AVAIL USED FREE USED LIMIT RESERVE PRI FILE N AME
syslog: pid: 10960 dev 6291456 169009 6122447 2.69% 0 - 1 64 (major) 2 (minor)
syslog: pid: 10960 reserve - 5590985 -5590985
syslog: pid: 10960 memory 15925771 3795255 12130516 23.83% "


What will be the cause of this fail? There are more than 80% of memory is free still server is doing paging.

Does anybody faced this type of issue? What will be exact reason for this?

OS Version is HP Unix 11.31.

Manoj K

[Prasanth V Aravind]

check kernel parameter usage history with kcusage.

Gudluck
Prasanth

[Prasanth V Aravind]

it seems these threads are much relevant.


http://forums13.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1274948890156+28353475&threadId=1208762

http://forums13.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1274948896355+28353475&threadId=748118

http://forums13.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1274948896657+28353475&threadId=179840

GUdluck
Prasanth

[Dennis Handly]

>The physical memory usage in the server is 14% (8Gb) still swapping is happening (page in and page out).

Where do you see 14 and where do you see paging? I see 23.83.

>There are more than 80% of memory is free still server is doing paging.

Why do you think it is paging? Have you looked at vmstat(1)?
Please provide the "swapinfo -tam" output when this happens.

>kernel parameter MAXDSIZ is very small.

Have you checked maxdsiz?

[ManojK_1]

Hi,

Please find the attachment for vmstat and swapinfo output.

I have verified the MAXDSIZ during the issue time it was only 38% used at that time.
May be the application is wrong or not inteligent enough to categorize the error.

Manoj K

[TTr]

Most likely the 169MB in the swap volume is idle. Maybe the server ran out of memory previously and stuff got swapped out to disk.

> Recently our apllication failed with the following error...

You would need to look at the memory/swap status right at the moment when you get the error.

What about maxdsiz? (It's a kernel parameter). What is its value? It may need to be increased. Java is asking for a 512MB chunk of memory.

[Bill Hassell]

> still swapping is happening (page in and page out).

page-in is a measurement of two very different conditions. A page-in occurs when a process starts or is loaded into memory for the first time. A page-in can also occur when a process is returned from swap space. Since there is no way to tell the difference in vmstat, pi (page-in) is only an indicator of how many process pages are being brought in. Thus, the only relevant metric for memory/swap usage is page-out (po). Note also that some swap space is also used for memory-mapped files.

> java.lang.OutOfMemoryError:

The suggestions are very generic for the actual cause. The most likely problem is local (heap) memory limits imposed by the current settings for maxdsiz and maxdsiz_64bit. These are defaulted to arbitrarily low values and the sysadmin is expected to raise these when valid programs require large amounts of RAM. I would set maxdsiz to 3800 MB and maxdsiz_64bit to 50% of your installed RAM.

Java does not investigate why a request for memory was rejected so you have to remove kernel limits first and then see if you need more swap space or RAM. Java has an multi-way startup for local memory assignments. When you specify a large amount of local RAM, for example 2200MB, then Java will start a different executable that can map 2.2GB.

[Steven E. Protter]

Shalom,

I've seen this before.

Java is allocating memory in huge portions out of a memory area that is undocumented in the OS and may not be accurately measured by the OS.

We had a similar problem with Oracle reports doing something similar, though overall memory usage statistics were not this far off. Oracle provided us a patch to resolve what was clearly an application issue.

Recommended steps:
1) OS patches recommended from Java
2) Obtain and install up to date Java version from http://www.hp.com/go/java

Ignite OS backup before implementing action item 1 or 2. If you have a test environment, test there first.

SEP

[chris huys_4]

Hi,

Add another 24Gbyte of device swapspace.

The standard practice on hp-ux, is still to create as much device swapspace as there is physical memory.

Only having 24Gbyte device swapspace on a system with 64Gbyte physical memory is a long way from that "standard practice".

48Gbyte of device swapspace vs. the 64Gbyte physical memory would look allready much better and it will also avoid that the system begins using pseudo swap to "reserve swapspace", with the current "load", when its out of device swapspace, like its doing now.

Greetz,
Chris
PS. Oh yes, logging a hp support call and getting crashinfo/kmeminfo tools and beginning to activily monitor, memory usage on this system, might also be a good idea. ;)

[Dennis Handly]

>Chris: The standard practice on hp-ux, is still to create as much device swapspace as there is physical memory.

This practice may be wasteful for large memory systems.