System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

Memory usage is only 14% still swapping is happening. why?

ManojK_1
Valued Contributor

Memory usage is only 14% still swapping is happening. why?

Hi,

We are having a HP Integrity server with 64Gb physical Memory and 24Gb Swap.

The physical memory usage in the server is 14% (8Gb) still swapping is happening (page in and page out).

Recently our apllication failed with the following error
"java.lang.OutOfMemoryError: requested 536870920 bytes for Chunk::new. Out of swap space?
Possible causes:
- not enough swap space left, or
- kernel parameter MAXDSIZ is very small. "

We got some error in syslog also as below
"syslog: Java out of memory messages are marked with pid:10960 in /var/adm/syslog/syslog.log.
syslog: pid: 10960 4Kb 4Kb 4Kb PCT START/ 4Kb DEV NO ./
syslog: pid: 10960 TYPE AVAIL USED FREE USED LIMIT RESERVE PRI FILE N AME
syslog: pid: 10960 dev 6291456 169009 6122447 2.69% 0 - 1 64 (major) 2 (minor)
syslog: pid: 10960 reserve - 5590985 -5590985
syslog: pid: 10960 memory 15925771 3795255 12130516 23.83% "


What will be the cause of this fail? There are more than 80% of memory is free still server is doing paging.

Does anybody faced this type of issue? What will be exact reason for this?

OS Version is HP Unix 11.31.

Manoj K
Thanks and Regards,
Manoj K
15 REPLIES
Prasanth V Aravind
Trusted Contributor

Re: Memory usage is only 14% still swapping is happening. why?

check kernel parameter usage history with kcusage.

Gudluck
Prasanth
Dennis Handly
Acclaimed Contributor

Re: Memory usage is only 14% still swapping is happening. why?

>The physical memory usage in the server is 14% (8Gb) still swapping is happening (page in and page out).

Where do you see 14 and where do you see paging? I see 23.83.

>There are more than 80% of memory is free still server is doing paging.

Why do you think it is paging? Have you looked at vmstat(1)?
Please provide the "swapinfo -tam" output when this happens.

>kernel parameter MAXDSIZ is very small.

Have you checked maxdsiz?
ManojK_1
Valued Contributor

Re: Memory usage is only 14% still swapping is happening. why?

Hi,

Please find the attachment for vmstat and swapinfo output.

I have verified the MAXDSIZ during the issue time it was only 38% used at that time.
May be the application is wrong or not inteligent enough to categorize the error.

Manoj K
Thanks and Regards,
Manoj K
TTr
Honored Contributor

Re: Memory usage is only 14% still swapping is happening. why?

Most likely the 169MB in the swap volume is idle. Maybe the server ran out of memory previously and stuff got swapped out to disk.

> Recently our apllication failed with the following error...

You would need to look at the memory/swap status right at the moment when you get the error.

What about maxdsiz? (It's a kernel parameter). What is its value? It may need to be increased. Java is asking for a 512MB chunk of memory.
Bill Hassell
Honored Contributor

Re: Memory usage is only 14% still swapping is happening. why?

> still swapping is happening (page in and page out).

page-in is a measurement of two very different conditions. A page-in occurs when a process starts or is loaded into memory for the first time. A page-in can also occur when a process is returned from swap space. Since there is no way to tell the difference in vmstat, pi (page-in) is only an indicator of how many process pages are being brought in. Thus, the only relevant metric for memory/swap usage is page-out (po). Note also that some swap space is also used for memory-mapped files.

> java.lang.OutOfMemoryError:

The suggestions are very generic for the actual cause. The most likely problem is local (heap) memory limits imposed by the current settings for maxdsiz and maxdsiz_64bit. These are defaulted to arbitrarily low values and the sysadmin is expected to raise these when valid programs require large amounts of RAM. I would set maxdsiz to 3800 MB and maxdsiz_64bit to 50% of your installed RAM.

Java does not investigate why a request for memory was rejected so you have to remove kernel limits first and then see if you need more swap space or RAM. Java has an multi-way startup for local memory assignments. When you specify a large amount of local RAM, for example 2200MB, then Java will start a different executable that can map 2.2GB.


Bill Hassell, sysadmin
Steven E. Protter
Exalted Contributor

Re: Memory usage is only 14% still swapping is happening. why?

Shalom,

I've seen this before.

Java is allocating memory in huge portions out of a memory area that is undocumented in the OS and may not be accurately measured by the OS.

We had a similar problem with Oracle reports doing something similar, though overall memory usage statistics were not this far off. Oracle provided us a patch to resolve what was clearly an application issue.

Recommended steps:
1) OS patches recommended from Java
2) Obtain and install up to date Java version from http://www.hp.com/go/java

Ignite OS backup before implementing action item 1 or 2. If you have a test environment, test there first.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
chris huys_4
Honored Contributor

Re: Memory usage is only 14% still swapping is happening. why?

Hi,

Add another 24Gbyte of device swapspace.

The standard practice on hp-ux, is still to create as much device swapspace as there is physical memory.

Only having 24Gbyte device swapspace on a system with 64Gbyte physical memory is a long way from that "standard practice".

48Gbyte of device swapspace vs. the 64Gbyte physical memory would look allready much better and it will also avoid that the system begins using pseudo swap to "reserve swapspace", with the current "load", when its out of device swapspace, like its doing now.

Greetz,
Chris
PS. Oh yes, logging a hp support call and getting crashinfo/kmeminfo tools and beginning to activily monitor, memory usage on this system, might also be a good idea. ;)
Dennis Handly
Acclaimed Contributor

Re: Memory usage is only 14% still swapping is happening. why?

>Chris: The standard practice on hp-ux, is still to create as much device swapspace as there is physical memory.

This practice may be wasteful for large memory systems.
Nafez ALNajjar
Frequent Advisor

Re: Memory usage is only 14% still swapping is happening. why?

Hi,
vhand daemon will give an indication of how busy your system with swapping try :
#ps -ef |grep vhand

also find the attached document wich may help in troubleshooting java memory related issues.
chris huys_4
Honored Contributor

Re: Memory usage is only 14% still swapping is happening. why?

>Chris: The standard practice on hp-ux, is
> still to create as much device swapspace as
> there is physical memory.

> Dennis: This practice may be wasteful for
> large memory systems.
Not as long as 4Gbyte of memory on a hp-ux itanium system costs 4 times the price of a 146Gbyte internal disk. ;)

Greetz,
Chris
ManojK_1
Valued Contributor

Re: Memory usage is only 14% still swapping is happening. why?


Hi,

I have raised the issue with HP and the application vendor and they were working on it.

If the issue got solved i will update the details in the forum.


Manoj K
Thanks and Regards,
Manoj K
SoorajCleris
Honored Contributor

Re: Memory usage is only 14% still swapping is happening. why?

Hi Manoj,

Your OS version is 11.31 and if you are talking about vhand deamon activities,this is as an expected behaviour.

Vhand handles aging and stealing pages in the filecache.

When you see this in 11.31 , don't worry untill and unless system has memory pressure.


Regards
Sooraj
"UNIX is basically a simple operating system, but you have to be a genius to understand the simplicity" - Dennis Ritchie
Mancboy
Valued Contributor

Re: Memory usage is only 14% still swapping is happening. why?

with java bases systems it is always better to have more swap than there is RAM.
Java will and does leak, we've seen java processes locked at 2Gb, take 15Gb.
HPUX reserves the memory in swap "just in case it needs to use it", if there isn't enough swap available to reserve, it won't schedule the job - I've seen this enough times after raising countless calls to HP on this.
Also, java will sap system memory - kmeminfo shows how poor HPUX is, if you have glance installed you can see the memory getting hived off to system over time.

HP still recommened twice the swap of memory, and even though I'm opposed to this waste of money, it has proven very useful in keeping the systems up and running.
So, to keep the system afloat, 64Gb swap, to guarantee no vhand / swap issues, 128Gb Swap.

Trust me, in the long Java run, it's worth it just to shut syslog up.
Rita C Workman
Honored Contributor

Re: Memory usage is only 14% still swapping is happening. why?

Did read every line above....so forgive me is this is covered.

But on that vmstat that shows page outs.....have you cleared the accumulators on vmstat to ensure you're seeing current issues, and not just old info.

man vmstat

vmstat -z

Then monitor for page-outs..

/Rita