Memory usage - confusing

[MohitAnchlia]

Java app runs on HP 11 box. Now after running my code through the profiler I see that only 200 MB of heap space is being utilized but when I run top command I see 200 MB of RES space and total size is 1 GB. I tried to look at profiler but can't see where it's using extra 800 MB of memory. This application extracts blob and then encrypts them and writes to the file system. There are 4 threads that extract different kind of data. Any suggestions on where memory is being utilized

[Hein van den Heuvel]

That's just the difference between 'virtual memory' or 'address space' and real, touched memory.

If a program 'mallocs' 1Gb and does nothing else, you might see SIZE=1GB and RES=1MB
And those values coudl be appearing on a 64GB physical memory system, or a 512MB system (if you can still find one that small).

Check out the Java command line options used, notably:

Xms = the initial/minimum Java heap size within the VM.
Xmx = the maximum Java heap size within the VM.

Google for: +java +memory +tuning for more info. Toss in +Xms or +Xmx as desired :-)

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[MohitAnchlia]

Is there a way to find out where it's sucking so much memory. When I did man on top the definition of SIZE is interesting:

--
Total virtual size of the process in kilobytes. This includes virtual sizes of text, data, stack, mmap regions, shared memory regions and IO mapped regions. This may also include virtual memory regions shared with other processes.
---
Interesting part is that virtual memory region could be shared with other processes.

1. So, is there a way to find out more detailed information about memory usage of a process ?
2. How to determine how much virtual memory is being shared by other processes ?

[Zeev Schultz]

If you have Glance you can map memory regions of a process. Mostly,for shared regions it would be *.so (shared libraries).

[MohitAnchlia]

I'll look at manual. But if somebody could also tell me what steps I need to follow to map a process to shared memory regions and what to deduce from it, and how to understand the output, would be helpful.

[Avishay_Cohen]

java uses garbage collector. a facility with different policies for different occations who does all memory utilization for proper java oprations. when a need for memory arrises, the GC according to its policy frees (or accuires more) memory from the OS.
what glance sees is the OS POV. http://www.javaworld.com/javaworld/jw-08-1996/jw-08-gc.html
http://java.sun.com/docs/hotspot/gc1.4.2/

[MohitAnchlia]

I am confused because when I do top I see SIZE as 1 GB and RES as 200 MB. But when I run it through the profiler total heap size allocated is just 200 MB by JVM. I see 800MB worth memory unaccounted for.

[Don Morris_1 · Accepted solution]

I don't know if Java is expressing the Heap in terms of real page usage (SZ) instead of virtual size (VSZ) or if you've just got 800Mb of shared libraries, stack, text, etc... all of which would show up in SIZE for top.

Try compiling the attached (+DD64 or +DD32 -D_PSTAT64), and my apologies in advance if it doesn't compile because I've used something too new... it _should_ be okay barring some printf() format conversion complaints.

Point it at the pid of your Java process with at least one level of verbosity ( -v -p ) and you should get a breakdown by object type for Virtual and Physical memory consumption. That should narrow things down.

Other alternatives are, as always, Glance -- any number of other programs/scripts folks have written to talk to pstat if you don't like this one, kmeminfo if you can get it from support, etc.

[Don Morris_1]

Argh! And I forget to attach the code while blathering on about how to use it. My apologies (and no points on this followup, certainly!)

[MohitAnchlia]

I am unable to download the attachment.

[Hein van den Heuvel]

I could download just fine.
Something wrong on your side / site.
need to click that 'go ahead it is safe' in the 'yellow bar' in a browser window? It might have appeared in a different browser window if you have multiple. Try again?

Just in case the extention ".C" caused a problem I'll re-attach Don's source as ".txt"

Also... google for pstat_getprocvm will get several alternatives.

fwiw,
Hein.

[MohitAnchlia]

I compiled and ran, but I just get this:

$ aCC -o meminfo 298591.c
$ meminfo -v -p 11765
VIRT/PHYS/LOCKED/SWAP summaries in pages.
System page size is 4096 or 0x1000 bytes.

[Don Morris_1]

Just to eliminate the obvious, compile with +DD64 or +DD32 -D_PSTAT64 as I requested. Calling pstat on a 64-bit kernel without handling 64-bit values is a good way to get garbage results [such as perhaps not finding the pid you though you should].

[MohitAnchlia]

How do I interpret these results ? Are these in bytes ?

PID 11765:
UNUSED TYPE consumes 0 VIRT, 0 PHYS, 0 LOCKED and 0 SWAP.
UAREA consumes 192 VIRT, 192 PHYS, 0 LOCKED and 192 SWAP.
TEXT consumes 16 VIRT, 15 PHYS, 0 LOCKED and 0 SWAP.
DATA/HEAP consumes 10223 VIRT, 10095 PHYS, 0 LOCKED and 10223 SWAP.
MAIN STACK consumes 576 VIRT, 96 PHYS, 0 LOCKED and 576 SWAP.
SYSV SHMEM consumes 0 VIRT, 0 PHYS, 0 LOCKED and 0 SWAP.
NULL DEREF consumes 0 VIRT, 0 PHYS, 0 LOCKED and 0 SWAP.
MEM MAPPED I/O consumes 0 VIRT, 0 PHYS, 0 LOCKED and 0 SWAP.
MMAP consumes 308674 VIRT, 54399 PHYS, 0 LOCKED and 45134 SWAP.
GRAPHICS SPECIFIC consumes 0 VIRT, 0 PHYS, 0 LOCKED and 0 SWAP.
GRAPHICS SPECIFIC consumes 0 VIRT, 0 PHYS, 0 LOCKED and 0 SWAP.
RSE STACK consumes 0 VIRT, 0 PHYS, 0 LOCKED and 0 SWAP.
PID 11765 Mb in Use:
Virt: 1248 Phys: 253 Swap: 219

[Don Morris_1]

No, those are all in pages -- hence the line about:
"VIRT/PHYS/LOCKED/SWAP summaries in pages"
followed by telling you how many bytes are in a page for your system [yes, it isn't always 4096 ;) ]: "System page size is 4096 or 0x1000 bytes".

So for your Data/Heap -- that process is using 10095*4096 = 41349120 bytes (40380kb... 39.43359375Mb) of Physical Memory [RAM].

MMAPs are the biggest Virtually at 1.17749Gb -- if this is the process you started the thread about, there's your answer. If you want to know the sizes of each object, add another "-v" option [verbose level greater than 1 spits out the virtual range in question for each virtual object in the process]. That may not have much meaning for you -- though in this case it would let you know if that's one big mmap or just a whole pile of little ones.

I'd note though that there's only around 212Mb in use (and if these are MMAPs of shared libraries, some of that cost is going to be independent of the process anyway... any process linked to the library will show at least some consumption for it).... so I really wouldn't think any of this is worth worrying about unless you're actually running out of swap or something.

[MohitAnchlia]

Thanks. So if 212MB is in use then why does it report the total size of 1.7 GB ? I know for sure it doesn't need more than 250 MB because I've run it in profiler

[Don Morris_1]

Two possibilities leap to mind:

1) The profiler reports your expected/worst case physical memory consumption for your direct allocations (RES, not SIZE).

2) Your profiler may report your expected virtual memory needs... but doesn't report every single virtual object which will be affiliated with your process when it runs (like a file being mmap()'d, the total virtual size of every shared library attached, etc.). This is my suspicion since you specifically mention you're profiling your heap (which usually means that somewhere there's a malloc() or brk() call... I don't think Java's odd enough to do this stuff on the Stack, though it might use private MAP_ANONYMOUS mmap() calls. Your DATA/Heap virtual usage is well within the profiling boundary -- it is the other mmap() objects in the process that are using the virtual address space.

[MohitAnchlia]

What really is mmap ? Is it something that could be shared between other processes too ? Does it mean that some space is reserved but could be used by other processes, if required ?

[Don Morris_1]

http://docs.hp.com/en/B2355-60130/mmap.2.html

Yes, mmap() created objects [files or Anonymous memory ranges] can be either shared or private. Yes, shared objects affect the virtual size of all processes attached to them.

At this point, perhaps you should look over:
http://docs.hp.com/en/1218/mem_mgt.html

[MohitAnchlia]

thanks I'll try to read the manual. When I look at glance I see FIle Name as

What does it mean and how can I trace that inode number to actual file ? One of these files is taking 90MB space.

[Don Morris_1]

I would think the simplest way would be to use lsof (http://hpux.cs.utah.edu/hppd/hpux/Sysadmin/lsof-4.78/)

[MohitAnchlia]

my application doesn't do anything in /opt directory. When I look at glance it shows several MEMMAP that refers to regular file pointing to /opt. When I look at lsof it also points me to /opt - no more details. So I am not able to understand why there are allocations pointing to /opt.

[Don Morris_1]

Well, obviously it does in code you don't know about (like libc or some other system library below your code itself...).

Just trace the thing if you can start a new one:
http://hpux.cs.utah.edu/hppd/hpux/Sysadmin/tusc-7.8/

[Dennis Handly]

>When I look at glance I see File Name as
>What does it mean and how can I trace that inode number to actual file?

If you know the filesystem, you can use brute force to find it:

find /opt -inum 22451

If it is /opt, it may be a shlib.