Available Memory Info Problem

[Vladimir Tihu]

With a program that uses pstat_getprocvm() call we find that real memory of all processes is Text+Data+Stack+Shared+MM: 1201 Mb (virtual 1768).
swpainfo -atm gives for the memory line 6340=available 4386=used 1954=free
top says 2091872K free (2043Mb)
The question is what occupies the memory up to what top or swapinfo report? Can it be the reserved memory swap? Or the dynamic kernel structures? The disk cache? Or the call to pstat_getprocvm() doesn't produce accurate results?

Thanks

[David_246]

Hi Vladimir,

See attached file.
Change the script and change :
for i in `UNIX95= ps -e -o vsz=Kbytes -o ruser -o pid,args=Command-Line | sort -rnk1 | grep -v Kbytes`
do
print "$i"
done

Regs David

[James Murtagh]

Hi Vladimir,

The pstat call is reporting the memory in user space (process usage).

Swapinfo isn't an accurate way of measuring memory usage, the memory section actually refers to psuedo swap.

Top will include system memory - the kernel data structures, kernel dynamic memory and the buffer cache etc, as you suspected.

If you want to find out approximately how much system memory used:

Use vmstat to find how much free memory pages you have. Subtract this from your physical memory (in pages)

# echo phys_mem_pages/D|adb -k /stand/vmunix /dev/kmem|tail -1|awk -F: '{print $2}'
# vmstat 1 1|tail -1|awk '{print $5}'

Find out how much buffer cache (pages) you are using:

# echo bufpages/D|adb -k /stand/vmunix /dev/kmem|tail -1|awk -F: '{print $2}'

So if you have already calculated process memory the system memory will be:

used memory - (process memory) - (buffer cache)

Obviously a lot easier if you have glance! :-)

Regards,

James.

[Vladimir Tihu]

Ran the script and found 2034104 Kbytes reclaimed; so there is still a missing cca 2000Mb unexplained (the used column from swapinfo); on the other hand, the script adds the sizes of all text segments but, since these are shared, the real reclaimed space is less (we have a number of processes that share the same text segment).

Thanks

[Vladimir Tihu]

Hi James,

Very useful commands, especially for finding the buffer cache size; seems that here resides the "lost" memory; I found that basically what top says it's free is what vmstat also reports and the difference (after including the process memory read with pstat call and eliminating duplicate text segments) is mainly this cache. Is there a way to find how the memory (pseudo) swap system handles memory (how much it reserves and uses)?

Thanks a lot

[James Murtagh]

Hi Vladimir,

The swapinfo command's "memory" section always leads to confusion! :-)

This memory section is just a slight of hand hpux uses to make the VM system think it has more swap space than it actually has. All processes need to reserve swap space before they run in case they need to page (any!) part of the process out. To do this the kernel uses a counter. It decreases the counter when a process starts and requests VM. This is where the reserve field comes in.

Consider an example. Using the old rule of thumb where swap=memory, on a 64GB system you would need 64GB disk space for swap. Bit of a waste as this server will probably be unlikely to page out in normal conditions. However, it may have to reserve massive amounts of the swap counter just to run. It we could take into account the physical memory the processes are occupying to increase this counter we could solve this problem, which is what psuedo swap does.

The normal rule of thumb for this is 75% of available physical memory. It is also equal to the amount of lockable memory your system has.

# dmesg|grep lockable
or
# echo lockable_mem/D|adb -k /stand/vmunix /dev/kmem

Regards,

James.

[Steven E. Protter]

swapinfo includes swap area stats.

Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 8388608 0 8388608 0% 0 - 1 /dev/vg00/lvol2
reserve - 132316 -132316
memory 3169288 171332 2997956 5%


That first line is my swap area, which is not used because the system isn't doing anything.

SEP

[James Murtagh]

Hi again Vladimir,

Maybe that wasn't the best explanation, told you it could be confusing! If you're like me you may prefer the numbers in an example. I'll use my workstation and work through it.

When the system starts up the swap space static counter swapspc_max is set to amount of device swap. The counter swapmem_max is set to the amount of psuedo swap.

# echo swapspc_max/D|adb -k /stand/vmunix /dev/kmem|tail -1|awk -F: '{print $2}'
131072
# echo swapmem_max/D|adb -k /stand/vmunix /dev/kmem|tail -1|awk -F: '{print $2}'
95268

So this indicates I have 512MB device swap and 372MB psuedo swap.

Initially the dynamic counter swapsps_cnt is set to swapspc_max and swapmem_cnt is set to swapmem_max. Every time a process reserves swap space this amount is decremented from swapspc_cnt.

# echo swapspc_cnt/D|adb -k /stand/vmunix /dev/kmem|tail -1|awk -F: '{print $2}'
71505

The way the swap is reserved the calculation for the swap in use is swapspc_max-(swapspc_max-swapspc_cnt), so from this we should have 281MB of swap in reserve.

Also, pseudo swap is only used as a last resort, when swapspc_cnt=0.

Lets check how this ties in with our swapinfo now:

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 512 0 512 0% 0 - 1 /dev/vg00/lvol2
reserve - 214 -214
memory 372 67 305 18%
total 884 281 603 32% - 0 -

So, we do indeed have 512MB device swap and 372MB pseudo swap. However, our reserve is 67MB short and the memory field seems to have it. Why is this considering what I have said before?

The 67MB is placed in the pseudo space field because it is already locked - it is the memory which cannot be paged (mainly system memory). So in effect it is already being used as pseudo swap would be if device swap was fully reserved.

I hope this made more sense.

Regards,

James.

[Bill Hassell]

Trying to figure out all the uses from memory in any Unix system is difficult. top's reports are the least useful. vmstat is a start but it doesn't give the whole picture. swapinfo shows reserved and actual usage plus pseudo swap assignments but not in an easy to read format.

HP-UX has many other uses for RAM that are hard to count. As mentioned, the buffer cache (likely dynamic on your system) might occupy asa much as 50% of your RAM (not good usually) and extensive use of shared memeory segments, shared libraries and memory mapped files will also lead to unaccounted memory.

About the only thigns you can control are the buffer cache size (kernel parameter) and application programs. And usually when sysadmins are asked about memory usage, it is due to a program that reports that it is out of memory. Unfortunately, out of memory is often a poor description of the real problem, either a kernel limit (maxdsiz or possibly maxssiz) or a user limit (ulimit). As long as you have a lot of swap, running out of memory is difficult.

Check the white papers in /usr/share/doc for memory and process management and pay close attention to the severe restrictions for 32bit programs. Lots of systems have many Gb of RAM but 32bit programs cannot take advantage of large RAM except within their limited addressing space. Same with shared memory and 32bit programs.

[Vladimir Tihu]

Hi everybody,

Sorry for the delay, I had to go to sleep!
I will read the documentation that you recommend.
For James, I must say the example started to enlighten me but I went check on my machine and noticed that swapspc_cnt is 1957260, swapspc_max is 2097152, swapinfo says reserve is 496Mb, memory used is 1176Mb, and the total dev used is 13Mb; so swapspc_max-swapspc_cnt is about 546Mb. I understand that the memory used means locked memory that doesn't have to be reserved for the process(es) but the rest of used memory? It must be system memory?
I find all the explanations very useful.

Thanks a lot and best regards

[James Murtagh]

Hi Vladimir,

Please keep asking the questions, I'll try and explain more each time, there is a lot of variables as Bill mentioned!

The important thing to remember is that the kernel is not a process and hence cannot reserve swap, either is the buffer cache and neither can be swapped out (bit simplistic but generally true) - they are fixed in memory. Hence they do not decrease the swapspc counters but the swapmem counters.

I think the best way to show this is if I fill my buffer cache and show you the effect it has on the reserve and memory lines. I'll show the swapinfo output before and after.

# swapinfo -tam|egrep "res|mem"
reserve - 202 -202
memory 372 68 304 18%

# echo bufpages/D|adb -k /stand/vmunix /dev/kmem|tail -1|awk -F: '{print $2}'
10726

# find / &

(wait till this finishes)

# echo bufpages/D|adb -k /stand/vmunix /dev/kmem|tail -1|awk -F: '{print $2}'
22976

# swapinfo -tam|egrep "res|mem"
reserve - 205 -205
memory 372 183 189 49%

So as you can see my memory field has increases a lot - simply due to the buffer cache increasing and adding to the locked memory in my system. As you have a large buffer cache this will be most of the count on your server too.

Regards,

James.

[Vladimir Tihu]

Hello,

I found what I was looking for; James' notes esp. were of considerable help, they helped me understand what seemed to be in the dark and provided a valuable base for better tuning the systems.
If it is not too much, for the moment I would like to ask just one more question:
James, in your "echos" you used standard input piped into adb -k /stand/vmunix...; these gives dynamic info (eg the swap dev counter, the count of free pages etc), but /stand/vmunix is a file on disk and it's timestamp is from when it was generated the last time, so where does this current info come from? Does adb go through memory mapped mechanism or some other way reads the system memory?

Best regards

Vladimir

[James Murtagh]

Hi Vladimir,

Glad I could help, it is certainly not obvious what swapinfo reports a lot of the time!

Adb is a debugger taking an object file and core file as arguments. In our case the kernel object (vmunix) is used and the core is kernel memory. The symbols we are using correspond to the entries in the symbol table of the vmunix object file, as it is not stripped. /dev/kmem is a pseudo driver for kernel memory access. We know these symbols will be in kernel memory as vmunix is loaded into memory when the system boots.

If you want to check other symbols you can find the value of using adb run:

# nm /stand/vmunix
or
# nm /stand/vmunix|awk -F\| '{print $8}'

as the symbols are the last field. If you know q4 you can also use:

q4> symbol

Regards,

James.

[Vladimir Tihu]

Thanks for adb explanation (I didn't use debuggers on hpux until now preferred writing C programs when possible, but they seem to be very useful).
Thanks James, thanks everybody.
Now I'll go do some tcpip programming then I'll return to hpux.

Best regards,

Vladimir