Understanding HP-UX memory usage

[Ed Loehr_1]

I'm missing some knowledge about HP-UX memory metrics, maybe someone can help...

On one of our ia64 11.23 DB boxes, I see the sum of the 'sz' column in 'ps -elf' (with 4096b pagesize) amounts to 4.4gb for all processes in the process table. I calculated that total by multiplying the ps numbers by 4096 and then summing them. My measurements do not agree with the man page, which says pagesize is _SC_PAGE_SIZE from unistd.h (3001 bytes). I wrote the attached simple C program to test via malloc + memset. I run the program, which mallocs and memsets the memory, and the sleeps long enough to let me look at the 'ps -elf' number. Empirically, the pagesize seems to be obviously 4096 (vs. unistd.h _SC_PAGE_SIZE = 3001). What am I missing?

Second, the box has a total of 16gb of RAM. Glance GBL_MEM_UTIL says it is 80% consumed. dbc_max_pct is 20% (3.2gb). If ps says 4.4gb is being used, and 3.2gb is allocated for the OS buffer cache, that leaves 16gb - 4.4gb - 3.2gb = 8.4gb unaccounted for. If pagesize is 3001 and not 4096, then even more memory is unaccounted for. How do I find it?

Thanks,
Ed

[Bill Hassell · Accepted solution]

The page size for HP-UX has been 4096 forever. Not sure what the 3001 number actually is since it is certainly an oddball number, certainly unrelated to calculating memory usage in HP-UX. The standard ps values are only part of the story. sz (and the UNIX95 feature vsz) don't account for some of the program pieces. Most often overlooked are the IPC features such as shared memory (see ipcs), shared libraries, memory mapped files and so on. For the complete picture, get a copy of the HP-UX Intenals book by the 2 Chris's (Cooper and Moore). Chapter 6 gets into the details.

[Ed Loehr_1]

The book sounds good, thanks for that tip, Bill. I found 3.4gb in shared mem via ipcs. Urgency leaves me with a couple immediate questions...

Glance says 16gb of RAM is 80% exhausted, with 2.9gb "free", and 3.2gb in dynamic buffer cache. My database server is asking for 110MB in shared memory at start-up, but won't start because of memory shortage. My DB server definitely demands more than shared memory alone. But it's difficult to conjecture my DB server is asking for 2.9gb or more in total memory (shared libs, memory-mapped files, etc). What gives? Must I assume the sumtotal of the DB server request exceeds 2.9gb??

As a rule of thumb, what percentage of physical RAM, as reported by glance, should I expect to be able to use?

Thanks,
Ed

[Ermin Borovac]

_SC_PAGE_SIZE is just a symbolic constant that can be used to get memory page size with getconf.

$ getconf _SC_PAGE_SIZE
4096

I think that your buffer cache is too large. With 16G of RAM you should look into changing the buffer cache size to 800M or 1G maximum.

[Bill Hassell]

Free memory in HP-UX is misleading, just like it is in vmstat. The total amount of memory (which is virtual) is found with swapinfo -tm. Your DBA can ask for 30Gb of shared memory for SGA if desired, as long as the combination of RAM and swap space exceeds that amount (plus space for processes which are sort of insiginifcant compared to 30Gb for shared memory). So you really don't care about free space from Glance. What you care about is paging (swap activity). The "m" screen in Glance will show paging, but don't pay attention to page-in which is the total of actual pagei-in *and* process starts. Only pay attention to page out where real data is sent to the swap area.

Now database apps are notorious for not telling you anything useful about memory problems. If the app is 32bits, the max local data area is just under 1Gb. Recompile for EXEC magic (cc options -Wl -N) and the program can reach about 1700 megs. Use chatr to set +q3p enable (2700 megs) and +q4p enable (3700 megs). Or recompile the app as a 64bit program and local data is unlimited (well, multiple terabytes of RAM).

Use ipcs -bmop to look at shared memory allocation. And note that the above recommendations are for PA-RISC 11i machines which may not translate 100% for ia64 at 11.23. Of course, the maxdsiz/maxdsiz_64 values (for local data) and shmmax values in the kernel must allow the program to grab lots of data. Otherwise, it will say it can't get more than 100 megs of local or shared memory (because of the kernel fences).

I would agree that the buffer cache is far too large. 500-900 megs is all you need, less if your DBA uses record buffering in SGA. That will likely not change your database program's problem. Also look at ulimit -a (specifically ulimit -d) to make sure the shell environment allows for large memory.

[Sandman!]

Link to the hp-ux memory management white paper...

http://docs.hp.com/en/1218/mem_mgt.html

[Bill Hassell]

Sandman provides an excellent link to the PA-RISC whitepaper, good reading for anyone wondering about memory management. Unfortunately, it is quite dated and applies only to 11.11 v1 PA-RISC. While 11.11 v2 is targeted at a converged HP-UX model, there are still some differences in the ia64 platform.

I've attached a small C program that demonstrates how much memory a program can obtain from your system depending on how it is compiled. If you have lots of swap space (16Gb or more), the 64bit version of this program may grab more than 20Gb with no problem. Don't worry, the program doesn't write anything so the RAM is just reserved and won't cause active programs to page out. All memory is returned when the program terminates. It is lots of fun to watch this program using Glance. Be sure to compile and chatr it each of the ways shown in the comments to see how even a 32bit program can grab almost 4Gb. Note also that settings for ulimit (shell), maxdsiz and maxdsiz_64 (kernel) will limit the program.

[Bill Hassell]

Sorry about the previous attachment, it can be opened if Wordpad or Word but Notepad will be unhappy with the Unix-style line terminators. Here is the PC-version.