RAM needs and swapinfo

[Delphine TALLARON]

Bonjour à tous,

First, if somebody post an answer, I would like to limit the discussion on results given by swapinfo command.

I know that there are other more appropriate tools as Glance, but I do not use Glance, I have informations collected with standard commands as vmstat, ps, sar, swapinfo, and it is on the basis of these collections that I lead my reflection.

Having said that, my 1st approach is to tempt an estimation of the needs of RAM from the swapinfo command. Maybe it is a totally silly idea . In that case tell me. Maybe it is necessary to complete with other commands as vmstat, ps, tell also. I think it is, but I plan to open later other threads with vmstat, ps, ...

I use only logical volumes as swap devices. Thus to estimate the minimum memory I need, I would tend to say that it is necessary to add what is used by the swap devices, what is reserved and, may be, current size of filesystem buffer cache (bufpages).

On the other hand is it necessary to integrate the memory line in the compute? If yes how, knowing that the parameter swapmem_on modifies considerably the result?

On an 8 Go system doing NOTHING, if swapmem_on=0 I obtain:

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
reserve - 322 -322
total 4096 322 3774 8% - 0 -

if swapmem_on=1 I obtain:

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
reserve - 325 -325
memory 8162 946 7216 12%
total 12258 1271 10987 10% - 0 -

I thus have a 949 Mo difference in used Mo, simply by activating or not the pseudo-swap

If on the same "doing nothing system", with swapmem_on=0, I had obtained the following results, I would feel less stupid :

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
reserve - 1271 -1271
total 4096 1271 2825 31% - 0 -


Obviously I can't use as is the value in memory line and must do corrections. But according to which principles?

Thanks in advance

DT

[Victor BERRIDGE]

Bonsoir,

Lets look at the bottom line. At the launch of a process the system allocates the same amount of swap space for it too should it need
to be paged out and although it may not be using the swap space,it's allocated

So you can allocate as much as the size of the swap...(device file)
swapmem_on lets you go beyond if you have more RAM...

All the best
Victor

[Victor BERRIDGE]

I forgot to add:
If the output from 'swapinfo' shows sufficient device swap (1.5-2 times physical memory, or higher - depending on application requirements), you should be able to disable it and reclaim some memory for use by system/user processes.
About the filesystem buffer cache:
It can penalize your system's vitrual memory management capabilities if not well tuned (and so have a major impact...)

[Delphine TALLARON]

Bonsoir,

"swapmem_on lets you go beyond if you have more RAM"

Ok Victor, I understant that. But you don't explain the 949 Mo difference, on the same system, just by switching swapmem_on.

I just had a more accurate reading of "man swapinfo". For Kb USED it says :

"The current number of 1-Kbyte blocks used for paging in the paging area. For the memory paging area, this count also includes memory used for other purposes and thus unavailable for paging."

So I have now a better understanding. Ouf ... But what is there inside this "memory used for other purposes" ?

If shared memory is in this scope I have no shared memory but standard ones (no Oracle SGA, nothing at all) for about 27 Mo :

IPC status from /dev/kmem as of Tue Dec 4 19:15:20 2007
T ID KEY MODE OWNER GROUP SEGSZ
Shared Memory:
m 0 0x411c018a --rw-rw-rw- root root 348
m 1 0x4e0c0002 --rw-rw-rw- root root 61760
m 2 0x4120723c --rw-rw-rw- root root 8192
m 3 0x00a5c581 --rw------- sfmdb users 10469376
m 294916 0x06347849 --rw-rw-rw- root root 65544
m 5 0x02fb07f1 --rw------- hpsmdb users 10469376
m 32774 0x0c6629c9 --rw-r----- root root 17841928
m 65543 0x491815d3 --rw-r--r-- root root 22908


If filesystem cache is also in this scope, I have almost no filesystem buffer cache (81 Mo) since dbc_min_pct and dbc_max_pct are 1% each :

echo "bufpages/2d" | adb /stand/vmunix /dev/kmem
bufpages:
20895 0

So, whatelse (G. Clooney ;-) ? HP-UX kernel itself ? Does anybody know what is behind "memory used for other purposes" ?

Thanks in Advance

DT

[Delphine TALLARON]

Re-bonsoir Victor,

You said "you should be able to disable it and reclaim some memory for use by system/user processes."

I understood through my readings that pseudo-swap is not RAM consuming. It's just a mechanism that allows processes to reserve more swap space than device swap give. So if pseudo-swap is OFF I will not gain much more available memory.

And filesystem buffer cache is very low on my system because it is a test system.

DT

[Geoff Wild]

Have a read through:

http://www.met.ca/itrc/index.php?option=com_content&task=view&id=52&Itemid=2

It gives a good explanation of swap..

And this next one is about memory:

http://www.met.ca/itrc/index.php?option=com_content&task=view&id=38&Itemid=2

Rgds...Geoff

[Don Morris_1]

Yes, it is a totally silly idea. swapinfo has very little (practically nothing) to do with RAM needs in the general case.

swapinfo returns the usage, reservation and total for swap resources. Period. Swap resources are not RAM. Swap resources by default are, however, required for *virtual* memory allocation -- as such swapinfo will determine how large (barring tricks like lazy swap) your total User virtual address space can be. But just because someone creates a big virtual object does not mean they consume the same amount (or any) RAM.

As to why enabling/disabling pseudo-swap drives up your total used -- it is simply because when pseudo-swap is enabled, all memory locked pages also reserve from pseudo-swap (since they can't be paged anyway). To prevent deadlock between kernel dynamic allocations (which can't be paged either and can be considered locked for all intents and purposes) and true locked pages, the kernel will "steal" pseudo-swap corresponding to the amount of lockable memory on the system it consumes. Your 949Mb increase is the memory being consumed within the kernel. Why it is 949Mb in particular no one can say from this output.

If you want to estimate RAM needs, use a tool that measures RAM. If you want to estimate virtual address space requirements, swapinfo is fine (or Glance, etc.)

[A. Clay Stephenson]

Swapinfo is a rather bad tool for your purposes --- and 8GiB is considered a rather small system these days so relying upon pseudoswap for a system that size usually doesn't make sense. Pseudoswap is really intended for larger systems (e.g those with 32GiB or so) with less device or filesystem swap space than physical memory. I have a number of systems that fit that description and they run just fine with much less swap than physical memory. In your case, it makes more sense to add enough device swap to at least equal physical memory and then disable pseudoswap. Pseudoswap serves no purpose when swap >= RAM.

In deciding if any additional swap is needed above that, you really need to ask yourself what are your goals. If your top priority is performance then 1X swap space is sufficient because the moment you actually begin to start paging out significantly is the moment when performance drops significantly. If your priority is to run as many processes as possible then add swap to your heart's content --- and as much as you and your users' patience will allow.

You have probably badly crippled your box by specifying such a small amount of buffer cache. 81MiB is a very small cache for simply the OS itself much less any application code. I would tune it up to at least 256MiB or so.

The best free tool to monitor swap usage is vmstat. The only metric of any real value in vmstat's output is the pageout (po) column. When that value climbs above 10-20 for significant periods of time, your performance drops very noticeably. In that case the answer is to add more RAM or reduce the memory load.

[Tim Nelson]

My simple 2cents.

If I had a system with 8GB of RAM I would configure 10GB ( maybe 12GB )of device swap and leave swapmem_on set to 1.

As mentioned psuedo swap really is only a math game that the kernel plays for large mem systems where the admin may not wish to allocate many GB of disk space for device swap. As your memory needs increase psuedo swap will decrease an the requirement for device swap will increase.

Another simple statement.
If you wish to use all 8GB of ram you must have a minimum of 8GB of device swap configured.

Try it. Turn off swapmem_on, remove all device swap except for the minimally required primary. Start up some processes.

[Jdamian]

Just a remark: you wrote that you "have informations collected with standard commands as vmstat, ps, sar, swapinfo"

Be aware of the WARNING in manual pages of sar command:

WARNINGS
Users of sar must not rely on the exact field widths and spacing of
its output, as these will vary depending on the system, the release of
HP-UX, and the data to be displayed.

Due to that, two adjacent values might be displayed "glued" (without blanks between them). That might misrepresent the values in your reports. For example:

$ sar -v -f /var/adm/sa/sa03

HP-UX xxxxx1p B.11.11 U 9000/800 12/03/07

00:00:00 text-sz ov proc-sz ov inod-sz ov file-sz ov
00:20:00 N/A N/A 1112/4096 0 2730/6592 0 8591/13405 0
00:40:00 N/A N/A 1177/4096 0 2823/6592 0 9951/13405664
01:00:00 N/A N/A 1169/4096 0 2787/6592 0 10099/13405 0
01:20:01 N/A N/A 1168/4096 0 2763/6592 0 10068/13405 0
01:40:00 N/A N/A 1221/4096 0 2756/6592 0 13296/134054561
02:00:00 N/A N/A 1162/4096 0 2750/6592 0 13189/134051775
02:20:00 N/A N/A 1156/4096 0 2746/6592 0 13174/1340592
03:40:00 N/A N/A 1193/4096 0 2657/6592 0 13395/13405873073
04:40:00 N/A N/A 1200/4096 0 2641/6592 0 13300/134051357774
05:20:00 N/A N/A 1198/4096 0 2633/6592 0 13382/13405994272
06:00:00 N/A N/A 1195/4096 0 2632/6592 0 13398/1340540113
06:20:00 N/A N/A 1200/4096 0 2640/6592 0 13400/1340587358
07:20:00 N/A N/A 1209/4096 0 2653/6592 0 13340/13405741615
07:40:00 N/A N/A 1131/4096 0 2657/6592 0 11628/13405198
08:00:00 N/A N/A 964/4096 0 2578/6592 0 6094/13405 0

[Delphine TALLARON]

Bonjour a tous,

First, thanks to every body for your answers.

Geoff, I am now confused between paging and swapping. I read :

- "If vhand finds any "non-active" pages, vhand will move those pages to the swap area.". I think that this activity is seen in vmstat with "po" column.

- "If available memory drops too low, another daemon called swapper will remove or swap out the entire process. Swapper will continue to push processes from physical memory to the swap area". I think it is swapping activity as we can see under column "so" in vmstat or sar -w

Yes, but I can read later "At HP-UX 10.x and up, the entire process is not swapped out, only parts of the process are swapped. The term for a portion of a process that is swapped is called paged.". So what difference between swapping and paging ?

Don, you said "But just because someone creates a big virtual object does not mean they consume the same amount (or any) RAM". I totally agree. But if we can imagine one second that each bit of those "big virtual objects" is in use, the "total Used" of swapinfo would be the necessary RAM. In other words, I could say that "total Used" is POTENTIAL need, a UPPER LIMIT which has FEW CHANCE TO BE REACHED ?

"Your 949Mb increase is the memory being consumed within the kernel. Why it is 949Mb in particular no one can say from this output.". There is no way to be more precise ? For example does current size of buffer cache impact "used Kb" in line memory ?


Clay, sizing of swap, using or not pseudo-swap are things rather clear to me (I think ;-) The fact that buffer cache is very low is desired for testing purposes : the system i am working on is a kind of "sandbox" (I don't know english term for this ?). But I do retain that "The best free tool to monitor swap usage is vmstat. The only metric of any real value in vmstat's output is the pageout (po) column. When that value climbs above 10-20 for significant periods of time, your performance drops very noticeably." It is a very pragmatic information.

Tim, "pseudo swap is on only a math game" seems to confirm my opinion : though "Pseudoswap serves no purpose when swap >= RAM." it doesn't affect global system performance.

Oscar, I wrote a portion of code to control number of columns of sar outputs. It was a good thing to recall it.


My main goal is to estimate reasonably the amount of memory I need on production systems. I will retain that swapinfo is not the good tool to do a reasonable estimation, but it will give something like a upper limit. It's a beginning. I will also retain that to estimate size of my kernel I can shrink dbc_max_pct to 1 and reboot in a "system_doing_nothing" state and take a look at memory line.

I still have to do a reasonable estimation ... but I will open a new thread for this as soon a syou have no more things to say in this one.

DT

[Yogeeraj_1]

Bonjour Delphine!

juste une petite clarification.

Are you trying to size new servers or trying to tune existing servers?

If you have existing servers, you can existing tools to determine the current usage and gauge accordingly.

If you are running an application, e.g. Oracle, you have the possibility to use the provided tools to view the performance of your systems (e.g. Oracle Enterprise Manager Database control)

cordialement,
Yogeeraj

[Don Morris_1 · Accepted solution]

Yes, certainly if you assume every last virtual page gets touched that would require the equivalent physical memory -- but what isn't reflected there are:

a) Physical memory requirements for the kernel at that time (current requirements are reflected via the pseudo-swap stealing if swapmem_on is enabled... but you don't know what the kernel will ask for based on existing user virtual objects)

b) Physical memory requirements for non-swap backed objects. Any mapping of a file (like the Text segment of a binary or a mmap()) where any changes made are reflected in the file (in other words, not a private/local mapping) is already "swapped" to the file itself.. and hence doesn't need space on the swap device. mmap() is pretty common -- so relying on swapinfo to approximate RAM is going to be missing that very large chunk.

In other words, your proposed Upper Limit is still a low-ball estimate even assuming it is possible to hit it on the workload... you simply don't know from swap reservations and consumptions alone.

Now I'll grant that if you're running Oracle and your SGA is 92% of the system it isn't as hard to be close... but then again, you tend to know what the RAM requirements are anyway to set that up.

Next question -- of course there are ways to be more precise... just not with the swapinfo output you gave us. Glance's Memory Report would at least tell you the physical memory consumed by Buffer Cache vs. File Cache vs. "Everything Else non-User" (System).

To get below *that* requires some very kernel intrusive stuff (since how memory is managed and handled in the kernel can change in patches, much less releases). The only way to get that type of information would be to ask your support contact for the kmeminfo tool which does that sort of intrusive reporting. Personally -- unless you think you've terribly mis-tuned the system (which I don't see signs of) -- I can't see what value there really would be in getting that level of detail. You can't change it (not being a kernel developer or driver writer), there's no memory pressure anyway (and since the kernel dynamic memory allocator favors caching for performance where there's no memory pressure, there may be memory which will come back to the system very easily when you ramp up the memory load you won't be able to pick out) and a good chunk of it is absolutely unavoidable (boot time allocations which depend solely on your hardware configuration, etc.)

Buffer cache does not consume swap, by the way (for the same reason file backed pages don't -- if the data is already out on storage and we keep it sync'ed up... you don't need to waste time putting it somewhere else). I was referring to dynamic kernel memory allocations [the equivalent to calling malloc() in the kernel].

Other points/questions:

Under memory pressure, vhand will run to page out pages of RAM from victim processes (sub-process memory reclamation). If the pressure gets bad enough, it will pick entire processes to be deactivated (halted) and swapper will then push all remaining pages in the process [including pages which can not be paged by vhand such as thread stacks and metadata describing per-page state] to swap. So paging is sub-process, swapping is whole process... really that simple.

I'll disagree on pseudo-swap serving no purpose when you have tons of swap but that's more from the point of view of symmetrical resource handling in the kernel and the fact that it doesn't hurt either. (There's a reason we removed the option to turn it off in 11iv3, after all...)

I agree with Yogeeraj that the best way to approximate RAM needs is to correlate your working set with your solution stack (and your solution stack providers frequently have tools/insight to help). The kernel is a small part of that -- and how the application drives kernel load should be accounted for in the providers' data.

[Delphine TALLARON]

Don,

special thanks for the time you took and the quality of your answers.

Yoggerai,

I try to understand what's happening on a production system on a strict OS point of view. Since we have "paging out" on this system, my guess was to determine how much RAM this systeme should have to avoid paging. It's not so obvious :-(

ISV consultants and dba are working together and our works will be soon correlate.

Thxs again every body

DT