howmuch free memory

[Geetam]

How do I find out whether I have enough memory for another database instance without swapping to disk. (sorry, I don't have Glance installed, yet)

Either I have never seen an answer to this question, or it was too difficult to remember...

Can you also remind me how to reduce buffers to get more memory for the database instance.

[Victor BERRIDGE]

Use vmstat
vmstat 1 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
0 0 0 10598 299827 20 21 0 0 0 0 0 0 149 53 8 3 89
0 0 0 10598 299800 11 0 0 0 0 0 0 331 526 86 35 1 64
0 0 0 10598 299800 8 0 0 0 0 0 0 365 824 140 21 3 76
0 0 0 11109 299800 6 0 0 0 0 0 0 353 702 126 1 2 97
0 0 0 11109 299800 4 0 0 0 0 0 0 344 596 113 2 -3 102

[James R. Ferguson]

Hi:

You can also get some indications from:

swapinfo -t

...JRF...

[Stefan Farrelly]

simply use top see how much free memory you have (FREE MEM). If its low then you wont want to add another database without freeing up some memory first by say adjusting the buffer cache or adding some more RAM. The free mem column from vmstat is in pages (4096bytes per page) so you need to do some math, use top, its easier.

If you want to reduce your buffer cache then check the kernel parameter bc_max_pct.
Try reducing it to free up some memory, build a new kenerl and reboot and see how much free ram you have now. To see if your system is swapping to disk then use swapinfo -mt and look at the used column. %0 used is great, no swapping.

[Anthony deRito]

Geetam, keep in mind that HP-UX is a virtual memory operating system. It works very well with only bringing in parts of the process that needs to be in memory in order to run it. In fact, it would rather keep most of the process in VM rather than in real memory in order to maintain a large free list.

What I am trying to say is that at the time you install and run the additional database instance, you will have to measure your real memory and virtual memory stats in order to determine if there is any memory pressure.

Use vmstat and swapinfo -tma to monitor memory usage. You can post the outputs here if you need help interpreting the values.

Tony

[CHRIS_ANORUO]

Hi Geetam,

Go through this attached document. It has all you can learn to tackle job.
You can also use sysdef; adb command (echo "bufpages/D" |adb /stand/vmunix /dev/mem) to determine your bufpages.
From kernel configuration, set swapmem_on=1;
bufpages=0
nbuf=0
So the bufpage caching will be dynamic and you have 75% of you total physical memory as virtual reserved memory.

[Bill Hassell]

HP-UX is indeed a virtual memory machine so free memory is not too important. What is important is when you add additional porcesses, that the paging rate is reasonable (the po column in vmstat). Page outs indicate that additional memory was rfequired and occupied memory had to be freed by writing a few pages to disk. This is normal.

However, if the page out rate is dozens to hundreds, then RAM is indeed too small as the processes are constantly fighting for RAM. HP-UX manages memory in a very complex manner such that a simple value like free memory won't tell the entire picture.

You'd be better off looking at Glance. You can load a copy immediately from Application CDROMs. If you like the results from Glance's screens, you can order a permanent codeword from HP.

[Antoanetta Naghiu]

All the above are valuable points of view... You said you do not have Glance installed, yet. Should I understand you could have no problems if you had Glance? Glance has a trial version that you can install and use it for 60 days. Calling HP, you can get 30 more days of evaluation copy... total of 90 days could give you the posibility to get an answer to your Oracle issue and as well to the management if baying or not...

[Geetam]

The reason why I don't want any swapping or paging, is that the memory used by an
Oracle database instance is almost all buffers for data etc. If that starts to
page to disk (however efficient that may be in HP-UX), performance will degrade.
(you would not want your disk buffers to be paged out/in, do you?)

Some of your replies seem to say: "try it and see". I am looking at a live system
and I do not want to take any risks.

My problem is knowing what is virtual memory and what is real memory...

I copy output from some commands below.
I see that 'vmstat' 1229 * 4K = 4916 is roughly the same as 4892K in 'top'.
Is this my answer? Just under 5Mb free? Or is virtual memory included in this
figure?

When I add up the last column of 'ipcs -mob' I get 887941788 (846M). Where does
that fit in the picture?

I have 1Gb RAM installed, I don't recognise that anywhere in the output below
(just making sure I understand what I see).

Or am I making a thinking error somewhere?

---------------
# vmstat
procs memory page
faults cpu
r b w avm free re at pi po fr de sr in
sy cs us sy id
0 0 0 23593 1229 2 0 1 0 0 0 5 501
403 77 1 0 99

---------------
# top
Memory: 62080K (11708K) real, 211796K (55696K) virtual, 4892K free Page# 1/11

---------------
# ipcs -mob
IPC status from /dev/kmem as of Fri Sep 1 15:51:22 2000
T ID KEY MODE OWNER GROUP NATTCH SEGSZ
Shared Memory:
m 0 0x411c0232 --rw-rw-rw- root root 0 348
m 1 0x4e0c0002 --rw-rw-rw- root root 1 31040
m 2 0x41200c15 --rw-rw-rw- root root 1 8192
m 104451 0xc468aa0c --rw-r----- oracle dba 33 713961472
m 6148 0xfc9e5780 --rw-r----- oracle dba 10 173940736

---------------
# swapinfo -t
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1048576 785912 262664 75% 0 - 1 /dev/vg00/lvol2
reserve - 262664 -262664
memory 748668 300276 448392 40%
total 1797244 1348852 448392 75% - 0 -

---------------
# echo "bufpages/D" |adb /stand/vmunix /dev/mem
Error from elf64_getehdr(application core file)
Not an Elf file: No Elf header
bufpages:
bufpages: 13106

---------------

[Stefan Farrelly]

Well, your right, you dont have much free memory at all. From your ipcs -sa command you can see how much shared memory is allocated for the existing Oracle Db already, over 700MB. And your swap is reaching 100% as you only have 1 Gb of RAM and Swap, but you can allow the use of more virtual memory (over the 1Gb physical RAM) if you increase your swapsize to say 2xphysical RAM.

So, if you want to add another Oracle DB the same size as the current one you will need another 700 MB of physical RAM and around another 2 Gb of swapspace. Then your server should perform fine without swapping.

[Anthony deRito]

Don't really see your concern. It has been suggested already that you add more swap. If you think you have a choice of how the HP-UX Memory Management System will deal with the memory regeons of your processes, this would be a wrong assumption. As mentioned before, HP-UX is a virtual operating system. It will do its best bring things into and out of real memory from VM. You have no control over this. The only way to hold things down in real memory is to use the plock() system call. Otherwise, a lot of page-ins will have to occur to bring in the memory regeons of a process only when they are needed.

It is very important to understand that this is very dirrent from page-out activities. Page-outs are your first indication of memory pressure.

The physical memory can be evaluated by looking at the RES values of your processes using top.

Tony

[John Palmer]

Recent versions of Oracle have an initialisation parameter 'lock_sga' which when set to true cause the SGA to be locked in memory.

This requires the MLOCK privilege for the dba group (see man 1m setprivgrp)

[Anthony deRito]

Thanks John for that info! I am going to inform our DBA of this as well. I was wondering how Oracle processes were able to have such a large resident set size as compared to other processes.

Tony

[Geetam]

Dear friends

Thank you for your contributions.

Can you please help me understand how you come to your conclusions. You understand these things, I want to learn to understand them. Quick answers are handy (and very much appreciated in emergencies), but they do not seem to convince me until I understand it myself.

[Geetam]

Stefan

I understand that 'ipcs' shows us that I have two instances of oracle, one using 700Mb, the other 170Mb.

Where do you see that swap is reaching 100%?

Why would I need to add swap if I add more physical RAM? (I trying to make sure I am not going to use swap space)

[John Palmer]

Your posted 'swapinfo -t' indicated that total swap usage was 75%:-

Your current swap space is made up of:-

device swap (/dev/vg00/lvol2) 1Gb
memory (75% of memory) .75Gb

Total 1.75Gb of which about 1.3Gb was either used or reserved.

You don't need to add more swap space just because you add RAM. In fact you would need less becacause as you've got 'swapmem_on' set, adding more RAM will increase the available 'memory' swap space by approx 75% of that added. So adding another 1Gb for instance would increase your 'memory' swap space from around 750Mb to 1500Mb.

You only need more swap space when you start to use the additional RAM for extra process data. For example, another Oracle SGA of 500Mb or increasing one of your existing SGA's would mean that the system would have to reserve (not necessarily use) that additional amount of swap space.

Remember that it is only data segments (private and shared) that require swap space. Text (code) segments are paged in from the file from which the program was loaded.

Hope this helps,
John

[Stefan Farrelly]

I know this is all a bit confusing. Even I get confused about it sometimes!

Anyway, HP recommed you have enough swapspace for double your RAM, minimum should be 1x RAM. The reason why ? HP-UX needs to reserve swapspace for any programs loaded into memory (not actually use the swapspace, just reserve it in case it is needed later if you run out of RAM). So, do you want it to reserve this swapspace on disk or in memory ?? much better to use the diskspace available as we need our precious RAM for other things - not reserving for future use for swapping - if swapping ever happens (you would hope not for performance reasons).

The reserve column from swapinfo shows you how much RAM is used by running processes that would need swapspace (paging space) if memory got really tight, in your case its 262MB worth.

The memory line of swapinfo indicates you have psuedo-swap enabled, and it shows current usage at 40%, or 300MB. This is not swapped out processes/memory. The only time memory will be used for paging space is if the device swap is fullup - in your case its 75% full, so memory is not yet being used for paging (and you wouldnt want it to - only in an emergency - as it will hit system performance hard gobbling up more of our precious RAM for paging space when we could be using disk instead!)
So, what on earth is this 40%, 300MB figure showing ? Its showing processes locked into memory - the main culprit it buffercache, your buffercache is set to 13106 pages, or 52 Megabytes (x4k), the rest by other daemons, probably database ones, which are locked into memory to maintain good performance of the database!

Anway, you have 1 GB of device swap, 1GB of Ram, pseudo-swap on (which at max is 75% of RAM) which gives you a total of 1.75Gb of available virtual memory. Try to use more than this and processes will fail to start.

In your case, of your 1 GB of RAM, the databases are using 770MB, the OS and other 'locked in memory' processes are using 300MB, other processes running which arent locked and can be paged out is 262MB worth, which gives a combined total of around 1.3GB of memory needed, and as it is you are using 785Mb of device swap already meaning you are actually trying using around 1.785MB of RAM, (I know we have a difference of around 400Mb here but its difficult to count precisely using the data so far available - but were close!) which is right on the limit without more device swap setup. This is shown by your free memory, only a few MB. The performance on your server now must be pretty poor ?

If you want to add another oracle instance of say 500MB then I would suggest for maximum performance you add another 1.5GB of RAM and another 2GB of device swap. Then you will see swapinfo -mt results where the %USED column for device swap stays at zero which means you have plenty of RAM free and nothing has been paged out = best performance.

[Geetam]

Thanks, John for explaining the output of swapinfo. I think I understand swapping/paging better now.

Stefan, my understanding is that speudo swap is never actually used (that would be the snake trying to eat its own tail). And, no, this system has good performance (and I want to keep it that way, or else...). It may have reserved lots of swap, and even paged some pages to disk (is that "paging in"?), but there is not other demand on memory, so these pages are still accessed from physical memory (it is not "paging out"?).

Can we go back to the original question: how much physical memory do I have free?

Can I do it this way?
Add up SEGSZ from ipcs -mob (847MB), then add 52MB for buffers used. Total 898MB used. Should I add memory used by kernel, etc? Anything else? Or can I assume that I have 1GB - 898MB = approx 100MB free? (if I want to live dangerously)

That is nothing like what the memory line in 'top' was telling me, but I think I may be on the right track here. What do you think?

Please explain your answers, so I can learn from this discussion. Thanks.

[Anthony deRito]

Geetham, you wrote:

"my understanding is that speudo swap is never actually used pseudo-swap is used as a last resort."

If virtual memory is used up, as would be seen by high (100%) values in the "PCT USED" column of swapinfo -tma, the memory management subsystem will use this portion of system memory. If you have sufficient amount of swap, there is no need to use pseudo-swap. Chances are, it will never be used.

"It may have reserved lots of swap, and even paged some pages to disk(is that "paging in"?),"

The memory management subsystem reserves the swap from the total available pool for all processes created. This memory is on disk. It is to be used by the virtual memory subsystem to dynamically page-out the total process. Chances are, it will not need to but it "reserves" the right to. As I mentioned before, pageing-out indicates memory pressure, paging-in is completely normal.

"Can we go back to the original question: how much physical memory do I have free? "

The available memory will change often because the memory mamangement subsystem is contantly trying to maintain a large free list. You need to look at the entire pool of memory... including the virtual memory and that is why there has been so much discussion on swap.

Tony

[Anthony deRito]

The first comment above should have read:

"my understanding is that speudo swap is never actually used"


Pseudo-swap is used as a last resort. If virtual memory is used up, as would be seen by high (100%) values in the "PCT USED" column of swapinfo -tma, the memory management subsystem will use this portion of system memory. If you have sufficient amount of swap, there is no need to use pseudo-swap. Chances are, it will never be used.

[Geetam]

Tony

Thanks for your explanations/corrections.

Can I ask a stupid question, to help me understand: How is it possible to actually use pseudo swap, because at that point there is no memory left to use for pseudo swap?

[Geetam]

Tony

Thanks for your explanations/corrections.

Can I ask a stupid question, to help me understand: How is it possible to actually use pseudo swap, because at that point there is no memory left to use for pseudo swap?

[Geetam]

I am (reluctantly) coming to the conclusion that is is not possible to find an answer to my original question. Maybe the question should be: How do I work out how much memory is used? (leaving out the word physical) and how much is locked and/or accessed so often that it is not paged to disk. Isn't there a way to predict how much more memory I can use before paging will affect performance? It is a production server, so 'try it' is not an option.

[Stefan Farrelly · Accepted solution]

You only start using pseudo-swap when your device swap if full, ie. swapinfo -mt shows type dev as 100% used. Yours is at 75% so your not far off it. this is pretty clear from the manpage on swapinfo. It has nothing to do with memory being at 100% usage or not, if fact memory usage must be >100% as tons of it must be swapped to disk - which causes swapinfo to show dev swap as 75% full in your case (ie. hundreds of MB swapped out). The system will always keep some memory free, so swapping/paging to disk is what it does when memory pressure results in more RAM being requsted than is available.

swapinfo -mt shows how much memory is locked (cant be swapped/paged out) - the memory column usage shows this figure - in your case above around 300MB.

Your server is already paging/swapping - run vmstat for a while and you will see the pi/po column showing paging going on. Youve got dev usage from swapinfo at 75%, which means hundreds of MB has been paged out. If your system wasnt paging then dev usage from swapinfo would be 0% - as I like to keep it on our production servers here.

I wouldnt like to add any more apps at all to your system without adding more physical RAM or reducing the memory requirements of your current apps.

swapinfo shows reserve at 260Mb. Thats how much more memory youve really got to work with before pseudo-swap will be used. Once this 260MB is paged out dev usage from swapinfo will be 100% and pseudo-swap will activate, and your server will crawl.

[Anthony deRito]

Hi Geetham, sorry for the late reply. Yes, as Stefan said, your only device swap you have available is currently being used by the memory management subsystem for pageout activities. This is VM that the system has set aside for your running processes. This is why HP-UX is called a virtual memory operating system. The concept of reserving memory in this way "ahead of time" allows your processes to keep running. And they are... thanks to VM.

The side affect is that at some point in time, and depending on the priority of the running processes on the run queue, processes will start to "deactivate". This means that parts of the processes will be paged-out to VM so that other process memory regeons can be paged-in so other processes gets their chance to run for its 1/10th of a second share. (Remember, page-ins are normal, page-outs not so good.)

There comes a point in time when the kernel is spending too much time trying to manage VM and physical memory and performing too many deactivations and page-outs that your system will start to "thrash". As you know this is not good.

It was my opinion from the start that you should add more swap. Disk is cheap. Add another 6 GB if you have it. Also make sure you have enough RAM. I would make 2GB the minimum. Psuedo-swap is your choice... I don't use it. You said this is a production server so cost should be a non-issue.

Then you need to keep a close eye on your vmstat values. Everything is there to let you know if you are experiencing memory pressure such as page-in and page-out values (pi, po), how many process are in the run queue (r), how many of those processes are blocked for resources (b), size of your free list (free), context switch rate (cs).

Tony