memory performance question 11.0 64 bit

[Penni Nussbaum]

I have an 11.0 system (64 bit) with memory (according to Glance) as follows..
phys mem 2 gb
user mem 1.36 gb
free mem 332.4 mb
sys mem 115.8
buf cache 204.8
total vm 765.5
active vm 723.3

This is an oracle database server. It keeps on getting memory shortage problems. There is 1 gig of swap. I don't have any more disks to add more swap. Pseudo swap is turned on.

bufpages = 0
db max =10
db min = 2
nbuf =0

Besides adding more memory, does anyone have tuning ideas? Also, the developer is sure that Oracle is using only 1 gig of memory. He wants to know why the OS requires 1 gig (He assumes this since we are getting memory errors when no one can telnet into the box).

[Thierry Poels_1]

hi,
If the server is mainly an Oracle database server then you should minimize the file cache:
set dbc_min & dbc_max = 2
just my 2 cents,
Thierry.

[A. Clay Stephenson]

First thing let's see how much shared memory Oracle is eating. ipcs -m -a. Also get some page/in and especially page/out rates via glance.

[A. Clay Stephenson]

Next question, you appear to have 3 oracle instances running. Is this correct?

[Penni Nussbaum]

yep, 3 instances.

[A. Clay Stephenson · Accepted solution]

Are you using raw disk for database I/O (or vxfs mount options mincache=direct, convosync=direct (both bypass the UNIX buffers)?
If so really reduce the buffer cache sizes so that Oracle is not buffering in both the SGA's and the Unix Block I/O Buffers. Thierry's suggestion is a good one. I would prefer to set bufpages to about 15360 (60MB) if this is a pure database server.

I would also ask the DBA to reduce his buffers
a bit (especially on any of the non-production instances) to see if this helps.

If you are using cooked files for datafiles/indexes and you have OnLineJFS I strongly suggest that you use the mount options mentioned above to avoid the double buffering. No database changes will be required.

Hopr some of this helps...

[A. Clay Stephenson]

One other thing Penni:
Are the glance snapshots you're sending taken when thinks are pretty good or when the box is a dog? Things don't look so terrible so far.
When things are bad it would also be nice to have some idea of run queue sizes as well.

[Thierry Poels_1]

Hi,
I agree with Clay; the system statistisch you showed us, don't look like an overloaded system.
And (also mentioned by Clay, I'm just repeating ;) check if you can shrink some SGA-sizes. They seem to consume a considerable amount of memory; but size depends on your setup, size of database, users, etc. so I'm not going to state that they really are to big.

One more thing I noticed is the amount of swapspace used: 95%. You might need to increase your swapspace.
Thierry.

[Carlos Fernandez Riera]

The amount of recomended swap is two times phisical memory and you have configured a half.

To add swap you need disk space but in 11.0 swap could be in any vg.

By the other hand, you there are running 3 oracle instances with shared memory sizes (SEGMENT SIZE) of about 200, 254 and 500 Mb each, this is 1 Gb, plus 10% of buffer cache get 1,2 Gb of memory used with out users.

If realy you can not increase swap ( that is what you need), try to reduce shared memory usage.

Run swapinfo... you are near to the message "not enough memory" ....

[Penni Nussbaum]

The statistics I showed are not when the system was low on memory. When the system is low on memory, no one can log on.

I realize I need to increase sap size, but there is no more disk space anywhere on this system. No free disks and the no gree space on any volume groups.

One of thehigh level users is questioning why the system needs almost 1 gig of memory (Oracle needing the other). I need to defend that 1 gig.

[Penni Nussbaum]

Regarding mincache=direct and convosync, it is my understanding these options need Online JFS, which we do not have here.

[Penni Nussbaum]

One more question. Although I only have 1 gig of swap with 2 gig of memory, I do have the kernel parameter swapmem_on set to 1. I thought turning this on helps alleviate the swap shortage.

[Carlos Fernandez Riera]

Perhaps this help:

When you start a new process ( like telnet) the kernel reserve space in swap to use if is needed. If you have not enough swap the system cant reserve swap then and then message issued is not " more swap needed" the message is not enought memory.

This is not a problem of memory, the problem is swap.

If you can not add disk, then reduce memory usage. That oracle instance of 505 Mb of shared memory maybe could be reduced ( raising db_block_buffers, or shared_pool_size.

Choosing between performance and accessibility... i dont need performance if i cant access!.

[Servio Cabrera]

I have seen, when you apply the kernel template for Monolitic DB server, it put swapmen=0, and I heard some experts dont recomend swapmem=1, for a DB server. But if you do that, you have to increase the swapspace even more to compensate. If you are mirroring the swap lv, try to break the mirror and use the freed space for another swap lv, so you will double. And again, reduce the db_max_pct to 2%

[Stefan Farrelly]

Youve simply got to have swap size set to the same as memory, at a minimum. You really need 2 GB of swap. Using pseudo-swap is not an alternative as this will cause memory contention and performance degredation, especially as you only have 300 odd MB free which means your using 700MB of pseudo swap instead of swapspace. Pseudo-sawp is a backup (for emergencies when your really out of memory) not as an alternative to swapspace.

Increase swapspace to 2GB and you will be surprised how much better your system performs.

[Penni Nussbaum]

Thanks. I cannot change kernel parameter because of the reboot, but I did unmirror my swap which doubled the size. I'll see what happens today.

[Carlos Fernandez Riera]

Shit man...

If you need a kings reponse you should say it before...

There was 3 hated saying what was the problem.

[Thierry Poels_1]

LOL Carlos. Yep Penni seems to be very selective in giving points ;)
Personally I would prefer some feedback on how things are going after the swap increase (you did really add swap space, not just decreased the mirror) than getting points without knowing what did the trick.
regards,
Thierry.

[A. Clay Stephenson]

Penni: You cheated when you told us that you had no more disks for swap and then ... oops I unmirrored swap. I took you at your word and was trying everything I could to avoid using filesystem swap which was the only option left.

[Penni Nussbaum]

I appreciate the help you have given me. Bcause the memory "really swap" problems where intermitent, I wanted to monitor the system an entire day to make sure all was O.K.
I never thought of unmirroring swap, to double the size, simple as the answer seems. I have always had 1 root disk mirrored exactly on a second disk.
My only concern now is if I loose a root disk (1 is mirrored to the other), I will again have swap problems until the disk is replaced.

[A. Clay Stephenson]

Yes that is a valid concern and I think that you have really only fixed the symptoms of your problem. You are probably swapping at least a portion of the SGA's now which will kill ORACLE performance. I would still try to reduce the SGA buffer's if you can't purchase additional memory. My approach to memory management is a little bit different. I purchase plenty of memory so that I don't have to swap and then only allocate 256-512MB of primary swap. To handle your kind of emergency situation I then enable filesystem swap at very low priority. (It's really never used but I don't then blow away tons of disk space for unused swap logical volumes with their mirrors.) You should really purchase OnLineJFS; it's worth the price and will make your life much easier.

[Stefan Farrelly]

Mr Stephenson,

Are you recommending a system with say 2GB of Ram (used for Oracle) but only setting up 256-512MB of swap ? (not including filesystem swap). If so I do not reccomend this to anyone. What happens if you server crashes with an HP-UX dump and you dont have 2GB of swap for the entire dump so cant process it (or get HP to) and find out the cause of your crash ? What are you going to say to your manager then ?

There are very good reasons why the default swap size in the HP-UX install program is set to 2x Physical RAM and the HP Sys Admin book says you should set it to a minimum of 2x RAM. OK, in the real world you could run as low as 1x RAM but certainly not any lower.

[Servio Cabrera]

Penni: Another thing that worth to have installed and configured is MWA (MeasureWare) agent and defined in the parm file, the oracle process and all the other application group, in order to have a nice application organization. You will have after a week a very nice data collected to analize with PerfView. Then you take a look of all the memory usage global and per application.

[Thierry Poels_1]

I have to agree with Stefan on this.
And I've got another thing to worry about: if you unmirrored primary swap you won't be able to boot anymore from the mirrored boot disk.
I've seen several sites where they prefer to take the risk above the price of additional hardware, until thing go wrong ....
regards,
Thierry.