Memory problems - swap at 100% - DOH!

[Timothy Czarnik]

Hey all,

RP7400, HP-UX 11.11, Oracle 8i. Oracle users complaining that they can't get connections. I seem to be having memory issues.

Output of swapinfo -atm:

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 2048 0 2048 0% 0 - 1 /dev/vg00/lvol2
reserve - 2045 -2045
total 2048 2045 3 100% - 0 -

vmstat info:

/tmp/tim # vmstat 5 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 0 0 140992 451423 9 4 0 0 0 0 0 1666 2414 572 3 1 95
1 0 0 140992 452135 36 0 0 0 0 0 0 1504 3758 564 6 1 93
1 0 0 150460 451121 25 0 0 0 0 0 0 1552 4634 599 6 2 92
1 0 0 150460 451121 7 0 0 0 0 0 0 1610 4164 604 4 1 95
1 0 0 155913 452136 20 0 0 0 0 0 0 1436 3337 457 3 1 96

It looks to me that I'm out of swap, but have memory available. Any ideas on where I can look next? Rebooting is a worst case scenario, so I'm hoping to not have to do that.

Any suggetions would be greatly appreciated!

Tim

[keith persons]

Well, the first suggestion would be to add a swap device if you have disk space available. You would need to add it to the /etc/fstab to continue using after reboot. This is probably the fastest short-term suggestion.

Longer term you might want to consider enabling psuedo-swap or memory swap - it is disabled on your system. The kernel parameter is swapmem_on - set to 1 to enable (you'll need to rebuild kernel and reboot).

Also, check maxswapchunks to insure you'll be able to address all swap configured.

[John Bolene]

The first thing to add is turning on pseudo-swap.

kernel parm
swapmem_on 1

yes a reboot is needed

after that you may need to add more swap

[James R. Ferguson]

Hi Tim:

I would turn on the kernel 'swapmem_on' to enable pseudoswap. You will need to reboot, of course.

You could also create additional device swap, but arming pseudoswap will certainly help.

Regards!

...JRF...

[A. Clay Stephenson]

If you can't reboot and don't have any free LVOL's available for device swap, you could add
some filesystem swap at low priority. The best answer is to of course add more memory along with an increase in swap or enabling pseudoswap.

[Martin Johnson]

Add more swap space. Make sure maxswapchunks is large enough to use all the space. Add an entry into /etc/fstab for the new space. Make sure the priority is the same as your other swap spaces.


HTH
Marty

[Timothy Czarnik]

Hey all,

Thanks for the quick replies!

Am I correct in thinking that the basic formula for swapsize is (physical memory *2)?

We have SAN disk via a VA7400 attached to this server. Any issues with putting additional swap space on this?

Tim

[Roger Baptiste]

<>

It depends on the memory configuration and system usage. For high memory systems , making the swap size twice the size of RAM is a waste of disk space.


<>

No problem in doing it. Just make sure the priority of the swap is the same and the sizes of the swap are the same.

In your current situation, you can create a temporary swap space (filesystem or raw) and use it, without rebooting the system.

HTH
raj

[Sajid_1]

hello,

That's is an OLD formula! When there was only a few amount of RAM available in each systems, this formulas was useful. But now almost all systems comes with a good amount of memory and the swap space should be calculated according to the usage only.

By observing the swap and memory usage, and if all Kernel parameters are tuned perfectly, then you can increase the swap space and you could easily calcaulate the amount of swap needed according to the usage.

For the other question, Yes! you can add the swap space on the disk array. Check the swap priority too

[MANOJ SRIVASTAVA]

Hi Timothy

You can run a simple program which give the o/p like this

Also there is small c program that you can run to get teh o/p in the following format if you want :.



Memory Stat total used avail %used
physical 32764.0 10640.1 22123.9 32%
active virtual 1784.9 245.5 1539.4 14%
active real 1963.9 267.1 1696.7 14%
memory swap 26522.5 5508.9 21013.6 21%
device swap 3332.0 3332.0 0.0 100%


I have attached the program all you ahve to do is to recomiple it and run it .


Manoj Srivastava

[Timothy Czarnik]

Hey all,

It is simply amazing the stuff you learn when fighting fires :)

There is no entry in my /etc/fstab for my current 2 GB swap (/dev/vg00/lvol2). Is this just assumed in the kernel somewhere?

Also, will turning swapmem_on in the kernel negate my need for additional swap. That is, should I turn swapmem_on in the kernel before I add more swap space, or do both at the same time.

It looks like I'll be rebooting over lunch anyhow.

Tim

[Sajid_1]

hello,

For getting details about the primary swap space:
# lvlnboot -v
# swapinfo -tam
# sam
When you add any additional file system swap, that will make an entry in the fstab file.

For the second question, I would first create the swap space (which you can do online) with SAM, check the outputs with swapinfo -t, and then do the kernel configuration. Once you reconfigure kernel, you need to restart the system!

[Bill Thorsteinson]

If Oracle is the main application on the system
you should have very little
swapping. Don't let the
shared memory segment start
swapping or performance will
tank big time.

It looks like you may not have the swap mounted if it isn't
already in fstab. If so mounting should help
immediately.

Oracle has had problems with memory leaks from time to time
but I think those are all patched. Last time I had a
server crash due lack of memory, it was a leak in a
monitoring program. Sys admin
said 2k physical memory free wasn't a problem.

[MANOJ SRIVASTAVA]

The default swap /dev/vg000/lvol2 will not be in /etc/fstab , what is the system memory , if you ahve lots of physical memeory then do the swapmem_on stuff if the real memopry is less then just add a secondary swap , this can be done online.



Manoj Srivastava

[Roger Baptiste · Accepted solution]

Hi,

Primary swap (i.e lvol2) won't be listed in fstab file. Pseudo-swap alone may not solve your swap problem . If youcan give the details of how much RAM your system has and what applications you are running, it would help in making a approximate estimate of usage.

If you have free disk space, it doesn't harm to throw in a extra swap volume. To create device swap do the following:

Identify a VG which has free space of 2Gb. (since it is the same as your primary swap size).

Then,
1. Create a contiguous logical volume:

lvcreate -n swap1 -L size -C y /dev/vgX

2. Turn swap on in the logical volume:

swapon /dev/vgX/swap1

IMPORTANT: The default priority for swap is "1". It is a recommended to have
all the swap devices set to the same priority.

3. Edit the /etc/fstab so that swap will be turned on at boot-up; add the
following line:


/dev/vgX/swap1 ... swap pri=1 0 0

***

Reboot is not necessary for the above three steps. If you want to enable pseudo swap, then you would need to run SAM, turn swapmem_on kernel bit to value 1 and rebuild the kernel, which would reboot the system .

HTH
raj

[Bill Hassell]

Since swap 100% unused, you don't have any problems with virtual memory. The po (page out) rate in vmstat is also very low. However, if you have changed the default setting of swapmem_on from 1 to 0, then you are indeed completely out of virtual memory.

Here's the simple explanation. When you set swapmem_on=0, *ALL* RAM for user processes must have a unique reservation spot on the swap disk. And if you have 2Gb of swap then you have 2Gb of usable RAM for processes. Since it appears that about 2Gb of RAM is in use, there is no space for any new processes because all the swap disk is currently mapped.

Now, change to swapmem_on=1 and now you will now have 3.5 Gb of virtual memory. This has been discussed several times before in the forums:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x6f537bb04b5cd611abdb0090277a778c,00.html

By the way, 2Gb is a bit too small for most Oracle systems. Your customers will be a lot happier with 4Gb and your DBAs can do more with SGA to improve performance.