accessing device swap

[Randy Tarrier]

Hello all,
I'm running a k460 w/ 3gb RAM. Just upgraded Oracle from 8.1.7.0 to 8.1.7.4 last weekend, now Oracle listener cannot allocate swap space for new processes. I usually run around 330 users, today it was around 276.
Here is sample output from the listener:
TNS-00510 Internal limit restriction exceeded

Cause: Too many files or sockets open simultaneously (or some other resource has been depleted).

Action: For further details, trace the operation for protocol details.
Pid 28053 received a SIGSEGV for stack growth failure.
Possible causes: insufficient memory or swap space,
or stack size exceeded maxssiz.
Memory fault(coredump)

I've noted other forums advising increasing maxssize & maxssize_64 bit, but mine is already huge.

# sysdef|grep max
dbc_max_pct 2 - - -
maxdsiz 262144 - 0-655360 Pages -
maxdsiz_64bit 491520 - 256-1048576 Pages -
maxfiles 2048 - 30-2048 -
maxfiles_lim 2048 - 30-2048 -
maxssiz 51200 - 0-655360 Pages -
maxssiz_64bit 51200 - 256-1048576 Pages -
maxswapchunks 2048 - 1-16384 -
maxtsiz 262144 - 0-655360 Pages -
maxtsiz_64bit 262144 - 256-1048576 Pages -
maxuprc 1971 - 3- -
maxvgs 20 - - -
netmemmax 0 - - -
# sysdef|grep sh
public_shlibs 1 - - -
shmem 0 - 0-1 -
shmmni 200 - 3-1024 -

Glance errs out w/following:
mi_ki_init - only able to allocate 2 bufsets
Not enough space.
midaemon: Tue Apr 24 15:02:51 2001
mi_ki_init - only able to allocate 1 bufsets
Not enough space.
midaemon: Tue Dec 11 10:57:00 2001
mi_create - Shared Memory Database initialization failed; vss = 14328260 bytes
Not enough space.

It seems Oracle SGA (750mb) & users are using about 800mb of memory at any time

# ipcs -mopb
IPC status from /dev/kmem as of Tue Jan 21 14:02:04 2003
T ID KEY MODE OWNER GROUP NATTCH SEGSZ CPID LPID
Shared Memory:
m 0 0x411c27a9 --rw-rw-rw- root root 0 348 1347 1347
m 1 0x4e0c0002 --rw-rw-rw- root root 1 31040 1347 1347
m 2 0x41200649 --rw-rw-rw- root root 1 8192 1347 1359
m 803 0x33180e2d --rw------- root root 2 1129408 2406 2412
m 4 0x6d1806a6 --rw-rw-rw- root root 2 47120 2406 2412
m 19405 0x6014f024 --rw-rw---- oracle oinstall 272 788545536 2284
4320
m 606 0x0c6629c9 --rw-r----- root sys 0 14328260 4975 4975
m 9 0xffffffff --rw-r--rw- root sys 0 22908 17360 17360
# swapinfo -mt
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1664 461 1203 28% 0 - 1 /dev/vg00/lvol2
reserve - 1203 -1203
memory 2182 2178 4 100%
total 3846 3842 4 100% - 0 -


My question:
Why don't these processes access the device swap not yet used?
Device swap % has increased from 25% to 28% over a three hour period,
while oracle has refused 3800 requests for connections.
Can you advise me?

[Sridhar Bhaskarla · Accepted solution]

Hi,

Your case seems to be simple. You are basically running out of virtual memory. (Total column in your swapinfo -t). Add more device swap and that will take care of your issue. It can't take swap out of device swap because the total reservation allowed is exceeded. This is one of the classic examples of using swapmem_on parameter for the servers that are low on memory.

-Sri

[Randy Tarrier]

Sridhar,
If I understand you correctly, the 1203 reserve is the difference between RAM and psuedoswap? If I increase device swap (or in my case, add fs-swap) it allows the reserve to grow, as new processes come in?
Thanks,
Randy

[Sridhar Bhaskarla]

Hi Randy,

1203 is not the difference between RAM and Pseudoswap. Initially you got 2182 MB out of RAM as the pseudoswap but it is no more available as you are completely using the memory. So, all the reservation has to come from only the deviceswap but you are already using 461MB out of it for "swapping". So the total swap available for you for reservation calculation is only 1664-461=1203. And you are hitting that limit already. So, in order for your system to spawn more processes, you will need to add more device swap. You can do it online and I believe you do not need commands to add secondary swap.

-Sri

[Bill Hassell]

Adding more swap is easy: swapon
If you simply locate an lvol with a few Gb that's not being used, just use swapon to add it on the fly and the problem will disappear. You can 'borrow' swap space from a filesystem if you like although it is about 30% slower and once used, it is never given back until a reboot occurs.

Too make the extra swap space permanent, add the entry to /etc/fstab. But if you are having performance issues, I would strongly recommend doubling your RAM so you don't have to swap at all. Swap (paging) can create 50:1 to 100:1 performance penalties on busy systems.

[Randy Tarrier]

Thanks guys,
I've added another 1.6 gb of swap to another lun, we'll see how that works tomorrow!