Who is using swap space?

[Alexey Borchev]

I've got Tru64 5.1A pk3, Oracle, swpamode=lazy.

The system configured so that Oracle fits in memory, and system does not use swap space at all.

From time to time (~1 per 3 months) someone consumes all the swap so Kernel finally kills
Oracle's pmon :-(((
so I've got to reboot the system.

Now I caught up the situation:
avrumo06> /sbin/swapon -s
Swap partition /dev/disk/dsk1h:
Allocated space: 1855713 pages (14.16GB)
In-use space: 522018 pages ( 28%)
Free space: 1333695 pages ( 71%)

Total swap allocation:
Allocated space: 1855713 pages (14.16GB)
In-use space: 522018 pages ( 28%)
Available space: 1333695 pages ( 71%)

vmstat shows 261K pages (Over 1 GB) free. (I can attach output if needed)

See ps aux output attached.

How can I identify which process used swap?
If i'll kill the process, will the swap space be released?

[Michael Schulte zur Sur]

Hi,

root 0 0.7 3.4 20.3G 554M ?? R < Mar 17 2-12:02:07 [kernel idle]

this seems to be a lot!
Can you post
sysconfig -q proc
sysconfig -q vm
sysconfig -q ipc
?

greetings,

Michael

[Ralf Puchner]

swapping is a normal system behaviour, there is no way to block oder prevent swapping! If the system is out of free pages it will swap. The ps output will also give the virtual memory information. There are also processes which always allocated swap space.

Be sure Oracle supports lazy mode, "supports" means here the company Oracle have tested and qualified and recommended this mode. It is not a technical aspect it is an aspect of "support".

If the system is out of virtual memory there is a process allocating and use the memory or there is a memory leak leading to such behaviour. Monitor your system to get the root cause of the problem.

[Alexey Borchev]

ipc:
msg_max = 8192
msg_mnb = 16384
msg_mni = 64
msg_tql = 40
shm_max = 16106127360
shm_min = 1
shm_mni = 128
shm_seg = 32
sem_mni = 16
sem_msl = 25
sem_opm = 10
sem_ume = 10
sem_vmx = 32767
sem_aem = 16384
sem_broadcast_wakeup = 1
max_kernel_ports = 93728
ssm_threshold = 8388608
ssm_enable_core_dump = 1
shm_allocate_striped = 1
shm_enable_core_dump = 1

proc:
max_proc_per_user = 320
max_threads_per_user = 1024
per_proc_stack_size = 8388608
max_per_proc_stack_size = 33554432
per_proc_data_size = 17179869184
max_per_proc_data_size = 17179869184
max_per_proc_address_space = 17179869184
per_proc_address_space = 17179869184
executable_stack = 0
autonice = 0
autonice_time = 600
autonice_penalty = 4
open_max_soft = 4096
open_max_hard = 4096
ncallout_alloc_size = 8192
round_robin_switch_rate = 60
sched_min_idle = 0
give_boost = 1
maxusers = 1024
num_wait_queues = 1024
num_timeout_hash_queues = 1024
enhanced_core_name = 0
enhanced_core_max_versions = 16
exec_disable_arg_limit = 0
dump_cores = 1
dump_setugid_cores = 0
executable_data = 0

vm:
ubc_minpercent = 5
ubc_maxpercent = 50
ubc_borrowpercent = 5
vm_max_wrpgio_kluster = 32768
vm_max_rdpgio_kluster = 16384
vm_cowfaults = 4
vm_segmentation = 1
vm_ubcpagesteal = 24
vm_ubcfilemaxdirtypages = 4294967295
vm_ubcdirtypercent = 40
ubc_maxdirtywrites = 5
ubc_maxdirtymetadata_pcnt = 70
ubc_kluster_cnt = 32
vm_ubcseqstartpercent = 50
vm_ubcseqpercent = 10
vm_csubmapsize = 1048576
vm_ubcbuffers = 256
vm_syncswapbuffers = 128
vm_asyncswapbuffers = 4
vm_clustermap = 1048576
vm_clustersize = 65536
vm_syswiredpercent = 80
vm_troll_percent = 4
vm_inswappedmin = 1
vm_page_free_target = 1024
vm_page_free_swap = 522
vm_page_free_hardswap = 16384
vm_page_free_min = 20
vm_page_free_reserved = 10
vm_page_free_optimal = 522
vm_swap_eager = 0
swapdevice = /dev/disk/dsk1h
vm_page_prewrite_target = 2048
vm_ffl = 1
ubc_ffl = 1
vm_rss_maxpercent = 100
anon_rss_enforce = 0
vm_rss_block_target = 522
vm_rss_wakeup_target = 522
kernel_stack_pages = 0
vm_min_kernel_address = 18446741891866165248
malloc_percpu_cache = 1
vm_aggressive_swap = 0
new_wire_method = 0
vm_segment_cache_max = 50
gh_chunks = 0
rad_gh_regions[0] = 0
rad_gh_regions[1] = 0
rad_gh_regions[2] = 0
rad_gh_regions[3] = 0
rad_gh_regions[4] = 0
rad_gh_regions[5] = 0
rad_gh_regions[6] = 0
rad_gh_regions[7] = 0
rad_gh_regions[8] = 0
rad_gh_regions[9] = 0
rad_gh_regions[10] = 0
rad_gh_regions[11] = 0
rad_gh_regions[12] = 0
rad_gh_regions[13] = 0
rad_gh_regions[14] = 0
rad_gh_regions[15] = 0
rad_gh_regions[16] = 0
rad_gh_regions[17] = 0
rad_gh_regions[18] = 0
rad_gh_regions[19] = 0
rad_gh_regions[20] = 0
rad_gh_regions[21] = 0
rad_gh_regions[22] = 0
rad_gh_regions[23] = 0
rad_gh_regions[24] = 0
rad_gh_regions[25] = 0
rad_gh_regions[26] = 0
rad_gh_regions[27] = 0
rad_gh_regions[28] = 0
rad_gh_regions[29] = 0
rad_gh_regions[30] = 0
rad_gh_regions[31] = 0
rad_gh_regions[32] = 0
rad_gh_regions[33] = 0
rad_gh_regions[34] = 0
rad_gh_regions[35] = 0
rad_gh_regions[36] = 0
rad_gh_regions[37] = 0
rad_gh_regions[38] = 0
rad_gh_regions[39] = 0
rad_gh_regions[40] = 0
rad_gh_regions[41] = 0
rad_gh_regions[42] = 0
rad_gh_regions[43] = 0
rad_gh_regions[44] = 0
rad_gh_regions[45] = 0
rad_gh_regions[46] = 0
rad_gh_regions[47] = 0
rad_gh_regions[48] = 0
rad_gh_regions[49] = 0
rad_gh_regions[50] = 0
rad_gh_regions[51] = 0
rad_gh_regions[52] = 0
rad_gh_regions[53] = 0
rad_gh_regions[54] = 0
rad_gh_regions[55] = 0
rad_gh_regions[56] = 0
rad_gh_regions[57] = 0
rad_gh_regions[58] = 0
rad_gh_regions[59] = 0
rad_gh_regions[60] = 0
rad_gh_regions[61] = 0
rad_gh_regions[62] = 0
rad_gh_regions[63] = 0
gh_min_seg_size = 4194304
gh_fail_if_no_mem = 1
private_cache_percent = 0
gh_keep_sorted = 0
gh_front_alloc = 1
replicate_user_text = 1
enable_yellow_zone = 0
boost_pager_priority = 0
kstack_free_target = 5
wire_audit_count = 0
vm_prepop_percent = 100

[Alexey Borchev]

Yes, I've got 16 GB Ram, 12 is given to Oracle, it seems Oracle is happy with it so far.

It's strange that Kernel idle maps not 16, but 20.5 GB.
Is it 16 GB Ram + Swap in-use space?

Any ideas about identifying resource user?
Otherwice i'll have to reboot my system - opportunity will be lost.

P.S.
It decreased slightly:
In-use space: 521399 pages ( 28%)

Maybe, it worth try to kill swapped out processes (which has W in ps aux)?

[Michael Schulte zur Sur]

Hi,

ubc_maxpercent = 50
this seems too much to me. I would reduce it to 20. You can do this online with syscongfig.

hth,

Michael

[Ralf Puchner]

Michael,

come on, this parameter is not within my coffee-cup.... ;-)

Alexey,

are you sure you configured the machine as described within the Oracle tuning guide? The values must be calculated depending your needs and reflecting your database setup. So posting the sysconfigtab informations doesn't really help to analyze the problem without knowing your complete setup and dependencies.

Use "collect" to monitor the system and to get a clue what is going on. But as described in my previous posting swapping is normal. With the tool you will get the time and memory usage also the dependencies (I/O etc.)

[Joris Denayer]

If this occurs after a couple of months, you might have a memory leak somewhere.
You can use the command
# vmstat -M
and monitor if a particular bucketsize and memory type is growing constantly.

Joris

[Alexey Borchev]

1) Michael, i'll try to
ubc_maxpercent = 50 -> 20
but I'll wait for less busy period.
As far as i understand, Trunix will not swap out filesystem cache - it will shrink it instead.

2) I've run vmstat -M see attachment.
Total memory being used from buckets = 99444208 bytes
Total free memory in buckets = 26347376 bytes
- i.e. here only 100 MB, I am looking for 4.5 GB.

3) I'll read & try collect, thanks.

4) Currently I've got 187K free mem pages,
and 520670 pages used in swap space.
-Why Unix does not read it back into memory?
- Swap usage is slightly decreased. May be some swapped out processes was terminated?

[Joris Denayer]

Hi Alexej,
A leak is not yet visible, short after a reboot.
Try this command again each week.

If there is a leak, you will see if one of the buckets/memory types is increasing.

Joris

[Alexey Borchev]

Swap space usage decreased:

Currently
In-use space 11551 pages ( 0%)

Oracle instance was re-started during weekend. (Cold backup)

Imho, some Oracle processes where swapped-out, then loaded back into memory, but swap space was still reserved for the process.
Swap space was released when process terminated.

[Joerg Schulenburg]

You can switch off swap editing /etc/sysconfigtab removing swapdevice entry.
You can add swap space on runtime if you need it again.
I thought that you can identify swapping process only indirect by looking for large
vsz-rss (ps aux), high syscall and idle activity.
I have problems using more than 16GB on
a 128GB GS1280 with 32 CPUs without swapping,
where the vm system
does strange things (see http://www.uni-magdeburg.de/urzs/marvel/vmbug3.html)

[Joerg Schulenburg]

>>As far as i understand, Trunix will not swap out filesystem cache - it will shrink it instead.

My experiences are otherwise. Tru64 takes swap instead of shrinking UBC.

>>Why Unix does not read it back into memory?

Unix does, Tru64 does not :(
I think the philosophy behint is, that the
pages read from the swap are cached and if
memory again is low only cached pages has to
be released.

>>swap usage is slightly decreased. May be some swapped out processes was terminated?

Probably. and because of reduced ubc_maxpercent new
processes are not swapped out again.

Did you mention your hardware?
NUMA?
What tells vmstat -R?

[Alexey Borchev]

GS1280, 8 CPU, 32 GB mem.

vmstat -R see in attachment.

[Alexey Borchev]

>>>>As far as i understand, Trunix will not swap out filesystem cache - it will shrink it instead.
>>My experiences are otherwise. Tru64 takes swap instead of shrinking UBC.
- How did You observed this? The behaviour seems absolutely rediculose, unless ubc_min_percent hit.

>>>>Why Unix does not read it back into memory?

>>Unix does, Tru64 does not :(
I think the philosophy behint is, that the
pages read from the swap are cached and if
memory again is low only cached pages has to
be released.
- Yes, i see the point. So, next time instead of swapping out, kernel will just discard the pages (if they are not "dirty").

[Joerg Schulenburg]

>>How did You observed this? The behaviour seems absolutely rediculose, unless ubc_min_percent hit.

I did vmstat -R 1 (or 10 instead of 1) and vmubc -tty -swap -memory (please look at the webpage mentioned in my GS1280 swap thread threadId=859125 for details). Read a big file to fill the UBC,
started a memory consuming testprogram and
swapping started before ubc_minpercent and borrowpercent was hit. I am not quite sure if
actu was not lowered or not lowered enough to
avoid swapping. I will have a look to my logfiles or repeat the experiment later, because I dont have time now and put the original data to my mentioned webpage.

Your attachement:
a snapshot is not enough, do a vmstat -R 60 (or so). vmubc would be good too.
Have a look to the free pages of each RAD. Looks like RAD0 and RAD1 where
temporary out of memory (high acti and pout).
If so, you triggered a kernel bug or
however HP call that misbehaviours (see my thread). But its not for sure, not enough data from you (and HPs engineers) to find out, what exactly goes wrong.