Simpler Navigation coming for Servers and Operating Systems
Coming soon: a much simpler Servers and Operating Systems section of the Community. We will combine many of the older boards, and you won't have to click through so many levels to get at the information you need. If you are looking for an older board and do not find it, check the consolidated boards, as the posts are still there.
Operating System - Tru64 Unix
cancel
Showing results for 
Search instead for 
Did you mean: 

Who is using swap space?

Alexey Borchev
Regular Advisor

Who is using swap space?

I've got Tru64 5.1A pk3, Oracle, swpamode=lazy.

The system configured so that Oracle fits in memory, and system does not use swap space at all.

From time to time (~1 per 3 months) someone consumes all the swap so Kernel finally kills
Oracle's pmon :-(((
so I've got to reboot the system.

Now I caught up the situation:
avrumo06> /sbin/swapon -s
Swap partition /dev/disk/dsk1h:
Allocated space: 1855713 pages (14.16GB)
In-use space: 522018 pages ( 28%)
Free space: 1333695 pages ( 71%)

Total swap allocation:
Allocated space: 1855713 pages (14.16GB)
In-use space: 522018 pages ( 28%)
Available space: 1333695 pages ( 71%)

vmstat shows 261K pages (Over 1 GB) free. (I can attach output if needed)

See ps aux output attached.

How can I identify which process used swap?
If i'll kill the process, will the swap space be released?
The fire follows shedule...
15 REPLIES
Michael Schulte zur Sur
Honored Contributor

Re: Who is using swap space?

Hi,

root 0 0.7 3.4 20.3G 554M ?? R < Mar 17 2-12:02:07 [kernel idle]

this seems to be a lot!
Can you post
sysconfig -q proc
sysconfig -q vm
sysconfig -q ipc
?

greetings,

Michael
Ralf Puchner
Honored Contributor

Re: Who is using swap space?

swapping is a normal system behaviour, there is no way to block oder prevent swapping! If the system is out of free pages it will swap. The ps output will also give the virtual memory information. There are also processes which always allocated swap space.

Be sure Oracle supports lazy mode, "supports" means here the company Oracle have tested and qualified and recommended this mode. It is not a technical aspect it is an aspect of "support".

If the system is out of virtual memory there is a process allocating and use the memory or there is a memory leak leading to such behaviour. Monitor your system to get the root cause of the problem.


Help() { FirstReadManual(urgently); Go_to_it;; }
Alexey Borchev
Regular Advisor

Re: Who is using swap space?

ipc:
msg_max = 8192
msg_mnb = 16384
msg_mni = 64
msg_tql = 40
shm_max = 16106127360
shm_min = 1
shm_mni = 128
shm_seg = 32
sem_mni = 16
sem_msl = 25
sem_opm = 10
sem_ume = 10
sem_vmx = 32767
sem_aem = 16384
sem_broadcast_wakeup = 1
max_kernel_ports = 93728
ssm_threshold = 8388608
ssm_enable_core_dump = 1
shm_allocate_striped = 1
shm_enable_core_dump = 1

proc:
max_proc_per_user = 320
max_threads_per_user = 1024
per_proc_stack_size = 8388608
max_per_proc_stack_size = 33554432
per_proc_data_size = 17179869184
max_per_proc_data_size = 17179869184
max_per_proc_address_space = 17179869184
per_proc_address_space = 17179869184
executable_stack = 0
autonice = 0
autonice_time = 600
autonice_penalty = 4
open_max_soft = 4096
open_max_hard = 4096
ncallout_alloc_size = 8192
round_robin_switch_rate = 60
sched_min_idle = 0
give_boost = 1
maxusers = 1024
num_wait_queues = 1024
num_timeout_hash_queues = 1024
enhanced_core_name = 0
enhanced_core_max_versions = 16
exec_disable_arg_limit = 0
dump_cores = 1
dump_setugid_cores = 0
executable_data = 0

vm:
ubc_minpercent = 5
ubc_maxpercent = 50
ubc_borrowpercent = 5
vm_max_wrpgio_kluster = 32768
vm_max_rdpgio_kluster = 16384
vm_cowfaults = 4
vm_segmentation = 1
vm_ubcpagesteal = 24
vm_ubcfilemaxdirtypages = 4294967295
vm_ubcdirtypercent = 40
ubc_maxdirtywrites = 5
ubc_maxdirtymetadata_pcnt = 70
ubc_kluster_cnt = 32
vm_ubcseqstartpercent = 50
vm_ubcseqpercent = 10
vm_csubmapsize = 1048576
vm_ubcbuffers = 256
vm_syncswapbuffers = 128
vm_asyncswapbuffers = 4
vm_clustermap = 1048576
vm_clustersize = 65536
vm_syswiredpercent = 80
vm_troll_percent = 4
vm_inswappedmin = 1
vm_page_free_target = 1024
vm_page_free_swap = 522
vm_page_free_hardswap = 16384
vm_page_free_min = 20
vm_page_free_reserved = 10
vm_page_free_optimal = 522
vm_swap_eager = 0
swapdevice = /dev/disk/dsk1h
vm_page_prewrite_target = 2048
vm_ffl = 1
ubc_ffl = 1
vm_rss_maxpercent = 100
anon_rss_enforce = 0
vm_rss_block_target = 522
vm_rss_wakeup_target = 522
kernel_stack_pages = 0
vm_min_kernel_address = 18446741891866165248
malloc_percpu_cache = 1
vm_aggressive_swap = 0
new_wire_method = 0
vm_segment_cache_max = 50
gh_chunks = 0
rad_gh_regions[0] = 0
rad_gh_regions[1] = 0
rad_gh_regions[2] = 0
rad_gh_regions[3] = 0
rad_gh_regions[4] = 0
rad_gh_regions[5] = 0
rad_gh_regions[6] = 0
rad_gh_regions[7] = 0
rad_gh_regions[8] = 0
rad_gh_regions[9] = 0
rad_gh_regions[10] = 0
rad_gh_regions[11] = 0
rad_gh_regions[12] = 0
rad_gh_regions[13] = 0
rad_gh_regions[14] = 0
rad_gh_regions[15] = 0
rad_gh_regions[16] = 0
rad_gh_regions[17] = 0
rad_gh_regions[18] = 0
rad_gh_regions[19] = 0
rad_gh_regions[20] = 0
rad_gh_regions[21] = 0
rad_gh_regions[22] = 0
rad_gh_regions[23] = 0
rad_gh_regions[24] = 0
rad_gh_regions[25] = 0
rad_gh_regions[26] = 0
rad_gh_regions[27] = 0
rad_gh_regions[28] = 0
rad_gh_regions[29] = 0
rad_gh_regions[30] = 0
rad_gh_regions[31] = 0
rad_gh_regions[32] = 0
rad_gh_regions[33] = 0
rad_gh_regions[34] = 0
rad_gh_regions[35] = 0
rad_gh_regions[36] = 0
rad_gh_regions[37] = 0
rad_gh_regions[38] = 0
rad_gh_regions[39] = 0
rad_gh_regions[40] = 0
rad_gh_regions[41] = 0
rad_gh_regions[42] = 0
rad_gh_regions[43] = 0
rad_gh_regions[44] = 0
rad_gh_regions[45] = 0
rad_gh_regions[46] = 0
rad_gh_regions[47] = 0
rad_gh_regions[48] = 0
rad_gh_regions[49] = 0
rad_gh_regions[50] = 0
rad_gh_regions[51] = 0
rad_gh_regions[52] = 0
rad_gh_regions[53] = 0
rad_gh_regions[54] = 0
rad_gh_regions[55] = 0
rad_gh_regions[56] = 0
rad_gh_regions[57] = 0
rad_gh_regions[58] = 0
rad_gh_regions[59] = 0
rad_gh_regions[60] = 0
rad_gh_regions[61] = 0
rad_gh_regions[62] = 0
rad_gh_regions[63] = 0
gh_min_seg_size = 4194304
gh_fail_if_no_mem = 1
private_cache_percent = 0
gh_keep_sorted = 0
gh_front_alloc = 1
replicate_user_text = 1
enable_yellow_zone = 0
boost_pager_priority = 0
kstack_free_target = 5
wire_audit_count = 0
vm_prepop_percent = 100
The fire follows shedule...
Alexey Borchev
Regular Advisor

Re: Who is using swap space?

Yes, I've got 16 GB Ram, 12 is given to Oracle, it seems Oracle is happy with it so far.

It's strange that Kernel idle maps not 16, but 20.5 GB.
Is it 16 GB Ram + Swap in-use space?

Any ideas about identifying resource user?
Otherwice i'll have to reboot my system - opportunity will be lost.

P.S.
It decreased slightly:
In-use space: 521399 pages ( 28%)

Maybe, it worth try to kill swapped out processes (which has W in ps aux)?
The fire follows shedule...
Michael Schulte zur Sur
Honored Contributor

Re: Who is using swap space?

Hi,

ubc_maxpercent = 50
this seems too much to me. I would reduce it to 20. You can do this online with syscongfig.

hth,

Michael
Ralf Puchner
Honored Contributor

Re: Who is using swap space?

Michael,

come on, this parameter is not within my coffee-cup.... ;-)

Alexey,

are you sure you configured the machine as described within the Oracle tuning guide? The values must be calculated depending your needs and reflecting your database setup. So posting the sysconfigtab informations doesn't really help to analyze the problem without knowing your complete setup and dependencies.

Use "collect" to monitor the system and to get a clue what is going on. But as described in my previous posting swapping is normal. With the tool you will get the time and memory usage also the dependencies (I/O etc.)








Help() { FirstReadManual(urgently); Go_to_it;; }
Joris Denayer
Respected Contributor

Re: Who is using swap space?

If this occurs after a couple of months, you might have a memory leak somewhere.
You can use the command
# vmstat -M
and monitor if a particular bucketsize and memory type is growing constantly.

Joris
To err is human, but to really faul things up requires a computer
Alexey Borchev
Regular Advisor

Re: Who is using swap space?


1) Michael, i'll try to
ubc_maxpercent = 50 -> 20
but I'll wait for less busy period.
As far as i understand, Trunix will not swap out filesystem cache - it will shrink it instead.

2) I've run vmstat -M see attachment.
Total memory being used from buckets = 99444208 bytes
Total free memory in buckets = 26347376 bytes
- i.e. here only 100 MB, I am looking for 4.5 GB.

3) I'll read & try collect, thanks.

4) Currently I've got 187K free mem pages,
and 520670 pages used in swap space.
-Why Unix does not read it back into memory?
- Swap usage is slightly decreased. May be some swapped out processes was terminated?

The fire follows shedule...
Joris Denayer
Respected Contributor

Re: Who is using swap space?

Hi Alexej,
A leak is not yet visible, short after a reboot.
Try this command again each week.

If there is a leak, you will see if one of the buckets/memory types is increasing.

Joris
To err is human, but to really faul things up requires a computer
Alexey Borchev
Regular Advisor

Re: Who is using swap space?

Swap space usage decreased:

Currently
In-use space 11551 pages ( 0%)

Oracle instance was re-started during weekend. (Cold backup)

Imho, some Oracle processes where swapped-out, then loaded back into memory, but swap space was still reserved for the process.
Swap space was released when process terminated.

The fire follows shedule...
Joerg Schulenburg
Frequent Advisor

Re: Who is using swap space?

You can switch off swap editing /etc/sysconfigtab removing swapdevice entry.
You can add swap space on runtime if you need it again.
I thought that you can identify swapping process only indirect by looking for large
vsz-rss (ps aux), high syscall and idle activity.
I have problems using more than 16GB on
a 128GB GS1280 with 32 CPUs without swapping,
where the vm system
does strange things (see http://www.uni-magdeburg.de/urzs/marvel/vmbug3.html)
Fighting for a better world with more penguins.
Joerg Schulenburg
Frequent Advisor

Re: Who is using swap space?

>>As far as i understand, Trunix will not swap out filesystem cache - it will shrink it instead.

My experiences are otherwise. Tru64 takes swap instead of shrinking UBC.

>>Why Unix does not read it back into memory?

Unix does, Tru64 does not :(
I think the philosophy behint is, that the
pages read from the swap are cached and if
memory again is low only cached pages has to
be released.

>>swap usage is slightly decreased. May be some swapped out processes was terminated?

Probably. and because of reduced ubc_maxpercent new
processes are not swapped out again.

Did you mention your hardware?
NUMA?
What tells vmstat -R?
Fighting for a better world with more penguins.
Alexey Borchev
Regular Advisor

Re: Who is using swap space?

GS1280, 8 CPU, 32 GB mem.

vmstat -R see in attachment.
The fire follows shedule...
Alexey Borchev
Regular Advisor

Re: Who is using swap space?

>>>>As far as i understand, Trunix will not swap out filesystem cache - it will shrink it instead.
>>My experiences are otherwise. Tru64 takes swap instead of shrinking UBC.
- How did You observed this? The behaviour seems absolutely rediculose, unless ubc_min_percent hit.

>>>>Why Unix does not read it back into memory?

>>Unix does, Tru64 does not :(
I think the philosophy behint is, that the
pages read from the swap are cached and if
memory again is low only cached pages has to
be released.
- Yes, i see the point. So, next time instead of swapping out, kernel will just discard the pages (if they are not "dirty").

The fire follows shedule...
Joerg Schulenburg
Frequent Advisor

Re: Who is using swap space?

>>How did You observed this? The behaviour seems absolutely rediculose, unless ubc_min_percent hit.

I did vmstat -R 1 (or 10 instead of 1) and vmubc -tty -swap -memory (please look at the webpage mentioned in my GS1280 swap thread threadId=859125 for details). Read a big file to fill the UBC,
started a memory consuming testprogram and
swapping started before ubc_minpercent and borrowpercent was hit. I am not quite sure if
actu was not lowered or not lowered enough to
avoid swapping. I will have a look to my logfiles or repeat the experiment later, because I dont have time now and put the original data to my mentioned webpage.

Your attachement:
a snapshot is not enough, do a vmstat -R 60 (or so). vmubc would be good too.
Have a look to the free pages of each RAD. Looks like RAD0 and RAD1 where
temporary out of memory (high acti and pout).
If so, you triggered a kernel bug or
however HP call that misbehaviours (see my thread). But its not for sure, not enough data from you (and HPs engineers) to find out, what exactly goes wrong.
Fighting for a better world with more penguins.