General
cancel
Showing results for 
Search instead for 
Did you mean: 

High system memory usage, max_dbc_pct is not high.

SOLVED
Go to solution
ricardor_1
Frequent Advisor

High system memory usage, max_dbc_pct is not high.

We have a 11.23 ia64 server running a couple of Oracle 10gR2 instances.

Some days ago the system started paging with no setup modifications.

The system has 20 GB RAM and the sum os the SGAs is surely not the problem (the DBAs reduced it to about 3 GB).

System memory utilisation is very high and we cannot figure out what´s happening:

$ /usr/sbin/swapinfo -mt
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 20448 4304 16144 21% 0 - 1 /dev/vg00/lvol2
reserve - 1636 -1636
memory 20431 19846 585 97%
total 40879 25786 15093 63% - 0 -


$ vmstat 1 10
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 2 0 1307872 19505 50 9 31 19 18 0 381 1790 10492 547 1 1 97
1 2 0 1307872 19314 181 19 410 349 122 0 4400 2995 8421 1009 0 2 98
1 2 0 1307872 19302 163 15 419 371 148 0 4501 2891 6931 928 0 3 97
1 2 0 1307872 19794 154 36 402 296 118 0 3994 2816 6411 860 0 3 97
1 2 0 1307872 19625 160 81 464 269 215 0 3533 2728 5955 848 1 3 96
1 2 0 1307872 19436 156 113 463 325 254 0 3500 2677 4927 804 1 1 98
1 2 0 1307872 19602 143 92 413 337 213 0 4518 2654 4365 752 0 2 98

Glance:

Total VM : 6.8gb Sys Mem : 18.7gb User Mem: 568mb Phys Mem : 20.0gb
Active VM: 5.0gb Buf Cache: 613mb Free Mem: 75mb FileCache: na

Buffer
dbc_max_pct 3 3 Immed
dbc_min_pct 3 3 Immed

Any clues?
14 REPLIES
ricardor_1
Frequent Advisor

Re: High system memory usage, max_dbc_pct is not high.

Just to mention, I currently have no root access to this machine, so I´m unable to run kmeminfo...
Dennis Handly
Acclaimed Contributor

Re: High system memory usage, max_dbc_pct is not high.

How much shared memory is in use? ipcs -ma
SoorajCleris
HPE Pro

Re: High system memory usage, max_dbc_pct is not high.

Hi,

what is "top" saying

Regards,
Sooraj
"UNIX is basically a simple operating system, but you have to be a genius to understand the simplicity" - Dennis Ritchie
ricardor_1
Frequent Advisor

Re: High system memory usage, max_dbc_pct is not high.

We had to reboot that server, but I did check "ipcs -am" (although I did not save the output).

The shared memory segments added up to about a little more than 3 GB, which was the amount of memory allocated to oracle´s SGA.

Top indicated low user CPU load. vhand was always active, demanding about 10% CPU.
Dennis Handly
Acclaimed Contributor

Re: High system memory usage, max_dbc_pct is not high.

>The shared memory segments added up to about a little more than 3 GB,

Hmm, that's not close to your 19.8 Gb used memory.
ricardor_1
Frequent Advisor

Re: High system memory usage, max_dbc_pct is not high.

We have another server with the very same behavior. The systems were cloned using ignite.

vmstat

procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 0 0 1736001 7933 64 19 38 14 25 0 348 621 6748 262 1 2 97
1 0 0 1736001 7861 34 7 27 3 25 0 48 579 7672 249 1 0 99
1 0 0 1733790 7861 27 5 21 2 20 0 38 560 6586 238 0 1 99
1 0 0 1733790 7861 21 4 16 1 16 0 30 541 5651 227 1 0 99

kmtune

dbc_max_pct 3 3 Immed
dbc_min_pct 3 3 Immed


ipcs -amIPC status from /dev/kmem as of Tue Aug 17 15:25:33 2010
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME
Shared Memory:
m 0 0x4124096a --rw-rw-rw- root root root root 0 348 857 857 17:28:06 17:28:06 17:27:59
m 1 0x4e0c0002 --rw-rw-rw- root root root root 3 61760 857 28382 15:25:20 15:25:20 17:27:59
m 2 0x41281a89 --rw-rw-rw- root root root root 2 8192 857 859 16:56:52 17:27:59 17:27:59
m 3 0x00a5c581 --rw------- sfmdb users sfmdb users 9 10469376 1445 1448 17:28:16 17:28:16 17:28:16
m 4 0x412405b5 --rw------- root root root root 1 4096 1588 2216 17:29:02 no-entry 17:29:02
m 106463237 0x2f1ccb73 --rw-rw-r-- root sys root sys 1 10488 21325 4130 11:59:55 no-entry 17:34:08
m 6 0x0c6629c9 --rw-r----- root root root root 2 22637624 1778 28258 15:24:58 15:25:33 17:28:54
m 7 0x06347849 --rw-rw-rw- root root root root 1 65544 1778 1786 17:28:54 17:28:54 17:28:54
m 8 0xffffffff --rw-r--r-- root root root root 0 22908 1772 28258 15:25:31 15:25:31 17:28:55
m 9 0x2f1ccb72 --rw-rw-r-- root sys root sys 1 5592 21329 4121 11:59:42 no-entry 17:34:08
m 36864010 0xcccaad98 --rw-r----- oracle dba oracle dba 14 132083712 27067 23443 14:53:02 14:58:04 20:24:58
m 1638411 0xadf4a31c --rw-rw---- oracle dba oracle dba 27 1077948416 22176 27478 15:18:05 15:18:47 8:51:22
m 163852 0x0128a0aa --rw-rw-rw- root sys root sys 2 962 26070 27029 18:36:12 no-entry 18:35:32
m 425997 0x00000000 D-rw------- root root root root 2 512000 18503 18503 22:05:59 no-entry 22:05:59
m 327694 0x00000000 D-rw------- root root root root 2 2162688 18503 18503 22:05:59 no-entry 22:05:59
m 327695 0x00000000 D-rw------- root root root root 2 14896 18503 18503 22:05:59 no-entry 22:05:59
m 426000 0x0128a493 --rw-rw-rw- root sys root sys 2 926 26070 2232 11:47:12 no-entry 11:47:12
m 294929 0x00000000 D-rw-rw-rw- root sys root sys 1 946 26070 26070 11:45:28 11:45:28 11:45:28
m 294930 0x0128a0ae --rw-rw-rw- root sys root sys 2 1022 26070 2125 11:45:30 no-entry 11:45:30
m 327699 0x0128a601 --rw-rw-rw- root sys root sys 2 926 26070 2233 11:47:13 no-entry 11:47:13
m 294932 0x0128a4af --rw-rw-rw- root sys root sys 2 994 26070 2142 11:45:37 no-entry 11:45:37
m 611909653 0x9b857a7c --rw-rw---- oracle dba oracle dba 26 482353152 9167 26668 15:11:33 15:13:11 11:01:11
m 262166 0x0128a4b9 --rw-rw-rw- root sys root sys 2 1062 26070 2144 11:45:38 no-entry 11:45:38
m 262167 0x0128a4bd --rw-rw-rw- root sys root sys 2 1076 26070 1273 10:37:31 no-entry 11:45:38
m 229400 0x0128a4bf --rw-rw-rw- root sys root sys 2 1028 26070 2145 11:45:38 no-entry 11:45:38
m 228032537 0xaf2d21c8 --rw-rw---- oracle dba oracle dba 28 935342080 26048 28357 15:25:07 15:25:07 17:56:14
m 235634714 0x0cd67ec0 --rw-rw---- oracle dba oracle dba 28 616570880 23329 28351 15:25:06 15:25:06 14:52:27
m 157679644 0x55f72024 --rw-rw---- oracle dba oracle dba 26 1627398144 26724 28276 15:24:58 15:25:28 17:59:14


(Oracle´s memory amount to less than 5 GB, this system has 8 GB RAM).


$ /usr/sbin/swapinfo -mt
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 12288 4771 7517 39% 0 - 1 /dev/vg00/lvol2
reserve - 3369 -3369
memory 8143 7359 784 90%
total 20431 15499 4932 76% - 0 -



Total VM : 8.2gb Sys Mem : 6.8gb User Mem: 846mb Phys Mem : 8.0gb
Active VM: 6.7gb Buf Cache: 244mb Free Mem: 62mb FileCache: na
MemFS Blk Cnt: na MemFS Swp Cnt: na
ricardor_1
Frequent Advisor

Re: High system memory usage, max_dbc_pct is not high.

kmeminfo:


tool: kmeminfo 7.11 - HP CONFIDENTIAL

unix: /stand/current/vmunix 11.23 64bit IA64 on host "hxdesdb2"

core: /dev/kmem live

link: Wed Nov 25 12:31:20 SAT 2009

boot: Wed Dec 16 16:26:00 2009

time: Wed Aug 18 10:27:13 2010

nbpg: 4096 bytes





----------------------------------------------------------------------

Physical memory usage summary (in page/byte/percent):



Physical memory = 2084610 8.0g 100%

Free memory = 16724 65.3m 1%

User processes = 205954 804.5m 10% details with -user

System = 1850903 7.1g 89%

Kernel = 1788365 6.8g 86% kernel text and data

Dynamic Arenas = 1641185 6.3g 79% details with -arena

M_IOSYS = 1444837 5.5g 69%

vx_global_pool = 23935 93.5m 1%

spinlock = 21554 84.2m 1%

vx_buffer_cache = 21216 82.9m 1%

mdep arena = 12467 48.7m 1%

Other arenas = 117176 457.7m 6% details with -arena

Super page pool = 5153 20.1m 0% details with -kas

Static Tables = 107709 420.7m 5% details with -static

pfdat = 48864 190.9m 2%

vhpt = 16384 64.0m 1%

nbuf = 11696 45.7m 1% bufcache headers

inode = 9526 37.2m 0%

text = 7475 29.2m 0% vmunix text section

Other tables = 13763 53.8m 1% details with -static

Buffer cache = 62538 244.3m 3% details with -bufcache

UFC file mrg = 0 0.0 0%

UFC meta mrg = 0 0.0 0%

Don Morris_1
Honored Contributor
Solution

Re: High system memory usage, max_dbc_pct is not high.

Can we see `kmeminfo -V -arena M_IOSYS` please? (That's not a very specific arena and I'd like to see if the object sizes taking up all that memory give a hint as to where the usage is stemming from).
Don Morris_1
Honored Contributor

Re: High system memory usage, max_dbc_pct is not high.

Also, `swlist -l patch |grep PHKL` and `swlist -l product |grep IPFilter` -- there are some fixes for past 11.23 M_IOSYS leaks that it would save time to know if you already have or not.
ricardor_1
Frequent Advisor

Re: High system memory usage, max_dbc_pct is not high.


Dynamic Arenas:


Variable arena "M_IOSYS" owns 1445287 pages (5.5gb):
Free objects represent 140.0kb (0%) of all memory in this arena.
idx objsz pages bytes % nobjs used free %
0 24 1443373 5.5g 100 181864998 181864802 196 0
1 56 910 3.6m 0 57330 57248 82 0
2 120 13 52.0k 0 403 367 36 9
3 248 101 404.0k 0 1515 1497 18 1
4 504 45 180.0k 0 315 130 185 59
5 1016 174 696.0k 0 522 503 19 4
6 2040 243 972.0k 0 243 243 0 0
15 4096 4 16.0k 0 4 4 0 0
16 8192 2 8.0k 0 1 1 0 0
23 36864 422 1.6m 0 46 46 0 0
ricardor_1
Frequent Advisor

Re: High system memory usage, max_dbc_pct is not high.

The attached files contains the requested commands.

We do have ipfilter installed:


IPF-HP A.11.23.15.01 HP IPFilter 3.5alpha5
PFIL-HP A.11.23.15.01 HP IPFilter PFIL Interface


You mean PHKL_37233? We have PHKL_37305 installed.
Don Morris_1
Honored Contributor

Re: High system memory usage, max_dbc_pct is not high.

PHKL_37305 was one of the ones I wanted to see, yes.

IPFilter has some fixes in A.11.23.17 that I thought might play into this - but those should show a 512 byte allocation pattern, not a 24 byte object pattern as you show.

That small of an allocation does make it harder to track down by just pawing through the existing allocations. Rather, monitoring the actual allocations to get the code path causing the leak should be the next step. Unfortunately, that involves more intrusive tools than kmeminfo by itself.

Contact support and have them work with you to get a leak analysis done on one or the other of the machines exhibiting the issue. (The tool involved to do this can most certainly negatively impact the system in some cases [the way it should be used for yours is fairly unobtrusive, but I suspect you'll need a reboot to be able to run the tool and see the leak in progress], so I don't want to just hand it to you and hope for the best).
ricardor_1
Frequent Advisor

Re: High system memory usage, max_dbc_pct is not high.

Don,

Thank you very much for your attention. We have an open support case and they asked if there were any removed LUNs from the system in the past, before the last reboot, which is the case.

I´ll submit your points and update this thread as soon as we find sometime. We may need to reboot this server anyway, since the existing paging rate is degrading performance considerably.
mvpel
Trusted Contributor

Re: High system memory usage, max_dbc_pct is not high.

My M_IOSYS arena has 14.5GB in it, nearly all of which are extra-large pages.

Don, what's the command needed to paw through the allocations?