filecache_max ignored?

[David G. Douthitt]

How can this output even be possible?

$ kcusage -h filecache_max
Tunable: filecache_max
Setting: 2147483648
Time Usage %
=============================================
Fri 05/02/08 16:35 CDT 4126244864 192.1
Fri 05/02/08 16:40 CDT 4126244864 192.1
Fri 05/02/08 16:45 CDT 4126244864 192.1
Fri 05/02/08 16:50 CDT 4126240768 192.1
Fri 05/02/08 16:55 CDT 4126244864 192.1
Fri 05/02/08 17:00 CDT 4126244864 192.1
Fri 05/02/08 17:05 CDT 4126240768 192.1
Fri 05/02/08 17:10 CDT 4126244864 192.1
Fri 05/02/08 17:15 CDT 4126244864 192.1
Fri 05/02/08 17:20 CDT 4126244864 192.1
Fri 05/02/08 17:25 CDT 4126244864 192.1
Fri 05/02/08 17:30 CDT - -

This usage was reported as 100% previously in the month, then it jumped to double.

This is in the context of searching for possible memory pressures on this system (an rp7420 with 18G of memory and 2 cells (working as a single cell right now).

[David G. Douthitt]

I should mention that the system is using HP-UX 11i v3 patched to near current (but not current) levels.

By tonight I expect to have this system patched to the current Quality Pack plus one critical patch.

[Don Morris_1]

Well, I've got some speculation almost all of which amounts to there being a bug here somewhere... but let's check the obvious:

What does kctune filecache_max say?
When you had the other Cell in, did you OLA it originally and OLD it or did you do static cell add/delete?

[David G. Douthitt]

kctune reports:

filecache_max 2147483648 2Gb Imm (auto disabled)
filecache_min 2147483648 2Gb Imm (auto disabled)

As far as the cells go, I didn't convert it to one cell or anything - it was set up that way when I inherited it. I'm new to the multiple cell architecture.

[Don Morris_1]

Oh, ok... you meant you've got a 2 cell platform with both cells in the single nPar then, right?

Back to the problem at hand -- you didn't change the filecache_max at any point from 2Gb I assume then either (if you aren't sure [aka maybe there's multiple sysadmins?] kclog would tell you)...

It may not tell us anything new (the tunable query should be accurate, but lets be sure) -- but compile and run the attached. [Note to anyone who picks this up -- this was a quick addition hack here, anyone who wants to use this in general including me at some point should figure out a cleaner way to compile v3 vs. v2 (where the new fields don't exist). ]

cc +DD32 -D_PSTAT64 -o mem_detail mem_detail.c

./mem_detail

should work.

[David G. Douthitt]

Memory Stat total used avail %used
physical 18381.0 16612.9 1768.1 90%
active virtual 9641.1 7376.5 2264.6 77%
active real 7411.5 5936.7 1474.8 80%
memory swap 17483.3 7632.2 9851.1 44%
device swap 28672.0 9119.0 19553.0 32%
Activations: 0 total, 0 rate. Deactivations: 0 total, 0 rate.
Reclaims from Swap: 10265 total (Up 0), 0 rate.
UFC statistics:
File Cache size: 0.00 Mb
File System Buffer Cache size: 0.00 Mb
Average CPU wait cycles for I/O: 134217743.
Megabytes wired for I/O: 0.00

What can you make of that?

[Don Morris_1]

That I should test this stuff even if it means rebooting to the work side of things, sorry.

Corrected version of the program attached (there was truncation of the values going on).

Sample output on one of my v3 boxes:

# ./mem_details
Memory Stat total used avail %used
physical 32631.2 7209.5 25421.7 22%
active virtual 690.7 496.3 194.4 72%
active real 284.6 206.3 78.3 73%
memory swap 31037.4 5137.3 25900.1 17%
device swap 32768.0 349.9 32418.1 1%
Activations: 0 total, 0 rate. Deactivations: 0 total, 0 rate.
Reclaims from Swap: 0 total (Up 0), 0 rate.
UFC statistics:
File Cache size: 6.51 Mb
File System Buffer Cache size: 2.32 Mb
Average CPU wait cycles for I/O: 7672650.
Megabytes wired for I/O: 0.00

[Don Morris_1]

One more cycle -- I don't like that you show a high physical consumption but a low user space. Makes me wonder where it went... so I added what kernel counters we can here.

Example:
Memory Stat total used avail %used
physical 32631.2 14600.1 18031.1 45%
active virtual 622.2 489.8 132.4 79%
active real 350.0 241.6 108.4 69%
memory swap 31037.4 6385.0 24652.4 21%
device swap 32768.0 351.0 32417.0 1%
Activations: 0 total, 0 rate. Deactivations: 0 total, 0 rate.
Reclaims from Swap: 0 total (Up 0), 0 rate.
RAM Mb claimed for System in boot: 3275.28
RAM Mb claimed for Kernel Dynamic memory: 4074.88
UFC statistics:
File Cache size: 7719.07 Mb
File System Buffer Cache size: 2.35 Mb
Average CPU wait cycles for I/O: 446862.
Megabytes wired for I/O: 0.00

[Don Morris_1]

Ok... found a PA cellular v3 box to try to be close to your environment.

First thing to note -- kcusage definitely adjusts the history based on the current value of the tunable - not the value as it was at the time stamp.

Example:
Fri May 2 18:02:24 PDT 2008
Tunable: filecache_max
Setting: 5880414208
Time Usage %
=============================================
Fri 05/02/08 17:05 PDT 232034304 3.9
Fri 05/02/08 17:10 PDT 231776256 3.9
Fri 05/02/08 17:15 PDT 231768064 3.9
Fri 05/02/08 17:20 PDT 231763968 3.9
Fri 05/02/08 17:25 PDT 231751680 3.9
Fri 05/02/08 17:30 PDT 232058880 3.9
Fri 05/02/08 17:35 PDT 231944192 3.9
Fri 05/02/08 17:40 PDT 231673856 3.9
Fri 05/02/08 17:45 PDT 231194624 3.9
Fri 05/02/08 17:50 PDT 233607168 4.0
Fri 05/02/08 17:55 PDT 865787904 14.7
Fri 05/02/08 18:00 PDT 1529065472 26.0

vmpamh2# kctune filecache_max=15%
==> Update the automatic 'backup' configuration first? n
* Future operations will ask whether to update the backup.
* The requested changes have been applied to the currently
running configuration.
Tunable Value Expression Changes
filecache_max (before) 5880414208 Default Imm (auto disabled)
(now) 1764124261 15%

Fri May 2 18:03:24 PDT 2008
Tunable: filecache_max
Setting: 1764124261
Time Usage %
=============================================
Fri 05/02/08 17:05 PDT 232034304 13.2
Fri 05/02/08 17:10 PDT 231776256 13.1
Fri 05/02/08 17:15 PDT 231768064 13.1
Fri 05/02/08 17:20 PDT 231763968 13.1
Fri 05/02/08 17:25 PDT 231751680 13.1
Fri 05/02/08 17:30 PDT 232058880 13.2
Fri 05/02/08 17:35 PDT 231944192 13.1
Fri 05/02/08 17:40 PDT 231673856 13.1
Fri 05/02/08 17:45 PDT 231194624 13.1
Fri 05/02/08 17:50 PDT 233607168 13.2
Fri 05/02/08 17:55 PDT 865787904 49.1
Fri 05/02/08 18:00 PDT 1755656192 99.5

So your original output would fit with filecache_max being halved... I'll see if I can reproduce that exact behavior next, though I expect I can.

Which leads me back to wondering if your /var/adm/kc.log shows a change to filecache_max after 17:25 CDT, since that would explain the whole thing. (Say someone tried lowering the file cache to get memory back to the system as a whole [although I'll point out that as long as filecache_min is lower, the cache should shrink if there's memory pressure for actual kernel or user allocations...]

[Don Morris_1]

Yup, that will definitely do it:

Fri May 2 18:07:25 PDT 2008
Tunable: filecache_max
Setting: 5880414208
Time Usage %
=============================================
Fri 05/02/08 17:10 PDT 231776256 3.9
Fri 05/02/08 17:15 PDT 231768064 3.9
Fri 05/02/08 17:20 PDT 231763968 3.9
Fri 05/02/08 17:25 PDT 231751680 3.9
Fri 05/02/08 17:30 PDT 232058880 3.9
Fri 05/02/08 17:35 PDT 231944192 3.9
Fri 05/02/08 17:40 PDT 231673856 3.9
Fri 05/02/08 17:45 PDT 231194624 3.9
Fri 05/02/08 17:50 PDT 233607168 4.0
Fri 05/02/08 17:55 PDT 865787904 14.7
Fri 05/02/08 18:00 PDT 1761681408 30.0
Fri 05/02/08 18:05 PDT 1933262848 32.9

vmpamh2# kctune filecache_max=15%
==> Update the automatic 'backup' configuration first? n
* Future operations will ask whether to update the backup.
* The requested changes have been applied to the currently
running configuration.
Tunable Value Expression Changes
filecache_max (before) 5880414208 Default Imm (auto disabled)
(now) 1764124261 15%

Fri May 2 18:08:25 PDT 2008
Tunable: filecache_max
Setting: 1764124261
Time Usage %
=============================================
Fri 05/02/08 17:10 PDT 231776256 13.1
Fri 05/02/08 17:15 PDT 231768064 13.1
Fri 05/02/08 17:20 PDT 231763968 13.1
Fri 05/02/08 17:25 PDT 231751680 13.1
Fri 05/02/08 17:30 PDT 232058880 13.2
Fri 05/02/08 17:35 PDT 231944192 13.1
Fri 05/02/08 17:40 PDT 231673856 13.1
Fri 05/02/08 17:45 PDT 231194624 13.1
Fri 05/02/08 17:50 PDT 233607168 13.2
Fri 05/02/08 17:55 PDT 865787904 49.1
Fri 05/02/08 18:00 PDT 1761681408 99.9
Fri 05/02/08 18:05 PDT 1933262848 109.6

Once kcusage has a chance to have a reporting stamp post tunable change, though:

Fri May 2 18:09:25 PDT 2008
Memory Stat total used avail %used
physical 12272.0 4723.5 7548.5 38%
active virtual 427.0 242.4 184.6 57%
active real 258.2 140.2 118.0 54%
memory swap 11216.0 2311.2 8904.8 21%
device swap 16370.0 314.0 16056.0 2%
0.0 Mb of pages actually paged out to swap.
Activations: 0 total, 0 rate. Deactivations: 0 total, 0 rate.
Reclaims from Swap: 0 total (Up 0), 0 rate.
RAM Mb claimed for System in boot: 1185.92
RAM Mb claimed for Kernel Dynamic memory: 1205.53
UFC statistics:
File Cache size: 1678.41 Mb
File System Buffer Cache size: 1.70 Mb
Average CPU wait cycles for I/O: 435809.
Megabytes wired for I/O: 0.00

Fri May 2 18:10:25 PDT 2008
Tunable: filecache_max
Setting: 1764124261
Time Usage %
=============================================
Fri 05/02/08 17:15 PDT 231768064 13.1
Fri 05/02/08 17:20 PDT 231763968 13.1
Fri 05/02/08 17:25 PDT 231751680 13.1
Fri 05/02/08 17:30 PDT 232058880 13.2
Fri 05/02/08 17:35 PDT 231944192 13.1
Fri 05/02/08 17:40 PDT 231673856 13.1
Fri 05/02/08 17:45 PDT 231194624 13.1
Fri 05/02/08 17:50 PDT 233607168 13.2
Fri 05/02/08 17:55 PDT 865787904 49.1
Fri 05/02/08 18:00 PDT 1761681408 99.9
Fri 05/02/08 18:05 PDT 1933262848 109.6
Fri 05/02/08 18:10 PDT 1761714176 99.9

[David G. Douthitt]

I don't know if this is fair or comparable, but I'm in process of rebooting and have already taken down the memory consumers (Caché databases) - but you can see the results anyway:

niagara # ./mem_detail
Memory Stat total used avail %used
physical 18381.0 9737.6 8643.4 53%
active virtual 781.4 441.1 340.4 56%
active real 462.9 275.3 187.6 59%
memory swap 17483.3 7240.1 10243.2 41%
device swap 28672.0 642.4 28029.6 2%
Activations: 0 total, 0 rate. Deactivations: 0 total, 0 rate.
Reclaims from Swap: 10432 total (Up 0), 0 rate.
RAM Mb claimed for System in boot: 1844.86
RAM Mb claimed for Kernel Dynamic memory: 3546.61
UFC statistics:
File Cache size: 2036.84 Mb
File System Buffer Cache size: 1901.54 Mb
Average CPU wait cycles for I/O: 692296.
Megabytes wired for I/O: 0.00

[David G. Douthitt]

This is after rebooting with all databases down:

Memory Stat total used avail %used
physical 18381.0 5442.8 12938.2 30%
active virtual 545.1 342.6 202.5 63%
active real 374.6 233.0 141.6 62%
memory swap 17483.3 4198.8 13284.5 24%
device swap 28672.0 417.4 28254.6 1%
Activations: 0 total, 0 rate. Deactivations: 0 total, 0 rate.
Reclaims from Swap: 0 total (Up 0), 0 rate.
RAM Mb claimed for System in boot: 1844.86
RAM Mb claimed for Kernel Dynamic memory: 801.17
UFC statistics:
File Cache size: 172.96 Mb
File System Buffer Cache size: 4.19 Mb
Average CPU wait cycles for I/O: 2512398.
Megabytes wired for I/O: 0.00

and after starting all databases with no users yet:

Memory Stat total used avail %used
physical 18381.0 6876.0 11505.0 37%
active virtual 6347.8 2196.5 4151.3 35%
active real 1822.9 791.6 1031.3 43%
memory swap 17483.3 4205.2 13278.2 24%
device swap 28672.0 6212.1 22459.9 22%
Activations: 0 total, 0 rate. Deactivations: 0 total, 0 rate.
Reclaims from Swap: 0 total (Up 0), 0 rate.
RAM Mb claimed for System in boot: 1844.86
RAM Mb claimed for Kernel Dynamic memory: 807.52
UFC statistics:
File Cache size: 267.01 Mb
File System Buffer Cache size: 4.19 Mb
Average CPU wait cycles for I/O: 22449961.
Megabytes wired for I/O: 0.00

And after the database applications started:

Memory Stat total used avail %used
physical 18381.0 9855.8 8525.2 54%
active virtual 8669.5 6758.4 1911.0 78%
active real 4615.1 3801.8 813.3 82%
memory swap 17483.3 4388.2 13095.2 25%
device swap 28672.0 8606.4 20065.6 30%
Activations: 0 total, 0 rate. Deactivations: 0 total, 0 rate.
Reclaims from Swap: 0 total (Up 0), 0 rate.
RAM Mb claimed for System in boot: 1844.86
RAM Mb claimed for Kernel Dynamic memory: 990.19
UFC statistics:
File Cache size: 2036.02 Mb
File System Buffer Cache size: 4.19 Mb
Average CPU wait cycles for I/O: 4765957.
Megabytes wired for I/O: 0.00