topic Re: The value of OS' file cache in Operating System - HP-UX

The value of OS' file cache

rustam_2 — Mon, 06 Sep 2010 10:16:22 GMT

Hi all,
Recently my server had Panic Shutdown. I called to support center and then they required to send crash file. In few days they determined that i have some problem with OS, such as:
...

3. A big value of memory uses for file system cache. Because i use Oracle DB there and DB uses own cache value.

Tunables
========
filecache_max = 47.6%
filecache_min = 4.8%
fs_meta_max = 46.6%
fs_meta_min = 1.0%

He recommended to turn off or decrease the file system cache value.

So what do you think, should i turn off or decrease this value? And do you know any documents where is the description of this operation. I mean where is the clear description of doing this trip.

Regards,
Rustam

Re: The value of OS' file cache

Michal Kapalka (mikap) — Mon, 06 Sep 2010 10:26:11 GMT

hi,

you could read this docu :

"HP-UX Performance Cookbook"

http://h21007.www2.hp.com/portal/download/files/unprot/devresource/Docs/TechPapers/UXPerfCookBook.pdf

mikap

Re: The value of OS' file cache

James R. Ferguson — Mon, 06 Sep 2010 12:20:26 GMT

Hi Rustam:

Have a look at the manpages for 'filecache_max(5)' as a guide.

Regards!

...JRF...

Re: The value of OS' file cache

rustam_2 — Tue, 07 Sep 2010 10:02:51 GMT

Thanks, mikap and JRF.
I read both material and decided to change this value. First i wanted to check the memory value. I found out strange value on my server. I have 2 servers prod and test, i attached screen of top command.
Prod has:
Memory: 5195184K (1353332K) real, 7307632K (1888260K) virtual, 395860K free Page
e# 1/22
So
-RAM is 8Gb - used 5.2 Gb of them.
-Virtual memory is 1Gb and used 700Mb

Test has:
Memory: 12330140K (3462780K) real, 21361628K (7866040K) virtual, 117708K free Pa
age# 1/32

How many GB are for TEST server is used?

Regards,
Rustam.

Re: The value of OS' file cache

Dennis Handly — Tue, 07 Sep 2010 20:14:35 GMT

Yes, your file cache numbers are too big for database servers.
Please provide the "swapinfo -tam" values for each machine.

Re: The value of OS' file cache

rustam_2 — Wed, 08 Sep 2010 03:33:18 GMT

Hi Dennis,
I attached the result of swapinfo tam od prod and test. Test uses 90% of swap memory and prod just 42%. Therefore he suggested to turn off or decrease cache info for FileSystem. What do you think?

Regards,
Rustam.

Re: The value of OS' file cache

Dennis Handly — Wed, 08 Sep 2010 07:53:13 GMT

>I attached the result of swapinfo -tam of prod and test.

(It would be more helpful to attach a text file, since I can't grep or diff pixels.)

>Test uses 90% of swap memory and prod just 42%. Therefore he suggested to turn off or decrease cache info for FileSystem. What do you think?

Yes, it makes a lot of sense. Lots more memory is used for caching? in the second case.

Re: The value of OS' file cache

rustam_2 — Wed, 08 Sep 2010 09:44:30 GMT

> (It would be more helpful to attach a text file, since I can't grep or diff pixels.)

For prod
# swapinfo -tam
Mb Mb Mb PCT
TYPE AVAIL USED FREE USED
dev 8192 0 8192 0%
reserve - 7078 -7078
memory 15555 2909 12646 19%
total 23747 9987 13760 42%

For test
# swapinfo -tam
Mb Mb Mb PCT
TYPE AVAIL USED FREE USED
dev 8192 5152 3040 63%
reserve - 3040 -3040
memory 15555 13328 2227 86%
total 23747 21520 2227 91%

>>Yes, it makes a lot of sense. Lots more memory is used for caching? in the second case.

So i have to turn off file cache value?

Regards,
Rustam.

Re: The value of OS' file cache

Dennis Handly — Thu, 09 Sep 2010 06:01:54 GMT

>So I have to turn off file cache value?

No, just reduce it to a smaller value. Check the value of filecache_max on prod and set test to that value. But if it is 50%, you could just set it to 6% or so.

Re: The value of OS' file cache

rustam_2 — Mon, 13 Sep 2010 05:15:57 GMT

Hi Dennis,
I checked the value of filecache_max on both server and there are default values - 8155217920. In test is used percentage is 98% and in prod is 16%. How can i check the value of filecache_max in %?

Regards,
Rustam

Re: The value of OS' file cache

chris huys_4 — Mon, 13 Sep 2010 06:18:56 GMT

Hi Rustam,

> How can i check the value of filecache_max
> in %?
The filecache_max percentage of the system is displayed in the crashinfo output that was sent by HP support.

>>>
Tunables
========
filecache_max = 47.6%
>>>

Or it can also be "counted".

>>>>>
For prod
# swapinfo -tam
Mb Mb Mb PCT
TYPE AVAIL USED FREE USED
dev 8192 0 8192 0%
reserve - 7078 -7078
memory 15555 2909 12646 19%
total 23747 9987 13760 42%
>>>>>

memoryswap shows "as available" 15555 mbyte. On hp-ux 11.31, contrary to previous OSversions, this means that the available amount of memoryswap, equals to the total amount of physical memory (RAM) installed in the system, so the "prod" system has 16Gbyte of physical ram.

filecache_max = 8155217920 = around 8Gbyte and 8Gbyte is 50% of 16Gbyte of physical ram, which is around the 47.6% that was found earlier by hp support.

And to change filecache_max to f.e. 10%, follow the following procedure.
>>>>>>>
ronin # ./crashinfo -c > ./ci_c.out
crashinfo (5.04)

[root@ronin]/var/tmp/crashinfo_1131_ia64
ronin # vi ci_c.out
[..]
Dump time Mon Feb 16 15:27:23 2009
System has been up 19 minutes.
[..]
Physical Memory = 2091669 pages (7.98 GB)
Free Memory = 1483738 pages (5.66 GB)
Average Free Memory = 1494234 pages (5.70 GB)
gpgslim = 7168 pages (28.00 MB)
lotsfree = 138718 pages (541.87 MB)
desfree = 7168 pages (28.00 MB)
minfree = 3328 pages (13.00 MB)
[..]
libp4 (9.344): Opening /stand/current/vmunix /dev/kmem
Tunables
========
filecache_max = 47.6%
filecache_min = 4.8%
fs_meta_max = 46.6%
fs_meta_min = 1.0%

Current Memory Usage
====================
filecache data = 55276 pages (215.92 MB)
filecache metadata = 16 pages (64 KB)
filecache total = 55292 pages (215.98 MB)
filecache total pct = 2.6%

ronin # kctune|grep filecache
filecache_max 4 074 840 064 Default Auto
filecache_min 407482368 Default Auto

47.6% * (2 091 669 *4 * 1024 ) = 47.6% * 8 567 476 224 = 4 001 011 396.608

ronin # smh

kernel configuration

tunables

filecache_max Auto 4 074 840 064 - 4074840064 5.4

10% of 8Gbyte = 800 Mbyte = 800 * 1024 * 1024 = 838860800 bytes = 838 860 800 bytes =

New setting[Expression/Value]: 10%

Recalculate

Do you want to hold the change until next reboot ?: ( ) Yes
(X) No

Back up the current configuration before applying change: (X) Yes
( ) No

modify

x -exit

# shutdown -ry 0

after reboot

ronin # kctune|grep filecache
filecache_max 814968422 10% Imm (auto disabled)
filecache_min 407482368 Default Auto
>>>>>>>>>>

Greetz,
Chris

Re: The value of OS' file cache

rustam_2 — Mon, 13 Sep 2010 07:59:17 GMT

Hi, Chris.
Thanks a lot, i got what to do.
1. command machinfo doesnt work.
# crashinfo
sh: crashinfo: not found.
# whereis crashinfo
crashinfo:
#

2. I did like this and a bit confused. smh-> kernel configuration-> tunables->modify.
There were 2 settings:
New settings [expression/values]:
New settings (evaluated):

So should type 10% for 'New settings [expression/values]:' and that's enough?

I want to restart my server in midnight when users will not working. Is it enough to check NO here -
>> Do you want to hold the change until next reboot ?: ( ) Yes
(X) No

Regards,
Rustam

Re: The value of OS' file cache

rustam_2 — Mon, 13 Sep 2010 08:15:04 GMT

Dennis,
>>>No, just reduce it to a smaller value. Check the value of filecache_max on prod and set test to that value.
But if it is 50%, you could just set it to 6% or so.

The value of filecache_max on prod and test are the same. The only different is that on TEST run 4 database instances and on prod just 1 instance.

Regards,
Rustam

Re: The value of OS' file cache

chris huys_4 — Mon, 13 Sep 2010 08:44:04 GMT

Hi Rustam,

> # crashinfo
> sh: crashinfo: not found.
crashinfo is a hp support utility, given out by hp support personel, so its normal that its not on the system.

> So should type 10% for 'New settings
>[expression/values]:' and that's enough?
Yes.

> I want to restart my server in midnight
> when users will not working. Is it enough
> to check NO here -
> >> Do you want to hold the change until
> next reboot ?: ( ) Yes
> (X) No
My own preference would be not to put NO there, the "smh part of the procedure takes only a minute", so if you need to hold it off untill midnight, why not execute the whole procedure at midnight ?

Greetz,
Chris

Re: The value of OS' file cache

Dennis Handly — Mon, 13 Sep 2010 13:12:39 GMT

>I want to restart my server in midnight when users will not working.

Why? There is no need to reboot, filecache_max(5) is a dynamic tunable.

>The value of filecache_max on prod and test are the same. The only different is that on TEST run 4 database instances and on prod just 1 instance.

Then make them both smaller or at least on TEST.

Re: The value of OS' file cache

rustam_2 — Tue, 14 Sep 2010 13:11:17 GMT

Hi Chris and Dennis,
I got all steps and whole concept of tasks. But have last question, yesterday i couldnt restart my server therefore couldn't change value of filechache_max.
So does this tunable is dynamic or dosn't?
If yes, then great if no then i should wait suitable day.

Regards,
Rustam

Re: The value of OS' file cache

Dennis Handly — Wed, 15 Sep 2010 09:26:26 GMT

>So is this tunable is dynamic or not?

That's what the documentation and command says:
$ /usr/sbin/kctune -v filecache_max
Tunable filecache_max ...
Can Change Automatic Tuning Enabled

Restrictions on Changing
These tunables are dynamic and automatic.

Re: The value of OS' file cache

chris huys_4 — Wed, 15 Sep 2010 11:30:00 GMT

Hi Rustam,

The filecache_max tunable is dynamic. No need for the "shutdown -ry 0" command, "see corrected procedure".

Greetz,
Chris

"Corrected procedure"

ronin # ./crashinfo -c > ./ci_c.out
crashinfo (5.04)

[root@ronin]/var/tmp/crashinfo_1131_ia64
ronin # vi ci_c.out
[..]
Dump time Mon Feb 16 15:27:23 2009
System has been up 19 minutes.
[..]
Physical Memory = 2091669 pages (7.98 GB)
Free Memory = 1483738 pages (5.66 GB)
Average Free Memory = 1494234 pages (5.70 GB)
gpgslim = 7168 pages (28.00 MB)
lotsfree = 138718 pages (541.87 MB)
desfree = 7168 pages (28.00 MB)
minfree = 3328 pages (13.00 MB)
[..]
libp4 (9.344): Opening /stand/current/vmunix /dev/kmem
Tunables
========
filecache_max = 47.6%
filecache_min = 4.8%
fs_meta_max = 46.6%
fs_meta_min = 1.0%

Current Memory Usage
====================
filecache data = 55276 pages (215.92 MB)
filecache metadata = 16 pages (64 KB)
filecache total = 55292 pages (215.98 MB)
filecache total pct = 2.6%

ronin # kctune|grep filecache
filecache_max 4 074 840 064 Default Auto
filecache_min 407482368 Default Auto

47.6% * (2 091 669 *4 * 1024 ) = 47.6% * 8 567 476 224 = 4 001 011 396.608

ronin # smh

kernel configuration

tunables

filecache_max Auto 4 074 840 064 - 4074840064 5.4

10% of 8Gbyte = 800 Mbyte = 800 * 1024 * 1024 = 838860800 bytes = 838 860 800 bytes =

New setting[Expression/Value]: 10%

Recalculate

Do you want to hold the change until next reboot ?: ( ) Yes
(X) No

Back up the current configuration before applying change: (X) Yes
( ) No

modify

x -exit

ronin # kctune|grep filecache
filecache_max 814968422 10% Imm (auto disabled)
filecache_min 407482368 Default Auto

Re: The value of OS' file cache

rustam_2 — Mon, 20 Sep 2010 04:04:12 GMT

Thanks for both of you, Dennis and Chris.
I changed and now the value is 14%. It doesnt required restart of OS, i just need stop and start my DB.

Regards,
Rustam

Re: The value of OS' file cache

rustam_2 — Mon, 20 Sep 2010 05:11:41 GMT

By the way, another question :).Sorry for that, i think i have to change min value also, isnt it?

# kctune|grep filecache
filecache_max 2283461590 14%Imm (auto disabled)

filecache_min 815521792 Default Auto

Or it is ok, when max is ~2.2 Gb and min is 820Mb ?

regards,
Rustam