system memory consumption

[Fabio Acquarone]

Hi all, (sorry for the long post but is an higlight that needs best explanation as possible)

I found a not well known (for my knowledge) behaviour on HP-UX11iv1 O.S.
System memory is increasing during the time after some activity from users!!!???? ....passing from 550Mbytes (occupied after the O.S. boot ) to 1.3GBytes !!!!.

The tool used to monitor RAM is GlancePlus.

I tried to find different systems (rp3440 principally) and i saw the same issue.

At the end, after some attempts, i discovered a indirect cause.

I saw that after an I/O peak (disk I/O burst) the System Memory is affected but also after a simple "find" command.

The increased amount in never deallocated by the O.S.

This is the first time that i see a such behaviour, during past sizing scalability benchmarks i never seen a significant increment of the System Memory after O.S. boot

Now, is it a standard behaviour??? or not??

Could be some kernel parameter affect it ??

Thank you very much.

[Jean-Luc Oudart]

Hi

could you post :
dbc_max_pct
dbc_min_pct
total amount of RAM

Regards
Jean-Luc

[TwoProc]

I think the disk cache for the OS disk system is kicking in for you.

This is regulated by the value of max_dbc_pct and min_dbc_pct.

What happened is that you searched a bunch of disk, so it is now cached for you should you need it.

Depending on how much ram is in the box (you didn't indicate) the amount of ram you indicated may or may not be "too much".

Review how much ram you have and what the percentage of disk cache you've got the thing set for (min and max_dbc). The max_dbc may still be set for 50% by default in the kernel. Which means that the system may grab up to 50% of your ram to buffer disk.

To check the values:

kmtune | grep -e max_dbc -e min_dbc

[Fabio Acquarone]

Thank you for the answer....but..

dbc_max_pct is set to 15
dbc_min_pct is set to 5

The total amount of RAM is 8GBytes.

The above settings are due to the expected behaviour regarding the BufferCache amount of memory.
In my system i see that the BufferCache remain to the minimum value ( 5% of total memory), instead the SystemMemory grow.
I repeat all is measured using GlancePlus.

Yesterday i found a cause...really strange...expecially after a reboot if i do a
"find / -name -print " i see an increment of the SystemMemory, roughly 200MBytes. If i do some work and the i start an application that internally i doing a "find" i see the same behaviour.

On PATCHING site i found the PHKL_32668 that is reporting the same aspects of my issue, but the installation unfortunatelly has not solved the issue.

I hope that this can help someone to understand well the issue.

Thank you all,

Fabio

[TwoProc]

Well Fabio,

15% of 8G is 1.2G - which is well within the amount of growth(1300MB-550MB=750MB) that you saw, right?

So, I still think it could be max/min dbc settings and subsequent caching.

[Fabio Acquarone]

Hi John,

i am attaching the values shown by GlancePlus. In such display the BufferCache
is not involved in the lack of memory (5% of total memory)...all i am discussing is correct if GlancePlus is showing rights values and,if so, the total amount of SystemMemory used is 18% of Phys.Mem

On SysTable page of Glance i can see that
the max buffcache is defined to 1.2GB (15% of Phys.Mem.)

Regards

[TwoProc]

OK, I see you've got 1.4G of system memory taken up - and 400 of that is buffer cache. So, I'm going to say its gotta be kernel config.

I've got an 8G test box running over right now and it has 1.5G of Sys Mem used - of which 250M is buffer cache. Leaving me using 1.25 Gig or so for System Memory. Not that much different than yours.

The problem is, I can't think of anything besides the buffer cache (that's kernel related) that would respond to file i/o in a big way with a growing memory footprint.

If you're using HFS (which I doubt), then there are a bunch of kernel params that start with "hfs_*" that have to do with memory - might want to check that stuff out.

... out of ideas ...

[Ted Buis]

Look at all your kernel parameters and see how they vary from the defaults. The good news is that you have 0 page-outs, so the memory usage doesn't create a real problem, at least yet. There are certainly other tables in HP-UX that can grow in size. There is a book by Chris Cooper and Chris Moore on hp-ux 11i internals if you want to read all the details.

[Fabio Acquarone]

Thank you Ted....i am proceeding in your direction from about a week but without any result. HP is helpping me also....

Could be possible that i am in a "normal" situation for HP-UX11i ??

Did you saw a such behaviour?
My issue is that i need to size systems for
our customers and a such behaviour could be
too application independent. I do not want
to have systems with oversized memory to avoid the issue....

Fabio

[Ted Buis]

HP-UX has a number of tables. Some are static and some are dynamic, and grow with usage. The kernel parameter are used to set the size of some of these tables, and some maximum kernel parameter values contain their growth. Larger than necessary values can waste RAM for static tables, while too large values for maximum sizes allow some tables to grow with diminishing returns. For example, the buffer cache can grow to its maximum value if there is no other memory pressure, but shrinking it back takes some time when memory pressure becomes more intense. In the book, "hp-ux 11i tuning and performance" by Robert Sauers, he recommends a fixed size cache for most database environments. Buffer cache over 500MB may be a waste, but it depends. Also, if you have OnLineJFS and are using Oracle, then you want to use mount options to avoid double buffering with Oracle and the buffercache. This not only wastes memory but wastes processesing power with unnecessary memory to memory transfers.

Each "open" system call adds an entry into the system file table. There are buffer cache pages, but there is also an inode/vnode cache in system memory. There is also an I/O tree table. I have seen systems with thousands of LUNs visible via the SAN. Each of these LUNs requires a entry in that table.

HP-UX 11i may require only 512MB to run (or crawl), but HP recommends starting at 2GB of RAM. This indicates to me that HP-UX 11i will page-in what is needed also. If there is no memory pressure, then it will consume more memory. System code does not need to be paged out, since it can always be paged in again when needed. Look for page-outs to determine if you need more RAM.

Applications like Oracle are smart about how much memory the system has and will try to use all of it, so no matter how much you have, you can expect a high utilization number. If you bought it, you may as well use it. Again, monitor page-outs to determine if you need more RAM, but if you have no page-outs, it doesn't imply that having less RAM would not reduce performance. Again, it depends.

[Fabio Acquarone]

Thank you again for your reply,

our application needs two different views on RAM sizing.

The first is related to CORE processes that needs a relative fixed amount of memory.
The second is related to a variable number of USER GUIs that must be present on the same system.

Now, i saw that running CORE processes we have an increasing of SystemMemory and i can't understand the reason for which SystemMemory is increasing so much.
Now the amount of extra system memory used by the O.S. is taking away Free Memory for users....and this is the real issue.

On a 8Gbyte PhysMem system i reached 1.4GBytes of System Memory with only CORE processes. If i add extra memory, is it all extra added memory usable for USER GUIs or it will be part used by system memory??

Related to the Buffer Cache i aggree with you, in the past we have reduced the amount of BufferCache limmiting it to the 15% of the PhysMemory, starting from a 5%.

I add that this behaviour was never highlighted before on previous benchmarking on HP-UX 11.00 and 11i...
..maybe is some added PATCH that emphasize the issue ???
I am still waiting from HP an answer after two weeks.

Regards

[Ted Buis]

Naturally, a patch could affect this, but I haven't searched the patch database. I assume that you have done that already.

The kernel parameter maxfiles controls the maximum size of the per process file descriptor table. Application code can change this with setrlimit(2), within the maxfiles_lim. So depending on the number of processes, this could be a factor.

What is the vps_pagesize?

Does the code use fork when it could use the more efficient vfork?

[Fabio Acquarone]

Tod,

we have following settings

max_filelim 3000
vps_pagesize 4

we use in some cases setrlimit and usually fork instead of vfork (our labs saw this...)

Fabio

[Fabio Acquarone]

Tod,

we have following settings

max_filelim 3000
vps_pagesize 4

we use in some cases setrlimit and usually fork instead of vfork (our labs saw this...)

i checked PATCH databases and i found some patch related to a similar behaviour but no usefull results occured after the installation

Fabio