Buffer Cache Utilization is very high on RX2660

[Shibin_2]

I have RX2660 server HP-UX Base OE ( 11.31 ); so no glance software available.

I have noticed that VM utilization is always 84% and above, but swap and memory utilization is below 30%.

             Mb      Mb      Mb   PCT  START/      Mb
TYPE      AVAIL    USED    FREE  USED   LIMIT RESERVE  PRI  NAME
dev        8192     210    7982    3%       0       -    1  /dev/vg00/lvol2
reserve       -     866    -866
memory    11667    3523    8144   30%
total     19859    4599   15260   23%       -       0    -

Below output taken on 1st of October.

      |          |      Sec  |Cache Mem   |     User |           |    Active |

Date        |  Time  |/Intvl |      Hit % |     Mem %|     Swap %|     VM %  |

10-01-12    |13:10:00|  300.0|      100.00|     30.75|      41.00|      83.90|

10-01-12    |13:15:00|  300.8|      100.00|     30.75|      41.00|      84.38|

10-01-12    |13:20:00|  300.0|      100.00|     30.75|      41.00|      84.23|

10-01-12    |13:25:00|  299.3|      100.00|     30.75|      41.00|      83.95|

10-01-12    |13:30:00|  300.0|      100.00|     30.75|      41.00|      84.19|

This system is using as Data Protector Cell server.    System has 12GB memory with 8GB Swap space configured. 

What could be the problem?  What's the solution?  Appreciate your valuable inputs.

[Dennis Handly]

(I assume you mean filecache for 11.31.)

>Below output taken on 1st of October.

What produced this output?  And did the swapinfo occur at the same time?

>What could be the problem?

Why do you think there is a problem?  Something not working or slow with the system?

[Shibin_2]

Hi Dennis,

Thanks for the response.  Yes.  This is HP-UX 11.31 BASE OE ( no glance available ) with Sept 2010 patch level.

The given report is taken from perf ( using OVPA extract tool ).   As you can see here, Cache memory and Virtual memory in its peak and the monthly performance report is ( graph ) is giving very bad report.

Any possible solution ?  I am thinking a patch level update; but I do not have relevent evidence to convey that patch update will resolve the issue.

Thanks in advance.

[Dennis Handly]

>monthly performance report is (graph) is giving very bad report.

Again, why do you think this is bad?

[Shibin_2]

It's DP cell server.   When all backup trigger during night time ( server time is CET ), Data Protector GUI becomes crawling. 

That's the simplest way I can explain.

[Dennis Handly]

>Data Protector GUI becomes crawling.   That's the simplest way I can explain.

That's, that's more useful info.

Anything else running on the server?  You may need more memory.

Can you get the vmstat(1) output?  Are the po/pi values high?

[Shibin_2]

Paging is very low level all the time. 

         procs           memory                   page                              faults       cpu
    r     b     w      avm    free   re   at    pi   po    fr   de    sr     in     sy    cs  us sy id
    2     0     0   325629  497473    0    0     7    1     0    0     0  37262  22620  8316   9  3 88
    2     0     0   325629  497242    0    0     4    0     0    0     0   1605  31237   272  20  3 77
    2     0     0   334296  497233    0    0     2    0     0    0     0   1594  32423   338  27  1 72
    2     0     0   334296  497217    0    0     0    0     0    0     0   1599  25125   344  27  0 73
    2     0     0   334296  497265    0    0     0    0     0    0     0   1621  19222   339  11  0 89

It's very strange, I think, and just virtual memory is high.  There are no other applicaiton installed other than Data Protector and HP OVO tools.

I am going to extract another report for this month to see the activity.  System has currently 12GB memory and suggestions came to add another 8GB.  However, I need to re-consider that option because if additional memory is not giving expected output, then the expenditure made will be in question.

[Dennis Handly]

>Paging is very low level all the time.

Then it isn't a memory bottleneck.

>Data Protector GUI becomes crawling

What's crawling?  GUI updates?

Or backup progress?  If the latter, it could be I/O or networking bottlenecks?

>This system is using as Data Protector Cell server.

What exactly does a Cell server do?  (I.e. What resources does it use/need?)

The source (disks) to backup?

Something to collect data from other systems?

>because if additional memory is not giving expected output, then the expenditure made will be in question.

Here is where glance is important.  Do you have the trial version available?

[Shibin_2]

Thanks for your response.  I have diverted my investigation to application level and found these lines in Omni debug.log file.

ipw_getaddrinfo(servername.domain.com) Network performance warning: took 15.000000 sec
ipw_getaddrinfo(servername.domain.com) Network performance warning: took 15.000000 sec
ipw_getaddrinfo(localhost) Network performance warning: took 20.000000 sec

Note: servername.domain.com - used here to hide original name.

Now, it seems more related to Network issue. 

But why just VM is above 85% and all other at normal ?

[Dennis Handly]

>it seems more related to Network issue.

Yes.

>But why just VM is above 85% and all other at normal?

I think the title is "Active VM %", which I'm not sure what it means.  If it means "used", I'm not sure why it didn't say that.

I do find "Active VM" on a glance output in:

The HP-UX 11i v3 Memory Management Subsystem

h20000.www2.hp.com/bc/docs/support/SupportManual/c02465072/c02465072.pdf

Page 4 seems to imply that "active' means "belonging to a process that is running or has run in the last 20 seconds".

This could mean that 85% of the "USED" VM is active?

[Shibin_2]

Apologize for the delayed response. 

Here we go with precise output on buffer cache.  You might be noticed the "%rcache" is always 100%.

<servername># sar -b 5 10

HP-UX <servername> B.11.31 U ia64    12/16/12

05:16:54 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
05:16:59       0      42     100       2      30      93       0       0
05:17:04       0      26     100       6      22      73       7       0
05:17:09       0       7     100       1       7      85       0       0
05:17:14       0      17     100       6      13      58       0       0
05:17:19       0       6     100       1       5      78       0       0
05:17:24       0      15     100       1       8      93       0       0
05:17:29       0      40     100       0      28      99       0       0
05:17:34       0       7     100       1       5      85       0       0
05:17:39       0       6     100       1       5      72       0       0
05:17:44       0      11     100       3      10      67       0       0

Average        0      18     100       2      13      84       1       0

[Dennis Handly]

>You might be noticed the "%rcache" is always 100%.

Read cache hits is a good thing.

[Shibin_2]

Again, apologize for delayed response. 

So with that we can conclude there are no issues at platform level.  correct?

[Dennis Handly]

>So with that we can conclude there are no issues at platform level.

Or you need a tool like glance to find the bottlenecks.

[Bill Hassell]

>> ipw_getaddrinfo(servername.domain.com) Network performance warning: took 15.000000 sec
>> ipw_getaddrinfo(servername.domain.com) Network performance warning: took 15.000000 sec
>> ipw_getaddrinfo(localhost) Network performance warning: took 20.000000 sec

This looks like a DNS issue (as opposed to a network issue).  getaddrinfo is a hostname or IP address query and follows the rules setup in nsswitch.conf as to where to obtain such information. Except for an occasional login or application query, these occur infrequently. But for reasons that escape me, many commercial backup programs insist on asking about the hostname over and over again, at least once for every file or directory being backed up. This could hundreds of requests per second when backing up small files, and a massive load on the DNS server.

Since rewriting the backup software is not an option, you can change the name resolver rules in nsswitch.conf to use the local /etc/hosts file with the hostname of all of your backup clients. Then change nsswitch.conf to use files first, then DNS. In doing so, the hosts file will be immediately cached and available from memory, way faster than a network query.

You GUI slowdown may also be related to an overloaded DNS server caused by the backup software.

[Matti_Kurkela]

If your environment contains a large number of hosts, and their names are already in DNS, you might consider setting up a "slave" DNS server on your backup server. The advantage is that once configured, it will automatically replicate all the information from your main DNS server - you don't have to keep the /etc/hosts file updated manually.

Such a slave DNS server does not need to be public (i.e. you don't have to indicate it in NS records for your domain(s)), and it can be dedicated to answer local queries only. This is sometimes called a "stealth-slave" DNS server configuration.

For the slave DNS server to work, the administrator of your master DNS server needs to allow zone transfer requests from the IP address of your new slave server, and also configure the master DNS server to send notify messages whenever data is updated on the master server. (If the slave server is listed in the NS records of a DNS zone, the notify messages may be sent automatically; but if you set up a stealth-slave, the notifications may have to be configured explicitly.)

A stealth-slave DNS server that is dedicated to providing information to localhost only takes a minimal amount of CPU power and RAM.

A stealth-slave may also be useful if your main DNS server does not like IPv6 address queries. The ipw_getaddrinfo() function is one of the new-style DNS resolver functions, and it is IP protocol version agnostic: unless the program specifies that it wants only IPv4 addresses, the function may query the DNS server for both types. Some older DNS servers will have problems with such combined queries. If you use e.g. a modern version of ISC BIND as your slave DNS server, this will not be an issue.