topic Re: filecache Utilization is very high on RX2660 in Operating System - HP-UX

Buffer Cache Utilization is very high on RX2660

Shibin_2 — Mon, 03 Dec 2012 07:26:41 GMT

I have RX2660 server HP-UX Base OE ( 11.31 ); so no glance software available.

I have noticed that VM utilization is always 84% and above, but swap and memory utilization is below 30%.

             Mb      Mb      Mb   PCT START/      Mb
TYPE      AVAIL    USED    FREE USED   LIMIT RESERVE PRI NAME
dev        8192     210    7982    3%       0       -    1 /dev/vg00/lvol2
reserve       -     866    -866
memory    11667    3523    8144   30%
total     19859    4599   15260   23%       -       0    -

Below output taken on 1st of October.

Date | Time |/Intvl | Hit % | Mem %| Swap %| VM % |

10-01-12 |13:10:00| 300.0| 100.00| 30.75| 41.00| 83.90|

10-01-12 |13:15:00| 300.8| 100.00| 30.75| 41.00| 84.38|

10-01-12 |13:20:00| 300.0| 100.00| 30.75| 41.00| 84.23|

10-01-12 |13:25:00| 299.3| 100.00| 30.75| 41.00| 83.95|

10-01-12 |13:30:00| 300.0| 100.00| 30.75| 41.00| 84.19|

This system is using as Data Protector Cell server. System has 12GB memory with 8GB Swap space configured.

What could be the problem? What's the solution? Appreciate your valuable inputs.

Re: filecache Utilization is very high on RX2660

Dennis Handly — Mon, 03 Dec 2012 19:05:07 GMT

(I assume you mean filecache for 11.31.)

>Below output taken on 1st of October.

What produced this output? And did the swapinfo occur at the same time?

>What could be the problem?

Why do you think there is a problem? Something not working or slow with the system?

Re: filecache Utilization is very high on RX2660

Shibin_2 — Wed, 05 Dec 2012 14:54:12 GMT

Hi Dennis,

Thanks for the response. Yes. This is HP-UX 11.31 BASE OE ( no glance available ) with Sept 2010 patch level.

The given report is taken from perf ( using OVPA extract tool ). As you can see here, Cache memory and Virtual memory in its peak and the monthly performance report is ( graph ) is giving very bad report.

Any possible solution ? I am thinking a patch level update; but I do not have relevent evidence to convey that patch update will resolve the issue.

Thanks in advance.

Re: filecache Utilization is very high on RX2660

Dennis Handly — Wed, 05 Dec 2012 16:30:34 GMT

>monthly performance report is (graph) is giving very bad report.

Again, why do you think this is bad?

Re: filecache Utilization is very high on RX2660

Shibin_2 — Wed, 05 Dec 2012 17:12:33 GMT

It's DP cell server. When all backup trigger during night time ( server time is CET ), Data Protector GUI becomes crawling.

That's the simplest way I can explain.

Re: filecache Utilization is very high on RX2660

Dennis Handly — Wed, 05 Dec 2012 18:38:49 GMT

>Data Protector GUI becomes crawling. That's the simplest way I can explain.

That's, that's more useful info.

Anything else running on the server? You may need more memory.

Can you get the vmstat(1) output? Are the po/pi values high?

Re: filecache Utilization is very high on RX2660

Shibin_2 — Thu, 06 Dec 2012 08:56:14 GMT

Paging is very low level all the time.

         procs           memory                   page                              faults       cpu
    r     b     w      avm    free   re   at    pi   po    fr   de    sr     in     sy    cs us sy id
    2     0     0   325629 497473    0    0     7    1     0    0     0 37262 22620 8316   9 3 88
    2     0     0   325629 497242    0    0     4    0     0    0     0   1605 31237   272 20 3 77
    2     0     0   334296 497233    0    0     2    0     0    0     0   1594 32423   338 27 1 72
    2     0     0   334296 497217    0    0     0    0     0    0     0   1599 25125   344 27 0 73
    2     0     0   334296 497265    0    0     0    0     0    0     0   1621 19222   339 11 0 89

It's very strange, I think, and just virtual memory is high. There are no other applicaiton installed other than Data Protector and HP OVO tools.

I am going to extract another report for this month to see the activity. System has currently 12GB memory and suggestions came to add another 8GB. However, I need to re-consider that option because if additional memory is not giving expected output, then the expenditure made will be in question.

Re: filecache Utilization is very high on RX2660

Dennis Handly — Sat, 02 Feb 2013 13:06:54 GMT

>Paging is very low level all the time.

Then it isn't a memory bottleneck.

>Data Protector GUI becomes crawling

What's crawling? GUI updates?

Or backup progress? If the latter, it could be I/O or networking bottlenecks?

>This system is using as Data Protector Cell server.

What exactly does a Cell server do? (I.e. What resources does it use/need?)

The source (disks) to backup?

Something to collect data from other systems?

>because if additional memory is not giving expected output, then the expenditure made will be in question.

Here is where glance is important. Do you have the trial version available?

Re: filecache Utilization is very high on RX2660

Shibin_2 — Fri, 07 Dec 2012 06:54:01 GMT

Thanks for your response. I have diverted my investigation to application level and found these lines in Omni debug.log file.

ipw_getaddrinfo(servername.domain.com) Network performance warning: took 15.000000 sec
ipw_getaddrinfo(servername.domain.com) Network performance warning: took 15.000000 sec
ipw_getaddrinfo(localhost) Network performance warning: took 20.000000 sec

Note: servername.domain.com - used here to hide original name.

Now, it seems more related to Network issue.

But why just VM is above 85% and all other at normal ?

Re: filecache Utilization is very high on RX2660

Dennis Handly — Fri, 07 Dec 2012 08:11:12 GMT

>it seems more related to Network issue.

Yes.

>But why just VM is above 85% and all other at normal?

I think the title is "Active VM %", which I'm not sure what it means. If it means "used", I'm not sure why it didn't say that.

I do find "Active VM" on a glance output in:

The HP-UX 11i v3 Memory Management Subsystem

h20000.www2.hp.com/bc/docs/support/SupportManual/c02465072/c02465072.pdf

Page 4 seems to imply that "active' means "belonging to a process that is running or has run in the last 20 seconds".

This could mean that 85% of the "USED" VM is active?

Re: filecache Utilization is very high on RX2660

Shibin_2 — Sun, 16 Dec 2012 04:20:23 GMT

Apologize for the delayed response.

Here we go with precise output on buffer cache. You might be noticed the "%rcache" is always 100%.

<servername># sar -b 5 10

HP-UX <servername> B.11.31 U ia64 12/16/12

05:16:54 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
05:16:59       0      42     100       2      30      93       0       0
05:17:04       0      26     100       6      22      73       7       0
05:17:09       0       7     100       1       7      85       0       0
05:17:14       0      17     100       6      13      58       0       0
05:17:19       0       6     100       1       5      78       0       0
05:17:24       0      15     100       1       8      93       0       0
05:17:29       0      40     100       0      28      99       0       0
05:17:34       0       7     100       1       5      85       0       0
05:17:39       0       6     100       1       5      72       0       0
05:17:44       0      11     100       3      10      67       0       0

Average 0 18 100 2 13 84 1 0

Re: filecache Utilization is very high on RX2660

Dennis Handly — Sun, 16 Dec 2012 10:45:14 GMT

>You might be noticed the "%rcache" is always 100%.

Read cache hits is a good thing.

Re: filecache Utilization is very high on RX2660

Shibin_2 — Fri, 01 Feb 2013 14:48:51 GMT

Again, apologize for delayed response.

So with that we can conclude there are no issues at platform level. correct?

Re: filecache Utilization is very high on RX2660

Dennis Handly — Sat, 02 Feb 2013 13:08:09 GMT

>So with that we can conclude there are no issues at platform level.

Or you need a tool like glance to find the bottlenecks.

Re: filecache Utilization is very high on RX2660

Bill Hassell — Mon, 04 Feb 2013 18:58:16 GMT

>> ipw_getaddrinfo(servername.domain.com) Network performance warning: took 15.000000 sec
>> ipw_getaddrinfo(servername.domain.com) Network performance warning: took 15.000000 sec
>> ipw_getaddrinfo(localhost) Network performance warning: took 20.000000 sec

This looks like a DNS issue (as opposed to a network issue). getaddrinfo is a hostname or IP address query and follows the rules setup in nsswitch.conf as to where to obtain such information. Except for an occasional login or application query, these occur infrequently. But for reasons that escape me, many commercial backup programs insist on asking about the hostname over and over again, at least once for every file or directory being backed up. This could hundreds of requests per second when backing up small files, and a massive load on the DNS server.

Since rewriting the backup software is not an option, you can change the name resolver rules in nsswitch.conf to use the local /etc/hosts file with the hostname of all of your backup clients. Then change nsswitch.conf to use files first, then DNS. In doing so, the hosts file will be immediately cached and available from memory, way faster than a network query.

You GUI slowdown may also be related to an overloaded DNS server caused by the backup software.

Re: filecache Utilization is very high on RX2660

Matti_Kurkela — Mon, 04 Feb 2013 19:44:10 GMT

If your environment contains a large number of hosts, and their names are already in DNS, you might consider setting up a "slave" DNS server on your backup server. The advantage is that once configured, it will automatically replicate all the information from your main DNS server - you don't have to keep the /etc/hosts file updated manually.

Such a slave DNS server does not need to be public (i.e. you don't have to indicate it in NS records for your domain(s)), and it can be dedicated to answer local queries only. This is sometimes called a "stealth-slave" DNS server configuration.

For the slave DNS server to work, the administrator of your master DNS server needs to allow zone transfer requests from the IP address of your new slave server, and also configure the master DNS server to send notify messages whenever data is updated on the master server. (If the slave server is listed in the NS records of a DNS zone, the notify messages may be sent automatically; but if you set up a stealth-slave, the notifications may have to be configured explicitly.)

A stealth-slave DNS server that is dedicated to providing information to localhost only takes a minimal amount of CPU power and RAM.

A stealth-slave may also be useful if your main DNS server does not like IPv6 address queries. The ipw_getaddrinfo() function is one of the new-style DNS resolver functions, and it is IP protocol version agnostic: unless the program specifies that it wants only IPv4 addresses, the function may query the DNS server for both types. Some older DNS servers will have problems with such combined queries. If you use e.g. a modern version of ISC BIND as your slave DNS server, this will not be an issue.