filecache Utilization is very high on RX2660

[Shibin_2]

Apologize for the delayed response. 

Here we go with precise output on buffer cache.  You might be noticed the "%rcache" is always 100%.

<servername># sar -b 5 10

HP-UX <servername> B.11.31 U ia64    12/16/12

05:16:54 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
05:16:59       0      42     100       2      30      93       0       0
05:17:04       0      26     100       6      22      73       7       0
05:17:09       0       7     100       1       7      85       0       0
05:17:14       0      17     100       6      13      58       0       0
05:17:19       0       6     100       1       5      78       0       0
05:17:24       0      15     100       1       8      93       0       0
05:17:29       0      40     100       0      28      99       0       0
05:17:34       0       7     100       1       5      85       0       0
05:17:39       0       6     100       1       5      72       0       0
05:17:44       0      11     100       3      10      67       0       0

Average        0      18     100       2      13      84       1       0

[Dennis Handly]

>You might be noticed the "%rcache" is always 100%.

Read cache hits is a good thing.

[Shibin_2]

Again, apologize for delayed response. 

So with that we can conclude there are no issues at platform level.  correct?

[Dennis Handly]

>So with that we can conclude there are no issues at platform level.

Or you need a tool like glance to find the bottlenecks.

[Bill Hassell]

>> ipw_getaddrinfo(servername.domain.com) Network performance warning: took 15.000000 sec
>> ipw_getaddrinfo(servername.domain.com) Network performance warning: took 15.000000 sec
>> ipw_getaddrinfo(localhost) Network performance warning: took 20.000000 sec

This looks like a DNS issue (as opposed to a network issue).  getaddrinfo is a hostname or IP address query and follows the rules setup in nsswitch.conf as to where to obtain such information. Except for an occasional login or application query, these occur infrequently. But for reasons that escape me, many commercial backup programs insist on asking about the hostname over and over again, at least once for every file or directory being backed up. This could hundreds of requests per second when backing up small files, and a massive load on the DNS server.

Since rewriting the backup software is not an option, you can change the name resolver rules in nsswitch.conf to use the local /etc/hosts file with the hostname of all of your backup clients. Then change nsswitch.conf to use files first, then DNS. In doing so, the hosts file will be immediately cached and available from memory, way faster than a network query.

You GUI slowdown may also be related to an overloaded DNS server caused by the backup software.

[Matti_Kurkela]

If your environment contains a large number of hosts, and their names are already in DNS, you might consider setting up a "slave" DNS server on your backup server. The advantage is that once configured, it will automatically replicate all the information from your main DNS server - you don't have to keep the /etc/hosts file updated manually.

Such a slave DNS server does not need to be public (i.e. you don't have to indicate it in NS records for your domain(s)), and it can be dedicated to answer local queries only. This is sometimes called a "stealth-slave" DNS server configuration.

For the slave DNS server to work, the administrator of your master DNS server needs to allow zone transfer requests from the IP address of your new slave server, and also configure the master DNS server to send notify messages whenever data is updated on the master server. (If the slave server is listed in the NS records of a DNS zone, the notify messages may be sent automatically; but if you set up a stealth-slave, the notifications may have to be configured explicitly.)

A stealth-slave DNS server that is dedicated to providing information to localhost only takes a minimal amount of CPU power and RAM.

A stealth-slave may also be useful if your main DNS server does not like IPv6 address queries. The ipw_getaddrinfo() function is one of the new-style DNS resolver functions, and it is IP protocol version agnostic: unless the program specifies that it wants only IPv4 addresses, the function may query the DNS server for both types. Some older DNS servers will have problems with such combined queries. If you use e.g. a modern version of ISC BIND as your slave DNS server, this will not be an issue.