System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

vhand starting up after backups kicks in

PatRoy
Regular Advisor

vhand starting up after backups kicks in

G'day!

Running HPUX 11.31. Backup software used is Dataprotector 6.0. Cell Manager is on MS Windows.

The system is used has a database server. Running Sybase 15. It as 48Gbs, where 36 are allocated to Sybase (through the use of mlock). Problem we have noticed is that when ever a backup starts, vhand will show up in the picture, using nearly 100% out of 800% of CPU. I'm "almost" sure the CPU utilization is normal for vhand.. it's still almost idle, since we've got 8 cores. Plus, vhand is just doing it's job right!? Problem is that vhand showing up tells me the system is paging out, causing system to be slow! (i.e. slow logins, lags, etc.)

I've got 48gbs of physical swap devices setup...

swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 10240 156 10084 2% 0 - 1 /dev/vg00/lvol2
dev 38912 156 38756 0% 0 - 1 /dev/vg01/swap
reserve - 35632 -35632
memory 46720 43827 2893 94%
total 95872 79771 16101 83% - 0 -


Now, one fix I found for this was to decrease the default filecache_max from 50% all the way down to 5% only! That forced the system to free up memory, which prevented paging out.

For one thing, isn't filecache dynamic? Why wouldn't it just decrease has DP would possibly need more memory? And DP needing a lot of memory??? Didn't think DP was a big memory eater!! Can can anyone clarify...

If reducing my caching is my solution, is 5% an appropriate number for a database server? That's mostly used for web server, NFS servers, etc.. right? Which we aren't using.

Thanks. Patrick.
6 REPLIES
PatRoy
Regular Advisor

Re: vhand starting up after backups kicks in

Take note that the swapinfo capture in my post isn't the one from when the backup was running... Just wanted to output swap setup...
Don Morris_1
Honored Contributor

Re: vhand starting up after backups kicks in

The filecache is dynamic if min is less than max -- but the important thing here is that *vhand* is what shrinks the filecache in v3. Hence the pageouts from vhand were very likely vhand pushing out filecache pages.

Is the system _really_ slow or you just assuming that because there are pageouts that it is?

Backup and the filecache is interesting because some backups use direct I/O, some don't, some look sequential, some don't, etc.... there have been issues raised in this space before (so the first thing I'd check is if you're up-to-date on patch levels). That said -- usually the DB programs operate with direct I/O on many levels, so the cache doesn't get used. Running with it at 5% is likely fine for your box and is worth monitoring -- if the performance stays acceptable, leave it there.
James R. Ferguson
Acclaimed Contributor

Re: vhand starting up after backups kicks in

Hi Patrick:

I'm not surprised that vhand() runs. You have memory pressure. You are swapping and have very little free memory.

The filecache is dynamic but you have allowed it to potentially grow quite large. You have already found that lowering it helps. I suggest you tune it down; evaluate swap activity with 'vmstat'; evaluate the hit ratio for your filesystem cache with 'sar' and begin to see where a sweeter spot resides. Certainly adding more physical memory would be desirable.

Do not equate the values for 'filecache_min' and 'filecache_max' directly to the values you used for 'dbc_min_pct' and 'dbc_max_pct'. Instead, measure and evaluate in your environment.

Regards!

...JRF...
PatRoy
Regular Advisor

Re: vhand starting up after backups kicks in

Don: yeah the system did come to be slow. It would kinda hang / lag sometimes.. logins would take a long time, etc. As for OS patches, it's pretty up-to-date.. as of end of march 2009. I'll need to check with the backup (dataprotector) guys to see if there's anything...

DP obviously seems to be doing "some" filecaching... has when it's not doing any backups, it goes back down to only about 1Gb... when DP runs, goes up to about 4Gb.. (before having reduced filecache_max). I was pretty sure DP was just straight Disk I/O... but then again, I don't know dataprotector.


James: my settings for min and max were 5% (min) and 50% (max), which were the defaults I believe. I've never used the 'sar' command. I'm guessing I would just use it with 'sar -b' to report buffer activity. I keep getting the following error when trying to run it:

sar: Can't open /var/adm/sa/sa14

How would I use this and what should I be looking at?

Pat (*UX junior Admin ;)




Patrick Wallek
Honored Contributor

Re: vhand starting up after backups kicks in

Hi Patrick,

You must specify some timing information for sar.

For example:

# sar -b 5 4

will post the results of 'sar -b' every 5 seconds, for 4 iterations. The first number is the delay in seconds between queries and the 2nd number is the number of time to run the query.

If you want the results once a minute for 30 minutes:

# sar -b 60 30

(60 = 60 second delay between results ; 30 = number of iterations)
James R. Ferguson
Acclaimed Contributor

Re: vhand starting up after backups kicks in

Hi (again) Patrick:

The problem with your 'sar' command is that it is expecting to open a previously collected file of data. You can do:

# sar -b 5 10

...for example, which will report buffer activity every 5-seconds for a total of 10-intervals. The "%rcache" column is the buffre read-hit ratio. See the manpages for more information.

Regards!

...JRF...