vhand processing

[David G. Douthitt]

I'm getting frustrated with vhand. I'm periodically seeing it take 15-20% of the CPU and up - with no pageouts (or little pageouts), no deactivations, no physical swap used, and free memory remaining (all taken from Glance metrics).

A lot of the recommendations in the forums seem to be too old and are not relevant to 11i v3: kmtune doesn't exist, using "desfree/D" command with adb results in "Bad address error", disk_sweep_seconds (?) doesn't exist as a tunable, and dbc_cache_max and dbc_cache_min don't exist. (RANT: It should've been called HP-UX 12i; this is nuts....)

Some of this I've resolved (it's now filecache_max and filecache_min for instance).

But why is vhand showing apparent evidence of memory pressure when other metrics show no swappiness and no sluggishness? And in one case, *raising* filecache_max made vhand disappear from the top list of CPU users just like that (unless it was only done just at the right time).

Both systems I've seen this on (Itanium rx4640s) have heavy disk IO (SAN) as a matter of course from database operations (no, not Oracle: Intersystems Caché).

What gives? Or do I just take it as given that vhand using more than (how much?) % CPU means a memory shortage, even with no evidence of processes "fighting" (or thrashing) for space?

[Rita C Workman]

Maybe you hit your own answer:
"...in one case, *raising* filecache_max made vhand disappear from the top list of CPU users just like that (unless it was only done just at the right time)."

The new parm filecache_max description states:
filecache_max Maximum amt of physical memory to be used for caching file I/O data (sounds like dbc_max_)
filecache_min Minimum guaranteed physical memory used for caching file I/O data (sounds dbc_min)

So raising your filecache_max afforded more physical memory for the I/O.

We may all need to watch and see if we need to tweek this one as we go to v11.31.

I'm just now getting into this O/S version on some new boxes - so thanks for the heads up!

Rgrds,
Rita

[James R. Ferguson]

Hi:

Use 'kctune' instead of 'kmtune' to look at your kernel parameters. Using your older'adb' syntax will probably require adding the '-o' switch for backwards compatability to PA-RISC syntax.

What is 'syncer' doing? High activity for this would point to the file cache.

Regards!

...JRF...

[Patrick Wallek]

Check into patch PHKL_38275:
Patch Description: 11.31 vm cumulative patch
http://www12.itrc.hp.com/service/patch/patchDetail.do?patchid=PHKL_38275&sel={hpux:11.31,}&BC=main|search|

One of the problems fixed in an older patch (PHKL_38174) is "vhand monopolizes the CPU, and on a single CPU machine or a vPar, the system can seem like a hang as user applications are starved for CPU cycles."

If you don't have PHKL_38174 or higher already installed, then try PHKL_38275.

Please note that PHKL_38275 has a dependency of PHKL_36261.

[David G. Douthitt]

Both of these machines are up to date with system patches - that is, they have the most recent quality packs installed and have all HP recommended patches (as shown by swainv and the HP patch analyzer).

Maybe I need to add some special patches?

I'm still waiting to see the June 2008 Quality Pack Bundles for HP-UX 11i v3.

[David G. Douthitt]

Just happened again: no swapping, CPU utilization was down, even disk utilization was down (according to the glance bars) - but system CPU usage was *very* high, vhand was running at 15+% of the CPU, and no swappiness.

Increasing filecache_max once again now made vhand disappear once again.

Memory on this 11i v3 Itanium rx4640 is 18G, filecache_max now stands at 4G (with min at 2.5G). Standard operations has been a fixed filecache at 3G.

[Patrick Wallek]

Check and see if you have the patches I referenced above installed. I do not know if they are part of the current quality packs or not.

[David G. Douthitt]

I do have kmeminfo (5.19) and caliper (4.4) both; perhaps one of these can help. I'm still learning about caliper.

[Dennis Handly]

>I'm getting frustrated with vhand. I'm periodically seeing it take 15-20% of the CPU

Do you have anything better for that CPU to be doing?
Have you talked to the Response Center?

[David G. Douthitt]

Haven't spoke to response center; recommended patches are not installed but can be. Since the patches require a reboot, it'll have to wait until later.

Running caliper showed me that a lot of time was being spent in the btree_purge_free() function (whatever that is: sounds like memory freeing algorithms). I searched for that on the web and on the ITRC: nothing.

vhand is running anywhere from 2% to 50% (!) now, and the filecache size dropped back to the reduced filecache_max (memory pressure?).

I still don't understand how vhand can think there is memory pressure when Free Mem is listed at 1.3Gb and the memory utilization bar is at about 75% (whereas the current/avg/high numbers are at 93%).

Frankly, it looks to me as if all of the standard measures of memory are worthless in 11i v3: HP scrambled everything.

[Don Morris_1]

Simple -- vhand manages memory pressure both system-wide and within narrower focuses (i.e. Kernel memory pressure but not system [yes, this is possible -- think about a lot of Floating memory which the kernel doesn't get to use], User but not system... and most especially on v3 -- *filecache*).

Given your description, the filecache was full and vhand was fighting whatever is generating all the cache traffic to get pages back (somewhere there was someone waiting for it). The btree purge would be it cleaning up metadata to represent pages in the filecache after freeing other pages there.

You don't see pageouts if the pages aren't dirty, for one... vhand has to just find the pages, determine they're clean and toss them back to the system.. no I/O required.

In any event - if you think vhand is running too much, get with support. There's always a possibility that you've hit an issue that needs to be worked. There's also a possibility that your workload simply generates cache thrashing and would benefit from a bigger filecache_max. (From your description of heavy non-Oracle disk I/O, I suspect this is being done as normal read/writes... which go through the filecache). And there's really no way you can triage this sort of thing on a web forum... you have to really see what vhand is doing relative to the memory pressure status / scan rates / etc.

As far as your other comments -- desfree doesn't exist as a single global variable anyway (and hasn't since 11.11!) Don't try to tweak it via adb, you're more likely to tromp on something beside the old variable that just is there to make reporting easier.

kmtune hasn't existed since 11.22 (or .23 -- I think .22, though). kctune replaced it.

dbc* as you know is now filecache_min / filecache_max.

[Don Morris_1]

Out of pure curiosity (since you mention you don't have the recommended patches)... what does "what /stand/vmunix |grep vm" report (yeah, there will be LVM in there as well, but I'd like to know if you're at least at 11.31.0803 or so).

[Rasheed Tamton]

vhand is related with memory bottle neck. As it is a high priority process, it will always use high CPU time. It is very strange you are not seeing any POs, deactivations.

The main kernel parameters related to vhand are lotsfree, desfree, and minfree.

[Rasheed Tamton]

Here is a good doc. detailing the performance issues on 11iV3. vhand is explained also.

http://h20219.www2.hp.com/hpux11i/downloads/perf%20troubleshooting%20trans.pdf

[David G. Douthitt]

The link to the Performance Troubleshooting Knowledge on Demand (http://www.hp.com/go/kod) was fantastic; I also got the slides to go with the transcript.

From looking at things, I'd just about say that this was a vhand bug; VM writes are down, Free Memory (listed by Glance in plain sight in the Memory screen) is 1.5-2Gb, and the only disk activity done by vhand is a minimal set of physical disk writes (not VM). Swap is untouched.

Raising filecache_max seems to improve things briefly; perhaps that's because vhand has nothing to do (?) and the bug - if there is one - only surfaces when vhand is busy.

The other thing is that the memory pressure in evidence may be in the DB itself; but that's another story (and hopefully now taken care of).

[Don Morris_1]

It may well be... hence why I'd like to know your patch level.

[David G. Douthitt]

There is a lot of output from your request (197 lines). I've attached it.

There's reference to B.11.31_LR FLAVOR=perf all over; is that what you were looking for?

[Don Morris_1]

One aspect of it, yeah. The rest is the PHKL_36* stuff on vm and vm_fs in particular.

Ok... so your VM patches are:

General VM as a whole: 11.31 LR
vm_asi (Address Space Interfaces): PHKL_36054
vm_fs (VM/FileSystems): PHKL_36242
vm_kmem (Kernel memory): 11.31 LR
vm_pmem (Physical memory): PHKL_35900
vm_resm (Resource Management): 11.31 LR
vm_swap (Swap): 11.31 LR
vm_xlat (Translation layer): PHKL_36009

Yeah, you're a little behind, I'm afraid... and there were some critical vhand/UFC tweaks (unfortunately, I know there's one more coming in that space but it affects uniprocessor small memory boxes more).

Scanning over some of them, my gut is that you've hit QXCR1000716656 in fact. One symptom there is when there's filecache but not system wide pressure, vhand can spend way too much time trying to prune the VM metadata affiliated with the file object (the btree pruning), wasting a bunch of cycles on work that isn't needed. [There were other pruning inefficiencies addressed there, but they had more to do with doing a better job when there actually was system pressure and we needed the kernel memory used for these structures back].

As such, at a bare minimum I'd recommend you move to PHKL_37452 as soon as possible (that's a 2 star rated patch from 11.31.0803 set, so fairly well aged and all. PHKL_38243 would be the current absolute latest, but it just came out so I wouldn't at all be surprised if you want to let it wait. It might help your performance if Intersystems Cache chatr's their binary to use large Text pages, but otherwise the only real updates are for systems under heavy system wide pressure, which you're not.