topic Re: vhand process is taking more CPU. in Operating System - HP-UX

vhand process is taking more CPU.

gb karki — Fri, 10 Apr 2009 03:27:35 GMT

Hi,

We have newly implemented rx2660 servers with 11i.v3 HP-UX OS after that most of the time vhand process is taking 10-15 % CPU.

Same time i have checked physical memory is showing 100% full through "kmeminfo" command.

This system is having 16 GB memory but oracle is taking 11.2 GB memory only.

please suggest me.

Regards
Karki

Re: vhand process is taking more CPU.

Venkatesh BL — Fri, 10 Apr 2009 03:46:05 GMT

how many cpus are there?
can you post the o/p of 'kmeminfo', 'kctune filecache_min filecache_max' and 'swapinfo -tam'?

Re: vhand process is taking more CPU.

Ganesan R — Fri, 10 Apr 2009 04:07:13 GMT

Hi Karki,

If vhand daemon is activated, system is running out of physical memory. vhand daemon is responsible for moving some process out to swap area and give that memory for new process again bring back the process to memory area.

You need to check the memory usage. If application really needs that much memory then you should consider to upgrade the memory.

Re: vhand process is taking more CPU.

gb karki — Fri, 10 Apr 2009 04:09:58 GMT

Hi,

Please find O/P.

----------------------------------------------------------------------
Physical memory usage summary (in page/byte/percent):

Physical memory = 4188969 16.0g 100%
Free memory = 15092 59.0m 0%
User processes = 2998648 11.4g 72% details with -user
System = 962245 3.7g 23%
Kernel = 962225 3.7g 23% kernel text and data
Dynamic Arenas = 624325 2.4g 15% details with -arena
FCACHE_ARENA = 61280 239.4m 1%
reg_fixed_arena = 51126 199.7m 1%
vx_global_kmcac = 50684 198.0m 1%
vx_inode_kmcach = 47664 186.2m 1%
misc region are = 46762 182.7m 1%
Other arenas = 366809 1.4g 9% details with -arena
Super page pool = 8700 34.0m 0% details with -kas
Static Tables = 262456 1.0g 6% details with -static
pfdat = 204539 799.0m 5%
vhpt = 32768 128.0m 1%
text = 10179 39.8m 0% vmunix text section
inode = 7616 29.8m 0%
bss = 5145 20.1m 0% vmunix bss section
Other tables = 2208 8.6m 0% details with -static
Buffer cache = 20 80.0k 0% details with -bufcache
UFC file mrg = 198777 776.5m 5%
tool: kmeminfo 8.09 - libp4 9.339 - libhpux 1.223 - HP CONFIDENTIAL
unix: /stand/current/vmunix 11.31 64bit IA64 on host "NDA-HCLT-RQ01"
core: /dev/kmem live
link: Fri Sep 26 20:08:16 IST 2008
boot: Fri Mar 13 06:13:25 2009
time: Wed Apr 8 17:49:08 2009
nbpg: 4096 bytes

----------------------------------------------------------------------
Summary of processes memory usage:

List sorted by physical size, in pages/bytes:

virtual physical swap
pid ppid pages / bytes pages / bytes pages / bytes command
10652 2338 3424055 13.1g 65643 256.4m 124132 484.9m disp+work
10817 2338 3424055 13.1g 57129 223.2m 123986 484.3m disp+work
24618 2338 3444666 13.1g 50554 197.5m 144813 565.7m disp+work
26365 2338 3436474 13.1g 50173 196.0m 136572 533.5m disp+work
24615 2338 3440570 13.1g 49401 193.0m 140688 549.6m disp+work
4161 2338 3436474 13.1g 49023 191.5m 136628 533.7m disp+work
24616 2338 3432378 13.1g 48270 188.6m 132387 517.1m disp+work
17964 2338 3436474 13.1g 47733 186.5m 136596 533.6m disp+work
7293 2338 3436474 13.1g 47385 185.1m 136580 533.5m disp+work
24798 2338 3432378 13.1g 47165 184.2m 132391 517.2m disp+work
26445 2338 3432378 13.1g 46695 182.4m 132455 517.4m disp+work
17155 2338 3428282 13.1g 46679 182.3m 128339 501.3m disp+work
27727 2338 3424055 13.1g 46533 181.8m 124148 485.0m disp+work
24619 2338 3428282 13.1g 46296 180.8m 128335 501.3m disp+work
5924 2338 3432378 13.1g 46191 180.4m 132451 517.4m disp+work
4163 2338 3432378 13.1g 46089 180.0m 132455 517.4m disp+work
2388 2338 3436343 13.1g 46043 179.9m 136590 533.6m disp+work
28902 2338 3428282 13.1g 45955 179.5m 128343 501.3m disp+work
2391 2338 3424055 13.1g 45723 178.6m 124212 485.2m disp+work
7954 2338 3428282 13.1g 45706 178.5m 128255 501.0m disp+work
9916 2338 3428282 13.1g 45147 176.4m 128275 501.1m disp+work
9749 2338 3428282 13.1g 44954 175.6m 128319 501.2m disp+work
24709 2338 3432378 13.1g 44787 174.9m 132459 517.4m disp+work
24252 2338 3436474 13.1g 44760 174.8m 136568 533.5m disp+work
2392 2338 3436343 13.1g 44084 172.2m 136562 533.4m disp+work
10130 2338 3424055 13.1g 43951 171.7m 124132 484.9m disp+work
24789 2338 3424186 13.1g 43800 171.1m 124218 485.2m disp+work
10498 2338 3424055 13.1g 43710 170.7m 124196 485.1m disp+work
2394 2338 3424055 13.1g 43473 169.8m 124148 485.0m disp+work
10159 2338 3424186 13.1g 42758 167.0m 124158 485.0m disp+work
25029 2338 3424055 13.1g 42421 165.7m 124148 485.0m disp+work
2395 2338 3419959 13.0g 40371 157.7m 119968 468.6m disp+work
2380 2338 3419959 13.0g 40053 156.5m 119870 468.2m disp+work
2374 2338 3432247 13.1g 40048 156.4m 132301 516.8m disp+work
2376 2338 3419959 13.0g 40035 156.4m 119952 468.6m disp+work
2375 2338 3424055 13.1g 40018 156.3m 124068 484.6m disp+work
2538 2338 3419959 13.0g 40014 156.3m 119952 468.6m disp+work
2533 2338 3419959 13.0g 40013 156.3m 119952 468.6m disp+work
2377 2338 3419959 13.0g 40013 156.3m 119952 468.6m disp+work
2535 2338 3424055 13.1g 40011 156.3m 124068 484.6m disp+work
2338 2284 2289174 8.7g 37817 147.7m 91301 356.6m disp+work
2494 2347 729512 2.8g 24212 94.6m 427991 1.6g jlaunch
10835 1 1473628 5.6g 9966 38.9m 16648 65.0m oracle
3134 1 19130 74.7m 9790 38.2m 10085 39.4m midaemon
10808 1 1219036 4.7g 9788 38.2m 14663 57.3m oracle
26381 1 1474652 5.6g 9777 38.2m 17724 69.2m oracle
7958 1 1474140 5.6g 9517 37.2m 17182 67.1m oracle
27730 1 1474159 5.6g 9490 37.1m 17187 67.1m oracle
4181 1 1474652 5.6g 9485 37.1m 17702 69.1m oracle
10142 1 1474140 5.6g 9481 37.0m 17183 67.1m oracle
24629 1 1474652 5.6g 9471 37.0m 17728 69.2m oracle
4179 1 1474652 5.6g 9468 37.0m 17705 69.2m oracle
7306 1 1474652 5.6g 9445 36.9m 17723 69.2m oracle
9977 1 1474140 5.6g 9442 36.9m 17183 67.1m oracle
2462 1 1474159 5.6g 9424 36.8m 17190 67.1m oracle
10500 1 1473884 5.6g 9414 36.8m 16922 66.1m oracle
8855 1 1219696 4.7g 9412 36.8m 15417 60.2m oracle
29705 1 1474652 5.6g 9399 36.7m 17722 69.2m oracle
9806 1 1474652 5.6g 9390 36.7m 17703 69.2m oracle
24625 1 1474652 5.6g 9377 36.6m 17702 69.1m oracle
2544 1 1474159 5.6g 9367 36.6m 17188 67.1m oracle
24623 1 1474652 5.6g 9358 36.6m 17701 69.1m oracle
26453 1 1474652 5.6g 9358 36.6m 17702 69.1m oracle
2455 1 1474159 5.6g 9356 36.5m 17189 67.1m oracle
24756 1 1474652 5.6g 9334 36.5m 17702 69.1m oracle
24627 1 1474652 5.6g 9333 36.5m 17701 69.1m oracle
8851 1 1221744 4.7g 9329 36.4m 17477 68.3m oracle
29707 1 1473884 5.6g 9313 36.4m 16924 66.1m oracle
28904 1 1474159 5.6g 9306 36.4m 17188 67.1m oracle
29703 1 1474396 5.6g 9302 36.3m 17443 68.1m oracle
24794 1 1474140 5.6g 9302 36.3m 17184 67.1m oracle
2451 1 1474671 5.6g 9289 36.3m 17725 69.2m oracle
24254 1 1474140 5.6g 9285 36.3m 17186 67.1m oracle
29741 1 1474140 5.6g 9281 36.3m 17202 67.2m oracle
5926 1 1474140 5.6g 9270 36.2m 17189 67.1m oracle
29739 1 1474140 5.6g 9258 36.2m 17184 67.1m oracle
29747 1 1474140 5.6g 9246 36.1m 17184 67.1m oracle
25003 1 1474652 5.6g 9243 36.1m 17697 69.1m oracle
17157 1 1474652 5.6g 9242 36.1m 17701 69.1m oracle
10162 1 1473884 5.6g 9237 36.1m 16924 66.1m oracle
17967 1 1474652 5.6g 9223 36.0m 17725 69.2m oracle
10668 1 1473628 5.6g 9195 35.9m 16664 65.1m oracle
25031 1 1473884 5.6g 9185 35.9m 16925 66.1m oracle
8849 1 1221744 4.7g 9090 35.5m 17461 68.2m oracle
===============================
# kctune filecache_min
Tunable Value Expression Changes
filecache_min 816039731 5% Imm (auto disabled)
# kctune filecache_max
Tunable Value Expression Changes
filecache_max 1305663569 8% Imm (auto disabled)
#
==================================
swap memory is 25 GB.

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 25600 12402 13198 48% 0 - 1 /dev/vg00/lvol2
reserve - 13185 -13185
memory 15565 8428 7137 54%
total 41165 34015 7150 83% - 0 -
#
=====================

Regards
Karki

Re: vhand process is taking more CPU.

Bart Paulusse — Fri, 10 Apr 2009 07:25:23 GMT

Hi Karki,

I see you have SAP running on that system.
Oracel takes a large chunk of your memory, but so does SAP. Every sap work process takes up memory but so does every user that is logged on in SAP. And that's even before they actualy start working and realy begin consuming memory.

I know SAP tells you to configure systems as if you have 150% memory because you have a large swap area, but as soon as your machine uses all of its real memory and swapping starts (vhand) then your perfomance goes out the window.

So either decrease you Oracle SGA, decrease your memory related sap parameters (decrease amount of users...) or by more memory.

vhand consuming 10 to 15% is actually quit reasonable when overloading your memory on SAP systems. I've had situations where vhand was useing almost all available cpu...

Regards,

Bart

Re: vhand process is taking more CPU.

Turgay Cavdar — Fri, 10 Apr 2009 07:44:46 GMT

vhand daemon is changed with the 11i.v3. Also check the vm patches.

vhand
The vhand daemonâ the System Pager daemon in the HP-UX kernelâ reclaims pages when there is memory pressure. In operating system versions earlier than HP-UX 11i v3, the primary role of vhand was to reclaim pages from the virtual memory Page Cache. With the implementation of the UFC, this role has been extended to include file cache memory. See Section 1, What is File Cache Memory ? for a definition of this term. Due to the extended scope, the vhand daemon is expected to be more active in HP-UX 11i v3 than in previous versions.
The vhand daemon uses a software technique called Cache Aging to maintain an optimal working set of file data in memory. Therefore, vhand keeps track of pages that have been accessed recently and of those that have not. Pages that have not been accessed recently are better candidates to be reclaimed. It is assumed that if pages have not been accessed in the recent past, they will not be accessed in the near future. When vhand reclaims pages, it may need to flush (page-out) modified data to the files before it can actually release the pages to the pool of free memory.
When vhand reclaims large size pages during memory pressure, it can entail an increased overhead due to large physical I/O request sizes and due to demotion of large size pages to smaller size pages. As mentioned in Section 1, Support for Variable Page Size and ccNUMA memory allocation, the file cache manager selects page sizes based on record sizes. An application that uses large record sizes to access file data can avoid the allocation of large size pages by specifying a smaller, preferred page size with fcntl() and fadvise().

Re: vhand process is taking more CPU.

smatador — Fri, 10 Apr 2009 07:45:09 GMT

Hi,
We had some performance issue with a SD-IA with 11.31, the server made many IO and we got specially trouble with the virtual memory so vhand before the release of september 2008.
The way, we solve many issue with memory was to patch with the latest vm cumulative patch PHKL_38651.
With the vm cumulative patch we got a stable box and the customers were very enthousiast and also very angry because during 4 month we search tuning oracle and suspect the array.
So i suggest you to tune your server to the most recent vm cumulative patch.
Hope it helps

Re: vhand process is taking more CPU.

Ganesan R — Fri, 10 Apr 2009 07:47:26 GMT

Hi Karki,

========================
User processes = 2998648 11.4g 72% details with -user
System = 962245 3.7g 23%
========================

You can't get more memory from system. 23% is quit normal. You need to dig into user processs. Does it really need 73% or improperly coded/configured?

Need to work with oracle/sap. If apps needs that much, then no other way other than upgrading the physical memory.

Re: vhand process is taking more CPU.

gb karki — Fri, 10 Apr 2009 08:34:59 GMT

Hi Bart,

Can we restict vhand process?
Not take more then 10 % CPU.

Regards
Karki

Re: vhand process is taking more CPU.

Bart Paulusse — Fri, 10 Apr 2009 08:51:23 GMT

You can only "restrict" vhand by decreasing your memory usage.
You are using more memory than you physically have, so vhand is doing what it's supposed to do, move memory pages to and from disk. This is a cpu and disk intensive operation.

Regards,
Bart

Re: vhand process is taking more CPU.

gb karki — Fri, 10 Apr 2009 08:59:18 GMT

Thanks Bart.

Re: vhand process is taking more CPU.

smatador — Fri, 10 Apr 2009 14:29:31 GMT

Hi,

Normally I agree at 100% with this" You can only restrict vhand by decreasing your memory usage." but in my experience and a call to the hp support (escalade), I would say you it's right but it's better after patching to the latest vm cumulative patch.

Because it's a newly install, I hope you have this patch, if yes forget my advise, if no look at the release.
http://www13.itrc.hp.com/service/patch/patchDetail.do?patchid=PHKL_38651&sel={hpux:11.31,}&BC=main|search|

Re: vhand process is taking more CPU.

Bill Hassell — Fri, 10 Apr 2009 20:24:40 GMT

vhand seems to be working normally. You do not have enough memory to run all the processes at the same time so vhand is trying to move processes in and out of the swap area. This creates enormous delays and most of the CPU time will be chewed up waiting for virtual memory reorganization. You can verify this by stopping enough processes to prevent further page outs (vmstat will show po in single digits). vhand will then go back to normal and performance will be as expected.

The only solution (with all the processes) is to double your RAM size. That will allow the processes to run at full speed.

Re: vhand process is taking more CPU.

gb karki — Mon, 13 Apr 2009 11:17:27 GMT

Hi Bill,

Now my system physical memory is 16 GB and SWAP space is 25 GB not double.

can you tell me this may be the issue?

Regards
Karki

Re: vhand process is taking more CPU.

Dennis Handly — Mon, 13 Apr 2009 13:45:08 GMT

>Now my system physical memory is 16 GB and swap space is 25 GB not double. can you tell me this may be the issue?

Yes, your memory should be larger.

Re: vhand process is taking more CPU.

Bill Hassell — Mon, 13 Apr 2009 16:58:31 GMT

> Now my system physical memory is 16 GB and SWAP space is 25 GB not double.
>
> can you tell me this may be the issue?

As Dennis said, you RAM is too small. I'm sure you meant to say: will it help to increase swap to 32 GB? And the answer is: Increasing swap simply means that you will run slower and slower. Using swap is very, very bad for performance. HP-UX is quite capable of running 250 GB of processes in 16 GB of RAM. You might have to wait an hour before you can get logged in but it works.

Paging out is designed for the occasional use where a large process is waiting for a long time (for example, a user that went to lunch) and can be moved out of memory to make room for other processes. Biut if all your processes are busy all the time, then paging out to swap is a very bad thing. Performance will drop to less than 1/10 of what you expect, maybe worse.

There is no substitute for RAM. Swap simply provides a very, very slow location to move programs out of memory (and back again). Tell your Oracle DBAs to reduce Oracle memory usage to 4GB and swap activity should be minimized (very little vhand activity).

Or buy more RAM.