topic Re: vhand process needs 100 % cpu in Operating System - HP-UX

vhand process needs 100 % cpu

Walter Kießl — Wed, 09 Apr 2008 06:27:56 GMT

Hi,

I have a rx2660 with hpux11.31. The system has 24 gb physical memory installed. According to gpm filecache is 1,1 gb, pseudo-swap is 22,8 g and free mem is 18 gb. Device swap ist 24 gb. Operating system is patched with all available patches up to february 2008.

I can start several oracle databases on it (free mem still >10gb) but after a few hours the system will suddenly have responstimes up to 1 hour (!). As the system is unusable in this situation very little information is available. What I could see is, that vhand is up to 100 % cpu, but gpm does not show page ins/out/faults nor network or disk activity, but plenty of free memory (about 5gb).

Has anyone faced an equal problem? How to deal with it? Is this related to NFS, networking, databases, disk-io, others?

I suspect that pseudo-swap is the cause for this problem. Is there a way in 11.31 to get rid of it or to shrink it to less than 1gb?

Thanks for your help!
Walter

Re: vhand process needs 100 % cpu

Safarali — Wed, 09 Apr 2008 06:50:48 GMT

Hi Walter what is the out put of

swapinfo -tam

sar -u 5 5

Regards
Safar

Re: vhand process needs 100 % cpu

Venkatesh BL — Wed, 09 Apr 2008 07:05:15 GMT

Try installing the latest vm patch on the system. If the problem persists, you could contact HP support for thorough investigation.

Re: vhand process needs 100 % cpu

Duncan Edmonstone — Wed, 09 Apr 2008 07:10:55 GMT

>> As the system is unusable in this situation very little information is available.

Presumably at this stage you just reboot it yes?

Instead you should be issuing a TC from the Management Processor - this will generate a core dump which you can send to HP support for analysis. They should be able to get some sort of indication of what is happening from the dump.

HTH

Duncan

Re: vhand process needs 100 % cpu

Eric Antunes — Wed, 09 Apr 2008 07:19:48 GMT

Hi Walter,

Can you attach your Oracle init.ora?

Best Regards,

Eric

Re: vhand process needs 100 % cpu

Rasheed Tamton — Wed, 09 Apr 2008 07:22:33 GMT

Hi Walter,

Vhand runs normally when your demands for MEM have exceeded the physical RAM.

Also did you check the link speed of your network card (HD/FD),

Otherwise, please provide more info as asked above.

Regards,
Rasheed.

Re: vhand process needs 100 % cpu

Walter Kießl — Wed, 09 Apr 2008 07:48:38 GMT

Hi to all,

thanks for your answers.

swapinfo -tam on the (now unused) system:

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 8192 0 8192 0% 0 - 1 /dev/vg00/lvol2
dev 16384 0 16384 0% 0 - 1 /dev/vg00/lvol10
reserve - 723 -723
memory 23352 3396 19956 15%
total 47928 4119 43809 9% - 0 -

Linkspeed is ok. We have transferrates of about 100MB/sec from our NFS server.

I've already patched with the latest vm patches and even opened a case/escalation with HP. But after 6 weeks we still have no solution for the problem (but a lot of tips like not using 11.31 or using FC instead of NFS)

The problem didn't arise when we started one test database on it. The situation described above arose, when we started our development databases. It is reproducable (it happened 4 times now). But as nobody can work with the databases my DBAs refuse to do another test.

The databases I transfered to this server have been running on our parisc-development-system in the same configuration for years - whithout problems.

As this server is our testsystem for the planned parisc-to-itanium-migration no one at my shop is willing to invest more time/money in this project. They are already seeking for alternatives to hpux ...

So I'm sorry that I cannot provide more information than this or do more testings.

But I still hope for some hints because information about 11.31 is rare.

Thanks again for your help!!!

Regards
Walter

Re: vhand process needs 100 % cpu

Rasheed Tamton — Wed, 09 Apr 2008 08:18:58 GMT

Please check this link - might give you some hints.

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1203773

Regards,
Rasheed Tamton.

Re: vhand process needs 100 % cpu

Dennis Handly — Wed, 09 Apr 2008 08:29:17 GMT

>I suspect that pseudo-swap is the cause for this problem. Is there a way in 11.31 to get rid of it or to shrink it to less than 1gb?

I doubt that pseudo-swap would ever be this problem, it's just a bookkeeping trick. And swapmem_on is obsolete on 11.31.

>vhand is up to 100% cpu, but gpm does not show page ins/out/faults nor network or disk activity

What does it show about the unified cache?
Perhaps vhand is so confused that is just burning CPU?

>but plenty of free memory (about 5gb).

How do you know this?

>Is this related to NFS, networking, databases, disk-I/O, others?

It wouldn't be related to NFS or networking (unless used for device swap?) but memory, I/O and databases.

>swapinfo -tam on the (now unused) system:

Can you collect swapinfo output continuously so you can get it just before your system becomes unusable?

Re: vhand process needs 100 % cpu

Wim Rombauts — Wed, 09 Apr 2008 08:49:03 GMT

Do you use any resource management software ?
Just a thought of course, but if you use WLM and limit the amount of available RAM per resource group, your system can go swapping for that resource group way before the total system is out of memory.

Is there any other reason you can think of that would make your system to try to squeeze your oracle databases into a subset of your system memory ?

Re: vhand process needs 100 % cpu

Rasheed Tamton — Wed, 09 Apr 2008 09:04:12 GMT

The above link contains ref. to this doc [The Unified File Cache (UFC) in HP-UX 11i v3]
http://h71028.www7.hp.com/ERC/downloads/4AA0-7157ENW.pdf

Check for the sections called What is file cache memory and vhand. The are few new kernel variables which are related to performance issues.

Regards.

Re: vhand process needs 100 % cpu

Steven E. Protter — Wed, 09 Apr 2008 09:41:17 GMT

Shalom Walter,

My look at your output shows no free memory or swap.

You system is effectively dead, spending all its time swapping processes to disk and back and doing no other meaningful work.

Take a look at the Oracle SGA settings and make some cuts, or run fewer instances than you are running.

SEP

Re: vhand process needs 100 % cpu

Wim Rombauts — Wed, 09 Apr 2008 13:39:17 GMT

I think you really need to run swapinfo and vmstat in a loop and write the output to a file. This way, you can check the memory status of the almost hanging system after rebooting.

Use something like :
while true
do
swapinfo -tam >> /tmp/swapinfo.log
sleep 1
done

and
vmstat 1 9999999

of course, you are free to choose your time interval (here set to 1 second, which may be too fast).

Re: vhand process needs 100 % cpu

Don Morris_1 — Wed, 09 Apr 2008 19:09:00 GMT

No, as Dennis said pseudo-swap isn't the issue.
(And as he also hinted at by virtue of the tunable being gone, no -- there's no way to disable pseudo-swap in v3 or later.)

And I suspect free memory isn't either per se -- this sounds like a vhand/file cache interaction to me. In fact, it sounds rather like QXCR1000589687 (of course, there's not a lot of data here either). I know you said that you can't spend any more time on this - but since you raised the escalation, you might mention that CR to your support contact and have them work with the lab to see what your options are as far as possible early/unofficial versions of the fix or scheduling, etc.