topic Re: HP-UX 11.31 System Utilization Too High in Operating System - HP-UX

HP-UX 11.31 System Utilization Too High

Teo Yi Fang — Thu, 10 Jul 2008 02:58:16 GMT

Hi All,

Now I am facing some problem that the system memory usage for my 11.31 running on rx6600 is very high. I can solve this issue on previous hp-ux version by tuning the dbc_max_pct and dbc_min_pct but in 11.31 this two parameters had been replaced by filecache_max and filecache_min. But seem like it is useless if I try to tune this two parameters in 11.31. Any other kernel parameter that I can tune to make the system memory usage lower?

Thanks.

Re: HP-UX 11.31 System Utilization Too High

Avinash Agarkar — Thu, 10 Jul 2008 03:47:04 GMT

hi,

Check how much total physical memory you have installed in you system, so your swap should be double then check if your application or database required more memory then increase the physical memory and swap too.

Set the proper piority of swap and check.

after this if you didn't get the performance then go for kernal tuning.

Thanks,
Avinash

Re: HP-UX 11.31 System Utilization Too High

YAQUB_1 — Thu, 10 Jul 2008 03:49:59 GMT

Hi,

put the "kctune" output here

BR/
Yaqub

Re: HP-UX 11.31 System Utilization Too High

Teo Yi Fang — Thu, 10 Jul 2008 05:19:33 GMT

Hi,

Attached is the output of "kctune"

Thanks.

Re: HP-UX 11.31 System Utilization Too High

Dennis Handly — Thu, 10 Jul 2008 08:36:43 GMT

Why do you care how much memory you are using? Is something failing?
Also what is the output of "swapinfo -tam"?

Re: HP-UX 11.31 System Utilization Too High

Teo Yi Fang — Thu, 10 Jul 2008 08:42:40 GMT

Hi,

Because the system memory usage is too high and causing the memory usage to hit 100% and causing the operations of the Oracle.

Below is the output of "swapinfo -tam"
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 8192 182 8010 2% 0 - 1 /dev/vg00/lvol2
dev 4096 182 3914 4% 0 - 1 /dev/vg00/lvol10
reserve - 11830 -11830
memory 11659 3993 7666 34%
total 23947 16187 7760 68% - 0 -

Re: HP-UX 11.31 System Utilization Too High

Dennis Handly — Thu, 10 Jul 2008 09:33:08 GMT

>Because the system memory usage is too high and causing the memory usage to hit 100% and causing the operations of the Oracle.

Since your filecache_* values are 2 to 3%, you have no choice but reduce the load on your system or add more swap or RAM. (Or reduce the Oracle SGA.)

What does "ipcs -am" show?

Re: HP-UX 11.31 System Utilization Too High

Teo Yi Fang — Thu, 10 Jul 2008 09:38:35 GMT

Ya....since like the only choice is to add more memory or the swap space.

By the way, this is the output of the "ipcs -am"

IPC status from /dev/kmem as of Thu Jul 10 18:34:44 2008
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME
Shared Memory:
m 0 0x411c016a --rw-rw-rw- root root root root 0 348 956 956 15:10:17 15:10:17 15:10:11
m 1 0x4e0c0002 --rw-rw-rw- root root root root 1 61760 956 29868 15:20:33 15:20:33 15:10:11
m 2 0x41202182 --rw-rw-rw- root root root root 1 8192 956 968 15:10:13 15:10:11 15:10:11
m 3 0x00a5c581 --rw------- sfmdb users sfmdb users 4 10469376 1610 1616 15:10:55 15:10:55 15:10:55
m 32772 0x06347849 --rw-rw-rw- root root root root 2 65544 1750 1777 15:11:05 15:11:02 15:11:01
m 688133 0x0c6629c9 --rw-r----- root root root root 2 19023064 1792 2421 17:42:16 17:42:26 15:11:02
m 32774 0x49189ebc --rw-r--r-- root root root root 0 22908 1777 1777 18:34:00 18:34:00 15:11:03
m 41025543 0x41bb6640 --rw-r----- oracle oinstall oracle oinstall 385 1593843712 6210 20007 18:34:42 18:34:43 19:07:49
m 7471112 0x03c0c360 --rw-r----- oracle oinstall oracle oinstall 108 1077948416 12659 22604 18:34:44 18:34:44 15:55:35
m 9 0x0000cace --rw-rw-rw- root sys root sys 0 2 22710 22710 20:08:26 20:08:26 10:16:05
m 2719754 0xed1553d4 --rw-r----- oracle oinstall oracle oinstall 35 1077948416 11333 22447 18:34:18 18:34:18 15:45:55

Re: HP-UX 11.31 System Utilization Too High

Dennis Handly — Thu, 10 Jul 2008 10:02:48 GMT

Using this on your ipcs -am output:
awk ' { print $5, $10 }'
OWNER SEGSZ
root 348
root 61760
root 8192
sfmdb 10469376
root 65544
root 19023064
root 22908
oracle 1593843712
oracle 1077948416
root 2
oracle 1077948416

That's 2.6 Gb for oracle.

Re: HP-UX 11.31 System Utilization Too High

Teo Yi Fang — Thu, 10 Jul 2008 11:22:05 GMT

It means that Oracle only use up 2.6GB out of my 12GB physical memory?

Thanks.

Re: HP-UX 11.31 System Utilization Too High

Dennis Handly — Thu, 10 Jul 2008 11:41:42 GMT

>It means that Oracle only use up 2.6GB out of my 12GB physical memory?

It is using 2.6 Gb of swap space for shared memory segments. Who knows what else it is using.

Re: HP-UX 11.31 System Utilization Too High

Khairul_1 — Thu, 10 Jul 2008 12:33:55 GMT

Hi Teo,

From your swapinfo -tam output it is showing swap is very low i.e. (8GB+4GB=12GB) and Memory 12GB. It is better to increase swap double or more and check that status.

Thanks
Khairul
Bangladesh

Re: HP-UX 11.31 System Utilization Too High

Don Morris_1 — Thu, 10 Jul 2008 13:32:13 GMT

1) I can't deduce pixels, sorry. What percentage is system? (Not to mention what's "too high"? Anything over 0%? Realistically, 11i v3 *will* eat around 10% for system, that's just the way things work. Above that is load dependent, and we'll need to understand where you're trying to get things).

2) When memory utilization hits 100% (in other words, when you hit pressure)... Oracle operations ---- do what, exactly? Slow down? Block entirely? Please be clear. Pressure is going to have an effect on the system, certainly -- letting system caches build until pressure is reached is default behavior and is expected. Pressure is what would cause system caches to flush down -- so understanding what impact you're seeing at the pressure point is essential to differentiate what you've given from "Fluctuating between User and Kernel consumption under load" to "Total load too much for box".

3) Patch levels? (Particularly core kernel VM patches?) Since LR there have been several fixes in the filecache space, especially when memory pressure is reached. You may simply need one. The patch family I'm talking about currently ends at PHKL_38449.

4) Your kctune output shows JFS. Have you looked at http://docs.hp.com/en/5992-0732/5992-0732.pdf and checked/considered your JFS tuning?

5) Compiling and running the attached will help narrow things down. (Alternately, if you happen to have picked up kmeminfo surreptitiously, run that).

6) Last but not least [and only if you didn't run kmeminfo] -- as root do:

#kwdb -q4 /stand/vmunix

q4> dbc_ceiling

q4> dbc_bufpages

q4> quit

And by the way -- I can flat out guarantee tuning filecache_min and filecache_max is not useless. They most definitely do what they're supposed to do... and if you set them both higher, I promise you'd see a difference (you'd either fail the kctune to raise min or be under pressure immediately depending on the state at the time of the request, I'd suspect the raise would fail from the output given). They don't control everything in the kernel, though -- so we need to narrow things down.

Re: HP-UX 11.31 System Utilization Too High

Teo Yi Fang — Fri, 11 Jul 2008 02:16:59 GMT

Hi Don,

Thanks for your explanation and advices.