topic Re: HP-UX v11.31 high System (SYS) Utilization in Operating System - HP-UX

HP-UX v11.31 high System (SYS) Utilization

zidercol — Thu, 13 Nov 2014 05:01:30 GMT

Hi everyone!

I have OS HP-UX v11.31 on IVM located on nPar of Supredome sx2000 and there are 2 SAP ECC (Dev & Qas) systems on it.

We saw some problems with performance before SAP sytems were merged. Now we have High performance problems exactly in SYS. I see it in "top" System utilization, in "uptime" load average, forks in sar are not so high. Parameter "filecache_max" in kcusage close to maximum value.

How I understand high SYS utilization is talking about a lot of system calls and problems with different type of I/O (network, HDD), but there is no any kind of high utilization in it (network SAN & LAN, disks). After hours of debates we've deceived to update some Kernel patches such as HostAVIOStor or upgrade HPVM to version 4.3, but realy cause of the problem we can't understand.

Cpu states:

CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS

0 0.87 8.9% 0.0% 53.9% 37.2% 0.0% 0.0% 0.0% 0.0%

1 0.71 0.7% 0.0% 77.0% 22.3% 0.0% 0.0% 0.0% 0.0%

2 0.82 44.1% 0.0% 28.3% 27.6% 0.0% 0.0% 0.0% 0.0%

3 0.76 1.7% 0.0% 54.3% 43.9% 0.0% 0.0% 0.0% 0.0%

4 0.74 1.5% 0.0% 65.8% 32.7% 0.0% 0.0% 0.0% 0.0%

5 0.63 2.3% 0.0% 66.0% 31.7% 0.0% 0.0% 0.0% 0.0%

6 0.62 0.9% 0.0% 40.1% 59.0% 0.0% 0.0% 0.0% 0.0%

--- ---- ----- ----- ----- ----- ----- ----- ----- -----

avg 0.73 8.5% 0.0% 55.7% 35.8% 0.0% 0.0% 0.0% 0.0%

#uptime
9:46am up 12 days, 1:05, 3 users, load average: 2.61, 1.54, 0.80

#uptime

12:02pm up 3 days, 21:34, 7 users, load average: 1.01, 0.89, 0.78

14:26:36 scall/s sread/s swrit/s fork/s exec/s rchar/s wchar/s
14:26:37 28705 1191 1240 3.81 4.76 1022050 78019
14:26:38 52674 2097 1318 24.36 24.36 2315816 895344
14:26:39 48340 1134 655 20.35 15.93 4842704 529218
14:26:40 35513 437 221 11.61 12.50 2638628 533943
14:26:41 70522 1413 933 12.75 11.76 2665412 164643

Average 46580 1196 837 14.12 13.33 2749804 420442

# kcusage
Tunable Usage / Setting
=============================================
filecache_max 2030202880 / 2042346291
maxdsiz 28712960 / 1073741824
maxdsiz_64bit 8976334848 / 17179869184
maxfiles_lim 168 / 63488
maxssiz 786432 / 134217728
maxssiz_64bit 1769472 / 1073741824
maxtsiz 9797632 / 100663296
maxtsiz_64bit 419430400 / 1073741824
maxuprc 64 / 4096
max_thread_proc 101 / 3000
msgmni 2 / 4096
msgtql 0 / 4096
nflocks 112 / 8192
ninode 2079 / 165888
nkthread 1303 / 250000
nproc 421 / 4200
npty 0 / 60
nstrpty 11 / 60
nstrtel 0 / 60
nswapdev 2 / 32
nswapfs 0 / 32
semmni 205 / 8192
semmns 810 / 16384
shmmax 6861881344 / 4398046511104
shmmni 99 / 4096
shmseg 12 / 512

# vmstat 1
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
5 2 0 7847886 1486629 0 0 12 2 0 0 10 2118 36763 1069 8 12 80
4 1 0 7311394 1486783 0 0 1 0 0 0 0 1914 24536 1485 8 12 81
4 1 0 7311394 1484867 0 0 0 0 0 0 0 2025 23807 1454 12 8 80
4 1 0 7311394 1488316 0 0 0 0 0 0 0 1936 22373 1302 9 32 58
4 1 0 7311394 1486311 0 0 0 0 0 0 0 1951 24623 1301 16 23 61
4 1 0 7311394 1491777 0 0 0 0 0 0 0 1931 25477 1274 3 20 77
4 1 0 7311394 1491777 0 0 0 0 0 0 0 1891 23210 1147 6 29 65

Re: HP-UX v11.31 high System (SYS) Utilization

Santosh Abraham — Tue, 18 Nov 2014 08:37:27 GMT

What does the "top" output show ? Does it show system daemons such as vhand/kmemdaemon showing high utilization ?

-santosh

Re: HP-UX v11.31 high System (SYS) Utilization

zidercol — Wed, 19 Nov 2014 05:12:31 GMT

I've checked the history and got information about vhand in top. This process really got more CPU resources than usual in the case of using HP Data Protector for backup (sar19.txt). In other cases vhand doesn't use a lot of resources, but problem can exist (sar17.txt). I've attached 2 files with information collected by sar. kmemdaemon doesn't use a lot of resources.

This system ():

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND

2 2 root 128 20 72K 64K sleep 224:43 1.85 1.85 vhand
2 2 root 128 20 72K 64K sleep 130:47 3.51 3.50 vhand
0 2 root 128 20 72K 64K sleep 127:45 3.75 3.75 vhand

Other systems:

1 ? 2 root 128 20 72K 64K sleep 141:34 0.14 0.14 vhand

Re: HP-UX v11.31 high System (SYS) Utilization

Santosh Abraham — Wed, 19 Nov 2014 08:31:30 GMT

There is some file activity, which causes the system pager (vhand) to execute more often -- but it does not seem to be too bad, and we can rule that out as the problem.

Can you share the 'top' O/P ? You mentioned that you were running SAP applications -- these tend to make a lot of mprotect() system calls. In some cases, these can spend a lot of time in the kernel. Do you see these apps as consuming most of the CPU cycles ?

What is the level of the VM cumulative patch on your system ?

You also mentioned that you were running at HPVM4.3. Any reason you cannot upgrade to the latest ones -- there have been a lot of performance fixes between 4.3 and 6.3 ?

-santosh

Re: HP-UX v11.31 high System (SYS) Utilization

zidercol — Thu, 27 Nov 2014 06:48:00 GMT

We've done:

1) In OS we did update HPVM AVIO from B.11.31.0903 to B.11.31.1003

2) In SAP we changed parameter "em/initial_size_MB" from 6500mb to 8192mb

3) parameter rdisp/gui_auto_logout from "0" to "3600"

4) Update core of SAP system ECQ.

At the first time System worked better with maximum load in time of backup (high sr and sys).

At the second time System used more CPU SYS resources (95% of all CPU resources).

Update from v.4.3 to 6.3, as the update from 4.2 to 4.3, should be done on Host and on all IVM. This is difficult on our landscape.

In the proccess log i'm see activity of Oracle applications, such as tnslsnr and sqlplus. There are a lot of proccess, twice more than usual and all about Oracle and SAP. They see delay then DB work with disks.

FEATURE11i B.11.31.1209.383a Feature Enablement Patches for HP-UX 11i v3, September 2012
HWEnable11i B.11.31.1209.383a Hardware Enablement Patches for HP-UX 11i v3, September 2012
QPKAPPS B.11.31.1209.383 Applications Patches for HP-UX 11i v3, September 2012

Re: HP-UX v11.31 high System (SYS) Utilization

Santosh Abraham — Tue, 02 Dec 2014 09:31:11 GMT

It's hard to make suggestions without running a few more tools and gathering more data. You are certainly running into some sort of issue with the HP-UX kernel. Whether this is a known fixed issue or a new one can be determined with more data. Do you have experience running Caliper ?
Since it appears that it is Orcale/SAP user processes spending too much time in the kernel, you maybe running into issues already fixed in the past.

I'd recommend that you install "11.31 vm cumulative patch" PHKL_43513 or later. This has fixes for some issues we see on SAP installations. Also, what is the value of the tunable 'mprotect_reduce_protid_on' ?

-santosh