- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - HP-UX
- >
- Re: HP-UX v11.31 high System (SYS) Utilization
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Discussions
Discussions
Forums
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО11-12-2014 09:01 PM
тАО11-12-2014 09:01 PM
Hi everyone!
I have OS HP-UX v11.31 on IVM located on nPar of Supredome sx2000 and there are 2 SAP ECC (Dev & Qas) systems on it.
We saw some problems with performance before SAP sytems were merged. Now we have High performance problems exactly in SYS. I see it in "top" System utilization, in "uptime" load average, forks in sar are not so high. Parameter "filecache_max" in kcusage close to maximum value.
How I understand high SYS utilization is talking about a lot of system calls and problems with different type of I/O (network, HDD), but there is no any kind of high utilization in it (network SAN & LAN, disks). After hours of debates we've deceived to update some Kernel patches such as HostAVIOStor or upgrade HPVM to version 4.3, but realy cause of the problem we can't understand.
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.87 8.9% 0.0% 53.9% 37.2% 0.0% 0.0% 0.0% 0.0%
1 0.71 0.7% 0.0% 77.0% 22.3% 0.0% 0.0% 0.0% 0.0%
2 0.82 44.1% 0.0% 28.3% 27.6% 0.0% 0.0% 0.0% 0.0%
3 0.76 1.7% 0.0% 54.3% 43.9% 0.0% 0.0% 0.0% 0.0%
4 0.74 1.5% 0.0% 65.8% 32.7% 0.0% 0.0% 0.0% 0.0%
5 0.63 2.3% 0.0% 66.0% 31.7% 0.0% 0.0% 0.0% 0.0%
6 0.62 0.9% 0.0% 40.1% 59.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.73 8.5% 0.0% 55.7% 35.8% 0.0% 0.0% 0.0% 0.0%
#uptime
9:46am up 12 days, 1:05, 3 users, load average: 2.61, 1.54, 0.80
#uptime
12:02pm up 3 days, 21:34, 7 users, load average: 1.01, 0.89, 0.78
14:26:36 scall/s sread/s swrit/s fork/s exec/s rchar/s wchar/s
14:26:37 28705 1191 1240 3.81 4.76 1022050 78019
14:26:38 52674 2097 1318 24.36 24.36 2315816 895344
14:26:39 48340 1134 655 20.35 15.93 4842704 529218
14:26:40 35513 437 221 11.61 12.50 2638628 533943
14:26:41 70522 1413 933 12.75 11.76 2665412 164643
Average 46580 1196 837 14.12 13.33 2749804 420442
# kcusage
Tunable Usage / Setting
=============================================
filecache_max 2030202880 / 2042346291
maxdsiz 28712960 / 1073741824
maxdsiz_64bit 8976334848 / 17179869184
maxfiles_lim 168 / 63488
maxssiz 786432 / 134217728
maxssiz_64bit 1769472 / 1073741824
maxtsiz 9797632 / 100663296
maxtsiz_64bit 419430400 / 1073741824
maxuprc 64 / 4096
max_thread_proc 101 / 3000
msgmni 2 / 4096
msgtql 0 / 4096
nflocks 112 / 8192
ninode 2079 / 165888
nkthread 1303 / 250000
nproc 421 / 4200
npty 0 / 60
nstrpty 11 / 60
nstrtel 0 / 60
nswapdev 2 / 32
nswapfs 0 / 32
semmni 205 / 8192
semmns 810 / 16384
shmmax 6861881344 / 4398046511104
shmmni 99 / 4096
shmseg 12 / 512
# vmstat 1
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
5 2 0 7847886 1486629 0 0 12 2 0 0 10 2118 36763 1069 8 12 80
4 1 0 7311394 1486783 0 0 1 0 0 0 0 1914 24536 1485 8 12 81
4 1 0 7311394 1484867 0 0 0 0 0 0 0 2025 23807 1454 12 8 80
4 1 0 7311394 1488316 0 0 0 0 0 0 0 1936 22373 1302 9 32 58
4 1 0 7311394 1486311 0 0 0 0 0 0 0 1951 24623 1301 16 23 61
4 1 0 7311394 1491777 0 0 0 0 0 0 0 1931 25477 1274 3 20 77
4 1 0 7311394 1491777 0 0 0 0 0 0 0 1891 23210 1147 6 29 65
Solved! Go to Solution.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО11-18-2014 12:37 AM
тАО11-18-2014 12:37 AM
Re: HP-UX v11.31 high System (SYS) Utilization
What does the "top" output show ? Does it show system daemons such as vhand/kmemdaemon showing high utilization ?
-santosh
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО11-18-2014 09:10 PM - edited тАО11-18-2014 09:12 PM
тАО11-18-2014 09:10 PM - edited тАО11-18-2014 09:12 PM
Re: HP-UX v11.31 high System (SYS) Utilization
I've checked the history and got information about vhand in top. This process really got more CPU resources than usual in the case of using HP Data Protector for backup (sar19.txt). In other cases vhand doesn't use a lot of resources, but problem can exist (sar17.txt). I've attached 2 files with information collected by sar. kmemdaemon doesn't use a lot of resources.
This system ():
CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
2 2 root 128 20 72K 64K sleep 224:43 1.85 1.85 vhand
2 2 root 128 20 72K 64K sleep 130:47 3.51 3.50 vhand
0 2 root 128 20 72K 64K sleep 127:45 3.75 3.75 vhand
Other systems:
1 ? 2 root 128 20 72K 64K sleep 141:34 0.14 0.14 vhand
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО11-19-2014 12:31 AM
тАО11-19-2014 12:31 AM
SolutionThere is some file activity, which causes the system pager (vhand) to execute more often -- but it does not seem to be too bad, and we can rule that out as the problem.
Can you share the 'top' O/P ? You mentioned that you were running SAP applications -- these tend to make a lot of mprotect() system calls. In some cases, these can spend a lot of time in the kernel. Do you see these apps as consuming most of the CPU cycles ?
What is the level of the VM cumulative patch on your system ?
You also mentioned that you were running at HPVM4.3. Any reason you cannot upgrade to the latest ones -- there have been a lot of performance fixes between 4.3 and 6.3 ?
-santosh
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО11-23-2014 09:16 PM - edited тАО11-26-2014 10:48 PM
тАО11-23-2014 09:16 PM - edited тАО11-26-2014 10:48 PM
Re: HP-UX v11.31 high System (SYS) Utilization
We've done:
1) In OS we did update HPVM AVIO from B.11.31.0903 to B.11.31.1003
2) In SAP we changed parameter "em/initial_size_MB" from 6500mb to 8192mb
3) parameter rdisp/gui_auto_logout from "0" to "3600"
4) Update core of SAP system ECQ.
At the first time System worked better with maximum load in time of backup (high sr and sys).
At the second time System used more CPU SYS resources (95% of all CPU resources).
Update from v.4.3 to 6.3, as the update from 4.2 to 4.3, should be done on Host and on all IVM. This is difficult on our landscape.
In the proccess log i'm see activity of Oracle applications, such as tnslsnr and sqlplus. There are a lot of proccess, twice more than usual and all about Oracle and SAP. They see delay then DB work with disks.
FEATURE11i B.11.31.1209.383a Feature Enablement Patches for HP-UX 11i v3, September 2012
HWEnable11i B.11.31.1209.383a Hardware Enablement Patches for HP-UX 11i v3, September 2012
QPKAPPS B.11.31.1209.383 Applications Patches for HP-UX 11i v3, September 2012
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО12-02-2014 01:31 AM
тАО12-02-2014 01:31 AM
Re: HP-UX v11.31 high System (SYS) Utilization
It's hard to make suggestions without running a few more tools and gathering more data. You are certainly running into some sort of issue with the HP-UX kernel. Whether this is a known fixed issue or a new one can be determined with more data. Do you have experience running Caliper ?
Since it appears that it is Orcale/SAP user processes spending too much time in the kernel, you maybe running into issues already fixed in the past.
I'd recommend that you install "11.31 vm cumulative patch" PHKL_43513 or later. This has fixes for some issues we see on SAP installations. Also, what is the value of the tunable 'mprotect_reduce_protid_on' ?
-santosh