HP-UX 11.31 SAP running slow and VHAND on TOP

[jsk_cbe]

Dear Admins,
We have SAP solution manager running on 11.31 BL860 Blade. We are facing the application running very slow and even commands on unix does'nt run fast. I see the Vhand process is on TOP and vmstat has only 7608 Free, out of 8GB physical memory. I dont have any VMs in this server.I tried reducing the filecache_min and max to 800MB earlier it was 1GB. But still no change. Can anyone suggest if any kernel parameter needs to be tuned.?

System: eosmsm1 Tue Feb 2 09:44:08 2010
Load averages: 0.05, 0.05, 0.11
259 processes: 204 sleeping, 53 running, 2 zombies
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.04 0.2% 0.0% 2.1% 97.7% 0.0% 0.0% 0.0% 0.0%
2 0.07 1.5% 0.0% 1.7% 96.9% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.05 0.8% 0.0% 2.0% 97.2% 0.0% 0.0% 0.0% 0.0%

System Page Size: 4Kbytes
Memory: 6260036K (4259652K) real, 16893884K (13368192K) virtual, 28872K free Page# 1/8

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 ? 2 root 128 20 72K 64K sleep 10:13 4.89 4.88 vhand
2 ? 4350 sm1adm 152 20 2965M 41148K run 9:15 3.62 3.62 jlaunch
0 ? 1617 cimsrvr 152 20 70164K 12900K run 12:26 2.75 2.74 cimservermain


eosmsm1@root:/> vmstat -S 5 13
procs memory page faults cpu
r b w avm free si so pi po fr de sr in sy cs us sy id
1 19 0 2973221 7100 348 348 18 14 4 0 157 478 5283 248 3 0 97
1 19 0 2973221 7174 0 0 617 250 257 0 11345 1058 6766 572 1 2 97
1 17 0 2940873 7166 0 0 706 317 339 0 11415 1105 3912 591 1 2 97
1 17 0 2940873 7140 0 0 791 456 339 0 7583 1198 3543 637 3 2 95
4 12 0 3403950 7155 0 0 761 453 368 0 9474 1284 13936 696 6 2 92
4 12 0 3403950 7131 0 0 875 266 418 0 8896 1281 7082 670 1 3 96
2 13 0 3381376 7208 0 0 842 227 517 0 12844 1177 3881 605 1 2 97
4 15 0 2990452 7111 0 0 800 243 523 0 8587 1165 2888 575 1 2 97
4 15 0 2990452 7133 0 0 830 290 539 0 10299 1207 3995 690 2 1 97
3 15 0 2859935 7211 0 0 799 380 580 0 13804 1259 3915 618 1 2 97
3 15 0 2859935 7095 0 0 910 345 577 0 12921 1473 5012 703 2 5 93
4 13 0 3085229 7058 0 0 918 312 735 0 14888 1373 9632 704 0 3 97



eosmsm1@root:/> vmstat 5 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
2 19 0 3127203 35738 1 0 20 15 4 0 174 480 5329 249 3 0 97
2 19 0 3127203 6728 76 0 1841 470 323 0 18432 1899 36193 1595 3 1 8 62
1 30 0 3492648 6726 98 0 1057 807 526 0 27942 1415 11882 751 4 4 92
1 68 0 2929081 10158 7 0 36 9036 355 0 6789 3906 1349 200 1 6 93
2 44 0 3178168 63971 88 0 1716 400 17 0 302 1830 4766 804 5 2 93

eosmsm1@root:/> vmstat 5 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 18 0 3482785 7784 1 0 20 15 4 0 175 480 5329 249 3 0 97
1 18 0 3482785 7498 110 0 884 714 375 0 16225 1327 5296 636 2 4 94
3 19 0 3481171 6828 117 0 1229 823 449 0 11254 1487 4843 678 2 3 95
3 19 0 3481171 6265 136 0 1398 891 964 0 9120 1535 19419 817 7 4 90
1 43 0 3495672 6489 74 0 628 8800 478 0 6117 3747 8175 482 3 8 88
eosmsm1@root:/> vmstat 5 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
3 8 0 3254047 246782 1 0 20 16 5 0 176 480 5335 249 3 0 97
3 7 0 2887165 215101 120 0 2968 4 0 0 12 6453 175046 8631 67 9 24
3 7 0 2887165 195028 65 0 2312 0 0 0 3 3687 146824 5372 42 3 56
4 4 0 2855116 141087 39 0 1681 0 0 0 0 3111 234690 3653 61 8 31
4 4 0 2855116 87022 21 0 1803 0 0 0 0 3254 204627 3795 79 6 15





eosmsm1@root:/> kctune filecache_min=400MB filecache_max=400MB
==> Update the automatic 'backup' configuration first? y
* The automatic 'backup' configuration has been updated.
* Future operations will update the backup without prompting.
NOTE: The requested changes could not be applied to the currently
running system, for the following reasons:
* The requested changes have been saved, and will take effect at
next boot.
Tunable Value Expression Changes
filecache_max (now) 4070334464 Default Auto
(next boot) 419430400 400MB
filecache_min (now) 407031808 Default Auto
(next boot) 419430400 400MB




Feb 3 08:54:56 eosmsm1 su: + 3 root-sm1adm
Feb 3 09:04:29 eosmsm1 SAPSM1_01[26658]: Unable to open trace file sapstartsrv.log. (Error 13 Permission denied) [ntservs server.cpp 1909]
Feb 3 09:05:02 eosmsm1 SAPSM1_00[26721]: Unable to open trace file sapstartsrv.log. (Error 13 Permission denied) [ntservs server.cpp 1909]
Feb 3 09:18:49 eosmsm1 su: + 3 root-sm1adm
Feb 3 09:20:23 eosmsm1 sshd[27262]: Accepted keyboard-interactive/pam for root from 10.20.22.133 port 49915 ssh2
Feb 3 09:23:25 eosmsm1 vmunix: Warning: The validity of the tunable values could not be completely verified, because the value of the tunable 'fs_meta_max' will not be known until the system is booted. The tunable values will be verified durin g boot. Please check the console messages during boot to see if there are any tunable value errors.
Feb 3 09:23:33 eosmsm1 vmunix: Warning: Failed to resize up (grow) the Metadata Memory Resource Group.Error code 12.


eosmsm1@root:/> kctune filecache_min=800MB filecache_max=800MB
* The automatic 'backup' configuration has been updated.
* The requested changes have been applied to the currently
running configuration.
WARNING: The previous change of the tunable 'filecache_max' to '400MB',
which was being held for next boot, has been discarded.
WARNING: The previous change of the tunable 'filecache_min' to '400MB',
which was being held for next boot, has been discarded.
Tunable Value Expression Changes
filecache_max (before) 4070334464 Default Imm (auto disabled)
(now) 838860800 800MB
filecache_min (before) 407031808 Default Imm (auto disabled)
(now) 838860800 800MB
eosmsm1@root:/>
eosmsm1@root:/> kcusage
Tunable Usage / Setting
=============================================
filecache_max 709681152 / 838860800
maxdsiz 5369856 / 1073741824
maxdsiz_64bit 48398336 / 17179869184
maxfiles_lim 78 / 4096
maxssiz 786432 / 8388608
maxssiz_64bit 786432 / 268435456
maxtsiz 7221248 / 100663296
maxtsiz_64bit 201326592 / 1073741824
maxuprc 24 / 4096
max_thread_proc 40 / 3000
msgmni 2 / 4096
msgtql 0 / 4096
nflocks 75 / 8192
ninode 1446 / 139264
nkthread 398 / 10000
nproc 169 / 4200
npty 1 / 60
nstrpty 3 / 60
nstrtel 0 / 60
nswapdev 1 / 32
nswapfs 0 / 32
semmni 30 / 8192
semmns 227 / 16384
shmmax 4311756800 / 17179869184
shmmni 10 / 512
shmseg 3 / 300
eosmsm1@root:/>
Regards
Satheesh
HP-UX/SAN Admin

[Shibin_2]

The kernel parameters filecache_min and max changes will only take effect on the next reboot time. So, you cannot get any immediate effect.

Btw, from the top output, vhand is sleeping; it's not eating any memory. You may please check, what other program is holding memory.

Try to use glance -m or other options.

[jsk_cbe]

Thanks for reply.
But i think the filecache_min taken effect already,as per the message and values.
I dont have glance.
Memory utilisation is like this from SMH of hp-ux.

Total Physical Memory Size 8.0 GB
Free Physical Memory Size 37.1 MB
Total Virtual Memory Size 39.6 GB
Free Virtual Memory Size 22.4 GB
Total Swap Space Size 39.6 GB

eosmsm1@root:/> swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 32768 5738 27030 18% 0 - 1 /dev/vg00/lvol2
reserve - 9345 -9345
memory 7764 2407 5357 31%
total 40532 17490 23042 43% - 0 -
any other suggestions.?

[Shibin_2]

Something is holding your memory. Use the following command to find, what are the top eaters.

UNIX95= ps -el -o pid,comm,sz | grep -i -v pid | sort -nr -k 3,3 | head -15

Other commands reference you can get it from this thread.

http://forums13.itrc.hp.com/service/forums/parseCurl.do?admit=109447627+1265180989370+28353475&CURL=%2Fcm%2FQuestionAnswer%2F1%2C%2C0xa56442308663d611abdb0090277a778c%2C00.html

Tnx

[Wim Rombauts]

I think the very first things to check are :
What is the SGA size of your database ?
What is the Cache size of SAP ?

One of them, or both, are way too high, and that causes your system to be so slow.

[BUPA IS]

Hello ,
Can you check and see if you have this patch on?
PHKL_38449: HANG vhand monopolizes cpu, kernel unable to resolve kernel memory pressure. It had something wrong with it as well and has been superceeded .

It is now part of the vm cumulative patch:
Patch Name: PHKL_40441 11.31 vm cumulative patch.

Mike

[jsk_cbe]

eosmsm1@root:/var/tmp> UNIX95= ps -el -o pid,comm,sz | grep -i -v pid | sort -nr -k 3,3 | head -15
29016 dw.sapSM1_DVEBMGS00 18144
29018 dw.sapSM1_DVEBMGS00 17426
29017 dw.sapSM1_DVEBMGS00 16442
29882 dw.sapSM1_DVEBMGS00 15276
29019 dw.sapSM1_DVEBMGS00 15058
29024 dw.sapSM1_DVEBMGS00 12841
8571 ora_q002_SM1 12718
29037 oracleSM1 12683
29052 oracleSM1 12680
29058 oracleSM1 12664
29884 oracleSM1 12649
29065 oracleSM1 12621
29031 oracleSM1 12603
29373 oracleSM1 12597
29040 oracleSM1 12593
eosmsm1@root:/var/tmp>