topic Re: High CPU utilization in Operating System - HP-UX

High CPU utilization

yulianto piyut — Tue, 05 Oct 2010 11:05:27 GMT

hi all,

We have server rx6600 run 11.31, used for java based application. The cpu utilization is very very very roughly and there is a question from us, when idle cpu under 5 % (output from vmstat), this server can not make a connection to the other server, like telnet or ssh.

Re: High CPU utilization

Hakki Aydin Ucar — Tue, 05 Oct 2010 13:37:59 GMT

if you know which processes use how much cpu and you can make a decision hw upgrade like memory OR add CPU to the system.
if you have glance, you can use it to monitor.

And you can check old threads about it:
http://www13.itrc.hp.com/service/james/search.do?prevQueryString=high+cpu+and+telnet&docType=Advisories&docType=Manuals&docType=EngineerNotes&docType=WhitePapers&docType=SecurityBulletins&docType=Patch&docType=Drivers&docType=Forums&dateSearchType=lastXdays&dateRange=all&startDateYear=1999&startDateMonth=0&startDateDay=1&endDateYear=2010&endDateMonth=9&endDateDay=5&resPerPage=20&sortCrit=rank&searchString=high+cpu+&submit1=Search+%C2%BB&searchCrit=allwords

Re: High CPU utilization

Patrick Wallek — Tue, 05 Oct 2010 14:29:31 GMT

>>server can not make a connection to the other server, like telnet or ssh.

Is this a statement that you cannot connect? Or is this a question as to whether or not it will connect?

CPU Usage should not affect your ability to connect to other servers from this. If the server is busy it may take a little longer to connect, but you should still be able to connect.

Re: High CPU utilization

yulianto piyut — Wed, 06 Oct 2010 01:57:29 GMT

hi patrick,

after idle cpu reach under 5%, my system can not make connection to the other server.

Re: High CPU utilization

Patrick Wallek — Wed, 06 Oct 2010 02:08:13 GMT

What happens when it tries to connect? Does it time out? Do you get an error? If you get an error, what is it?

Re: High CPU utilization

Kapil Jha — Wed, 06 Oct 2010 02:32:45 GMT

Is it like always or sometime.....how u come to a conclusion that when CPU is IDLE 5% ie its used for 95%, server can not make connection,

did you check what is the process taking other 95% of CPU, the number of connection.

CPU usage does not seems very relevent to connection making, probably process??

BR,
Kapil+

Re: High CPU utilization

yulianto piyut — Wed, 06 Oct 2010 03:56:14 GMT

I run vmstat in putty session, when i see idle cpu under 5, i try to make connection from this server to other, for example ssh to other server.

$ ssh 10.1.81.102
ssh: connect to host 10.1.81.102 port 22: Can't assign requested address
$ ssh 10.1.81.102
ssh: connect to host 10.1.81.102 port 22: Can't assign requested address
$ ssh 10.1.81.102
ssh: connect to host 10.1.81.102 port 22: Can't assign requested address

when idle cpu is high, we can make connection

$ ssh 10.1.81.102
Enter PASSCODE:

This issue came from my application team, when server have idle cpu under 5%, the application can not make connection to other application.
the usage cpu in this server is very roughly:

procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
65 11 0 1714665 2110625 0 0 0 2 0 0 0 6379 353346 7069 8 91 1
18 29 0 1703773 2110018 0 0 0 4 0 0 0 5352 211788 4520 8 90 1
18 29 0 1703773 2110023 0 0 0 3 0 0 0 5541 176018 5371 7 92 1
85 1 0 1704624 2110400 0 0 0 1 0 0 0 4735 562224 3148 11 88 1
85 1 0 1704624 2110142 0 0 0 0 0 0 0 5281 244355 2779 6 93 1
44 1 0 1660340 2108558 0 0 0 2 0 0 0 5159 201377 3785 8 91 0
44 1 0 1660340 2108571 0 0 0 2 0 0 0 5401 143178 2611 8 91 2
67 1 0 1664617 2109336 0 0 0 0 0 0 0 5040 293371 25450 13 74 13
67 1 0 1664617 2114123 0 0 0 7 0 0 0 8869 311093 12737 18 5 78
8 0 0 1670032 2107441 0 0 0 1 0 0 0 11863 941897 14195 45 5 50
8 0 0 1670032 2107972 0 0 0 0 0 0 0 15961 1594273 20221 60 7 33
46 0 0 1712046 2107913 0 0 0 3 0 0 0 18016 1693501 21419 57 8 35
46 0 0 1712046 2108934 0 0 0 6 0 0 0 9977 824418 10322 8 88 4
53 3 0 1706961 2108931 0 0 0 1 0 0 0 6696 395754 6385 10 88 2
53 3 0 1706961 2109007 0 0 0 0 0 0 0 5367 284599 5882 8 91 1

Re: High CPU utilization

Kapil Jha — Wed, 06 Oct 2010 04:26:16 GMT

Pretty strange, can you please check if at this time ssh is taking high CPU?

One more thing you experience connection problem for ssh or for other other ports as well.

Probably try
telnet IP port_number

at high CPU usage.

BR,
Kapil+

Re: High CPU utilization

yulianto piyut — Wed, 06 Oct 2010 05:40:54 GMT

Hi Kapil,

I have tried to make telnet, ssh to another server in high cpu utilization, and failed.
Process ssh is small, from top output, top of five is java, because application run in java platform.

System: vasprdpapp6 Wed Oct 6 13:38:21 2010
Load averages: 4.87, 5.02, 4.94
201 processes: 152 sleeping, 48 running, 1 zombie
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 4.58 10.9% 0.0% 89.1% 0.0% 0.0% 0.0% 0.0% 0.0%
2 5.09 7.9% 0.0% 92.1% 0.0% 0.0% 0.0% 0.0% 0.0%
4 5.28 9.9% 0.0% 90.1% 0.0% 0.0% 0.0% 0.0% 0.0%
6 4.95 5.9% 0.0% 94.1% 0.0% 0.0% 0.0% 0.0% 0.0%
8 5.01 2.0% 0.0% 98.0% 0.0% 0.0% 0.0% 0.0% 0.0%
10 4.85 5.0% 0.0% 95.0% 0.0% 0.0% 0.0% 0.0% 0.0%
12 4.64 8.9% 0.0% 90.1% 1.0% 0.0% 0.0% 0.0% 0.0%
14 4.57 7.9% 0.0% 92.1% 0.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 4.87 7.8% 0.0% 92.2% 0.0% 0.0% 0.0% 0.0% 0.0%

System Page Size: 4Kbytes
Memory: 3235516K (2961064K) real, 7150796K (6685088K) virtual, 8431600K free Page# 1/41

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
8 ? 19967 apps 152 20 1305M 890M run 44660:51 376.46 375.80 java
4 ? 15502 apps 152 20 1172M 796M run 1289:27 249.56 249.12 java
2 ? 19547 apps 152 20 1407M 376M run 45:51 3.00 2.99 java
2 ? 20928 apps 152 20 1399M 361M run 52:16 2.76 2.75 java

need for some kernel configuration to tune?

bash-4.0# kctune
Tunable Value Expression Changes
NSTREVENT 50 Default
NSTRPUSH 16 Default
NSTRSCHED 0 Default
STRCTLSZ 1024 Default
STRMSGSZ 0 Default
acctresume 4 Default
acctsuspend 2 Default
aio_iosize_max 0 Default Immed
aio_listio_max 256 Default Immed
aio_max_ops 2048 Default Immed
aio_monitor_run_sec 30 Default Immed
aio_physmem_pct 10 Default Immed
aio_prio_delta_max 20 Default Immed
aio_proc_max 0 Default Immed
aio_proc_thread_pct 70 Default Immed
aio_proc_threads 1024 Default Immed
aio_req_per_thread 1 Default Immed
allocate_fs_swapmap 0 Default
alwaysdump 0 Default Immed
audit_memory_usage 5 Default Immed
audit_track_paths 0 Default Auto
copy_on_write 1 Default Immed
core_addshmem_read 0 Default Immed
core_addshmem_write 0 Default Immed
create_fastlinks 0 Default
default_disk_ir 0 Default
diskaudit_flush_interval 5 Default Immed
dma32_pool_size 268435456 Default
dmp_rootdev_is_vol 0 Default
dmp_swapdev_is_vol 0 Default
dnlc_hash_locks 512 Default
dontdump 0 Default Immed
dst 1 Default
dump_compress_on 1 Default Immed
dump_concurrent_on 1 Default Immed
executable_stack 0 Default Immed
expanded_node_host_names 1 1 Immed
fcache_fb_policy 0 Default Immed
fcache_seqlimit_file 100 Default Immed
fcache_seqlimit_system 100 Default Immed
fcd_disable_mgmt_lun 0 Default Immed
fclp_ifc_disable_mgmt_lun 0 Default Immed
filecache_max 815521996 5% Imm (auto disabled)
filecache_min 489313197 3% Imm (auto disabled)
fr_statemax 800000 Default
fr_tcpidletimeout 86400 Default
fs_async 0 Default
fs_symlinks 20 Default Immed
ftable_hash_locks 64 Default
gvid_no_claim_dev 0 Default
hires_timeout_enable 0 Default Immed
hp_hfs_mtra_enabled 1 Default
intr_strobe_ics_pct 80 Default Immed
io_ports_hash_locks 64 Default
ipf_icmp6_passthru 0 Default
ipl_buffer_sz 8192 Default
ipl_logall 0 Default
ipl_suppress 1 Default
ipmi_watchdog_action 0 Default Immed
ksi_alloc_max 33600 Default Immed
ksi_send_max 32 Default
lcpu_attr 0 Default Auto
max_acct_file_size 2560000 Default Immed
max_async_ports 4096 Default Immed
max_mem_window 0 Default Immed
max_thread_proc 3000 3000 Immed
maxdsiz 4000000000 4000000000 Immed
maxdsiz_64bit 8589934592 8589934592 Immed
maxfiles 60000 60000
maxfiles_lim 60000 60000 Immed
maxrsessiz 8388608 Default
maxrsessiz_64bit 8388608 Default
maxssiz 209715200 209715200 Immed
maxssiz_64bit 268435456 Default Immed
maxtsiz 1073741824 1073741824 Immed
maxtsiz_64bit 1073741824 1073741824 Immed
maxuprc 1024 1024 Immed
mca_recovery_on 1 Default Auto
msgmbs 8 Default Immed
msgmnb 16384 Default Immed
msgmni 512 Default Immed
msgtql 1024 Default Immed
ncdnode 150 Default
nclist 8292 Default
ncsize 8976 Default
nflocks 4096 Default Auto
nfs2_max_threads 8 Default Immed
nfs2_nra 4 Default Immed
nfs3_bsize 32768 Default Immed
nfs3_do_readdirplus 1 Default Immed
nfs3_jukebox_delay 1000 Default Immed
nfs3_max_threads 8 Default Immed
nfs3_max_transfer_size 1048576 Default Immed
nfs3_max_transfer_size_cots 1048576 Default Immed
nfs3_nra 4 Default Immed
nfs4_bsize 32768 Default Immed
nfs4_max_threads 8 Default Immed
nfs4_max_transfer_size 1048576 Default Immed
nfs4_max_transfer_size_cots 1048576 Default Immed
nfs4_nra 4 Default Immed
nfs_portmon 0 Default Immed
ninode 8192 Default
nkthread 28723 28723 Immed
nproc 16404 16404 Immed
npty 60 Default
nstrpty 60 Default
nstrtel 60 Default
nswapdev 32 Default
nswapfs 32 Default
numa_policy 0 Default Immed
pa_maxssiz_32bit 83648512 Default
pa_maxssiz_64bit 536870912 Default
pagezero_daemon_enabled 1 Default Immed
patch_active_text 1 Default Immed
pci_eh_enable 1 Default
pci_error_tolerance_time 1440 Default Immed
process_id_max 30000 Default Auto
process_id_min 0 Default Auto
remote_nfs_swap 0 Default
rng_bitvals 9876543210 Default
rng_sleeptime 2 Default
rtsched_numpri 32 Default
sched_thread_affinity 6 Default Immed
scroll_lines 100 Default
secure_sid_scripts 1 Default Immed
semaem 16384 Default
semmni 2048 Default
semmns 4096 Default
semmnu 256 Default
semmsl 2048 Default Immed
semume 100 Default
semvmx 32767 Default
shmmax 1073741824 Default Immed
shmmni 400 Default Immed
shmseg 300 Default Immed
streampipes 0 Default
swchunk 4096 4096
sysv_hash_locks 128 Default
tcphashsz 0 Default
timeslice 10 Default
timezone 420 Default
uname_eoverflow 0 0 Immed
vnode_cd_hash_locks 128 Default
vnode_hash_locks 128 Default
vol_checkpt_default 10240 Default
vol_dcm_replay_size 262144 Default
vol_default_iodelay 50 Default
vol_fmr_logsz 4 Default
vol_max_bchain 32 Default
vol_max_nconfigs 20 Default
vol_max_nlogs 20 Default
vol_max_nmpool_sz 4194304 Default Immed
vol_max_prm_dgs 1024 Default
vol_max_rdback_sz 4194304 Default Immed
vol_max_vol 8388608 Default
vol_max_wrspool_sz 4194304 Default Immed
vol_maxio 256 Default
vol_maxioctl 32768 Default
vol_maxkiocount 2048 Default
vol_maxparallelio 256 Default
vol_maxspecialio 256 Default
vol_maxstablebufsize 256 Default
vol_min_lowmem_sz 532480 Default Immed
vol_mvr_maxround 256 Default
vol_nm_hb_timeout 10 Default
vol_rootdev_is_vol 0 Default
vol_rvio_maxpool_sz 4194304 Default Immed
vol_subdisk_num 4096 Default
vol_swapdev_is_vol 0 Default
vol_vvr_transport 1 Default
vol_vvr_use_nat 0 Default
volcvm_cluster_size 16 Default
volcvm_smartsync 1 Default
voldrl_max_drtregs 2048 Default
voldrl_min_regionsz 512 Default
voliomem_chunk_size 65536 Default
voliomem_maxpool_sz 4194304 Default
voliot_errbuf_dflt 16384 Default
voliot_iobuf_default 8192 Default
voliot_iobuf_limit 131072 Default
voliot_iobuf_max 65536 Default
voliot_max_open 32 Default
volpagemod_max_memsz 6144 Default Immed
volraid_rsrtransmax 1 Default
vps_ceiling 16 Default Immed
vps_chatr_ceiling 1048576 Default Immed
vps_pagesize 16 Default Immed
vx_maxlink 32767 Default
vx_ninode 0 Default Immed
vxfs_bc_bufhwm 0 Default Immed
vxfs_ifree_timelag 0 Default Immed
vxtask_max_monitors 32 Default
bash-4.0#

Re: High CPU utilization

Hakki Aydin Ucar — Wed, 06 Oct 2010 07:19:29 GMT

>Yulianta:ssh: connect to host 10.1.81.102 port 22: Can't assign requested address

Are you able to ping to this address at the time of CPU utily is %95 .

When you attempt to telnet did you get the same error OR what ?

Re: High CPU utilization

yulianto piyut — Wed, 06 Oct 2010 10:07:16 GMT

hakki,

for telnet have the same error. in that situation, system can ping to another server (10.1.81.102).

Re: High CPU utilization

Roland van Lochem — Wed, 06 Oct 2010 10:15:13 GMT

Hi,

A while ago we had the same error which kept us busy for quite some time.

Have a look at http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1366892

The error message indicates you probably have a lack of anonymous ports on the system.

Regards,
Roland

Re: High CPU utilization

yulianto piyut — Wed, 06 Oct 2010 11:23:59 GMT

Hi Roland,

Thx for your information, We have tried to add the range of ephemeral port range, default is 49152 through 65535, changed to 30000 through 65535. Idle cpu never reach under 5%, the connection from output netstat -an reach above 50000.

do you have link for download xportshow?

Re: High CPU utilization

Roland van Lochem — Thu, 07 Oct 2010 04:11:27 GMT

Hi Yulianto,
log a call at HP support and ask them for xportshow.
You can however find out if you are short of anonymous ports by extending the range and then check if connections can be made again. Then you will have to find out why.

Regards,
Roland