topic Re: CPU 100% in Operating System - HP-UX

CPU 100%

tom quach_1 — Mon, 13 Jun 2005 03:48:16 GMT

Hello,
Sorry for this long list.
this is a RP8400 with 2cpu and 16Gb of RAM.
The memory, disks activity are low, but the CPU usage was 99 to 100%.
Someone please check these parameters and let me know what is wrong with it.

Regards,
Tom

Parameter Current Dyn Planned Module Version
===============================================================================
NSTRBLKSCHED - - 2
NSTREVENT 50 - 50
NSTRPUSH 16 - 16
NSTRSCHED 0 - 0
STRCTLSZ 1024 - 1024
STRMSGSZ 65535 - 65535
acctresume 4 - 4
acctsuspend 2 - 2
aio_listio_max 256 - 256
aio_max_ops 2048 - 2048
aio_physmem_pct 10 - 10
aio_prio_delta_max 20 - 20
allocate_fs_swapmap 0 - 0
alwaysdump 1 - 1
bcvmap_size_factor 2 - 2
bootspinlocks - - 256
bufcache_hash_locks 128 - 128
bufpages 0 - (NBUF*2)
chanq_hash_locks 256 - 256
core_addshmem_read 0 Y 0
core_addshmem_write 0 Y 0
create_fastlinks 0 - 0
dbc_max_pct 10 - 10
dbc_min_pct 5 - 5
default_disk_ir 0 - 0
desfree - - 0
disksort_seconds 0 - 0
dmp_rootdev_is_vol 0 - 0
dmp_swapdev_is_vol 0 - 0
dnlc_hash_locks 512 - 512
dontdump 0 - 0
dskless_node - - 0
dst 1 - 1
effective_maxpid - - ((NPROC<=30000)?30000:(NPROC*5/4))
eisa_io_estimate - - 0x300
enable_idds 0 - 0
eqmemsize 15 - 15
executable_stack 1 - 1
fcp_large_config 0 - 0
file_pad - - 10
fs_async 0 - 0
ftable_hash_locks 64 - 64
hdlpreg_hash_locks 128 - 128
hfs_max_ra_blocks 8 - 8
hfs_max_revra_blocks 8 - 8
hfs_ra_per_disk 64 - 64
hfs_revra_per_disk 64 - 64
hp_hfs_mtra_enabled 1 - 1
hpux_aes_override - - 0
initmodmax 50 - 50
io_ports_hash_locks 64 - 64
iomemsize - - 40000
ksi_alloc_max 32928 - (NPROC*8)
ksi_send_max 32 - 32
lotsfree - - 0
max_async_ports 50 - 50
max_fcp_reqs 512 - 512
max_mem_window 0 - 0
max_thread_proc 64 - 64
maxdsiz 0x10000000 - 0x10000000
maxdsiz_64bit 0x40000000 - 0X40000000
maxfiles 60 - 60
maxfiles_lim 1024 Y 1024
maxqueuetime - - 0
maxssiz 0x800000 - 0X800000
maxssiz_64bit 0x800000 - 0X800000
maxswapchunks 2048 - 2048
maxtsiz 0x4000000 Y 0X4000000
maxtsiz_64bit 0x40000000 Y 0X40000000
maxuprc 75 Y 75
maxusers 512 - 512
maxvgs 10 - 10
mesg 1 - 1
minfree - - 0
modstrmax 500 - 500
msgmap 42 - (2+MSGTQL)
msgmax 8192 Y 8192
msgmnb 16384 Y 16384
msgmni 50 - 50
msgseg 2048 - 2048
msgssz 8 - 8
msgtql 40 - 40
nbuf 0 - 0
ncallout 7235 - (16+NKTHREAD)
ncdnode 150 - 150
nclist 8292 - (100+16*MAXUSERS)
ncsize 9916 - (NINODE+VX_NCSIZE)+(8*DNLC_HASH_LOCKS)
ndilbuffers 30 - 30
netisr_priority - - -1
netmemmax - - 0
nfile 7822 - (16*(NPROC+16+MAXUSERS)/10+32+2*(NPTY+NSTRPTY+NSTRTEL))
nflocks 200 - 200
nhtbl_scale 0 - 0
ninode 4796 - ((NPROC+16+MAXUSERS)+32+(2*NPTY))
nkthread 7219 - (((NPROC*7)/4)+16)
nni - - 2
no_lvm_disks 0 - 0
nproc 4116 - (20+8*MAXUSERS)
npty 60 - 60
nstrpty 60 - 60
nstrtel 60 - 60
nswapdev 10 - 10
nswapfs 10 - 10
nsysmap 8232 - ((NPROC)>800?2*(NPROC):800)
nsysmap64 8232 - ((NPROC)>800?2*(NPROC):800)
o_sync_is_o_dsync 0 - 0
page_text_to_local - - 0
pfdat_hash_locks 128 - 128
public_shlibs 1 - 1
region_hash_locks 128 - 128
remote_nfs_swap 0 - 0
rtsched_numpri 32 - 32
scroll_lines 100 - 100
scsi_max_qdepth 8 Y 8
scsi_maxphys 1048576 - 1048576
sema 1 - 1
semaem 16384 - 16384
semmap 66 - (SEMMNI+2)
semmni 64 - 64
semmns 128 - 128
semmnu 30 - 30
semmsl 2048 Y 2048
semume 10 - 10
semvmx 32767 - 32767
sendfile_max 0 - 0
shmem 1 - 1
shmmax 0x4000000 Y 0X4000000
shmmni 200 - 200
shmseg 120 Y 120
st_ats_enabled 0 - 0
st_fail_overruns 0 - 0
st_large_recs 0 - 0
st_san_safe 0 - 0
streampipes 0 - 0
swapmem_on 1 - 1
swchunk 2048 - 2048
sysv_hash_locks 128 - 128
tcphashsz 0 - 0
timeslice 10 - (100/10)
timezone 420 - 420
unlockable_mem 0 - 0
vas_hash_locks 128 - 128
vnode_cd_hash_locks 128 - 128
vnode_hash_locks 128 - 128
vol_checkpt_default 10240 - 10240
vol_dcm_replay_size 262144 - (256*1024)
vol_default_iodelay 50 - 50
vol_fmr_logsz 4 - 4
vol_max_bchain 32 - 32
vol_max_nconfigs 20 - 20
vol_max_nlogs 20 - 20
vol_max_nmpool_sz 4194304 - (4*1024*1024)
vol_max_prm_dgs 1024 - 1024
vol_max_rdback_sz 4194304 - (4*1024*1024)
vol_max_vol 8388608 - (8*1024*1024)
vol_maxio 256 - 256
vol_maxioctl 32768 - 32768
vol_maxkiocount 2048 - 2048
vol_maxparallelio 256 - 256
vol_maxspecialio 256 - 256
vol_maxstablebufsize 256 - 256
vol_min_lowmem_sz 524288 - (512*1024)
vol_mvr_maxround 256 - 256
vol_nm_hb_timeout 10 - (10)
vol_subdisk_num 4096 - 4096
vol_vvr_transport 1 - (1)
vol_vvr_use_nat 0 - (0)
volcvm_cluster_size 16 - 16
volcvm_smartsync 1 - 1
voldrl_max_drtregs 2048 - 2048
voldrl_min_regionsz 512 - 512
voliomem_chunk_size 65536 - (64*1024)
voliomem_maxpool_sz 4194304 - (4*1024*1024)
voliot_errbuf_dflt 16384 - 16384
voliot_iobuf_default 8192 - 8192
voliot_iobuf_limit 131072 - 131072
voliot_iobuf_max 65536 - 65536
voliot_max_open 32 - 32
volraid_rsrtransmax 1 - 1
vps_ceiling 16 - 16
vps_chatr_ceiling 1048576 - 1048576
vps_pagesize 4 - 4
vx_fancyra_enable 0 - 0
vx_maxlink 32767 - 32767
vx_ncsize 1024 - 1024
vx_ninode 0 - 0
vxfs_max_ra_kbytes 1024 - 1024
vxfs_ra_per_disk 1024 - 1024
vxtask_max_monitors 32 - 32

Re: CPU 100%

Laurent Menase — Mon, 13 Jun 2005 03:56:23 GMT

what process is running? ( look with top or glance )

Re: CPU 100%

Senthil Kumar .A_1 — Mon, 13 Jun 2005 04:05:01 GMT

Hi Tom,

One of the rules in performance diagnosis is "IT DEPENDS". Here you have mentioned that CPU usage is 100 % and you have given the lsit of Kernel parameters.

This info does not help the forumers to suggest what to tune unless pointed what you server is used for. I mean for what application.

Secondly , 100 % usage of CPU never indicates that there is a CPU bottle neck in your system. The 100 % usage in conjuction with the CPU "run queue" for stipulated amount of time will give us the true picture.

So please take the following output

sar -A -o /tmp/sarout 60 60

then attach the file /tmp/sarout here...

And the system performance is ofcourse subjective and need to consult your user community whether they are facing any response issues from the server.

if indeed there is a issue then we need to get the offending process by using "Glance plus"... app or...need to consider the performace tuning based on the app your are hosting..that includes both "syatem kernel tuning" aswell as "application tuning"..

Regards.

Re: CPU 100%

Eknath — Mon, 13 Jun 2005 04:13:18 GMT

Hi Tom,

If you have glance plus installed, check by running #glance
It will show you what causing the issue.

If glance plus is not available use #top

find out which process(s) eating more cpu time something like this shown in attachment. These will be on top (may be thats why name top)

Cheers !!!
eknath

Re: CPU 100%

Senthil Kumar .A_1 — Mon, 13 Jun 2005 04:18:07 GMT

Hi tom,

Before i'm pelted with stones for Plagiarism...

My earlier inputs where as a result of consulting ..a book..

"hp-ux 11i tuning and performance
by robert f.sauers,chris p.ruemmler and peter s.weygant".

If there is a complex performance problem ..cosider consulting this book and be prepared for the future..

This is worth a read......

Regards.

Re: CPU 100%

tom quach_1 — Mon, 13 Jun 2005 05:12:23 GMT

Thank you all for the info.
i know, it is hard to diagnosis with out further details. Just would like to ask you that is there any unusual setting in these parameters.
my glance and top were showing different.
the glance shown 100%cpu and the top shown something less. Do you know why?

Regards,
Tom

Re: CPU 100%

Laurent Menase — Mon, 13 Jun 2005 05:14:32 GMT

take glance.
top gives sampled data.

Re: CPU 100%

tom quach_1 — Tue, 14 Jun 2005 05:38:48 GMT

Hi All,
attached are sarout.ap1 and sarout.ap2
please check.
Regards,

Tom

Re: CPU 100%

tom quach_1 — Tue, 14 Jun 2005 05:39:42 GMT

Hi All,
this is sarout.ap2
Regards,

Tom

Re: CPU 100%

Pedro Cirne — Tue, 14 Jun 2005 05:51:15 GMT

Hi,

Post the output of the following:

sar -d 5 5
sar -u 5 5
top

Enjoy :-)

Re: CPU 100%

tom quach_1 — Tue, 14 Jun 2005 06:04:18 GMT

Hi Pedro,
right now is the end of the shift, so the systems are normal.I wil post it tomorrow when systems show 100% cpu .

Regards,

Tom

Re: CPU 100%

Cem Tugrul — Tue, 14 Jun 2005 08:37:56 GMT

Tom,
Have you ever had a patch management recently?
On the other hand,
if you use glance what is the PID of the process which busy on %100 cpu?
try to give more detail...

Good Luck,

Re: CPU 100%

KVK — Tue, 14 Jun 2005 14:44:34 GMT

I have seen your output their it's CPU IDLE time % is 99.8 . no problem . when you are finding the performance problem while user increases & application runs ? What type of application is running ?
please post the following output if system CPU usage 100%

#sar -M -u 5 10
#top -d2 -s10 -f /tmp/topperf.txt ==>post this doc

Re: CPU 100%

tom quach_1 — Wed, 15 Jun 2005 02:07:56 GMT

Hello,

Today, the system does not experience 100% cpu as yesterday.
Regards,
Tom

#top -d2 -s10 -f /tmp/topperf.txt

System: sunuxap1 Tue Jun 14 23:56:31 2005
Load averages: 0.26, 0.18, 0.12
279 processes: 265 sleeping, 14 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.27 29.8% 0.0% 3.9% 66.3% 0.0% 0.0% 0.0% 0.0%
1 0.25 23.2% 0.0% 4.2% 72.6% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.26 26.5% 0.0% 4.0% 69.5% 0.0% 0.0% 0.0% 0.0%

Memory: 588236K (117968K) real, 753224K (172188K) virtual, 12967332K free Page#
1/18

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 pts/ti 20872 teddy 154 20 31128K 5412K sleep 0:29 51.80 51.10 quiz
1 ? 1449 root -16 20 27440K 6516K run 16:25 0.69 0.69 midaemon
1 ? 35 root 152 20 7424K 7424K run 2:08 0.66 0.66 vxfsd
1 pts/tb 13766 yoyo 154 20 33896K 7520K sleep 0:05 0.64 0.64 PH8SRVN
0 pts/tb 21084 chis 154 20 13284K 2516K sleep 0:00 1.19 0.47 PH8SRVN
0 pts/tg 19846 root 154 10 27580K 3032K sleep 0:28 0.35 0.35 glance
0 pts/tb 16750 aska 154 20 33384K 7152K sleep 0:00 0.32 0.32 PH8SRVN
1 ? 3158 root 152 20 16340K 4520K run 0:07 0.16 0.16 rep_serv
er
1 ? 3162 root 152 20 13772K 2816K run 0:03 0.16 0.16 agdbserv
er
1 ? 1207 root 158 20 2036K 412K sleep 2:18 0.15 0.15 sh
0 ? 3165 root 152 20 13388K 2292K run 0:08 0.12 0.12 alarmgen
0 ? 1687 root 152 20 23040K 3912K run 0:06 0.10 0.10 vxsvc
0 pts/tb 15488 eliza 154 20 33256K 7008K sleep 0:00 0.09 0.09 PH8SRVN
1 ? 1283 root 154 20 2920K 468K sleep 0:18 0.08 0.08 pwgrd
0 ? 11421 batch 154 21 18996K 2448K sleep 0:00 0.08 0.08 fmqread
1 ? 1892 root 152 10 5328K 940K run 0:00 0.08 0.08 memlogd
System: sunuxap1 Tue Jun 14 23:56:41 2005
Load averages: 0.30, 0.19, 0.12
279 processes: 265 sleeping, 14 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.31 21.0% 0.0% 2.9% 76.2% 0.0% 0.0% 0.0% 0.0%
1 0.28 37.2% 0.0% 4.7% 58.1% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.30 29.1% 0.0% 3.8% 67.1% 0.0% 0.0% 0.0% 0.0%

Memory: 587860K (127672K) real, 752876K (184612K) virtual, 12967064K free Page#
1/18

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 pts/ti 20872 teddy 154 20 31128K 5412K sleep 0:34 53.02 52.58 quiz
0 pts/to 21146 khsiung 154 20 31156K 5228K sleep 0:00 4.08 1.06 qtp
0 pts/tb 21084 chis 154 20 32232K 5968K sleep 0:00 1.44 0.91 PH8SRVN
0 ? 1449 root -16 20 27440K 6516K run 16:25 0.76 0.75 midaemon
0 ? 35 root 152 20 7424K 7424K run 2:08 0.67 0.67 vxfsd
1 pts/tb 13766 yoyo 154 20 33896K 7520K sleep 0:05 0.45 0.45 PH8SRVN
0 pts/tb 16750 aska 154 20 33384K 7152K sleep 0:00 0.33 0.33 PH8SRVN
1 pts/tg 19846 root 154 10 27580K 3032K sleep 0:28 0.33 0.33 glance
1 pts/tb 15488 eliza 154 20 33256K 7008K sleep 0:00 0.32 0.32 PH8SRVN
1 ? 3158 root 152 20 16340K 4520K run 0:07 0.16 0.16 rep_serv
er
1 ? 3162 root 152 20 13772K 2816K run 0:03 0.16 0.16 agdbserv
er
1 pts/ta 21081 root 178 20 7012K 700K run 0:00 0.25 0.16 top
0 ? 1207 root 158 20 2036K 412K sleep 2:18 0.14 0.14 sh
0 ? 3165 root 152 20 13388K 2292K run 0:08 0.12 0.12 alarmgen
0 ? 1687 root 152 20 23040K 3912K run 0:06 0.10 0.10 vxsvc
1 ? 12843 root 154 20 2432K 376K sleep 0:09 0.09 0.09 cognos1d
sunuxap1:[root]:/home/teddy/WORK

#sar -d 5 5

HP-UX sunuxap1 B.11.11 U 9000/800 06/15/05

00:01:06 device %busy avque r+w/s blks/s avwait avserv
00:01:11 c0t6d0 2.99 0.50 7 57 4.45 11.99
c4t0d0 2.59 0.50 5 48 5.17 10.71
c2t6d0 1.40 1.25 6 190 7.22 14.99
c4t2d0 1.20 0.50 5 187 5.25 8.71
00:01:16 c0t6d0 2.20 0.50 4 26 4.51 9.77
c4t0d0 1.20 0.50 3 19 4.68 6.89
c2t6d0 0.20 0.50 0 2 0.31 9.81
00:01:21 c0t6d0 7.39 0.50 17 102 4.44 7.11
c4t0d0 5.39 0.50 13 87 4.53 5.69
c2t6d0 0.20 0.50 0 6 4.93 7.92
c4t2d0 0.20 0.50 0 6 4.95 7.16
00:01:26 c0t6d0 5.20 0.50 11 85 3.79 12.20
c4t0d0 3.80 0.50 7 67 3.72 11.74
c2t6d0 0.60 1.82 4 210 11.79 10.45
c4t2d0 0.60 1.55 4 208 7.30 5.50
00:01:31 c0t6d0 2.20 0.50 5 61 4.00 11.61
c4t0d0 1.80 0.50 3 50 4.95 10.36

Average c0t6d0 4.00 0.50 9 66 4.24 9.93
Average c4t0d0 2.96 0.50 6 54 4.51 8.43
Average c2t6d0 0.48 1.44 2 82 8.90 12.74
Average c4t2d0 0.40 0.96 2 80 6.13 7.25

#sar -u 5 5

HP-UX sunuxap1 B.11.11 U 9000/800 06/15/05

00:02:10 %usr %sys %wio %idle
00:02:15 33 8 1 58
00:02:20 33 15 0 52
00:02:25 34 10 1 55
00:02:30 34 14 0 52
00:02:35 33 11 1 56

Average 33 12 0 55