system is slow

[Fadia Almarei]

Dear All
my system is slow how can check the cause of this problem
BR,
Fadia

[RAC_1]

You have not provided any data. Do you have glance installed??

start with glance.
run glance and let us know which one is hitting the limit-cpu, network, swap, disk??

Anil

[Fadia Almarei]

no i do not know what glance is , but i see the top command and see that almost every thing is ok, and this is a shot from it
"
Load averages: 0.87, 0.78, 1.04
359 processes: 328 sleeping, 27 running, 4 zombies
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 1.24 30.0% 0.0% 15.3% 54.8% 0.0% 0.0% 0.0% 0.0%
1 0.49 14.7% 0.4% 6.2% 78.8% 0.0% 0.0% 0.0% 0.0%
--- ---- - - - - - - - -----
avg 0.87 22.4% 0.2% 10.7% 66.7% 0.0% 0.0% 0.0% 0.0%

Memory: 1393060K (750900K) real, 5661520K (3628408K) virtual, 3444K free Page#
1/30

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
1 ? 19960 oracle8i 154 20 667M 7196K sleep 0:03 9.86 7.30 oracleOA1
0 pts/1 17993 apps11i 154 20 68980K 13860K sleep 0:02 5.09 5.08 f60webmx
1 ? 2 root 128 20 32K 32K sleep 3:04 3.73 3.72 vhand
1 pts/1 8090 apps11i 154 20 78244K 17008K sleep 1:38 3.32 3.31 f60webmx
0 ? 6933 oracle8i 154 20 703M 1308K sleep 0:55 3.00 3.00 oracleOA1
0 ? 6901 oracle8i 154 20 706M 4660K sleep 0:35 2.93 2.92 oracleOA1
1 pts/1 3177 apps11i 152 20 251M 79972K run 12:52 2.26 2.252.41 java
0 ? 6931 oracle8i 154 20 665M 756K sleep 0:54 2.21 2.21 oracleOA1
1 pts/1 8590 apps11i 154 20 72780K 10352K sleep 0:56 2.06 2.05 f60webmx
0 ? 2791 apps11i 152 20 544M 45008K run 22:34 1.96 1.96 java

[Fadia Almarei]

by the way i have oracle data base on my system which is HPUX11i

[RAC_1]

Looks like you have cpu bottlenck. There are many oracle processed waiting for cpu. Also java process is taking a lot of memory.

do following command and post.

glance -B
glance -g

Anil

[HGN]

Hi

Like others who had mentioned here the reasons can be many , You had mentioned Oracle is running you might have to tune the kernel to use the resources.

http://docs.hp.com/hpux/onlinedocs/939/KCParms/KCparams.OverviewAll.html
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=665998

The first has kernel parameter tuning details and the second link for one of the thread which talks about kernel tuning. Hope both these things will be useful.

The main thing now qould be to find out what is taking the resource and which resource(memory,cpu or i/o ) which can determined by glance output.

Rgds

HGN

[Geoff Wild]

Check out your system with sar and vmstat:

For CPU:
sar 5 20

Disk:
sar -d 5 20

Memory/cpu:
vmstat 5 20

What are the zombies?

ps -el |grep Z

Rgds...Geoff

[Fadia Almarei]

1- the average output for cpu
Average 12 2 13 73

2-the average output for disk
Average c1t2d0 13.95 153.43 44 1366 423.98 15.14
Average c1t0d0 2.83 0.51 7 127 4.66 5.60
Average c2t0d0 16.38 0.51 35 880 4.84 7.81
Average c2t2d0 7.19 0.50 18 626 4.97 5.59

3- the output for virtual memoryvmstat
sar 5 20
procs memory page
faults cpu
r b w avm free re at pi po fr de sr in
sy cs us sy id
2 0 0 803094 3248 27 12 28 14 4 0 308 707
1632 586 8 2 90
1 0 0 830808 1079 13 0 117 134 6 0 1595 697
0691 680 38 9 53
1 0 0 830808 1067 7 0 43 96 8 0 1042 543
5111 555 4 4 92
1 0 0 798909 1093 28 0 80 32 89 0 379 587
7776 588 7 3 90
1 0 0 798909 1058 27 17 67 39 29 0 226 559
4127 558 1 1 98
1 4 0 827266 1367 13 4 23 19 8 0 76 492
2691 494 1 1 98
1 4 0 827266 1353 10 1 9 5 1 0 24 504
2441 500 10 2 88
1 1 0 813939 1339 2 0 5 0 0 0 7 458
5163 472 5 0 95
1 1 0 813939 1328 6 0 3 0 0 0 1 448
3023 456 2 1 98
3 4 0 756982 1315 0 0 3 0 0 0 0 486
3757 530 6 1 93
3 4 0 756982 1190 5 0 44 91 0 0 33 670
5045 648 17 2 81
2 3 0 886670 3334 4 0 20 865 293 0 7548 899
4016 778 30 6 65
2 3 0 886670 8679 24 19 52 387 198 0 2604 860
4804 945 37 3 60
8 0 0 870850 8672 18 6 21 125 64 0 852 589
2707 631 7 1 92
8 0 0 870850 8623 14 3 9 40 20 0 278 466
2449 480 4 4 92
2 0 0 777558 9545 16 0 12 12 5 0 89 593
1457 626 18 6 75
2 0 0 777558 9119 13 1 8 3 0 0 28 527
7227 646 15 1 84
1 0 0 831414 9084 6 0 21 0 0 0 8 475
7413 659 8 1 91
1 0 0 831414 8327 18 1 71 0 0 0 1 534
4690 560 5 1 93
2 0 0 843072 6621 20 0 86 0 0 0 0 516
3719 560 9 1 90

4-$ ps -el | grep Z
F S UID PID PPID C PRI NI ADDR SZ WCHAN TT
Y TIME COMD

[Fadia Almarei]

another point that i do not have glance on my system ,so how could i get it

[Kyri Pilavakis]

You have 4 Zombies...Kill those Zombies and see the difference... ps -el | grep Z.
Hopefully once removed your system will be ok, otherwise more investigation is needed.

kyris

[Fadia Almarei]

first what is zombines and how to kill them and what do they affect the system

[Kyri Pilavakis]

In simple terms Zombies are dead processes which should be monitored and removed. Otherwise they can create havoc with your system's performance.
If they are still on your system, remove them using the kill command or kill -9 command. If they keep re-appearing then you need to find out what causes them.
kyris

[Alzhy]

First the initial assessment based on what you've submitted:

1. You've significant WaitIO's
2. You have occasional page scans (sscan rate at times are unusually high)
3. Disk c1t2d0 (OS/Swap disk?) has very significant queuing!

Possibles:

Memory lack or improper memory configuration
- what is your memory? (swapinfo -atm)
- what is your SHM settings? Do you have SHMMAX set to contain your Oracle instance's SGA?
- What is the size of your SGA?
- Is c2t1d0 your OS disk? (send lvlnboot -v output)

Please send outputs of:

swapinfo -atm
cat /stand/system
ipcs -a
model

[Fadia Almarei]

1-# swapinfo
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 978508 3215796 23% 0 - 1 /dev/vg00/lvol2
reserve - 2636100 -2636100
memory 1563480 385292 1178188 25%

2-# cat /stand/system
* Drivers and Subsystems

CentIf
CharDrv
DlkmDrv
GSCtoPCI
PCItoPCI
arp
asio0
asp
autofsc
beep
bs_osm
btlan
c720
c8xx
cachefsc
cb
cdfs
cifs
clone
core
cxperf
dev_config
dev_olar
devkrs
diag0
diag1
diag2
dlkm
dlpi
dmem
dmp
echo
evp
fcT1
fcT1_cntl
fcT1_fcp
fcms
fcp
fcp_cdio
fcparray
fcpdev
fcpmux
fddi4
ffs
foreign
framebuf
func0
gelan
graph3
hcd
hid
hpstreams
hub
i2o_cdio
iether
igelan
inet
iomem
iop_drv
ip
ipmi
ipmi_psm
ite
kepd
klog
krs
lasi
lba
ldterm
lv
lvm
maclan
netdiag1
netqa
nfs_client
nfs_core
nfs_server
nfsm
nms
nuls
olar_psm
olar_psm_if
pa_generic_psm
pa_psm
pat_psm
pci
pckt
pipedev
pipemod
prm
ptem
ptm
pts
rawip
rpcmod
sad
sapic
sba
sc
sctl
sdisk
sio
stape
stcpmap
strlog
strpty_included
strtelnet_included
tape2
tape2_included
tcp
td
telm
tels
timod
tirdwr
tlclts
tlcots
tlcotsod
token_arp
tun
udp
ufs
uipc
usbd
vol
vols
vxdmp
vxfs
vxportal
vxvm
wsio


* Kernel Device info

dump lvol


* Tunable parameters

STRMSGSZ 65535
dnlc_hash_locks 512
maxdsiz_64bit 0X40000000
maxfiles 2048
maxssiz 0X800000
maxssiz_64bit 0X800000
maxswapchunks 4096
maxtsiz 0X4000000
maxtsiz_64bit 0X40000000
maxuprc 2146
maxusers 510
nfile (15*NPROC+2048)
nflocks 4096
npty 1024
nstrpty 1024
nstrtel 1024
semmni 3320
semmns 6640
shmmax 0X40000000
shmmni 512
shmseg 32

3-# ipcs -a
IPC status from /dev/kmem as of Wed Nov 3 09:37:53 2004
T ID KEY MODE OWNER GROUP CREATOR CGROUP CBYTES
QNUM QBYTES LSPID LRPID STIME RTIME CTIME
Message Queues:
q 0 0x3c1c02ac -Rrw--w--w- root root root root 0
0 16384 0 0 no-entry no-entry 15:49:02
q 1 0x3e1c02ac --rw-r--r-- root root root root 0
0 264 0 0 no-entry no-entry 15:49:02
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH
SEGSZ CPID LPID ATIME DTIME CTIME
Shared Memory:
m 0 0x411c025e --rw-rw-rw- root root root root 0
348 539 539 15:49:08 15:49:08 15:49:02
m 1 0x4e0c0002 --rw-rw-rw- root root root root 2
61760 539 541 7:08:26 15:49:08 15:49:02
m 2 0x4120180e --rw-rw-rw- root root root root 2
8192 539 541 7:08:26 15:49:02 15:49:02
m 3 0x301c63c0 --rw-rw-rw- root root root root 3
1048576 1084 1098 15:50:59 no-entry 15:49:24
m 5636 0x49182c35 --rw-r--r-- root root root root 0
22908 1268 1268 15:49:51 15:49:51 15:49:51
m 2053 0x431c39d0 --rw-rw-rw- daemon root daemon root 2
5767168 1426 1484 11:15:45 11:15:44 15:49:54
m 3590 0x00000000 D-rw------- root root root root 7
1052672 1438 1438 15:49:59 no-entry 15:49:59
m 1543 0x00000000 D-rw------- www other root root 7
184324 1443 1443 15:50:02 no-entry 15:50:02
m 23560 0xad582298 --rw-r----- oracle8i dba oracle8i dba 117
594780160 12987 22262 9:35:58 9:36:19 3:45:35
m 521 0x00000000 D-rw------- apps11i dba apps11i dba 10
2658308 13408 13408 7:00:08 no-entry 7:00:08
m 522 0x00000000 D-rw------- apps11i dba apps11i dba 8
2658308 13431 13431 7:00:10 no-entry 7:00:10
T ID KEY MODE OWNER GROUP CREATOR CGROUP NSEMS
OTIME CTIME
Semaphores:
s 0 0x411c025e --ra-ra-ra- root root root root 1 1
5:49:08 15:49:02
s 1 0x4e0c0002 --ra-ra-ra- root root root root 2 1
5:49:04 15:49:02
s 2 0x4120180e --ra-ra-ra- root root root root 2
7:08:26 15:49:02
s 3 0x00446f6e --ra-r--r-- root root root root 1 n
o-entry 15:49:11
s 4 0x00446f6d --ra-r--r-- root root root root 1 n
o-entry 15:49:11
s 5 0x01090522 --ra-r--r-- root root root root 1 n
o-entry 15:49:11
s 6 0x311c63c0 --ra-ra-ra- root root root root 1 1
5:50:59 15:49:24
s 7 0x4184df5b --ra-ra-ra- root root root root 1 n
o-entry 15:49:31
s 8 0x4184df5c --ra-ra-ra- root root root root 1 1
5:49:31 15:49:31
s 9 0x61200ded --ra-ra-ra- root root root root 1 1
5:50:26 15:49:44
s 10 0x73200ded --ra-ra-ra- root root root root 1
2:30:48 15:49:44
s 11 0x70200ded --ra-ra-ra- root root root root 1
2:30:48 15:49:44
s 12 0x69200ded --ra-ra-ra- root root root root 1 1
5:50:26 15:49:44
s 13 0x75200ded --ra-ra-ra- root root root root 1 1
5:49:45 15:49:44
s 14 0x63200ded --ra-ra-ra- root root root root 1 1
5:49:45 15:49:44
s 15 0x64200ded --ra-ra-ra- root root root root 1 1
5:50:28 15:49:44
s 16 0x66200ded --ra-ra-ra- root root root root 1 n
o-entry 15:49:44
s 17 0x6c200ded --ra-ra-ra- root root root root 1 1
5:50:26 15:49:44
s 18 0x6d200ded --ra-ra-ra- root root root root 1 1
5:57:04 15:49:44
s 19 0x6f200ded --ra-ra-ra- root root root root 1 n
o-entry 15:49:44
s 20 0x410c0508 --ra-ra-ra- root root root root 1 1
5:49:47 15:49:47
s 21 0x49182c35 --ra-r--r-- root root root root 1 1
5:49:51 15:49:51
s 3342 0x4c1c39d0 --ra-r--r-- daemon root daemon root 1 1
5:49:53 15:49:53
s 23 0x52200ded --ra-ra-ra- root root root root 1 1
5:57:04 15:50:28
s 24 0x4184dfb3 --ra-ra-ra- root root root root 1 n
o-entry 15:50:59
s 25 0x4184dfb4 --ra-ra-ra- root root root root 1 1
5:50:59 15:50:59
s 49826 0xe4ee60c0 --ra-r----- oracle8i dba oracle8i dba 204
9:37:52 3:45:36

4-# model
9000/800/L1500-7x

[Ted Buis]

The HP-UX performance cookbook might help.
Check your dynamic buffer cache kernel parameters. I would recommend OnlineJFS with Oracle. See details in the cookbook.

[Alzhy]

Fadia,

It looks like you have a serious memory issue to start with. And you have both a an Oralce DB instance with 500+ MB SGA plus APP server. Your sar stats and disk stats (is c1t2d0 your OS disk?) indicate memory lack due to high memory scan rates and page outs. BTW is this a newly migrated system? If not, has this been this "slow" before? Is this an 11.0 or 11i system?

You may want to do the following to free up some memory if you cannot add to the (2GB? of memory you have)

trim down max_dbc_pct to say 20%.
enable direct IO on your Oracle Datafile mountpoints .


Also, based on your IPCS output.. it appears you may have 150+ Oracle sessions -- is this normal for this environment? have you rebooted this machine?

[Fadia Almarei]

thanks nelson
my system is HP 11i, and it is the first time to have this problem ,and i reboot the machine .