system is slow

[Fadia Almarei]

first what is zombines and how to kill them and what do they affect the system

[Kyri Pilavakis]

In simple terms Zombies are dead processes which should be monitored and removed. Otherwise they can create havoc with your system's performance.
If they are still on your system, remove them using the kill command or kill -9 command. If they keep re-appearing then you need to find out what causes them.
kyris

[Alzhy]

First the initial assessment based on what you've submitted:

1. You've significant WaitIO's
2. You have occasional page scans (sscan rate at times are unusually high)
3. Disk c1t2d0 (OS/Swap disk?) has very significant queuing!

Possibles:

Memory lack or improper memory configuration
- what is your memory? (swapinfo -atm)
- what is your SHM settings? Do you have SHMMAX set to contain your Oracle instance's SGA?
- What is the size of your SGA?
- Is c2t1d0 your OS disk? (send lvlnboot -v output)

Please send outputs of:

swapinfo -atm
cat /stand/system
ipcs -a
model

[Fadia Almarei]

1-# swapinfo
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 978508 3215796 23% 0 - 1 /dev/vg00/lvol2
reserve - 2636100 -2636100
memory 1563480 385292 1178188 25%

2-# cat /stand/system
* Drivers and Subsystems

CentIf
CharDrv
DlkmDrv
GSCtoPCI
PCItoPCI
arp
asio0
asp
autofsc
beep
bs_osm
btlan
c720
c8xx
cachefsc
cb
cdfs
cifs
clone
core
cxperf
dev_config
dev_olar
devkrs
diag0
diag1
diag2
dlkm
dlpi
dmem
dmp
echo
evp
fcT1
fcT1_cntl
fcT1_fcp
fcms
fcp
fcp_cdio
fcparray
fcpdev
fcpmux
fddi4
ffs
foreign
framebuf
func0
gelan
graph3
hcd
hid
hpstreams
hub
i2o_cdio
iether
igelan
inet
iomem
iop_drv
ip
ipmi
ipmi_psm
ite
kepd
klog
krs
lasi
lba
ldterm
lv
lvm
maclan
netdiag1
netqa
nfs_client
nfs_core
nfs_server
nfsm
nms
nuls
olar_psm
olar_psm_if
pa_generic_psm
pa_psm
pat_psm
pci
pckt
pipedev
pipemod
prm
ptem
ptm
pts
rawip
rpcmod
sad
sapic
sba
sc
sctl
sdisk
sio
stape
stcpmap
strlog
strpty_included
strtelnet_included
tape2
tape2_included
tcp
td
telm
tels
timod
tirdwr
tlclts
tlcots
tlcotsod
token_arp
tun
udp
ufs
uipc
usbd
vol
vols
vxdmp
vxfs
vxportal
vxvm
wsio


* Kernel Device info

dump lvol


* Tunable parameters

STRMSGSZ 65535
dnlc_hash_locks 512
maxdsiz_64bit 0X40000000
maxfiles 2048
maxssiz 0X800000
maxssiz_64bit 0X800000
maxswapchunks 4096
maxtsiz 0X4000000
maxtsiz_64bit 0X40000000
maxuprc 2146
maxusers 510
nfile (15*NPROC+2048)
nflocks 4096
npty 1024
nstrpty 1024
nstrtel 1024
semmni 3320
semmns 6640
shmmax 0X40000000
shmmni 512
shmseg 32

3-# ipcs -a
IPC status from /dev/kmem as of Wed Nov 3 09:37:53 2004
T ID KEY MODE OWNER GROUP CREATOR CGROUP CBYTES
QNUM QBYTES LSPID LRPID STIME RTIME CTIME
Message Queues:
q 0 0x3c1c02ac -Rrw--w--w- root root root root 0
0 16384 0 0 no-entry no-entry 15:49:02
q 1 0x3e1c02ac --rw-r--r-- root root root root 0
0 264 0 0 no-entry no-entry 15:49:02
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH
SEGSZ CPID LPID ATIME DTIME CTIME
Shared Memory:
m 0 0x411c025e --rw-rw-rw- root root root root 0
348 539 539 15:49:08 15:49:08 15:49:02
m 1 0x4e0c0002 --rw-rw-rw- root root root root 2
61760 539 541 7:08:26 15:49:08 15:49:02
m 2 0x4120180e --rw-rw-rw- root root root root 2
8192 539 541 7:08:26 15:49:02 15:49:02
m 3 0x301c63c0 --rw-rw-rw- root root root root 3
1048576 1084 1098 15:50:59 no-entry 15:49:24
m 5636 0x49182c35 --rw-r--r-- root root root root 0
22908 1268 1268 15:49:51 15:49:51 15:49:51
m 2053 0x431c39d0 --rw-rw-rw- daemon root daemon root 2
5767168 1426 1484 11:15:45 11:15:44 15:49:54
m 3590 0x00000000 D-rw------- root root root root 7
1052672 1438 1438 15:49:59 no-entry 15:49:59
m 1543 0x00000000 D-rw------- www other root root 7
184324 1443 1443 15:50:02 no-entry 15:50:02
m 23560 0xad582298 --rw-r----- oracle8i dba oracle8i dba 117
594780160 12987 22262 9:35:58 9:36:19 3:45:35
m 521 0x00000000 D-rw------- apps11i dba apps11i dba 10
2658308 13408 13408 7:00:08 no-entry 7:00:08
m 522 0x00000000 D-rw------- apps11i dba apps11i dba 8
2658308 13431 13431 7:00:10 no-entry 7:00:10
T ID KEY MODE OWNER GROUP CREATOR CGROUP NSEMS
OTIME CTIME
Semaphores:
s 0 0x411c025e --ra-ra-ra- root root root root 1 1
5:49:08 15:49:02
s 1 0x4e0c0002 --ra-ra-ra- root root root root 2 1
5:49:04 15:49:02
s 2 0x4120180e --ra-ra-ra- root root root root 2
7:08:26 15:49:02
s 3 0x00446f6e --ra-r--r-- root root root root 1 n
o-entry 15:49:11
s 4 0x00446f6d --ra-r--r-- root root root root 1 n
o-entry 15:49:11
s 5 0x01090522 --ra-r--r-- root root root root 1 n
o-entry 15:49:11
s 6 0x311c63c0 --ra-ra-ra- root root root root 1 1
5:50:59 15:49:24
s 7 0x4184df5b --ra-ra-ra- root root root root 1 n
o-entry 15:49:31
s 8 0x4184df5c --ra-ra-ra- root root root root 1 1
5:49:31 15:49:31
s 9 0x61200ded --ra-ra-ra- root root root root 1 1
5:50:26 15:49:44
s 10 0x73200ded --ra-ra-ra- root root root root 1
2:30:48 15:49:44
s 11 0x70200ded --ra-ra-ra- root root root root 1
2:30:48 15:49:44
s 12 0x69200ded --ra-ra-ra- root root root root 1 1
5:50:26 15:49:44
s 13 0x75200ded --ra-ra-ra- root root root root 1 1
5:49:45 15:49:44
s 14 0x63200ded --ra-ra-ra- root root root root 1 1
5:49:45 15:49:44
s 15 0x64200ded --ra-ra-ra- root root root root 1 1
5:50:28 15:49:44
s 16 0x66200ded --ra-ra-ra- root root root root 1 n
o-entry 15:49:44
s 17 0x6c200ded --ra-ra-ra- root root root root 1 1
5:50:26 15:49:44
s 18 0x6d200ded --ra-ra-ra- root root root root 1 1
5:57:04 15:49:44
s 19 0x6f200ded --ra-ra-ra- root root root root 1 n
o-entry 15:49:44
s 20 0x410c0508 --ra-ra-ra- root root root root 1 1
5:49:47 15:49:47
s 21 0x49182c35 --ra-r--r-- root root root root 1 1
5:49:51 15:49:51
s 3342 0x4c1c39d0 --ra-r--r-- daemon root daemon root 1 1
5:49:53 15:49:53
s 23 0x52200ded --ra-ra-ra- root root root root 1 1
5:57:04 15:50:28
s 24 0x4184dfb3 --ra-ra-ra- root root root root 1 n
o-entry 15:50:59
s 25 0x4184dfb4 --ra-ra-ra- root root root root 1 1
5:50:59 15:50:59
s 49826 0xe4ee60c0 --ra-r----- oracle8i dba oracle8i dba 204
9:37:52 3:45:36

4-# model
9000/800/L1500-7x

[Ted Buis]

The HP-UX performance cookbook might help.
Check your dynamic buffer cache kernel parameters. I would recommend OnlineJFS with Oracle. See details in the cookbook.

[Alzhy]

Fadia,

It looks like you have a serious memory issue to start with. And you have both a an Oralce DB instance with 500+ MB SGA plus APP server. Your sar stats and disk stats (is c1t2d0 your OS disk?) indicate memory lack due to high memory scan rates and page outs. BTW is this a newly migrated system? If not, has this been this "slow" before? Is this an 11.0 or 11i system?

You may want to do the following to free up some memory if you cannot add to the (2GB? of memory you have)

trim down max_dbc_pct to say 20%.
enable direct IO on your Oracle Datafile mountpoints .


Also, based on your IPCS output.. it appears you may have 150+ Oracle sessions -- is this normal for this environment? have you rebooted this machine?

[Fadia Almarei]

thanks nelson
my system is HP 11i, and it is the first time to have this problem ,and i reboot the machine .