hp-ux 11.11 kernel param+Oracle 9.2.0 performance+General performance

[Cem Tugrul]

Hi forum,
i have RP2450 9000/800/SD16000 with 16Gb Ram
hp-ux 11.11 Oracle 9.2.0+BaaN application.

when i moni system via glance the Disk Utilization seems %100 and responsible deamon
is ora_dbw0_baan01
my kernel params are;
baan01:/#kmtune
Parameter Current Dyn Planned Module Version
===============================================================================
NSTRBLKSCHED - - 2
NSTREVENT 50 - 50
NSTRPUSH 16 - 16
NSTRSCHED 0 - 0
STRCTLSZ 1024 - 1024
STRMSGSZ 65535 - 65535
acctresume 4 - 4
acctsuspend 2 - 2
aio_listio_max 256 - 256
aio_max_ops 2048 - 2048
aio_physmem_pct 10 - 10
aio_prio_delta_max 20 - 20
allocate_fs_swapmap 0 - 0
alwaysdump 1 - 1
bcvmap_size_factor 2 - 2
bootspinlocks - - 256
bufcache_hash_locks 128 - 128
bufpages 0 - (NBUF*2)
chanq_hash_locks 256 - 256
core_addshmem_read 0 Y 0
core_addshmem_write 0 Y 0
create_fastlinks 0 - 0
dbc_max_pct 4 - 4
dbc_min_pct 2 - 2
default_disk_ir 0 - 0
desfree - - 0
disksort_seconds 0 - 0
dmp_rootdev_is_vol 0 - 0
dmp_swapdev_is_vol 0 - 0
dnlc_hash_locks 512 - 512
dontdump 0 - 0
dskless_node - - 0
dst 1 - 1
effective_maxpid - - ((NPROC<=30000)?30000:(NPROC*5/4))
eisa_io_estimate - - 0x300
enable_idds 0 - 0
eqmemsize 15 - 15
executable_stack 1 - 1
fcp_large_config 0 - 0
file_pad - - 10
fs_async 0 - 0
ftable_hash_locks 64 - 64
hdlpreg_hash_locks 128 - 128
hfs_max_ra_blocks 8 - 8
hfs_max_revra_blocks 8 - 8
hfs_ra_per_disk 64 - 64
hfs_revra_per_disk 64 - 64
hp_hfs_mtra_enabled 1 - 1
hpux_aes_override - - 0
initmodmax 50 - 50
io_ports_hash_locks 64 - 64
iomemsize - - 40000
ksi_alloc_max 51232 - (NPROC*8)
ksi_send_max 32 - 32
lotsfree - - 0
max_async_ports 50 - 50
max_fcp_reqs 512 - 512
max_mem_window 0 - 0
max_thread_proc 64 - 64
maxdsiz 0x40000000 - 0X40000000
maxdsiz_64bit 0x80000000 - 0X80000000
maxfiles 2048 - 2048
maxfiles_lim 2048 Y 2048
maxqueuetime - - 0
maxssiz 0x8000000 - 0X8000000
maxssiz_64bit 0x40000000 - 0X40000000
maxswapchunks 16384 - 16384
maxtsiz 0x4000000 Y 0X4000000
maxtsiz_64bit 0x40000000 Y 0X40000000
maxuprc 5763 Y (NPROC*9/10)
maxusers 500 - 500
maxvgs 50 - 50
mesg 1 - 1
minfree - - 0
modstrmax 500 - 500
msgmap 16386 - (2+MSGTQL)
msgmax 8192 Y 8192
msgmnb 65536 Y 65536
msgmni 8192 - 8192
msgseg 30000 - 30000
msgssz 128 - 128
msgtql 16384 - 16384
nbuf 0 - 0
ncallout 11239 - (16+NKTHREAD)
ncdnode 150 - 150
nclist 8100 - (100+16*MAXUSERS)
ncsize 12672 - (NINODE+VX_NCSIZE)+(8*DNLC_HASH_LOCKS)
ndilbuffers 30 - 30
netisr_priority - - -1
netmemmax - - 0
nfile 32768 - 32768
nflocks 6404 - 6404
nhtbl_scale 0 - 0
ninode 7552 - ((NPROC+16+MAXUSERS)+32+(2*NPTY))
nkthread 11223 - (((NPROC*7)/4)+16)
nni - - 2
no_lvm_disks 0 - 0
nproc 6404 - 6404
npty 300 - 300
nstrpty 60 - 60
nstrtel 60 - 60
nswapdev 10 - 10
nswapfs 10 - 10
nsysmap 12808 - ((NPROC)>800?2*(NPROC):800)
nsysmap64 12808 - ((NPROC)>800?2*(NPROC):800)
o_sync_is_o_dsync 0 - 0
page_text_to_local - - 0
pfdat_hash_locks 128 - 128
public_shlibs 1 - 1
region_hash_locks 128 - 128
remote_nfs_swap 0 - 0
rtsched_numpri 32 - 32
scroll_lines 100 - 100
scsi_max_qdepth 8 Y 8
scsi_maxphys 1048576 - 1048576
sema 1 - 1
semaem 16384 - 16384
semmap 6402 - (SEMMNI+2)
semmni 6400 - 6400
semmns 6400 - (SEMMNI)
semmnu 6400 - (SEMMNI)
semmsl 2048 Y 2048
semume 10 - 10
semvmx 32767 - 32767
sendfile_max 0 - 0
shmem 1 - 1
shmmax 0x300000000 Y 0X300000000
shmmni 200 - 200
shmseg 120 Y 120
st_ats_enabled 0 - 0
st_fail_overruns 0 - 0
st_large_recs 0 - 0
st_san_safe 1 - 1
streampipes 0 - 0
swapmem_on 1 - 1
swchunk 4096 - 4096
swspConsMin - - 2 hsx -
swspLogMin - - 1 hsx -
swspNotifyMin - - 0 hsx -
swsp_trc_flags - - 0x10001002 swspBus -
sysv_hash_locks 128 - 128
tcphashsz 0 - 0
timeslice 10 - (100/10)
timezone 420 - 420
unlockable_mem 0 - 0
vas_hash_locks 128 - 128
vnode_cd_hash_locks 128 - 128
vnode_hash_locks 128 - 128
vol_checkpt_default 10240 - 10240
vol_dcm_replay_size 262144 - (256*1024)
vol_default_iodelay 50 - 50
vol_fmr_logsz 4 - 4
vol_max_bchain 32 - 32
vol_max_nconfigs 20 - 20
vol_max_nlogs 20 - 20
vol_max_nmpool_sz 4194304 - (4*1024*1024)
vol_max_prm_dgs 1024 - 1024
vol_max_rdback_sz 4194304 - (4*1024*1024)
vol_max_vol 8388608 - (8*1024*1024)
vol_maxio 256 - 256
vol_maxioctl 32768 - 32768
vol_maxkiocount 2048 - 2048
vol_maxparallelio 256 - 256
vol_maxspecialio 256 - 256
vol_maxstablebufsize 256 - 256
vol_min_lowmem_sz 524288 - (512*1024)
vol_mvr_maxround 256 - 256
vol_nm_hb_timeout 10 - (10)
vol_subdisk_num 4096 - 4096
vol_vvr_transport 1 - (1)
vol_vvr_use_nat 0 - (0)
volcvm_cluster_size 16 - 16
volcvm_smartsync 1 - 1
voldrl_max_drtregs 2048 - 2048
voldrl_min_regionsz 512 - 512
voliomem_chunk_size 65536 - (64*1024)
voliomem_maxpool_sz 4194304 - (4*1024*1024)
voliot_errbuf_dflt 16384 - 16384
voliot_iobuf_default 8192 - 8192
voliot_iobuf_limit 131072 - 131072
voliot_iobuf_max 65536 - 65536
voliot_max_open 32 - 32
volraid_rsrtransmax 1 - 1
vps_ceiling 16 - 16
vps_chatr_ceiling 1048576 - 1048576
vps_pagesize 4 - 4
vx_fancyra_enable 0 - 0
vx_maxlink 32767 - 32767
vx_ncsize 1024 - 1024
vx_ninode 0 - 0
vxfs_max_ra_kbytes 1024 - 1024
vxfs_ra_per_disk 1024 - 1024
vxtask_max_monitors 32 - 32

Who could check or tell me what is wrong on my
kernel param?or give any link about what i have to change on my kernel params?is there any recomendations about my system spec and kernel?Which kernel params of my system might be wrong or missing?

Please urgent help!!!

Greetings,

Ps:if you want more detail i can send...

[Jean-Luc Oudart]

Hi,

Is this a new application or has this application behaviour just changed ?

which exact version do you run ?
select * from v$version ?
(have you applied oracle patches for oracle 9.2)

Regards
Jean-Luc

[Geoff Wild]

Only thing I see is shmmax - mine is set to:

shmmax 0x400000000


But you say it's disk utilization - what is on theat disk?

Is it a particular lvol? IE- - do you have redo logs on same disk as indexes?

Some guidelines I follow:

APPENDIX D - Oracle Requirements

Vgcreate volume groups as per the following standard (these are minimum)

Preamble: VG00 - VG09 are internal disks, VG10 and higher are reserved for EMC.

VG00
The root volume group VG00 should be mirrored and only contain OS files.
Primary swap should be 1 x memory
/ - 140MB (HP Default) set higher - 200 MB
/usr - 2GB
/usr/openv - 512MB
/var - 4GB
/tmp - 512MB
/stand - 256MB
/opt - 2GB
/home - 512MB with quotas (32MB soft, 64MB hard)
/var/opt/perf/datafiles - 2GB
If adding CCMON - then /opt/cctool - 256MB
Note: /var/tmp should be a symbolic link to /tmp. /tmp should be 1777 (sticky bit).

VG01
/app - 512MB
/usr/local - 512MB
/app/admin - 512MB
/var/adm/crash - 1 x memory + 512MB
VG02
File system swap
VG03
Applications
VG04-VG09
Reserved

VG10 - VG19
Log volumes (redo, archive, etc)

VG20 - VG256
Data volumes

Miscellaneous

Don't use strict/contiguous when creating logical volumes.
Create file systems with "large file" support (fsadm).
Mount file systems with "delaylog".
Oracle
The following is the typical oracle layout:

Volume Size File System

vg01 4 Gig - /app/oracle
vg01 500 Meg - /app/dbcorp
vg30 6 Gig - /data/oracle/irsp/data01
vg30 6 Gig - /data/oracle/irsp/data02
vg30 6 Gig - /data/oracle/irsp/data03
...
Can all be on one mount point if need be, but this probably will reduce performance.

vg31 2 Gig - /data/oracle/irsp/indx01

vg31 2 Gig - /data/oracle/irsp/indx02

vg31 2 Gig - /data/oracle/irsp/indx03

...

Again, can all be on one mount point if need be, but this probably will reduce performance.

vg12 12 Gig - /data/oracle/irsp/arch
Should be on a different mount point than dataXX and indxXX as this will benefit from different mount flags.

vg10 8 Gig - /data/oracle/irsp/exports
Also should be on a different mount point than dataXX and indxXX as this will benefit from different mount flags.

vg10 1.5 Gig - /data/oracle/irsp/redo01a
vg11 1.5 Gig - /data/oracle/irsp/redo01b
redo01a & redo01b on different physical volumes

vg10 1.5 Gig - /data/oracle/irsp/redo02a
vg11 1.5 Gig - /data/oracle/irsp/redo02b
redo02a & redo02b on different physical volumes

Rgds...Geoff

[Sridhar Bhaskarla]

Hi,

Kernel parameters have little to do with the disk utilization.

Post your 'sar -d 5 5' 'sar -b 5 5' 'sar 5 5' outputs. Also post the following details.

1. How big are your LUNs.
2. How many VGs you have and with how many disks.
3. How big are your LVs.
4. What's your storage system.

-Sri

[Steven E. Protter · Accepted solution]

I'd collect some performance data with the past recommendations, or get more comprehensive with the script I'm attaching.

This is a comprehensive link on performance tuning:

http://www1.itrc.hp.com/service/cki/search.do?category=c0&docType=Security&docType=Patch&docType=EngineerNotes&docType=BugReports&docType=Hardware&docType=ReferenceMaterials&docType=ThirdParty&searchString=UPERFKBAN00000726&search.y=8&search.x=28&mode=id&admit=-682735245+1102350851284+28353475&searchCrit=allwords

SEP

[Fred Ruffet]

Better thing would be to post here your initbaan01.ora file or spfilebaan01.ora rather than your kernel params.

One more thing : How many CPUs are there on this box ? Are you using Raw devices ? Asynch-ios ?

Regards,

Fred

[Fred Ruffet]

as you seem to look for perfs :
http://h21007.www2.hp.com/dspp/files/unprotected/database/HP3KOracle.ppt
is a VERY good doc.

Regards,

Fred

[Zinky]

Your kernel params are absolutely fine.

Be aware that if you're using your default Measureware/Glance configuration - that 100% disk utilization in Glance SHOULD NOT BE TOTALLY BE BELIEVED in.. The default glance/mwa config will give glance users a 100% DIsk utilisation graph/stat even if just one LUN is reaching 90-100% Utilisation.

Your best gauges to tell you if you've an I/O (a serious one at that are):

1. sar -d (you've queue lenghts gt 0, gt. 20 access/service times

2. sar (CPU states where %WIO is consistently high -- say close to 50%)

3. vmstat (you've many blocked processes)

4. Oracle complains that the system is really waiting in I/O states.

What storage do you have? an EVA perhaps?

[Cem Tugrul]

Hi Forum,
Thank's for all response;

As an addition about my system;

i have EVA HSV Storage with 6 Disk Enclosure
(each Enc has 14 bay) and they are fully
with %50 72Gb and %50 146 GB Disks.and i have
4 Virtual Disk Grp.

baan01:/#bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1048576 162752 881312 16% /

baan01:/#vmstat
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
7 9 0 790759 1831864 64 4 3 0 0 0 9 5118 148314 20867 31 11 58


/dev/vg00/lvol1 251696 43984 182536 19% /stand
/dev/vg00/lvol7 2097152 1384792 712360 66% /var
/dev/vg00/lvol10 4194304 2075963 1985954 51% /var/adm/crash
/dev/vg00/lvol6 2097152 1208776 881480 58% /usr
/dev/vg00/lvol5 409600 36016 371088 9% /tmp
/dev/vg00/lvol4 2097152 1443528 648552 69% /opt
/dev/vg00/lvol8 409600 2784 403704 1% /home
/dev/vg07/ezisnew 52416512 177040 51831392 0% /ezisnew
/dev/vg08/lvol1 204800000 131081632 73142504 64% /baandat_ytl
/dev/vg01/lvol1 41934848 16357104 25379440 39% /users
/dev/vg02/lvol1 5238784 4453444 774384 85% /oracle
/dev/vg03/lvol1 15724544 8992716 6631640 58% /baan
/dev/vg04/lvol1 471793664 399937232 71295208 85% /baandat
/dev/vg05/lvol1 20967424 226536 20578920 1% /archivelog
/dev/vg06/lvol1 104841216 56797640 47670464 54% /archivedat
/dev/vgrman/lvrman 1044480 653612 366478 64% /RMAN
/dev/vgomni/lvetcoptomni
53248 2040 48068 4% /etc/opt/omni
/dev/vgomni/lvvaroptomni
3072000 2186429 830284 72% /var/opt/omni
/dev/vgomni/lvoptomni
770048 641447 120565 84% /opt/omni

baan01:/#cat /etc/fstab
# System /etc/fstab file. Static information about the file systems
# See fstab(4) and sam(1M) for further details on configuring devices.
/dev/vg00/lvol3 / vxfs delaylog 0 1
/dev/vg00/lvol1 /stand hfs defaults 0 1
/dev/vg00/lvol9 ... swap defaults 0 0
/dev/vg00/lvol4 /opt vxfs delaylog 0 2
/dev/vg00/lvol5 /tmp vxfs delaylog 0 2
/dev/vg00/lvol6 /usr vxfs delaylog 0 2
/dev/vg00/lvol7 /var vxfs delaylog 0 2
/dev/vg00/lvol8 /home vxfs delaylog 0 2
/dev/vg00/lvol10 /var/adm/crash vxfs delaylog 0 2
/dev/vg00/lvol11 ... swap pri=1 0 0
/dev/vg04/lvol1 /baandat vxfs rw,suid,largefiles,delaylog,datainlog 0 2
/dev/vg07/ezisnew /ezisnew vxfs rw,suid,largefiles,delaylog,datainlog 0 2
/dev/vg08/lvol1 /baandat_ytl vxfs rw,suid,largefiles,delaylog,datainlog 0 2

My srv also have 8 cpu,and mirrored (vg00)disk;
--- Physical volumes ---
PV Name /dev/dsk/c0t6d0
PV Status available
Total PE 4374
Free PE 586
Autoswitch On

PV Name /dev/dsk/c3t6d0
PV Status available
Total PE 4374
Free PE 586
Autoswitch On

baan01:/#sar 5 5

HP-UX baan01 B.11.11 U 9000/800 12/07/04

08:45:50 %usr %sys %wio %idle
08:45:55 57 14 18 10
08:46:00 42 15 29 14
08:46:05 49 16 24 10
08:46:10 40 15 31 13
08:46:15 42 16 28 14

Average 46 15 26 12

baan01:/#sar -d 5 5

HP-UX baan01 B.11.11 U 9000/800 12/07/04

08:46:49 device %busy avque r+w/s blks/s avwait avserv
08:46:54 c0t6d0 2.99 0.50 6 33 4.06 9.00
c3t6d0 2.40 0.50 4 26 4.44 6.84
c20t1d1 28.34 0.51 179 7425 5.05 3.26
c20t0d0 0.20 0.50 1 8 4.47 2.27
c20t0d2 3.59 0.54 21 191 4.22 1.96
c20t0d3 100.00 0.63 1589 27146 5.17 4.00
08:46:59 c0t6d0 7.62 0.50 13 85 4.57 8.02
c3t6d0 5.61 0.50 9 57 4.40 6.73
c20t1d1 23.65 0.52 170 7181 5.12 2.93
c20t0d0 0.80 0.50 2 25 3.85 3.02
c20t0d1 0.20 0.50 0 2 5.80 3.42
c20t0d2 6.21 0.59 55 595 4.37 1.57
c20t0d3 100.00 1.07 1674 36648 5.48 4.05
c20t0d5 0.60 0.50 1 58 4.73 8.12
08:47:04 c0t6d0 6.60 0.50 11 41 4.90 6.44
c3t6d0 5.20 0.50 10 34 5.25 4.68
c20t1d1 35.40 0.55 236 8672 5.15 2.62
c20t0d2 3.80 1.14 24 246 4.10 2.07
c20t0d3 100.00 0.91 1730 30187 5.44 4.14
c20t0d5 0.60 0.50 1 58 6.23 4.77
08:47:09 c0t6d0 2.99 0.50 7 56 5.01 5.93
c3t6d0 2.99 0.50 5 40 5.03 5.41
c20t1d1 37.72 0.54 324 13051 5.16 2.47
c20t0d2 2.40 0.84 24 342 4.13 1.35
c20t0d3 99.80 0.65 1681 28331 5.22 3.96
c20t0d5 0.40 0.50 1 51 3.62 10.76
08:47:14 c0t6d0 1.00 0.50 3 24 2.55 7.16
c3t6d0 1.00 0.50 3 23 2.37 6.48
c20t1d1 25.80 0.53 266 11696 5.09 1.98
c20t0d0 0.20 0.50 0 6 5.97 0.36
c20t0d2 3.60 0.56 20 242 3.96 1.92
c20t0d3 100.00 0.73 1633 28442 5.27 4.17
c20t0d5 0.40 0.50 1 51 6.86 5.43

Average c0t6d0 4.24 0.50 8 48 4.51 7.29
Average c3t6d0 3.44 0.50 6 36 4.61 5.86
Average c20t1d1 30.19 0.53 235 9606 5.12 2.58
Average c20t0d0 0.24 0.50 1 8 4.21 2.57
Average c20t0d2 3.92 0.71 29 323 4.20 1.72
Average c20t0d3 100.00 0.80 1661 30146 5.32 4.07
Average c20t0d1 0.04 0.50 0 0 5.80 3.42
Average c20t0d5 0.40 0.50 1 44 5.32 7.23

baan01:/#sar -b 5 5

HP-UX baan01 B.11.11 U 9000/800 12/07/04

08:47:57 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
08:48:02 2247 13742 84 459 575 20 0 0
08:48:07 2261 12995 83 597 626 5 0 0
08:48:12 2003 13235 85 586 657 11 0 0
08:48:17 2084 14569 86 595 669 11 0 0
08:48:22 1640 12651 87 729 739 1 0 0

Average 2047 13439 85 593 653 9 0 0

baan01:/#swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
dev 8192 0 8192 0% 0 - 1 /dev/vg00/lvol9
dev 36000 0 36000 0% 0 - 1 /dev/vg00/lvol11
reserve - 7818 -7818
memory 13160 823 12337 6%
total 61448 8641 52807 14% - 0 -

[Cem Tugrul]

finally my init.ora;

control_files = (/baandat/baan01/system/control01_baan01.ctl,
/baandat/baan01/system/control02_baan01.ctl,
/baandat/baan01/system/control03_baan01.ctl)
background_dump_dest = /oracle/admin/baan01/bdump
core_dump_dest = /oracle/admin/baan01/cdump
user_dump_dest = /oracle/admin/baan01/udump
log_archive_start = true
log_archive_dest = /archivelog/baan01
log_archive_format = baan01_log_arc_%t_%S

db_name = baan01

open_cursors = 4096
db_files = 255
db_file_multiblock_read_count = 8
db_block_size = 8192
db_block_buffers = 300000
shared_pool_size = 1000000000
java_pool_size = 50331648
log_checkpoint_interval = 1000000000
processes = 1200
open_cursors = 4000
dml_locks = 5000
log_buffer = 1048576
max_dump_file_size = 10240
sort_area_size = 1048576
compatible = 9.2.0.0.0
undo_management = auto
nls_sort = binary
timed_statistics = true
undo_retention = 72000
remote_login_passwordfile=exclusive
utl_file_dir = /u04/globus/out
utl_file_dir = /u04/globus/dispo
job_queue_processes = 4