topic Re: high disk IO kernel params in Operating System - HP-UX

high disk IO kernel params

michael_210 — Fri, 03 Jan 2003 16:45:37 GMT

Any suggestions that would reduce Disk IO from 100%??

HPUX 11 /N9000 server/Oracle8.1.7

kmtune query

NSTRBLKSCHED 2
NSTREVENT 50
NSTRPUSH 16
NSTRSCHED 0
STRCTLSZ 1024
STRMSGSZ 65535
acctresume 4
acctsuspend 2
aio_listio_max 256
aio_max_ops 2048
aio_physmem_pct 10
aio_prio_delta_max 20
allocate_fs_swapmap 0
alwaysdump 0
bootspinlocks 256
bufcache_hash_locks 128
bufpages (NBUF*2)
chanq_hash_locks 256
create_fastlinks 0
dbc_max_pct 20
dbc_min_pct 5
default_disk_ir 0
desfree 0
dnlc_hash_locks 64
dontdump 0
dskless_node 0
dst 1
eisa_io_estimate 0x300
eqmemsize 15
fcp_large_config 0
file_pad 10
fs_async 0
ftable_hash_locks 64
hdlpreg_hash_locks 128
hfs_max_ra_blocks 8
hfs_ra_per_disk 64
hpux_aes_override 0
initmodmax 50
io_ports_hash_locks 64
iomemsize 40000
km_disable 0
ksi_alloc_max (NPROC*8)
ksi_send_max 32
lotsfree 0
max_async_ports 50
max_fcp_reqs 512
max_mem_window 0
max_thread_proc 256
maxdsiz 0X40000000
maxdsiz_64bit 0X0000000050000000
maxfiles 2048
maxfiles_lim 2048
maxqueuetime 0
maxssiz 0X01500000
maxssiz_64bit 0X00900000
maxswapchunks 1024
maxtsiz 0X09000000
maxtsiz_64bit 0X0000000050000000
maxuprc 2000
maxusers 400
maxvgs 10
mesg 1
minfree 0
modstrmax 500
msgmap (2+MSGTQL)
msgmax 8192
msgmnb 16384
msgmni 180
msgseg 2048
msgssz 8
msgtql 40
nbuf 0
ncallout (16+NPROC)
ncdnode 150
nclist (100+16*MAXUSERS)
ncsize (NINODE+VX_NCSIZE)
ndilbuffers 30
netisr_priority -1
netmemmax 0
nfile (32*(NPROC+16+MAXUSERS)/10+32+2*(NPTY+NSTRPTY+NSTRTEL))
nflocks 1000
nhtbl_scale 0
ninode ((NPROC+16+MAXUSERS)+32+(2*NPTY))
nkthread (((NPROC*7)/4)+16)
nni 2
no_lvm_disks 0
nproc (20+16*MAXUSERS)
npty 400
nstrpty 400
nstrtel 400
nswapdev 10
nswapfs 10
nsysmap ((NPROC)>800?2*(NPROC):800)
nsysmap64 ((NPROC)>800?2*(NPROC):800)
num_tachyon_adapters 5
o_sync_is_o_dsync 0
page_text_to_local 0
pfdat_hash_locks 128
public_shlibs 1
region_hash_locks 128
remote_nfs_swap 0
rtsched_numpri 32
scroll_lines 100
scsi_maxphys 1048576
sema 1
semaem 16384
semmap (SEMMNI+2)
semmni 1800
semmns 4096
semmnu 600
semume 10
semvmx 32767
sendfile_max 0
shmem 1
shmmax 0X70000000
shmmni 300
shmseg 120
st_ats_enabled 1
st_fail_overruns 0
st_large_recs 0
streampipes 0
swapmem_on 1
swchunk 2048
sysv_hash_locks 128
tcphashsz 0
timeslice (100/10)
timezone 420
unlockable_mem 0
vnode_cd_hash_locks 128
vnode_hash_locks 128
vps_ceiling 16
vps_chatr_ceiling 65536
vps_pagesize 4
vx_ncsize 1024
vx_ninode 0
vx_noifree 0
vxfs_max_ra_kbytes 1024
vxfs_ra_per_disk 1024

thanks
mike

Re: high disk IO kernel params

Robert Thorneycroft — Fri, 03 Jan 2003 16:50:28 GMT

You really need to investigate why your disk IO is at 100% rather than looking into kernel parameters. Even if you could find a parameter which would throtle the IO back you would just be creating a system bottleneck which is probably even worse than any you are already experiencing!

Kind regards,

Robert Thorneycroft

Re: high disk IO kernel params

michael_210 — Fri, 03 Jan 2003 17:17:52 GMT

The usuage is due to multiple oracle
processes running concurrently. I am working with devs to improve the process scripts.

i was just curious if making any adjustments in the params would provide any benefit..
thanks

Re: high disk IO kernel params

Steven E. Protter — Fri, 03 Jan 2003 17:24:20 GMT

I don't think this is a kernel problem.

Are you using asyncrhonous access?

is /etc/privgroup set properly(dba MLOCK).

Also, you might want to look into your disk layout. It makes a difference.

Steve

Re: high disk IO kernel params

Patrick Wallek — Fri, 03 Jan 2003 17:24:43 GMT

More likely you need to look at how your LVOL's are laid out across the disks. Are you using striping? Do you have datafiles and redo logs on the same disks? Questions like that need to be answered and that information taken into account when laying out the VGs and LVs that will contain your Oracle databases.

Re: high disk IO kernel params

Robert Thorneycroft — Fri, 03 Jan 2003 17:27:46 GMT

If the disk utilization is coming from your Oracle process there is not really a lot you can do about it other than:
1) Tune your Oracle SQL queries so that they use better access methods to gather the data they are after
2) Increase the size of your SGA so that more data resides in memory so repeated disk accesses are not necessary.
3) Go buy a nice fast disk array from EMC or HP.

There is also a problem know with JFS 3.3 and IO throttling but this does not sound to be the issue you are experiencing as it tend to have the opposite effect.

Sorry I could not be of any further help.

Kind regards,

Robert Thorneycroft.

Re: high disk IO kernel params

michael_210 — Fri, 03 Jan 2003 17:37:55 GMT

The data live on an array in different lvols.connected via fiber.
oracle binaries are on the server itself.

i tried increasing SGA but couldnt get higher than 980mb.
if i increase shmmax this should allow the oracle SGA to grow ..

what i am curious about is if i increase
maxswapchunks and max_thread_proc
and /or change db_max_pct..

Will there be any improvement..?????
regards
mike

dev/vg00/lvol3 143360 106698 34399 76% /
/dev/vg00/lvol1 83733 43814 31545 58% /stand
/dev/vg00/lvol8 3670016 3194233 446389 88% /var
/dev/vg00/lvol7 770048 590443 168410 78% /usr
/dev/vg00/lvol4 536576 224977 292681 43% /tmp
/dev/vg01/lvol2 2048000 744358 1222238 38% /rman
/dev/vg01/lvol1 4096000 2727318 1283333 68% /pslogs
/dev/vg00/lvol6 1024000 966631 53969 95% /opt
/dev/vg00/lvol9 8192000 5525051 2500298 69% /hr/hroracle8i
/dev/vg01/lvol4 4096000 2104 3838035 0% /hr/hrarch
/dev/vg00/lvol5 102400 30906 67030 32% /home
/dev/vg00/lvol10 102400 1133 94945 1% /fin/finoracle8i
/dev/u06/lvol8 34865152 1342256 33261008 4% /fs/fsorasys
/dev/u06/lvol63 8912896 10930 8623788 0% /fs/fstools
/dev/u06/lvol9 34865152 4611488 30017376 13% /fs/fsredos
/dev/u06/lvol62 20709376 6294932 14189228 31% /fs/fstemp
/dev/u06/lvol5 177209344 22592096 153409392 13% /fs/fsarch
/dev/u06/lvol61 41156608 40684568 468360 99% /fs/fsrbs
/dev/u06/lvol1 141557760 26592432 114067216 19% /fs/fsdata
/dev/u06/lvol4 70778880 22204808 48194600 32% /fs/fsindex
/dev/u06/lvol2 70778880 14902680 55439736 21% /fs/fsexports
/dev/u06/lvol3 34865152 12586280 22104832 36% /fs/fsdata2
/dev/u06/lvol10 34865152 31460688 3377880 90% /fs/fsdata3
/dev/u06/lvol11 34865152 29363552 5458632 84% /fs/fsindex2

Re: high disk IO kernel params

Chris Vail — Fri, 03 Jan 2003 17:58:43 GMT

I concur with the others that say that there isn't much you can do here from a kernel tuning standpoint. There is, however, a lot you can do from an Oracle perspective to fix this.
Usually, the application dumps all of its data into a single tablespace. This is easy and convenient for the programmers, but is tough on the machines. Without redesigning the application, about all a sysadmin can do is stripe the filesystem on which the tablespace resides across several disks. HP, EMC and others sell systems that do this very nicely. This can provide a substantial increase in disk speed--especially with sequential reads that can come from cache. Random read slowness is usually impervious to hardware fixes.
Best deal is to redesign the application(s) to use more than one tablespace. It is far, far better to have several (even dozens) of tablespaces spread across numerous filesystems than a single (or very few) tablespaces on a single (or few) filesystems. The more filesystems in use, the more the system can access multiple data streams at the same time. Ultimately, a disk drive is its own bottleneck. Spread the load as much as possible amongst as many as possible.
Lastly, check to see that the application(s) are not doing full table scans, and that they're indexed as fully as possible. All by itself, using indexes--and otherwise properly designing the Oracle database--can yield a tenfold increase in I/O speed.
Use iostat to find out which filesystems are getting hammered. Usually, you'll find that only a few are so busy. Whichever tablespace, index or other file is so busy is the first one to move to faster and better i/o systems, or be subject to a redesign.
This is not a fast/easy fix. As a rule, Oracles suggestions for kernel tuning parameters should be followed, and modified only very carefully, and only after exhausting other possibilities.
Good Luck
Chris

Re: high disk IO kernel params

Mott Given — Fri, 03 Jan 2003 18:13:49 GMT

What tool says you disk I/O is 100%? A lot of vendor tools do not give correct results, especially if the I/O is being done to a file on a RAID device.

How long was the measurement interval where the disk was this
busy?

Mott Given

Re: high disk IO kernel params

michael_210 — Fri, 03 Jan 2003 18:21:22 GMT

Mott

I am using Glance.

regards
mike

Re: high disk IO kernel params

James R. Ferguson — Fri, 03 Jan 2003 18:48:03 GMT

Hi Mike:

As already noted, the most performance gains are probably going to be realized with I/O configuration (striping (extent-based or true stripes)); improved SQL queries; and mount options.

For instance, if you are using VxFS filesystems and if you have OnlineJFS, then you can mount the filesystems which contain datafiles and indices with 'delaylog, nodatainlog, convosync=direct, mincache=direct'. For the archive and redo logs use 'delaylog, nodatainlog'. This allows the datafiles I/O to bypass the Unix buffer cache.

In fact, you might decrease your 'dbc_min_pct' and 'dbc_max_pct' to something like <2-5> and <5-8> respectively. This will return some of the buffer cache memory.

You asked about 'max_thread_proc' and 'macswapchunks'. Neither will improve performance per se. 'max_thread_proc' is a fence for the number of threads a process may have. If you are running into this with an errno value of EAGAIN, then you might need to increase it.

'maxswapchunks' governs how much swap space can be defined. If you are unable to define sufficient swap, then you increase 'maxswapchunks'.

Regards!

...JRF...

Re: high disk IO kernel params

Mott Given — Fri, 03 Jan 2003 19:10:03 GMT

Glance does not give accurate info on things like disk utilization if the disks are on a RAID device.

Mott Given

Re: high disk IO kernel params

Steven E. Protter — Sun, 05 Jan 2003 01:39:39 GMT

mike,

give some of these nice people some points for trying. They are trying to help.

I've read the other suggestions, and I'm still thinking patches or disk layout.

Steve

Re: high disk IO kernel params

Brian Watkins — Mon, 06 Jan 2003 19:54:21 GMT

Sounds like you'll need to work with the DBA's to optimize their SQL lookups and look at how the database is actually being laid out on the disks in your array.

When dealing with an active database, the more physical spindles you can spread the I/O across, the better, especially when it comes to I/O intensive operations like redo and archive logs.

And don't make your SGA too big...if you give too much memory to the database, you could end up causing other problems (excessive swapping, print spooling delays, etc.)