topic Re: Kernel Tuning - nproc + dbc_max_pct in Operating System - HP-UX

Kernel Tuning - nproc + dbc_max_pct

rmueller58 — Tue, 21 Jun 2005 14:16:42 GMT

We've been hitting the wall periodically on one of our UX systems. HP/UX 11.0 N Class, 3gb RAM, Dual CPU,

In reading the "intro to performance tuning" sent to me by tech team, I monitored buffer cache which appears to maybe be one of the possible "issues" %wcache averaged about 51%, HP recommends having this about 75%. It recommends lowering the dbc_max_pct from the default of 50 to something less.

I tried modifying the property in SAM and it indicated via error that there is a dependency on the nproc kernel parameter. My kernel parameter for nproc is 3072.

For the "kernel" gurus, What the he** should I do? change the nproc? and is so to what?

Any thought appreciated ..

Rex M - ESU#3 Omaha NE

Re: Kernel Tuning - nproc + dbc_max_pct

Mel Burslan — Tue, 21 Jun 2005 14:21:48 GMT

to the best that I know, if your dbc_max_pct is taking up more than 700-750MB of memory space, it is almost being wasted. Having said that, HP's suggestion is to reduce the default 50% mark to somewhere between 5% and 10%, I usually use 8% on my 8-12 GB physical memory containing machines running as oracle databases at the SAP's back end. Haven't heard a peep about it from any users.

I have not seen this kernel parm being tied to nproc but again, stranger things has happened so, I do not want to speculate on this one. Maybe someone else knows the relation and I can read it as well.

Re: Kernel Tuning - nproc + dbc_max_pct

MZ_1 — Tue, 21 Jun 2005 14:28:35 GMT

it's a SAM thing. use kmtune -s dbc_max_pct=10 (or whatever you decide), then mk-kernel, then kmupdate, and reboot.

bottom line is kmtune not SAM

Re: Kernel Tuning - nproc + dbc_max_pct

rmueller58 — Tue, 21 Jun 2005 15:22:10 GMT

Mark,

This would replace my existing /stand/vmunix?
Correct?

Re: Kernel Tuning - nproc + dbc_max_pct

Robert Bennett_3 — Tue, 21 Jun 2005 15:55:35 GMT

Rex - you can also create your own tunable kernel fileset for a SAM application.

#cd /usr/sam/lib/kc/tuned
#ll
cp one of the files to a new one like 11.tun.
vi this file with the tunable parameters needed
Here is an example of this file:
# 11.00 tuned kernel parameters
#
# Format:
# 1st line = title of tuned parameter set
#
# Lines or words (delimited by white space) beginning with # are comments.
#
# Blank lines are ignored.
#
# Parameter defs can take either of two forms:
# 1. parameter min max
# The parameter's value must fall into the range of min-max,
# inclusive. If parameter is less than min, it is raised to
# min; if greater than max, it is lowered to max. Otherwise
# it is left unchanged.
#
# Min and/or max can be "--" (two dashes), indicating there is no
# corresponding lower and/or upper bound, respectively.
#
# Min and/or max can be arithmetic formulas, and can reference
# other parameters. Any parameters referenced in a formula must
# already be defined earlier in the parameter set. No white
# space may occur within a formula. Parentheses are recognized
# and can be nested.
#
# 2. parameter value
# The parameter's value will be set to the value given. Value
# can be an arithmetic formula as described above.
#
STRCTLSZ 0
STRMSGSZ 8192
maxusers 124
nproc 1024
bufpages 0
create_fastlinks 1
nbuf 0
dbc_min_pct 4
dbc_max_pct 10
maxfiles 200
maxfiles_lim 2048
nfile 2800
nflocks (nproc)
ninode 15000
msgtql (nproc*10)
msgmap (msgtql+2)
msgmax 32768
msgmnb 65535
msgmni (nproc)
msgssz 128
msgseg 32767
semaem 16384
semmni (nproc*5)
semmap (semmni+2)
semmns (semmni*2)
semmnu (nproc-4)
semume 64
semvmx 32768
shmmax 0x40000000
shmmni 512
shmseg 32
maxvgs 80
maxswapchunks 4096
nswapdev 25
swapmem_on 1
unlockable_mem (maxusers*10)
maxuprc 200
timeslice 10
npty 512
sema 1
mesg 1
shmem 1
no_lvm_disks 0
netmemmax 0

Note - this is just an example of one of our tuned sets - use prejudice.

Once you have saved this file you will be able to apply it via SAM > Kernel Configuration > Configurable Parameters > Actions > Apply Tuned Parameter Set > Pick the one you just made > OK.

This is just another way to achieve what you want and may be analogous to using an elephant gun to kill a mosquito, but it's nice to make your own tuned set to use when you're building a server or two.

Hope this helps

Re: Kernel Tuning - nproc + dbc_max_pct

AMAT SA — Tue, 21 Jun 2005 18:43:11 GMT

Once we had a visit from HP Performance Guru who actually wrote some performance books in HP. His thumb rule was to have dbc_max_pct at =<400MB. Secondly, nproc is a parameter which is a threshold per my understanding and would not effect the system performance. Other things which can significantly improve the system performance is to reduce the sememu and semnu if your Kernel size is about 75MB. The thumb rule is to get it between 4 and 8 MB for HPUX 11.0 per his advise.

How do you find that is by running 'size /stand/vmunix'

Another note: check if you are running Oracle the SGA size. This should be between 50 to 70% of the system physical memory...

Prasad.

Re: Kernel Tuning - nproc + dbc_max_pct

rmueller58 — Wed, 22 Jun 2005 08:26:58 GMT

My application vendor and I went through tuning Informix quite a last night.

I did not get the "kernel" tweak completed yet. I will review this today and make the appropriate modifications.

Re: Kernel Tuning - nproc + dbc_max_pct

rmueller58 — Wed, 22 Jun 2005 08:32:28 GMT

I have several different tun files..

2 dr-xr-xr-x 2 support support 1024 Oct 4 2003 .
2 dr-xr-xr-x 3 bin bin 1024 Nov 21 2000 ..
4 -r--r--r-- 1 bin bin 1987 Jun 22 1999 cae_wkst.tun
4 -r--r--r-- 1 bin bin 1592 Aug 31 1999 eew32.tun
4 -r--r--r-- 1 bin bin 1678 Aug 31 1999 eew64.tun
4 -r--r--r-- 1 bin bin 1536 Aug 31 1999 mew32.tun
4 -r--r--r-- 1 bin bin 1622 Aug 31 1999 mew64.tun
6 -r--r--r-- 1 bin bin 2761 Oct 27 1997 oltp_cli.tun
6 -r--r--r-- 1 bin bin 2765 Jun 22 1999 oltp_sa.tun
6 -r--r--r-- 1 bin bin 2762 Feb 8 2003 oltp_ser.tun
4 -r--r--r-- 1 bin bin 1724 Jun 8 2001 sde.tun
4 -r--r--r-- 1 bin bin 1271 Oct 27 1997 vct_ser.tun

Not sure which one to use as the model script.

Is there a way to get the existing parameters from /stand/vmunix and create a modification of the existing kernel with a tun file?

Re: Kernel Tuning - nproc + dbc_max_pct

Eric Antunes — Wed, 22 Jun 2005 09:12:41 GMT

Hi Rex,

Forgget those templates... Each site/system has its own particularity.

For your 3Gb RAM, set dbc_max_pct to 10 and dbc_min_pct to 7. shmmax also must be something like 2/3 RAM, 2147483648 in your case.

Furthermore, here are some kernel parameters I've changed from the default - the 64 bits ones have no impact for me but I list them anyway (don't change any parameter that you have with higher values):

maxdsiz to 536870912, maxfiles 1024, maxssiz to 134217728, maxssiz_64bits to 1073741824, maxswapchunks to 16384, maxtsiz to 134217728, maxuprc to (NPROC*9/10), maxusers to 150, msgmni to (NPROC), msgseg to 32767, msgtql to (NPROC), nfile to (15*NPROC+2048), nflocks to 4096, ninode to 2048, nproc to 400, nstrpty to 60, semmni to 4096, semmns to (SEMMNI*2), semmnu to (NPROC-4), shmmni to 512, swchunk to 4096 and vps_ceiling to 64.

Best Regards,

Eric Antunes

Re: Kernel Tuning - nproc + dbc_max_pct

Mel Burslan — Wed, 22 Jun 2005 09:36:06 GMT

The answer to your question

___________
Is there a way to get the existing parameters from /stand/vmunix and create a modification of the existing kernel with a tun file?
___________

yes there is.

run command

kmtune | awk {'print $1"\t"$2'}> /tmp/kern_parms

then edit this file to make it look like a .tun file. It already is very similar anyways. Then move the file to/usr/sam/lib/kc/tuned location. The file format is not going to be pretty but it will still work.

hope this helps

Re: Kernel Tuning - nproc + dbc_max_pct

rmueller58 — Wed, 22 Jun 2005 09:39:07 GMT

Thanks Mel,

I will see what can do from here now..

If I tune dbc_max_pct is it a realtime change or does it require reboot.

Re: Kernel Tuning - nproc + dbc_max_pct

Eric Antunes — Wed, 22 Jun 2005 09:44:40 GMT

Hi,

Every kernel change needs a reboot.

That's why you should tune all the parameters you need on a Test environment and test those changes before doing it on Production environment.

Best Regards,

Eric Antunes

Re: Kernel Tuning - nproc + dbc_max_pct

Robert Bennett_3 — Wed, 22 Jun 2005 10:11:27 GMT

Rex -

Here is a good reference on tuning for you.

http://h21007.www2.hp.com/dspp/files/unprotected/devresource/Docs/TechPapers/UXPerfCookBook.pdf

As far as a model script - the question would be 32 or 64 bit OS - and then look at the choices you have. We have an N running 11i-64 OS w/8 CPUs and 8G memory that used to be a production box and is now used for testing. Here's the tunable file we used:

N4000 11i.00 tuned kernel parameters
#
# Format:
# 1st line = title of tuned parameter set
#
# Lines or words (delimited by white space) beginning with # are comments.
#
# Blank lines are ignored.
#
# Parameter defs can take either of two forms:
# 1. parameter min max
# The parameter's value must fall into the range of min-max,
# inclusive. If parameter is less than min, it is raised to
# min; if greater than max, it is lowered to max. Otherwise
# it is left unchanged.
#
# Min and/or max can be "--" (two dashes), indicating there is no
# corresponding lower and/or upper bound, respectively.
#
# Min and/or max can be arithmetic formulas, and can reference
# other parameters. Any parameters referenced in a formula must
# already be defined earlier in the parameter set. No white
# space may occur within a formula. Parentheses are recognized
# and can be nested.
#
# 2. parameter value
# The parameter's value will be set to the value given. Value
# can be an arithmetic formula as described above.
#
STRCTLSZ 1024
STRMSGSZ 65535
bufpages 0
create_fastlinks 0
dbc_max_pct 15
dbc_min_pct 5
default_disk_ir 0
dnlc_hash_locks 512
fs_async 0
ksi_alloc_max (NPROC*8)
maxdsiz 0x80000000
maxdsiz_64bit 0x100000000
maxssiz 0x8000000
maxssiz_64bit 0x40000000
maxtsiz 0x4000000
maxtsiz_64bit 0x40000000
maxfiles 2048
maxfiles_lim 2048
maxswapchunks 16384
maxuprc (NPROC-5)
maxusers 124
maxvgs 128
mesg 1
msgmap (NPROC+2)
msgmax 32768
msgmnb 65535
msgmni (NPROC)
msgseg (NPROC*4)
msgssz 128
msgtql (NPROC*10)
nbuf 0
ncallout (NKTHREAD+1)
ncsize (((8*NPROC+2048)+VX_NCSIZE)
netmemmax 0
nfile (15*NPROC+2048)
nflocks (NPROC)
ninode (8*NPROC+2048)
no_lvm_disks 0
nproc 4096
nstrpty 60
npty 512
nswapdev 25
scsi_max_qdepth 128
sema 1
semaem 16384
semmap ((NPROC*2)+2)
semmni (NPROC*2)
semmns ((NPROC*2)*2)
semmnu (NPROC-4)
semume 64
semvmx 32768
shmem 1
shmmax 0x80000000
shmmni 512
shmseg 32
swapmem_on 1
timeslice 10
unlockable_mem (maxusers*10)
vpx_ceiling 64

Please don't use this verbatim - but use it as a guide for your tuning. We spent days putting this .tun together 4-5 years ago with the help of Steve Ciullo - the author of the reference mentioned above. It would be something to look at after you read the reference.

I hope I'm not burying you with information that is more than you need to change a kernel parm or two. In other words - I hope this helps.

Re: Kernel Tuning - nproc + dbc_max_pct

Mel Burslan — Wed, 22 Jun 2005 10:16:28 GMT

Well, I have to disagree with Eric here. Not all kernel changes require a reboot anymore since the introduction of DLKM (dynamically loadable kernel modules) but in your case, you will need to reboot as the dbc_max_pct & dbc_min_pct are static parameters in general.

If you are wondering how to figure out the Dynamic Parms, Run command

kmtune | grep " Y "

the string " Y " is actually answer to setting "is the parameter Dynamic" in the kmtune command output.

Hope this clears it a bit

Re: Kernel Tuning - nproc + dbc_max_pct

rmueller58 — Wed, 06 Jul 2005 12:44:38 GMT

Thanks All,

I am going to change the kernel tonight.

Re: Kernel Tuning - nproc + dbc_max_pct

rmueller58 — Wed, 06 Jul 2005 14:59:23 GMT

Eric

I wish I had the luxury of a decent test system, but unfortunately in an educational system testing is not among the highest priorities, although my inclination is to not do it to err on the side of caution.

I am getting beat up by a director, a manager who in turn are getting flailed by our school clients because of sh*tty performance. So I have to try something..

I am sending out RFQ's for memory upgrade but on this RP7400/N4000 they (HP) seems to be awful proud of the memory (to the tune of $4500/stick of 512). I'd need 4, I am hoping to get approval on 4 1gb sticks.

I am going to take Mel's idea and run with it after I apply the latest QPK and HDW updates. I only going to modify the parameters in question.

Below is my existing "kern_parms", I noticed in that nbuf and bufpages are 0 and (NBUFS*2) or 0..
I will set my dbc_max_pct/dbc_min_pct to reflect what Eric Recommended, But Eric also made several other recommendation I am needing to research more to understand fully the implications.

If any of you see any further Ideas post back I'd appreciate it..

Thanks..
Rex Mueller
Educational Service Unit #3
Omaha, NE

Parameter Value
===============================================================================
NSTRBLKSCHED 2
NSTREVENT 50
NSTRPUSH 16
NSTRSCHED 0
STRCTLSZ 1024
STRMSGSZ 65535
acctresume 4
acctsuspend 2
aio_listio_max 256
aio_max_ops 2048
aio_physmem_pct 10
aio_prio_delta_max 20
allocate_fs_swapmap 0
alwaysdump 0
bootspinlocks 256
bufcache_hash_locks 128
bufpages (NBUF*2)
chanq_hash_locks 256
create_fastlinks 0
dbc_max_pct 50
dbc_min_pct 5
default_disk_ir 0
desfree 0
disksort_seconds 0
dnlc_hash_locks 64
dontdump 0
dskless_node 0
dst 1
eisa_io_estimate 0x300
eqmemsize 15
fcp_large_config 0
file_pad 10
fs_async 0
ftable_hash_locks 64
hdlpreg_hash_locks 128
hfs_max_ra_blocks 8
hfs_ra_per_disk 64
hpux_aes_override 0
initmodmax 50
io_ports_hash_locks 64
iomemsize 40000
km_disable 0
ksi_alloc_max (NPROC*8)
ksi_send_max 32
lotsfree 0
max_async_ports 50
max_fcp_reqs 512
max_mem_window 0
max_thread_proc 256
maxdsiz 262144000
maxdsiz_64bit 0x0000000040000000
maxfiles 1024
maxfiles_lim 1024
maxqueuetime 0
maxssiz 0x00800000
maxssiz_64bit 0x00800000
maxswapchunks 1604
maxtsiz 262144000
maxtsiz_64bit 0x0000000040000000
maxuprc 512
maxusers 512
maxvgs 10
mesg 1
minfree 0
modstrmax 500
msgmap (2+MSGTQL)
msgmax 8192
msgmnb 16384
msgmni 50
msgseg 2048
msgssz 8
msgtql 40
nbuf 0
ncallout 2064
ncdnode 150
nclist (100+16*MAXUSERS)
ncsize (NINODE+VX_NCSIZE)
ndilbuffers 30
netisr_priority -1
netmemmax 0
nfile 51548
nflocks 2048
nhtbl_scale 0
ninode ((NPROC+16+MAXUSERS)+32+(2*NPTY))
nkthread 2048
nni 2
no_lvm_disks 0
nproc 3072
npty 512
nstrpty 512
nstrtel 512
nswapdev 10
nswapfs 10
nsysmap ((NPROC)>800?2*(NPROC):800)
nsysmap64 ((NPROC)>800?2*(NPROC):800)
num_tachyon_adapters 0
o_sync_is_o_dsync 0
page_text_to_local 0
pfdat_hash_locks 128
public_shlibs 1
region_hash_locks 128
remote_nfs_swap 0
rtsched_numpri 32
scroll_lines 100
scsi_max_qdepth 8
scsi_maxphys 1048576
sema 1
semaem 16384
semmap (SEMMNI+2)
semmni 4096
semmns 4096
semmnu 30
semmsl_override 2048
semume 10
semvmx 32767
sendfile_max 0
shmem 1
shmmax 2048000000
shmmni 512
shmseg 256
st_ats_enabled 0
st_fail_overruns 0
st_large_recs 0
st_san_safe 0
streampipes 0
swapmem_on 1
swchunk 2048
sysv_hash_locks 128
tcphashsz 0
timeslice (100/10)
timezone 420
unlockable_mem 0
vnode_cd_hash_locks 128
vnode_hash_locks 128
vps_ceiling 16
vps_chatr_ceiling 65536
vps_pagesize 4
vx_maxlink 32767
vx_ncsize 1024
vx_ninode 0
vx_noifree 0
vx_nothrottle 1
vxfs_max_ra_kbytes 1024
vxfs_ra_per_disk 1024

Re: Kernel Tuning - nproc + dbc_max_pct

rmueller58 — Wed, 06 Jul 2005 15:58:03 GMT

Eric,

when you mention setting Values with (NPROC) or (NPROC*9/10) etc, do you mean calculate the value based on the value of NPROC or just use the "TEXT" NPROC??

Re: Kernel Tuning - nproc + dbc_max_pct

rmueller58 — Wed, 06 Jul 2005 16:08:43 GMT

# diff kern_parms kern_edit
22,23c22,23
< dbc_max_pct 50
< dbc_min_pct 5
---
> dbc_max_pct 10
> dbc_min_pct 7
52c52
< maxdsiz 262144000
---
> maxdsiz 536870912
57,59c57,59
< maxssiz 0x00800000
< maxssiz_64bit 0x00800000
< maxswapchunks 1604
---
> maxssiz 134217728
> maxssiz_64bit 1073741824
> maxswapchunks 16384
62c62
< maxuprc 512
---
> maxuprc (NPROC*9/10)
71,72c71,72
< msgmni 50
< msgseg 2048
---
> msgmni (NPROC)
> msgseg 32767
74c74
< msgtql 40
---
> msgtql (NPROC)
83,84c83,84
< nfile 51548
< nflocks 2048
---
> nfile (15*NPROC+2048)
> nflocks 4096
86c86
< ninode ((NPROC+16+MAXUSERS)+32+(2*NPTY))
---
> ninode 2048
90c90
< nproc 3072
---
> nproc 400
92c92
< nstrpty 512
---
> nstrpty 60
113,114c113,114
< semmns 4096
< semmnu 30
---
> semmns (SEMMNI*2)
> semmnu (NPROC-4)
120c120
< shmmax 2048000000
---
> shmmax 21474836478
129c129
< swchunk 2048
---
> swchunk 4096
137c137
< vps_ceiling 16
---
> vps_ceiling 64
#

Re: Kernel Tuning - nproc + dbc_max_pct

Eric Antunes — Thu, 07 Jul 2005 04:04:52 GMT

Hi Rex,

After reading your kernel listing, I suggest you change the following parameters (I'm not refering the 64bit parameter assuming your OS is 32 bits):

maxdsiz to 536870912, maxssiz to 134217728, maxswapchunks to 16384, maxtsiz to 134217728, nproc to 3072 (no changes, just refering it...), maxuprc to (NPROC*9/10), maxusers to 512 (no changes, just refering it...), msgmni to (NPROC), msgseg to 32767, msgtql to (NPROC), nfile to (15*NPROC+2048), nflocks to 4096, ninode to 2048: start with this value and check with "sar -v" if it doesn't overflow. If it overflows increase it. See my stats:

#sar -v 6 10

HP-UX B.11.00 U 9000/800 07/07/05

10:02:55 text-sz ov proc-sz ov inod-sz ov file-sz ov
10:03:01 N/A N/A 265/400 0 1598/2048 0 3611/8058 0
10:03:07 N/A N/A 265/400 0 1598/2048 0 3611/8058 0
10:03:13 N/A N/A 266/400 0 1603/2048 0 3614/8058 0
10:03:19 N/A N/A 265/400 0 1606/2048 0 3624/8058 0
10:03:25 N/A N/A 265/400 0 1589/2048 0 3611/8058 0
10:03:31 N/A N/A 265/400 0 1573/2048 0 3609/8058 0
10:03:37 N/A N/A 265/400 0 1565/2048 0 3602/8058 0
10:03:43 N/A N/A 265/400 0 1556/2048 0 3571/8058 0
10:03:49 N/A N/A 265/400 0 1551/2048 0 3571/8058 0
10:03:55 N/A N/A 265/400 0 1534/2048 0 3571/8058 0

The inod-sz column is the inode size and ov > 0 indicates overflow.

..., semmns to (SEMMNI*2), semmnu to (NPROC-4), swchunk to 4096 and vps_ceiling to 64.

Everything that is between () is a formula.

Best Regards,

Eric Antunes

Re: Kernel Tuning - nproc + dbc_max_pct

Eric Antunes — Thu, 07 Jul 2005 04:18:29 GMT

Hi again,

This kernel changes are more easy to do in SAM since it validates, for each parameter, if your new value/formulas is consistant with other parameters and for the whole system.

Of course, also change dbc_max_pct to 10, dbc_min_pct to 7 and shmmax to 2147483648. Check "sar -b 6 100" for you buffer cache ratio. Here is an example (my stats with only 10 iterations):

#sar -b 6 10

HP-UX B.11.00 U 9000/800 07/07/05

10:16:21 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
10:16:27 3 121 97 10 10 5 0 0
10:16:33 4 252 98 10 13 25 0 0
10:16:39 4 499 99 12 11 0 0 0
10:16:45 24 500 95 29 176 83 0 0
10:16:51 4 526 99 30 62 52 0 0
10:16:57 7 208 97 20 23 11 0 0
10:17:03 0 27 99 5 6 6 0 0
10:17:09 1 383 100 12 14 14 0 0
10:17:15 16 1535 99 46 555 92 0 0
10:17:21 34 1143 97 30 60 50 0 0

Average 10 519 98 21 93 78 0 0

"%rcache" is you buffer cache ratio: you just have to reach more than 90% here.

Best Regards,

Eric Antunes