topic Re: buffer cache in HP 11.11 ? in Operating System - HP-UX

buffer cache in HP 11.11 ?

Sammy_2 — Wed, 28 Sep 2005 09:15:00 GMT

Seeing slow disk perf from all disks (local or San disks) . Cpu , memory looks good. Trying to increase buffer cache to see any improvement

have 33gb of memory but buffer cache is only 46.9mb (via glance) ?Isnt' buffer cache controlled by dbc_min_pct
and dbc_max_pct ? I change dbc_max_pct from 10 to 30 but thru glance I see buffer cache as only 46.9mb ? Why ?

dbc_max_pct 30 - 30
dbc_min_pct 5 - 5
====================
Total VM : 2.31gb Sys Mem : 1.73gb User Mem: 2.30gb Phys Mem: 32.0gb
Active VM: 1.65gb Buf Cache: 46.9mb Free Mem: 27.9gb Page 1 o

Re: buffer cache in HP 11.11 ?

A. Clay Stephenson — Wed, 28 Sep 2005 09:30:37 GMT

Buffer cache is controlled by dbc_xxx_pct unless bufpages is set to a non-zero value. In that case, buffer cache is fixed to bufpages X 4KB. Under 11.11 (with large amounts of memory), I typically use bufpages to fix the buffer cache at about 800-1600MB.

Re: buffer cache in HP 11.11 ?

Geoff Wild — Wed, 28 Sep 2005 09:31:26 GMT

First off - you don't need dbc_max_pct @ 30...

should be set to 2 (same for dbc_min_pct).

That should give you 655 MB of buffer cache.

Now - as to why glance isn't displying the correct ammount? I don't know...

Do you have kmeminfo on your system?

See docid: KBRC00011764

http://www2.itrc.hp.com/service/cki/search.do?category=c0&prevQueryString=KBRC00011764&mode=id&searchString=KBRC00011764&searchCrit=allwords&docType=Security&docType=Patch&docType=EngineerNotes&docType=BugReports&docType=Hardware&docType=ReferenceMaterials&docType=ThirdParty&dateRange=all&search.x=29&search.y=11

Rgds...Geoff

Re: buffer cache in HP 11.11 ?

Sammy_2 — Wed, 28 Sep 2005 10:00:00 GMT

Clay,
I changed bufpages to 0 and changed 3 and 10 for dbc_min and max. But glance still show 46.9mb. I have 32gb memory on system. Odd ?
kumtune show bufpages to be 0 but when i run
# echo bufpages/D | adb /stand/vmunix /dev/kmem
bufpages:
bufpages: 12000

shows 12000. that was the old value of (nbuf*2) but I had changed it to 0 before re-gen the kernel.

# kmtune|egrep -i 'buf|dbc'
bufcache_hash_locks 128 - 128
bufpages 0 - 0
dbc_max_pct 10 - 10
dbc_min_pct 3 - 3

B3692A GlancePlus C.03.72.00 09:54:24 ojrdhes3 9000/800 Current Avg High
--------------------------------------------------------------------------------
CPU Util SU | 2% 3% 9%
Disk Util F FV | 7% 16% 24%
Mem Util S SU U | 13% 13% 13%
Swap Util RR | 5% 5% 5%
--------------------------------------------------------------------------------
MEMORY REPORT Users= 1
Event Current Cumulative Current Rate Cum Rate High Rate
--------------------------------------------------------------------------------
Page Faults 4794 7100 494.2 682.6 3294.2
Page In 2005 3035 206.7 291.8 1471.4
Page Out 0 0 0.0 0.0 0.0
KB Paged In 536kb 804kb 55.2 77.3 382.8
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Reactivations 0 0 0.0 0.0 0.0
Deactivations 0 0 0.0 0.0 0.0
KB Deactivated 0kb 0kb 0.0 0.0 0.0
VM Reads 31 47 3.1 4.5 3.1
VM Writes 4 4 3.1 0.3 0.4

Total VM : 2.30gb Sys Mem : 1.73gb User Mem: 2.31gb Phys Mem: 32.0gb
Active VM: 1.63gb Buf Cache: 46.9mb Free Mem: 27.9gb Page 1 of 1

# echo bufpages/D | adb /stand/vmunix /dev/kmem
bufpages:
bufpages: 12000

Thanks

Re: buffer cache in HP 11.11 ?

Jeff Lightner_1 — Thu, 29 Sep 2005 08:10:50 GMT

If you want to use dynamic buffer cache you have to make sure both bufpages and nhbuf are 0.

Assuming this is the case already then "dynamic" is the clue for why you only see what you do. In dynamic buffer cache the cache only grows as needed though over time it will normally grow to whatever you set dbc_max_pct to.

Personally I don't use DBC - I prefer to hard set my buffer cache to 600 MB on 11.11 systems. The caching algorithm for most purposes is not efficient beyond that level and having it allocated up front means I don't have to worry about it when doing other memory research.

Re: buffer cache in HP 11.11 ?

Stuart Abramson — Thu, 29 Sep 2005 08:23:22 GMT

nbuf and bufpages both come set to 0 default, and we don't mess with them. We set dbc_max_pct to 10% or so (smaller on bigger systems). The default "max" is 50% and that just wastes RAM, if you don't change it when you install the system.

Re: buffer cache in HP 11.11 ?

Stuart Abramson — Thu, 29 Sep 2005 08:25:34 GMT

Your real problem is why is your disk I/O slow. Fix that.

How do you know it's slow?

How are you measuring?

What was it BEFORE. Did you change something?

sar -d

Are ALL disks slow, or just some?

Re: buffer cache in HP 11.11 ?

baiju_3 — Thu, 29 Sep 2005 08:32:00 GMT

Hi All ,

I was reading this inetersting thread .I checked my QA System with 64 GB memory , and dbc max and min set as 10 and 5 .I could see that buf cache is 6.4GB in glance .

Now few questions .

1)Is buf cache is only used for disk I/O ?

2)Is amount of buffer cache reserved varies depending on the usage ?

3)Is it possible to monotor buf cache usage .

Your answers will be appreciated .

Thanks,
BL.

Re: buffer cache in HP 11.11 ?

Sammy_2 — Thu, 29 Sep 2005 09:10:08 GMT

I am testing by using the following command: (basically tarring up abt 368mb home directory on local disks.
glance shows disks peak to 100% for 7-10 seconds. does not matter if tar a directory on local or EMC San disks.
same bad performace.

# timex tar -cvf TEST.TAR /home/sam01
..
..

real 21.70
user 3.49
sys 4.03

# ll *TAR
-rw-r--r-- 1 root sys 368424960 Sep 29 09:56 TEST.TAR

output of "glance -u"
======================

------------------------------------------------------------------------------------------------------------------------------
Disk Detail for: 0/1/1/0.1.0 Bus Type: scsi
Controller: na Product ID: ST373454LC Vendor: HP 73.4G
--------------------------------------------------------------------------------
Log Reads : na Phy Reads : 562.0 Util (%): 100 q=0 (%): 91
Writes : na Writes : 370.8 Q Len : 3.2 0 Rd Bytes: na Rd Bytes: 6192.4 2 Wt Bytes: na Wt Bytes:16865.7 4--------------------------------------------------------------------------------
Mapping Information

I never see Wt Bytes in glance -u output jump to more than 17000 but on similiar less
powerful HP systems , same command finishes in 10-12sec. Wt bytes often go upto 20000 - 23000
on another hpux 11.11 system.

Nothing changed as this is happening in several new Hp ux 11.11 boxes managed by
different unix admins.

=======

Oracle Dba exports are taking 10 slower than
on other systems. But this is how I am
doing a test
==========

no matter what disks I write to , it is slow. Dba do export on EMC8830 disks and just to make sure,i did a tar test on local SEAGATE disks, and perf. was lot slower than other boxes where I did the same test. Both systems are very much idle. No cpu issue, no memory issue

Re: buffer cache in HP 11.11 ?

A. Clay Stephenson — Thu, 29 Sep 2005 09:35:44 GMT

Your fundamental problem is that your buffer cache is too small (only 48MB). The buffer cache lookups in 11.11 are more efficient than those of earlier HP-UX versions and thus larger buffer caches (within reason) work well. On boxes with large amounts of memory (and yours qualifies), I would set the buffer cache to somewhere between 1200-1600MB as a good sweet spot. I would not use dynamic buffer cache because the values will change as you add or remove physical memory. Set nbuf=0 and bufpages=307200 (1200MB) or bufpages=409600 (1600MB).

Your other problem is that you are measuring too many things at once with your tar test. The idea is to isolate the i/o components as much as possible.

For example, to do a test write to a disk, you don't want to involve reading and you don't want the filesystem or buffer cache overhead because that's just noise. Use a device node as an input device and a raw device as output.

e.g.

timex dd if=/dev/zero bs=256k count=400 of=/dev/rdsk/c1t5d0

You could also use a raw logical device:
of=/dev/vg01/rlvol1

You would reverse the process to do a test read:
timex dd if=/dev/vg01/rlvol1 bs=256k count=400 of=/dev/null

Each of these will do 100MB's of I/O.

Now do the same commands but up a step in the "food chain" - block devices.

Using /dev/dsk/c1t5d0 rather than /dev/rdsk/c1t5d0 or /dev/vg01/lvol1 rather than /dev/vg01/rlvol.

Finally introduce fully cooked files (but still use the device nodes) so that
/dev/dsk/c1t5d0 becomes "/u01/myfile".

This will give you the basic data to make intelligent choices about where your actual problems lie.

Re: buffer cache in HP 11.11 ?

Sammy_2 — Thu, 29 Sep 2005 09:55:33 GMT

clay, I will try that.

timex dd if=/dev/zero bs=256k count=400 of=/dev/rdsk/c1t5d0

Before I do this, The above command is not going to blank out
c1t5d0 ? Is that right?
I got data on that disk

Re: buffer cache in HP 11.11 ?

A. Clay Stephenson — Thu, 29 Sep 2005 10:10:51 GMT

Wrong. dd will write 100MB's of zero's to whatever output device you point it at. This is a destructive test. The read tests writing to /dev/null will be non-destructive. Your best best is to find an unused section of a VG and create an LVOL. Doing the tests on an LVOL is almost as low-level as doing it on the disk device itself.

Re: buffer cache in HP 11.11 ?

Sammy_2 — Thu, 29 Sep 2005 12:38:53 GMT

I am writing to a disk not part of any vg
Do i need to create /dev/zero file. Will

#timex dd if=/dev/zero bs=256k count=400 of=/dev/rdsk/c0t1d0
/dev/zero: No such file or directory
dd: cannot open /dev/zero

real 0.02
user 0.00
sys 0.01

Re: buffer cache in HP 11.11 ?

James R. Ferguson — Thu, 29 Sep 2005 12:46:35 GMT

Hi SammY:

You say that you are running 11.11 but don't have /dev/zero --- that's strange.

You can verify this with:

# lssf /dev/* 2> /dev/null|grep "mm"

If /dev/zero is truly missing, create it thusly:

# mknod /dev/zero c 3 0x000004
# chown bin:bin /dev/zero
# chmod 444 /dev/zero

Regards!

...JRF...

Re: buffer cache in HP 11.11 ?

Sammy_2 — Thu, 29 Sep 2005 13:12:30 GMT

JRF,
My fault. To test the command, I actually ran that test on hpux 11.0 (and not realize it). I see /dev/zero is on 11.11 system now.

Re: buffer cache in HP 11.11 ?

baiju_3 — Thu, 29 Sep 2005 13:20:21 GMT

Pls do not run this

timex dd if=/dev/zero bs=256k count=400 of=/dev/rdsk/c0t1d0

This is destructive , you will end up in corrupting data on c0t1d0

thx,
-bl.

Re: buffer cache in HP 11.11 ?

Sammy_2 — Tue, 04 Oct 2005 09:09:26 GMT

The problem was with the brocade port connected to EMC frame. We switched all prim path to alternate and all was fine. Clay dd test were useful in doing little comparison before and after the switch from prima to secondary path.

Re: buffer cache in HP 11.11 ?

Sammy_2 — Tue, 04 Oct 2005 09:10:35 GMT

All those who suggested increasing the buffer cache also helped as the system is now faster than what it was when prim path was working fine.