topic Re: write performance too slow / changing memory cash in Operating System - HP-UX

write performance too slow / changing memory cash

medali.benchaabane — Tue, 22 Aug 2006 10:56:46 GMT

HI
I have a two node cluster HPUX 11.11 connected to an EVA 3000.
I created a new file system on the eva and presented it to each system one by one.
I tried to write some date on it and compare the results.
on test system
# time cp test.tmp test.tmp.1
real 0m46.34s
user 0m0.03s
sys 0m16.98s
#
on 1st production system
On BSCS1
bscs1:/test_perf2 #time cp test.tmp test.tmp.1
real 1:11.5
user 0.0
sys 25.8
bscs1:/test_perf2 #

On BSCS2
bscs2 root:/test_perf2 #time cp test.tmp test.tmp.1
real 7:57.0
user 0.2
sys 34.2
bscs2 root:/test_perf2 #

the second production system is 7 times slower than the test system which is crazy.

I enclosed some performance results.

I enclosed also the result of
STM > TOOLS > UTILITY > RUN > LOGTOOL > FILE > RAW SUMMARY

this thread is opened as consequence of this one http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1052343

where do you thing the problem is, most probably it's related to momory cache, even with the help of the support we couldn't find where the problem is and how to fix it.
thanks.

Re: write performance too slow / changing memory cash

IT_2007 — Tue, 22 Aug 2006 11:02:07 GMT

First I would check where cp command being invoked from PATH? then look for any error messages in syslog. Also check IO timeout value on the disk.

Also check network settings on the othernode which you have problems like it might set to 100HD with Auto negotiation which will put it slow.

Re: write performance too slow / changing memory cash

medali.benchaabane — Tue, 22 Aug 2006 11:26:49 GMT

Hi,

the cp is voked from the same file system on the same server, so the data is not moving on the network.

this is the file system created

/dev/test_perf1/lvol1
2048000 1606 1918502 0% /test_perf1

the cp is on the same server
cd /test_perf1
cp /test_perf1/test.tmp test.tmp1

it couln't be pb of the network?

Re: write performance too slow / changing memory cash

A. Clay Stephenson — Tue, 22 Aug 2006 11:35:26 GMT

You really need to isolate the problem better when you both read and write from a filesystem at the same time, it's difficult to know what you are actually measuring. A much better test is to read from a device node and write to a device node.

e.g. as a read test:

timex dd if=myfile of=/dev/null bs=64k

a write test:
timex dd if=/dev/zero bs=64k count=1000 of=myfile

It's even better if you do not use a fully cooked file such as myfile but instead write directly to a raw device. This will isolate the filesystem overhead from the disk i/o.

raw read test:
timex dd if=/dev/rdsk/c1t5d0 bs=64k count=1000 of=/dev/null bs=64k

raw write test: NOTE THIS S DESTRUCTIVE and will overwite any filesystem data that happens to be on a disk so know what you are doing.
timex dd if=/dev/zero bs=64k count=1000 of=/dev/rdsk/c1t5d0

Man dd for details.

Re: write performance too slow / changing memory cash

spex — Tue, 22 Aug 2006 11:42:14 GMT

Hi,

Check if filesystem mount options differ between the nodes:

bscs1:/ # mount -v | grep /test_perf2
bscs2:/ # mount -v | grep /test_perf2

PCS

Re: write performance too slow / changing memory cash

medali.benchaabane — Tue, 22 Aug 2006 11:55:28 GMT

here is the results :
bscs2 root:/tmp #time dd if=/dev/test_perf2/rlvol1 bs=64k count=1000 of=/dev/null bs=64k
1000+0 records in
1000+0 records out

real 9.4
user 0.0
sys 0.1
bscs2 root:/tmp #

bscs2 root:/tmp #time dd if=/dev/zero bs=64k count=1000 of=/dev/test_perf2/rlvol1
1000+0 records in
1000+0 records out

real 1.2
user 0.0
sys 0.1

bscs1:/dev #time dd if=/dev/test_perf2/rlvol1 bs=64k count=1000 of=/dev/null bs=64k
1000+0 records in
1000+0 records out

real 7.4
user 0.0
sys 0.1

bscs1:/dev #time dd if=/dev/zero bs=64k count=1000 of=/dev/test_perf2/rlvol1
1000+0 records in
1000+0 records out

real 1.4
user 0.0
sys 0.2
bscs1:/dev #

thanks

Re: write performance too slow / changing memory cash

medali.benchaabane — Tue, 22 Aug 2006 12:01:30 GMT

mount -v | grep test_perf2
/dev/test_perf2/lvol1 on /test_perf2 type vxfs log,nodatainlog on Tue Aug 22 18:59:12 2006

Re: write performance too slow / changing memory cash

Steven E. Protter — Tue, 22 Aug 2006 12:07:06 GMT

Shalom,

These days write performance is often dependent on the configuration of LUNS. If they are not configured for raid 1 or raid 10 write performance will be awful in write intensive systems.

It is common for performance to get hammered due to poor cache performance, but I agree at this point the cause has not been identified.

http://www1.itrc.hp.com/service/cki/search.do?docType=Security&docType=Patch&docType=EngineerNotes&docType=BugReports&docType=Hardware&docType=ReferenceMaterials&docType=ThirdParty&searchString=UPERFKBAN00000726&searchCrit=allwords&search.x=28&admit=-682735245+1156266342722+28353475&category=c0&mode=id&search.y=8

This document will give you more to look at in eliminating local causes of this issue.

SEP

Re: write performance too slow / changing memory cash

medali.benchaabane — Tue, 22 Aug 2006 12:14:36 GMT

sorry but I had no response in this link

Note: We're sorry but your search produced 0 results

Re: write performance too slow / changing memory cash

Michael Steele_2 — Tue, 22 Aug 2006 20:16:44 GMT

Looking at your sar -b (* buffer activity report *) rcache can be increased from 78% to 98% - 100%. Paste in your values for

bufpages
Pages of static buffer cache

dbc_min_pct
Minimum dynamic buffer cache

dbc_max_pct
Maximum dynamic buffer cache

nbuf
Number of static buffer headers

Looking at the rest like sar -q I don't see a lot of activity so I have to ask if you took these reports while the system was stressed? Perhaps a 15 min. cron job would be inorder for a historical reference over 24 hours and a week.

For swap use vmstat 5 5 and swapinfo.

Re: write performance too slow / changing memory cash

medali.benchaabane — Wed, 23 Aug 2006 03:18:42 GMT

Hi,
these are my values for
bscs2 root:/tmp #cat /stand/system | grep dbc_min_pct
dbc_min_pct 2
bscs2 root:/tmp #cat /stand/system | grep dbc_max_pct
dbc_max_pct 10
bscs2 root:/tmp #cat /stand/system | grep bufpages
bscs2 root:/tmp #cat /stand/system | grep nbuf

The system is at normal activity not stressed

Re: write performance too slow / changing memory cash

Michael Steele_2 — Tue, 29 Aug 2006 03:13:16 GMT

I'd lower my dbc_max_pct to 5. But this is very oracle sensitive so I'd verify with dba's first, just to cover your....