Slow internal disk

[Scott Minter]

Why is my throughput so low?

Performing:
time dd if=file1 of=/dev/null
results in 24.3 seconds or approximately 4.11MB/s transfer rate.

Screen copy of action:

104858624 Feb 22 14:14 redo01.log
$ time dd if=redo01.log of=/dev/null
204802+0 records in
204802+0 records out

real 24.3
user 12.3
sys 4.0

This is from the internal disk on an L2000. The disk is 18.2 GB LVD 10K rpm internal (A5803A).

When I ran this just now glance showed:
cpu 3%
disk 20%
mem 44%
swap 31%

so there was activity on the system. However, I have run the test after hours with all other file system unmounted and I never get more than 7MB/s throughput.

I am patched up through the December 2004 Support Plus CD.

I have patches PHKL_30511 and PHKL_32090.

HP has suggested the following patches as well:
PHCO_30698 LVM commands cumulative patch
PHKL_30257 Fibre Channel Mass Storage Patch
PHKL_30607 VxVM,EMC,PM,vPar,slpq,KRS,DUMP,LVM,VM,DaS,IF
PHKL_31216 LVM Cumulative Patch; LVM OLR;

and I will load them as well.

After loading patches I have not seen any signifant increase.

I also have external raid arrays - ultra160 and see slow throughput with them as well (6-7MB/s throughput, 1MB/s backing up from Openview Data Protector).

I think I'm missing something very obvious here. Any suggestions?

[harry d brown jr]

Have you tried to increase the block size (bs=###)?

live free or die
harry d brown jr

[Scott Minter]

I have not tried increasing the block size.

Could that be slowing down these disks that much (from 160MB/s to 6-7MB/s)?

[harry d brown jr]

well the DEFAULT is 512 bytes.

thus 204802 * 2 IO actions (+/- a few cached).

So increase your bs to a bigger number!

(see man dd)

live free or die
harry d brown jr

[Leif Halvarsson_2]

Hi,

You will NEVER get 160MB/s from one disk, 160MB/s is the theoretical speed of the bus, not when transfering data to the magnetic platters. Specification for a 10k disk says 39-80MB/s sustained transfer rate (I belive, depending on inner/outer tracks).

Increasing blocksize for dd may help or, use a better benchmark tool (e.g. bonnie).

[Scott Minter]

I'm fully aware I won't get 160MB/s, but I should be getting performance than 4MB/s.

Changing the block size increased the throughput tremendously. See new output below:

$ time dd bs=32768 if=redo01.log of=/dev/null
3200+1 records in
3200+1 records out

real 3.9
user 0.0
sys 0.7

That is 25MB/s. Still, a lot lower than the 39-80, but an improvement all the same.

I will try to find bonnie and test with it.

[harry d brown jr]

[root@vpart1 /var/appl]# ls -l xrated
-rw-rw-rw- 1 root sys 665931776 Sep 22 09:27 xrated
[root@vpart1 /var/appl]# timex dd if=xrated of=/dev/null
58655+0 records in
58654+0 records out

real 4.18
user 2.87
sys 0.94

[root@vpart1 /var/appl]# date;timex dd if=xrated of=/dev/null;date
Tue Feb 22 15:35:05 EST 2005
1300648+0 records in
1300648+0 records out

real 1:32.35
user 1:03.35
sys 20.34

Tue Feb 22 15:36:37 EST 2005
[root@vpart1 /var/appl]# date;timex dd if=xrated of=/dev/null bs=1024;date
Tue Feb 22 15:38:20 EST 2005
650324+0 records in
650324+0 records out

real 22.00
user 1.53
sys 12.65

Tue Feb 22 15:38:42 EST 2005
[root@vpart1 /var/appl]# date;timex dd if=xrated of=/dev/null bs=2048;date
Tue Feb 22 15:38:56 EST 2005
325162+0 records in
325162+0 records out

real 19.11
user 0.75
sys 8.57

Tue Feb 22 15:39:15 EST 2005
[root@vpart1 /var/appl]# date;timex dd if=xrated of=/dev/null bs=4096;date
Tue Feb 22 15:39:36 EST 2005
162581+0 records in
162581+0 records out

real 33.36
user 0.38
sys 5.90

Tue Feb 22 15:40:10 EST 2005
[root@vpart1 /var/appl]# date;timex dd if=xrated of=/dev/null bs=8192;date
Tue Feb 22 15:40:30 EST 2005
81290+1 records in
81290+1 records out

real 37.94
user 0.20
sys 5.72

Tue Feb 22 15:41:08 EST 2005
[root@vpart1 /var/appl]# date;timex dd if=xrated of=/dev/null bs=16384;date
Tue Feb 22 15:41:31 EST 2005
40645+1 records in
40645+1 records out

real 25.35
user 0.10
sys 4.95

Tue Feb 22 15:41:56 EST 2005
[root@vpart1 /var/appl]# date;timex dd if=xrated of=/dev/null bs=32768;date
Tue Feb 22 15:42:12 EST 2005
20322+1 records in
20322+1 records out

real 24.86
user 0.06
sys 4.63

Tue Feb 22 15:42:37 EST 2005
[root@vpart1 /var/appl]# date;timex dd if=xrated of=/dev/null bs=65536;date
Tue Feb 22 15:43:14 EST 2005
10161+1 records in
10161+1 records out

real 21.11
user 0.03
sys 4.50

Tue Feb 22 15:43:36 EST 2005
[root@vpart1 /var/appl]# date;timex dd if=xrated of=/dev/null bs=131072;date
Tue Feb 22 15:43:52 EST 2005
5080+1 records in
5080+1 records out

real 20.25
user 0.02
sys 4.39

Tue Feb 22 15:44:13 EST 2005
[root@vpart1 /var/appl]# date;timex dd if=xrated of=/dev/null bs=262144;date
Tue Feb 22 15:44:34 EST 2005
2540+1 records in
2540+1 records out

real 18.91
user 0.01
sys 0.56

Tue Feb 22 15:44:53 EST 2005
[root@vpart1 /var/appl]# date;timex dd if=xrated of=/dev/null bs=524288;date
Tue Feb 22 15:45:40 EST 2005
1270+1 records in
1270+1 records out

real 18.97
user 0.01
sys 0.54

Tue Feb 22 15:45:59 EST 2005
[root@vpart1 /var/appl]# date;timex dd if=xrated of=/dev/null bs=1048576;date
Tue Feb 22 15:46:09 EST 2005
635+1 records in
635+1 records out

real 18.75
user 0.00
sys 0.50

Tue Feb 22 15:46:28 EST 2005
[root@vpart1 /var/appl]#


I guess there's a limit to how much BS you can use :-))

live free or die
harry d brown jr

[Xavier Gutierrez_2]

Hi, Scott.

I made some tests in the past and the best results were always achieved with a block size of 4MB (bs=4096k), no more, no less.

Later on, I spoke about this with our Account Support Engineer and he said that this was because that was exactly the size of the buffer of those disks.

If you happen to do some tests on block size, make sure you combine both block size and count options so that you come with a steady amount of data to transfer in the operation. This way you can compare times. Also, make sure the amount of transferred data is big enough to notice the difference...

Hope this helps,

Xavier.

[Scott Minter]

Even with changing the block size to 8196 I still only get 8MB/s throughput.

Any idea why my throughput would be so lousy on this disk (I've run the test when I'm the only user on the disk and still poor performance).

[Leif Halvarsson_2]

Hi,
I tried the dd command on one of our systems.

if was a very large (1.2GB) database file, of /dev/null.

With bs=4096 I got about 30MB/s. No significant improvement with bloksize above 4k.

With blocksize <4k user and sys time begin to increase but I still got 20MB/s with bs=512.

Disk was mirrored 15k.

[Scott Minter]

Do you have the same setup as I describe above (ultra160 card)?

[A. Clay Stephenson · Accepted solution]

One of the things that you are doing that really skews your results is the use of cooked files. You really aren't measuring disk performance or blocksize but rather the efficiency of the filesystem and the buffer cache. You need to get the filesystem and buffer cache out of the picture and read from the raw device.

e.g.

dd if=/dev/rdsk/c1t5d0 bs=256k count=400 of=/dev/null

[Scott Minter]

Thank you Clay. You are absolutely correct.

Now I'm getting 28MB/s throughput. My throughput is limited by my drive channel in the SCSI array at 40MB/s.

This would indicate my performance is being hurt by the file system wouldn't it?