dd for disk performance, buffer cache, and hardware-level cache

[A. Daniel King_1]

Hi, folks.

I've got some easy points for anyone wishing to point out the virtues and/or shortcomings of dd and system cache at several levels.

Background:

HP-UX 11.0, V-Class box, various disk architectures.

Does dd use buffer or any other file-system cache at the OS level? I'm trying to run some disk-level performance testing - something like:

$time dd if=some_known_big_file of=/dev/null bs=10k

Is this a real measure of disk performance? Does this test cache more than disk performance? Will this time vary if I repeat the test?

I believe that, even if the OS is not caching, the machine and/or the disk units probably ARE caching. I know that EMC Symmetrics have quite a bit of cache, and I believe that even individual disk units have some cache. How about HP Galaxy disk? Jamaica?

I note that:

$time dd if=some_known_big_file of=/dev/null ibs=10k obs=10k

is quite a bit slower.

Is there a better test than dd?

Thanks.

[harry d brown jr]

To gauge performance I'd use glance, with multiple "dd"'s running, until I saturated the IO.

live free or die
harry

[Stefan Farrelly · Accepted solution]

If you use dd to a rlvol then no, it does NOT use the unix buffer cache. If you dd an lvol or a filename then yes, it does use the unix buffer cache.

Yes, dd will be quicker on a rerun. Especially if youre using say an EMC or XP or any sort of disk array with cache on it, as even if you dd a raw lvol it will still get cached up on the disk array and thus subsequent runs will be faster. Thus - always use the first run as a benchmark as its not in cache.
Even using an individual disk with raw lvol subsequent runs will be slightly quicker as most disks have a little cache on them - so either pick the first run or average over a few runs.

The answer is yes, dd is the best tool for measuring disk performance at a low level - unless you purchase a highlevel tool to do it.

EMC's come with tons of cache so subsequent runs are always tons faster. Jamaica's are dump disks - but even individual disks have a little cache so subsequent runs will be slightly quicker.
I dont know about HP Galaxy disks - what on earth are they ? if new then almost certainly they have lots of cache.

[James R. Ferguson]

Hi Daniel:

One comment. If you want to begin to measure disk metrics, use the raw device (/dev/rdsk/cXtYdZ) and make sure the input and output block sizes are the same. You can use the 'count' option to constrain the transfer to a given amount (size) of your choice. See the 'dd' manpages for more information.

Regards!

...JRF...

[A. Clay Stephenson]

One thing that jumped out at me was your command:
"dd if=some_known_big_file of=/dev/null ibs=10k obs=10k".

That is very different from this seemingly equivalent command:"dd if=some_known_big_file of=/dev/null bs=10k".

In the former, there is an extra step that dd itself handles within its own process space (regardless of any additional UNIX buffer cache that might be involved); it is the copy from the input buffer to the output buffer. By specifying only 'bs=10k' both input and output use a common buffer and the process's copy operation is avoided.

You should also test writing from a device file as well.

dd if=/dev/zero of=/dev/rdsk/cXtYdZ bs=10k; the /dev/zero will supply an endless stream of ASCII NUL's.

As has been stated, dd is at best only a fair tool for measuring disk performance but the least you can do is get rid of the overhead of the extra copy operations to provide a better measurement of what you are actually trying to gauge.

[A. Daniel King_1]

Galaxy - This is how I've heard some of the HP NIKE disk referenced. Am I the only one who has heard this?

[A. Daniel King_1]

I wonder, though, why specifying the obs and ibs could be so much significantly slower?

It's been somewhere in the vicinity of three times slower.

Could there potentially be a CPU bottleneck skewing the results?

[A. Clay Stephenson]

The buffer copy overhead can be quite significant but you also need to be very careful that you compare apples to apples. If you first do your test reading from a cooked file and with both an input buffer and an output buffer and then soon retest with a common buffer, your result are going to be skewed because there is a very good chance that the second test used many of the blocks in the UNIX buffer cache. You can eliminate that bit of skewing by comparing result copying from a raw device.

You might have a CPU or memory bottleneck but I rather doubt it.

[Bill Hassell]

Your dd command has been crippled by the tiny blocksize. Dump the obs and ibs and just use bs=128k. You will see a dramatic increase in performance.

As far as a disk metric, dd is pretty meaningless. Not even a backup behaves like dd (unless you are backing up raw disks). There are reams of articles on how to characterize disk performance but most of it boils down to the same thing: large sequential transfers on separate channels is the fastest way to move data. Since the real world seldom behaves this way, modern disk arrays incorporate massively large data caches.

By the way Nike and Jamaica are VERY old technology. Today, you would look at HP's VA-series of disk arrays using fibre interfaces for maximum performance.

[Tim D Fulford]

OK

I'll tell you what I did... We needed to move 220GB of data from FC60 to VA7400 (bigger disks, more cache), but did not have a VA7400 to play with for the tests to evaluate the migration time...

We did a variety of tests, one of which was

dd if=/dev/vgXX/rlvolY bs=4m of=/dev/null
This took about 1 minute per GB, we chose 4MB as the block size as this is the Extent size of the Physical volume.

dd if=/dev/vgXX/rlvolY bs=4m of=/dev/vg00/rtest
This took longer 2 minutes per GB.

We tried multiple dd's but this only slowed things down (we used a SAME structure on the FC60, therefore the dd'd interfere with each other)

So we ended up doing single dd's in serial (script) and the whole thing took about 1min 25 secs per GB, so the writing phase added about 50%.

From the above I would conclude
o dd was a fairly good way of moving data 'round
o If you use SAME in the volume groups DO NOT use parallel dd's
o Writting to /dev/null would overestimate a good disk-subsystem.

Regards

Tim