Will dd copy filesystems faster than cpio if you have large FS w/ 350K files?

[John Knapp]

We have a situation where we have to migrate a filesystem from one disk array to another. This filesystem has 31+GBs of capacity @ 95% utiiization but with over 350,000 files. Test runs of a find -xdev -depth -print | cpio pdumv process took over 3.75 hours to complete.

One of our developers suggested we do a block copy ... I tried using the example on a small FS with few files:

dd if=/dev/vg03/test1 of=/dev/vg04/test2 bs=36b

The dd was slower than the logical cpio copy but has anyone done benchmark test in which at a certain point the dd command is more efficient/expedient that a logical cpio operation?

PS: we have to use naitive unix command ... tape backup/restore is nice but I think disk-to-disk will still be faster.

[Helen French]

Hi John,

This is an old document, but can give you some idea about what you are looking for:

http://us-support2.external.hp.com/cki/bin/doc.pl/sid=c11687120d3bcb7e92/screen=ckiDisplayDocument?docId=200000007949999

HTH,
Shiju

[Wim Rombauts]

One thing to consider :

cpio copies files from one filesystem to the other. The disks used to create these filesystems are not important for cpio.

dd copies harddisk blocks from one disk (or volume) to the other. You need identical or compatible disks to create a filesystem on the target that still works. If you are using Logical Volume manager, dd won't work.

So dd is only an option if you copy a simple filesystem (non-LVM) from one disk to another disk with identical size.

If you use logical volumes ? Maybe you could use pvmove ? it could take some time, but your filesystem stays online.
If I'm right : add the new disk box to your volumegroup, use pvmove to migrate the LV's on the old disk box to the new disk box.

[MANOJ SRIVASTAVA]

Hi John


Just a thought , while cpio reads data , dd reads blocks even if there is no data writtin on it , if we have a full file system then i beleive it will faster to do a dd on that FS than cpio .


Manoj Srivastava

[John Knapp]

The target disk array has physical volumes larger than the source so under 10.20, pvmove sound great but I don't think I can use it with the limitation (vg).

I was wondering if anyone in a lab coat has determined at what size logical volume with a certain number of files, dd out performs cpio ???

All you have given good suggestions/information ... thank you

[John Carr_2]

Hi

I have always been led to believe dd does a bit by bit copy of the device so the capacity of the disk does not come into question. it is hard to do a comparason without creating a mirror of your environment because so many factors are involved such as amount of RAM disk access speed...

dd will be faster the larger you increase block size but if you use cpio on a filesystem with say 1MB of data which is 100MB in size of course cpio will be faster its only backing up the data not the filesystem.

good luck
John.

[Bill Hassell · Accepted solution]

dd ... bs=36b

Wow, that will REALLY be very slow. Change the bs value to bs=64k or even bs=128k and try the measurement again. You want to give dd the largest blocksize possible. bs greater than 128k has a diminshing effect since the I/O's may be split at 128k.