fastest way to copy tons of data ?

[Edward Sedgemore]

Ive got to copy hundreds of GB of lvm stuff from one set of disks to some others, whats the fastest way to do it ??
cpio, tar, pax, gtar, cp, mv ??

[Stefan Farrelly · Accepted solution]

dd is the fastest way as it bypasses the unix driver and goes straight to the disks (if you use raw lvols to copy). But, if your logical volumes are on average < 50% used it wont be quicker to use dd, use cpio instead.

eg, to use dd do;

dd if=/dev/vgXX/rlvolY of=/dev/vgNEW/rlvolZ bs=1024k

You should even run many at once, spawn them off in the background with & this will make your copy even faster.

[T. M. Louah]

Please refer to Knowledge base document "Comparison of Backup Methods and their supported features"
Document Id : CJW9304231450 Old but good!

.: dd is only an option if you copy a simple filesystem with no LVM, from one disk to another disk with identical size as it copies HDD blocks. In LVM dd is slower.

.: Make sure that you are not copying files that are larger than 2GB as only dd (and fbackup) over-throw this constrain.

.: tar command uses pax, & it is widely accepted to perform data migration try this:
# cd dir_from; tar cf - . |(cd dir_to;tar xf -i)
tar cf - means send data to stdout, & it tells tar xf to look to stdin for data.

G'd luck
T??

[Corthouts Carlo]

Best way to about it is with dd.

[Edward Sedgemore]

Whats that bit in the Knowledgebase document that says; in LVM dd is slower ??

[Stefan Farrelly]

nothing to worry about. It means if you use dd instead of cp to copy a logical volume without using the raw lvol. ie;
dd if=/dev/vgXX/lvolY ...

To get the performance benefit of dd use the raw lvol (/dev/vgXX/rlvolY) - but only if the source and destination lvol sizes are identical.

I dont like articles like this one where they state something which is unclear and can be misleading. It should be comprehensive and categorically state that dd'ing of raw lvols is the fastest method to copy (as long as you dont have <50% of it with date on, in which case cpio or tar may be faster).

[James R. Ferguson]

Hi Edward:

I presume that you want to "move" the data from one set of disk to another.

If this is the case, and you have MirrorDisk/UX (and I hope you do), and the disk you want to copy is LVM-based, then you can avoid downtime while copying your data from the old to the new disk.

Start by 'lvcreate'ing each logical volume to be transfered on the new disk. Do the creation with a zero-size to be able to choose the physical volume on which you want to allocate the logical extents with 'lvextend'.

Now use 'lvmerge' to merge the two volumes into one. This process resynchronizes data previously contained in the destination path with data in the source path, destroying the data in the destination path.

Finally, use 'lvreduce' to remove the original logical volume from the original disk.

I have used this process serveral times to migrate data from one EMC SAN to another with a minimum of downtime.

Regards!

...JRF...

[Damian O'Hara_1]

Edward,

We solved this another way - using mirrordir!

We also wanted to move large amounts of data from an old EMC Symm to a new one with the minimum of downtime.

We couldn't use LVM to mirror the old volumes to the new ones because the old ones were striped - HP's Support Centre advised that this isn't possible.

We decided to compile a Linux tool on HP-UX and use it to mirror one filesystem to another. The first time it runs it takes a several hours but the second time it only transfers the differences - a matter of minutes!

We've since used this on Solaris 2.6 and SuSE Linux and it was the key tool to complete our migration.

Have a look at http://mirrordir.sourceforge.net/mirrordir.php

Damian