How can I speed up the cp command?

[Radnour Acton-Page_1]

We have just ported across our database system from NT to UNIX. At the end of the month we copy the database files in order to archive them. In NT this copy took 40(ish) mins but in UNIX took 1 hour 30 mins for the same amount of files/data. I was using cp -r to do the copy. Is there any way of speeding up the process ?

[Steven Sim Kok Leong]

Hi,

Below are some factors you might want to be concerned with:

1) What is your filesystem block size? The larger the filesystem block size, the faster it takes for cp to complete.

# fstyp /dev/vgXX/lvolYY

2) What is your filesystem type? Is it HFS or JFS? JFS is usually faster than HFS.

# fstyp /dev/vgXX/lvolYY

3) Are there other harddisks on the same controller as the harddisk containing this filesystem? Are the currently writes on this filesystem? How busy is the disk containing the source and destination filesystems before you perform the copy? Worse, you are reading and writing to the same disk (which has only one spindle).

# sar -d 1 100

4) Are you mirroring the source and/or destination filesystems? Are these mirrors on the same controller? How busy are the harddisks containing these filesystems?

# lvdisplay -v /dev/vgXX/lvolYY
# sar -d 1 100

Hope this helps. Regards.

Steven Sim Kok Leong

[Olivier LEGRAND]

Hi,

If you just want to archive your databases, have a look on snapshots file systems (very fast)

http://docs.hp.com/
and search snapshot


Regards

[Olivier LEGRAND]

Hi,

Is a maximum of 30 secondes too fast for you ????

Regards

[Justo Exposito]

Hi,

If the files are in the same volume use mv and not cp. This is instantaneus.

Regards,

Justo.

[Radnour Acton-Page_1]

would splitting the copy from one command to several running simultaneously speed up the process ?

[Darrell Allen]

The fastest method I know to copy a directory from one disk to another is:

$ cd /source_dir
$ find . | cpio -pudlmv /destination_dir
omit ???v??? (verbose) option from cpio for more speed

Darrell

[harry d brown jr]

Why not install MirrorUX and be done with it? It would always be in SYNC and would only take a second to split the mirror and bang you have a copy of your data. You would create a filesystem (Logical Volume) large enough to hold your "database files" and such. Then you create a mirrored copy of it, and when month end rolls around you split the mirror, and mount the "mirrored" copy (which you should backup to tape!).

Then you can be happy again!

live free or die
harry

[Olivier LEGRAND]

Hi Harry,

And if you have a crash when you break the mirror ???


A snapshot is more secure


Regards

[harry d brown jr]

Olivier,

Depending upon the timing, this could be true, but the data is on two different disk drives, because most people don't mirror LV's on the same physical disk.

In your scenario, if a piano fell out of the sky and crushed the server and the disk drives losing all of the data, then "snapshot" wouldn't mean much, right ;-))))


live free or die
harry

[Jeff Schussele]

Guys,

If you're concerned about the vulnerability window when you split out a mirror to archive - then do what we did back in my admin days at Cat with our OpenMail msg store - USE A TRIPLE MIRROR.

Split it out to archive & you still have 2 copies on-line. The possibility of both drives failing in the same time period is almost up there with Harry's piano theory ;)

This is ideal for any type of dynamic, large datastore. Furthermore you can mount it out to another server if perf degradation on the app server is a concern during backup.

ALSO...you get the added benefit of being able to use that 3rd for other scenarios....like when some bonehead deletes something he/she "absolutely needs back now!" You can incremental restore tape to it, run up the app on another server pointing to it & get it back fairly easily w/o having to climb too many mountains.

Just my thoughts,
Jeff