topic Re: Time to do a basic cp in Operating System - HP-UX

Time to do a basic cp

James Clay_2 — Mon, 24 May 2004 06:45:31 GMT

Pardon my ignorance but I expeerienced some strange issues last night when copying some files on my system, and was hoping someone could explain what was going on.

Filesystem A had 22GB worth of data contained in 42 files, it took ~45min to copy to a new filesystem.
filesystem B had 7GB worth of data in over 42000 files, it took +2 hours to copy to a new filesystem.

Both the filesystems had the same mount options etc. and did not differ from what was there originally. I did one cp after the other, and did not experience major disk contention etc.

Any ideas as to why this would happen?

Cheers

Re: Time to do a basic cp

Robert-Jan Goossens — Mon, 24 May 2004 06:59:49 GMT

Hi,

Large files do copy faster then a lot of small files. Large files can use the lan interface more efficient, if you use the cp command it copies file after file. If you have a lot of small files, this will take a long time.

Create a tar archive before you copy the files, this will save a lot of time.

Hope this helps,
Robert-Jan

Re: Time to do a basic cp

Keith Bevan_1 — Mon, 24 May 2004 07:00:56 GMT

James,

You may want to provide a little more information for the forum members, so they can evaluate more precisely !

Are the file systems A & B on the same physical server ?

Are the filesystems on the same logical volume, volume group, physical disks, or are your running nfs mounts for example ?

What was the system usage at the time of the cp (ie disk i/o, memory, process priority) ?

The more precise and detailed information you can provide the more likely you are to receive beneficial advice.

Keith

Re: Time to do a basic cp

Bruno Ganino — Mon, 24 May 2004 07:35:02 GMT

This is normal !
The system must take present attribute of many files and to copy it one to one.
HTH
Bruno

Re: Time to do a basic cp

Kent Ostby — Mon, 24 May 2004 07:35:22 GMT

Also, for each file, the system has to go through the overhead work to find a new inode to link it into and to set aside the space where its going to write the data.

With the large files, it only has to do these I/Os 42 times and then it can get on with the business of writing out the data.

Best regards,

Kent M. Ostby

Re: Time to do a basic cp

doug mielke — Mon, 24 May 2004 07:50:51 GMT

also with large directories the creation of the file list alone can be time consuming.

a dd copy would ignore most of this structure, and even with many files, would have copied the 7 gig faster.

Re: Time to do a basic cp

A. Clay Stephenson — Mon, 24 May 2004 08:38:24 GMT

The biggest overhead in copying the larger number of small files is the fact that directory has to be rewritten each time a file is added. The system also has to use locking mechaisms to ensure that only one process is actually writing to the directory at any moment. This overhead completely swamps the fact that the total data volume was ~ 3X larger in the run with a small number of individual files.

Re: Time to do a basic cp

James Clay_2 — Mon, 24 May 2004 09:16:35 GMT

Thanks for that.

For the record I was copying data within the system onto new LV's in different VG's. I think next time I'll tar the whole lot up :-)

Re: Time to do a basic cp

Mark Grant — Mon, 24 May 2004 09:22:12 GMT

Actually, I'm not sure tar is going to help you too much when moving all the stuff to new LV's. Even with one big tar archive, you'll still need to extract that tar archive on the other side and therefore still need to muck about with all those directories that A.Clay mentions.