Copy takes too much time.

[Rajesh SB]

Hi Experts,

I have copied a directory recursively, which is size of 198Mb and directory was containing near 198000 files. It took near 8 Hours to copy.

The command used,

# cp -R /siemens .
Both the source and destination were on same file system.

OS : HPUX 11.00

What would be the copy speed? Why it took this much time. Any pointer.

Thanks in advance,

Regards,
Rajesh

[Pete Randall]

Rajesh,

For this type of situation I always use cpio:

find /siemens -print |cpio -pdumxl .


Pete

[Rajesh SB]

Hi Pete,

You mean to say, what ever time taken by cp is normal?

Regards,
Rajesh

[curt larson_1]

What would be the copy speed

that is hard to say without knowing the kind of hardware and software that your using.

Why it took this much time

1) a flat directory, 198000 files all in the same directory is going to slow things down.

2) small files, your average file size seems to be about 1k. lots of small files are slower then a few large ones.

3) possible hitting the same disk drive (source and destination were on same file system), if you are your causing the disk heads to more back and forth on reads and writes.

[doug mielke]

could it have been a typo resulting in something like this

cp -r /admin /admin/data

could loop, maybe never finish.

[Todd McDaniel_1]

I would prefer to use tar command like this.

cd /orig/dir1 ; tar cf - . | ( cd /orig/dir2 ; tar xf - )

Simple cp is not really meant for that functionality imho. cpio and tar are meant to handle large numbers of files.

By your statement you had 198MB and 190k files... Is each file 1mb in size? If you are using -r recursive, make sure the destination is not a sub-directory of the dir you are cp'ing from, or you will have a looping command.

[Todd McDaniel_1]

Oops bad math... 190MB / 190k files is 1k files, at least I think it is... /ponders.

No points here...

[Bill Hassell]

The cpio solution is generally the fastest but always use puddle-move for the options:


cd /siemens
find . | cpio -pudlmv /newdirectory

Be sure to cd to the old location and use find . rather than find /siemens to put the files in the right location. Many users leave out -v which shows the progress, and others put in -x which should never be used (device files should never be copied except in very unusual situations). This syntax will maintain ownership and permissions and proceed much faster than cp. NOTE: It will fail to copy files larger than 2 Gbytes (largefiles).

[David Child_1]

Is this a vxfs file system? One other reason for the slowness of the 'cp' (and may also slow down cpio, ls, etc.) is the 'size' of the directory. I ran into a similar situation where an application would create and delete all these small files. The number of files at any one time was maybe 2000, but the directory still contained all the entries from the deleted files and was very large (as shown by ls -ld ). Even 'ls' ran relatively slow until I 'defraged' the directory:

fsadm -F vxfs -D will report on the current state (note: you have to run it against the root of the file system and all subdirectories will get processed.

fsadm -F vxfs -d will reduce all the directories under the listed mount point. On our system it greatly improved performance of any command that read/write into the directory.

Hope that made some sense. Its still early and I haven't had any caffine yet.
David