fastest way to copy files ??

[Stuart Abramson_2]

I'm copying files from one Oracle SID to another on the same system. Some 70 GB on EMC disks.

What is the fastest way to do this?

I normally do it one of two ways:

1. cd /dir
find . -depth -xdev -print | cpio -pdum /newdir

where: /newdir already exists!

2. cd /fromdir
tar cf - . | (cd /todir; tar xf - )

However, in this case I don't have any links or special files. I could use "cp".

So far I have copying using Method 1 above (find/cpio) and I'm getting about 1 GB/15 mins or 4 GB/hr, which will take me 17 hours. Too long.

Could I split it up into multiple streams and they won't interfere which each other?

[Chris Vail]

Yeah, you can split it up into multiple streams, and run 'em all in the background.
However, you're getting REALLY SLOW speed, and were I you, I'd look into why that is. On my EMC systems, I can copy 700GB in 4 hours using tar.

Chris

[Robert-Jan Goossens]

Hi,

One small suggestion is remove the -p option from cpio.

# find . -depth -xdev -print | cpio -cmudv /newdir

Hope it helps,

Robert-Jan.

[Shannon Petry]

cpio would be the fastest method of copy, as you have given the example for. Note that large files may/may not copy. Last I checked this was not supported by HP, same for dd. (I have never had problems, but that does not mean you wont).

Looking at your speed though, there is definately something wrong. I can copy GB/minute on a C3750 with ftp and rcp, so something is broken or... are these files in use?

If the files are not in use, then look hard at your network and make sure your settings are all as expected. lanadmin -x will give settings of the devices.

The only time I see performance this bad, is on 100baseT full duplex networks where the HP keeps switching or is always running at 1/2 duplex.

If it's duplexing, then
lanadmin -S 100FD
will change the settings to full duplex. Make this permanent by editing /etc/rc.config.d/hpbase100conf

If the lan interface is okay, maybe a bad port on the switch.

Regards,
Shannon

[Mark Greene_1]

Go with cp:

FILELIST=`find . -depth -xdev -print`

for FILENAME in `echo $FILELIST`; do

cp $FILENAME /newdir/$FILENAME

done

If you do not have multiple HBA's on your system, then no matter what method you use in Unix, you are dragging the data over the fibre and through the single HBA twice, which is what well may be constricting your bandwidth. Splitting the copy process into streams is going to compound the issue.

To be thorough, you can check the fibre controller for errors. Look in the syslog to. You may have an iffy cable or HBA.

HTH
mark

[Stuart Abramson_2]

1. Why will cpi "-p" slow down my copy?

2. tar is SLOWER than cpio. I'm getting 6 GB/hr with both tar and cp.

3. I'm on the SAME system. there is no LAN involved.

4. 700 GB/4 hrs = 175 GB/hr. That's pretty fast.

5. It takes 2 hours to reload this SID from tape. 70 GB/2 hours = 35 GB/hr tape to disk.

[Stuart Abramson_2]

I have multiple HBAs and EMC PowerPath.

[Stuart Abramson_2]

Well, I have calmed down a little, and here is where I stand:

From 3:30 - 4:00 PM:
cpio: 4 GB/hr
cp: 8 GB/hr
tar: 6 GB/hr

At 4:00 PM:
two copies at same time:
cp: 60 GB/hr

So, I suspect that something was going on on my system, and I didn't know it, and it must have just stopped. So all my previous timings were completely "invalid", and I will proceed and just hope to finish in time.

Thanks for all your help.

[Yogeeraj_1]

hi,

i had a similar question a few months ago and got some great replies from the gurus here.

see

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xb20b5bd3782dd711abdc0090277a778c,00.html

hope this helps!

best regards
Yogeeraj

[Robert-Jan Goossens]

Hi Stuard,

I made a big mistake, sorry sorry.

The command you used for cpio is the fastest. I mixed the verbose and pass-through options.

Hope you accept my apologies,

Kind regards,

Robert-Jan.
0 point please for both answers.

[Massimo Bianchi]

Hi,
some notes:

- cp has problems with files bigger than 2G, better use dd

- if you use dd there are tweo differest ways:

SLOW:
write a script that for each file does a

dd if=/oracle/SID/infile1 of=/oracleNEW/SID/infile1 bs=1024k

and start them in parallel, at max 2* CPU#.


VERY FAST:
create a set of logical volumes in the destination exaclty of the same size of the sources you want to copy.

the name is of no matter.

and start them in parallel, at max 2 per CPU#

umount all the logical volumes you want to copy (and stop oracle :) )

for each lvol

dd if=/dev/yourvg/RlvolSOURCE of=/dev/yournewvg/RlvolDEST bs=1024k



HTH,
Massimo

things to pay attention to:
- use rlvol for better performances
- umount all lv
- exaclty same sizes.


Using this trick i copie a 350Gb DB in 150 minutes

[TwoProc]

I've also seen really slow copying when your file system is fragmented. Online JFS can reorg it for you, or, even easier, if you have nothing else on the destination file system, just do a "newfs" on it.

[Bruno Vidal]

Hi,
Few month ago, I had to move 6Tb from XP256 to XP512, and I had less than 12h to do that. The fastest way to copy has been to use dd, because it is possible to give the block size, and it is the same with EMC, this kind of system are really efficient with a block size of 256kor 512k
try to timex this:
dd if=/tmp/my_file of=/my_dest_dir bs=512k
It is the quickest way to do a copy. The only difficulty has been to write a script to do the copy by using dd recursivly and change the owner to matche the original one. But try it, it is really quicker than any other copy command.

cheers.

[John Poff]

Hi,

Just to clarify a point made by Massimo, I have used 'cp' to copy files larger than 2 Gb on a system and it works fine. It will handle files larger than 2 Gb.

JP

[Chris Vail]

Our DBA's set the size of their .dbf's to 33 GB. The cp, mv and most other HPux commands choke on this size. Even tar and zip will choke.
We had to compile GNU tar for these systems. It is a lot faster than the HP version, and is a true 64 bit application, so it can handle the larger file sizes.
Absolutely the fastest way to copy files is with dd on the raw filesystems. This is still a 32bit facility on HPux 10.2, 11.0, and 11.11. Use it with caution, and verify every byte. However, even the sum command trips up on 64 bit apps.
If you have EMC, PowerPath etc....see if you can make a BCV copy. T'is fastest of all, and uses no clock cycles. Just copy your standard to BCV, split it, then reverse the process against the new standard device.


Chris

[Rodney Hills]

If you are copying a whole file system and have a limited time window for the copy, you could use disk mirroring.

Create the file system on the destination disks, define the mirror and HPUX will replicate the data onto the other disks. Best part is you have no down time! Even updates to the original files will be updated to the mirror.

Another benefit of mirrored disks is that you can split the mirror off and have a copy of your data set that is not changing and you can get a clean backup. The live system can continue updating the one copy. Once the backup is complete, the mirror can be remerged into the system and it will be automatically updated with the changes from the original.

This only works on entire file systems though, not individual directories or files.

HTH

-- Rod Hills