System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

Copying files from one FS to another

 
javedk
Advisor

Copying files from one FS to another

Hi,
I would like to copy files from one system to another. The file system is around 500GB and it has thousands of small files. I have tried the below below options. Could you kindly advice the fastest way to do this.

1) cpio, cp, fbackup from source FS to new FS ; This is very slow because of the huge number of small files.

2) dd if=/dev/vg01/rlvol1 of=/dev/vgtest/rlvol1 bs=1024k; The problem here is my destination LV is 750GB and hence I loose 250GB of space here.

3) The option I had was to create the new LV of 500GB size and copy the files using "dd" and then extend the filesystem. But I wanted to increase the PE size for the new VG and hence if I create a 500GB file system it will be few extends more than the source LV. I tried the "dd" command and fsck-ed the FS and mounted and it is fine. Is this option fine?


Thanks,
Javed.
14 REPLIES
Michael Steele_2
Honored Contributor

Re: Copying files from one FS to another

Hi

a) I use find . -depth -exec tar -cvf /tmp/file.tar {} \;

Then verify the contents with
tar -tvf /tmp/file.tar

Then ftp the tarball over and extract.

cd /dir/dir
tar -xvf file.tar

b)Yes, and in LVM you can only extend the logical volume at this point, no harm on the dest or source systems.

lvextend -L 750m /dev/vg/lvol
fsadm -b 750m /filesystem mount point

c) Two problems.

1) You can only increase the PE at time of vg creatation. So forget this unless you make a new vg.

2) The few extends concern is not a big worry either. At worse the dest file system (* in the new vg and new lvol *) will need to be increased a little. This has to do with for example, a 4 MB extend and a 16 MB extend and the file system blocks. All the blocks may not conveniently fit into the new 16 MB extent.
Support Fatherhood - Stop Family Law
Olivier Masse
Honored Contributor

Re: Copying files from one FS to another

Personally, I don't like using dd to copy files as userland tools will guarantee filesystem consistency. And I'm not fond of cpio either but that's another story :). If you're allowed to unblock remsh for the time of the copy, you can try to pipe tar directly in a remsh (ssh will compress/encrypt and this could possibly end up being slower). Tar should be able to gobble up small files quickly. I think that vxdump/vxrestore could be even faster but it is not as trivial to use.

This is an example:
tar cf - /source_fs | remsh server "tar xvf -"
(might need some tweaking, I can't reproduce it here)

Good luck
Suraj K Sankari
Honored Contributor

Re: Copying files from one FS to another

Hi,

The easiest way is take a backup from 1st system and restore the same into second system.

Suraj

Re: Copying files from one FS to another

>Michael: I use find . -depth -exec tar -cvf /tmp/file.tar {} \;

There are so many things wrong with trying to use find and tar. Instead use pax(1) or cpio(1).

Or if you want to copy everything, just tar at the top.
melvyn burnard
Honored Contributor

Re: Copying files from one FS to another

Well, if it is to another file system on the same server, the simplest is to use pax to copy across.
If on another server, you would be best copying to tape using pax, then recover.I would NEVER use dd, as it is a bit by bit copy, and has no "intelligence" to skip "bad" bits

man pax
My house is the bank's, my money the wife's, But my opinions belong to me, not HP!
Steven Schweda
Honored Contributor

Re: Copying files from one FS to another

A Forum search for keywords like, say,
tar pipeline
(or "pax", or whatever) should find many
examples of methods which don't involve steps
like "Then ftp the tarball [...]".

> tar cf - /source_fs | remsh server "tar xvf -"

I tend to avoid absolute paths (like, say,
"/source_fs") with "tar", but yes, something
like that. One can add "mkdir" and "cd",
too, as needed.

> [...] you would be best copying to tape
> using pax [...]

Because there's no network connection, or
why? This must be some new meaning of
"best".
Michael Steele_2
Honored Contributor

Re: Copying files from one FS to another

Dennis:

Can you explain?

Last I checked cpio couldn't handle files > 2 GB but that was a decade ago, nor does cpio pass through easily restore across file systems, forcing the backup procedure to not cross mount points.

'Pax', ok. Use it with ignite and from the cli once in awhile but aside from is being able to read tar and cpio file formats, what's the advantage?
Support Fatherhood - Stop Family Law

Re: Copying files from one FS to another

>Michael: Last I checked cpio couldn't handle files > 2 GB but that was a decade ago

I wasn't recomending cpio but denigrating tar.
Both pax and cpio work with find. Your example would overwrite /tmp/file.tar with each file.

>nor does cpio pass through easily restore across file systems, forcing the backup procedure to not cross mount points.

I'm not sure I understand this? You mean a cpio pipeline doesn't allow you to copy multiple file systems?

>aside from is being able to read tar and cpio file formats, what's the advantage?

Because it has the best of tar and cpio's interfaces. It can take files from the command line or from stdin. It also allows files > 8 Gb. And allows you to rename paths/files when you restore.

What's broken about pax is that it doesn't have tar's -C directory nor does it restore SUID bits.
Sagar Sirdesai
Trusted Contributor

Re: Copying files from one FS to another

Hi Javed,

rsync utility is the ideal option

Pleaae download this utility and use it.

See man page examples for a better understanding of the usage


Sagar
Sharma Sanjeev
Respected Contributor

Re: Copying files from one FS to another

Hi Javed

@
dd if=/dev/vg01/rlvol1 of=/dev/vgtest/rlvol1 bs=1024k; The problem here is my destination LV is 750GB and hence I loose 250GB of space here.


After dd your destination FS will come to 500 GB, but you can just do extendfs or fsadm.
again it will grow upto 750GB, no need to do lvextend just after dd do extendfs or fsadm

Regards
Sanjeev
Everything is Possible as " IMPOSSIBLE" word itself says I M POSSIBLE
Tiziano Contorno _
Valued Contributor

Re: Copying files from one FS to another

I personally always use fbackup | frecover pipe (it can handle BIG files, permissions, link mamagement, etc.).

Have a look at the man pages, here's my one liner:

/usr/sbin/fbackup -0i /source_path -f - | (cd /target_path && /usr/sbin/frecover -rosvXf -)

Beware that frecover create a directory named as the source in the target (you have to manually mv ..).

Regards.

Michael Steele_2
Honored Contributor

Re: Copying files from one FS to another

geez. are people still adding to this? and still no points? brutal.
Support Fatherhood - Stop Family Law
javedk
Advisor

Re: Copying files from one FS to another

Sorry I was on holiday and could not assign points. Thank you very much for the response.

The advantage i found with dd is that I can save the time required for backup. I can straight away do a dd and then mount the filesystems and check if it's fine.

I have tried this on my test servers which has Oracle database and it worked fine.

Is the bad block issue mentioned above a real concern.
javedk
Advisor

Re: Copying files from one FS to another

the saving time I mentioned is compared to using a backup software like DP.

the biggest advantage I found was that I did not have to worry about links. > 2GB files etc.