topic Re: Needed: Large and Fast system backup in Operating System - Linux

Needed: Large and Fast system backup

kenny chia — Mon, 17 Feb 2003 03:20:01 GMT

Hi
I need to backup a linux system daily. There are many files and directories (about 2G total) and creating a tar.gz file from them everyday will take up too much CPU resources.

I am thinking of this
1) Create a tar.gz from all the files to be backup

Run a daily program that
1) Search for new or modified files. These files will overwrite or be appended to the tar.gz file
2) Search for deleted files. These files will be deleted from the tar.gz file.

Any ideas on how to implement 1) and 2) ?

Thanks

Re: Needed: Large and Fast system backup

Stuart Browne — Mon, 17 Feb 2003 04:45:48 GMT

If creating a tar/gz file is too much CPU over-head, then you've got a seriously under-powered server there. Given that I can do this on my old PPro's without creating much of a serious dent in usage ability, I'd like to know what you're doing to this poor machine!

Tar it's self cannot do file removals based on removed files, so you'd have to create your own routines of which say "is this file still there?" which would be even worse CPU overhead.

For that you it's probably better you use something like 'rdist' or 'rsync' (which are traditionally used for remote-system backup of a given file-set).

As you seem not to be concerned with creating 'backups' on the same system, one of these would be ideal. You could then tar/compress tar of the backup structure for a decent snapshot.

Re: Needed: Large and Fast system backup

Balaji N — Mon, 17 Feb 2003 06:38:27 GMT

this could be done by maintaining a list of files with their md5 sums. every time generate a list of all files with their md5 sums and if there is a change between these two, then the file is modified and take a backup.

i guess this is a crude way of taking backup and would suggest using rsync.
check out the link for more info.
http://samba.org/rsync/index.html
-balaji

Re: Needed: Large and Fast system backup

kenny chia — Mon, 17 Feb 2003 07:30:01 GMT

rsync looks like a good idea but it doesn't seem to work on tar.gz files. Anyway if I can get a large hard drive. I might implement it without tar.gz

Re: Needed: Large and Fast system backup

Balaji N — Mon, 17 Feb 2003 08:16:24 GMT

yes. that would be the best way to go. rsync directly the files instead of the .tar.gz approach on a new drive.
-balaji