Needed: Large and Fast system backup

[kenny chia]

Hi
I need to backup a linux system daily. There are many files and directories (about 2G total) and creating a tar.gz file from them everyday will take up too much CPU resources.

I am thinking of this
1) Create a tar.gz from all the files to be backup

Run a daily program that
1) Search for new or modified files. These files will overwrite or be appended to the tar.gz file
2) Search for deleted files. These files will be deleted from the tar.gz file.

Any ideas on how to implement 1) and 2) ?

Thanks

[Stuart Browne]

If creating a tar/gz file is too much CPU over-head, then you've got a seriously under-powered server there. Given that I can do this on my old PPro's without creating much of a serious dent in usage ability, I'd like to know what you're doing to this poor machine!

Tar it's self cannot do file removals based on removed files, so you'd have to create your own routines of which say "is this file still there?" which would be even worse CPU overhead.

For that you it's probably better you use something like 'rdist' or 'rsync' (which are traditionally used for remote-system backup of a given file-set).

As you seem not to be concerned with creating 'backups' on the same system, one of these would be ideal. You could then tar/compress tar of the backup structure for a decent snapshot.

[Balaji N]

this could be done by maintaining a list of files with their md5 sums. every time generate a list of all files with their md5 sums and if there is a change between these two, then the file is modified and take a backup.

i guess this is a crude way of taking backup and would suggest using rsync.
check out the link for more info.
http://samba.org/rsync/index.html
-balaji

[kenny chia]

rsync looks like a good idea but it doesn't seem to work on tar.gz files. Anyway if I can get a large hard drive. I might implement it without tar.gz

[Balaji N]

yes. that would be the best way to go. rsync directly the files instead of the .tar.gz approach on a new drive.
-balaji