Home backup project

[John Collier]

I'm sure I am making more of this than I need to, but I am looking for the most simple and reliable way to make a mirror image of the /home/{user} directory from two different boxes on a single (separate) file server.

In order for this to be considered a success for my personal requirements, I have ruled out a tar command for the simple reason that I want to be in a position to simply change the user's home path to the fileserver at a moment's notice if the need ever comes up so that the potential user interruption is taken down to a minimum.

This means that I will need a complete backup of the directory structure under each user's home directory including any and all files, hidden or not, in use or not, at the time the backup is run and I would like to be able to schedule this out under cron to run at specific times.

Given the potential size of the modern /home/{user} directory under modern Linux distributions with all the graphical components and everything else now common, and the relative slow speed of the file server hardware being used, I also desire the solution to include a provision to compare the files to be copied to the ones already on the file server and skip any that have not changed since the last backup was performed.

All three boxes in question are running Fedora Core 4 with the only major difference between the distros being that the file server is a 32 bit system and the two boxes to be backed up are running 64 bit systems (which should make no big difference). They are all connected via a fast ethernet switch running full duplex connections.

Major question is simply this -- is there a program out there that I can use to accomplish this simply or would it be better done through a custom script?

Second part of the question would be -- if there is a program, what is it and if a script would be better, would someone be kind enough to help with one since my scripting skills are virtually nonexistent?

All input appreciated. Any questions will be answered to the best of my ability.



Bunny will be awarded when solution has been implemented and tested.

[Stuart Browne]

I was recently introduced to this, which sounds interesting:

http://www.drbd.org/

It allows for mirroring of a filesystem over the network to a remote filessytem..

Possibly this will be good for you.

[Ivan Ferreira · Accepted solution]

I'll go by rsync. Rsync is a remote copy service that you can schedule. The rsync command copies only modified files, and can do on the fly compression to transmit the data. It's very easy to configure, and you can use a secure communication channel using SSH to transfer the files.

[David Timms]

rsync also has some cool options like:
* make dest same as source (ie if a file is deleted from source - the backup will also be deleted - good when people move folders around, or delete cd iso's etc, so that the mirror machine doesn't become full.
* make backup - if a file changes, move the old backup into a certain folder, for later reference. sort of cvs'ish
* as mentioned, backups are differential per file, but also within files - even insertion of text into the start of a file that has already been backed up will be analyzed and appropriate minimal diffs transmitted (perhaps compresses) to the backup machine.
* there is a script available (I forget where) that also saves data transfer in the case where a file / folder is renamed on the server. I think it does this by keeping track of file sizes, and then comparing contents.

I use rsync to both internal mirror machines, and offsite mirrors. In both cases rsync is instructed to use ssh for encryption. Mostly, the 3x daily rsync's complete in about 5 minutes, unless great changes to data.

The offsite is far more variable because of the link speed (128kb/sec). We check the rsync logs, and if the sync is taking too long (continues into next working day), we arrange to bring the box in and do the rsync locally (getting it complete), then return it offsite where the nightly syncs take less than an hour.

[Huc_1]

I would also recommend the use of rsync with a set of scripts using cron to schedule.

This is easy to secure using ssh and is very flexible...has many option + the man page examples are very clear.

That is what I use for my own set up

Jean-Pierre Huc

[dirk dierickx]

rsync, what else?!

[Stephen Barratt]

I guess I'm only echoing what's already been said, but I implemented exactly this setup myself last week using rsync. I have a main file server for a small business (suse 10) with a second backup server at a remote site (suse 10), connected through hardware VPN.

The on-site machine is the main server, but I wanted to be able to switch the small number of windows desktop users to the alternative server at short notice if the main server failed, leaving a slower but still fully functional system.

I set up a new "backup" user on each system, and then set up an rsa authenticated ssh link from the file server to the backup. Then I setup keychain to automatically authenticate the link without me having to input the ssh passphrase.

I then made a (very) simple script that runs rsync to backup an entire directory tree from the server to the backup and set up a cron job to periodically run the script.

I used this page for information on rsync options:
http://rsync.samba.org/ftp/rsync/rsync.html

this page for information on keychain:
http://www.gentoo.org/proj/en/keychain/index.xml

and "Linux Cookbook" published by O'Reilly for general instructions.

I am a newcomer to running servers and linux, so sorry if I'm just telling you what you already know, but your situation sounded identical to mine!

If I can be of further help, drop me an email at barratt_sab at msn dot com and I'll try to pass on what I learned.

Stephen

[Guru Dutta]

Hi,

I guess rsync is an open source utility which can be used for the purpose stated above.More information on the same can be found here:-
http://www.samba.org/rsync/

[John Collier]

Stuart,

As always, I can count on you to point out the not-so-obvious. While I have to admit to liking the general idea behind the info you provided, I also have to admit that it seems like a bit of overkill for this little project.

I really do appreciate your pointing me toward it, however. I have a future project I am considering that it may actually help with and I will definitely keep it in mind.



It would seem that rsync has the majority of the votes so far. I will do some research into the subject and the options prior to attempting to implement it. After all, I have only heard of it, not used it.

I notice that there was a suggestion to use this with a set of simple scripts. Curious question: What type of simple scripts for what reason?

Anyway, I will let you know how it goes.

BTW, as far as the bunny goes, I will most likely have to award it to the first suggestion for rsync if this does what I looking for. It is also highly possible that I could award additional bunnies to offerings of more detailed info that will further my cause.

Bottom line. Points have not been, and will not be, forgotten.

[Stuart Browne]

RSync as a simple backup solution is a good one. I know I've used it many times in the past.

The good news about setting it up is this:

If you've got passwordless SSH, then you're 99% done!

There is an 'rsync' service on systems, but as you aren't really serving files out publically (just wanting to pull specific 'protected' structures), you can ignore it's existance.

Using SSH's pathing is all you need to know, so something like this:

rsync -az -e ssh user@remote:/path /local/

If you want to ignore files, you have an exclude list. You also have the ability to follow deletions (by adding '--delete').

Quick explanation as to the arguments in use:

-a = -rlptgoD =
recursive
copy links
preserve permissions
preserve times
preserve groups
preserve owners
copy devices
-z =
compress data stream
--delete =
delete files that don't exist on senders server

Basically a simple remote-mirror :)

As for scripting it, well, you can put a line like the above directly into crontab if you wish. It'll spit some details to cron's mail which you'll receive whenver it runs.