Home backup project

[John Collier]

I'm sure I am making more of this than I need to, but I am looking for the most simple and reliable way to make a mirror image of the /home/{user} directory from two different boxes on a single (separate) file server.

In order for this to be considered a success for my personal requirements, I have ruled out a tar command for the simple reason that I want to be in a position to simply change the user's home path to the fileserver at a moment's notice if the need ever comes up so that the potential user interruption is taken down to a minimum.

This means that I will need a complete backup of the directory structure under each user's home directory including any and all files, hidden or not, in use or not, at the time the backup is run and I would like to be able to schedule this out under cron to run at specific times.

Given the potential size of the modern /home/{user} directory under modern Linux distributions with all the graphical components and everything else now common, and the relative slow speed of the file server hardware being used, I also desire the solution to include a provision to compare the files to be copied to the ones already on the file server and skip any that have not changed since the last backup was performed.

All three boxes in question are running Fedora Core 4 with the only major difference between the distros being that the file server is a 32 bit system and the two boxes to be backed up are running 64 bit systems (which should make no big difference). They are all connected via a fast ethernet switch running full duplex connections.

Major question is simply this -- is there a program out there that I can use to accomplish this simply or would it be better done through a custom script?

Second part of the question would be -- if there is a program, what is it and if a script would be better, would someone be kind enough to help with one since my scripting skills are virtually nonexistent?

All input appreciated. Any questions will be answered to the best of my ability.



Bunny will be awarded when solution has been implemented and tested.

[Stuart Browne]

I was recently introduced to this, which sounds interesting:

http://www.drbd.org/

It allows for mirroring of a filesystem over the network to a remote filessytem..

Possibly this will be good for you.

[Ivan Ferreira · Accepted solution]

I'll go by rsync. Rsync is a remote copy service that you can schedule. The rsync command copies only modified files, and can do on the fly compression to transmit the data. It's very easy to configure, and you can use a secure communication channel using SSH to transfer the files.

[David Timms]

rsync also has some cool options like:
* make dest same as source (ie if a file is deleted from source - the backup will also be deleted - good when people move folders around, or delete cd iso's etc, so that the mirror machine doesn't become full.
* make backup - if a file changes, move the old backup into a certain folder, for later reference. sort of cvs'ish
* as mentioned, backups are differential per file, but also within files - even insertion of text into the start of a file that has already been backed up will be analyzed and appropriate minimal diffs transmitted (perhaps compresses) to the backup machine.
* there is a script available (I forget where) that also saves data transfer in the case where a file / folder is renamed on the server. I think it does this by keeping track of file sizes, and then comparing contents.

I use rsync to both internal mirror machines, and offsite mirrors. In both cases rsync is instructed to use ssh for encryption. Mostly, the 3x daily rsync's complete in about 5 minutes, unless great changes to data.

The offsite is far more variable because of the link speed (128kb/sec). We check the rsync logs, and if the sync is taking too long (continues into next working day), we arrange to bring the box in and do the rsync locally (getting it complete), then return it offsite where the nightly syncs take less than an hour.

[Huc_1]

I would also recommend the use of rsync with a set of scripts using cron to schedule.

This is easy to secure using ssh and is very flexible...has many option + the man page examples are very clear.

That is what I use for my own set up

Jean-Pierre Huc

[dirk dierickx]

rsync, what else?!

[Stephen Barratt]

I guess I'm only echoing what's already been said, but I implemented exactly this setup myself last week using rsync. I have a main file server for a small business (suse 10) with a second backup server at a remote site (suse 10), connected through hardware VPN.

The on-site machine is the main server, but I wanted to be able to switch the small number of windows desktop users to the alternative server at short notice if the main server failed, leaving a slower but still fully functional system.

I set up a new "backup" user on each system, and then set up an rsa authenticated ssh link from the file server to the backup. Then I setup keychain to automatically authenticate the link without me having to input the ssh passphrase.

I then made a (very) simple script that runs rsync to backup an entire directory tree from the server to the backup and set up a cron job to periodically run the script.

I used this page for information on rsync options:
http://rsync.samba.org/ftp/rsync/rsync.html

this page for information on keychain:
http://www.gentoo.org/proj/en/keychain/index.xml

and "Linux Cookbook" published by O'Reilly for general instructions.

I am a newcomer to running servers and linux, so sorry if I'm just telling you what you already know, but your situation sounded identical to mine!

If I can be of further help, drop me an email at barratt_sab at msn dot com and I'll try to pass on what I learned.

Stephen

[Guru Dutta]

Hi,

I guess rsync is an open source utility which can be used for the purpose stated above.More information on the same can be found here:-
http://www.samba.org/rsync/

[John Collier]

Stuart,

As always, I can count on you to point out the not-so-obvious. While I have to admit to liking the general idea behind the info you provided, I also have to admit that it seems like a bit of overkill for this little project.

I really do appreciate your pointing me toward it, however. I have a future project I am considering that it may actually help with and I will definitely keep it in mind.



It would seem that rsync has the majority of the votes so far. I will do some research into the subject and the options prior to attempting to implement it. After all, I have only heard of it, not used it.

I notice that there was a suggestion to use this with a set of simple scripts. Curious question: What type of simple scripts for what reason?

Anyway, I will let you know how it goes.

BTW, as far as the bunny goes, I will most likely have to award it to the first suggestion for rsync if this does what I looking for. It is also highly possible that I could award additional bunnies to offerings of more detailed info that will further my cause.

Bottom line. Points have not been, and will not be, forgotten.

[Stuart Browne]

RSync as a simple backup solution is a good one. I know I've used it many times in the past.

The good news about setting it up is this:

If you've got passwordless SSH, then you're 99% done!

There is an 'rsync' service on systems, but as you aren't really serving files out publically (just wanting to pull specific 'protected' structures), you can ignore it's existance.

Using SSH's pathing is all you need to know, so something like this:

rsync -az -e ssh user@remote:/path /local/

If you want to ignore files, you have an exclude list. You also have the ability to follow deletions (by adding '--delete').

Quick explanation as to the arguments in use:

-a = -rlptgoD =
recursive
copy links
preserve permissions
preserve times
preserve groups
preserve owners
copy devices
-z =
compress data stream
--delete =
delete files that don't exist on senders server

Basically a simple remote-mirror :)

As for scripting it, well, you can put a line like the above directly into crontab if you wish. It'll spit some details to cron's mail which you'll receive whenver it runs.

[John Collier]

OK, as promised I have done some reading on this.

Now I have come up with a possible command to try and I would like to bounce it off of all of you and see what sticks and what stinks.

According to the info I read (36 or so printed pages from the I-net link provided above) I think I want to run a command line of

rsync -caKWi -delete-during /home/{user} /path/to/nfs_mounted/backup/directory/

Now, each host has nfs mounts for the fileserver. Based on this fact, I think I could even get rid of the -W option if I read the defaults correctly.

Question: If I am pointing the rsync command at two 'local' directories even if the destination is truly an nfs mount, I really won't have to worry about the ssh or any other remote connection, right?

Does anybody see a flaw in this command or line of thinking that they would like to share?

[Stuart Browne]

It looks good to me.

And yes. Because you're using NFS, you don't have to worry about the '-e ssh' or anything like that. It's all treated 'local'.

[John Collier]

Well, I see bunnies hopping through here shortly.

The rsync worked just as you all said it would.

All I have to do now it put it into a cron for her user and we are off to the races.

One more question before I close this thread off (if I may be so bold):

When I ran the command that I last listed here as my own home cooked version, everything passed through just as I would have expected with one glaring exception.

The last line that was shown from the command output was:

rsync error: some files could not be transferred (code 23) at main.c(789)

Now I am sure that I have simply overlooked something, but I am not seeing where this output was listed in the info that was on the site linked above or anywhere else.

Can somebody give me a clue as to what most likely went wrong and what (if anything) can or should be done to resolve it in the future?

[Steven E. Protter]

SEP

I like Stuart's first exapmle.

I'm however trying to get it done with rsync and found a useful link.

http://oldmill.uchicago.edu/~wilder/Security/rsync/

SEP

[Yogeeraj_1]

hi john,

Also have a look at the following:

http://rsnapshot.org/faq.html


hope this helps too!

kind regards
yogeeraj

[John Collier]

Steven,

Thanks for the link. Gives one ideas on how to script it out if they wish. Good info on firewalls and some other quirks as well.

Unfortunately, no information found there pertaining to the errors I see at the end of each run.



Yogeeraj,

Interesting thought. If it would have come in first, perhaps I would have gone that way. As it is, I think I will be happy with the rsync solution for my needs. As always, I still appreciate your input.


Still searching for the reason it would tell me that some files could not be copied. If anybody has a hint on where I overlooked that at, I would still be grateful.

[John Collier]

Ivan wins the race for the biggest bunny since he was the very first to suggest rsync. A few others were handed out as well since there were some really good additional entries with further info and links to great resources.

As far as my last questions regarding why there were constant error messages about files that could not be copied, I found I was the one shooting myself in the foot with that one. I was running the command line under a regular user logon, not root. Once it was run using the root account, all was good and happy in rsync land.

Footnote for anybody else considering this as an option for his or her own network(s):

I included an option in my command line that adds an extra level of comparison between the files being copied and the existing files (refer to the info pages linked above for more intimate details on the options). While this is a good thing if you want to be as thorough as possible, it does GREATLY lengthen the time it takes to actually perform the operation.

I would strongly suggest that you read the options available for rsync and perhaps do several test runs to see which ones work best for your particular environment. It is very likely that my choice of options will not be what you would want.

The other options that were provided also look promising and I am sure that they would work wonderfully in the right environment. I do not want to discount them at all. Personally, I am glad that this is my thread. For this reason, I know that I will never loose the information on these other possible resources. In another environment, they may very well be a better choice.

Many thanks to you all for all the wonderful input and the patience you have shown.

I still say this is the absolute BEST user forum on the face of this planet!