Verifying Backups

[Simon Abbott]

Hello,

We use fbackup to backup our system every night. My boss has asked me to plan a shedule for verifying the backups.

I have no reference point of other systems to compare ours to and neither has my boss so I'd like to draw upon your experience if I may. As you could say that running frecover with the -N option and a 0 return value was good or you could go all the way to recovering a whole database and starting it up and running a query.

Could anyone suggest a good 'middle way' for us to work towards that will garuntee a successful backup most often. Are there any best practices for this?

Also, my boss has suggested that we verify the backup on another tape drive to prove that it can be read on a drive other than the one it was written on. Is this necisary?

Cheers,

Simon.

[Robert Gamble · Accepted solution]

Simon,

Your boss is correct. You should regulary verify that your backups work properly. CYA!!

Since you use fbackup, it should be pretty easy to do. Using frecovery, you could conduct an audit of the last file on the tape(s). (fbackup does things alphabetically on a level 0 backup) That would force the whole tape(s) to be read and verify that a file can be restored.

Good Luck!

[Patrick Wallek]

There are lots of ways you can do something like this.

The way I would probably do it is to run frecover and just recover the file listing from the tape. This will read the entire tape and write out the list of files to a temp file, but won't really restore anything. (The -N option is probably the best bet here).

You could also pick out selected files to restore. If you want to be thorough, pick different files from all locations of the tape (beginning, middle, end).

If you have a test system you could try restoring the whole tape.

It is probably a good idea to try a tape drive on a machine other than the machine that created the tape. Just make sure the tape drive you are using is compatible. If you make your backups on a DDS3 drive, then only DDS3 or DDS4 drives will be able to read the tape.

Just keep in mind that while you can run all the verifications on the backup you want, that is still no guarantee that the tape will work at the most critical time. Unfortunately things can happen to magnetic media that at any point in time, and render the tape useless.

The person in charge of our DR once asked me how to make sure all of our tapes were 100% good all the time. I told him there isn't a way. As long as you use magnetic media, you will have a chance of media failures at the most inopportune time.

Now if you could write to DVD's at the rate that you do tapes......

[John Carr_2]

Hi

i agree with your boss use another tape drive on another server and do an frecover of say one or two files to a temp directory.
This will prove the tape to be good.

John.

[harry d brown jr]

Simon,

It doesn't matter "how" you verify your backups, as long as you can do so. From experience, I've seen some rather "thick" recovery manuals, but these only address the issue: how do we restore. They never take the PRO-active stance: how do we make sure we can restore!

You are on the right track. I only wish others would follow.

live free or die
harry

[A. Clay Stephenson]

Hi Simon:

This is a non-trivial question. As a first step, I would at least do an frecover -N -v and on another tape drive. I have seen several cases (especially DAT's) in which a drive could read/write to itself but could not to another drive. I have seen one instance of that condition in a DLT7000 drive as well. That's the easy part of the question. Now here's the more difficult part: Frecover -N or even a full restore elsewhere can only verify that what was in your graph file was actually backed up; in other words, your backup could be absolutely perfect and yet utterly useless because someone modified the schema and you missed one database file. It is absolutely essential that whoever does the backup configuration is kept informed of any changes.
Even then, the ONLY way to ABSOLUTELY, POSITIVELY know is to do a full restore to a new location or host and start the database.

I use OmniBack and this is my scheme to know for sure:

1) Each backup is copied to another drive using a different drive to read the media than originally created the backup. In order for the omnimcopy command to succeed, the original media and backup must be valid. (The copy is stored offsite).

2) At least once per month, one of the copied media is selected randomly and restored to a sandbox environment. The database is started and a few SQL statements are executed to verify the operation.

-------------------------------------------

This may be overkill but I sleep very soundly; in my case, all of this is completely automated.

[Frank Quinteros]

Simon,

fbackup is the 'backup' utility of choice for many operations, but not the only one, remember tar and cpio. If Disaster Recovery is the concern, lets not forget HP's Ignite-UX recovery utility. The make_tape_recovery -AvIt (cron entry) will 'backup' your entire root volume group into one tape which you can later use to 'recover' the OS with on the same or alternate server. Ignite-UX has been modified to include VG's other that root and can be found for free on the Application CD's.

[Simon Abbott]

Thank you every one.

Those were exactly the kind of ideas I was looking for - I'll start cracking on a plan now...

Thanks again for your time,

Simon.

[John Carr_2]

Simon

please let everyone know how you go on your reply is very vaulable to the search engine and our curiosity

John.

[Bill Hassell]

A couple of notes: fbackup's -N is actually quite extensive as a test. Not only does it check the tape drive's status in reading the tape, but internal checksums generated by fbackup as well as headers are also checked.

tar and cpio should only be used to exchange small files as they cannot backup largefiles and they lack a central index and error recovery code.

Ignite/UX is a disaster recovery procedure only, not a general backup technique. As noted in the man pages, there are a number of considerations concerning recovering the boot volume. Rehearsing a vg00 restore on a test machine is very important, including writing down all the extra steps to get the restored system back to normal.

One of the important Ignite/UX issues is the vulnerability of the /dev directory. The names of device files and the hardware path to which they belong is not guarenteed to match the original system. Device files are created in the order they are discovered. When the machine was new, there was minimal I/O, but as devices were added, new device files are created. But when the system is recovered, the device files are created with a backplane scan and not necessarily is the same order (and names) as the current system.

This will affect the restored system as the names of device files are stored in lvmtab and other config files such as backup software config files.

[Ronald Cogen]

Hello,

fbackup will usually create an 'indices' file listing the contents of the tape based on the tape header. This list does not reflect the actual content of the tape. On the baseis of this fact it is important to have an exact figure of how many bytes are to be backed up and how many bytes will fit on the tape you are using. Otherwise, you may get a big surprise.
I would not recommed using DDS as they are not very reliabele. DLT or something similar would be better.
Personally, I think frecover -N ... should suffice, because it lists what is actually on the tape. If you have enough disks to actually restore large amounts of data, well then more power to you. But, at about $500 per GB disk, that is a luxury for us.