Testing backups - best practice

[Mark Blonde]

This may seem like a strange question. We are reviewing our entire backup strategy. As a consequence of this, management has asked to test our backups periodically by running test restores. However, because of the large amount of servers we have we can only do this so often.

Is there a best practice/industry standard for how often we should test restores?

Thanks,
Mark

Thanks,
Mark

[A. Clay Stephenson · Accepted solution]

There is really no answer here except "it depends". Your tests really have to address two distinct areas: 1) Backup Integrity -- is the backup a faithful reproduction of the data? 2) Backup usefulness -- does this backup capture everything needed to restore the application to operation?

It is perfectly possible for a backup to be absolutely perfect (in the sense that it copied everything accurately) and perfectly useless. A good example would be a backup of a running database without first putting the database in a backup mode.

There are also two very different ways to do backups: 1) Inclusive 2) Exclusive. Always strive to use the latter. Essentially backup everything except what you know you don't need. I have seen far too many horror stories of backup scripts that require a list of things to backup. That method is truly asking for an "oops".

The best test restores involve restoring to a sandbox environment and then actually bringing the application up. If that works you know you have a very good backup. I typically schedule those monthly for most of my servers.

I am a firm believer that each critical backup should be tested for accuracy. This might simply be running the verification phase associated with most commercial backup solutions. My method is more robust. During the day, I make a copy of the previous night's backup; i.e. a medium to medium logical copy. That does two things for me: 1) Because this is a logical copy (as opposed to a blind bit-by-bit copy) I know that if the copy succeeds the original must be an accurate backup (no media errors). 2) I now have two copies of the backup --- one remains in the library for immediate possible use and the other is stored off-site for disaster recovery purposes.

It has been my experience that those who plan well for restores never need to use them but ...

Food for thought, Clay

[Chuck Ciesinski]

Mark,

First let me apologize for the lateness of the response. IMHO, it's not just the data you need to be testing for accuracy and completeness. You need to address the procedures, the connectivity, system types, availablitiy of resources, assuming you are simulating a D/R test, getting resources to the site which costs $$$$$ and time, making sure that your IGNITE procedures and tapes are available, getting cooperation and licenses from third parties for
your D/R systems.

As far as frequency, I agree that 'it depends' is the answer that only you and your management can decide based on what your company does.

Food for thought,

Chuck Ciesinski

[Mobeen_1]

Mark,
There is no real inductry standard on how often you should test your restores. This really depend on
1. The size of your data
2. Criticality of your data
3. Availability of resources
Based on these factors you need to come out with some time thats acceptable to the customers.

Staying on this subject.....
Some of the things that need to be considered in addition to just testing backups/archives or restores and retrieves are

1. Are the backup/archive times acceptable
to your customers or well within the
SLA.

2. The same goes with regard to restores or
retrieves.

If your customers are not happy with the backup or restore performance then you may have to consider doing things like the following to refine your entire backup/restore design
1. Identify the bottleneck
(Some times large data files will
contribute to the bottlenecks while
other times a file system containing
millions of small files does)
2. Refine your backup/archive or
restore/retrieve strategy by asking
yourself some of the following questions

Q1) Can i backup instead of archive for
some type of data and vice-versa?
Q2) Do i get better performance by doing a
tar of some files before backing up?
Q3) Do the archives finish faster when they
are written to the tape instead of
being written to disk and then onto
tape?

I am not sure if you are using native backup tools or are using some enterprise backup software for this. If you are using something like TSM (Tivoli Storage Manager) , then a lot of these will be put in place and also you will have a DR module etc.

Some relevant link
http://www.disasterrecoveryworld.com

regards
Mobeen

[The Real MD]

I feel only slightly overwhelmed, from the previous three, very comprehensive answers.

but I feel there are primarily two streams when dealing with data protection

the first being business continuity. Which is primarily concerned building fault tolerant systems where there is no single point of failure. Which can cope with something as simple as disk failures right up to no access the building for whatever reason being fire, flood or martians.

the second method is taking copies of the data as frequently as possible to ensure recovery to a known point in time.

At the end of the day it all comes down to cost, what is the cost benefit analysis of never having down time against being able to recover from a tape. If your business is of the on-line auction type then down time will cost you money in lost business and probably by the second.

If you have never tested a restore, then you should test you can perform a restore. Ideally, its always an eye opener to try and recover from a simulated disaster recovery situation from any offsite backups that are available. Assuming not all the key people are available and using/followung the documentation to recover to a known position. To keep it as real as possible.

Whatever solution you come up with make sure the management are educated about it's limitations, as well what it can do.

I hope this helps

Martin.

[Trond Haugen]

In addition to what has already been said about D/R and recovering systems do also remember to test for recovering a single file "form before it was destroyed last Tuesday"...

Regards,
Trond

[Mobeen_1]

Martin,
Thanks for chipping in with some usefull information. Yes, Business continuity is an important thing and that needs to be at the back of ones mind when formulating any kind of data protection procedures.

I consider Business Continuity to be at the top level under which you can have things like Disaster Recovery. Technically Disaster Recovery Planning is just one aspect of ones Business Continuity Plan, some other items like Contigency Plans could be part of BCP as well

rgds
Mobeen

[Antoniov.]

Mark,
sorry if is too late.
Thi article can help you
http://h71000.www7.hp.com/openvms/journal/v3/backup_strategies.html

@Antoniov

[Wim Van den Wyngaert]

The only way to know if your backup is realy complete : restore it and continue production on the restored system. If this doesn't work, you have the original system to get the missing pieces.

Otherwise you will get a disaster and at that moment you will discover the missing pieces. And won't have them.

Wim