Use DR servers for Test / Dev purpose during normal times

[Hanry Zhou]

I am not sure if this is right place to post this message.

Since DR servers are sitting there and doing nothing during normal times, and it seems a kind of waste. If I use them for Test/Dev servers, will there be any issues?

One thing I feel about is that it should be difficult to configure, or to switch when real disaster is coming. Could anybody please layout the general concerns and what steos I should setup in genral to seperate Test/Dev environment from "Production to be" environment.

Thanks!

[Mel Burslan]

It is mostly security of the servers related. Development servers are generally open to anyone from developers, to QA testers to Production users and maybe via some extranet connections, to some 3rd party service providers, like off-shore developers etc.

When it comes to production it needs to be locked down pretty securely. If you do this securing from the beginning, then you may hamper the access of internal people other than production support to these servers. Also, in production, depending on the nature of your application, it might be subject to SOX or PCI audits, which means extensive logging and log retention policies. Although this aspect is more of a political concern than a sysadmin one.

Another point is, if your production data is residing on SAN, the SAN volumes should not be zoned to this server while it is in DEV mode, to prevent "accidents" So, in the case the internal disks are not enough, you will need a separate set of SAN volumes.

Depending on the nature and the resiliency requirements of the DR site, you may need to worry about a clustering component for the DR servers.

The best bet is, considering this is HPUX, to configure the server to the production role it is expected to play when a disaster strikes, take a ignite image of the server and reconfigure it to its DEV role and take another ignite image. When the disaster recovery is needed, reload the OS and all related components from the first ignite image you took and go. When the original configuration gets restored, blast the DEV image and you are golden.

Caveat Emptor: If you replace parts in this server, say a fiber-optic adapter, the WWN will change but your PROD config will depend on the old WWNs. So, it is a prudent practice to enact the PROD config periodically, say every 3 or 6 months to make sure it is in good standing and if not, update the necessary parts and take a new ignite image.

My 2 cents

[Doug O'Leary]

Hey;

From my experience, one of the issues you may face is that the people doing the development and testing can be unwilling to give up their systems when required for the DR test.  The previous post mentioned a potential issue about remnants being on the system from the developers and testers.  

Both those issues can be worked - one through coordination and policy and the other through your DR test.

1.  make_net_recovery of the DR and prod systems prior to the test

2.  close prod systems onto DR

3.  recover apps

4.  when done with the test, recover DR systems onto the DR systems.

Cloning the system should effectively remove any potential dev/test remnants.  

Hope that helps.

Doug O'Leary

[Hanry Zhou]

I feel I am missing some key and basic points on how this should be doen, Please be patient with me.

So, let me make sure I understand you correctly. In order to make the server to be a Test server in the normal time and a Prod server in DR time, I have to prepare two ignite images one is for Test, and another is Prod, when realy DR really happens, I need to blow away Test env, and load Prod image, and restore application, and everything else from Prod backups. When DR is finished, I need to load Test image, and restore everything else from Test backups.

What I just said is correct steps?

[Doug O'Leary]

Hey;

It sounds like over kill, but that, with one caveat, would be the best way.  The one caveat is that you would want to have extra disk space that you can attach to your test system so that you're not blowing away your test data to run the DR.

The reason you want to go to this extreme is to ensure that there is nothing remaining from the test system that could hinder or help your DR exercise.  In a nutshell, if you're rebuilding your system, you're probably going to be rebuidling it from scratch.  Anything left over from the test box is soemthing that's not going to be there in your real disaster recovery.  So, to have a clean test, you need to have a clean system.

That's perfect world.  There are a number of areas that you could cut some corners; however, you have to know that you're cutting corners and make the decision to do so.

So, perfect world:

completely blow away test, rebuild prod on new disks.  

Option 1:  Blow away test and use the disks from test.  At the end of test, just rename the database and use the DR exercise as a database refresh.

Option 2: leave the OS there but restore configuration files.  Leave the disks there but use them after the DR as a database refresh.

You could probably take this just about anywhere you want to go.  My suggestion is take a step back and remember the goals of your DR test.  You want to make sure you can rebuild your production system in case it ever goes away for any reason.  Can you be sure you can do that if you're not testing the DR plan on a clean system?

Anyway; hope that clears things up.

Take care;

Doug O'Leary

[Hanry Zhou]

Doug,

I feel we may be talking about different things.

I am not talking about exercising DR test on an existing test server.

We are going to setup a DR environment, and the new server will resides on another remote  site of our company as DR site. We plan to use the server as the test server in normal times, and then it will become the product server when real DR is happening. One thing I did not mention early is that we will also plan to use the storage (or oracle) replicating tool to update data between the local production server in DC site,and this remote server which should be functioning as Test server while it is receiving updated production data.

The server on the remote site is  Test server in normal times and receive production data updates, and then switch to DR production server when real disaster is happening. I am not asking for DR test or exercises.

I am asking for help and  the general steps to achieve that.

Thanks for your patience.

[chris huys_4]

Hi Harry,

> The server on the remote site is  Test server in normal times and receive production data updates, and then

> switch to DR production server when real disaster is happening. I am not asking for DR test or exercises.

Read up on HP Serviceguard metrocluster documentation on docs.hp.com.

Designing Disaster Recovery Clusters using Metroclusters and Continentalclusters, September 2010 (B7660-90028) is a good start.

http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c02464583/c02464583.pdf

Greetz,

Chris