Disaster Recovery Plan - Info needed

[Karthik S S]

Hi,

My question here is going to be vague. I want to learn more on Disaster Recovery. Need your help in that. Any links/Docs/Notes are welcome.

Secondly, have any of you implemented HP-UX/NetApp Filer based SnapMirroring across different sites for DR?. If yes could you please provide me with more details. How to integrate this with Disaster Recovery. How to get the applications run when one of the site fails with the use of mirrored data on the other side???

I need more details on this as we are planning to come up with a DR plan for our company.

Thanks
Karthik S S

[melvyn burnard]

You could take a read of some of the white papers etc at:
http://docs.hp.com/hpux/ha/

[twang]

Some information you might need to keep in record:
# dmesg
# sysdef
# strings /etc/lvmtab
# kmtune
# lsdev
# landadmin -x ..options ..
# print_manifest
# swlist
# swlist -lfileset
# cat /etc/system
# cat /etc/resolv.conf
# cat /usr/local/bin/*

[Michael Steele_2]

Any DR plan begins with capturing the current state of each server's O/S with Ignite make_tape_recovery tapes which some SA's make as frequently as once a week, certainly, they should be made with every change of the server, i.e., adding patches. I make copies every Friday night and archive the last one of the month for a year. This is because sometimes errors aren't caught for several weeks so keeping copies for a year are important.

Regarding "...HP-UX/NetApp Filer based SnapMirroring..." can you supply a product number and double check the product name. Can find anything and this is not the norm.

[twang]

there are some advices if using ignite-ux:
- run make_tape_recovery regularly

- do it every time you apply patches or system administration task.

- test backup

[Stefan Farrelly · Accepted solution]

Were thinking very seriously about doing DR usnig Netapp Filers and SnapArchive - notSnapMirror. SnapMirror licenses are expensive, snapArchive licenses are 1/4 the cost, but the quickest you can update is every hour (SnapMirror can be a low as every minute) but for DR you only really want to archive your data to your DR site every day/night, so SnapArchive is best.

What we would do is use Netapps filers here for our data storage. Keep some spare HP servers and more Netapps filers at our DR site. Every night the Netapps do a Raid-4 parity snapshot (almost instantaneous) and copy the blocks which have changed (incremental) to the DR site, so basically our data is only a day old. If we need to go to DR we up the HP server at our DR site, mount the filesystems from the DR Netapps filer, and were up and running.

The only issue is using SnapArchive only allows read-only access to your data at your DR site, but if you need to activate DR to go live you call Netapps and get a license to make it read-write (basically pay them some money) In future they said they will automtate this.

Sounds great in theory. Ive been to a recent Netapps user group meeting and some customers demoed their setups using this, and its working fine. Some have been using it for 1-2 years.

Its benefits are Netapps and SnapArchive are cheap - much cheaper than anything from EMC for example (We also investigated using EMC), and only data which is changed gets copied - reducing network size/costs, and its tons faster than backing everything to tape each night and then having to restore all the tapes at a DR site (which would take us days!). We could activate the DR site in an hour or 2 using Netapps+SnapArchive.

[Ron Brown_2]

We used NetApp filers for a variety of tasks at my last job. We enabled the SnapMirror function, then later implemented SnapArchive.

The filers formed the backend for a linux server cluster running SMTP and POP3 for > 120000 accounts; spools and account information were stored there.

They also held databases for the MySQL backend used by the mail and DNS clusters, and held web sites running under Linux, Solaris, and Irix.

This was > 2 years ago and the clusters are all still humming away under this configuration. The cost is well worth the peace of mind it brings, if one can sell it to management. This is especially the case if a server ever fails and you tell management about it after the server is fixed, rather than management hearing about it from Support or Sales.

[John Meissner]

1- I have been involved in DR for the past year and a half with my company and am the primary contact for DR. In the past we have been using a 3rd party vendor to host our DR's - This works but it's expensive and we did have some data loss due to time of last backups. For our DR vendor we were/are using Sungard. I've developed a set of scripts (with a lot of help from this forum) to rebuild the External Volume groups (EMC disks).

2 - We are currently developeing a 2nd site strategy that will allow us to use modified Service Gard scripts to fail over our data to a 2nd site. We replicate the data offsite a third paryt company (who controls the data flow) and then to out 2nd site. At the 2nd site we keep 2 copies of this data. a backup copy in case the production copy dies durring transfer. This way the most data we'll lost is aproximately 4 to 8 hours of data.

After we get this process complete I could post the scripts on this topic - but they're a work in progress at this point.

We found it cheaper to use a 2nd site to recover to instead of a 3rd party. If you have any further questions for me aobut any details feel free to ask.

[Caesar_3]

Hello!

What you talk's here is almost the same thing
that my company start to build.
Some network with servers and workstations (HPUX and linux based),
NAS server from NetApp as sentral storage and
another NetApp in another place.
The NetApp has function that he made mirror (not the mirror that made for disks, much smarter) on the other one and can made it even on slow network.
If you have problem with the first place (disaster) so you have all the data on the other one same as in the moment of disaster
so you can continue to work on the other place
with no problem.

Caesar

[Karthik S S]

Hi,

Thanks for all your valuable inputs.

Ron/John/Caesar: Can you please give me more inputs on Snap Archive??. Is it a Offline or Online Storage. With SnapMirror the data is replicated online on a remote filer. Is it the same with Archiving also??. And also I need some Info. on configuring the Applications to run on the DR site incase of a disaster. As we are having only the data on the DR site (On NetApps) how to get the applications work there? Do we need to have the same kind of servers at DR Site also as in the Remote Site??

Is there any good Docs. on planning for DR??

Thanks
Karthik

[Stefan Farrelly]

SnapArchive is online storage, same as SnapMirror. As I said the only difference is the data at the DR site is read-only, not read-write. If you need to activate your DR site you need a license to convert the DR filer from read-only to read-write.

You will need a server at your DR site attached to the filer there, with all your applications installed just as you have at your live site. If you need to activate DR you up the server, and then mount all your filesystems & data so that it looks the same as at your live site. Then you can use it.
We plan to keep 1/2 as many HP servers as we have at our main site at DR.

Take a look at the netapps.com site in their tech library - tons of info there, eg;

http://www.netapp.com/tech_library/3057.html

[John Meissner]

Karthik,

You don't need to have the SAME servers or even the same type of server. You just need to have similar servers (example L & N calss)

As for how we worked out the transfer or data we wrote a script to handle that.

I'll attach all the scripts we 're using currently - some of them still need work but here they are.

[John Meissner]

Here's 2 of 3

[John Meissner]

3 of 3

This is the modified MC Service Gard script

[Karthik S S]

Hi Stefan,

But I could not find any Info. regarding Snap Archiving in the NetApp Sites ( search resulted no docs ..)

Karthik

[Stefan Farrelly]

Youre right, its a new product so nothing on the netapps site about it, yet. I should have made myself clearer, sorry;

SnapArchive is exactly the same as SnapMirror except;

1. cost is 1/4
2. data at remote site is read-only (not read-write).
3. if you do failover to DR site you need to purchase a license to change SnapArchive into SnapMirror, then you can mount read-write on your servers and startup your apps.

The examples we had at the netapps usergroup meeting last week were SnapMirror and SnapArchive (most of those who had been using it for 1-2 years were using SnapMirror, but some had converted to the new SnapArchive as the license is so much cheaper).

So the SnapMirror documentation on the netapps site is the same (almost) for SnapArhive.

[Karthik S S]

Hi Stefan,

But how long does it take to obtain the license and convert the data on the remote DR site to be read/write which is stored using Snap Archive??. Is the downtime nominal?? Do you have any whitepapers or documented something on Snap Archive??

Pl. help

Thanks
Karthik S S

[Ron Brown_2]

techssk:

SnapArchive is online storage and is strictly read only. You will configure the repository during Filer setup (it can be added later) and it eats a chunk of your disk space.

We had a site in Las Vegas, NV, USA that was a complete mirror of the main data center in St. Louis, MO, USA. All data and services were duplicated between the sites and traffic could be routed between the two in case of emergency. Expensive, but one has to weigh to cost of downtime against the hardware and engineering costs.

You can find more information at http://www.netapp.com.

[Ron Brown_2]

That URL Stefan posted is really all you need to investigate further. Good show!

[Mike Fisher_5]

Ron

I'n interested in this "SnapArchive" that you & Stefan mention
[it being cheaper than SnapMirror until you need more than just "read" functionality]

I've looked at your link above &
can't navigate my way to a reference to "SnapArchive"

Have I missed something?

Mike Homer Simpson Fisher
P.S. No points please Karthik
This is for my curiosity
& not a contribution

[Karthik S S]

Hi,

Sorry to bother you again ... :-(

Can somebody post more on SnapArchiving???


Regards,
Karthik S S

[Stefan Farrelly]

Everyone;

SnapArchive s almost identical to SnapMirror. Its new, and thus no direct info on it on the netapps website yet.

For all technical info on SnapArchive refer to SnapMirror. Its the same code! There are only a few differences as ive highlighted in an earlier reply (cheaper and read-only and can only snap every hour at the most).

I got all this from a Netapps user conference and discussing with lots of people from Netapps in London last week.

Cheers,

Stefan

[Ron Brown_2]

Mike:

It's been a while since I did this, but take a look here:

http://www.netapp.com/tech_library/3066.html

and here:

http://www.netapp.com/tech_library/index_dpr.html#toc