HP-UX OS Recovery Strategies - Disaster Recovery

[Alzhy]

On our Solaris environments - we settled and have been successful in using mirrors of our Running OS disks which we offsite to our DR site and update regularly. When DR (tests or otherwise comes), all we have to do is plug in those disks, do a few tweaks and we're live to perform the actual restores of apps and data.

On HP-UX environments we are presented with 3 options:

1. Disk Mirrors/Clones as our Solaris strategy
2. Individual Re-Ignition of OS Backup tapes
3. Rebuild our Ignite Server, restore our NetIgnite backups onto that Ignite server and re-ignite/restore the other servers.

I reckon Option 1 will be the fastest to getting us back to a point where we are ready to restore apps and data compard to Option 2 and 3.

Our environment consists of a couple of nPars on rp8420s and a SuperDome. Approximately 12 server environments.

What do you guys think?


BTW, we do not havae a Hot-site DR...

[Patrick Wallek]

The problem I see with using DiskMirrors/Clones is the VG configuration.

If you have different hardware and your disk path is slightly different then you are going to have issues when the machine(s) attempt to boot and activate VG00. You will also have issues with the SCSI paths in the 'lvlnboot' information.

Most of these issues can probably be solved by booting into LVM maintenance mode initially, but that will take time.

I personally prefer option 2. If you have make_tape_recovery tapes, I would go that route, although you would still have issues if you have any vPars configured. Restoring those will still require your Ignite server.

[baiju_3]

Hi,

Creating Ignite recovery tapes and sending that to offsite is advicable .

You can boot from this Ignite tape and Ignite the servers incase of DR .

Using these Ignite Recovery Tapes ,there is no need of setting up an Ignite Server at DR site for restore , this tape is bootable and will do the entire job .

Use make_tape_recovery command to create Ignite tapes.

Also if you are creating Ignite over network for these servers regularly on your primary site ,these Images can be converted to Bootable tape Images and send to the DR site ,thus avaiding to have Tape drives and managing media on individual servers.

Only you need to have a DAT drive on the ignite server ,for this method to work .

If you are interested I can give more details on how to convert network Images to bootable tape images .


Thanks,
-BL.

[Jean-Luc Oudart]

Even if you could clone the hard drive you should still run your ignite backup and keep them offsite.

We ussually have 2 valid tapes and also keep 2 generations.

my 2c.

Regards
Jean-Luc

[Steven E. Protter]

Here was my strategy when I was in charge of a US site:

1) All servers had a tape drive and produced a weekly make_tape_recovery tape which was rotated off site. vg00.
2) All servers produced a weekly make_net_recovery which was backed up with Veritas and that tape was rotated off site.
3) Oracle servers were brought down for cold backup. Snapshots were tested but not implemented, that was the 24/7 strategy anyway and I belive Online JFS snapshots are a sound strategy.
4) Veritas netbackup backed up all authorized mountpoints, windows and HP-UX.

This strategy was tested by a complete loss of data on the disk array, March 21, 2005. HP-UX servers were the first effected servers to restore production service.

There were problems, but those were caused by variances from the strategies above. The servers didn't boot off the array so the make_tape_recovery backups were not tested last March, though they were tested in an HP Performance Center and on the test server.

I honestly don't know the strategy where I am now, but with 200+ Linux and HP-UX servers and me not the sole admin but part of a team, I imagine the strategy is different.

SEP
Unix Systems Administrator
Unix Systems Team
News Data Service Technologies(NDS)
Jerusalem, Israel

[Rita C Workman]

Here's one thought,
So you'd need compatible servers(not necessary exactly the same models) at both ends. And the arrays at both ends. Now to mirror your disk arrays (EMC has SRDF and HP has CA); SG cluster software to handle site-2-site failover depending on how far apart your remote site is would define with version to use.
Now this is the most expensive, but it is also the fastest way to get back up and running.

Rgrds,
Rita

[Marvin Strong]

I would think the method should be determined by your DR contract. If you contracted exact hardware, or similar hardware. The ignite tape method is useable. And I have had good success.

I have participated in a few DR scenerios and have found the ignite tape to be a very good one. HP OS is rebuilt in 1 hour give or take. And combined with some scripts that I used to keep non-vg00 information. VG's also recreated in short order. Followed by recovering the backup server. Which if it is an HP server, including the backup database/software in the ignite tape is also useful. Then you just need to tweak the backup software to use new hardware paths to your library, and drives.

I recommend Weekly make_tape_recoveries. With a 4 week rotation.

Of course if your DR contract doesn't promise any type of hardware this might not be the best solution.

[Alzhy]

Our DR contract as I've indicated is not a hot site but guarantees that we'll have compatible CPUs, same exact HBAs, network, fibre-channel, storage that we spec'd. SuperDome nPars may be recovered on rp8420 npars for instance. vPars on smaller machines maybe recovered to larger machines.

Option 1 -- which is esentially having periodic disk clones (via dd copies) it seems is the fastest per our tests. We use DS2100s/DS2110s and we also spec these disk enclosures at DR as our boot subsystems. So come DR, we just plug in the disk clones, bott off VxVM/LVM Maintenance mode, correct the configs, iopaths, etc.. run script to rebuild/carve LVM/VxVM diskgroups and we're ready for restoral of applications and data. About a quarter of the time it will take to boot and re-ignite from tape...

[Patrick Wallek]

It sounds as if you have done thorough testing with Option 1. As long as you are aware of, and comfortable with, the changes that need to be done in LVM maintenance mode you should be set.

I agree that option 1 would be faster than ignite tapes.

I'd say if it works for you, go for it.

[Marvin Strong]

I would say either option 1 or 2, will be good for you.

Option 1 is certainly the fastest. And as Patrick, stated as long as you are comfortable with LVM, should be a good choice. And since that is implemented already on your Solaris side, it should be easy to get management buyoff(at least in theory).

[Steve Andrews_6]

I believe the more options available to you in a true disaster the better off you will be: 1. Ignite make_tape_recovery and keep off-site. 2. make_net_recovery to an Ignite Server - back it up with make_tape_recovery and keep off-site. 3. Mirror the boot disk with MirrorDisk-UX keeping this mirror in sync and 4. Mirror the boot disk with MirrorDisk-UX again breaking the mirror just in the two primary get corrupted.

[Eric Antunes]

Hi Nelson,

I would go trought option 3 for HP-UX OS backup and at least a monthly tape for apps/binaries and weekly tape for database.

More important yet: keep all those tapes in a different location (more than 50Km) than your production site...

Best Regards,

Eric Antunes