Creating Service Guard pair on an active system

[Christopher LaPlante_1]

Currenly we have a pair of servers slated to become a service guard pair. Due to time constraints to go live. We want to put one server into production and build it into a cluster some time later after it has been put into productions. How difficult is it to create a service guard pair on a live system? Would this be an easy task to do? Can it be done without bring down the production server? What types of steps should we take up front on the server to make building it into a cluster easier?

[Ross Zubritski]

Chris,

You can bring up a "single" node cluster. However, I would cluster the two boxes together now. Then simply package the applications and bring them on line as needed.

Regards,

RZ

[John Meissner]

Setting up a SG cluster isn't exactly a small ammount of work to begin with. I've done some both from the get go and also after it's been put into production. I'd say the hardest part of implimenting a SC cluster on a production box is arranging the outages for testing. I've spent a good deal of time arranging outages and sometimes it takes our application teams several months to agree on an outage date. The work isn't much harder... but several times I've had one of the application teams not get their part ready but we didn't discover this until mid-outage. then we had to scrap what we were doing and fail back to the original server and setup yet another fail over.

Now don't let this discourage you from doing it after the fact... but I just wanted to tell you some of the problems i've seen.

[John Poff]

Hi,

The first thing I would suggest to make things easier is to sit down and plan out exactly what your final, production cluster will look like. It will help you figure out the road map to get there.

As for the cluster, you can go ahead and create the cluster without any packages and bring it up. That way, when you finally get some downtime on the production application you'll just have to make the package and add it to the cluster. You could go as far as creating your package config and control scripts, so that when you get your downtime you are just making the package.

On the other hand, if you have it all planned out [shared disks, networking, etc.] and setup correctly ahead of time, you can make a cluster pretty quickly. It really depends on how complicated your cluster will be, how many packages you need to configure and test, how much down time you have, and how much experience you have with MC/SG. Plus, the disadvantage of doing the cluster ahead of time would be that if something simple like the heartbeat timed out, your box might TOC and try to failover even though you don't have any packages defined yet, which would be bad if it was your production server.


JP

[Michael Steele_2]

I'd tend to agree with Mr. Meissner on this one. I've built three SG clusters and there are some HW fault testing that has to be performed before release into a production environment. The network failovers in particular has to be reviewed. For example, the mistaken use of cross over cables with the heartbeat LAN can confuse some people. Go through a switch or at least a hub with your heartbeat LAN configuration and test its failover.

A good SG consultant should be able to take you through the necessary steps. Spend 3 weeks planning and testing your failovers with a test xclock test package and then put one node out there and build the live application package into it. In general, this is a enabling of the package, but not all together.

I'd recommend an HP SG course otherwise. The advanced SG course is for 3 days and should be enough.

[melvyn burnard]

First, you may want to take a read of the Managing MC/ServiceGuard manual at
http://docs.hp.com/hpux/ha
Then I recommend you do the 5 day Hands On wiht MC/ServiceGuard at an HP Education centre.
Planning and implementing a cluster takes time, thought, and is not to be taken lightly.
Having installed/configured over 100 clusters over the last 8 years, htere is ALWAYS a little "gotcha" hidden away somewhere, and the first hurdle you have to overcome is decide if your application is compatible wiht ServcieGuard, i.e. it can work in a cluster. There are many apps that do not work in a clustered environment.

Just my two cents worth

[Stephen Doud]

Hello Chris,

As specialists of ServiceGuard in the HP Solution center, both Melvyn and I have seen our share of frustrated administrators. Hence Melvyn's valuable advice.
Using the manual as your guide, you might find this document, available to contract-bearing customers via the ITRC knowledge database:

UMCSGKBRC00010342
TITLE: ServiceGuard Cluster Implementation Summary

It is entirely possible to load ServiceGuard and create a cluster on set of nodes (HPUX servers) without rebooting. If this is a two-node cluster, they will need a common LAN to pass heartbeat packets and sync the cluster status, and they will need a shared LVM volume group or external quorum server to handle arbitration should a failure occur in the heartbeat LAN.

As stated previously, SG takes some time to master. The manual is a great place to do self-study before implementation.

-S.