Time estimate for SG implementation

[Sean OB_1]

Hello!

I have to put together a time estimate for a service guard implementation.

I have never done a front to back implmentation so I'm not sure what kind of time estimate to make.

The estimate is for a 2 node cluster running Oracle, attached to an HDS SAN.

Servers are rp54xx with dual quad ethernet and dual fibre to the san.

Can someone give me some direction on how much time it will take to do a design and implementation of this cluster?

Any and all thoughts are appreciated.

Thanks,

Sean

[melvyn burnard]

In all honesty, this really is hte "how long is a piece of string" question.
There are so many variables in this equation, that the biggest question would be where to start.

For someone who is an axpert at SAN's, SG, Oracle etc, coul;d probably do it in a day or so, provided all hte hardware is ready and connected, and the details of hte configuration you want are available, e.g. lvol sizs, ip addresses, etc.
For someone who has not really done htis, it could be a week or even more. Would you be looking at using a consultant to do this? If so, they may have some idea of timeframes.
It may be worthwhile taking a look at hte docs at:
http://docs.hp.com/hpux/ha

[John Poff]

Hi Sean,

I agree with Melvyn. Putting together a MC/SG cluster isn't real hard, but there are lots of little things that can trip you up. The first cluster I setup took me about three days, and it was very simple. The key to it all is design and planning. If you understand what you are trying to do before you start it will be pretty easy. Assuming that you are comfortable with networking and SANs, it should be pretty easy. I'd shoot for a day to design and plan it, two or three days to put together the cluster, and a day to test it. If things go well and you get it done sooner than that, you'll be a hero!

JP

[A. Clay Stephenson]

Well Sean, I did one very similar to yours over the Easter holidays. The actual MC/SG cluster setup running Oracle on a 2-node cluster was done in 90 minutes!

This really only represents the time needed to convert the Oracle instance running standalone on one host to a fully functional package. The two boxes were also fully patched with all the required prerequisite patches for MC/SG. It does include all the time needed to export/import the VG's, setup the additional LAN connections, and install MC/SG and the latest MC/SG cumulative patch.

I then spent another hour trying to crash it and making sure that the failover was handled correctly.

I only tell you this to illustrate that the MC/SG part is rather easy. The hard part is making certain that the boxes are already very robust so that drive / card / SCSI cable failures are handled long before MC/SG ever comes into play. The vast majority of your planning is going to be spent on identifying single-points of failure (SPOF's) and deciding how to deal with them.

The things that need careful attention:
1) Power
2) HVAC
3) Loss of router(s) and switches
4) Loss of disk drive(s) and/or arrays

I can tell you that I've never done one that takes more than a week to design
or more than about two weeks to implement - assuming that the equipment is on-hand and that you don't have to find some union guy to install a new power connection or go through a complicated engineering change order regime that has to be passed up through six levels of management.

On the other hand, I've seen firms take six months to do the same thing.

[John Meissner]

The Part that usually takes the longest for use is the land cables and the DNS entries... This is becuase our telecoms team handles this so we have no control over how long that will take.

After that, provided all your servers are setup and installed and the LAN and hardware is finished...

It usually takes us about a day or two to get everything setup. Probably two days more often. And than you have to figure on any "other" issues you encounter in your day. so you may want to push the SLA (service level agreement) back a few more days to give you a buffer. Then you have testing which usually only lasts for about 4 hours provided nothing goes wrong with the application which we've had a few instances of. The application team will come up and say... oh... we forgot to change these ENV variables....
the acutal installing of MCSG and implimenting it i'd say 2 days... but make it more for you own sanity.

[Uday_S_Ankolekar]

It depends..! It depends on how are you going to design SG cluster. It depends on how many packages, depends on redundany factors, depends on script to be involved in SG packages. Testing SG for hardware failure , software failure etc... and depends on your expertise. Look for some good documentations in KB, search for umcsg.

Good Luck,
-USA..

[Sridhar Bhaskarla]

Sean,

It depends on the site and the person doing the work. I will put down the high level tasks and you can estimate the time.

1. Get redundant network connections for the package networks.
2. Get the heartbeat network connections
3. Get the disks shared between the servers
4. Configure the redundant power sources
4. Prepare the configuration files - cluster as well package configuration. Write the package scripts.
5. Do the File System and Volume group work. This includes creating volume groups,filesystems, exporting the volume groups and importing them on the failover server etc.,
6. Apply the cluster and package configuration. If you have done all the above, this won't take more than 30 mins.
7. Run the cluster and Test the package.
8. Do the same on the failover server.

-Sri

[Rainer von Bongartz]

As already said this depends very much on the skills of the person involved.

With good MC/SG, Volume Manager , SAN and ORACLE skills this will be implemented in 1 day and then count another 2 day for excessive testing.

Without good knowledge of the above you should be prepaired for a two week implementation period.

Regards
Rainer

[Karthik S S]

Hi Sean,

Setting up a 2 node cluster shouldn't take much of time once you have proper design and plan in place. Create a checklist containing the following,
1. OS and Applications required
2. hardware inventory
3. Number of users and groups needs to be created
4. package IP and hostnames
5. Connectivity diagram

Decide upon what are all the applications that need to be HA enabled and check if those applications are supported by MC Service Guard toolkit. If not write a package script yourself. Also decide on failover policies. Make sure you have enough LAN cards ( atleast 4 -- 2 for the main network and 2 for the heartbeat ) ... get the hostnames registerd for the package applications ... etc.


Enjoy clustering,

Karthik

[Steve Lewis]

Speaking as someone who used to train people in implementing MCSG I have to say that Clay's answer is spot on.

The time to build the cluster and packages is small (unless you also have to install Oracle!). On my course, we built the cluster (including hardware adjustment of SCSI controller IDs and plugging the network cables in the first morning, built the package (a simple one consisting of a filesystem and IP) in the afternoon, tested switchovers the following day. All this included lectures on each part followed by practicals.

The real time involved is in
- testing
- documentation
- training the customer engineers how to use it on a daily basis

You must test every part of the cluster and package until it switches over - then you test the switchback. As Clay says in his profile - if it aint bust - you ain't done yet.

[Duncan Edmonstone]

I'm going to agree with Steve on this... actually physically setting up the cluster is the easy bit, and shouldn't take more than a couple of days to get a base 'vanilla' cluster configuration.

BUT

The reason for implementing MC/ServiceGuard is to enhance application availability, so you now need to think about how this cluster is actually going to work within your compute environment. This means:

LOTS of testing - not just 'does it failover?', but 'what happend to application functionality after failover?' after a controlled failover? after a TOC/panic? what happens when network components fail? what happens when storage components fail? The list goes on.

Documentation - detailing all the tests, all the procedures for managing the cluster and recovering the application in specific situations etc.

Training & Familiarity - your sysadmins need to know how to manage MCSG - your DBAs need to know what differences it makes to managing the database etc.

Basically before even thinking about ServiceGuard I'd urge you to do two things:

1. Look into the best practices found in ITIL (here's a good place to start:
http://www.itil.org/itil_e/index_e.html

2. Buy and read this excellent book on HA systems:

http://www.amazon.com/exec/obidos/tg/detail/-/0471356018/qid=1054801614/sr=8-1/ref=sr_8_1/102-3926701-4206518?v=glance&s=books&n=507846

HTH

Duncan

[Victor BERRIDGE]

Hi,
I agree with all the previous replies, the conversion isnt where you will spend time, its getting every thing ready before starting. More time you spend studying the case and knowing what your are going to do - less time you will spend in implemeting and testing (no waiste of time).
For someone new to ServiceGuard Id advice to take some advice from some consultant (HP has some) if you can afford it because the success depends mostly on the design of your cluster
(and the goal you are to meet...)
Saying all this for a newcomer lets say 2 weeks to study the design of the cluster and getting everything (hardware-software) ready
and a week to do the implementation and testing
(1-2 d impl. 3-4 days intensive testings)

All the best
Victor

[Jarle_1]

HP charges about 10000 Euro for the service including full doc, SPOF analysis and app integration of one oracle package .

Rgds Jarle