fail-over to a dedicated or DEV server?

[Hanry Zhou]

We are considering to setup a SG with 2 nodes in it. One is production server, another fail-over server. Now, the question is what is the down side to use the Development or Test server as the fail-over server?

There is no question about it that it'd better to use dedicated server, however, for saving money purpose, we want to use the Dev server as the fail-over. Can anybody please let me know in the detail of what problems or impacts we will have to use this option? what the extra headache we will have?

Thanks,

[Alan Meyer_4 · Accepted solution]

well, there are 2 scenerios that could happen on a fail over.

1. The production package runs in addition to the DEV/TEST environments and performance is impacted accordingly for everyone accros the board.

2. The production package stops the DEV/TEST environments and takes over the standby machine completely. Impact to production is only affected by the differences in hardware between the production and standby machines. Impact to DEV/TEST is tremendous because during failover, DEV/TEST are completely down.

Maybe you can upsize the DEV/TEST box to account for more throughput during a fail over event.

Just my thoughts.

[Tim Nelson]

This question is the question of the day when it comes to service guard failover.

From a reliability and ease of management a dedicated server is the way to go. It's only configuration and purpose is to failover. From a cost stand point it is hard to justify.

If the standby server is going to be used for other purposes you must take into consideration that it must be configured to support your developement and the failed over environment. i.e. host files, network, logins, permissions, and any other environments needed. Of course you can script any and all changed needed into the SG scripts but this does add complexity and a wrench into the failover process.

If you want something that is going to fail over reliably go with a dedicated server. If this cannot be done financially take every difference that exists into consideration when creating your SG scripts.

Don't forget you also have to fail back, and hopefully with little impact to your development server

[A. Clay Stephenson]

One of the major impacts is ability of users to access both development and production environments. This can be a major headache during auditing. The other thing is that you have to make sure that your development box has enough resources to run the package and keep the development environment up.

Let me suggest a Plan B that sounds insane at first glance but is really a very good approach. Use both of your "good, fast" boxes in the MC/SG Cluster and go out a buy a dog (e.g. an old K580 or so on the used equipment market where they are dirt cheap) for development. Your developer's will hate you because the compiles are so slow --- BUT, in reality, compile/links are at most 1% of the development so the actual impact in trivial. Be prepared because they will complain to your managers but I've done enough development to know that slow boxes are a good thing. Over time even the developers will come to appreciate this approach because of fewer performance complaints. Code that will run reasonably well on a SLOW development box has a much greater chance of running well in production. It's very common for development code to suffer performance problems when moved to production but by doing your testing on a slower box, you improve your chances of success enormously.

[Alan Meyer_4]

One thing to also consider is the complexity of the package that is run on the cluster. Does it interface with other servers(Like SAP appication servers) where the length of time to switch a package could be considerably longer? thereby affecting the existing environments more as it competes for resources.

[Hanry Zhou]

Hi,

"One of the major impacts is ability of users to access both development and production environments"

Can you please explore it a little bit more, what you mean by that?

I would assume, when fail-over happens, the development function will be shut off, and the application software will be umounted, production version of app.sofware will be mounted. So users should not able to do any development job.

[A. Clay Stephenson]

If this is any anyway related to Financial systems so that it is subject to Sarbannes-Oxley legislation then you will get dinged bigtime if those who are able to access deveopment are also able to access production. The concern is that developers will be able to alter code and/or data in production without rigorous controls being in place. It's not enough to say they don't do it; you have to be able to demonstrate that they can't do it.

[Mel Burslan]

Quote
________________________________________________
I would assume, when fail-over happens, the development function will be shut off, and the application software will be umounted, production version of app.sofware will be mounted. So users should not able to do any development job.
________________________________________________

As the wise-men said one time, assumption is the mother of all "mess"-up's.

First, if you configure your production and dev/test applications as separate packages, there is no need to shut down dev/test when production fails over to this node. You sure can force this but then you will lose major time for development, i.e., your company will pay developers to sit on their butts. After a couple of days of fail-over scenario, one persone will have a revelation and say "why don't we use both applications at the same time. SG can do that according to HP". And they will realize the performance penalties are not that big of a deal under these circumstances. At least tolerable compared to paying staffers not to do any work. Then Clay's Sarbanes-Oxley people start banging on your door with questions like "how do you keep the development users from touching your production data ?" or "show us your audit trail, your production data has not been compromised" and believe me, I am telling this from a very recent experience, digging up this data is neither fun nor quick. It almost justifies buying another fast server let alone a slow clunker.

[Dave La Mar]

Hanry -
Our shop adopted exactly what Clay explained as Plan B.
And yes, there was initially a lot of gripping from developers, at first. That eventually leveled off and is no longer a burning issue with the developers.

As stated, Pland B gets my vote.

Regards,

dl

[Devender Khatana]

Hi,

Yes, Clay's option seems to be more adequate as it will cost very low and will still be able to serve for the DEV environment all the time. The actual use of the idle node in the cluster can be done for some purposes which are not quite important and you can live without them for a few days if required. Another use of the node can be used for some packages which use minimal resources.

In that event the system can be used to run both packages at the same time with only a small reduce in the perforamnce of production( Only during downtime of one node) that anyway you will have to opt on the cost of the cost saving.

HTH,
Devender

[Hanry Zhou]

As I can read from this thread, the main concern with this option is the auditing part, but, we can copy the passwords account over to the test/failover server when the failover happens. In this way, we can ensure that the data on the failover server will not be comprimized by the developers because they don't have any login accesses to the adoptive node. Right?

[A. Clay Stephenson]

That solution would not pass my audit; you don't seem to grasp that just because you can
move passwords over doesn't mean that you will. You must have a documented procedure in place along with the scripts that actually do this and prove somehow that development users are truly isolated.

In principle, with very tight controls, you could actually allow deveopment to continue while production also runs on the same node. The difficultly is actually proving that the controls are adequate. One of the real problems I find when complex schemes are in place (no matter how valid) is getting the auditors to understand; I've yet to be impressed by any auditor's technical prowess. I'm not saying that there aren't extremely UNIX competant auditors out there but rather that I have yet to meet one during any of my audits.

[generic_1]

Buy a little bigger 2nd box and use a NPAR or VPAR on it. VPAR could dynamically reallocate resources which could allow you to test how much rsources your new changes will need too which is cool.