Arguments against "production" and "test/Dev enviroments on same server(s)

[Matt Powell]

Greetings,

This is not a 'technical' question particularly, nor is it a security issue specifically, however the answer(s) may involve technical issues and a robust, readily available set of answers might be helpful to this community.

Presently I am a UNIX admin in a truely heterogeneous large "non-profit" healthcare IS environment. HPUX, AIX, OpenVMS, Tru64, AS/400, Linux, and of course NT/2000/Novell.

Many of our more complex, enterprise scale applications require test/development environments as part of the ongoing support of the application over the product life-cycle.

In some cases we have dedicated test/dev servers. Increasingly however, turnkey solutions providers are peddling to the non-technical decision makers, the notion of hosting the "test" and "production" environments of their products on the same servers.

It is my opinion that it is generally poor practice to have test/development environments co-exist with critical production enviroments on production servers. Although there are exceptions to every rule, I maintain that from the standpoint of system integrity, the risks generally far outweigh the perceived benefits.

My position on this topic has resulted in numerous heated debates with non-technical IS management.

I would like to solicite feedback from this group on this topic.

Are there well-reasoned arguments for/against this practice? Is anyone aware of 'best practice' standards ( SAGE, ISO, ISSA, IEEE, etc) that address this particular issue? Are there whitepapers, articles, or other documents people are aware of that address this issue?

I have been somewhat unsuccessful finding existing literature on this, so our discussion here might be particularly valuable.

Thank you,

Matt Powell
UNIX System Admin
Providence Health System
Matt.Powell@providence.org
( NOTE: This message also posted to /business issues/disaster recovery planning )

[Steven E. Protter]

Before hardware got cheap, my employer, a non-profit ran things the same way.

In 1995 we went prod on our first HP-UX system and it contained our production and test/development environments.

We obviously saved a lot of money, but we had problems.

1) We needed to test security issues, and to do that we loosened security which caused us to be vulnerable in our production environment.

2) When we were performance testing our new code the prduction environment frequently ground to a halt. This was because the server wasn't powerful enough AND because we couldn't lay out disk in a way to keep prod and development from bothering each other. This problem can be solved by a more powerful machine, more memory and a bigger disk.

3) Runaway jobs in either environment halted the entire server.

4) While doing a version upgrade of our database server, the install program whacked the configuration for our production environment. We took a production down lasting nearly half a day. Obviously stupidity could still cause this by sticking the cd in the wrong server, but that wasn't the case here.

Within two years, we bought a new production server and turned the old one into a development server.

This past year we went to a three tier environment. We have a true sandbox, for testing new release, patches and code. Development staff have their own box which doesn't get upgraded until the kinks get worked out. We have a production box.

With the cost of hardware, even if you obtain it over a number of years, the reasons above are good ones for having different boxes.

That being said, with a powerful enough machine and a really sharp administrator, you can go with the one box solution. Just don't promise the users anything with regards to uptime.

SEP

[John Meissner]

you have 2 ways of looking at this.

1- I would not run developtment on the same server as production.
Reason: if the dev gets really hosed you could impact production. We separate all our developement and production onto seperate servers. Keeps things neat... running smooth... and eliminates confusion about what is what.

2- you could install a server with a virtual machine. This way you could have 5 OS's running simultaneously. If one craps out you don't have to shut down production to fix it. My company recently had IBM come in and go over this solution with us. very nice.

[A. Clay Stephenson]

If I'm only allowed two machines (yes, I meant two) then I would definitely run Test/Development on the Production box - with a few restrictions (e.g. : maxdsiz,maxssiz set to conservative values so that no rogue process under development can grab all the resources; a box powerful enough to easily run both environements; very restrictive permissions so that the development group can't get into production).
The problem in large-scale development projects is that the Development environment may be nearly as critical as production in that the development costs might be thousands of dollars / hr. - so downtime in Deveopment gets escalated very quickly.


Box number two is used as a sandbox; this is where OS patches, Oracle releases, etc. are tested along with a test environment.

The best approach is 3 boxes
1) Production
2) Test/Development
3) Sandbox

You could, of course, use one box with VPARS but the concept is the same. I would much rather purchase used Test/Development and Sandbox platforms (e.g. K-boxes are dirt cheap) and use the 3-box approach with only 1 newer,faster production box than have only 2 boxes no matter how new.

One more model is the 3-box approach; Test/Devel and Production comprise a ServiceGuard Cluster while a 3rd box - often an old beast - becomes the sandbox. This model is often an easier "sell" to management.

[Martin Johnson]

A couple of years back we had a developer bring down a "test" database for reloading. Unfortunately, it was a production database and took over 8 hours to restore (backup tapes were offsite). Good arguement for separate boxes for production and development. Also a good arguement to make sure database names in development are different from production.

Test/development environment by their nature are more unstable than production. When these environments are on the same system, the instability can affect production. If your managers are like mine, they don't want to hear that development took out production. The best way to prevent this is to have the environments on separate boxes.

HTH
Marty

[Paula J Frazer-Campbell]

Hi

Test/Dev on a production server is a disaster waiting to happen.

No matter how careful to set the environmental variables and put rules in place it will one day go wrong.

I had the experience of a developer thinking - (rare occurrence) that he was on the dev server and decided to clear the data from the database so he could test in a clean environment, but the problem was he was on the live production server and he zapped our data - thank god for my mirror copy which by chance I had broken away only 5 mins prior to this.

From that day on development and test environments NEVER get anywhere near my production servers and I do not even leave them an entry in passwd.

If they require access I set it up - watch them carefully and when finished I delete that login.

Test/DEv on Production server ?

Defiantly NOT recommended !!!


Paula

[Pete Randall]

Matt,

To keep it short and sweet:

The only way I would consider merging the two would be with VPARs and even then I'm not sure.


Pete

[Matt Powell]

Thank you all for your timely responses and "war stories". I really appreciate it.

Interestingly, this particular problem - the risks to "production" environments from co-hosted "test/dev" environments - doesn't seem to be covered very well in the literature related to Change Management and/or Security best practices. It seems like a "no-brainer" in so many ways to those of us that do this work - yet it also seems somewhat under represented (IMHO) in the body of Admin/security/change control literature. The Control Objectives for Information and Related Technology, (COBIT) standards for example, "sort of" address the related problems, but only in a high-level way.

Has anyone seen references from standards organizations that basically say "don't do this...."?

Might this be grist for a good white paper...?

[Rita C Workman]

Test & Prod on same box.
NO...absolutely...NO.

Why vendors peddle this scenario is so they can peddle their wares. With no concern for the ramifications of what can (and will) happen if something in test goes awry.

Let's forget things going bad..the fact is you are taking resources away from your production system. That is plain and simple, even a totally non-tech person should be able to grasp that concept.

Now from what you mention about your shop....your spread across so many platforms and with each comes the support contracts. Want to save a few $$$...see how much you can consolidate on fewer platforms. Yes, I know there are folks out there that are probably shaking their head since I said that...but support $$$ are becoming ridiculous, and if consolidating your platforms could reduce some costs (not to mention the number of vendors trying to sell you stuff), than I say do it.

Well that's my 2 cents...and so far it got me 2-RP8400 and a 7410 (..all nicely loaded..)

Rgrds,
Rita

[John Payne_2]

You wanted answers, and you got them.

With all-in-one, how do you really 'test'? We do like Clay was saying, but we call them Prod, Stage, and R&D. The staging environment is a near the production environment as possible, and it is where we do a final test before deployment.

What do you do when you need a patch? Just patch and cross your fingers? What do you do when development accidently do something that hammers your box? How do you guarentee performance of Prod with test/dev sitting there?

The answer is that it is a touchy situation in any of these instances. If your company is not that concerned about uptime in prod, then go ahead. Otherwise, I wouldn't

Hope ithelps

John

[harry d brown jr]

Matt,

You say you work for a HEALTH Care org. And HERE IS YOUR AMMO:


If you have both PRODUCTION and development on a server - and I do not mean a physical server like a partitioned server, I'm talking about both production users and developers/test/QA/implementation sharing the same OS, is a violation of the HIPPA Act and must cease immediately!

Additionaly, NO PRODUCTION DATA can EVER be used for any development/test/QA/Implementation testing. This is a serious problem and must be rectified!

They need to fork over the money and bring their systems up to date! Or they face FELONY charges, jail time, lost business and HUGE fines!

Bottom line, you won the argument! Prepare your purchase orders!

live free or die
harry

[Paula J Frazer-Campbell]

Matt

I cannot think of any literature that specifically says, "Don???t do it".


IMHO

But the experience of the people at the frontline is worth far more than a few lines on a paper or in a book.

We have either seen or experienced first had what can and does happen if this environment exits.

This is not just your average forum - many participants are degree level educated in various disciplines Engineering, Physics (the nuclear type) and Electronic engineering and many write the HP manuals and also instruct.

Because experience and knowledge has not been written down does it make it less valid that a few line of text on a paper/ book.

Paula
BSc

[Mark Greene_1]

Matt,

I work for a healthcare company in NH, so I feel your pain. Maybe we can meet somewhere near Boston and comisserate over a drink!

Here's what various vendors have to offer:

Sun sells a test/development suite:

http://wwws.sun.com/software/products/dev_platform/home_devplat.html

Microsoft actually recommends a controlled, segregated testing environment:

http://www.microsoft.com/windows2000/techinfo/planning/client/deploy.asp

And, they provide some instruction on how to set it up:

http://www.microsoft.com/technet/treeview/default.asp?url=/technet/prodtechnol/exchange/exchange55/plan/setting.asp

Dell offers some detail on their Data Center Best Practices:

http://www.dell.com/us/en/esg/topics/power_ps3q00-mcgrath.htm

Here's a data base project plan from Standford:

http://www.stanford.edu/group/isds/dms/DBA_services6.html

note: "Identify server/instance needs for development, test and production environments"

They also mention seperate development and production servers later in the doc.

You should be able to go through your regular support contacts for IBM and HP to inquire if they have white papers on the subject. For a small fee, there are white papers and case studies you can get from Gartner as well.

HTH
mark

[Benedict Adamson]

You've had many replies saying DON'T DO it. I wouldn't want to. I guess you're thinking your managers are insane. I'd like to come at it from a different angle. Consider (humour me here) that the managers are'nt stupid (that is they can make good decisions...) but they are ignorant (... given good information).

With a recession biting, they presumably have a greatrly cut budget. Perhaps they DO believe that combining development with production will cause problems, but that the total expected cost of those problems is less than the total expected cost of having separate computers. That's the only rational management decision given limited resources, even if it is a messy one technically. They have accurate information for the expected cost of separate computers, because they already have some.

Perhaps their estimate of the expexted cost of a combined environment is inaccurate because they do not understand the likelyhood and magnitude of problems. If so, one approach might be to carefully describe to them the likelihood and magnitude of the risks.

Another aspect of this is, I'm sorry to say, to CYA. If you've expressed your concerns in writing, they can not reasonably hold you accountable for the problems. As you would be a (the?) sysadmin for the combined system, don't forget to point out that the combined system will require more emergency sysadmin, so the quality and rapidity of other sysadmin tasks will inevitably suffer. If you have an explicit commitment to a given level of service, demand that the commitment is reduced.

[Stefan Farrelly]

We seem to have found a good solution to this; run Vpars which runs 2 versions of HP-UX on the same physical server, one for Dev, one for Prod. Very handy! we use the free version so no cost either (all that means is that dev is limited to 1 cpu).

Generally we dont run dev on the same server as prod, and nor would I want to for many reasons others have already listed, but lately we have been running development databases on the production server without any problems at all. We have the odd issue with say different versions of java, but with some careful planning both can co-exist. It depends on your application really - if they need different kernel parameters or frequent reboots to test something then they cant coexist.