Seeking arguments against "Production" and "Test/Dev" environments on same server

[Matt Powell]

Greetings,

This is not a 'technical' question particularly, however the answer(s) may involve technical issues and a robust, readily available set of answers might be helpful to this community.

Presently I am a UNIX admin in a truely heterogeneous large "non-profit" healthcare IS environment. HPUX, AIX, OpenVMS, Tru64, AS/400, Linux, and of course NT/2000/Novell.

Many of our more complex, enterprise scale applications require test/development environments as part of the ongoing support of the application over the product life-cycle.

In some cases we have dedicated test/dev servers. Increasingly however, turnkey solutions providers are peddling to the non-technical decision makers, the notion of hosting the "test" and "production" environments of their products on the same servers.

It is my opinion that it is generally poor practice to have test/development environments co-exist with critical production enviroments on production servers. Although there are exceptions to every rule, I maintain that from the standpoint of system integrity, the risks generally far outweigh the perceived benefits.

My position on this topic has resulted in numerous heated debates with non-technical IS management.

I would like to solicite feedback from this group on this topic.

Are there well-reasoned arguments for/against this practice? Is anyone aware of 'best practice' standards ( SAGE, ISO, ISSA, IEEE, etc) that address this particular issue? Are there whitepapers, articles, or other documents people are aware of that address this issue?

I have been somewhat unsuccessful finding existing literature on this, so our discussion here might be particularly valuable.

Thank you,

Matt Powell
UNIX System Admin
Providence Health System
Matt.Powell@providence.org

[M_10]

As your non-technical managers to sign-off on a document which permits the testing of software on a production machine. This individual(s) will be ultimately responsible for any liabilities and down times incurred. Everyone, in every industry (aerospace, car racing, medicine, etc.) uses prototyping, testing, etc. on non-production/non-critical environments. Test racing cars are used to evaluate changes.

Any changes, no matter how small should be tested before being implemented into the production environment. Many times, there may not be an adverse reaction, but those few times that things blow up - it's nice to know that only you were brought down, and not other clients.

My 2 cents worth.

[Jasti Kalyan Babu]

I encouter this once in a while.
I usually follow the methodology described in the earlier reply. Ask for a signoff overriding your recommendations.

[Catherine Benoit]

The field of business continuity is generally one that is misunderstood by many managers, and accordingly it is no surprise that some would suggest using a production machine for software/hardware testing.

People that make such decisions are irresponsible if only for the reason that they gamble the company money and precious time.

Justifications for not using production equipment have to go through financial considerations. For example, put a price tag on work lost and time lost if a particular machine is down. Etc... and compare against the cost of the test machine.

You???ll also be reluctant to fully test an application on a production unit. Hence an incomplete test.

Catherine

[Martin Johnson]

Not only should test/development environments be on different servers from production, the database/file names used in these environments should be different. A couple of years ago, a developer decided to delete a development database. Unfortunately, 1) he was on a production system and 2) the database names for development and production were the same. The result was a production database being deleted. It took 8 hours to restore the database - 4 hours was waiting for the backup tapes to be returned from the vault.

By their very nature, test/development environments are more unstable than production. Unstable environments have more downtime. They can take a system down more frequently. How would your managers like to explain to their customers that the systems were unavailable because of a bug in some test code?

And then there are the times when a production problem is found. Do you really want to debug a production problem that crashes the system in development when development is on the same system as production????

Ideally, your development system should be an exact duplicate of production. In reality, that is seldom the case. Unfortunately, that means you won't be able to test for all problems in development. Recently, we discovered a memory problem in one of our applications in production. It was not discovered because the development system only has 2 GBs of memory, while the production system has 4 GBs. The problem could only be replicated on a 4 GB machine.

HTH
Marty

[Chuck Ciesinski]

Matt,

In our manufacturing enviroment, we would NEVER consolidate the development and production environments on the same server for primary business systems. We have done that for legacy applications which we made the decision not to migrate or convert to another platform. As the others have said, it's just to risky! Think of when you need to do an upgrade to your O/S. Does a technical manager say just do it. NO. He wants it tested, same with patches. What about ORACLE or some other database? Same concept. Test with key customers that have a sense of what they are doing before you let the real code breakers have at it. We have seven servers dedicated to SAP.
1 database server, 3 application servers, 1 test box, 1 playbox (sandbox), and 1 knowledge base server. This does not include our HA (ServiceGuard) systems.

My $.04 for inflation,

Chuck Ciesinski