Post patch bundle install testing

[AndyMueller]

All, I am curious to know what you guys test after completing an HP-UX patch bundle install? Do any of you have a "Test Plan" that you complete? We generally test ServiceGuard Failover, Ignite image creation, ssh, bastille, EMS & OVO functionality. Anything else we should be checking?

Thanks, Andy

[Ivan Krastev]

It depends from the server usage. At least syslog/rclog and applications log should be examined after the upgrade.

regards,
ivan

[Steven E. Protter · Accepted solution]

Shalom,

I have a written test plan of important server functionality.

This includes testing procedures, and expected results.

After a major patch install, I execute the plan and roll back if major functionality is missing.

Prior to any patch, I back up the vg00 boot volume group with make_tape_recovery or make_net_recovery

SEP

[AndyMueller]

Steven,

I know that depending on the server's function (application) test plans may differ. Can you drill down in little more detail, what specifically you test? (especially OS level related).
Thanks, Andy

[Steven E. Protter]

Since I love earning points, an example.

Lets say I have an oracle app.

Tests:
ps -ef | grep instance name
This insures oracle processes are up.

I run a script that reads data from oracle tables to make sure the tables are actually online.

I test any oracle apps using a test account, which I fight the programmers to have to insure the app is up.

I do an ssh login to make sure that service is up since updates tend to "bork" it.

Expected results vary with the test.

I don't test everything, but I pick enough tests to insure that critical systems are up when I head out of the office.

I'd like to know how you test bastille, the individual apps?

If there is a web server, I browse it to make sure it works.

SEP

[Bob E Campbell]

While this is not exactly a test, to me the key is having an IUX recovery image and/or a DRD clone of the original environment.

While it does not increase confidence it does limit the risks to a defined level.

Beyond that if multiple systems are involved I would look at a rolling update that brought the new environment into production in stages with the most important systems coming online last.

[UVK]

Andy,

Apart from the regular checks you mentioned. I also moniter the system performance after any sort of patching(as i have bad experience when bundle invloves any PHKL's) to see any unusual CPU, io, mem usage. And enguage the relevant apps teams to have them test their apps and take a sign off.

Cheers,
uvk

[AndyMueller]

Steven, for Bastille, I will un-bastille the server, then run bastille -b to read the current configuration back in. I make sure that that services we don't allow (ftp & telnet for instance) are still disabled.

Thanks for some of the items you had listed. Given by the limited responses, it appears that many may not conduct extensive "formal" testing, or don't follow a stringent test plan. I am developing this for my organization and was hoping for a bit more feedback to add to my procedures.

Andy

[AndyMueller]

UVK, excellent points, thank you!

[AndyMueller]

Thank you Bob, we are doing precisely that. Any formal testing you conduct AFTER the patch bundle has been installed?

Andy

[Bob E Campbell]

I come from the world that creates the bundle. For us the primary concern is if the installation proceeded as expected.

For that I recommend:

1. Run of swverify(1M) before and after

2. Review of messages in swagent.log

For those of you with applications it comes down to the investment you have made to certify your environment. Hands on testing is expensive and does not reproduce well.

If you are interested in the system level aspects I might recommend running the standard performance benchmarks or "standards" tests before and after. If HP is doing our job these should never have an issue. See http://www.opengroup.org/testing/testsuites/ for starters.

What I really recommend is the testing we cannot do. In an ideal world I would have a simulated production load so that my applications themselves are the test. That takes an investment...

[Bob E Campbell]

Oh, one more thought...

Over the course of time as failures are encountered a test that recreates the problem should be generated. The most likely failures are the ones you have already seen. If you can only have one test, this would be my vote for the best return on investment.

[James R. Ferguson]

Hi Andy:

I'd begin by agreeing with Bob: make sure you have a current Ignite or DRD image in the event all goes poorly.

Depending upon how you patch, for example whether or not you are applying an individual patch or two to correct a specific issue or whether or not you are performing a "routine" update will determine what you test and what your back-out plan may be.

In some cases, you can 'swremove' a patch. In other cases, an Ignite or DRD recover may be necessary (e.g. kernel patches that cause you problems).

If you apply the standard patch bundles (either from CD/DVD media, via a download, or (best) via SWA), then you have the highest probablity that everything will work. The standard HP-UX bundles offered every 6-months, comprise the most generally tested set for most environments.

Reading the patch notes (for whatever you intend to install) _before_ patching avoids surprises. This applies particularly to any patches with special instructions!

Rebooting your server _before_ applying a large set of patches may eliminate surprises or at least un-cover any issues _not_ related to the patch session.

Running 'swverify \*' or the 'check_patches' script _before_ a patch installation _and_ paying stict attention to the output is excellent insurance against surprises and/or problems.

If at all possible, deploy patches or patch bundles on a test server before doing the same in production.

Regards!

...JRF...

[Steven E. Protter]

Shalom,

I definitely endorse the concept of having a test server that mirrors production and do Quality Assurance there before production.

That does not negate the need for a test plan prior to deployment in production.

All of these are important steps and Quality Assurance is good for all concerned, including system administrators.

Involving the users in Quality Assurance makes the process more transparent and produces higher quality results. It makes them stakeholders.

SEP