HPE Community read-only access December 15, 2018
This is a maintenance upgrade. You will be able to read articles and posts, but not post or reply.
Hours:
Dec 15, 4:00 am to 10:00 am UTC
Dec 14, 10:00 pm CST to Dec 15, 4:00 am CST
Dec 14, 8:00 pm PST to Dec 15, 2:00 am PST
System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

Post patch bundle install testing

 
SOLVED
Go to solution
AndyMueller
Frequent Advisor

Post patch bundle install testing

All, I am curious to know what you guys test after completing an HP-UX patch bundle install? Do any of you have a "Test Plan" that you complete? We generally test ServiceGuard Failover, Ignite image creation, ssh, bastille, EMS & OVO functionality. Anything else we should be checking?

Thanks, Andy
13 REPLIES
Ivan Krastev
Honored Contributor

Re: Post patch bundle install testing

It depends from the server usage. At least syslog/rclog and applications log should be examined after the upgrade.

regards,
ivan
Steven E. Protter
Exalted Contributor
Solution

Re: Post patch bundle install testing

Shalom,

I have a written test plan of important server functionality.

This includes testing procedures, and expected results.

After a major patch install, I execute the plan and roll back if major functionality is missing.

Prior to any patch, I back up the vg00 boot volume group with make_tape_recovery or make_net_recovery

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
AndyMueller
Frequent Advisor

Re: Post patch bundle install testing

Steven,

I know that depending on the server's function (application) test plans may differ. Can you drill down in little more detail, what specifically you test? (especially OS level related).
Thanks, Andy
Steven E. Protter
Exalted Contributor

Re: Post patch bundle install testing

Since I love earning points, an example.

Lets say I have an oracle app.

Tests:
ps -ef | grep instance name
This insures oracle processes are up.

I run a script that reads data from oracle tables to make sure the tables are actually online.

I test any oracle apps using a test account, which I fight the programmers to have to insure the app is up.

I do an ssh login to make sure that service is up since updates tend to "bork" it.

Expected results vary with the test.

I don't test everything, but I pick enough tests to insure that critical systems are up when I head out of the office.

I'd like to know how you test bastille, the individual apps?

If there is a web server, I browse it to make sure it works.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Bob E Campbell
Honored Contributor

Re: Post patch bundle install testing

While this is not exactly a test, to me the key is having an IUX recovery image and/or a DRD clone of the original environment.

While it does not increase confidence it does limit the risks to a defined level.

Beyond that if multiple systems are involved I would look at a rolling update that brought the new environment into production in stages with the most important systems coming online last.
UVK
Trusted Contributor

Re: Post patch bundle install testing

Andy,

Apart from the regular checks you mentioned. I also moniter the system performance after any sort of patching(as i have bad experience when bundle invloves any PHKL's) to see any unusual CPU, io, mem usage. And enguage the relevant apps teams to have them test their apps and take a sign off.

Cheers,
uvk
-------------------------------------------
Like it or worked !! Click kudos !!
AndyMueller
Frequent Advisor

Re: Post patch bundle install testing

Steven, for Bastille, I will un-bastille the server, then run bastille -b to read the current configuration back in. I make sure that that services we don't allow (ftp & telnet for instance) are still disabled.

Thanks for some of the items you had listed. Given by the limited responses, it appears that many may not conduct extensive "formal" testing, or don't follow a stringent test plan. I am developing this for my organization and was hoping for a bit more feedback to add to my procedures.

Andy
AndyMueller
Frequent Advisor

Re: Post patch bundle install testing

UVK, excellent points, thank you!
AndyMueller
Frequent Advisor

Re: Post patch bundle install testing

Thank you Bob, we are doing precisely that. Any formal testing you conduct AFTER the patch bundle has been installed?

Andy
Bob E Campbell
Honored Contributor

Re: Post patch bundle install testing

I come from the world that creates the bundle. For us the primary concern is if the installation proceeded as expected.

For that I recommend:

1. Run of swverify(1M) before and after

2. Review of messages in swagent.log

For those of you with applications it comes down to the investment you have made to certify your environment. Hands on testing is expensive and does not reproduce well.

If you are interested in the system level aspects I might recommend running the standard performance benchmarks or "standards" tests before and after. If HP is doing our job these should never have an issue. See http://www.opengroup.org/testing/testsuites/ for starters.

What I really recommend is the testing we cannot do. In an ideal world I would have a simulated production load so that my applications themselves are the test. That takes an investment...
Bob E Campbell
Honored Contributor

Re: Post patch bundle install testing

Oh, one more thought...

Over the course of time as failures are encountered a test that recreates the problem should be generated. The most likely failures are the ones you have already seen. If you can only have one test, this would be my vote for the best return on investment.
James R. Ferguson
Acclaimed Contributor

Re: Post patch bundle install testing

Hi Andy:

I'd begin by agreeing with Bob: make sure you have a current Ignite or DRD image in the event all goes poorly.

Depending upon how you patch, for example whether or not you are applying an individual patch or two to correct a specific issue or whether or not you are performing a "routine" update will determine what you test and what your back-out plan may be.

In some cases, you can 'swremove' a patch. In other cases, an Ignite or DRD recover may be necessary (e.g. kernel patches that cause you problems).

If you apply the standard patch bundles (either from CD/DVD media, via a download, or (best) via SWA), then you have the highest probablity that everything will work. The standard HP-UX bundles offered every 6-months, comprise the most generally tested set for most environments.

Reading the patch notes (for whatever you intend to install) _before_ patching avoids surprises. This applies particularly to any patches with special instructions!

Rebooting your server _before_ applying a large set of patches may eliminate surprises or at least un-cover any issues _not_ related to the patch session.

Running 'swverify \*' or the 'check_patches' script _before_ a patch installation _and_ paying stict attention to the output is excellent insurance against surprises and/or problems.

If at all possible, deploy patches or patch bundles on a test server before doing the same in production.

Regards!

...JRF...
Steven E. Protter
Exalted Contributor

Re: Post patch bundle install testing

Shalom,

I definitely endorse the concept of having a test server that mirrors production and do Quality Assurance there before production.

That does not negate the need for a test plan prior to deployment in production.

All of these are important steps and Quality Assurance is good for all concerned, including system administrators.

Involving the users in Quality Assurance makes the process more transparent and produces higher quality results. It makes them stakeholders.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com