Methods and Views on Patching

[steven Burgess_2]

Hi all

I would like to open a question about patching.

1. How often do you patch your HPUX servers ?

2. Do you have custom bundles per server or apply one standard bundle then relevent patches per server

3. How many use the custom patch manager ?

4. What method do you use of applying your patches to multiple servers (ignite etc)

5. When you apply your patches do you investigate each individual patch to understand exactly what it's doing

6. With regard to pro active patching, how do you present this to the business

7. Who's job is it to test the patch before it goes in

Any more views and thoughts appreciated

Thanks in advance

Steve

[Steve Steel · Accepted solution]

Hi

Look at

http://www.software.hp.com/SUPPORT_PLUS/

http://www.software.hp.com/SUPPORT_PLUS/faq.html

http://www.software.hp.com/SUPPORT_PLUS/info.html


Will answer all your questions


Steve Steel

[Donald Kok]

Hi Steve,

1. I patch when there is need to. I don't fix something that ain't broken.
2. If needed, I the support CD, or specific (newer) patches from the net.
3. Do not use it.
4. Do not have much servers, so I do them one by one.
5. No.
6. Do not do pro-active patching. Sometimes patches are redrawn, I never want to have had these patches.
7. HP! Patches goes to opeational machines immediately.

Donald

[Robert-Jan Goossens]

Hi Steve,

I'm going to ansew for both HPUX and Solaris,

-1- Twice a year

-2- Custom patch bundles and relevent patches for each server.

-3- I use the custom patch manager to check for extra patches.

-4- Install them for every server.

-5- I investigate every single patch.

-6- We do not use pro active patching, we let the rest of the world test the patches, install the patch bundles 3 months after it is released.

-7- Unix sysadmin and the technical application desingers.

We start patching on our test servers ---> shadow servers ---> production servers.

Kind regards,

Robert-Jan.

[Pete Randall]

Hi Steve,

1. When needed - if there's something broken, I'll patch it. Once in a great while I may put on a Quality Pak but not very often - I've caused more problems doing regular patching than I've ever cured.

2. See #1 - if I do feel the need to patch it's pretty much specific to the server having the problem, so I guess you could call that a "custom bundle per server".

3. The first time I tried it, it came up with a list of hundreds of patches - see #1 - no way would I put on that many patches at one time.

4. I don't have enough servers to have any elaborate scheme. When a patch(es) needs to be applied globally, I walk through each machine individually. That way I can be sure that the application goes OK.

5. Yes.

6. I don't - see #1.

7. Mine, if I need to put a patch on that I'm not sure about, it goes on my sandbox, then, after a testing period, into my development environment where the programmers can beat on it, then, after a shakedown period, onto production.

You're welcome

Pete

[H.Merijn Brand (procura]

0. Also read http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x136128c64656d71190080090279cd0f9,00.html

1. Not-reboot patches as soon as they come out and *I* think it will improve our system in general (e.g. C-compiler patches & security patches). Reboot patches every three month after the ExtSW has been released and received.

2. For the non-reboot patches patches mentioned in 1., might diff per system. Standard ExtSW bundles are applied on reception ASAP on the test system, and 3 weeks later on the production machine if all goes well.

3. Depends on the complexity of the wanted patch. I get the announcements on a regular basis by mail, and if there will be a period in front of me where I think that one of those patches will improve my work (e.g. NFS performance which is still *VERY* poor between two 11.00 servers), I might schedule that in, and these might well be fetched through the patch manager due to multiple dependencies. Single (easy) patches, I fetch from ftp.

4. swinstall on every server

5. Yes. Uhh, not *all* patches. Some I apply blindly, because I don't use it anyway :) (e.g. I don't give a sh.. about tar/cpio/csh patches, since I don't use them. I use GNU tar, GNU cpio and tcsh)

6. I don't :) We have a small buisiness, and I'm the only one to decide what is patched on HP servers, when and why.

7. Mine, and mine only :(

Enjoy, have FUN! H.Merijn

[Stefan Farrelly]

1. Main production servers; only apply security patches routinely. QPK bundles every year or two. Development servers; QPK bundles whenever they come out (including diags) - to test for stability for many months before applying to live prod servers.

2. Standard bundles only, never use custom bundles (as HP havent tested how the patches in them work together whereas the standard bundles are tested by HP extensively before releasE).

3. Never use CPM for reason cited in 2.

4. Simply use SD to install them (swinstall). All servers install from a central depot server (pull method, to a local depot on each machine first, then install from local depot if it copied aok. If you install over network from a remote depot and the network drops you can completely screw your server!).

5. Only follow bundle release notes for individual patches mentioned there, dont check every patch.

6. Our business only go for proactive patching for security patches, nothing else. How you can you explain to ths business you want a lot of downtime to install a massive QPK bundle on a prod server when it has the potential to completely screw the server (if say the kernel wont build after it?) - especially when the QPK bundle is not essential (unlike security patches which are).

7. HP test the QPK bundles extensively first, we test them on dev servers for months first, and a single prod server for a few more months before rolling out, just to be safe.


7.

[John Poff]

Hi Steve,

1. We try to patch quarterly.

2. We have been getting a standard bundle per server type and patching similar boxes with that bundle. Now we are looking at building custom patch bundles for each box.

3. We don't, but I'd like to start doing it.

4. Currently we are patching them individually, but we used to have a large patch depot [until someone stole our storage!] and we are probably going to make one again.

5. I get the weekly patch notification via e-mail from the ITRC and I'll review what it soming out, but when we get a patch bundle we just go with it. We've have had great success with our patches and in five years we've only had to back out a couple of patches, so we don't worry about it.

6. We sold our business on it early and they like it.

7. We put our patches on test and development boxes first and let them run for a month. It would be nice to have a sandbox system to run them on from our end but we don't have that luxury.

JP

[BFA6]

Hi Steve,

1. Don't patch unless there is a problem
2. Usually download any required patches from the net
3. Nope
4. Generally do not patch multiple servers.
5. Usually an applied patch is a because of a call with HP
6. No active patching.
7. Testing - what's that :-)

Regards,

Hilary

[Jeff Schussele]

Hi Steve,

1) Twice a year

2) We have standard bundles *except* for our V-class - it gets a custom bundle.

3) Sort of - our contract calls for HP to bundle custom sets of patches that we - the SAs - agree upon. We call this our HP Recommended bundle & it along with the HWE & QPK bundles are what we install twice/yr.

4) Every HP SA has a certain # of systems that they are the primary SA on. It's the responsibility of designated primaries to install the patches on those servers. Unless we get the opportunity to install on multiple servers at once - rarely - we generally install them locally.

5) We follow the recall list & exclude/remove any that pop up there, but we generally trust the HWE & QPK bundles. But yes we DO investigate each on the HP Recommended bundle.

6) We tell them it's a mandatory part of normal system maintenance. The only choice we give them is - when.

7) Patches go in this order
A) Development
B) Test
C) Staging
They then "break-in" there for a while & then finally go in
D) Training & Production
Again, it's the responsibility of the primary SA to monitor the system for adverse effects of the patch install.

Later,
Jeff

[eran maor]

Hi Steva

1. if think that you need to patch your system when installing the system and then do a CPM to your system every 3 month .

2. i m advising my cu to install the last support plus bundle ( QPK and HWE ) and to do a CPM for each server .

3. all of our css and pss customer is using CPM and it is highly recommend .

4. my advise is to do a maintence windows every 3 month to install the patches .
remember to do a make_tape_recovery before the install and also to try to compile the kernel to see if there is a problem with the kernel .

5. i will check the rating of the patch .
dont install patches with rating of 1 star * .

6. installing the level of patches that is recommend by hp will help you to prevert problem and it is saving you problem in the futate .

7. it is the system admin job .

i will do a test in 1 computer to see if there is problem but if your system are diff. from each other a simple test will not help you .

remember to stop oracle , omniback and perview and mwa and to do a test to the kernel with compiling it .

most of the problem is not with the patches and are before you install your patches .

[A. Clay Stephenson]

As a general rule, I patch on a quarterly basis. For those of you in the "It ain't broke so don't fix it camp" my question is "How do you know?". If you read through the patch release notes, you will find that a fairly large fraction of them deal with "possible data corruption". The very last thing that you want is little bits and pieces of data corruption that you might not find for months.

My method is to first apply the full patchset and Hardware Enablement bundles to a sandbox and test there. Next, the patches are applied
to the Test/Development boxes; finally (usally after two or three weeks) a maintenance window is scheduled for production.

At this point, you know how long the patches will take to apply and you know that they are safe for your environment.

While it's a good tool, I rarely use the CPM. I install the entire patchsets - with very rare exceptions.

I typically use multiple copies of the install media and do multiple patches on the production boxes simultaneously.

I don't bother to read over the patches until I've installed on the Sandbox. I then know which will actually be applied and then I read over the release notes for each one and look for possible impact; that gives me a very good way to look for possible hiccups in the sandbox and test/development environments.

I've never had any real difficulty scheduling maintenance windows as long as plenty of lead time is available AND I NEVER exceed my requested time window. In order to make your case, just drag out one of the "possible data corruption" release notes and your battle is won.

It's your job to test along with the DBA's and any application support people's job as well.


I am rapidly approaching 4 years of zero unplanned production downtime so I have no intention of changing my maintenance policies. My best piece of advice is get yourself a sandbox - not the same as a test environment. A sandbox will more than pay for itself in the prevention on unplanned downtime.

[Jeff Schussele]

Clay,

Quick question:

How do you simulate load on that sandbox system?
Specifically, what do you do to insure you have an environment & workload as similar as possible to your other environments?

We've found problems with patches that did not appear until the system reached certain levels rarely touched - even in production.
To me it seems that scenario is almost impossible to duplicate outside of production.

Curiously,
Jeff

[Steven E. Protter]

When I took over my shop, the attitude was, if if it ain't broke, don't fix it.

That made our last server rollout hell, because critical patches were missing and omniback and a dozen other functions didn't work the first day we ran the new system.

I patch my servers with large patch sets quarterly and two weekends a month with critical patches for secruity or oracle functionality.

I build my own custom batches based on the task at hand.

I am moving to a new setup where I patch an Ignite server and then roll out vg00 to two other rp5450 servers. Thus far its testing all right, though I'm worried about Ignite properly pushing nsswtich.conf and the hosts files . I'll resolve that.

Its my job to test patches in a test environment, though I am allowed to draft the dba and users. I have a list of things to check, open this app, look up account, stuff like that.

For production servers, same thing, if its a reallhy big deal managment might keeep me company and test the Sunday morning after I patch.

We have adopted what I call a sane but agressive strategy. We don't patch for the sake of patching, but we do try hard to stay current, especially with security issues and Oracle.

I was once burned by putting in a pax patch that let Ignite pack up 8 Gig files. The pax_iux utility that unpacks on the Ignite client barfed and could not unpack the files. It took four weeks, including a bout with the flu to figure that out, but hey, I learned how to use tusc.

Even with that one failure, the strategy is sound and the users rarely notice when I roll a new server into production.

SEP

[steven Burgess_2]

Hi all

Thanks for the responses

A.Clay I'm interested in your reply from the question that Jeff has just put to you

I finish for the weekend shortly.

Keep them coming will pick up again on Monday

Thanks

Steve

[Marco Santerre]

1) In my case, what I try to do is patch my systems every six months with the Support Plus CD.

2) When it comes to my production servers, I do ask for a patch analysis to be done by HP at least once a year after I've just patched the system with the Support Plus CD. In this case, mostly to get more relevant patches for the specific applications, like oMniback or things like that.

3) I do use the custom patch manager, but not as much as I should unfortunately. I use mostly for information at this point rather than patching.

4) In my case, I rather use a simple depot, rather than Ignite because almost all my servers have got a little something that's different. So I will swinstall the patches but from a depot, rather than the CD.

5) I do investigate some patches, but not all of them. I'm often interested in network , and Kernel mostly to see what changes are made and see the probably impact it may have.

6) I admit this one is tough, but often enough, my development servers are often done ahead of time, therefore proving that (relatively) all patches work for one is a good thing for the business case. Another thing is if I can find certain patches that are really critical that would help as well. Unfortunately, I don't hold all the secrets to dealing with managers and they sometimes (though not too often) don't listen to me.

7) Mine

I often find, that when you have a whole slew of servers, even if you do try to move ahead with them, you'll often find yourself behind for several reason. I lok at one server in particular here that I have a problem patching cause I haven't been able to get a downtime period for it for almost a year. So I find that it's good to have standards in regards to patching but I sometimes have problems adhering to my own standards.

[Sridhar Bhaskarla]

Hi Steve,

My method is same of Jeff as we work together :-).

However,

C) Staging
They then "break-in" there for a while & then finally go in
D) Training & Production

Our staging servers are supposed to be the DR for the production. So they match production in all aspects. Before we patch production, we patch Staging servers and do loadtest for critical application. We make sure neither functionality nor the performance is impacted negatively before rolling them into production.

I am against "if it isn't broken, then dont fix it" rule for patches.

-Sri

[A. Clay Stephenson]

How do I test?

After loading the patches and going through very routine tasks like printing, starting Oracle and doing a few simple queries, starting an Oracle listener and doing additional queries. I then startup the the main applications (e.g. ERP, HR, CAD) and run automated test scripts. There are several good vendors out there. As an example, http://www.qalinks.com/

You will do a lot of work on the front-end setting up the test scripts but they more than pay for themselves. These same scripts are used whenever one of the application service packs are applied.

[Michael Tully]

Hi Steve,

1. We try to patch twice a year. More often if there are particular problems or security related patch releases. If we patched more oftern we would'nt do anything else.
2. We use the standard patch bundle.
3. Don't use custom patch manager
4. Use a CD as a patch depot. All systems are updated from these.
5. As we test a patch bundle in it's entirety, we find any problems prior to rolling out to a production server.
6. We sell it like vitamins. It is better to prevent a sickness before it occurs. I am lucky enough that the business trusts us enough to do these things, as long as they are tested thoroughly, we go through our change control mechanism and they will like when we propose or negotiate a different time.
7. We are luck enough to have quite a few development systems, we also have plenty of developers willing to try and get the systems keel over ..... Seriously this actually irons out any problems.

Cheers
Michael

[Dave Unverhau_1]

Steve,

Since you're looking for different views on patching, I'll throw one in that comes from a little different angle...

My job with HP is to provide proactive patch analysis and patching assistance to our customers with PSS support contracts. HP's goal with regard to proactive patching is to minimize the need for reactive patching and unplanned downtime (shut the barn door before you find the cows grazing in the neighbor's garden...).

I try to have my customers patch at least twice per year, with custom patch bundles that I create using a tool similar to CPM (with some careful manual oversight).

When I started doing this job a couple of years ago, I found that most of my customers had no interest in proactive patch analysis and patching, and I had to chase after them. I expect that this was due to bad patching experiences or horror stories from other users. Now, most of my customers come to me on a pretty regular basis and whenever they're experiencing a behavior that they perceive to be related to, or fixable by, a patch. They seem to be very pleased with the proactive patching process.

Granted, proactive patching *can* introduce problems, but the incidence of this is inversely proportional to the time and care spent researching and selecting the patches, and reading the special installation instructions. Almost all of the post-patching complications I have been involved with resulted from failure to follow these instructions. A few were due to patches that were later found to have a problem, but that number is very small if only patches with a reting of 2 or 3 are selected for a bundle. The only exception I make to this rule is for security patches and critical patches when I can see that the critical patch addresses an issue that the customer is experiencing.

Some of my customers have contacted me to say how pleased they were because the proactive patching process had reaped benefits for them beyond their expectations (like fixing performance problems that they didn't know they had, or plugging security holes before their management presented them with a mandate to address said security issue.

Here are some documents at docs.hp.com related to patching that you may or may not have already seen:

"Patching Mission Critical Systems":
http://docs.hp.com/hpux/onlinedocs/os/patching_mc.pdf

"Patching Usage Models" (much of the same text, but worth worth scanning, as well):
http://docs.hp.com/hpux/onlinedocs/os/patch_usage.pdf

"Patch Management Guide for HP-UX 11.x" (patching-101):
Patch Management Guide for HP-UX 11.x

I hope this has been worth your consideration!

Best Regards,

Dave

[Dave Unverhau_1]

Oops...didn't include the link for the patch management guide...sorry!

http://docs.hp.com/hpux/pdf/5967-3578.pdf

Regards,

Dave

[V. Nyga]

Hi Steve,

we have a HP-UX enviroment for CAD only, so no oracle, sendmail or anything else I read often here at the forum with patching...

1. Only when needed (Never touch a running system)
2. Both made yet - Y2K-bundle for example or HP-made bundles for graphic problems.
3. In the Internet? No. Only from HP or distributor.
4. swinstall (only one server - clients one-by-one)
5. No
6. No pro active patching (without Y2K)
7. No test possible - but I sit next to the users.

Volkmar

[Balaji N]

Hi Steve,

We maintain two sets of environment. One for development and one for testing. I dont maintain any production environments


1. Approx, once every three months.
2. We have a customized bundle based on requirements.
3. No. We dont use it.
4. Install it individually on the servers. (around 60 of them and we need to plan downtime and stuff like that. and we dont patch all servers since we maintain different releases of our s/w as well on it)
5. no. there is a separate team for this.
6. i guess i have never been pro-active (bad, but we dont maintain prod servers on the net.) normally, i recomend a patch only if face some problem and after investigation find that, it has been fixed by patch. else the patch teams take care of it.
7. the patch delivery team who gives us the bundle.

hth
-b-

[John Payne_2]

I patch every 6 months. I use the QPK's. I install the current QPK in R&D and let it burn in for 3 months.

Then we go live on the production servers.

It's my job to test the patch to make sure the patches don't trash the box. Then we get the patches on Staging machines (which I consider prod...)and test our apps (The applications guys look at their apps, and then any problems we verify was a change they made, not caused by the patches. :) ). Then we move the patches to real production.

As far as getting buy-off from management, several years ago we did not patch. We had a machine crash, and the Response Center wanted us to apply 50 patches. (Think back to how easy it was to pull down patches a few years back.) After applying the patches, the problem was fixed. We blamed the problem on our not having applied any patches. We have been applying the QPK's for about 3 years now (And it's predessessor), and I have not had a system crash due to an HPUX software error. We just do not have them anymore. This is our justification to management anytime they ask now. Also, almost without fail, when an app problem occurs and the application vendor support asks if we have patches applied, the answer is yes. It makes my job a whole lot easier, because the applications people can just go back to the vendor and tell them to 'try it again' instead of pestering us to patch a machine and then find out they still have a problem...

Hope it helps

John

[harry d brown jr]

Steve,


1. The servers I'm responsible for, the R&D ones, I patch often, especially when I reconfigure a system. What we do in production is another story, a VERY SAD story indeed! It's sad, but we rolled out 80 servers with 11i, and three months later I found that we were 300 patches behind! Or should I say WERE 300 patches behind, because I called them on it to get the machines more up-to-date.

2. No, each system has it's own personality, and therefore I use the custom patch manager.

3. I do, always. And I live on the edge, because I choose the LATEST! I have YET to have any issue with using this method!

4. Each get's their own!

5. No, I like to drink too much and that would cut into my social hour!

6. Simple: Pathces fix things that are broken.

7. Implementation


The biggest issue I have with people "selectively" choosing patches, is that they don't have a clue as to what they are selecting and not selecting. The idea behind patching should be that you are fixing BROKEN things.

I gave a list of patches to an SA the other day and they came back and said that a few of them had "warnings" associated with them, and of course I said: And those warnings are what? I sat down with them and went through each Patch Warning and showed them that those warnings don't apply in our situation.

live free or die
harry