Opinions on HP patch scheme?

[Richard Hubbell]

I was curious to hear how others feel about the patch mechanism
used by HP? I find it to be very convoluted, error prone, and archaic.

There are so many combinations and dependencies and patch recalls,
etc., etc. Do others feel this way? Or have you been dealing with
the patch scheme for so long that you have just learned to live
with all its shortcomings?

[Mark Mitchell]

I stay away from patches as much as I can. I know people who load the bundles as soon as they come out (shotgun approach) and fight to fix problems every time. I wait until I have a problem which is a little more rare now that I am on 11.0. And then evaluate the patch by going to the website after printing my swlist and draw a tree diagram starting with what I need and move back from there. I have gone through this process in the past to find out that the recommended patch didn't apply to my server, or fixed something I didn't have.

This site has a section under the patch database main, then patch database, then search by patch ID. A better interactive screen comes up with the patch and dependencies. This is nice because you don't have to weed through the patches themselves to find dependency information. After I have it down, I go to ftp.hp.com and get the patch by saving to target rather than cut and paste where mistakes can occur. If there are more than one patch and more than one requires a reboot, I make a patch depot to do a quicker install. Here are the details someone wrote up way back when

.HP patch Depot
1. Obtain the set of patches you want to install in your depot.

For example:

[PHCO_7891/PACHRDME/English] , [PHCO_9348/PACHRDME/English] ,
[PHKL_9361/PACHRDME/English] , [PHSS_7726/PACHRDME/English] ,
[PHSS_8966/PACHRDME/English] , [PHSS_9400/PACHRDME/English] ,
[PHCO_8353/PACHRDME/English] , [PHKL_8376/PACHRDME/English] ,
[PHKL_9569/PACHRDME/English] , [PHSS_8667/PACHRDME/English] ,
[PHSS_9201/PACHRDME/English]

HP patches as delivered by the HP Response Center or by HP web sites are
shar format files consisting of a product depot and README file.

2. Unshar the patches:

# for i in PH*
do
sh $i
done

3. Combine all these separate depots into one depot. To do this, use the
swcopy command. First, create the directory to store the patches:

# mkdir /tmp/patches

4. Now take the patch depots and copy them into the target depot:

# for i in PH*depot
do
swcopy -s ${PWD}/$i * @ /tmp/patches
done

5. Verify the contents of the depot:

# swlist -d @ /tmp/patches

Anyway, that sums up my feeling on it and some good procedures.

[Anthony Goonetilleke]

Yes I tend to agree.. I am in the proccess of preparing a patch document at the moment and basically I have come to the following conclusions:
- Only install patches from the support plus CD (less chance of being recalled)
- Only install critical/corruption/hardware patches unless needed by applications
- Use the patch manager to keep an eye one your systems this is a good indication of how many patches your system is behind.

A word of caution though never install all the patches the patch manager reccommends as this will cause problems on your system there are still some issues with it that need to be looked at IMHO

[Rick Garland]

There are many different opinions as to how to manage the patching of a system. The patch bundles are only OK but they do tend to cause additional problems that what you might be facing. The patches on the Support CD are often the better patches as these have been tested for a period of time and are better suited to specific problems. HP will tell you to load the bundles every quarter when they come out but if you want to take that approach, start on the development machines before you load on production servers. In another life, we would put the patch bundles on development systems for 3 months before they were even considered to go on production. In this manner, the patches on production servers were 1 behind. But if there was a problem (and there often was) it was development that had the problem and not production.

Do not load all patches from the bundles. Chances are, you do not have all the services that the bundles apply to and thus you would be loading patches that you would not need, thus taking disk space.

At present, patches are installed primarily on an as needed basis with research into the patches for the specific services that we are using.

[Stefan Farrelly]

Im a bit astounded by some of the replies about this by others.

HP did a big survey last year sometime, over 80% of HP's customers use the quarterly General Release patch bundles. Those who dont are in the minority and ive yet to see a good argument from any of them as to why they dont use them ??
HP rigorously tests the GR bundles, and then they get tested for 3 months on HP's internal projects, and feedback give, before they release them to any external customers so theres tons of testing done. OK, they did have a problem with the GR bundle from Dec 99 which caused systems not to come up after installation but this was tracked down to human error at HP, and systems have now been put in place to prevent this recurring (partly due to pressure of bi-monthyl bundles in 99, now back to quarterly bundles to allow more time for testing before release).
Almost every site ive worked at use the GR bundles, those that dont tend to have more problems than those who do (more time sorting out individual patches, new releases of them, shoud I install that patch or not? etc.). Let HP do what theyre good at, just install the GR bundles when they come out, but follow standard procedure, install it 1 at a time on your server, least important server first, just lessen any possible risk. Standard admin procedure, whatever your doing.

As for HP's patch management, its far better than Suns or IBM's or even Dec's. There was a long post about this a while ago and those who use multiple flavours agree HP's patch management is the best.

[Duncan Edmonstone]

I have to agree with Stefan here, we have been applying the GR patches to our systems regularly for three years now without any issues. Nobody is forcing you to apply the patches as soon as you receive them after all. What we do is have a policy of not installing any patch which is less than six weeks old, that way we can have confidence that all those baffling early adopter bleeding edge types will have rooted out any bad patches before we get to them.

In my experience most of the reluctance people have to do patching is brought on by bad experiences during the actual install process, which can be avoided with a little preparation (checking that the IPD is clean, and that all patches are configured before commencing; running swinstall -p to check for disk space issues; taking an Ignite backup etc.)

And also carrying out regular patching helps avoid these (all too familiar) conversations with the response centre:

CUS: I'm having problems with XYZ command
HP: Oh, do you have the latest patch 12345 for that command?
CUS: No...
HP: Well install that patch - it will fix your problem - let me just check the dependencies for you... You'll also need to install the following patch dependencies 12346, 12347, 12348, 12349 etc. etc.

And you may as well have installed the GR bundle anyway!

[James R. Ferguson]

Richard:

I side with Stefan & Duncan. I use the GR bundles (usually once or twice a year). I start with my least critcal server, load the patches; run for a week or two; and continue upgrading my other servers week-to-week.

My feeling, too, is that the GR bundles represent well-tested patch colletions. I don't particularly like to dissect patch interdependencies and feel that you run a greater risk of non interoperability among patches by picking and choosing.

If you take an extreme approach where you never load patches until you think you absolutely need to, then you will definitely run into the "...so do you have patch-X installed?..." when you least have the time (read 'downtime') to get the patch installed to enable the business need in the first place.

Another way to look at this is that it "pays" HP to maintain the highest levels of quality assurance in their products. If they didn't endeavor to do a good job (and I think they do) then as customers, we might as well look elsewhere. By using the pre-tested GR bundles, I am letting HP do some of the work that I pay for with my support contracts.

...JRF...

[Richard Hubbell]

The forums format makes it hard to reply to messages, so don't blame me!
:^)

-------------------original message

Stefan Farrelly

September 10, 2000 08:31 AM GMT [ N/A ]



Im a bit astounded by some of the replies about this
by others.

Astounded? You have been doing it too long then.


HP did a big survey last year sometime, over 80%
of HP's customers use the quarterly General

Is that 80% of the customers that responded to the survey?



Release patch bundles. Those who dont are in the
minority and ive yet to see a good argument from
any of them as to why they dont use them ??
HP rigorously tests the GR bundles, and then they
get tested for 3 months on HP's internal projects,
and feedback give, before they release them to any
external customers so theres tons of testing done.
OK, they did have a problem with the GR bundle
from Dec 99 which caused systems not to come up
after installation but this was tracked down to
human error at HP, and systems have now been put

What other kinds of errors are we talking about here? If not human
are patches machine generated?

in place to prevent this recurring (partly due to
pressure of bi-monthyl bundles in 99, now back to
quarterly bundles to allow more time for testing
before release).
Almost every site ive worked at use the GR
bundles, those that dont tend to have more
problems than those who do (more time sorting out
individual patches, new releases of them, shoud I
install that patch or not? etc.). Let HP do what
theyre good at, just install the GR bundles when
they come out, but follow standard procedure,
install it 1 at a time on your server, least important
server first, just lessen any possible risk. Standard
admin procedure, whatever your doing.

As for HP's patch management, its far better than
Suns or IBM's or even Dec's. There was a long post
about this a while ago and those who use multiple
flavours agree HP's patch management is the best.

Not better than sgi and just because you think it's better than
some other platform doesn't mean that it's good. It leaves far too much
room for introducing big problems.

The custom patch manager should be taken offline immediately.
Unfortunately it is presented under the guise
of something that actually works.
The developers of the CPM just don't have their eye on the target.

[Richard Hubbell]

---original message-----

Duncan
Edmonstone
September 10, 2000 09:15 AM GMT


I have to agree with Stefan here, we have been

Guess you've been at it too long also.

applying the GR patches to our systems regularly
for three years now without any issues. Nobody is

I doubt this but ... what're you're systems used for?


forcing you to apply the patches as soon as you
receive them after all. What we do is have a policy
of not installing any patch which is less than six
weeks old, that way we can have confidence that all
those crazy early adopter bleeding edge types will
have rooted out any bad patches before we get to
them.


Interesting. "crazy early adopter bleeding edge types" are those the same as
those users that have critical issues that are looking for fixes but can't find them
types?


In my experience most of the reluctance people
have to do patching is brought on by bad
experiences during the actual install process, which

Why should people have to do so much work that computers are so well
suited for? Because the entire patch scheme is a kludge and is so complicated
that HP can not figure out how to make it work without human intervention.

can be avoided with a little preparation (checking
that the IPD is clean, and that all patches are
configured before commencing; running swinstall -p
to check for disk space issues; taking an Ignite
backup etc.)

And also carrying out regular patching helps avoid
these (all too familiar) conversations with the
response centre:

CUS: I'm having problems with XYZ command
HP: Oh, do you have the latest patch 12345 for that
command?
CUS: No...
HP: Well install that patch - it will fix your problem -
let me just check the dependencies for you... You'll
also need to install the following patch
dependencies 12346, 12347, 12348, 12349 etc. etc.

Yeah, it's the customer's fault! That's a gem.


And you may as well have installed the GR bundle
anyway! ;)

[Mark Mitchell]

I think that my point has been made before but I can put it more simply. Don't blast a functional system with patchs. If there isn't a need for them they you don't have to suffer the down time like all of the folks in DEC 99. I have never had a problem loading patches in the past, I just don't see the logical need for it.

[Duncan Edmonstone]

My new comments marked **
---original message-----

I have to agree with Stefan here, we have been

Guess you've been at it too long also.

** Richard, I'm sure you don't mean that having some experience makes me unable to comment fairly? What do you mean ?

applying the GR patches to our systems regularly
for three years now without any issues. Nobody is

I doubt this but ... what're you're systems used for?

**A range of uses - mostly database servers, with some application and some systems management functions. If your question was asking whether the systems are used in a 'Production' environment - the answer is yes.


forcing you to apply the patches as soon as you
receive them after all. What we do is have a policy
of not installing any patch which is less than six
weeks old, that way we can have confidence that all
those baffling early adopter bleeding edge types will
have rooted out any bad patches before we get to
them.


Interesting. "baffling early adopter bleeding edge types" are those the same as
those users that have critical issues that are looking for fixes but can't find them
types?

** No! obviously I won't attempt to inject any humour into my posts from now on! It's just a fact that a lot of people load patch CDs as soon as they receive them - I'm not trying to be rude to them or any other folk - I guess there must be all kinds of valid reasons for doing this.

In my experience most of the reluctance people
have to do patching is brought on by bad
experiences during the actual install process, which

Why should people have to do so much work that computers are so well
suited for? Because the entire patch scheme is a kludge and is so complicated
that HP can not figure out how to make it work without human intervention.

** This seems like an argument for doing away with sysadmins altogether! After all isn't nearly everything we do something that ultimately the computer 'should' be able to do without human intervention.

can be avoided with a little preparation (checking
that the IPD is clean, and that all patches are
configured before commencing; running swinstall -p
to check for disk space issues; taking an Ignite
backup etc.)

And also carrying out regular patching helps avoid
these (all too familiar) conversations with the
response centre:

CUS: I'm having problems with XYZ command
HP: Oh, do you have the latest patch 12345 for that
command?
CUS: No...
HP: Well install that patch - it will fix your problem -
let me just check the dependencies for you... You'll
also need to install the following patch
dependencies 12346, 12347, 12348, 12349 etc. etc.

Yeah, it's the customer's fault! That's a gem.

** Huh? No - it's still HPs fault for having such a COMPLEX patching mechanism - I never said it was a great system - I merely said that in my opinion it was worthwhile patching regularly.

And you may as well have installed the GR bundle
anyway!

**I've just re-read your intitial post Richard, you were looking for opinions of the patching mechanism weren't you? I was just giving mine. I guess I must be one of the people who according to your initial post has "just learned to live with all its shortcomings".