Best Practices on OpenVMS patches on a static system

[Gary H. Mims]

Definition of static system from my perspective: We are running OpenVMS 8.31H1, Cache, and Multinet 5.3A. We are not adding additonal products--especially OpenVMS products. The variables are users that login to application software written for Cache. The Cache version is new. Essentially this is my definition of a static OpenVMS system. Our current OpenVMS system came online in Feburary of 2010. Since that time we have had problems due to Cache and Multinet. Both have been resolved. For maintenance, I reboot everything once a month.

My boss thinks that we need to be 100% up to date on all of the OpenVMS patches, regardless of their priority.

On our previous hardware and OpenVMS 7-3.2, I basically ignored all patches, except for a firmware patch that was necessary for our SAN. In essentially ignoring the OpenVMS patches for 7-3.2, we never experienced a single downtime incident due to OpenVMS. That alone attests to the stability of OpenVMS.

Back on track, we reboot our current static system once a month.

As to my thinking on any OpenVMS patches, I take the side of "if it ain't broke, let's don't try to fix it", regardless of the latest patches from HP on OpenVMS.

From my perspective, any introduction to a patch introduces an element of chaos to a static system. We don't know how it will effect Cache or Multinet. Proof of stabilty lies in the monthly reboots without crashes in between.

So what is the "Best Practice" in this environment? I say that if we upgrade Cache, or Multinet, then it is time to look at the OpenVMS patches at all levels.

But, in between those times, I believe the safest "Best Practice" is to ignore the montly OpenVMS updates, and to only apply the big patches when they are released.

I can say that with our previous hardware and version of 7-3-2, I did minimal patching, and in 6 years, we never had a single crash that was due to OpenVMS. On our new system, we are simply using newer versions of Cache and Multinet.

My response to my boss, is that in being on the leading edge of OpenVMS with patches, we expose ourself to the "bleeding edge."

To summarize, I would like to hear from other OpenVMS systems managers on Best Practices in applying patches to a static system.

[Jan van den Ende]

Gary,

(when WE still controlled things) we tried a somewhat less conservative strategem:

1 .preferably be some 6 months behind on new versions and pathces
2. actively scan (OpenVMS.org, ITRC, comp.os.vms) for any adverse effects. Skip any reported patches.
3. Install patches to a sandbox. Try to trow everything at it that may be relevant to production ( ah, but reality is a surprisefull bitch!)
4. Evaluate "critical" patches for relevance to OUR config. Skip if irrelevant; until included in UPDATE bundles. (but the relevant patches in there usually were installed before the bundle)
5. NEVER shut down - except single nodes for "rolling reboots".

Full 365 * 24 service for 10 years.

Of course, that all changed drastically when IT was outsourced, and a totally different strategem was dictated which essentially reduced us to mindless drones.

-- I no longer have any knowledge how the site fares the last couple of years.

Proost.

Have one on me.

jpe

[Ian Miller.]

I would look at using the UPDATE patches. When a new UPDATE kit appears, wait for a few weeks, download it and install on a test system. Do testing for a few months then install on production (probably about the time the next UPDATE kit is announced).
This would then be a quarterly patch regime. You could stretch this to 6 months or a year by looking at every 2nd UPDATE kit or every 3rd UPDATE kit.
Note that UPDATE kits are supposed to consist of previously released patches.
You can buy a patch service as part of your support contract with HP.

[Robert Gezelter]

Gary,

I essentially agree with Jan and Ian. Being at the "bleeding edge" is potentially a problem for a production system.

However, I would not consider a system "static". While the system may be "static" in the grand-scale sense, this is almost never true in the details. For example, consider a bug relating to disk error handling on a particular device. If your system never hits an error, then the bug is never a problem. When one encounters the error, it is a different story.

With OpenVMS, I have rarely needed a regular reboot. I have encountered applications that have memory leaks, and need restarting, but not the underlying system.

Quarterly path kits applied to a sandbox 30-days after releases, tested for two to three months on the test system is certainly one balance.

The OP does not note which platform (Alpha/Integrity) is in use. Certainly, virtual environments make creating test systems far easier.

- Bob Gezelter, http://www.rlgsc.com

[SDIH1]

I agree with the other respondents, patch conservatively.

Also consider this: most patches are published for a reason: someone else ran into a problem with the software you are running before you ran in to that problem yourself. The software is broken. By not applying patches you choose not to fix what is broke.

I do not understand the monthly reboot at all, unless you have found a specific issue and found no other way to solve it than by a reboot.

[SDIH1]

Something else: by most people it's considered bad practice to combine changes that have no strict dependency. You said you would consider VMS patches at the time an application update is due. If this means that you would install VMS patches at the same time as the application updates, then you would really complicate finding the root cause of any problems arising after that: it could be VMS, it could be the application.

[Hoff]

What are your production downtime costs?

As for your question...

You have at least two choices here.

+ Do what your boss wants. Yes, this is an approach that is not without risks. This approach can also be somewhat more expensive (without any patch-related problems), as (if you don't "batch" the patches together and apply them in bigger granules) the usual pre- and post-upgrade data archives do lengthen your production downtime.

+ Follow a more conservative approach and to patch on a less frequent schedule (eg: quarterly) in the production environment (obviously barring specific process-critical errors) and to not patch in any "new" patches.

Monthly reboots can be an entirely reasonable practice in some production environments, and particularly if you're clustered. These reboots allow you to test the resumption of operations, and can allow you to roll in upgrades and patches (the so-called rolling upgrade), and to get (good, complete, consistent) copies of otherwise active disks.

As for best-practices, get copies of your data before and after the patch application. This can be via BACKUP or via splitting out (quiescent) shadowset member volumes, etc. This is your recovery path if Bad happens. The before-patch archive is the rollback path; your exit strategy should a patch go wrong. The after-patch archive is to get a consistent archive for DT purposes.

And get a test environment available before deploying patches into a production environment, too.

No, I don't trust any software vendor to always work. Not for a production application, and specifically not for life-critical applications, nor various classes of manufacturing applications.

This decision is the boss's call regardless, and s/he may have good reason(s) for asking for this fast-patch procedure.

[Hoff]

Apropos of patches:

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1444509

[Gary H. Mims]

Thanks for all of the responses, they have been helpful.

We have a single new Itanium server which replaced our Alpha ES-47. So I don't have a test environmet to work with.

To clarify one point: I would never mix patches and new software at the same time. Our current standing on OpenVMS is a quaterly patch, and two others--which unpon reading the details, will not touch anything we use. Each patch requires a reboot. None of these patches are critical.

On the other hand, I have two Multinet patches which are important, but not critical. Our IT Security group is messing with FTP's and somehow the patches got corrupted. I missed my maintenance window in trying to get them ready.

I want the Multinet patches to exist in the system for a while before applying the OpenVMS patches.

Based on your great responses in the thread, the consensus is that you would only recommend applying the quarterly patches, and at the least try to track down the importance of the two other patches.

Thanks,

Gary

[Hoff]

For the basic OS testing stuff, you can snag a used rx2600-class box and some disks and a single- or dual-core VMS license for around US$2000.

The hardware itself regularly shows up used for US$300 to US$500, plus some parts, or somewhat more than that if you want a vendor that will support the stuff and answer your calls.

When last I checked, the base FOE license was around US$900 per core, and (for testing) you don't really need both cores.

I haven't checked the OpenVMS I64 base license prices again since the base license got renamed BOE.

If you're running critical production and are concerned around maintaining uptime, do yourself a favor and ask for a target-practice box.