Patch installations

[Wim Van den Wyngaert]

Currently I refuse to install VMS patches without requalifying the applications.

Has anyone had problems with VMS or with (middleware) applications after applying patches on their system ?

Only answer if you had problems ... and only for VMS 6.2 and higher.

Wim

[labadie_1]

Yes of course :-)

The last time, I upgraded to Vms 7.3-2, and after applying SYS V 0100, the boot hanged.

Various boot conversational, startup_p1 to MIN and set/startup=opa0: to no avail

I booted to CD, renamed the culprit file and all was OK.

as they say

After installation of the VMS732_SYS-V0100 ECO
kit, subsequent reboot attempts fail after
SYSINT has created the STARTUP process and
LOGINOUT begins to run. The failure message
is:

Error opening primary input file SYS$INPUT,
Error in device name or inappropriate device type

[Uwe Zessin]

Myself, I didn't have any problems as I no longer work as a system manager, but I will respond anyway.

I have seen reports about patches (SYS, for example) that did break TCPIP(SMTP) or Pathworks/ Advanced Server or, (GULP?!) created an unbootable system.

[labadie_1]

ideally, one should have a dev node, and several test nodes (alpha test, beta test...) and of course similar computers (cpu/model...).

If your test node is an alpha station 500 and the production node a GS 160/320/1280, you may miss some points, because on the test node you can't "stress" enough the application.

But often CFO will refuse to have a similar node...

[Jan van den Ende]

Wim,

after opgrading 7.3-1 to 7.3-2 (latest patches as of mid-july) we had trouble with an application that uses some Progress communications routines (I know, no longer supported but... we depend havily on them, replacement due 2001, but not yet qualified) ) After re-linking the application became available, but didn't communicate with another app, as intended. (which in itself seemed to run OK, and communicates with app #3). Relinking app #2 repaires the communication with #1 (now both working fine, also tohether) but.. no more communication with #3. At this moment I am trying very hard to get the objects or the sources, or have the supplier do a 7.3-2 link (though I believe they are still at 7.1-some).
Until this is satisfactorily solved, we have to remain a mixed-version cluster...


Jan

[John Abbott_2]

I've had a problem with an old basic application which resulted in some unusual screen formatting problems after upgrading vms. There was an "issue" with the Basic RTL (DEC$BASRTL) it's hard to remember exactly what, as it was a long time ago - under vms v6.n. DEC fixed it (we were able to revert back to the previous image until a workaround was in place).

Never any issues with COBOL after vms upgrades, if that helps.

We have had some problems when upgrading pathworks. The patch applies ok but as a BDC we've seen issues communicating with a W2K PDC. In the past we've had to save the shares etc. re-configure pathworks into it's own domain as a PDC and re-introduce it to the W2K PDC as a BDC and re-apply the shares etc. This was logged with support, so my guess is that it *might* be fixed.

I can only add, just double check the patches you download have not been withdrwan or superseded by the time you apply them, we hit this once and had to restore (the system wouldn't reboot) and download the replacement patch.

[John Abbott_2]

Sorry, one more point worth mentioning. Note the order you apply the patches in test and at what point you reboot and stick to this when you roll it out.

Try not to apply too many patches together without rebooting. I've seen someone else apply a whole bunch of patches that resulted in the system being unbootable. When he applied them individually and rebooted there were no problems. I'm sure this was just back luck :-) and I think the bottom line is to go for the UPDATE* kits where possible and reboot each time.

[Uwe Zessin]

Patches that DO REALLY require a boot give out an extra hint and it is good practice to follow them. When it comes to patch order I try to install early issued ones first. They are supposed to have generation numbers with their files and I sometime see that a file is retained from an earlier patch, but OpenVMS has become a terribly complicated beast and one never knows...

[Uwe Zessin]

Oh, my. I am afraid you need to extend your refusal to applications as well:
-----
HP AXPVMS V73_MGMTAGENTS V3.0-36: HP Management Web Server

A current version of TCPIP was found. If you have not enabled SNMP, please
enable it now.

Insert the following lines in SYS$MANAGER:SYSHUTDWN.COM:
@pcsi$destination:[sysupd]wbem_check.com
$ show log pcsi$destination
%SHOW-S-NOTRAN, no translation for logical name PCSI$DESTINATION
$

-----
Of course I can find out where this procedure is really located (SYS$UPDATE:), but it isn't very encouraging if even so simple things are wrong.

[Volker Halle]

Uwe,

yes, I've already reported this problem with V73_MGMTAGENTS V3.0-36 and it will be fixed in a 'future version'.

The WBEM shutdown (WBEM$SHUTDOWN.COM) also won't work if it is not run from SYSTEM (or the account from which WBEM had been started) - also reported.

Some of these 'products' offer mail addresses for feedback and feedback from experienced system managers (including a suggestion for a solution), seem to be welcome.

Regarding the generic question about problems after patch installations, there is no generic answer. If you want to be sure (as 'sure' as you can get), try to test your application(s) after each patch installation on an appropriately configured test system.

Volker.

[Alon Jacob]

Labadie - what did you mean by "culprit file"?

Thanks.

[Uwe Zessin]

'culprit file' is the file that is responsible for the problem. Changing the name makes the previous version available unless you have deleted this one already.

[Andy Bustamante]

How about having a patch recalled by Engineering the morning after you installed it? No I didn't roll back, but I watched that system closely until the fixup came out. One of the "SYSxx" patches for 7.2-1 broke Pathworks as I recall. I knew that from comp.os.vms where the problem was posted with a work around.

My in house rules include not installing anything that hasn't been out for a least 4 weeks, 6 months for an o/s upgrade. My first internal system was upgraded to 7.3-2 last week. We test on internal systems before allowing any patch or upgrade near a production site. The only issue I've ever seen in a production site, was with ZIP/UNZIP after upgrading from 7.1 to 7.2-1. There was an upgraded release and I've since added ZIP/UNZIP to to testing. Internally I bundle and test ECOs into internal levels. Our core application hasn't seen any problems with VMS upgrades.

[Doug Phillips]

I've experienced about as many problems from *not* patching as from patching. Now, my S.O.P. is to wait at least one year before upgrading VMS after a new version appears and read the patch release notes on-line as they come out.

The 7.3-2 batch of patches is growing so rapidly that it's hard to keep up with them, and if I remember, there are around 10 reboots required if one follows the instructions (not counting layered products)!

I *really* wish that HP would publish a patch sequence list that tells what patches can be installed together and in which sequence without a reboot.

The UPDATE kit seems to be setup like this, because many of the patches said "reboot required" before they were bundled into one kit, and not all of those do a reboot (IIRC). But when there are 12 or more "kits not included in update kit", and 8 of them ask for reboot... that's a lot of (expletive deleted;-) reboots!

Doug

[Wim Van den Wyngaert]

I have a GS160 interbuilding cluster running very sophisticated software (using tcp, x25, decnet, X) and all is written inhouse. It is handling stock exchange orders in real time and keeps this banks situation in memory (the new applic needs a lot more power for doing less). If I patch and TCP is not working in the same way, or there is a small memory leak, or whatever, my head is cut off.
And testing the application is almost impossible.

So I'm especially interested in problems that caused application failures (by preference even unrunable).

Wim

[Jan van den Ende]

Wim,
if I project my experience (V7.3-1 => V7.3-2) into your description of your situation, then I WOULD invest the effort of re-linking your apps.
_MOST_ of ours apps were just fine, but:
We have one bundle of 5 interacting applics.
4 of them worked fine, also together. #5 ACCVIO'd. After re-linking #5 that one functioned, but #1 did not communicate with it (2,3,4 DID!). Re-link #1, then that communicated nicely with #5, but NOT with 2 anymore!. Re-link #2, and we are back in business (note: #3 & #4 were NOT re-linked).
Historic detail:
#5 originally linked under VMS 7.1; #1 under V7.1-2, #2 under V7.1-2, #3 under V7.2-1.
So far looks consistent, but.. #4 under V6.2!!!
No real basis for trustworthy guidelines there eigther.
Bottom line:
Wanna be on the safe side? Re-link!

hth.

Jan

[Lawrence Czlapinski]

1. I do UPDATE patch, non-reboot patches, then reboot patches with instability patches last.
Then I do one reboot.
2. EDIT/TPU problem during patch updates: If using EDIT/TPU from a console manager window or OPA0:, be sure your window size is 24 lines. Otherwise the changes on the screen often don't match what you're actually changing.

[Willem Grooters]

"If it ain't broke, don't fix it."

If your software relies on non-documented features, internal structures that are not normally accesable using system services or system variables - you ARE required to re-link, eventually re-compile your software. You, and your programmers should know that this is the case.

Caution is Ok, HP should be more aware of the implications and test the software and installations more thoroughly (and re-do some engineering (I mean: porting) since I think that introduced quite some problems)

Anyway: consider the update-procedure I have used for a critical system for years:

1. Keep your system disk as clean as possible. You should need a small one only.
2. Be sure to have multiple, identical backup copies of your system disk (a relatively small (cheap)disk will normally be sufficient).
3. Boot your system from such a copy.
4. Now do the update, reboot (from that disk) if required.
5. Run your application(s) and determine whether all works as expected.
6. If it does, let the system run for some time monitoring all processes, and if no problem is found, promote that to be the system disk.
First, make multiple image copies of that disk, store them in a safe place. If your original system disk is a shadow set, create one now of the NEW one. Then, when in a cluster, reboot the other node(s) (one by one).
If there are problems, revert to original system disk and reboot from that one. To solve the problems, you need a second system, boot that from the updated disk to analyze and fix the problem(s). (you could do that on this machine, of course)

At the time, a one-day period with average load and access was enough to tackle any problems. We only had minor ones one time, that were solved by re-linking part of the application.
However, be aware that when autogen is run after upgrade, you may have to run over a longer period and under heavier load, to get new feedback. Then run autogen again and reboot.

HTH

Willem

[Wim Van den Wyngaert]

Willem,

We have the problem that to relink an application you need an owner. And that one is missing. If you relink it yourself, you become the owner. And that's the last thing I want : owner of some complicated, undocumented system.

We have programs linked under 6.2 in 1997 running on 7.3. Luckely without problems. Also : applications refuse to take relinked programs into production. And no manager that forces them to do so ...

The migration of 6.2 to 7.3 has costed at least 1000 mandays ...

Wim

[Jan van den Ende]

Wim,


The migration of 6.2 to 7.3 has costed at least 1000 mandays ...


And the generated extra cost of the business not going smoothly!


And no manager that forces them to do so ...


Is there NO way to use quote1 to get anyone in the chain of upwards managers to become aware of the real validity of your requests, and the risks of ignoring them?

I thought I knew about irresposible managers, but you are setting a new standard!

Probably all you are left with now is 'cover your ass' with a heavy load of memo's pointing out the risks (and repeat that until at least one of them gets annoyed enough to respond in a way that PROVES they HAVE received your warning.

But it remains sad that these things keep happening, time and time again, at all levels...

I think all that is left now is to wish you strength (& luck)

Jan

[Antoniov.]

Wim,
I've got same problem.
I've fresh installed Vms V7.3-2; I have to test my software application, linked under V6.2; I can test in august, becuase no user are connected.
I remember when I ported form vax vms V5.0 to alphavms V6.2 I met a very weird problem, solved after modified software.
I can't help you about RAID or clustering but I'll report about system and application running.

Cheers
Antonio Vigliotti

[Wim Van den Wyngaert]

Jan,

To give an example : we have an interbuilding cluster of 2 nodes and a quorum station. Software running on it is not ready for drp. I made a list of all things that would go wrong in case the major node would go down (1 year ago). I gave it to my boss. No reaction. The boss his boss. No reaction. The DRP team : they reacted by asking the application teams if they saw problems. They answered no. DRP team closed my remarks.

Until mid July : double memory problems on the major node. 90% of my inventory went wrong and some other things I didn't see went wrong too. Nobody said a word of my inventory but only now are they taking action (for the 90%, the 10% must first prove to go wrong).

Note that this is a big progress because something is done. But I'm sure that within a month, 10% of the 90% will be solved and the incident is forgotton.

Wim

[Antoniov.]

After installation of Advanced Server V7.3, on OpenVms V7.3-2, I downloaded Adv/Serv ECO3 dated July, 20th 2004 by http://www4.itrc.hp.com/service/patch/search.do?pageContextName=openvms::&searchContextName=openvms:alpha:7.3-2&searchType=patch.text.searchTypeBrowse&pageNumberStr=1&pageSizeStr=25&BC=patch.breadcrumb.main|
During installation receive a messagge NCL unknow command. I suppose it happens because it isn't running DecNet-plus but DecNet IV.

Antonio Vigliotti

[John Gillings]

To all

Relinking.

You should NOT have to relink user mode applications when upgrading OpenVMS. If you find a situation where you believe it is necessary PLEASE log a case with a Customer Support Centre so it can be investigated. We have VAX images linked on VAX/VMS V1.0 in the regression test suite which MUST run on the latest version of OpenVMS in order to pass.

Requalification.

You should not have to requalify applications across dash releases. Again, if you find a case that contravenes this rule, PLEASE report it.

Patch stability.

There is a "hold" mechanism for patches where problems are discovered. Over the last three years, there have been a few cases of patches placed on hold, but none for a patch released for more than two weeks (I don't recall any over one week, but I'm being cautious!).

Patch order.

The simplest rule is to order patches by dependencies, and then chronologically. Where there is no dependency, the order should not matter. If you have a problem, please report it.

Reboots.

Read the "reboot requirement" paragraph carefully. Very occasionally it will state that a reboot MUST be performed immediately after installation. If it's not a "MUST" then you can install multiple patches followed by a single reboot. If you're really cautious, and have plenty of time, then reboots between each installation is certainly the safest approach, but it's NOT necessary unless clearly spelled out in the ECO summary. Keep track of your installation order and use the RECOVERY_DATA and PRODUCT REMOVE PATCH feature if you need to back out.

Problems in general after patches or upgrades...

OpenVMS's stability is, in some ways, it's worst enemy. Since systems can stay up for years between reboots, startup procedures are rarely completely debugged and volatile changes can accumulate on a system over time which are lost at the next reboot. So, although you may experience a problem immediately after a patch or upgrade, it's usually the reboot itself that "causes" the problem.

A recent example logged with the CSC, a customer upgraded, then the system crashed with a weird bugcheck on the first reboot after AUTOGEN. Why? Becuase there was an obscure SYSGEN parameter set to an unreasonable value in MODPARAMS. We dated it to more than a year earlier when the customer logged a case with the CSC. Back then the problem was identified, but the customer used SYSGEN to adjust the parameter directly and rebooted. He hadn't touched the system since. So the crash was laying dormant waiting for the next AUTOGEN for whatever reason.

So, if you want to play very cautiously, next time you want to perform an upgrade or patch installation, do a reboot FIRST to confirm there are no latent problems. Especially if it's been a long time since the last reboot.

Again, PLEASE, PLEASE report any problems you experience so they can be properly diagnosed and corrected (if found to be a fault in the product). It does nobody any good to assume the cause of a particular problem and adopt overly restrictive policies as a consequence.

[Wim Van den Wyngaert]

John,

You are right and you are wrong.

Yes it should not be necessary to relink/requalify programs.
But no, it is necessary to do it because even HP makes errors. Unless you are willing to take the risk.

Wim