Upgrade VMS 7.3-2 to 8.3

[Rich Hearn]

Hi Folks,

I'm going to be doing an upgrade of VMS on 2 ES47's in a cluster and I'm wondering if I can "get into trouble" by doing the following:

1 Get shadow copy of on-line cluster system disks
2 Move to DS20 test hardware
3 Remove all HSV disk & network connections from DS20 - no way to be seen on-line with live systems
4 Perform the upgrades specifying it will be in a cluster
5 Shutdown & do not re-boot for autogen until back on the ES47's the day of the upgrade (or should I do the autogens on the DS20's?)

Can anyone see if I'm setting myself up for problems? The goal is to minimize system level downtime (yet upgrade VMS) to allow for application upgrade also

Tnx for thoughts/ideas,
Rich

[Volker Halle · Accepted solution]

Rich,

why not complete the upgrade on the DS20 ? You can still run @AUTOGEN on the ES47s after moving the disk back.

Think about what may be changing on the live ES47 systems after taking a snapshot of the system disk (password changes, any system-mgmt level changes, accounting, auditing, errlog, jobs in queues) - or do you have all those files on a disk other than the system disk ?

Also take into account possible changes to SYSUAF and RIGHTSLIST during the upgrade.

Volker.

[Robert Gezelter]

Rich,

Before I say anything else, in many cases a system upgrade such as you describe should be a painless process with no issues visible to users. Indeed, using a rolling upgrade (one node at a time), it should be possible to accomplish the transition with zero impact on users, just some careful system management of the user workload. However, as I will comment later in this post, planning for the unexpected is what separates the successes from the crises.

While it is not always feasible, I try to move SYSUAF, RIGHTSLIST, and related files to a common location that will not be replaced as part of the system disk change. Alternatively, you can point to the existing files on the current system disks and just bootstrap off of another "volume".

I would take this opportunity to create a REPRODUCIBLE way of cloning the system disk for the test configuration. It is an activity that is likely to be needed in the future, and it is a high payback activity. In particular, I would strongly consider using a separate system root for the 8.2 testing, so as to preserve the system root specific files unmodified. With some care, it is possible to then boot the production roots without having to do surgery on them twice.

Once testing is completed on the test system, I would copy the system pack to a local disk or storage array volume on the production side, and boot ONE of the production nodes from it. I would be postured (see my earlier comment about the cluster common files remaining where they are) for the possibility that a retreat may be required for unexpected reasons. OpenVMS is very good at not causing this problem, but I have seen sites blindsided by unanticipated applications issues, and the ability to "fall back" and come back later without damage is, as I said earlier, what makes the difference.

I hope that above is helpful. If I have been unclear, please let me know.

- Bob Gezelter, http://www.rlgsc.com

[Rich Hearn]

Volker

Thanks for the thoughts on all the possibilities of changes after getting a copy of the system disk and doing the upgrade - very worth a 3rd & 4th look at what all changes - particularly the sysmgr acct.

Bob,

Oh to be able to do a rolling upgrade - alas, application (Cache - GE/IDX) does not allow - mandatory app chg with vms upg - not backward compatible. Cloning the sys disk is always reproducible - thanks to VMS & shadowing (gotta love it!) A "retreat" of VMS would be by "keeping" 1 of the 2 current system disks as a "backup" while the other sys disk becomes the "new" shadow member for v8.3. Booting one of the nodes isn't going to be a possibility unfortunately. Your post is clear and I thank you for your thoughts too.

Both of you give me some fresh "perspectives" to consider and I consider them well worth the time to think about.

Thank you again to both of you,
Rich

[Jan van den Ende]

Rich,

having been trough this many times, I can only concur with Volker and Bon.

Rolling upgrades are soooo nice!.

Not knowing Cache, I can only guess about that part, but we also had one app with similar issues.
You (temporarily) need some extra disk space, but can you not do a simultanious app "rolling upgrade"?
ie, do the OS upgrade "rolling". and the app as well.
Now, the node that uses the updated system disk also uses the updated app.
Clever use of logical names should enable that.

Of course, doing a sandbox (your DS20) exercise is indicated in a sensitive production environment.

Success!

Proost.

Have one on me.

jpe

[Willem Grooters]

If no data conversion or software adaption is required (other than the Cache runtime): complete the update on DS20, reboot it to join the cluster. In that case, you can do some testing before applying the updates to your live system.

To be completely safe, and you have the hardware and software environment (and time and propably budget), and your production environment is setup nicely (that is: as little references to physical disk as possible), you can think of taking an image copy of all disks that are touched in the upgrades, and update that environment, or update the original environment, either from the disks upadted on the DS20, or immediately from that ES47) and finsih installation (including AUTOGEN). Next, boot the other ES47 from that environment.
If something breaks, you can use the saved environment to continue processing and restore the original.

I concur with Bob that pulling all 'changeble' system files off the system disk (SYSUAF, RIGHTLIST, PROXY, ACCOUNTING, AUDIT....) is generally a good idea. certainly if you have multiple system disks (as will be if you have the DS20 pulled into the cluster).

WG

[Rich Hearn]

Jan

1st, Thank You for your thoughts. I appologize to all for not being more clear in my original posting - this is a 2 node VMS cluster (v7.3-2)... it is also a 2 node Application (Cache) Cluster. This is also made more "interesting" by the fact that the current version of the application (v5.0.20) does not function with v8.3 of VMS. That means an upgrade to the application as well (to v5.0.21). Because both nodes (both O/S & App pieces) work as a real cluster, the only way to upgrade the application is to shut it down - the 2 application versions (current & new) are incompatible with each other as well as with VMS versions. I will confess, I've never been "bored" at work :^)


Willem,

Thank You for your thoughts also. Data conversion is not required, but MUMPS recompiles are required (about 3 hours on an idle
ES47) - lots of routines. Because I'm using a copy of the "live" nodes in the test system (DS20), I can not boot the upgraded disk into the live cluster to test, but it would be nice if I could. I really don't want to go changing the SCSNODE & SCSSYSTEMID to allow that - there are enough things I can get wrong already :^)

The system disks are individual to each node, shadowed, and a copy of each current disk will be kept aside as backup - just in case of a problem. Going back will not be simple due to the fact that the application would also have to go back (another 3 hour re-compile) that would not make a lot of people happy.

I am in total agreeance with Bob's thoughts about the "changable" system files - I'll be taking steps to integrate them with the upgraded disks after they're booted.

Rich

[Volker Halle]

Rich,

adding a 3rd root (for the DS20) to the system disk(s) while doing the upgrade on the DS20 may not be a bad idea. Then you could boot the DS20 from it's own root and by just changing VAXCLUSTER from 0 to 2 or vice versa, you could bring that node into the cluster temporarily. While running V7.3-2 and V8.3 in the same cluster may not be 'supported for production', it should not create any problems. And why not compile the apps on the DS20 (if you have enough disks available), even if it may take a bit longer...

Volker.

[Robert Gezelter]

Rich,

A sidelight on the discussion.

If the disks are all at the same OS level, then switching to different roots on a common system disk is a far more efficient use of resources. Especially in a SAN environment, increasing members of mirror sets allows more efficiency in caching and other areas.

It would require a look, but recompiling the routines for 8.3 should not be a roadblock, even in the situation described. The more critical obstacle is the data. If the routines can be recompiled on the DS20, and then the common system pack can be updated, then it is possible to quickly fall back to the 7.3-2 system and applications if a problem is encountered.

If the data files need to be transitioned, it is a different story.

This approach requires admittedly more effort up front, but it is far more robust in the event of an unanticipated problem at a later stage in the process. I always counsel my clients with 24x7 systems that it is not the update process that is most critical, it is the planning for the potential, however small, that things will not go as planned, and a [temporary] fallback will be required to meet operational commitments.

- Bob Gezelter, http://www.rlgsc.com

[Jim Geier_1]

Rich,

In July I upgraded a GE Healthcare (IDX) 3-ES45 cluster with a single shadowed system disk from OpenVMS 7.3-2 and Cache' 5.0.20 to 8.3 and 5.0.21 and was completely done in less than 2 hours. According to GE Healthcare and InterSystems, recompiling all generated routines was not necessary for the minor upgrade.

We do have a non-production cluster of an ES40 and a DS20e with 11 Cache' environments, so I had an opportunity to practice.

My biggest gotcha is remembering to clear the shadow flag on the system disk being upgraded. Seems I almost always forget that, even though it is well-documented. All other aspects of the upgrade went very smoothly and well. I also applied many patch kits to bring the systems to the most current patches since we only get a scheduled outage once every 3 or 4 months.

My biggest frustration is doing the upgrade in the middle of the night, when I'd rather be sleeping. My second biggest frustration is that InterSystems and GE Healthcare do not allow for rolling upgrades.

Suggestion: if your ES47 systems have local disks, do an image restore of the distribution CD to the hard disk -- booting from the distribution is so much faster that way. I keep a copy of the distribution CD on a local disk in all five of the systems for whatever future use and or need arises, like making a standalone backup of the system disk, or defragmenting the system disk, etc.