PCSI-E-EXEPSTFAIL, product supplied EXECUTE POSTINSTALL procedure failed

[Robert_Boyd]

I frequently see this error when attempting to install multiple patch kits. The workaround I've been using for the last couple of years on V7.3-2 and V8.2 has been installing 1 kit at a time. This gets really tedious when I'm applying patches to several systems at once, since I have to keep up with which patches have been applied to each system as I go through a set of them. We generally apply a number of patch kits at one sitting, so as to minimize the number of outages we take for this kind of maintenance.

I know that sometimes there are dependencies that require sequential installation of patch kits. I'm getting this error when I attempt to install patches that have no order dependencies with each other.

I logged this problem with HP and the word I got back is "well, you just need to install 1 patch kit at a time as a workaround" -- I already KNEW that! Thankyouverymuch.

Here's some additional info: I usually am required to log everything fairly thoroughly so I use the following DCL to set up for doing patch kit installs:

$ nodename = f$edit(f$getsyi("nodename"),"trim")
$ os_ver = f$edit(f$getsyi("VERSION"),"TRIM")-"V"-"."-"-"
$!
$ DEFINE/SYS/EXE NO_ASK$BACKUP TRUE
$ DEFINE/SYS/EXE NO_ASK$REBOOT TRUE
$ DEFINE/SYS/EXE ARCHIVE_OLD NO
$ DEF/SYS/EXE AXPVMS$PCSI_EXECUTE_VERIFY YES
$ DEF/SYS/EXE AXPVMS$PCSI_LOG_TRACE BOTH
$ DEF/SYS/EXE PCSI$$SAVE_RECOVERY_DATA YES
$ DEF/SYS/EXE PCSI$LOG TRUE
$ DEF/SYS/EXE PCSI$SOURCE PATNOD::patch_root:['nodename'.'os_ver']
$ DEF/SYS/EXE PCSI$TRACE TRUE


And the command I usuallly use is something like

$PRODUCT INSTALL */SAVE/LOG

Do you have any ideas about what would be generating the error I get?

Portion done: 90%
%PCSI-I-CREFIL, created $30$DKB0:[SYS0.SYSMGR.PCSI$WRK21200C84.][SYSUPD]PCSI_POSTINSTALL.COM;1
%PCSI-I-CREDIR, created directory $30$DKB0:[SYS0.SYSMGR.PCSI$SCR21200C84]
%PCSI-I-SPAWNSUB, spawning interactive subprocess PCSI$0C84_I0004 to run command
%PCSI-I-SPAWNCMD, @PCSI$SOURCE:[SYSUPD]PCSI_POSTINSTALL.COM
%PCSI-I-SPAWNEND, interactive subprocess PCSI$0C84_I0004 exit status: %X100001BC
%PCSI-I-DELDIR, deleted directory $30$DKB0:[SYS0.SYSMGR]PCSI$SCR21200C84.DIR;1

%PCSI-E-EXEPSTFAIL, product supplied EXECUTE POSTINSTALL procedure failed
-SYSTEM-F-NOLOGNAM, no logical name match
%PCSI-E-OPFAILED, operation failed
Terminating is strongly recommended. Do you want to terminate? [YES]
%PCSI-E-CANCEL_WIP, termination resulted in an incomplete modification to the system
%PCSI-I-PRCLOGOUT, non-interactive subprocess PCSI$0C84_N0001 has logged out

[Hoff]

Try completely and entirely separate installations, from unique processes.

Or pull the kits apart, and look. There's probably a logical name left dangling, and subsequent installations are tripping over it. Most of the kits likely have some sort of a verification mechanism, or can have one added. (There were some changes made to PCSI in this area, too, IIRC.)

Your PCSI$SOURCE definition is not what I'd try -- I'd spend a little disk space during the installation, and haul the kits over locally. PCSI is not known for its parsing abilities, and this looks a little far afield.

A more interesting discussion with HP would be around the links of more automated patches -- if you ask for a specific bugfix or such, that's what you're going to get an answer for. But if you really want an easier mechanism to load in a bunch of ECOs, ask for that.

Or do what HP says, and apply the CO kits one at a time...

[Jon Pinkley]

I have never tried this; is it possible to do these installations from batch? If so you could initialize a batch queue with a job limit of one, and submit each update as separate job. The queue would provide the single threading, and would allow you to control the order of installation. Since each job would be a "fresh" process, each new process would pick up any changes to the DCL command tables.



Wouldn't it be nice if you could just

$ vms_update ip_address_of_server_with_updates

from a system with an internet connection, and it could figure out what needed to be updated, figure out dependencies, etc. and ask if you wanted a quick or custom install?

For systems without an internet connection, or for a site with multiple systems to update, it should be be possible to have a local system act as the update server.

This is one area that MS Windows has VMS beat hands down, in my opinion. I hate to say it; but I do believe it.

The one thing VMS has that makes updates easier is that you can have multiple nodes booting off a common system disk, but for sites with 7x24 uptime requirements for the cluster, that isn't an option.

I think it would be possible to automate the update process, and still give the user control of what was being updated.

[Volker Halle]

Robert,

I've seen this as well - from time to time. It's cleary a bug in either PCSI or in one of the patch kits.

If HP doesn't want to invest in the troubleshooting of this problem, you would have to do it yourself or live with the workaround.

I can provide a recent example, for anyone, who's willing to invest in troubleshooting:

- install VMS732_UPDATE-V0900

- then install the following list of patches in one go with $ PRODUCT INSTALL * /SOURCE=...

BACKUP-V0700, TRACE-V0300, FIBRE_SCSI-V1000, MOUNT96-V0200, TZ-V0300, SYS_V1100

I didn't use any of the logicals you've used, but this operation failed with the same error during PCSI postprocessing. Installing the patches individually - even in the same logged-in process, worked fine.

Volker.

[Volker Halle · Accepted solution]

Robert,

each patch includes a PCSI_POSTINSTALL.COM, which looks like it's being automatically generated by the VKC kit build procedure at HP, when a new patch is being created.

Some of the older patch kits (e.g. VMS82A_GRAPHICS-V0100) include a SET NOON at the beginning of PCSI_POSTINSTALL.COM and an EXIT 1 at the end. They should not be able to cause this problem.

More recent patch kits (like VMS732_BACKUP-V0700) do not include a SET NOON and may therefore suffer from any error causing the PCSI_POSTINSTALL.COM procedure to abort.

The following 2 commands in those procedures may cause the problem you've seen:

$ log_def = F$TRNLNM("ARCHIVE_OLD")
$ if log_def .nes. "" then deassign ARCHIVE_OLD/job

The guidelines for the ARCHIVE_OLD logical (at the bottom of each patch ECO cover letter) give the following command as an example:

$ DEFINE/JOB ARCHIVE_OLD NO

but PCSI_POSTINSTALL.COM does not limit the F$TRNLNM check to just the job table.

When you encounter that problem the next time, consider to delete your system-wide ARCHIVE_OLD logical name and try PRODUCT INSTALL * again, before dropping back to the single patch-by-patch installation workaround.

Volker.

[Volker Halle]

Robert,

there seems to exist a customer advisory with Document ID: 'c00630847' about this problem dated from 23-MAR-2006.

http://mail.openvms.org:8100/Lists/alerts/Message/258-P.txt

The solution cited may not have been completely and universally implemented - even in the most recent patch kits.

So it's worth another call to HP - after you've collected some more information.

Volker.

[Volker Halle]

Robert,

looks like the problem is this:

PCSI_PRECONFIGURE.COM looks at the logical ARCHIVE_OLD and defines an appropriate ARCHIVE_OLD logical in the /JOB_TABLE.

PCSI_POSTINSTALL.COM deletes the ARCHIVE_OLD logical in the job-table, if any (!) ARCHIVE_OLD logical exists.

Now if you install more than one patch in a single PRODUCT INSTALL command, all the PCSI_PRECONFIGURE.COM files for all patches will be run first - and will define (or re-define) the ARCHIVE_OLD logical in the job table.

Once each individual patch has been installed, PCSI_POSTINSTALL.COM will be run and the first invocation will delete the ARCHIVE_OLD logical in the job table. The next invocation of PCSI_POSTINSTALL.COM (for the 2nd patch) will see the system-wide ARCHIVE_OLD logical will incur the NOLOGNAM error when trying DEASS/JOB and abort the PRODUCT INSTALL operation.

As the Customer Advisory indicates, you can safely continue with the operation by answering NO to the termination question (ONLY in case of the SYSTEM-F-NOLOGNAM error in the PCSI_POSTINSTALL.COM phase).

The problem could have been prevented like this (in PCSI_POSTINSTALL.COM):

$ log_def = F$TRNLNM("ARCHIVE_OLD","LNM$JOB")
$ if log_def .nes. "" then deassign ARCHIVE_OLD/job

i.e. only try to delete the ARCHIVE_OLD logical in the JOB_TABLE, if it really exists in the JOB_TABLE.

Another workaround would be to deassign the system-wide ARCHIVE_OLD logical.

Now you have enough information and even a suggestion for a solution to log another case with HP...

Volker.

PS: I have tested this by simply installing VMS83A_ERRFMT-V0100 and VMS83A_TZ-V0100 on a native V8.3 system with PRODUCT INSTALL * and an existing system-wide definition for ARCHIVE_OLD.

[Hoff]

Yes, this is a bug in the ECO kits, and arguably in PCSI itself -- as the kits should not be maintaining nor propagating out any externally-visible state. (That there's DCL down underneath is pointing toward the usual "fun" of VMSINSTAL, a wonderful and powerful tool, replete with the occasional weird failures.) As I mentioned some time back, I'd suggest use of separate processes, one per installation.

Based on the answer you received from HP on this, I'd tend to empirically assume that this sort of interference will continue to occasionally arise among various ECO kits -- and would take steps to avoid it if I needed to "slipstream" ECO kits. (And if VMSINSTAL is any guide, there can be even more subtle manifestations of inter-kit weirdness.) And I'd therefore use a process per installation, and quite possibly a process not in the same job tree.

A per-install private process is most certainly inelegant, but it is likely effective and appropriately defensive.

The alternative to this approach, and what I'd tend to expect to see HP implement, is a check of the logical names and file system changes spilling out of the kit. A before and after picture of the job, group and other shared logical name tables, and of the file system outside of the kit-intended modifications.

[Robert_Boyd]

Thanks to everyone who replied. I figured this error message had something to do with the logical names.

Hoff, I'm not sure why you say PCSI would have any problem with a properly constructed logical name pointing to a remote directory via DECnet. I've used this construct for years on VAX and Alpha systems. It's just pointing to NODE::DEVICE:[PATCHES.MYNODE] -- nothing magic there. In some cases I have copied files over just to speed things up a tiny bit -- PCSI has never had any trouble with that aspect either way.

What I don't understand is why PCSI is set up to deassign the ARCHIVE_OLD logical name in the first place. I realize that in theory you might want to say yes to one patch kit and no to another one and then yes again to the next, but why deassign it after each invocation of PCSI? Let the user keep track of it and if they want to change it, they can re-define it. When I'm installing patches, I'm going to do a whole set all the same, and then when I'm done I might do another set the other way.

Since my ticket with HP is still open, I'm going to submit some additional detail suggest that the post installation procedures check that the logical is defined in the JOB table before attempting to deassign it. This is in the category of basic correct DCL/Logical Name Table programming.

Robert

[Robert_Boyd]

Jon and Hoff,

It does appear to me that a more streamlined approach to patch management would be immensely helpful.

I posted once before that I have an elaborate combination of DCL downloads patch kits, scans release notes, and does a fairly good job of sorting out which kits have been superseded and files them away for safekeeping, leaving the current ones in a nice neat directory. In addition, I've written some additional code that generates a map of what is on each node in my environment and checks that against the current set of patches and builds a list of "needed" patches for each system. The result is a set of directories by nodename and OpenVMS version number with appropriate links to the patch kits.

At least once per month I run the job that does all this processing and then look at the list of patch kits generated for each node and make some assessment of how soon we should be trying to apply the various patches based on criticality.

I've thought about posting some of this code. There are some parts of it that are written in DCL that probably should be re-done in perl to make it more easily maintainable.

If anybody is just dying for something fun to do, I'll be glad to share the code and see what comes of it.

Robert

[Bill Hall]

Robert,

I also typically install multiple ECOs in a command procedure using FAL. I've never had an issue with this. However, my procedure is "hard coded" with ECOs in a specific installation order and each ECO is installed with a $PRODUCT INSTALL . This way I can do my own "post install" if an ECO requires some it.

Why don't you add a loop to your procedure to F$SEARCH through your PSCI$SOURCE directory and PRODUCT INSTALL each PCSI kit you find?

Bill

[Hoff]

>>> Hoff, I'm not sure why you say PCSI would have any problem with a properly constructed logical name pointing to a remote directory via DECnet. <<<

Why? Long experience chasing weirdnesses. It *should* work, but then I've seen various weirdness ensue with this construct. (Subtle stuff, such as file dates, can get snarled as FAL truncates the values. Sometimes stuff tips over rather more overtly, because nobody thought to test that particular combination.)

As for the process improvements, I've long thought that the ECO kit information should be distributed using XML. With the exception of a line or three of XML tags at the top of an ECO text file, the first big block of the document can easily be straight human-readable text -- as it is now -- and the rest of the file (down at the bottom) can be the actual business end of the XML. This allows code such as yours to more reliably spot changes to the ECO via the XML -- most folks are presently using file dates for this task, and every so often that goes off the rails. With XML, everybody can receive specific information, superseded or prerequisite kits, MD5 checksums, and other such related to the specific ECO. All in a format that is established and extensible and easily parsed. (This topic is certainly fodder for adding a blog entry or three...)