topic Re: Error with VMS732_Update v14 Install in Operating System - OpenVMS

Error with VMS732_Update v14 Install

Rick Dyson — Mon, 10 Dec 2007 03:30:21 GMT

I installed this newest update on a system I have that is otherwise caught up and got this error:

Portion done: 0%...10%...20%...30%...40%...50%...60%...70%...80%
%PCSI-I-PRCOUTPUT, output from subprocess follows ...
%LIBRAR-W-HISTERR, error accessing library update history in DISK$SIPLE_73:[SYS0.SYSCOMMON.][SYSLIB]STARLET.OLB;5
-RMS-E-EOF, end of file detected

%PCSI-E-MODREPLERR, error replacing module C$$MAIN in library DISK$SIPLE_73:[SYS0.SYSCOMMON.][SYSLIB]STARLET.OLB
%PCSI-E-OPFAILED, operation failed
Terminating is strongly recommended. Do you want to terminate? [YES]

I did terminate on this node, however on another one that was not critical, I experimented and let it continue. At this time, that node seems OK but I am not sure if I want to blindly override the recomendation on more production critical nodes until I know where and why I am getting the error.

Does anyone have any insight?

Rick

Re: Error with VMS732_Update v14 Install

Volker Halle — Mon, 10 Dec 2007 07:41:14 GMT

Rick,

this sounds like the module update history in STARLET.OLB is corrupt. There was a recent problem on OpenVMS I64 regarding similar symptoms (see VMS83I_LIBRAR-V0100), but it seems to have been Itanium-only.

Is STARLET.OLB;5 your current library ? Did you try $LIBRARY/LIST/HISTORY/FULL SYS$LIBRARY:STARLET.OLB;5 ?

This error has apparently not been reported before. You will not directly 'see' this type of corruption, unless you try to delete or replace that (or any other) module, which would typically only happen during upgrades or patch installations.

Volker.

Re: Error with VMS732_Update v14 Install

Bart Zorn_1 — Mon, 10 Dec 2007 07:44:04 GMT

Hello Rick,

At the first attempt to install UPDATE V14 on our test system, I encountered the same error.

I just tried to reinstall the update on the same system, and the error did not happen again!

Because I had planned to install ECO's on our production systems last weekend, and UPDATE V14 does not contain anything new which I had not verified before, I installed UPDATE V14 on production. No error here!

YMMV!

Bart Zorn

Re: Error with VMS732_Update v14 Install

Rick Dyson — Mon, 10 Dec 2007 07:49:22 GMT

Bart: Thank you for the info! I have not tried a 2nd time, but will now to see. I too had everything up to date so it was theoretically just a formality!

Volker: I did not sniff around with library commands, but may just to see what I get. I have never done anything manually to that library outside of running ECOs or Installs, etc.

Thanks!
rick

Re: Error with VMS732_Update v14 Install

Volker Halle — Mon, 10 Dec 2007 12:01:01 GMT

Rick,

I've just installed VMS732_UPDATE-V1400 on an OpenVMS Alpha V7.3-2 SSB system with just VMS732_PCSI-V0300 installed previously. No problem.

So the STARLET.OLB update history corruption must have happened during installation of a previous patch.

After installing VMS732_UPDATE-V1400, the following command lists lots of module deletions and replacements in the Library Update History section:

$ libr/lis/full/hist sys$share:starlet.olb

This kind of problem may be very hard to diagnose. You would have to collect all patch installation history information from all sites seeing this problem and try to figure out, under which circumstances this problem would show up. Collecting such information would only make sense at HP, but as V7.3-2 is out of support...

Until then, I would suggest to check STARLET.OLB prior to installation of VMS732_UPDATE-V1400 with to above LIBR/LIS command.

Volker.

Re: Error with VMS732_Update v14 Install

Duncan Morris — Mon, 10 Dec 2007 12:05:48 GMT

Rick,

I have also experienced this error - but on only 1 system.

The update went in with no issues on 2 DS20e systems with 4GB, but failed on a DS10 with only 256MB of memory.

From the lis/his/fu listings, the process appears to have completed OK - even though I did an explicit continue following the "error". The C$$MAIN module appeared to have been correctly inserted when compared with the DS20e systems.

I would suspect that it might be related to quotas in some way (although the DS10 account appeared to have ample).

The failing LIBRARIAN phase is certainly a heavy consumer of virtual memory.

Repeating the install on the DS10 came up with no problems, second time around.

Regards,

Duncan

Re: Error with VMS732_Update v14 Install

Volker Halle — Mon, 10 Dec 2007 12:14:20 GMT

Duncan,

did you also get RMS-E-EOF as the system service failure code for the %LIBRAR-W-HISTERR error ? I've seen EXQUOTA errors being reported on a situation like this, but EOF due to quota problems would not be VMS-like.

Volker.

Re: Error with VMS732_Update v14 Install

Duncan Morris — Mon, 10 Dec 2007 12:16:11 GMT

Volker,

it was exactly the same error as reported by Rick.

I was also suprised by the EOF error!

Duncan

Re: Error with VMS732_Update v14 Install

Duncan Morris — Mon, 10 Dec 2007 13:02:07 GMT

Hi,

the DS10's small memory was probably a red herring, as I also have the issue on another larger system.

The problem is repeatable provided I abort the update at the prompt, so I will look at it in more detail later today.

Duncan

Re: Error with VMS732_Update v14 Install

Volker Halle — Mon, 10 Dec 2007 13:26:08 GMT

Duncan,

for further tests, please always check STARLET.OLB with LIBR/LIS/FULL/HIST before installing the update kit. If STARLET.OLB is o.k. before the update and you get the error during the update, it must be an error in the update procedure.

Maybe the 'replace' worked and the new module has been correctly replaced, but accessing and/or writing the record in the update history failed.

After a successful installation, I see the following entries (from a LIBR/LIS=x.x/FULL/HIST SYS$SHARE:STARLET.OLB):

...
Max. Number history records: 50 Library history records: 50
...

C$$MAIN Ident V4.5-108 Inserted 10-DEC-2007 12:21:02 14 symbols
C$$MAIN_COND_HAND Ident V4.5-28 Inserted 10-DEC-2007 12:21:03 6 symbols
...
SYSTEM deleted 1 module on 10-DEC-2007 12:17:43
C$$MAIN_COND_HAND

SYSTEM deleted 1 module on 10-DEC-2007 12:17:46
C$$MAIN
...

SYSTEM inserted 182 modules on 10-DEC-2007 12:23:48
C$$MAIN
C$$MAIN_COND_HAND
C$$MEMFUNC
...

Volker.

Re: Error with VMS732_Update v14 Install

George_145 — Mon, 10 Dec 2007 13:34:42 GMT

This problem originally came up on a V8.3 Integrity ACRTL patch kit. When that was researched, the problem was determined to be an Integrity-only Librarian issue. Looks like that may have been incorrect. I've forwarded this to engineering so they can take a look at this.

When the ACRTL kit is installed, Starlet.olb gets corrupted. This doesn't show up until the second time you install a kit that modifies STARLET.OLB. Exactly what happens when you have installed the ACRTL kit and then install the UPDATE kit, that also includes the ACRTL kit. Once the corruption has occured, you can do a LIBRARY/COMPRESS to fix it. The node that completed the installation is ok. Performing the LIBRARY/COMPRESS will correct the corruption. On the node with the installation that was terminated, you can either let the install continue and then do the COMPRESS or do the COMPRESS, which will fix the corruption, then do the installation which will succeed with no error. But, it will re-corrupt the library so you will need to do a second LIBRARY/COMPRESS after the installation.

The LIBRARY/COMPRESS will NOT fix the root cause of the problem. We will need a new patch kit to do that. Here is the problem description from the LIBRAR kit that corrected the problem for V8.3 Integrity:

"When the LIBRARIAN utility updates STARLET.OLB, library corruption can occur
that would interfere with the ability to delete modules from STARLET.OLB.
This problem typically has been seen after installing OpenVMS ACRTL patch
kits and UPDATE patch kits which include the ACRTL patch kits. These kits
delete modules from STARLET.OLB. Once such a patch kit has been installed,
and the library corrupted, subsequent installations of patch kits that
delete modules from STARLET.OLB could result in error messages similar to
the following:

%PCSI-E-MODDELERR, error deleting module
C$FTIME_UTC_XPG4 from library
DISK$I64SY1:[SYS0.SYSCOMMON.][SYSLIB]STARLET.OLB

%PCSI-E-OPFAILED, operation failed

It is important to note that this problem is not limited to patch installations but could occur anytime the STARLET.OLB is modified. Also, although the library is corrupted, preventing module deletion, all of the data in the library modules is correct and unaffected.

The new images in this kit will prevent this problem from occurring in the future but will not fix an already corrupted STARLET.OLB. To fix any corruption, this patch kit will also perform a LIBRARY/COMPRESS on STARLET.OLB which will eliminate the corruption."

George Pagliarulo
ECO Release Process
OpenVMS Sustaining Engineering
Hewlett-Packard Company
e-mail: george.pagliarulo@hp.com

Re: Error with VMS732_Update v14 Install

Volker Halle — Mon, 10 Dec 2007 13:52:47 GMT

George,

thanks.

Is there a way to 'detect' a corrupted STARLET.OLB ?

Or would it be simpler to just always do a LIBR/COMPRESS after installing a patch, which modifies STARLET.OLB ?

Volker.

Re: Error with VMS732_Update v14 Install

Volker Halle — Mon, 10 Dec 2007 13:58:02 GMT

George,

it does not seem to be documented in the patch release notes, wheter a patch actually modifies STARLET.OLB (checked VMS83A_ACRTL-V0*00 as an example).

Volker.

Re: Error with VMS732_Update v14 Install

Duncan Morris — Mon, 10 Dec 2007 14:25:51 GMT

George,

many thanks for that information.

I can confirm that the node with the "repeatable" problem was able to run the update without issue, following a lib/compress run. (I have also repeated the lib/compress subsequent to the update).

Duncan

Re: Error with VMS732_Update v14 Install

George_145 — Mon, 10 Dec 2007 14:35:17 GMT

Well, I mis-spoke, or wrote. The problem originally was reported on V8.2-1. It was fixed there and on V8.3. Usually, when we update a library we put in a notation in the image header information that says something like "update to starlet.olb"....usually.

We do not often update STARLET.OLB. Once the I64 LIBRAR kits are installed the problem is corrected and you don't need to worry about it. Same with Alpha, once this problem is corrected. If we have to issue any other kit that that updates STARLET.OLB before the problem is corrected, that kit will do a LIBRARY/COMPRESS as part of the install - same as the I64 LIBRAR patch kits do to correct the corruption.

Re: Error with VMS732_Update v14 Install

Rick Dyson — Mon, 10 Dec 2007 18:35:57 GMT

Everyone has been busy this morning! I went to bed after the over-night maint and woke to all kinds of interesting activity! I went sprinkling points everywhere. Never easy how to do that, but there were lots of spots to sprinkle.

So, to summarize, it seems like this might be same or similar problem recently seen on v8.x systems (I am v7.3-2 or lower everywhere). And Engineering is apparently aware of it and working on it. I have a service contract, but had not exercised it but wanted to put feelers out to this group first. Seems to have worked!

George: Do you need someone to log a formal service request on this?

I admit I have not yet absorbed everything written in this thread completely, but what I am not clear on is whether I need to run the mentioned Lib/Compress command or not? And, if so, when? Before applying the Update ECO? After continuing thru error? Or some other combo with applying ECO, backing out after error, then compressing, and then re-applying ECO?

Thanks for all your help and info, everyone!

Happy Holidays!
Rick

Re: Error with VMS732_Update v14 Install

Volker Halle — Mon, 10 Dec 2007 18:48:52 GMT

Rick,

thanks a lot for reporting this problem. Now the footprint of this problem is documented for everyone to find. This has also alerted the OpenVMS community and even OpenVMS engineering - with a little help ;-)

Also a workaround has been provided:

$ SET DEF SYS$COMMON:[SYSLIB]
$ LIBR/COMPRESS STARLET.OLB

This needs to be executed AFTER installing any patch, which modifes STARLET.OLB (to remove the possible corruption)

OR

BEFORE installing any patch, which may modify STARLET.OLB to prevent the HISTERR during install.

LIBR/COMPRESS just creates a new version of STARLET.OLB with corrected and intact library update history data.

The above has to be done until the underlying problem has been fixed (expect a new set of VMSxxx_LIBRAR-V0n00 patches for OpenVMS Alpha V7.3-2 up to V8.3).

Volker.

Re: Error with VMS732_Update v14 Install

George_145 — Mon, 10 Dec 2007 18:56:33 GMT

If the LIBRARY/COMPRESS is done before installation of a kit that modifies STARLET.OLB then it also has to be done AFTER the installation. The first COMPRESS fixes any existing corruption but installing the kit will re-corrupt the library so the COMPRESS has to be repeated. If it were me, I'd do the pre-install compress and then the post install compress rather than continuing from an installation that got an error. Better history - I don't want to look at an installation log 6 months later see there was an error and wonder if anything was ever done about it.

Engineering is looking at this problem. It is not confirmed yet that it is the same base problem cause.

George Pagliarulo
ECO Release Process
OpenVMS Sustaining Engineering
Hewlett-Packard Company
e-mail: george.pagliarulo@hp.com

Re: Error with VMS732_Update v14 Install

Rick Dyson — Mon, 10 Dec 2007 19:05:48 GMT

Thank you Volker! That helps a lot. You too George for the suggestiong to keep it as clean as possible.

My primary hospital nodes went smoothly this morning, so whatever lead to this did not happen to them (two node cluster with shared, common system disk). I also had a smooth install to a stand-alone system for another client last night.

The system where I am seeing the problem is one with 3 nodes, all independent system disks. I generally do rolling updates and always start with the old, very slow small DECServer 3000 that I have in the cluster just to have a 3 node cluster. The other two nodes are where work is done and the disks are mounted, etc.

Is there a need of a formal service problem request from me?

Rick

Re: Error with VMS732_Update v14 Install

George_145 — Mon, 10 Dec 2007 19:22:50 GMT

Yes, please file a service request.

On the nodes that did not have a problem, what versions were they ; did they have the ACRTL kit installed or a previous UPDATE kit that included an ACRTL kit? If not, that is why you did not see the problem. The corruption, *if* it is the same, occurs on the first installation but the error is not seen until a second installation tries to update the corrupted STARLET.OLB. So, if you did not see the error, you still have a corrupted STARLET.OLB and need to do the LIBRARY/COMPRESS

George Pagliarulo
ECO Release Process
OpenVMS Sustaining Engineering
Hewlett-Packard Company
e-mail: george.pagliarulo@hp.com