Error with VMS732_Update v14 Install

[Rick Dyson]

I installed this newest update on a system I have that is otherwise caught up and got this error:

Portion done: 0%...10%...20%...30%...40%...50%...60%...70%...80%
%PCSI-I-PRCOUTPUT, output from subprocess follows ...
%LIBRAR-W-HISTERR, error accessing library update history in DISK$SIPLE_73:[SYS0.SYSCOMMON.][SYSLIB]STARLET.OLB;5
-RMS-E-EOF, end of file detected

%PCSI-E-MODREPLERR, error replacing module C$$MAIN in library DISK$SIPLE_73:[SYS0.SYSCOMMON.][SYSLIB]STARLET.OLB
%PCSI-E-OPFAILED, operation failed
Terminating is strongly recommended. Do you want to terminate? [YES]

I did terminate on this node, however on another one that was not critical, I experimented and let it continue. At this time, that node seems OK but I am not sure if I want to blindly override the recomendation on more production critical nodes until I know where and why I am getting the error.

Does anyone have any insight?

Rick

[Volker Halle]

Rick,

this sounds like the module update history in STARLET.OLB is corrupt. There was a recent problem on OpenVMS I64 regarding similar symptoms (see VMS83I_LIBRAR-V0100), but it seems to have been Itanium-only.

Is STARLET.OLB;5 your current library ? Did you try $LIBRARY/LIST/HISTORY/FULL SYS$LIBRARY:STARLET.OLB;5 ?

This error has apparently not been reported before. You will not directly 'see' this type of corruption, unless you try to delete or replace that (or any other) module, which would typically only happen during upgrades or patch installations.

Volker.

[Bart Zorn_1]

Hello Rick,

At the first attempt to install UPDATE V14 on our test system, I encountered the same error.

I just tried to reinstall the update on the same system, and the error did not happen again!

Because I had planned to install ECO's on our production systems last weekend, and UPDATE V14 does not contain anything new which I had not verified before, I installed UPDATE V14 on production. No error here!

YMMV!

Bart Zorn

[Rick Dyson]

Bart: Thank you for the info! I have not tried a 2nd time, but will now to see. I too had everything up to date so it was theoretically just a formality!

Volker: I did not sniff around with library commands, but may just to see what I get. I have never done anything manually to that library outside of running ECOs or Installs, etc.

Thanks!
rick

[Volker Halle]

Rick,

I've just installed VMS732_UPDATE-V1400 on an OpenVMS Alpha V7.3-2 SSB system with just VMS732_PCSI-V0300 installed previously. No problem.

So the STARLET.OLB update history corruption must have happened during installation of a previous patch.

After installing VMS732_UPDATE-V1400, the following command lists lots of module deletions and replacements in the Library Update History section:

$ libr/lis/full/hist sys$share:starlet.olb

This kind of problem may be very hard to diagnose. You would have to collect all patch installation history information from all sites seeing this problem and try to figure out, under which circumstances this problem would show up. Collecting such information would only make sense at HP, but as V7.3-2 is out of support...

Until then, I would suggest to check STARLET.OLB prior to installation of VMS732_UPDATE-V1400 with to above LIBR/LIS command.

Volker.

[Duncan Morris]

Rick,

I have also experienced this error - but on only 1 system.

The update went in with no issues on 2 DS20e systems with 4GB, but failed on a DS10 with only 256MB of memory.

From the lis/his/fu listings, the process appears to have completed OK - even though I did an explicit continue following the "error". The C$$MAIN module appeared to have been correctly inserted when compared with the DS20e systems.

I would suspect that it might be related to quotas in some way (although the DS10 account appeared to have ample).

The failing LIBRARIAN phase is certainly a heavy consumer of virtual memory.

Repeating the install on the DS10 came up with no problems, second time around.

Regards,

Duncan

[Volker Halle]

Duncan,

did you also get RMS-E-EOF as the system service failure code for the %LIBRAR-W-HISTERR error ? I've seen EXQUOTA errors being reported on a situation like this, but EOF due to quota problems would not be VMS-like.

Volker.

[Duncan Morris]

Volker,

it was exactly the same error as reported by Rick.

I was also suprised by the EOF error!

Duncan

[Duncan Morris]

Hi,

the DS10's small memory was probably a red herring, as I also have the issue on another larger system.

The problem is repeatable provided I abort the update at the prompt, so I will look at it in more detail later today.

Duncan

[Volker Halle]

Duncan,

for further tests, please always check STARLET.OLB with LIBR/LIS/FULL/HIST before installing the update kit. If STARLET.OLB is o.k. before the update and you get the error during the update, it must be an error in the update procedure.

Maybe the 'replace' worked and the new module has been correctly replaced, but accessing and/or writing the record in the update history failed.

After a successful installation, I see the following entries (from a LIBR/LIS=x.x/FULL/HIST SYS$SHARE:STARLET.OLB):

...
Max. Number history records: 50 Library history records: 50
...

C$$MAIN Ident V4.5-108 Inserted 10-DEC-2007 12:21:02 14 symbols
C$$MAIN_COND_HAND Ident V4.5-28 Inserted 10-DEC-2007 12:21:03 6 symbols
...
SYSTEM deleted 1 module on 10-DEC-2007 12:17:43
C$$MAIN_COND_HAND

SYSTEM deleted 1 module on 10-DEC-2007 12:17:46
C$$MAIN
...

SYSTEM inserted 182 modules on 10-DEC-2007 12:23:48
C$$MAIN
C$$MAIN_COND_HAND
C$$MEMFUNC
...


Volker.

[George_145]

This problem originally came up on a V8.3 Integrity ACRTL patch kit. When that was researched, the problem was determined to be an Integrity-only Librarian issue. Looks like that may have been incorrect. I've forwarded this to engineering so they can take a look at this.

When the ACRTL kit is installed, Starlet.olb gets corrupted. This doesn't show up until the second time you install a kit that modifies STARLET.OLB. Exactly what happens when you have installed the ACRTL kit and then install the UPDATE kit, that also includes the ACRTL kit. Once the corruption has occured, you can do a LIBRARY/COMPRESS to fix it. The node that completed the installation is ok. Performing the LIBRARY/COMPRESS will correct the corruption. On the node with the installation that was terminated, you can either let the install continue and then do the COMPRESS or do the COMPRESS, which will fix the corruption, then do the installation which will succeed with no error. But, it will re-corrupt the library so you will need to do a second LIBRARY/COMPRESS after the installation.

The LIBRARY/COMPRESS will NOT fix the root cause of the problem. We will need a new patch kit to do that. Here is the problem description from the LIBRAR kit that corrected the problem for V8.3 Integrity:

"When the LIBRARIAN utility updates STARLET.OLB, library corruption can occur
that would interfere with the ability to delete modules from STARLET.OLB.
This problem typically has been seen after installing OpenVMS ACRTL patch
kits and UPDATE patch kits which include the ACRTL patch kits. These kits
delete modules from STARLET.OLB. Once such a patch kit has been installed,
and the library corrupted, subsequent installations of patch kits that
delete modules from STARLET.OLB could result in error messages similar to
the following:

%PCSI-E-MODDELERR, error deleting module
C$FTIME_UTC_XPG4 from library
DISK$I64SY1:[SYS0.SYSCOMMON.][SYSLIB]STARLET.OLB

%PCSI-E-OPFAILED, operation failed

It is important to note that this problem is not limited to patch installations but could occur anytime the STARLET.OLB is modified. Also, although the library is corrupted, preventing module deletion, all of the data in the library modules is correct and unaffected.

The new images in this kit will prevent this problem from occurring in the future but will not fix an already corrupted STARLET.OLB. To fix any corruption, this patch kit will also perform a LIBRARY/COMPRESS on STARLET.OLB which will eliminate the corruption."

George Pagliarulo
ECO Release Process
OpenVMS Sustaining Engineering
Hewlett-Packard Company
e-mail: george.pagliarulo@hp.com

[Volker Halle]

George,

thanks.

Is there a way to 'detect' a corrupted STARLET.OLB ?

Or would it be simpler to just always do a LIBR/COMPRESS after installing a patch, which modifies STARLET.OLB ?

Volker.

[Volker Halle]

George,

it does not seem to be documented in the patch release notes, wheter a patch actually modifies STARLET.OLB (checked VMS83A_ACRTL-V0*00 as an example).

Volker.

[Duncan Morris]

George,

many thanks for that information.

I can confirm that the node with the "repeatable" problem was able to run the update without issue, following a lib/compress run. (I have also repeated the lib/compress subsequent to the update).

Duncan

[George_145]

Well, I mis-spoke, or wrote. The problem originally was reported on V8.2-1. It was fixed there and on V8.3. Usually, when we update a library we put in a notation in the image header information that says something like "update to starlet.olb"....usually.

We do not often update STARLET.OLB. Once the I64 LIBRAR kits are installed the problem is corrected and you don't need to worry about it. Same with Alpha, once this problem is corrected. If we have to issue any other kit that that updates STARLET.OLB before the problem is corrected, that kit will do a LIBRARY/COMPRESS as part of the install - same as the I64 LIBRAR patch kits do to correct the corruption.

[Rick Dyson]

Everyone has been busy this morning! I went to bed after the over-night maint and woke to all kinds of interesting activity! I went sprinkling points everywhere. Never easy how to do that, but there were lots of spots to sprinkle.

So, to summarize, it seems like this might be same or similar problem recently seen on v8.x systems (I am v7.3-2 or lower everywhere). And Engineering is apparently aware of it and working on it. I have a service contract, but had not exercised it but wanted to put feelers out to this group first. Seems to have worked!

George: Do you need someone to log a formal service request on this?


I admit I have not yet absorbed everything written in this thread completely, but what I am not clear on is whether I need to run the mentioned Lib/Compress command or not? And, if so, when? Before applying the Update ECO? After continuing thru error? Or some other combo with applying ECO, backing out after error, then compressing, and then re-applying ECO?

Thanks for all your help and info, everyone!

Happy Holidays!
Rick

[Volker Halle · Accepted solution]

Rick,

thanks a lot for reporting this problem. Now the footprint of this problem is documented for everyone to find. This has also alerted the OpenVMS community and even OpenVMS engineering - with a little help ;-)

Also a workaround has been provided:

$ SET DEF SYS$COMMON:[SYSLIB]
$ LIBR/COMPRESS STARLET.OLB

This needs to be executed AFTER installing any patch, which modifes STARLET.OLB (to remove the possible corruption)

OR

BEFORE installing any patch, which may modify STARLET.OLB to prevent the HISTERR during install.

LIBR/COMPRESS just creates a new version of STARLET.OLB with corrected and intact library update history data.

The above has to be done until the underlying problem has been fixed (expect a new set of VMSxxx_LIBRAR-V0n00 patches for OpenVMS Alpha V7.3-2 up to V8.3).

Volker.

[George_145]

If the LIBRARY/COMPRESS is done before installation of a kit that modifies STARLET.OLB then it also has to be done AFTER the installation. The first COMPRESS fixes any existing corruption but installing the kit will re-corrupt the library so the COMPRESS has to be repeated. If it were me, I'd do the pre-install compress and then the post install compress rather than continuing from an installation that got an error. Better history - I don't want to look at an installation log 6 months later see there was an error and wonder if anything was ever done about it.

Engineering is looking at this problem. It is not confirmed yet that it is the same base problem cause.

George Pagliarulo
ECO Release Process
OpenVMS Sustaining Engineering
Hewlett-Packard Company
e-mail: george.pagliarulo@hp.com

[Rick Dyson]

Thank you Volker! That helps a lot. You too George for the suggestiong to keep it as clean as possible.

My primary hospital nodes went smoothly this morning, so whatever lead to this did not happen to them (two node cluster with shared, common system disk). I also had a smooth install to a stand-alone system for another client last night.

The system where I am seeing the problem is one with 3 nodes, all independent system disks. I generally do rolling updates and always start with the old, very slow small DECServer 3000 that I have in the cluster just to have a 3 node cluster. The other two nodes are where work is done and the disks are mounted, etc.

Is there a need of a formal service problem request from me?

Rick

[George_145]

Yes, please file a service request.

On the nodes that did not have a problem, what versions were they ; did they have the ACRTL kit installed or a previous UPDATE kit that included an ACRTL kit? If not, that is why you did not see the problem. The corruption, *if* it is the same, occurs on the first installation but the error is not seen until a second installation tries to update the corrupted STARLET.OLB. So, if you did not see the error, you still have a corrupted STARLET.OLB and need to do the LIBRARY/COMPRESS

George Pagliarulo
ECO Release Process
OpenVMS Sustaining Engineering
Hewlett-Packard Company
e-mail: george.pagliarulo@hp.com

[Rick Dyson]

All of my systems in this case are v7.3-2 with all patches applied prior to Update v14 EXCEPT the LAT v2, which I then skipped since it was listed as being included in Update v14. I also applied Verify v2. Sometimes before Update, sometimes after.

As for ACRTL:

product show history *acrtl*
----------------------------------- ----------- ----------- --------------------
PRODUCT KIT TYPE OPERATION DATE AND TIME
----------------------------------- ----------- ----------- --------------------
DEC AXPVMS VMS732_ACRTL V3.0 Patch Install 08-MAY-2006 00:40:12
DEC AXPVMS VMS732_ACRTL V2.0 Patch Install 09-JAN-2006 01:12:56
DEC AXPVMS VMS732_ACRTL V1.0 Patch Install 13-OCT-2005 16:00:36
----------------------------------- ----------- ----------- --------------------


Since May 2006, I have about 3 screens (~75 lines) of Product installs and removes. Is it plausible all of those installs did not use STARLET? Way outside my knowledge base!

I will get a call logged for this, too.

[John Gillings]

STARLET.OLB causes way more than its fair share of problems for installations and upgrades. It's a pity because although it's frequently referenced (ie: in just about every LINK command), it's very rarely actually used, because most LINK references are resolved from shareable image libraries, rather than the object library. Most systems would run quite happily with a dummy (empty) STARLET.OLB.

Also, don't forget to decompress those libraries! LIBRARY/COMPRESS is a throwback to the days when system disks were sub 100MB, so it was worth the additional CPU to save a handful of blocks. These days it's a silly tradeoff. It sounds like the only useful purpose for /COMPRESS is as a kind of "/VERIFY_AND_REPAIR".

[Jon Pinkley]

@John

$ @SYS$UPDATE:LIBDECOMP REDUCE STARLET.OLB ! DATA compressed with DCX

is not the same as

$ LIBR/COMPRESS STARLET.OLB ! create new version of STARTLET.OLB without deleted members.

The second is similar to MAIL> COMPRESS

Doing a LIBRARY/COMPRESS STARTLET.OLB will not change the way the elements are stored (whether they are REDUCED or EXPANDED).

John's recommendation to use sys$update:libdecomp to decompress the data is valid for libraries that are used. Almost certainly, the help libraries should be expanded. On the other hand, for libraries that are never used, it really makes little difference.

Attached is example log demonstrating the above claims, using the smallest library that LIBDECOMP knows about (NTA.TLB).

Jon

[Volker Halle]

Rick,

$ LIBR/LIS=x.x/HIS/FULL SYS$LIBRARY:STARLET.OLB will show you the Library Update History entries at the tail of the x.x listing file.

If modules had been deleted and replaced you should be able to match the date/time of the history entries with the PROD SHOW HIST dates. This way you can find out, which update has replaced modules in your STARLET.OLB. This may help OpenVMS engineering figure out, which update may have corrupted your STARLET.OLB

Volker.

[Steve Schultz_1]

Rick,
Update V14 does not include LAT v2. If you look at the release notes, you will see it contains LAT v1. The master ECO list does say that the update kit has LAT v2, but this is wrong. After installing the update kit myself, I did verify that LAT v2 was not part of the kit.