Rolling upgrade V7.3-2 to V8.3 problems

[Jan van den Boogaard]

Hi all.

I am interested in the experience of people in this forum on doing a rolling upgrading 7.3-2 Alpha to 8.3 Alpha. I myself am running into crashes.
I upgrade a ES45 from a single system disk on SAN to 8.3. Another node in the cluster keeps on running 7.3-2 on a shadowed system disk in the SAN in the meantime.
After the upgrade (which itself is succesful), on the first boot from 8.3 it runs into ACCVIO.
(see attachment)

The upgrading node does not mount any SAN disk mounted by the other node.

Any ideas what might be wrong? Is this scenario unsupported? Thnx, Jan.

[Jan van den Ende]

Jan,

rather little to go on so far, but
>>>
virtual address=0000000000000000
<<<
pretty much indicates addressing something for which the address has NOT been filled in.

Usually the info should include the name of the image to be forced to exit!

Without any more info to go on, just MAYBE
>>>
The operator console and logfile will not be enabled.
Change OPC$OPA0_ENABLE & OPC$LOGFILE_ENABLE in SYLOGICALS.COM
to enable them.
<<<
indicates that you ARE trying to do some addressing on a not-yet-mounted drive?

Is this system disk using the STK setup?
Can you try a boot with STK disabled
(boot conversational, set STARTUP_P1 to "MIN", and continue. Log in as SYSTEM, and try to run every step of the bootstrap one-by-one by hand. See what happens)

hth

Proost.

Have one on me.

jpe

[Volker Halle]

Jan,

the first question to be answered is: which image is getting the ACCVIO ? Then find out the command invoking that image and try to guess, what this image might be trying to do and why it may be failing.

Boot conversational and enable full verify during STARTUP.COM:

>>> b -fl n,1 (with n=system root)
SYSBOOT> SET STARTUP_P2 "VPC"
SYSBOOT> CONT

Volker.

[Jan van den Ende]

Volker,

agreed!

>>>
SYSBOOT> SET STARTUP_P2 "VPC"
<<<

Jan, do that, but MAKE SURE you have output recording with sufficient buffer hooked to the console!

It might well be, that even a STARTUP_P1 "MIN" will crash even before you can start single startup steps, which is where my suspicion goes because of the missing image name.

The P2 setting will also trap those.

Proost.

Have one on me.

jpe

[Anton van Ruitenbeek · Accepted solution]

Jan,

To do a supported rolling upgrade you must have the original STARTUP and corresponding SYSTARTUP procedures. You may only modify the SYS$SYSTARTUP etc. If you modify the procedures whitch may not modified and during upgrade these procedures are modified by HP there is another chalenge.
Make sure your systemarchitecture will support rolling upgrades.

AvR

[James Cristofero]

Appears something may have been over-wriiten with the upgrade.

Like Anton said, upgrade could have move a new S*.com into your startup stream. Looks like between SYCONFIG and SYLOGICALS.

Do you have the correct SYCONFIG? A Converation boot or boot from CD will allow you to examine the contents. Someone may have left something in the SYS$SYSROOT that may not have made it over, or been superceded.

Also, if a SAN environment, sometimes I've seen the device not be "readY" by the time SYCONFIG runs and you have to put a loop checking for the device either in SYCONFIG or SYLOGICALS (depending on speed of precessor).

[Robert_Boyd]

I had a problem with LANACP during an upgrade to V8.3. HP has all the particulars about my case.

My upgrade would not complete successfully because on the reboot to run AUTOCONFIGURE the system would crash when it tried to fire up LANACP.

I wound up doing numerous things to debug/work around this problem. I finally succeeded in booting the system in interactive mode and set/startup OPA0:

Once the system was up far enough to bring up the STARTUP process I was able to run LANCP and force a convert on the database files BEFORE LANACP could be started. That turned out to be the thing that helped me turn the corner.

I then put the INS_STARTUP.COM back as the startup procedure and rebooted to let it finish its reconfigure and final reboot.

I also discovered that since then, there is a patch kit available. If you should have the luxury of building a hard drive replica of the install CD, you can place the necessary patch kits on the drive before beginning the upgrade. This makes it possible to install the base system and use the new feature for locating and installing patch kits to bring the system past this wretched problem before it ever reboots.

I like to create a directory tree
[PATCHES.1ST]
[PATCHES.2ND]
[PATCHES.REMAINDER]
This way if there are groups of patches that must be installed in a particular order it's easier to do the right thing.

Robert

[Jan van den Boogaard]

Thanks for all hints, I will try 'm out soon as when i get back to work in the morning.
Could it also maybe have something to do with MSCP_LOAD parameter as VMS seems to crash there ? Jan

[Peter Zeiszler]

Is your 8.3 shadow system disk a different unit number than your 7.3-2 shadow disk?

[Volker Halle]

Jan,

the register printout is from an IMAGE incurring an improperly handled condition. The MSCP message is just the most recent message printed by STARTUP.COM before this happens. The OPC$* message is printed after executing SYLOGICALS.COM. The unexpected image termination is anywhere inbetween.

Volker.

[John Abbott_2]

FWIW I upgraded a GS80 on a SAN from V7.3-2 (fully patched) to V8.3 + UPDATE V2 and all patches listed in the master ECO list, excluding LAN V1 & ACMELDAP V2 no problems before for after (well, 7 days and counting...)

Regards
John.

[Jan van den Boogaard]

I did the startup_p1 MIN and startup_p2 VPC, see output attached, but then it didnt crash.
I believe Anton is right. I suspect highly the site specific startup stuff, so I'll focus on that for the moment.
Thnx sofar.

[Jan van den Ende]

Jan,

next time you post a .LOG file, please rename to .TXT before postin ( the PITD firewall does not even allow lowercase .txt extension!)
Some (at least my) browser does NOT know that .LOG means ASCII, and after binary transfer it takes some effort to read it.

So now, the next try will be with STARTUP_P1 empty, and STARTUP_P2 again for full output?

Success.

Proost.

Have one on me.

jpe

[Jan van den Boogaard]

Found the location of the ACCVIO.

The commands SHOW SHADOW and SHOW SHADOW /POLICY=HBMM cause the accvio. The command SHOW SHADOW/POLICY=HBMM/NAME=DEFAULT does not give an ACCVIO, strangely enough.

Those commands were somewhere in a sitespecific startup. So:
1) yes, cause was in the site startup.
2) the command crashing is SHOW SHADOW

It might have something to do with upgrading from a 7.3-2 system with HBMM kit installed to 8.3. I didnt have these problems on a system that I installed straight from 8.3 DVD, and using the same site startup....

so the problem is little more pinpointed now.

[Jan van den Ende]

Jan,

Great!.

So now the issue is:

WHY does the upgraded disk crash, where the fresl installed does NOT.

What version/patch level of HBMM is/was on the 7.3-2 system?

Is there any difference in SYSGEN SHD* & SHAD* params?

Proost.

Have one on me.

jpe

[Jan van den Ende]

Jan,

Great!

So now the issue is:

WHY does the upgraded disk crash, where the fresl installed does NOT.

What version/patch level of HBMM is/was on the 7.3-2 system?

Is there any difference in SYSGEN SHD* & SHAD* params?

Proost.

Have one on me.

jpe

[Robert Brooks_1]

From talking with the shadowing engineer a couple of weeks ago, this is a known problem. He knows exactly what the issue is, and how to fix it.

I'd formally contact your HP support centre to make the problem resolution process moves along quickly.


-- Rob

[Jon Pinkley]

Jan van den Boogaard,

Several questions.

Please verify that there are no 7.3-2 versions of images in the sys$specific directories. Under some circumstances an image gets placed in the sys$specific area instead of the sys$common area, and the upgrade only places things in the sys$common area. In this case, after the upgrade most images will not be found in the sys$specific area (intended behavior), but the images that do exist in the specific directory will be used instead of the ones in the common directory.

What do you get from the following command?

$ directory sys$system:setshoshadow/date/size=all

If you see two flles, one in sys$specific and one in sys$common, you should determine the version of each. If the one in sys$specific is an older version (perhaps from a patch), you should rename it to something like .exe_old and then reboot.

You can determine the image ident with the following command:

$ analyze/image/select=(ident,link) sys$system:setshoshadow

or for sys$common version

$ analyze/image/select=(ident,link) sys$common:[sysexe]setshoshadow

You should still open a call as suggested by Robert Brooks.

Jon

[Thomas Ritter]

While this answer may not help, I thought I would comment on the way we perform upgrades. We NEVER perform rolling upgrades on major upgrades. Always a new empty disk and complete install from scratch. This is an approach adopted because our outage windows are very very small and because it works so well. We also design our systems so that the system disks are basically read only. Our layered products installations and patch installations are always in batch mode. This way when two or more administrators are performing upgrades, absolute certainly is assured. No patch or product installation will be missed.

[Robert Brooks_1]

Unless the problem you've discovered is quite different from the one that's already known by the guy within VMS Engineering who deals with shadowing, it's likely that you have done nothing wrong with your upgrade. The SETSHOSHADOW image is not particularly dependent on any of the shadowing SYSGEN params. As Jon Pinkley pointed out, an image incorrectly placed in SYS$SPECIFIC, as opposed to SYS$COMMON:, would be a bad deal, as that image is quite dependent on shadowing-specific data structures that do change from version to version, and it's almost a guarantee that a V7.3-2 image would be really unhappy on a V8.3 system.

The point of the above is to suggest that spending any time figuring out what went wrong is not likely to be worth your time; it's likely that the fault is ours.

-- Rob

[Jan van den Boogaard]

I will formally contact the local HP support centre, thanx Robert. That seems the most obvious to do now.

To answer the other questions:

$ directory sys$system:setshoshadow/date/size=all

Directory SYS$COMMON:[SYSEXE]

SETSHOSHADOW.EXE;1 603/604 29-JUN-2006 18:19:09.69

Total of 1 file, 603/604 blocks.
$

so just 1 image luckily.

The HBMM kit is VMS732_HBMM-V0200 from september 2004 which is in the VMS732_UPDATE-V1100 consolidated kit.

And Thomas, a non-rolling upgrade could be more safe, but the concept of rolling upgrade is so nice and tempting: upgrading easily while users can keep on working.
By the way, what you mean by: "Our layered products installations and patch installations are always in batch mode." ?

[Thomas Ritter]

Jan, this link shows how one can batch patches and installs. Some products can only be installed interactively. We run 4 system disks in our four node cluster and one so called common disk. If you have many patches to install the batch approach makes work easy.

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1073507

Here is an example of installing a patch in batch mode. Notice we check to ensure the system account is used.

$ username = f$edit(f$getjpi("","USERNAME"),"COLLAPSE")
$ if username .nes. "SYSTEM"
$ then
$ write sys$output "Use the SYSTEM account."
$ exit 1
$ endif
$ saved_directory = f$environment("DEFAULT")
$ set default kits73:[vms731_ecos]
$ DEFINE/SYS NO_ASK$BACKUP TRUE
$ DEFINE/SYS NO_ASK$REBOOT TRUE
$ DEFINE/JOB ARCHIVE_OLD NO
$!
$ PROD INSTALL VMS731_SHADOWING/PRODUCER=DEC/BASE=AXPVMS-
/VER=V2.0/SAVE_RECOVERY_DATA
$!
$ DEASSIGN/SYS NO_ASK$BACKUP
$ DEASSIGN/SYS NO_ASK$REBOOT
$!
$ set def 'saved_directory'
$ prod show history
$ exit 1

[Jan van den Boogaard]

There was a small bug and a workaround from HP.

Before the upgrade, on the 7.3-2 side, remove the HBMM policies with these kind of commands:
$ set shadow /policy=HBMM=(master=*) /name=DEFAULT /delete

and
$ set shadow /disable=hbmm dsa
$ set shadow /policy=hbmm /delete dsa

Check with $ show log/table=h* that no logicals remain.
Then do a rolling upgrade of the first node to 8.3 and re-define the default policy on 8.3:

$ set shadow /policy=HBMM=(master=*) /name=DEFAULT

Then boot the other nodes from the upgraded system disk.

I tested it, problems solved!

[Jan van den Boogaard]

Thanx for all help.