SYSTARTUP_VMS runs twice during reboot

[Russ Carraro]

We have an Alpha 2100 running OpenVMS 7.1, which when booting, after running systartup_vms.com procedure runs it again. This is the end of systartup_vms.com:

$ say " Starting DECevent V3.4..."
$ @SYS$STARTUP:DECEVENT$STARTUP.COM
$ say ""
$!
$ say " Set Interactive Logins..."
$ startup$interactive_logins == 200
$ say ""
$!
$ exit

Here's the console log after the first running:

Starting DECevent V3.4...
Starting: DSA0:[SYS0.DIA$TOOLS.][MANAGER]DECEVENT_START.TMPCOM; UIC:[DIA$MANAG
]
/JOB_=10000/AST_=325 /BUFF=100000/ENQU=2000/EXTE=20480/FILE=100 /IO_B=80/IO_D=
0/MAXI=10240/PAGE=130000/QUEU=20 /SUBP=12/SWAP/WORK=8192/PRIV=ALL/PROCESS_NAME
ECEVENT_EVFMS1
$ run sys$system:loginout -
/uic=[DIA$MANAGER] -
/input=DSA0:[SYS0.DIA$TOOLS.][MANAGER]DECEVENT_START.TMPCOM; -
/output=DSA0:[SYS0.DIA$TOOLS.MANAGER]DECEVENT_START.LOG -
/JOB_=10000/AST_=325 /BUFF=100000/ENQU=2000/EXTE=20480/FILE=100 /I
B=80/IO_D=100/MAXI=10240/PAGE=130000/QUEU=20 /SUBP=12/SWAP/WORK=8192/PRIV=ALL/
OCESS_NAME="DECevent_EVFMS1"
%RUN-S-PROC_ID, identification of created process is 6120028E

Set Interactive Logins...

%SET-I-INTSET, login interactive limit = 200, current interactive value = 0
%DCL-S-SPAWNED, process SYSTEM_1 spawned
%DCL-I-SUPERSEDE, previous value of DECW$USEXLIBXPG4 has been superseded
NODE == "EVFMS1"
********************************************
Mounting production CLUSTER COMMON disks...


After the second running:

$ run sys$system:loginout -
/uic=[DIA$MANAGER] -
/input=DSA0:[SYS0.DIA$TOOLS.][MANAGER]DECEVENT_START.TMPCOM; -
/output=DSA0:[SYS0.DIA$TOOLS.MANAGER]DECEVENT_START.LOG -
/JOB_=10000/AST_=325 /BUFF=100000/ENQU=2000/EXTE=20480/FILE=100 /I
B=80/IO_D=100/MAXI=10240/PAGE=130000/QUEU=20 /SUBP=12/SWAP/WORK=8192/PRIV=ALL/
OCESS_NAME="DECevent_EVFMS1"
%RUN-F-CREPRC, process creation failed
-SYSTEM-F-DUPLNAM, duplicate name

Set Interactive Logins...

%SET-I-INTSET, login interactive limit = 200, current interactive value = 3
SYSTEM job terminated at 20-JUL-2010 20:43:46.98

Accounting information:
normal info

Any ideas what's causing the extra running is appreciated. Thanks.

[Hoff · Accepted solution]

Turn on the debugging via the STARTUP_P2 system parameter, and log the startup. There are some details on that toward the end of this page:

http://labs.hoffmanlabs.com/node/192

And also search all files on your system disk for references to the startup here:

SEARCH SYS$SYSDEVICE:[*...]*.*/WINDOW=0 SYSTARTUP_VMS

The above SEARCH intentionally searches everything, and you'll have to look at and decide about each.

It's unlikely, but I've also seen errors from "creative" settings in the specified startup:

SYSMAN> SHOW /STARTUP

which should be SYS$SYSTEM:STARTUP.COM

And OpenVMS V7.1 begat many fixes, enough of which led to the release of V7.1-2; you'll want to start loading (all of) the patches.

[John Gillings]

Russ,

As you know, a normal startup runs only once, so if it's executing a second time, something else must be calling it (other than the startup.

Normally SYSTARTUP_VMS is called from VMS$LPBEGIN-050_STARTUP. You can see the full seqence of procedures with:

SYSMAN> STARTUP SET DATABASE STARTUP$STARTUP_VMS
SYSMAN> STARTUP SHOW FILE

As Hoff suggests, check for calls from other procedures, but also consider that the repeated lines could possibly have been accidently pasted into another file (maybe SYLOGICALS.COM?), so maybe check that theory by searching for DECEVENT$STARTUP in other files.

I'd add a heading to SYSTARTUP_VMS:

$ self=F$PARSE(";",F$ENVIRONMENT("PROCEDURE"))
$ name=F$PARSE(self,,,"NAME")
$ node=F$GETSYI("NODENAME")
$ now=F$CVTIME(,"ABSOLUTE")
$ WRITE SYS$OUTPUT F$FAO("!AS executing on !AS at !AS",name,node,now)

You could add other debugging information, for example, F$ENVIRONMENT("DEPTH")

Another trick would be to add SET VERIFY at the END of the procedure, that way you'll see the remainder of the calling procedure verified.

Worst case, if you really can't figure out where it's coming from, separate the actions in the procedure which can be repeated from those which must only happen once. Use a process logical name to leave a breadcrumb:

$ IF .NOT.F$TRNLNM(name+"_DONE")
$ THEN
$ ! stuff which must only be done once
$ ...
$ DEFINE 'name'_DONE TRUE
$ ENDIF

(note - I'm using symbol "name" in the above code so you can cut and paste the code into other procedures without having to change it).

[Robert Gezelter]

Russ,

Having done many things with SYSTARTUP_VMS.COM and other components of the STARTUP process over the years, I would strongly agree with Hoff's comment about the trace. Something is most likely invoking SYSTARTUP_VMS.COM a second time, the trace will indicate precisely where.

For thoroughness, I would also suggest listing both startup databases using SYSMAN STARTUP SHOW FILE (I apologize for the lack of precision, the hotel I am in at the instant solely allows HTTP/HTTPS, I cannot get to my systems to verify the precise command).

- Bob Gezelter, http://www.rlgsc.com

[Russ Carraro]

Thanks Hoff, John and Robert.

I've searched everything for startup_vms and only found it in VMS$LPBEGIN-050_STARTUP.

The startup database looks good, too:
Phase Mode File
------------ ------ ---------------------------------
INITIAL DIRECT VMS$INITIAL-050_VMS.COM
INITIAL DIRECT VMS$INITIAL-050_LIB.COM
INITIAL CALLED VMS$INITIAL-050_CONFIGURE.COM
DEVICES DIRECT VMS$DEVICE_STARTUP.COM
PRECONFIG DIRECT IPC$STARTUP.COM
PRECONFIG DIRECT VMS$SPIRALOG_STARTUP.COM
CONFIG DIRECT VMS$CONFIG-050_VMS.COM
CONFIG DIRECT VMS$CONFIG-050_ERRFMT.COM
CONFIG DIRECT VMS$CONFIG-050_CACHE_SERVER.COM
CONFIG DIRECT VMS$CONFIG-050_CSP.COM
CONFIG DIRECT VMS$CONFIG-050_OPCOM.COM
CONFIG DIRECT VMS$CONFIG-050_AUDIT_SERVER.COM
CONFIG DIRECT VMS$CONFIG-050_JOBCTL.COM
CONFIG DIRECT VMS$CONFIG-050_LMF.COM
CONFIG DIRECT VMS$CONFIG-050_SHADOW_SERVER.COM
CONFIG DIRECT VMS$CONFIG-050_SECURITY_SERVER.COM
BASEENVIRON DIRECT VMS$BASEENVIRON-050_VMS.COM
BASEENVIRON DIRECT VMS$BASEENVIRON-050_SMISERVER.COM
BASEENVIRON DIRECT VMS$BASEENVIRON-050_LIB.COM
BASEENVIRON DIRECT DECDTM$STARTUP.COM
BASEENVIRON DIRECT LICENSE_CHECK.EXE
LPBEGIN DIRECT VMS$LPBEGIN-050_STARTUP.COM


I've added some "debug" commands to the startup and hopefully I can get more info the next time the system boots. Unfortunately it's a 24x7x365 mfg. cluster so it may take several months.

Thanks, again, for your feedback.

[Verne Britton]

Ignoring any licensing issues for a moment, from a purely technical point of view, testing this would be perfect for Personal Alpha ...

that is, configure PA to boot into your cluster, then play with it rebooting all day long (assuming the cluster state transition pauses can be tolerated).

Verne

[Hoff]

DECwindows that far back could do some screwy stuff.

Given your environmental constraints, I'd not bother with the investigation. At most, I'd place a set-up test (eg: test for a logical name that a previous start would have created) at the top of the startup to detect a previous run, and exit.

You're running an ancient version (January 1997), and something that most folks upgraded off of a decade or more ago (because V7.1-2, December 1998, fixed a boat-load of bugs in V7.1). Those that were prohibited from an upgraded ended up installing a said boat-load of patches individually.

Given you're running production, I'm mildly surprised you're going after this sort of stuff at all. The sequences I'm familiar with at most sites will usually have a test configuration running an exact clone of the main environment (sometimes exclusive of the control systems, sometimes with emulated control). And my preference: you don't mess with production, save through a release process, and the release process keeps production and test in synch.

Alpha emulation might be an option here, depending on what I/O and system hardware is required. Folks on this old a version can be using it for many reasons, and one of which can be specific hardware needed as part of production. If you're not tied to the hardware, then emulation is an option (for testing) and used Alpha hardware is cheap these days, and used and VMS-capable HP Itanium boxes are regularly well under US$1000. (Getting to Itanium is a larger project but - with the exception of the cluster license - the licenses and hardware are all far less costly than Alpha was at a comparable time in its product life.)

My immediate suspect on that extra invocation (on no evidence) would be DECwindows. But without a trace of the startup, it's hard to say exactly what's going on here.

[Russ Carraro]

Thanks Verne and Hoff,

A personal Alpha would be nice, but even if I had one the customer would probably nix the testing.

These are legacy clusters that the customer has running at several locations around the country. They investigated migrating several years ago but it couldn't be done for several reasons, so Alphas running 7.1 (a couple 7.1-1H2) is were it stays.

[Wim Van den Wyngaert]

2 files exists that define what has to be executed in each phase. The first one
(is also executed before the second one) is called sys$startup:vms$vms.dat and
the second one sys$startup:vms$layered.dat. For each phase found in the phases
file, the startup actions are searched in each of these files.

1. Did you check the laayered file too ?
2. Is there may be a forgotten "on error" active that moves the execution back to the beginning of the startup ?
3. Did you did a search in *.* (and not *.com) as Hoff said ?
4. May be some startup files are not on the system disk. Do a search on these other disks too.

Wim

[Robert Gezelter]

Russ,

I concur with Wim, it would also be useful to check the STARTUP$STARTUP_LAYERED database also (both databases are used during the startup). Since there is no guard against a file being listed in BOTH databases, a possible error would be for someone to have entered a file in two places.

As to the use of Virtual Alpha and similar tools, I would suggest a more benign alternative than adding a member to the production cluster. Personal Alpha can also be used to create a virtual "play" doppelganger of the production environment on an isolated network. This "play" environment can be used to exercise many contingencies that involve risk to the production cluster, without any risk in the real world (e.g., airlines do most of the "challenging" training scenarios in full motion simulators; having a mishap in a simulator is perhaps embarrassing, but it does not represent a danger to life, limb, hardware, or innocent bystanders).

Such an environment has many, many uses. It also increases the certainty of successful emergency responses on the real production cluster.

- Bob Gezelter, http://www.rlgsc.com