STARTUP process hang

[Martin Vorlaender]

Hi,

the STARTUP process on our VMS 7.2-2 +ECOs system hangs, like it seems, after finishing the "normal" startup procedure. The bad consequence is that OPA0: is being blocked by it. After logging in remotely and $STOP STARTUP OPA0: is okay again.

A reboot with STARTUP_P2="VDC" shows:

%SET-I-INTSET, login interactive limit = 160, current interactive value = 0
%STDRV-I-INDXFILE, using component file 1
%STDRV-I-EMTYPHAS, phase LPBEGIN has no entries in file 1.
%STDRV-I-WAITSUB, waiting for subprocess completion(s)...

the last line being repeated again and again.

The funny thing is that our SYSMAN startup database is empty, i.e. not a single entry in LPBEGIN, LPMAIN, LPBETA, or END.

Has anyone seen this behaviour? What to do about it?

cu,
Martin

[Joseph Huber_1]

Martin, the startup database being empty is rather normal (unless You expect to have some layered products to startup).

So the phase after SET-I-INTSET is after SYSTARTUP_VMS execution, and probably spawned DECWindows startup , and that seems to have a problem to finish.
Just a guess, but there usually is no other candidate in this phase.
Have some DECW*.log files showing errors ?
Evtl. do a conversational boot and set
WINDOW_SYSTEM=0 to verify the problem is DECWindow startup.

[Antoniov.]

Martin,
sorry, but I've never seen this behaviour.
I think you can just boot with conversional mode.
On my V7.3-2+ECOs I can read any value you ask for, just for some comparattion when it's possible.

Antonio Vigliotti

[Martin Vorlaender]

Sepp,

as this is a server, we define DECW$IGNORE_DECWINDOWS to TRUE is SYSTARTUP_VMS, so it's unlikely that DECwindows is responsible for this mess. ;-)

As a consequence, there are no DECW processes or logfiles.

cu,
Martin

[Ian Miller.]

check the subprocess count of the STARTUP process. STARTUP.COM is checking this.
$wait_for_subprocess:
$if f$getjpi("","PRCCNT").eq.stdrv$process_count then goto done_subprocess_wait
$if stdrv$verbose then -
stdrv$say f$fao("%STDRV-I-WAITSUB, waiting for subprocess completion(s)...")
$wait 00:00:02
$goto wait_for_subprocess
$done_subprocess_wait:

[Martin Vorlaender]

Ian,

on the next reboot, I'll have a look at STARTUP's sub-processes. I've found that chunk of code in STARTUP.COM, too.

Unfortunately, I don't have the code flow in STARTUP.LOG, as some &$?=* command file did an unconditional SET NOVERIFY :-(

cu,
Martin

[Antoniov.]

Martin,
here there is an AtW question like your trouble
http://h71000.www7.hp.com/wizard/wiz_9645.html
It doesn't appear useful but I hope can give you some clue.

Antonio Vigliotti

[Volker Halle]

Martin,

as you've done a STOP STARTUP, did you check accounting ? If there had been a sub-process, it should have been terminated as well and maybe left a trace in an accounting record.

Volker.

[Phillip Thayer]

Is it possible that your startup process spawned a process to do something and used a /NOWAIT. If so then it if it is not completed by the time your startup completes it will have this kind of problem. Go through all the startup procedures and see if this is the case.

Phil

[Robert Gezelter]

Joseph,

The STARTUP database for the LAYERED products (STARTUP$STARTUP_LAYERED) starts out empty. The STARTUP database for OpenVMS itself (STARTUP$STARTUP_VMS) is far from empty.

Also, consider connecting to the machine with SYSMAN SET ENVIRONMENT instead of remotely logging in, it will work earlier in the STARTUP sequence.

Logs woule be extremely useful, as would full shows of BOTH databases.

You can also use the startup databases as debugging tools themselves by adding files into the various stages.

- Bob Gezelter, http://www.rlgsc.com

[Antoniov.]

Unfortunately, I don't have the code flow in STARTUP.LOG, as some &$?=* command file did an unconditional SET NOVERIFY :-(

Martin,
$ MCR SYSMAN
STARTUP SET OPTION /VERIFY/OUTPUT=FILE
give you a way to see detailed startup.log
Type MCR SYSMAN HELP STARTUp for furthermore information.

Antonio Vigliotti

[Joseph Huber_1]

Bob, yes of course, it is STARTUP$STARTUP_LAYERED which is initially empty. But this is the default database after invoking SYSMAN, and probably what Martin wondered about being empty.

Back to the subprocess waiting: if Martin can't find something spawned explicitely from the startup_vms tree, I still think it's worth to look after DECWindows problems.
DECW$IGNORE_DECWINDOWS being defined, just means DECW$STARTUP is exiting immediately, but it is executed if system parameter WINDOW_SYSTEM is 1.
(I always have WINDOW_SYSTEM=1 on servers, to have all DECW DECW software defined for client applications, and just define DECW$IGNORE_WORKSTATION to inhibit DECW server startup).
DECW$STARTUP deassigns the logical after first call. So check, if it is still defined at the time of the "hang"..
One can also define DECW$IGNORE_SUBPROCESS to see if the behaviour changes.

[Wim Van den Wyngaert]

If I remeber correctly, the quotas of the startup process come from the PQL_M* parameters. Did you lower any of them ? E.g. pql_mprclm ?

Wim

[Uwe Zessin]

Hey, why not take a crash dump and hand it over to Volker?

[Willem Grooters]

Bot a solution, merely a (obvious) hint:



I'd think it is possible to find out what process causes the hang, so that would ease finding the culprit. And as specified, accounting may give some clue as well.

Willem

[Martin Vorlaender]

All,

thanks a lot for all your suggestions - food for thought...

Volker:
thanks for hinting towards accounting

Phillip:
You'd expect me to go through *all* VMS startup procedures??! Mind you, it's not some homegrown startup procedure we're talking about here, it's VMS' SYS$SYSTEM:STARTUP.COM. But thanks for the /NOWAIT hint - I'll have a look.

Bob:
Thanks for the SYSMAN hint. Last time the situation occurred was during a power outage. Unfortunately, this server is our best system, so the quickest to boot, so no chance to have another VMS system with SYSMAN up... (unless I get permission to plug my AlphaStation into the UPS).
I've already plugged some procedures into the layered database to get an idea of what stdrv$process_count looks like.

Antonio:
STARTUP_P2="VDC" is the same as
SYSMAN> STARTUP SET OPTIONS /CHECKPOINTING /OUTPUT=FILE /VERIFY=FULL
so I do have /VERIFY and code flow initially.

Sepp:
I'll follow that road further down, and remove DECW$IGNORE_DECWINDOWS from the startup.

Wim:
AFAIK, no PQL parameters have been lowered (PQL_MPRCLM is 10, PQL_DPRCLM is 32). I'd expect to see some error message in STARTUP.LOG if a process could not be created.

Uwe:
:-) Quite an idea...

Willem:
My wording was poor. The system is up and running (not hanging). The annoying thing is that OPA0: is blocked by the STARTUP process looping.

Again, thanks to all so far. I'll keep you updated (though it could take a while, as I'm going to take my well-earned vacation soon).

cu,
Martin

[Wim Van den Wyngaert]

Martin,

In the phase LPBEGIN you started a number of things thru your startup_vms.com. This created some subprocesses of the STARTUP process. The STARTUP process is waiting until ALL subprocesses have terminated. Since at least one was still active (spawn/nowait ?), it didn't stop waiting.

But you entered STOP STARTUP. So that subprocess MUST have stopped and there should be something in accounting. Did you check ? And operator log file and audit and console output ? Based upon the start time of the process that stopped it must be possible to find out what it was (compare with other accounting info).

Enjoy your holiday. The forecast for Belgium for next week seemd sunny. Hope the same for Germany or whatever country you visit.

Wim

[Martin Vorlaender]

Re: accounting

Indeed I have a subprocess that finished at the same time as the STARTUP process (see attachment) - too bad I'm not doing image accounting. I'll change that...

Thanks, Wim, for the good wishes. I believe Portugal (which is where we're going) could use some of the rain that flooded the Alps...

cu,
Martin

[Antoniov.]

Martin,
your attachment doesn't help but it show your process is waiting form something because processor time is around zero.

Antonio Vigliotti

[Ian Miller.]

the subprocess is a child of the STARTUP process but what was it doing. I guess you have to catch one to see.

[Wim Van den Wyngaert]

Could you post the complete accounting listing starting from the boot ? Gives us an exercise while you're getting overheated in Portugal.

Wim

[Martin Vorlaender]

All,

using the PID from the accounting record and STARTUP.LOG I have narrowed it down to either something in the TCPware, WBEM agent, or AMDS startup.

Too bad the code flow didn't get logged at that point of the startup - BTW, the culprit is syconfig_open3d.com (don't ask me why it's in there; must be a relict of days long gone...)

cu,
Martin

[Martin Vorlaender]

Wim,

here you are. Have fun with it.

cu,
Martin

[Lawrence Czlapinski]

Martin: That sysconf file is for Digiatal Open 3D. 1 Prerequisite Software

To install DIGITAL Open3D on an Alpha workstation, you must
have already installed:

o OpenVMS Alpha Operating System Version 7.1, 7.1-1H1, or
7.1-1H2

o DECwindows[TM] Motif[R] Version 1.2-4 or higher for
OpenVMS Alpha

DIGITAL Open3D also requires the following OpenVMS Alpha classes for full functionality:

o OpenVMS Alpha required save set

o Network support-where full DECwindows network functionality is necessary

o Programming support-necessary to develop programs using OpenGL[R]

o OpenVMS Alpha workstation support

http://www1.aclabs.com/MasterIndex/installation_guide/installation_guide_00a5161c.txt
Lawrence

[Antoniov.]

===[Off topic post]===
Lawrence,
I read your post. I invite you to read here
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=950878

Thank you!

Antonio Vigliotti
===[Please no pts here]===