topic Re: STARTUP process hang in Operating System - OpenVMS

STARTUP process hang

Martin Vorlaender — Mon, 22 Aug 2005 09:15:39 GMT

Hi,

the STARTUP process on our VMS 7.2-2 +ECOs system hangs, like it seems, after finishing the "normal" startup procedure. The bad consequence is that OPA0: is being blocked by it. After logging in remotely and $STOP STARTUP OPA0: is okay again.

A reboot with STARTUP_P2="VDC" shows:

%SET-I-INTSET, login interactive limit = 160, current interactive value = 0
%STDRV-I-INDXFILE, using component file 1
%STDRV-I-EMTYPHAS, phase LPBEGIN has no entries in file 1.
%STDRV-I-WAITSUB, waiting for subprocess completion(s)...

the last line being repeated again and again.

The funny thing is that our SYSMAN startup database is empty, i.e. not a single entry in LPBEGIN, LPMAIN, LPBETA, or END.

Has anyone seen this behaviour? What to do about it?

cu,
Martin

Re: STARTUP process hang

Joseph Huber_1 — Mon, 22 Aug 2005 09:42:43 GMT

Martin, the startup database being empty is rather normal (unless You expect to have some layered products to startup).

So the phase after SET-I-INTSET is after SYSTARTUP_VMS execution, and probably spawned DECWindows startup , and that seems to have a problem to finish.
Just a guess, but there usually is no other candidate in this phase.
Have some DECW*.log files showing errors ?
Evtl. do a conversational boot and set
WINDOW_SYSTEM=0 to verify the problem is DECWindow startup.

Re: STARTUP process hang

Antoniov. — Mon, 22 Aug 2005 09:43:56 GMT

Martin,
sorry, but I've never seen this behaviour.
I think you can just boot with conversional mode.
On my V7.3-2+ECOs I can read any value you ask for, just for some comparattion when it's possible.

Antonio Vigliotti

Re: STARTUP process hang

Martin Vorlaender — Mon, 22 Aug 2005 09:49:29 GMT

Sepp,

as this is a server, we define DECW$IGNORE_DECWINDOWS to TRUE is SYSTARTUP_VMS, so it's unlikely that DECwindows is responsible for this mess. ;-)

As a consequence, there are no DECW processes or logfiles.

cu,
Martin

Re: STARTUP process hang

Ian Miller. — Mon, 22 Aug 2005 09:51:15 GMT

check the subprocess count of the STARTUP process. STARTUP.COM is checking this.
$wait_for_subprocess:
$if f$getjpi("","PRCCNT").eq.stdrv$process_count then goto done_subprocess_wait
$if stdrv$verbose then -
stdrv$say f$fao("%STDRV-I-WAITSUB, waiting for subprocess completion(s)...")
$wait 00:00:02
$goto wait_for_subprocess
$done_subprocess_wait:

Re: STARTUP process hang

Martin Vorlaender — Mon, 22 Aug 2005 09:55:34 GMT

Ian,

on the next reboot, I'll have a look at STARTUP's sub-processes. I've found that chunk of code in STARTUP.COM, too.

Unfortunately, I don't have the code flow in STARTUP.LOG, as some &$?=* command file did an unconditional SET NOVERIFY :-(

cu,
Martin

Re: STARTUP process hang

Antoniov. — Mon, 22 Aug 2005 10:08:51 GMT

Martin,
here there is an AtW question like your trouble
http://h71000.www7.hp.com/wizard/wiz_9645.html
It doesn't appear useful but I hope can give you some clue.

Antonio Vigliotti

Re: STARTUP process hang

Volker Halle — Mon, 22 Aug 2005 11:44:22 GMT

Martin,

as you've done a STOP STARTUP, did you check accounting ? If there had been a sub-process, it should have been terminated as well and maybe left a trace in an accounting record.

Volker.

Re: STARTUP process hang

Phillip Thayer — Mon, 22 Aug 2005 13:07:13 GMT

Is it possible that your startup process spawned a process to do something and used a /NOWAIT. If so then it if it is not completed by the time your startup completes it will have this kind of problem. Go through all the startup procedures and see if this is the case.

Phil

Re: STARTUP process hang

Robert Gezelter — Mon, 22 Aug 2005 13:46:27 GMT

Joseph,

The STARTUP database for the LAYERED products (STARTUP$STARTUP_LAYERED) starts out empty. The STARTUP database for OpenVMS itself (STARTUP$STARTUP_VMS) is far from empty.

Also, consider connecting to the machine with SYSMAN SET ENVIRONMENT instead of remotely logging in, it will work earlier in the STARTUP sequence.

Logs woule be extremely useful, as would full shows of BOTH databases.

You can also use the startup databases as debugging tools themselves by adding files into the various stages.

- Bob Gezelter, http://www.rlgsc.com

Re: STARTUP process hang

Antoniov. — Tue, 23 Aug 2005 02:11:22 GMT

Unfortunately, I don't have the code flow in STARTUP.LOG, as some &$?=* command file did an unconditional SET NOVERIFY :-(

Martin,
$ MCR SYSMAN
STARTUP SET OPTION /VERIFY/OUTPUT=FILE
give you a way to see detailed startup.log
Type MCR SYSMAN HELP STARTUp for furthermore information.

Antonio Vigliotti

Re: STARTUP process hang

Joseph Huber_1 — Tue, 23 Aug 2005 03:27:48 GMT

Bob, yes of course, it is STARTUP$STARTUP_LAYERED which is initially empty. But this is the default database after invoking SYSMAN, and probably what Martin wondered about being empty.

Back to the subprocess waiting: if Martin can't find something spawned explicitely from the startup_vms tree, I still think it's worth to look after DECWindows problems.
DECW$IGNORE_DECWINDOWS being defined, just means DECW$STARTUP is exiting immediately, but it is executed if system parameter WINDOW_SYSTEM is 1.
(I always have WINDOW_SYSTEM=1 on servers, to have all DECW DECW software defined for client applications, and just define DECW$IGNORE_WORKSTATION to inhibit DECW server startup).
DECW$STARTUP deassigns the logical after first call. So check, if it is still defined at the time of the "hang"..
One can also define DECW$IGNORE_SUBPROCESS to see if the behaviour changes.

Re: STARTUP process hang

Wim Van den Wyngaert — Tue, 23 Aug 2005 05:10:36 GMT

If I remeber correctly, the quotas of the startup process come from the PQL_M* parameters. Did you lower any of them ? E.g. pql_mprclm ?

Wim

Re: STARTUP process hang

Uwe Zessin — Tue, 23 Aug 2005 05:30:12 GMT

Hey, why not take a crash dump and hand it over to Volker?

Re: STARTUP process hang

Willem Grooters — Wed, 24 Aug 2005 01:44:54 GMT

Bot a solution, merely a (obvious) hint:

I'd think it is possible to find out what process causes the hang, so that would ease finding the culprit. And as specified, accounting may give some clue as well.

Willem

Re: STARTUP process hang

Martin Vorlaender — Wed, 24 Aug 2005 02:28:43 GMT

All,

thanks a lot for all your suggestions - food for thought...

Volker:
thanks for hinting towards accounting

Phillip:
You'd expect me to go through *all* VMS startup procedures??! Mind you, it's not some homegrown startup procedure we're talking about here, it's VMS' SYS$SYSTEM:STARTUP.COM. But thanks for the /NOWAIT hint - I'll have a look.

Bob:
Thanks for the SYSMAN hint. Last time the situation occurred was during a power outage. Unfortunately, this server is our best system, so the quickest to boot, so no chance to have another VMS system with SYSMAN up... (unless I get permission to plug my AlphaStation into the UPS).
I've already plugged some procedures into the layered database to get an idea of what stdrv$process_count looks like.

Antonio:
STARTUP_P2="VDC" is the same as
SYSMAN> STARTUP SET OPTIONS /CHECKPOINTING /OUTPUT=FILE /VERIFY=FULL
so I do have /VERIFY and code flow initially.

Sepp:
I'll follow that road further down, and remove DECW$IGNORE_DECWINDOWS from the startup.

Wim:
AFAIK, no PQL parameters have been lowered (PQL_MPRCLM is 10, PQL_DPRCLM is 32). I'd expect to see some error message in STARTUP.LOG if a process could not be created.

Uwe:
:-) Quite an idea...

Willem:
My wording was poor. The system is up and running (not hanging). The annoying thing is that OPA0: is blocked by the STARTUP process looping.

Again, thanks to all so far. I'll keep you updated (though it could take a while, as I'm going to take my well-earned vacation soon).

cu,
Martin

Re: STARTUP process hang

Wim Van den Wyngaert — Wed, 24 Aug 2005 05:15:29 GMT

Martin,

In the phase LPBEGIN you started a number of things thru your startup_vms.com. This created some subprocesses of the STARTUP process. The STARTUP process is waiting until ALL subprocesses have terminated. Since at least one was still active (spawn/nowait ?), it didn't stop waiting.

But you entered STOP STARTUP. So that subprocess MUST have stopped and there should be something in accounting. Did you check ? And operator log file and audit and console output ? Based upon the start time of the process that stopped it must be possible to find out what it was (compare with other accounting info).

Enjoy your holiday. The forecast for Belgium for next week seemd sunny. Hope the same for Germany or whatever country you visit.

Wim

Re: STARTUP process hang

Martin Vorlaender — Wed, 24 Aug 2005 07:22:48 GMT

Re: accounting

Indeed I have a subprocess that finished at the same time as the STARTUP process (see attachment) - too bad I'm not doing image accounting. I'll change that...

Thanks, Wim, for the good wishes. I believe Portugal (which is where we're going) could use some of the rain that flooded the Alps...

cu,
Martin

Re: STARTUP process hang

Antoniov. — Wed, 24 Aug 2005 07:44:24 GMT

Martin,
your attachment doesn't help but it show your process is waiting form something because processor time is around zero.

Antonio Vigliotti

Re: STARTUP process hang

Ian Miller. — Wed, 24 Aug 2005 09:15:12 GMT

the subprocess is a child of the STARTUP process but what was it doing. I guess you have to catch one to see.