Securing the console port on an ES47

[Volker Halle]

John,

analysis of forced crashes is a 'art by itself' ! You MUST find an operation in the system, which is 'hanging', typically the LOGIN attempt of the system manager on the console terminal. Then you can work from that 'hanging' operation and find out what's causing this. It needs a lot of experience and patience to dig through a forced cash and find the problem - if there was one !

Your colleague, who had forced that crash, certainly had a reason for that. Which operation did he try ? And what did 'not happen', when he tried it ?

Volker.

[John A. Beard]

If by forced crash you are refering to the one on the 17th, then this is what occured.

After receiving the call that users could neither access existing connections or initiate new ones, my colleague connected to the console port via MBM. On this particular occasion (sometimes we cannot get this far)he was able to actually see the >>> prompt and he initiated a CRASH command. After writing out to the file the server then proceded to boot.

[Volker Halle]

John,

what is the setting of the console environment variable AUTO_ACTION on your servers ?

If it's set to HALT (it should be RESTART), then it can happen that the system just HALTS, i.e. drops to the console prompt by itself for some kinds of errors (machine check in PAL mode, kernel mode HALT, etc.) ! Then nothing would work, you connect to the console and 'see' the console prompt >>> ... If you then type CRASH, it will look like an operator forced crash and the problem itself is masked out.

You should strongly consider to use some toll to ALWAYS RECORD the console output of your systems: Console Manager, Console Works or similar.

Art,

is AMS enable to record console output ?

Volker.

[Volker Halle]

John,


and that overrides the console setting of HALT


So I bet AUTO_ACTION is actually set to HALT on your systems.

This is NOT the suggested setting, as you will completely loose ALL information, if your systeme unexpectedly halts. If AUTO_ACTION is set to RESTART, the console will try to restart the CPU, OpenVMS will detect this is a restart attempt after an unexpected HALT, write a crash and reboot. This will cause all of memory to be saved (inclduing possible errlog entries).

So to further diagnose these kind of problems, you MUST set AUTO_ACTION to RESTART and/or record ALL console output from all systems.

Volker.

[Art Wiens]

"Art,

is AMS enable to record console output ?

Volker. "

Yes, if it's enabled to do so. Attached is an example of how it can be configured, and overlayed with what you can see if you choose "View Console Log ...".

Cheers,
Art

[John A. Beard]

Hi Volker,

Your are correct in your assumption, the restart control is set to HALT on all the boxes.

We did this on the strngth of bing told by HP some time ago that to capture valid dump info we needed to have the server in question remain at the chevron prompt and from there issue a CRASH command whenever the problem occured.

I'm probably wrong on this, but I believe this course of action was recommended because otherwise we would loose critical information if the server simply rebooted.

We have had over 40 such instances in the last 16 months, and as originally stated, nobody appears to know whether or not this can be attributed hardware, software or a combination of both.

Alowing for all these possibilities, are you saying that if we reset the System Restart Control to RESTART then the system will detect the existance of a problem, write out to the crash dump file (with more meaningful information)and perform a full reboot.

I mentioned before that sometimes when we try and connect through MBM we cannot see anything at all. In those cases we have performed a POWER OFF/ON from MBM. In such cases I assume we are up the creek without a paddle in terms of capturing any meaningful information as to what was going on in the system at the time the problem occured.

I still have to check up on Console Manager/Works, etc...I haven't had the time to do it yet.

John

[Volker Halle]

John,

as we don't yet know, what the problem really is, it is MOST important to capture ALL console output from those systems.

The easiest way is this: on your 'favourite' operating system, start a couple of Putty sessions and log on to all those ES47 consoles. Turn ON Putty session logging into a file with a unique name for each Putty window, name the Putty windows accordingly. Do all this via RDP (Remote Desktop) and just disconnect that RDP-session, do NOT log out, as this will kill all the Putty Windows.

Then when a problem happens, log on to this 'poor man's console manager system' (via RDP) and look at the console of the system, doing whatever is necessary to capture information and restart the ES47.

Using a real console manager product (AMS etc.) would be preferred, but needs some time to set up.

Regarding AUTO_ACTION: the worst thing you could set this to is: REBOOT - as this will destroy ALL possible information and leave no trace whatsoever.

The next best thing is: RESTART, as this will automatically capture a dump and reboot the server.

Only set it to HALT, if you know of any commands, that need to be executed manually, if the problem happens. If you just type >>> CRASH, setting RESTART is much better.

Volker.

[John A. Beard]

...jut a quick update to inform everybody following this thread with bated breath, it looks like there is some serious action being planned by the good folks in the backline team.

Because these events have been of a random nature it might be a while before I can report back here on further progress. We have already set AUTO_ACTION to RESTART on the 4 test and development servers but being a 24x7 shop, getting access to the other production boxes may take a couple of weeks.

Until suct time as I can provide you with further information I would just want to once again thank everyone for thir time and input into this thread....as always, much appreciated.

[UpstateRob]

We also had many unexplained Crashes that started right after I put the MBM on the general network. You cannot do it; the only way to make the MBM available is by taking the ethernet cable and plugging it directly into the back of something else, whether it is a real Alpha management station or unix box with extra ethernet cards, or as we are going to try, a Windows Server with lots of extra ports.