Securing the console port on an ES47

[John A. Beard]

Hi,

We want to prevent (as part of a test) users from being able to issue a CTL/P command from the console port on any of our ES47s (2P Drawer Control Panel). We currently connect to the console via MBM. Drawer 0 has had it's switch set to the "secure" position, but when we connected to the node we were still able to issue the CTL/P command and have the box down to the >>> prompt.

Can anyone please tell us what we may have missed in setting this up.

[Peter Zeiszler]

Isn't there a "set control_P off" command available?

[John A. Beard]

Hi Peter,

I'm afraid that I have never seen or heard of that setting before.

Apart from that SRM setting, should the 'Secure' switch settng have prevented CTL/P from working?

[Peter Zeiszler]

We normally don't use the secure setting since we have to work on systems remotely through terminal servers and have to be able to do the CTRL_P. We actually have in our setup "set control_p on" so we have that ability. Thats what made me think setting it off would disable the CTRL_P functions.

[Jur van der Burg]

I would never let any normal user on the console. It's used for more besides ctrl-p, for example if the uaf cannot be accessed you can login on the console without a password.

Jur.

[John A. Beard]

We are well aware of the issues surrounding security and console access. I am not going to go into the details here as to why we are attempting to prevent staff from issuing CTL/P, all I am asking is why when we set the switch on the front pannel to SECURE were we still able to issue CTL/P and bring the system down to the >>> prompt.

I cannot find anything that relates to setting contol_p to off, that is why I am seeking confirmatin on the issue.

[Art Wiens]

According to:

http://h18002.www1.hp.com/alphaserver/download/es47_es80_gs1280_ug_rev3.pdf

page 26 (pdf page 32)

"Secure - All partitions are powered on. Commands issued via the LAN, control panel, or the MBM CLI which change the state of the system are prevented and receive an error response. If main power fails and returns, the system will power up all partitions, regardless of its soft state at the time of the power failure."

A CTL/P could be definately be said to "change the state of the system", but I think it has more to do with the power state ie. you can't power off or delete a partition with the switch in the secure position.

I think you might be SOL but support may be able to give a better answer.

Hopefully those allowed physical console access will not do something stupid, but I imagine we're talking operators here ;-)

Cheers,
Art

[Zeni B. Schleter]

I have not tested this in a long while but I thought that any input from the console was blocked including the control-P. We have VT420s as consoles. I know we had to enable the switch just to issue a "B" command. More than once I have powered off partitions when trying to enable the switch.

I did not modify in console settings.

[John A. Beard]

There is no physical terminal/console connected to the port, and we gain access via MBM (unles one of people working at the site gains entry to the computer room and connects via his laptop)

The confusion we have in place all started when we were told by HP that once the partition was set to SECURE nobody would be able to issue the likes of a CTL/P command....and obviousuly that is not the case.

[Art Wiens]

In our site we have a two nic Alpha Management Station, with one nic on our general network and the other nic in a private/closed VLAN with the SMC NAT routers to the ES47's.

The only way to gain access to the MBM/SRM is through the AMS and if you don't have the proper credentials in AMS, you don't get access to any console.

Cheers,
Art

[John A. Beard]

The whole reason we are doing this is that we are experiencing a number of system crashes that have yet to be properly explained (not sufficient data in dumps, etc). We have already upgraded everything possible in the VMS and firmware department, and now HP is saying that it potentially might be down to line noise coming being directed onto the console port.

The concept of securing the console port was so that any future similar incidences would not cause the servers to drop down to the SRM prompt...this issues has nothing to do with the who can or cannot gain acess to the port.

[Art Wiens]

Do you have an AMS? You can set it to record the console logging that you can scroll back through to see what possibly might have happened.

BTW, if you have no physical console connected, how can HP say it's line noise on the console port? Solar flares? :-)

Cheers,
Art

[John A. Beard]

By no physical connection I meant that there is no PC or VT device permanently attached. We have cables attached through the SMC NAT routers to the ES47's.

[Volker Halle]

John,

re: unexplainable crashes

The only type of system crashes, which could be explained by 'noise on the console line', would be CPUSANITY or CPUSPINWAIT crashes. This should be visible in the CLUE CONFIG data and the state of the CPUs. I've never heard something like this before, it seems a bit far-fetched ...

Volker.

[John A. Beard]

Hi Volker,

This whole business has proved to be very strange on a number of fronts.

Just to add some complexity to the issue, this problem has cropped up on all eight ES47s spread across two sites spanning a 12 month period.

It has affected both production servers and boxes that would have had absolutely no activity taking place.

I am due to speak with someone from HP in the next 5 minutes and I will mention the points you raised previously.

[Art Wiens]

It's pretty hard to believe that "line noise" on an ethernet cable could form a Ctl/P character but "anything's possible". I would push HP a bit harder and escalate to someone who could properly interpret those crash dumps. Perhaps Volker! ;-) (I know he doesn't work for HP ... maybe they can sub-contract it :-)

Cheers,
Art

[John A. Beard]

HP has had what crash dumps were available and could find nothing. The error logs did not show any hardware errors also.

On some occasions, we would connect to MBM but could not get a response from the server in question. We had no option but to issue a power off and power on command and that overrides the console setting of HALT and rebooted the box.

[Art Wiens]

I've had "similar" experiences not being able to access the console ... not sure if you have an AMS and/or if you tried, but I found that doing a "Broadcast to connected users ..." (available by right-clicking the MBM platform console entry) cleared up whatever was "stuck" and I was then able to use the console.

FWIW,
Art

[Bob Blunt]

John, if you're not getting good dump analysis and you have a valid software support contract then you NEED to have the case elevated, and note that word, ELEVATED to the VMS backline, most of whom are still loosely attached to Colorado Springs. Choose ONE case that's already open, if they've closed it have it reopened.

IF you aren't getting to the VMS backline then speak with your local field engineering manager. They can open an escalation and have the REAL big guns engaged, but there are tons of processes involved with an escalation and it can be cumbersome. Start with a polite but very FIRM insistance that you need the VMS backline. There are other steps that can be taken if you're not being taken seriously but that can get real ugly.

bob

[Volker Halle]

John,

if your systems are set up correctly, $ TYPE CLUE$HISTORY should show the crash history (1 line per crash). If you can make this information available in an attached .TXT file, I'll have a look at the types of crashes and might be able to give further advice.

For each crash, there should also be a CLUE$COLLECT:CLUE$node_ddmmyy_hhmm.LIS file, which contains the most important footprint data for each crash. Feel free to mail those files to me.

Volker.

[John A. Beard]

Hi Volker,

I am attaching the CLUE info pertaining to one of the 8 servers in question.

Although there have been more instances on this particular server, we were not able to always get to the SRM prompt from MBM in order to initiate a crash command. On those occasions we had to issue a power off/power on command from MBM whic automaticly rebooted the server (despite AUTO_ACTION being set to HALT).

[Art Wiens]

I'll ask again, do you have an Alpha Management Station? ie. you don't need to go through the MBM console to get to the SRM console (if you set up the AMS appropriately). If I recall, doing the MBM/SRM access seemed to be a bit of hit and miss ... going straight to the SRM works without issue. And like I said, AMS gives you more control of who can access any consoles and some very granular monitoring points.

A 2 nic Linux box for AMS and a private VLAN for the ES47 side works well for us.

Cheers,
Art

[John A. Beard]

Hi Art,

Sorry for not responding to the question earlier...the answer is no, we don't have AMS but I have just started to look into what it would take to set it up.

[Volker Halle]

John,

this crash footprint certainly looks like a software problem ! Whether it's an error in OpenVMS itself or third-party code remains to be seen.

It's an ACCVIO in process_management code. The code tries to access an invalid S2 space address. The symbolization of the failing PC is misleading. There is no lock manager code in PROCESS_MANAGEMENT. I would guess, that the failing code stream is in the Kernel Thread Support routines.

The crash happens during image rundown (in kernel mode) of the MQ series Execution Controller.

For further confirmation, one would need the source listing of the failing code from this OpenVMS build (from 29-AUG-2007). This information is ONLY available within OpenVMS engineering !

Please have these crashes escalated to HP OpenVMS engineering for further analysis.

The fact that CPU 1 is HALTed in this crash has absolutely nothing to do with 'noise on the console line'. It's becasue the invealid exception happened on CPU 0 and this CPU stopped (all) the other CPUs in the system while handling the bugcheck.

Feel free to mail me the other CLUE files.

Volker.

[John A. Beard]

Hi Volker,

Thanks for all your efforts.

I know it's difficult to include all the relevant history in cases like this, but apart from attaching some additional clue listings I should also mention a few other facts surrounding this case.

The server for which I sent the first listing has actually "hung" a total of nine times since Jan '09(including the incident associated with the listing I attached earlier). There was only one clue listing in sys$errorlog, and it seems no other clue files were geneated for all the successive incidences.

In the cases where we were able to connect to SRM we issued the crash command and sent the dump, errorlog and MBM information off to HP.

The reference to potential "line noise" was give as a possible cause and not a definitive answer.

On another occasion one of my colleagues attempted to connect to the console after being informed that the system had appeared to freeze. As soon as he issued a connect command from MBM, the system dropped down the chevron prompt. The dump file for this event actually showed the last piece of activity to have been the connection attempt by my colleague. My colleague had prior to this pinged the box but the command timed out.

I am attaching a clue file from another node that is located in a different sub-net. This particular incident occured only last Saturday.

FYI From January '09 to date we have experienced a total of 40 such events across all 8 VMS instances (4 ES47s in P2 drawers)