topic locked out in Operating System - OpenVMS

locked out

dschwarz — Fri, 07 Sep 2007 01:30:23 GMT

We have a MicroVax 3100/96 running OpneVMS 6.2.

Yesterday I reboored the system to start some new and changed applications, all of them running as detached processes.

After rebooting I did some interactive tests on
the system, everything worked fine.

In the early afternoon, I tried to log in to
the system to do some additional checks but
the system did not let me in:
"User authorization failure"

I tried different username/password combinations, but the system refused any login even at the console terminal.

The application was still running perfectly so
reboot was/ist not an option during production time.

There are no device errors on the system but in the OPERATOR.LOG file I found a suspicious entry:

%%%%%%%%%%% OPCOM 6-SEP-2007 10:04:44.43 %%%%%%%%%%%
Message from user SYSTEM on MYNODE
%SECSRV-E-QIOFAILED, security server QIO on client mailbox failed
%SYSTEM-S-NORMAL, normal successful completion

What has happened with the system ?
Is there a chance to 'repair' this without reboot ?
Is there a chance not to repair this after reboot ?
What is the meaning of the OPCOM message ?

Thank you very much in advance.

Re: locked out

Willem Grooters — Fri, 07 Sep 2007 05:28:44 GMT

Looks like the audit or security server has a problem with (one of) the mailboxes - or is gone.
Check devices AUDSRV$CONTROL_MAILBOX and SECSRV$MAILBOX, and look for processses AUDIT_SERVER and SECURITY_SERVER: Persence, status?

You could check these using SDA.

If the proces is gone, check accounting to find out what caused it to die.

a\After you gatheered the information, the next issue is to find out what caused the problem.

Re: locked out

dschwarz — Fri, 07 Sep 2007 05:39:09 GMT

Both processes are hibernating:

OpenVMS V6.2 on node MYNODE 7-SEP-2007 12:34:00.40 Uptime 1 04:02:51
...
00000209 AUDIT_SERVER HIB 10 310 0 00:00:00.80 539 805
...
0000020D SECURITY_SERVER HIB 10 51 0 00:00:00.16 1724 689

Both mailboxes exist:

"AUDSRV$CONTROL_MAILBOX" = "_MBA12:" (LNM$SYSTEM_TABLE)

Device MBA12:, device type local memory mailbox, is online, record-oriented
device, shareable, mailbox device.

Error count 0 Operations completed 2
Owner process "" Owner UIC [SYSTEM]
Owner process ID 00000000 Dev Prot S:RWPL,O:RWPL,G,W
Reference count 1 Default buffer size 65535

"SECSRV$MAILBOX" = "_MBA13:" (LNM$SYSTEM_TABLE)

Device MBA13:, device type local memory mailbox, is online, record-oriented
device, shareable, mailbox device.

Error count 0 Operations completed 53
Owner process "" Owner UIC [SYSTEM]
Owner process ID 00000000 Dev Prot S:RWPL,O:RWPL,G:RWPL,W:RWPL
Reference count 1 Default buffer size 8192
Device access control list:
(IDENTIFIER=SECSRV$OBJECT,ACCESS=READ+PHYSICAL+LOGICAL+CONTROL)
(IDENTIFIER=SECSRV$CLIENT,ACCESS=WRITE+PHYSICAL)
(IDENTIFIER=SECSRV$COMMUNICATION,ACCESS=WRITE+PHYSICAL)
(IDENTIFIER=*,ACCESS=NONE)

SDA is no an option at the moment, no chance for interactive login.

Re: locked out

Robert Gezelter — Fri, 07 Sep 2007 05:44:46 GMT

dschwarz,

Are there any potentials for "non-interactive" login?

- can you get to the system using SYSMAN?
- is it possible to use DECnet to run a batch job on the system?

How are you determining the information in your last posting?

- Bob Gezelter,http://www.rlgsc.com

Re: locked out

John Abbott_2 — Fri, 07 Sep 2007 06:00:29 GMT

> I tried different username/password combinations, but the system refused any login even at the console terminal.

What happens if you try the system account username from the system console - can you twice to the password prompt and get in ?

Did you changed any SYSGEN parameters before the reboot ?

Patch anything ?

J.

Re: locked out

dschwarz — Fri, 07 Sep 2007 06:21:39 GMT

Robert:

Non interactive login is possible.
I used decnet task-to-task via proxy access.
What can I do with SYSMAN ?

John:

The system resides some miles away, so I have
no access to the console terminal today. Next
chance will be on sunday.

No SYSGEN changes, no patches.
I activated NFS-Client for future use.

TCP/IP stack is:
Digital TCP/IP Services for OpenVMS VAX Version V4.2 - ECO 4

very old, I know.

Re: locked out

Robert Gezelter — Fri, 07 Sep 2007 07:05:52 GMT

dschwartz,

Take a look at the documentation for SYSMAN SET ENVIRONMENT/NODE command.

If you can do DECnet proxy access, is the proxy only for a non-privileged account?

- Bob Gezelter, http://www.rlgsc.com

Re: locked out

Wim Van den Wyngaert — Fri, 07 Sep 2007 07:15:27 GMT

I tried a login on a system with success. Security server didn't do a io or pagefault.

But when I give invalid username and password it does. Or when there is an intrusion and the username + p is valid.

Do you have an intrusion ?

May be the message you see has nothing to do with this problem ?

Wim

Re: locked out

Daniel Fernandez Illan — Fri, 07 Sep 2007 07:16:08 GMT

dschwarz

Obiously.

Are you check the interactive limit value and intrusion database?

Saludos.
Daniel.

Re: locked out

dschwarz — Fri, 07 Sep 2007 07:16:40 GMT

Robert,

I have all privileges.

Re: locked out

dschwarz — Fri, 07 Sep 2007 07:36:56 GMT

Wim,

maybe the opcom message has nothing to do with the problem.

Daniel,

interactive limit is 64 as usual.

SHOW INTRUSION :
%SYSTEM-F-MBFULL, mailbox is full

There must be a lot of entries in the database. ! ??

Re: locked out

David Jones_21 — Fri, 07 Sep 2007 08:05:43 GMT

I'd think the QIOFAILED opcom message and the MBFULL message on show intrusion are related. Whatever is supposed to be reading the mailbox isn't, so messages got saved in the mailbox's message queue until it reached it's quota.

Re: locked out

John Abbott_2 — Fri, 07 Sep 2007 08:09:27 GMT

Does anyone recall having to increase SYSGEN DEFMBXBUFQUO and/or DEFMBXMXMSG to accommodate large activity ? I vaguely recall something under an older version of VMS VAX or Alpha, but since upgrading to 7x and 8x I tidied up my modparams and cannot find anything (i.e. the settings these days are >= to what I had by default).

The problem I had was definately related to the audit server, although maybe it could have been more to do with a mailbox filter we have...

J.

Re: locked out

Hein van den Heuvel — Fri, 07 Sep 2007 08:24:24 GMT

Can you flush the mailbox(en) by $typing them, or better still, $copying the contents into a file?

Did you try stopping and restarting the audit and security servers?

$ SET SERVER SECURITY/EXIT
$ SET AUDIT/SERVER=EXIT
$ @SYS$SYSTEM:STARTUP AUDIT_SERVER
$ SET SERVER SECURITY/START

btw... Did you make sure you there is still space left on the system dusk or whatever disk the audit file is redirected to?

Re: locked out

Robert Gezelter — Fri, 07 Sep 2007 10:05:01 GMT

dschwartz,

Ok, if you have access to a privileged account, then it is highly likely that it is possible to resolve this situation without access to the console.

It would also appear that you have access to another system locally or over your internal network.

This can be an involved process, but it is likely better than being locked out of the system until Sunday (particularly in that the problem may cascade in an unexpected way).

It is difficult to give instructions in the forum on how to go through this step-by-step; as the precise steps depend on what exactly has happened.

The basic steps are to use SYSMAN or DECnet Task-to-Task to execute a series of commands or command files to diagnose and then correct the problem. Most of the involved components can be restarted without rebooting the system. While this is not a common, everyday event, it is a standard part of OpenVMS.

It is possible, but not guaranteed, that no reboot will be needed to clear this problem.

If I can be of additional assistance, please let me know. (Obviously, while I do monitor this forum, I am working on various projects at my desk, so private communications are faster).

- Bob Gezelter, http://www.rlgsc.com

Re: locked out

dschwarz — Sun, 09 Sep 2007 10:02:21 GMT

Now it's sunday afternoon and I have rebooted the system.
Everything seems to work fine including interactive logins.
Before rebooting I tried to stop and restart SECURITY_SERVER and AUDIT_SERVER. This did not work with SECURITY_SERVER.
I also tried to flush/copy the mailbox SECSRV$MAILBOX
==> 25 additional operations in a SHOW DEVICE MBAxx.
After that I tried SHOW INTRUSION and got SECSRV-F-SRVREPLYTIMEOUT.
I used HELP/MESSAGE ... and decided to reboot.

Thank you guys for your help.

Re: locked out

dschwarz — Mon, 17 Sep 2007 02:36:57 GMT

Reboot has 'solved' the problem