Security_server process is using 100% CPU

[Piet Timmers_1]

I have a security_server process consuming a lot off CPU.

Stop and start: same result.
Prio set to 0 to avoid CPU problems.
OpenVMS V7.3-1

Everything looks ok, but it is still using a lot off CPU.

Anyone??

Greetings,
Piet

[Jan van den Ende]

Piet,

my first suspect would be some monitoring setting wrong.
For instance, if you enable auditing for all file access, that DOES take a lot (although in that case, I would expect AUDIT_SERVER to trash).
Can you stop & start the security server, (and maybe take the file to an other system) and look what got registrered?
I would not be surprised if that points to an "AHA".

Success.

Have one on me.

jpe

[Volker Halle]

Piet,

does the SECURITY_SERVER only consume CPU time or is it doing 'real work', i.e. also doing lots of BUFIOs, DIRIOs or Pagefaults.
Check with SHOW PROC/CONT/ID=

Look at the right-hand column. Any numbers increasing except CPU time ? If so, it's doing some work. Otherwise it's probably looping...

Volker.

[Piet Timmers_1]

Jan, Volker,

The problem is on a two node cluster, and both processes are consuming CPU and buffered IO only, no pagefaults, no direct IO's.
On 08:43:34
CPU: 0 03:32:24.76
Buffered IO: 7719201

On 08:44:15
CPU: 0 03:32:38.86
Buffered IO: 7726813

So a lot off CPU and buffered IO.

When I stop and start both processes, the new processes act the same, very CPU consuming.

Accoring the regular system manager the problem started after the switch to daylight saving.

Greetings,

Piet

[Wim Van den Wyngaert]

Do you have DTSS ? Try restarting it.
I already had problems with decevent, desta, symbionts and home made stuff.

Wim

[Wim Van den Wyngaert]

Piet,

Check with accouting what processes were created, if any. Did you also check the system log file for messages ? And audit ?

Wim

[Piet Timmers_1]

Wim,

DTSS was active, and NTP was active, so I have disabled DTSS on both nodes.

No special messages in audit and other logging. Until the end off march it has worked as expected.

Greetings,
Piet

[Wim Van den Wyngaert]

Strange. And show intrussion ?

[Volker Halle]

Piet,

the SECURITY_SERVER recieves messages from other components in the system through it's mailbox. Does the operations count increase rapidly ?

$ SHOW DEV/FULL SECSRV$MAILBOX

Does the SYS$MANAGER:SECURITY_SERVER_ERROR.LOG exist, does it have any error messages ?

SECURITY_SERVER does PROXY processing and CIA processing (Compound Intrusion Analysis).

Did you check SHOW INTRUSION ?

Volker.

[Jan van den Ende]

Piet,

what was the effect of disabling DTSS?
Have you confirmed DTSS really stopped?

Proost.

Have one on me.

jpe

[Piet Timmers_1]

Wim, Volker,

The mailbox secsrv$mailbox is very active,

Operations completed: 9571086 at 10:41:50
9580286 at 10:42:34
That explains the buffered IO.

There are only a few records with show intrusion, no more than one record every 2-3 minutes.

Greetings,

Piet

[Piet Timmers_1]

Jan,

I checked that the process is really gone.
The effect was that NTP is doing its work, as can be seen in the log file sys$specific:[tcpip$ntp]tcpip$ntp_run.log with a message "clock adjustments will resume, DTSS no longer active"

Greetings,

Piet

[Wim Van den Wyngaert]

Piet,

Does show intr/old give anything ?

Wim

[Volker Halle · Accepted solution]

Piet,

as I said, SECURITY_SERVER is either doing:

- DECnet proxy processing
- Intrusion analysis

It should be receiving commands through it's mailbox and sending replies back to the 'clients'. Are there other processes on the system doing lots of BUFIOs ( MONI PROC/TOPBIO ) ?

You may be able to 'catch' a mailbox message in the SECSRV$MAILBOX.

$ ANAL/SYS
SDA> SHOW DEV MBAn: ! n = SECSRV mbx unit no.
SDA> VALI QUE UCB

If there are any messages in the queue, try

SDA> EXA @UCB;100

It should show the first mailbox message in the queue ? Does any field look like a PID ?

You could also check, whether other processes have channels assigned to that mailbox:

$ ANAL/SYS
SDA> SET OUT x.x
SDA> SHOW PROC/CHAN ALL
SDA> EXIT
$ SEA x.x MBAn: ! n= SECSRV mbx unit

Volker.

[Piet Timmers_1]

Everybody,

There is another process TNT_SERVER consuming a lot off CPU and buffered IO. As far as I know, nobody is using the OpenVMS Management software. Is it possible that there is a relation with security_server.

Greetings,

Piet

[Volker Halle]

Piet,

just try to SUSPEND the TNT_SERVER process and see whether the SECURTIY_SERVER BUFIOs will stop. Then you know for sure.

Volker.

[Wim Van den Wyngaert]

We stopped using TNT because every time my boss had done some work with it, I found a looping process.

Wim

[Piet Timmers_1]

Bingo,

TNT_SERVER suspended, security_server is HIB now, and act like normal.

So not completely solved, but know I know where to look at.

Thanks.

Piet

[Volker Halle]

Piet,

you may just want to stop and restart TNT_SERVER. There are TNT$STARTUP.COM and TNT$SHUTDOWN.COM in SYS$STARTUP:

This process may be another one on the list of those having a problem with daylight savings time switch...

Volker.

[Wim Van den Wyngaert]

We used TNT 3.0 A on 6.2 and had problems without time changes. May be this is solved in the mean time.

Wim