topic Re: Security_server process is using 100% CPU in Operating System - OpenVMS

Security_server process is using 100% CPU

Piet Timmers_1 — Tue, 12 Apr 2005 04:06:14 GMT

I have a security_server process consuming a lot off CPU.

Stop and start: same result.
Prio set to 0 to avoid CPU problems.
OpenVMS V7.3-1

Everything looks ok, but it is still using a lot off CPU.

Anyone??

Greetings,
Piet

Re: Security_server process is using 100% CPU

Jan van den Ende — Tue, 12 Apr 2005 07:43:36 GMT

Piet,

my first suspect would be some monitoring setting wrong.
For instance, if you enable auditing for all file access, that DOES take a lot (although in that case, I would expect AUDIT_SERVER to trash).
Can you stop & start the security server, (and maybe take the file to an other system) and look what got registrered?
I would not be surprised if that points to an "AHA".

Success.

Have one on me.

jpe

Re: Security_server process is using 100% CPU

Volker Halle — Tue, 12 Apr 2005 09:24:53 GMT

Piet,

does the SECURITY_SERVER only consume CPU time or is it doing 'real work', i.e. also doing lots of BUFIOs, DIRIOs or Pagefaults.
Check with SHOW PROC/CONT/ID=

Look at the right-hand column. Any numbers increasing except CPU time ? If so, it's doing some work. Otherwise it's probably looping...

Volker.

Re: Security_server process is using 100% CPU

Piet Timmers_1 — Wed, 13 Apr 2005 01:47:22 GMT

Jan, Volker,

The problem is on a two node cluster, and both processes are consuming CPU and buffered IO only, no pagefaults, no direct IO's.
On 08:43:34
CPU: 0 03:32:24.76
Buffered IO: 7719201

On 08:44:15
CPU: 0 03:32:38.86
Buffered IO: 7726813

So a lot off CPU and buffered IO.

When I stop and start both processes, the new processes act the same, very CPU consuming.

Accoring the regular system manager the problem started after the switch to daylight saving.

Greetings,

Piet

Re: Security_server process is using 100% CPU

Wim Van den Wyngaert — Wed, 13 Apr 2005 02:59:41 GMT

Do you have DTSS ? Try restarting it.
I already had problems with decevent, desta, symbionts and home made stuff.

Wim

Re: Security_server process is using 100% CPU

Wim Van den Wyngaert — Wed, 13 Apr 2005 03:01:50 GMT

Piet,

Check with accouting what processes were created, if any. Did you also check the system log file for messages ? And audit ?

Wim

Re: Security_server process is using 100% CPU

Piet Timmers_1 — Wed, 13 Apr 2005 03:22:20 GMT

Wim,

DTSS was active, and NTP was active, so I have disabled DTSS on both nodes.

No special messages in audit and other logging. Until the end off march it has worked as expected.

Greetings,
Piet

Re: Security_server process is using 100% CPU

Wim Van den Wyngaert — Wed, 13 Apr 2005 03:27:28 GMT

Strange. And show intrussion ?

Re: Security_server process is using 100% CPU

Volker Halle — Wed, 13 Apr 2005 03:28:24 GMT

Piet,

the SECURITY_SERVER recieves messages from other components in the system through it's mailbox. Does the operations count increase rapidly ?

$ SHOW DEV/FULL SECSRV$MAILBOX

Does the SYS$MANAGER:SECURITY_SERVER_ERROR.LOG exist, does it have any error messages ?

SECURITY_SERVER does PROXY processing and CIA processing (Compound Intrusion Analysis).

Did you check SHOW INTRUSION ?

Volker.

Re: Security_server process is using 100% CPU

Jan van den Ende — Wed, 13 Apr 2005 03:40:21 GMT

Piet,

what was the effect of disabling DTSS?
Have you confirmed DTSS really stopped?

Proost.

Have one on me.

jpe

Re: Security_server process is using 100% CPU

Piet Timmers_1 — Wed, 13 Apr 2005 03:44:50 GMT

Wim, Volker,

The mailbox secsrv$mailbox is very active,

Operations completed: 9571086 at 10:41:50
9580286 at 10:42:34
That explains the buffered IO.

There are only a few records with show intrusion, no more than one record every 2-3 minutes.

Greetings,

Piet

Re: Security_server process is using 100% CPU

Piet Timmers_1 — Wed, 13 Apr 2005 03:49:21 GMT

Jan,

I checked that the process is really gone.
The effect was that NTP is doing its work, as can be seen in the log file sys$specific:[tcpip$ntp]tcpip$ntp_run.log with a message "clock adjustments will resume, DTSS no longer active"

Greetings,

Piet

Re: Security_server process is using 100% CPU

Wim Van den Wyngaert — Wed, 13 Apr 2005 03:56:17 GMT

Piet,

Does show intr/old give anything ?

Wim

Re: Security_server process is using 100% CPU

Volker Halle — Wed, 13 Apr 2005 04:00:01 GMT

Piet,

as I said, SECURITY_SERVER is either doing:

- DECnet proxy processing
- Intrusion analysis

It should be receiving commands through it's mailbox and sending replies back to the 'clients'. Are there other processes on the system doing lots of BUFIOs ( MONI PROC/TOPBIO ) ?

You may be able to 'catch' a mailbox message in the SECSRV$MAILBOX.

$ ANAL/SYS
SDA> SHOW DEV MBAn: ! n = SECSRV mbx unit no.
SDA> VALI QUE UCB

If there are any messages in the queue, try

SDA> EXA @UCB;100

It should show the first mailbox message in the queue ? Does any field look like a PID ?

You could also check, whether other processes have channels assigned to that mailbox:

$ ANAL/SYS
SDA> SET OUT x.x
SDA> SHOW PROC/CHAN ALL
SDA> EXIT
$ SEA x.x MBAn: ! n= SECSRV mbx unit

Volker.

Re: Security_server process is using 100% CPU

Piet Timmers_1 — Wed, 13 Apr 2005 04:29:54 GMT

Everybody,

There is another process TNT_SERVER consuming a lot off CPU and buffered IO. As far as I know, nobody is using the OpenVMS Management software. Is it possible that there is a relation with security_server.

Greetings,

Piet

Re: Security_server process is using 100% CPU

Volker Halle — Wed, 13 Apr 2005 04:31:52 GMT

Piet,

just try to SUSPEND the TNT_SERVER process and see whether the SECURTIY_SERVER BUFIOs will stop. Then you know for sure.

Volker.

Re: Security_server process is using 100% CPU

Wim Van den Wyngaert — Wed, 13 Apr 2005 04:34:26 GMT

We stopped using TNT because every time my boss had done some work with it, I found a looping process.

Wim

Re: Security_server process is using 100% CPU

Piet Timmers_1 — Wed, 13 Apr 2005 04:36:42 GMT

Bingo,

TNT_SERVER suspended, security_server is HIB now, and act like normal.

So not completely solved, but know I know where to look at.

Thanks.

Piet

Re: Security_server process is using 100% CPU

Volker Halle — Wed, 13 Apr 2005 04:39:09 GMT

Piet,

you may just want to stop and restart TNT_SERVER. There are TNT$STARTUP.COM and TNT$SHUTDOWN.COM in SYS$STARTUP:

This process may be another one on the list of those having a problem with daylight savings time switch...

Volker.

Re: Security_server process is using 100% CPU

Wim Van den Wyngaert — Wed, 13 Apr 2005 04:45:23 GMT

We used TNT 3.0 A on 6.2 and had problems without time changes. May be this is solved in the mean time.

Wim