topic Re: Priority increases automatically in Operating System - OpenVMS

Priority increases automatically

Shankar Bose — Wed, 26 Mar 2008 13:12:17 GMT

In a clustered OpenVMS 8.2 multiple CPU ALPHA environments the priority of a process which is triggered with a base priority of 8 creeps up to 63 within few days.
Though the system is currently not facing any problem due to this, but a process that has been a time slot based one moves to real time is not welcomed, also it is running with priority higher than SWAPPER, which is running at priority 27.
Is there any way of restriction it?

Re: Priority increases automatically

Wim Van den Wyngaert — Wed, 26 Mar 2008 13:30:05 GMT

Start in batch :

$b:
$ set proc/id=xxx/prio=8
$ wait 00:01
$ goto b

But why is it happenning ?

Wim

Re: Priority increases automatically

Hoff — Wed, 26 Mar 2008 13:31:36 GMT

OpenVMS should not and does not normally increase priority from timeshare to realtime.

This is almost certainly local code, or local commands, or a local user that's responsible; a trigger within the local environment. (If this were a general error in OpenVMS, there would likely be more reports of this around.)

Enable system service auditing and use it to find out which process here is calling $setpri. The altpri privilege can be audited. Use "set audit /enable=privilege=success=altpri", IIRC. (Somewhat surprisingly, there's no way to directly audit $setpri calls. Weird. HP should consider an enhancement there.)

On no evidence, I would review the code of the image itself, and whatever code you might have around that calls the $setpri system service.

Re: Priority increases automatically

Michael Moroney — Wed, 26 Mar 2008 13:48:48 GMT

Even SWAPPER running at priority of 27 seems odd. I thought its priority was fixed at 16.

Re: Priority increases automatically

Shankar Bose — Wed, 26 Mar 2008 14:29:15 GMT

We are trying to trace why it is happening. The mentioned process is a server process and on client request it starts threads to serve it. There may be some ripple affect changing the priority of the process. There are around 90 application processes running with priority 7, 8. We could see that other process also keeps changing priorities but they are within time base range. I will start doing the audit, and see whether that helps.

Re: Priority increases automatically

Jon Pinkley — Wed, 26 Mar 2008 14:46:23 GMT

What are you using to view the process priority?

Can you paste the output here?

Re: Priority increases automatically

Shankar Bose — Wed, 26 Mar 2008 14:51:40 GMT

$ SH PROCESS/ID=nnnnnnn/CONTI
and watch the field Cur/base priority in the display.

Re: Priority increases automatically

Wim Van den Wyngaert — Wed, 26 Mar 2008 14:53:56 GMT

That are 2 fields. Is it cur or base going to 63 ?

Re: Priority increases automatically

Shankar Bose — Wed, 26 Mar 2008 15:00:54 GMT

Both are drifting

Re: Priority increases automatically

Jon Pinkley — Wed, 26 Mar 2008 15:05:26 GMT

And the base priority is increasing?

If so, as Hoff said, it is almost certainly not something that is part of stock standard VMS that is modifying it.

The only thing the VMS schedular modifies is the current priority, and that only up to base + PRI$_TICOM (boost of 6, which is immediately dropped by one when it is scheduled). And that will never promote the current priority above 15 (it will not move it into the realtime priorities).

So what other software do you have running? Any "fair share" priority adjusters?

RE:"Both are drifting"

When the base is in the realtime range, the current priority should be the same as the base, i.e. VMS shouldn't be applying dynamic priority adjustments to real time processes. Is that consistent with what you see?

Jon

Re: Priority increases automatically

Hoff — Wed, 26 Mar 2008 18:09:06 GMT

I'd have to assume that there's some local $setpri code (lurking somewhere) here that's the trigger, and it has apparently gone off the rails.

Scheduling code that alters the process priority tends to either be implemented correctly (rare, but possible), or (the vast majority of cases I've encountered) the process scheduling code hosts a few priority inversions and some slightly-flaky run-time code.

ALTPRI privilege can be semi-hazardous, and inversions can be easy to trigger and hard to debug.

The OpenVMS class scheduler was implemented a while back to make rolling your own scheduler somewhat easier.

Re: Priority increases automatically

Shankar Bose — Wed, 02 Apr 2008 08:59:56 GMT

Security audit (SECURITY) on TRINA, system id: 65473
Auditable event: Privilege used
Event information: ALTPRI used to elevate base priority of process ($SETPRI)
Event time: 31-MAR-2008 15:09:14.58
PID: 000A266C
Process name: CIS_TRSP04
Username: SPC_SST_SVR
Process owner: [SORT,SPC_SST_SVR]
New priority: 30
Old priority: 18
Target PID: 000A326C
Privileges used: ALTPRI
Posix UID: -2
Posix GID: -2 (%XFFFFFFFE)

Audit report output posted. Any clue for it.

Re: Priority increases automatically

Jon Pinkley — Wed, 02 Apr 2008 09:15:14 GMT

I don't recognize the process name or username. But again, with near certainty, this isn't part of VMS.

Do you have a username SPC_SST_SVR in your SYSUAF.DAT file? That would be one clue.

$ set default sys$system ! in case sysuaf isn't defined
$ mcr authorize show user spc_sst_svr

Is the process with PID: 000A266C still around?

$ set proc/priv=all
$ analyze/system
SDA> set proc/in=000A266C
SDA> show proc/chan

This will show what files this process has open, that can give you a clue too.

Re: Priority increases automatically

Shankar Bose — Wed, 02 Apr 2008 09:25:41 GMT

We have a username SPC_SST_SVR, the application is invoked by the batch user, and its priority is 9.

Attached is the SDA output.

Process index: 026C Name: CIS_TRSP04 Extended PID: 000A266C
--------------------------------------------------------------------

Process active channels
-----------------------

Channel CCB Window Status Device/file accessed
------- --- ------ ------ --------------------
0010 7FF5E000 00000000 $1$DKA201:
0020 7FF5E020 86D4EA80 $1$DKA201:[CSC_ENV_1.APPLIC.SPICE.LIB]SPC_WDG_RUN_SVR.COM;1
0030 7FF5E040 86BEA2C0 $1$DKA201:[CSC_ENV_1.APPLIC.SHARE.PACKAGE]CIS.EXE;1
0040 7FF5E060 868C11C0 $1$DKA202:[VMS$COMMON.SYSLIB]SECURESHRP.EXE;1 (section file)
0050 7FF5E080 868CD5C0 $1$DKA202:[VMS$COMMON.SYSEXE]DCL.EXE;1 (section file)
0060 7FF5E0A0 868C1F40 $1$DKA202:[VMS$COMMON.SYSLIB]DCLTABLES.EXE;510 (section file)
0070 7FF5E0C0 86E9F840 $1$DKA201:[DATALOG.CIS.LOGS]CIS.13MAR2008;1
0080 7FF5E0E0 86F7BE80 $1$DKA201:[CSC.ENV4.LOGS.APC_SCRATCH]APC_APE_DET1_00A768A0F78B7AA2_000007.COM_TMP;1
0090 7FF5E100 868C5940 $1$DKA202:[VMS$COMMON.SYSLIB]PTHREAD$RTL.EXE;1 (section file)
00A0 7FF5E120 868C2540 $1$DKA202:[VMS$COMMON.SYSLIB]LIBOTS.EXE;1 (section file)
00B0 7FF5E140 868C2140 $1$DKA202:[VMS$COMMON.SYSLIB]LIBRTL.EXE;1 (section file)
00C0 7FF5E160 868C4240 $1$DKA202:[VMS$COMMON.SYSLIB]CMA$TIS_SHR.EXE;1 (section file)
00D0 7FF5E180 868C5540 $1$DKA202:[VMS$COMMON.SYSLIB]DPML$SHR.EXE;1 (section file)
00E0 7FF5E1A0 868C7DC0 $1$DKA202:[VMS$COMMON.SYSLIB]DEC$FORRTL.EXE;1 (section file)
00F0 7FF5E1C0 868C53C0 $1$DKA202:[VMS$COMMON.SYSLIB]NCSSHR.EXE;1 (section file)
0100 7FF5E1E0 868C3640 $1$DKA202:[VMS$COMMON.SYSLIB]LBRSHR.EXE;1 (section file)
0110 7FF5E200 868C5DC0 $1$DKA202:[VMS$COMMON.SYSLIB]DECC$SHR_EV56.EXE;1 (section file)
0120 7FF5E220 86EB1240 $1$DKA200:[oracle.product.1010.lib32]libclntsh.so;1 (section file)
0130 7FF5E240 868C65C0 $1$DKA202:[VMS$COMMON.SYSLIB]CMA$RTL.EXE;1 (section file)
0140 7FF5E260 868C7AC0 $1$DKA202:[VMS$COMMON.SYSLIB]CMA$OPEN_RTL.EXE;1 (section file)
0150 7FF5E280 86A85840 $1$DKA202:[VMS$COMMON.SYSLIB]CMA$LIB_SHR.EXE;1 (section file)
0160 7FF5E2A0 86A86040 $1$DKA202:[VMS$COMMON.SYSLIB]CMA$OPEN_LIB_SHR.EXE;1 (section file)
0170 7FF5E2C0 86A842C0 $1$DKA202:[VMS$COMMON.SYSLIB]DTSS$SHR.EXE;1 (section file)
0180 7FF5E2E0 86A89740 $1$DKA202:[VMS$COMMON.SYSLIB]DCE$LIB_SHR.EXE;1 (section file)
0190 7FF5E300 868C3DC0 $1$DKA202:[VMS$COMMON.SYSLIB]SECURESHR.EXE;1 (section file)
01A0 7FF5E320 86E78C00 $1$DKA201:[CSC_ENV_1.APPLIC.DATABASE.EXE]DBMANAGER.EXE;1 (section file)
01B0 7FF5E340 86EB49C0 $1$DKA201:[CSC_ENV_1.APPLIC.SPICE.LIB]SPC_ERR_SHARE.EXE;1 (section file)
01C0 7FF5E360 86A863C0 $1$DKA202:[VMS$COMMON.SYSLIB]DCE$SOCKSHR_DNET_IV.EXE;1 (section file)
01D0 7FF5E380 86A87600 $1$DKA202:[VMS$COMMON.SYSLIB]DCE$KERNEL.EXE;1 (section file)
01E0 7FF5E3A0 86A86DC0 $1$DKA202:[VMS$COMMON.SYSLIB]DCE$SOCKSHR_DNET_OSI.EXE;1 (section file)
01F0 7FF5E3C0 86A87000 $1$DKA202:[VMS$COMMON.SYSLIB]DCE$SOCKSHR_IP.EXE;1 (section file)
0200 7FF5E3E0 00000000 Busy BG25204:

Re: Priority increases automatically

Jon Pinkley — Wed, 02 Apr 2008 21:40:23 GMT

Shankar,

I assumed you were the system manager. If not, do you know who is responsible for installing applications on this machine?

The first place I would look is the command file:

$1$DKA201:[CSC_ENV_1.APPLIC.SPICE.LIB]SPC_WDG_RUN_SVR.COM;1

See if it has any info. The program that was running at the time you looked was

$1$DKA201:[CSC_ENV_1.APPLIC.SHARE.PACKAGE]CIS.EXE;1

You may find more clues in:

0070 7FF5E0C0 86E9F840 $1$DKA201:[DATALOG.CIS.LOGS]CIS.13MAR2008;1
0080 7FF5E0E0 86F7BE80 $1$DKA201:[CSC.ENV4.LOGS.APC_SCRATCH]APC_APE_DET1_00A768A0F78B7AA2_000007.COM_TMP;1

It is evidently communicating with something over the network, as evidenced by the

0200 7FF5E3E0 00000000 Busy BG25204:

This may be tied into a database as evidenced by:

0120 7FF5E220 86EB1240 $1$DKA200:[oracle.product.1010.lib32]libclntsh.so;1 (section file)

and

01A0 7FF5E320 86E78C00 $1$DKA201:[CSC_ENV_1.APPLIC.DATABASE.EXE]DBMANAGER.EXE;1 (section file)

So find out where this package came from, and contact them. Either CIS.EXE or something else being executed by the SPC_WDG_RUN_SVR.COM command procedure is what is responsible for changing the process priorities.

This looks like a case of what Hoff meant by his comment "Scheduling code that alters the process priority tends to either be implemented correctly (rare, but possible), or (the vast majority of cases I've encountered) the process scheduling code hosts a few priority inversions and some slightly-flaky run-time code."

There is an electrical circuit emulator called SPICE, but whether this is related to that, I have no idea. Since that SPICE is normally CPU intensive, I can't imagine anyone wanting to increase its priority, if anything I would want to let it run at low priority and make use of cycles that would otherwise go to the idle loop. So, I am not convinced this has anything to do with the SPICE described at http://en.wikipedia.org/wiki/SPICE

There really isn't much more we can do at this point. You need to do some local research and find out what the application is doing, and why it is trying to adjust priorities.

Good luck,

Jon

Re: Priority increases automatically

Shankar Bose — Tue, 08 Apr 2008 08:38:37 GMT

Thanks for the extended help.