topic Re: CPU affinity and subprocess/threads in Operating System - OpenVMS

CPU affinity and subprocess/threads

Willem Grooters — Wed, 06 Sep 2006 07:17:08 GMT

A collegue ran into the following issue:
A multi-threaded application runs fine on a single-CPU system but in a multi-user environment, it crashes. It looks like a problem that several thraeds run on diffeernt CPU's and sometimes, the main thread is activated by different thredas on different CPU's causing the application to die.
As said, it's just on a multi-CPU machine. On a single_CPU system, there is no problem.

We thimnk the problem might be solved in setting the process's affinity to one CPU. If this is a real multi-threaded application, that would solve the problem because all threads would run in that process's context.
Is this true?

Another - for curiosity: If a process's affinity is set to one (or more) processes, is that setting inherited by subprocesses started by that proces?

Willem

Re: CPU affinity and subprocess/threads

Ian Miller. — Wed, 06 Sep 2006 09:28:34 GMT

sounds like a syncronisation problem where the application can't cope multiple threads running concurrently.

A workaround would be to not allow the application to use multiple kernel threads.
Link it without kernel threads or use (V8.2) command
SET IMAGE/FLAGS=NOMKTHREADS

Re: CPU affinity and subprocess/threads

Ian Miller. — Wed, 06 Sep 2006 09:30:56 GMT

On earlier versions of VMS use
THREADCP/DISABLE=MULTIPLE_KERNEL_THREADS XX.EXE

Re: CPU affinity and subprocess/threads

Volker Halle — Wed, 06 Sep 2006 11:23:54 GMT

Willem,

see HELP SET PROC/AFF
...
As with the other SET PROCESS qualifiers, the bit operations
occur on the current process if no /IDENTIFICATION qualifier
or explicit process name parameter is specified. Specifying a
process name does not imply that all kernel threads associated
with the process are affected; the SET PROCESS command affects
only the initial kernel thread of a multithreaded process.

An application with such a behaviour clearly has a synchronization bug, that just does not show up, until you run the application really in parallel on multiple CPUs.

Volker.

Re: CPU affinity and subprocess/threads

Willem Grooters — Wed, 06 Sep 2006 11:41:48 GMT

Volker,
That implies that SET PROCESS/AFFINITY=CPUxx won't help.

Ian,
I'm not certain on the VMS version but I'm quite certain it's 7.3-something. So 8.2 commands are not applicable.

I fully agree it might be a synchronisation issue. Not really a surprise ;-). To solve the problem it will be needed to redo this part, I guess.

Re: CPU affinity and subprocess/threads

Kris Clippeleyr — Wed, 06 Sep 2006 13:33:37 GMT

Willem,
I'm not sure how badly the following will affect other applications/processes on the system, but couldn't you just simply disable the creation of kernel threads altogether (setting the system parameter MULTITHREAD to 0 (zero)). We once solved a problem by doing so. The app (actually one process of the app) was also multithreaded, and very CPU hungry. If we allowed kernel threads, all CPUs (four of them) were busy serving that single process. By preventing kernel threads to be created, we could keep 1 CPU busy with that process, while the other 3 could service the rest.
Hope this helps,
Kris (aka Qkcl)

Re: CPU affinity and subprocess/threads

Ian Miller. — Wed, 06 Sep 2006 15:15:09 GMT

SET IMAGE/FLAGS=NOMKTHREADS only works on Itanium images so THREADCP would be the correct command (its been around for serveral versions).

As Kris said you can disable kernel threads completely but it would be better to fix the application.

Re: CPU affinity and subprocess/threads

Thomas Ritter — Wed, 06 Sep 2006 20:02:03 GMT

Willem, we locked a number of processes to a single CPU in our 4 node 4 CPU 7.3-2 environment. The developers got ambitious with their multithreading and we suffered numerious bugchecks. After the processes were assigned to a single CPU no more problem ! I am sure there are problems with VMS in this area of multi-threading. In spite of conceptual brilliance of some these applications developed, when assigned to a single CPU, did not measurably affect the applications throughput. I all for keeping things simple.

Re: CPU affinity and subprocess/threads

Volker Halle — Thu, 07 Sep 2006 00:24:38 GMT

Thomas,

how did you 'lock a number of processes to a single CPU' ? That's the question.

SET PROC/AFF ?
THREADCP/DISABLE=MULTIPLE_KERNEL_THREADS ?

Volker.

Re: CPU affinity and subprocess/threads

Thomas Ritter — Thu, 07 Sep 2006 00:29:26 GMT

Try this

$ set proc "''process_name'"/AFFINITY/SET=1/permanent

Re: CPU affinity and subprocess/threads

Thomas Ritter — Thu, 07 Sep 2006 00:32:20 GMT

And with a $show cpu/all extract

CPU 1 State: RUN CPUDB: 81921880 Handle: 00000E90
Owner: 00000640 Current: 00000640 Partition 0 (WIZ22)
ChgCnt: 0 State: Present, In-Console, Reassignable
Process: * None *
Capabilities:
System: QUORUM RUN RAD0
Slot Context: C94A0400
CPU - State..........: RC, PA, PP, CV, PV, PMV, PL
Type...........: EV68 (21264/EV68CB 21264C), Minor = 5
Speed..........: 1000 Mhz
Variation......: VAX FP, IEEE FP, Primary Eligible
Serial Number..: AY20408008
Revision.......:
Halt Request...: 0
Software Comp..: 8.4
PALCODE - Revision Code..: 1.98-01
Compatibility..: 43
Max Shared CPUs: 4
Memory Space..: Physical = 00000000.00000000 Length = 0
Scratch Space..: Physical = 00000000.00000000 Length = 0
Bindings:
WIR_CGI_FOXPRD PID = 2C80190C Reason: Affinitized to this CPU
WIR_CGI_FOXPRD PID = 2C80110C Reason: Affinitized to this CPU
WIR_CGI_FOXPRD PID = 2C80090C Reason: Affinitized to this CPU
Fastpath:

Re: CPU affinity and subprocess/threads

Volker Halle — Thu, 07 Sep 2006 00:44:07 GMT

Thomas,

thanks.

It looks like actually the process and all its threads are bound to that CPU. This seems to contradict the help text of SET PROC/AFF.

Does $ SHOW SYS/MULTI/FULL show 3 kernel threads for process WIR_CGI_FOXPRD ?

Volker.

Re: CPU affinity and subprocess/threads

Thomas Ritter — Thu, 07 Sep 2006 00:50:32 GMT

OpenVMS V7.3-2 on node WIZ22 7/09/06 15:49:20.88 Uptime 2 13:43:57
AlphaServer ES45 Model 2B
Pid Process Name State Pri I/O CPU Page flts Pages
2C800818 ACME_SERVER HIB 8 111 0 00:00:00.53 384 538 M
2C801018 HIB 8
2C801818 HIB 10
[SYSTEM] 4304Kb
2C80090C WIR_CGI_FOXPRD HIB 4 79389 0 00:01:19.19 1483 1891 M
2C80110C HIB 4
2C80190C HIB 6
[WIZARD_100,ADDR_FOXPRD] 15128Kb

Re: CPU affinity and subprocess/threads

Volker Halle — Thu, 07 Sep 2006 01:05:59 GMT

Thomas,

thanks. So this seems to indiciate, that the help text for SET PROC/AFF is not correct.

Volker.

Re: CPU affinity and subprocess/threads

Guy Peleg — Thu, 07 Sep 2006 01:45:44 GMT

>multi-threaded application runs fine on a
>single-CPU system but in a multi-user
>environment, it crashes

Can you provide some more details on
how it crashes? System crash? process crash?
ACCVIO?

Guy

Re: CPU affinity and subprocess/threads

Willem Grooters — Thu, 07 Sep 2006 04:15:01 GMT

Just a process crash - data attached.

Re: CPU affinity and subprocess/threads

Volker Halle — Thu, 07 Sep 2006 05:27:00 GMT

Willem,

what does SDA> EXA/INS 808F8290 return ? What module is the exception PC in.

Although this 'may be' an underlying application problem, the exception PC and the most recent call frames on the traceback stack are from OpenVMS/PTHREAD/JAVA i.e. HP code !

Could you create a process dump for this situation (SET PROC/DUMP in LOGIN.COM or SET PROC/DUMP/PID=xxx) ?

Make sure you're up-to-date on PTRHEAD patches.

Volker.

Re: CPU affinity and subprocess/threads

Volker Halle — Thu, 07 Sep 2006 05:56:24 GMT

Willem,

while you're in SDA, please also check

SDA> EXA/INS 8083D830

This is more likely the real exception PC. The values in the vms condition printout from PHTREADs may be misleading.

The code most likely at fault here is in PTHREAD$RTL or JAVA$HPI_SHR, which are the most recent callers.

Volker.

Re: CPU affinity and subprocess/threads

Volker Halle — Thu, 07 Sep 2006 06:14:14 GMT

Willem,

Another - for curiosity: If a process's affinity is set to one (or more) processes, is that setting inherited by subprocesses started by that proces?

You can easily test this for yourself:

$ set proc/aff/set=0/perm
$ SHOW PROC
...
Soft CPU Affinity: off
Hard CPU Affinity Mask: 00000001
$ SPAWN
$ SHOW PROC
...
Soft CPU Affinity: off
$

The Hard CPU affinity is not inherited by the spawned subprocess.

Volker.

Re: CPU affinity and subprocess/threads

Thomas Ritter — Thu, 21 Sep 2006 19:00:01 GMT

Willem, what's the progress ? Did you try to set lock the process(s) to a single CPU ?