CPU affinity and subprocess/threads

[Willem Grooters]

A collegue ran into the following issue:
A multi-threaded application runs fine on a single-CPU system but in a multi-user environment, it crashes. It looks like a problem that several thraeds run on diffeernt CPU's and sometimes, the main thread is activated by different thredas on different CPU's causing the application to die.
As said, it's just on a multi-CPU machine. On a single_CPU system, there is no problem.

We thimnk the problem might be solved in setting the process's affinity to one CPU. If this is a real multi-threaded application, that would solve the problem because all threads would run in that process's context.
Is this true?

Another - for curiosity: If a process's affinity is set to one (or more) processes, is that setting inherited by subprocesses started by that proces?

Willem

[Ian Miller.]

sounds like a syncronisation problem where the application can't cope multiple threads running concurrently.

A workaround would be to not allow the application to use multiple kernel threads.
Link it without kernel threads or use (V8.2) command
SET IMAGE/FLAGS=NOMKTHREADS

[Ian Miller.]

On earlier versions of VMS use
THREADCP/DISABLE=MULTIPLE_KERNEL_THREADS XX.EXE

[Volker Halle]

Willem,

see HELP SET PROC/AFF
...
As with the other SET PROCESS qualifiers, the bit operations
occur on the current process if no /IDENTIFICATION qualifier
or explicit process name parameter is specified. Specifying a
process name does not imply that all kernel threads associated
with the process are affected; the SET PROCESS command affects
only the initial kernel thread of a multithreaded process.


An application with such a behaviour clearly has a synchronization bug, that just does not show up, until you run the application really in parallel on multiple CPUs.

Volker.

[Willem Grooters]

Volker,
That implies that SET PROCESS/AFFINITY=CPUxx won't help.

Ian,
I'm not certain on the VMS version but I'm quite certain it's 7.3-something. So 8.2 commands are not applicable.

I fully agree it might be a synchronisation issue. Not really a surprise ;-). To solve the problem it will be needed to redo this part, I guess.

[Kris Clippeleyr]

Willem,
I'm not sure how badly the following will affect other applications/processes on the system, but couldn't you just simply disable the creation of kernel threads altogether (setting the system parameter MULTITHREAD to 0 (zero)). We once solved a problem by doing so. The app (actually one process of the app) was also multithreaded, and very CPU hungry. If we allowed kernel threads, all CPUs (four of them) were busy serving that single process. By preventing kernel threads to be created, we could keep 1 CPU busy with that process, while the other 3 could service the rest.
Hope this helps,
Kris (aka Qkcl)

[Ian Miller.]

SET IMAGE/FLAGS=NOMKTHREADS only works on Itanium images so THREADCP would be the correct command (its been around for serveral versions).

As Kris said you can disable kernel threads completely but it would be better to fix the application.

[Thomas Ritter]

Willem, we locked a number of processes to a single CPU in our 4 node 4 CPU 7.3-2 environment. The developers got ambitious with their multithreading and we suffered numerious bugchecks. After the processes were assigned to a single CPU no more problem ! I am sure there are problems with VMS in this area of multi-threading. In spite of conceptual brilliance of some these applications developed, when assigned to a single CPU, did not measurably affect the applications throughput. I all for keeping things simple.

[Volker Halle]

Thomas,

how did you 'lock a number of processes to a single CPU' ? That's the question.

SET PROC/AFF ?
THREADCP/DISABLE=MULTIPLE_KERNEL_THREADS ?

Volker.

[Thomas Ritter]

Try this

$ set proc "''process_name'"/AFFINITY/SET=1/permanent

[Thomas Ritter]

And with a $show cpu/all extract

CPU 1 State: RUN CPUDB: 81921880 Handle: 00000E90
Owner: 00000640 Current: 00000640 Partition 0 (WIZ22)
ChgCnt: 0 State: Present, In-Console, Reassignable
Process: * None *
Capabilities:
System: QUORUM RUN RAD0
Slot Context: C94A0400
CPU - State..........: RC, PA, PP, CV, PV, PMV, PL
Type...........: EV68 (21264/EV68CB 21264C), Minor = 5
Speed..........: 1000 Mhz
Variation......: VAX FP, IEEE FP, Primary Eligible
Serial Number..: AY20408008
Revision.......:
Halt Request...: 0
Software Comp..: 8.4
PALCODE - Revision Code..: 1.98-01
Compatibility..: 43
Max Shared CPUs: 4
Memory Space..: Physical = 00000000.00000000 Length = 0
Scratch Space..: Physical = 00000000.00000000 Length = 0
Bindings:
WIR_CGI_FOXPRD PID = 2C80190C Reason: Affinitized to this CPU
WIR_CGI_FOXPRD PID = 2C80110C Reason: Affinitized to this CPU
WIR_CGI_FOXPRD PID = 2C80090C Reason: Affinitized to this CPU
Fastpath:

[Volker Halle]

Thomas,

thanks.

It looks like actually the process and all its threads are bound to that CPU. This seems to contradict the help text of SET PROC/AFF.

Does $ SHOW SYS/MULTI/FULL show 3 kernel threads for process WIR_CGI_FOXPRD ?

Volker.

[Thomas Ritter]

OpenVMS V7.3-2 on node WIZ22 7/09/06 15:49:20.88 Uptime 2 13:43:57
AlphaServer ES45 Model 2B
Pid Process Name State Pri I/O CPU Page flts Pages
2C800818 ACME_SERVER HIB 8 111 0 00:00:00.53 384 538 M
2C801018 HIB 8
2C801818 HIB 10
[SYSTEM] 4304Kb
2C80090C WIR_CGI_FOXPRD HIB 4 79389 0 00:01:19.19 1483 1891 M
2C80110C HIB 4
2C80190C HIB 6
[WIZARD_100,ADDR_FOXPRD] 15128Kb

[Volker Halle]

Thomas,

thanks. So this seems to indiciate, that the help text for SET PROC/AFF is not correct.

Volker.

[Guy Peleg]

>multi-threaded application runs fine on a
>single-CPU system but in a multi-user
>environment, it crashes

Can you provide some more details on
how it crashes? System crash? process crash?
ACCVIO?

Guy

[Willem Grooters]

Just a process crash - data attached.

[Volker Halle]

Willem,

what does SDA> EXA/INS 808F8290 return ? What module is the exception PC in.

Although this 'may be' an underlying application problem, the exception PC and the most recent call frames on the traceback stack are from OpenVMS/PTHREAD/JAVA i.e. HP code !

Could you create a process dump for this situation (SET PROC/DUMP in LOGIN.COM or SET PROC/DUMP/PID=xxx) ?

Make sure you're up-to-date on PTRHEAD patches.

Volker.

[Volker Halle]

Willem,

while you're in SDA, please also check

SDA> EXA/INS 8083D830

This is more likely the real exception PC. The values in the vms condition printout from PHTREADs may be misleading.

The code most likely at fault here is in PTHREAD$RTL or JAVA$HPI_SHR, which are the most recent callers.

Volker.

[Volker Halle]

Willem,


Another - for curiosity: If a process's affinity is set to one (or more) processes, is that setting inherited by subprocesses started by that proces?


You can easily test this for yourself:

$ set proc/aff/set=0/perm
$ SHOW PROC
...
Soft CPU Affinity: off
Hard CPU Affinity Mask: 00000001
$ SPAWN
$ SHOW PROC
...
Soft CPU Affinity: off
$

The Hard CPU affinity is not inherited by the spawned subprocess.

Volker.

[Thomas Ritter]

Willem, what's the progress ? Did you try to set lock the process(s) to a single CPU ?

[Thomas Ritter]

Hello ?

[Willem Grooters]

Sorry for the delay but I hadn't heard from my collegue on the issue for weeks. I just asked him, he has pointed this developers and system managers on this thread but AFAWK they haven't done ANYTHING yet....But he'll contact them on the issue shortly so you'll be informed.

[Willem Grooters]

I guess they have given up, I didn't hear anything on the subject for weeks.