Processes waiting in MUTEX State

[Shailender_1]

Yesterday AMDS reported 8 process in MWAIT. I checked couple of individual process detail in amds and found that the WS size, ws quota was at 64000(Maximum WS for a process). Server is ALPHA ES40 with 5GB of Memory. Per my understanding process goes into Mwait when it needs more memory resource but in this case processes were already at the maximum range of memory can be allocated(64K pages). my Question here is if my 8 process were in mwait state and was allocated 64K pages each what should be the memory utilization of my server with 5GB of memory. Why I am asking this is, I generated the memory utilization for server and find that the memory utilization was only 40% which looks weared to me as per my inderstanding.

After checking the server I found


2023313B CSPPRI_113 MUTEX 8 1087 0 00:00:00.27 402 452 S
2022D93D CSPPRI_135 MUTEX 8 262 0 00:00:00.04 333 383 S
2022C93E CSPPRI_83 MUTEX 8 273 0 00:00:00.06 333 383 S
2022B53F CSPPRI_238 MUTEX 8 265 0 00:00:00.05 334 384 S
2022D140 CSPPRI_19 MUTEX 8 266 0 00:00:00.05 334 384 S
20225D44 CSPPRI_50 MUTEX 8 255 0 00:00:00.05 334 384 S
2022754C CSPPRI_143 MUTEX 7 389 0 00:00:00.09 334 384 S
2021D216 CSPPRI_250 MUTEX 8 179 0 00:00:00.04 311 358 S

[Hoff]

Using SDA, look up the value is in the process PCB efwm cell for these processes, and post it.

http://h71000.www7.hp.com/wizard/wiz_8023.html
http://h71000.www7.hp.com/wizard/wiz_0006.html

You'll usually either find a big negative value for a data structure (the JIB), or a small value that can be translated from RSNDEF.

The Internals and Data Structures Manual (The IDSM) is a good resource for understanding the plumbing.

Stephen Hoffman
HoffmanLabs

[Ian Miller.]

On recent versions of VMS
ANAL/SYSTEM
SDA> SHOW SUMMARY
will produce a display similar to SHOW SYSTEM but may do a better job in interpreting what the process is waiting for.

You need to get the value in PCB$L_EFWM as Hoff said. Post the results of
ANAL/SYSTEM
SDA> SHOW PROCESS/ID=2022213B


PWAIT$SDA can help in determining what the process is waiting for
http://eisner.encompasserve.org/~miller/pwait_sda010.zip

[Shailender_1]

All the process were in the Mutex state for about 20 minutes and then completed with success. I don't have these processes in the system running now so can't provide the data for analysis. Per amds all these processes had allocated 64K of WS , I was expecting an abrupt increase in the memory utilization of system for that period but didn't see any abrupt change in the memory utilization of the server it was about 40% only.

[labadie_1]

99% (at least) of the mutex states are false mutexes and in fact just a problem of quota depletion.

Next time post the value asked by Ian and Hoff
If it is

1 means bytlm problem
2 means tqelm problem

And forget about wsquota, wsextent, wssize or such quotas. It has no relationship at all with a mutex state.

[Hoff]

If you should wish to know the details around how the mutual exclusion (mutex) semaphore construct is implemented and used on OpenVMS, skim the IDSM; the OpenVMS Internals and Data Structures Manual.

Mutexes activity is a very normal and very common synchronization mechanism on OpenVMS, and mutexes are acquired and released all the time.

What happened here was that these ended up being held for rather longer than usual. Most mutex-related activity is very fast, and is thus not (usually) visible on the SHOW SYSTEM display.

Details from the PCB$L_EFWM cell are often the central key to determining whether this was one of various quotas (and as labadie writes, it's usually not the working set), or if this was some other mutex-protected resource involved.

There are mutexes hanging off all manner of structures, and off various devices.

Again, the IDSM describes the basic mutex semaphore mechanism in some detail.

Stephen Hoffman
HoffmanLabs

[Shailender_1]

I could retrieve the SDA output I had collected during the process was live. Please find attached for refrence

Thanks for all your help

[Jur van der Burg]

That's a giveaway:

State WTTQE Flags 00000000

Timer entries remaining 0/16

The process runs out of Timer queue quota (TQElm). You might want to increase that for the involved processes.

Jur.

[Ian Miller.]

CSPPRI_157
Event flag wait mask=JIB address so it was waiting for a pooled quota bytlm or TQELM

and from
"Timer entries remaining 0/16"

I'd say it was TQELM.

[Volker Halle]

Look at the usage of timer TQEs in those processes. SDA> SHOW PROC/TQE

Consider setting a realistic TQE limit for those processes based on how many TQE timers you expect the process to have currently outstanding.

Volker.