topic Re: Processes waiting in MUTEX State in Operating System - OpenVMS

Processes waiting in MUTEX State

Shailender_1 — Thu, 15 Mar 2007 19:41:11 GMT

Yesterday AMDS reported 8 process in MWAIT. I checked couple of individual process detail in amds and found that the WS size, ws quota was at 64000(Maximum WS for a process). Server is ALPHA ES40 with 5GB of Memory. Per my understanding process goes into Mwait when it needs more memory resource but in this case processes were already at the maximum range of memory can be allocated(64K pages). my Question here is if my 8 process were in mwait state and was allocated 64K pages each what should be the memory utilization of my server with 5GB of memory. Why I am asking this is, I generated the memory utilization for server and find that the memory utilization was only 40% which looks weared to me as per my inderstanding.

After checking the server I found

2023313B CSPPRI_113 MUTEX 8 1087 0 00:00:00.27 402 452 S
2022D93D CSPPRI_135 MUTEX 8 262 0 00:00:00.04 333 383 S
2022C93E CSPPRI_83 MUTEX 8 273 0 00:00:00.06 333 383 S
2022B53F CSPPRI_238 MUTEX 8 265 0 00:00:00.05 334 384 S
2022D140 CSPPRI_19 MUTEX 8 266 0 00:00:00.05 334 384 S
20225D44 CSPPRI_50 MUTEX 8 255 0 00:00:00.05 334 384 S
2022754C CSPPRI_143 MUTEX 7 389 0 00:00:00.09 334 384 S
2021D216 CSPPRI_250 MUTEX 8 179 0 00:00:00.04 311 358 S

Re: Processes waiting in MUTEX State

Hoff — Thu, 15 Mar 2007 20:51:03 GMT

Using SDA, look up the value is in the process PCB efwm cell for these processes, and post it.

http://h71000.www7.hp.com/wizard/wiz_8023.html
http://h71000.www7.hp.com/wizard/wiz_0006.html

You'll usually either find a big negative value for a data structure (the JIB), or a small value that can be translated from RSNDEF.

The Internals and Data Structures Manual (The IDSM) is a good resource for understanding the plumbing.

Stephen Hoffman
HoffmanLabs

Re: Processes waiting in MUTEX State

Ian Miller. — Fri, 16 Mar 2007 04:51:38 GMT

On recent versions of VMS
ANAL/SYSTEM
SDA> SHOW SUMMARY
will produce a display similar to SHOW SYSTEM but may do a better job in interpreting what the process is waiting for.

You need to get the value in PCB$L_EFWM as Hoff said. Post the results of
ANAL/SYSTEM
SDA> SHOW PROCESS/ID=2022213B

PWAIT$SDA can help in determining what the process is waiting for
http://eisner.encompasserve.org/~miller/pwait_sda010.zip

Re: Processes waiting in MUTEX State

Shailender_1 — Fri, 16 Mar 2007 04:59:26 GMT

All the process were in the Mutex state for about 20 minutes and then completed with success. I don't have these processes in the system running now so can't provide the data for analysis. Per amds all these processes had allocated 64K of WS , I was expecting an abrupt increase in the memory utilization of system for that period but didn't see any abrupt change in the memory utilization of the server it was about 40% only.

Re: Processes waiting in MUTEX State

labadie_1 — Fri, 16 Mar 2007 05:27:33 GMT

99% (at least) of the mutex states are false mutexes and in fact just a problem of quota depletion.

Next time post the value asked by Ian and Hoff
If it is

1 means bytlm problem
2 means tqelm problem

And forget about wsquota, wsextent, wssize or such quotas. It has no relationship at all with a mutex state.

Re: Processes waiting in MUTEX State

Hoff — Fri, 16 Mar 2007 14:23:55 GMT

If you should wish to know the details around how the mutual exclusion (mutex) semaphore construct is implemented and used on OpenVMS, skim the IDSM; the OpenVMS Internals and Data Structures Manual.

Mutexes activity is a very normal and very common synchronization mechanism on OpenVMS, and mutexes are acquired and released all the time.

What happened here was that these ended up being held for rather longer than usual. Most mutex-related activity is very fast, and is thus not (usually) visible on the SHOW SYSTEM display.

Details from the PCB$L_EFWM cell are often the central key to determining whether this was one of various quotas (and as labadie writes, it's usually not the working set), or if this was some other mutex-protected resource involved.

There are mutexes hanging off all manner of structures, and off various devices.

Again, the IDSM describes the basic mutex semaphore mechanism in some detail.

Stephen Hoffman
HoffmanLabs

Re: Processes waiting in MUTEX State

Shailender_1 — Mon, 19 Mar 2007 04:54:02 GMT

I could retrieve the SDA output I had collected during the process was live. Please find attached for refrence

Thanks for all your help

Re: Processes waiting in MUTEX State

Jur van der Burg — Mon, 19 Mar 2007 09:00:37 GMT

That's a giveaway:

State WTTQE Flags 00000000

Timer entries remaining 0/16

The process runs out of Timer queue quota (TQElm). You might want to increase that for the involved processes.

Jur.

Re: Processes waiting in MUTEX State

Ian Miller. — Mon, 19 Mar 2007 09:05:08 GMT

CSPPRI_157
Event flag wait mask=JIB address so it was waiting for a pooled quota bytlm or TQELM

and from
"Timer entries remaining 0/16"

I'd say it was TQELM.

Re: Processes waiting in MUTEX State

Volker Halle — Mon, 19 Mar 2007 12:58:16 GMT

Look at the usage of timer TQEs in those processes. SDA> SHOW PROC/TQE

Consider setting a realistic TQE limit for those processes based on how many TQE timers you expect the process to have currently outstanding.

Volker.