Problem with Temporary Mail Box

[#1]

Hi,

One process P1 is creating a temporary mail box using a system function SYS$CREMBX.

Second process P2 is trying to establish a connection with temporary mail box created by P1 using system function SYS$ASSIGN, but SYS$ASSIGN function is sometimes returning SS$_IVDEVNAM.

Can anyone help me why SYS$ASSIGN is sometimes returning IVDEVNAM error.

Note: Its a intermittent problem, means most of the times sys$assign is working properly i.e returning SS$_NORMAL and assigning a channel but sometimes (aprox. 1 out of 10) sys$assign is returning IVDEVNAM.

What I know about IVDEVNAM is that, sys$assign will return this value if device i.e. mailbox is no more existing. Can I know what can cause mailbox to be deleted on its own or what could be the other reason that sys$assign is returning IVDEVNAM.

Looking forward for quick help.

Regards,
ajaydec

[Hein van den Heuvel]

This is 100% certain to be a programming error
Typical causes

- mailbox was temporary and creator is already gone.
- Logical name LNM$TEMPORARY_MAILBOX not alsways correct.
- uninitialized (C) variable / buffer overflow
- hardcoded length restrictions (3 or 4 char mbx name?)
- creator failed to check error (no mode channels, pool, ...)

When you write 1 out of 10 is that
- once per day, 1 out of 10 days.
- several times per minute/hour from one process?
- all at the same time from multiple processes, some of which fail?

Finally, how does p2 know the name of the mailbox to use? hardcoded logical name?

hth,
Hein.

[#3]

Hi Hein,


- mailbox was temporary and creator is already gone.
Creator is not gone, its there

- Logical name LNM$TEMPORARY_MAILBOX not alsways correct.
I have already make sure LNM$TEMPORARY_MAILBOX logical is always correct.

- uninitialized (C) variable / buffer overflow
Sorry, I don't understand what you mean by uninitialized (C) variable can you elaborated it, there is no buffer overflow.

- hardcoded length restrictions (3 or 4 char mbx name?)
There is no hard coded restriction, mbx name can be of any length.

- creator failed to check error (no mode channels, pool, ...)
Creater check for SS$_NORMAL return value from SYS$CREMBX, if return value is other than SS$_NORMAL, then creator will log the error.

how does p2 know the name of the mailbox to use? hardcoded logical name?
P2 came to know the mailbox using a logical, its not hardcoded logical, logical value will depend on the mailbox name created by the process p1.

When you write 1 out of 10 is that
- once per day, 1 out of 10 days.
- several times per minute/hour from one process?
- all at the same time from multiple processes, some of which fail?

I'll try to elaborate the situation once again in detail.
We have a product which start 10 process P1-P10. When we start the product all the 10 process will got started and when we shutdown it, all the 10 process will get shutdown.
So first of all P1 got started after that P2, P3 and so on.
P1 is the main process and it will communicate with every other process P2-P10. So, P2-P10 will create a temporary mailbox ( mailbox name will be something like MBA453, MBA454, MBA455 and so on...)
(don't get confused I am interchanging P1 and P2 from previous)
Now P1 will try to establish a channel with mailbox created by each and every individual process. But sometimes P1 process is not able to establish channel with mailbox created by P2 or P3 or P4... or P10 and is giving IVDEVNAM error. And most of the times P1 is able to establish channel with each and every mailbox.

Also note:
1) This problem occurs only on a multiprocessor system. When I run the product on single-processor system, its not giving the IVDEVNAM error.
2) This problem occurs only during the startup of the product.

Hope I didn't confuse much.

Regards,
-ajaydec

[Kris Clippeleyr]

Ajaydec,


We have a product which start 10 process P1-P10. When we start the product all the 10 process will got started and when we shutdown it, all the 10 process will get shutdown.
So first of all P1 got started after that P2, P3 and so on.
P1 is the main process and it will communicate with every other process P2-P10. So, P2-P10 will create a temporary mailbox ( mailbox name will be something like MBA453, MBA454, MBA455 and so on...)
(don't get confused I am interchanging P1 and P2 from previous)
Now P1 will try to establish a channel with mailbox created by each and every individual process. But sometimes P1 process is not able to establish channel with mailbox created by P2 or P3 or P4... or P10 and is giving IVDEVNAM error. And most of the times P1 is able to establish channel with each and every mailbox.


Now there's a possibility of mis-synchronisation.
Are you sure that all processes (P2 thru P10) have created there mailboxes, before process P1 tries to assign channels to them?
On a multi-CPU system (and even on a single CPU system), P1 might already be trying to assign channels before all other processes have had a change to create the mailboxes.

What are you using for synchronisation?

Regards,
Kris (aka Qkcl)

[Hein van den Heuvel]

Ok much clearler, and as Q says, clearly broken. Just too bad it seemed to work correctly for a while.

>> Now P1 will try to establish a channel with mailbox created by each and every individual process.

So how does P1 know that its slaves have gotten to the point of creating the mailbox?

>> 1) This problem occurs only on a multiprocessor system.

That makes it 100% proof that the application code is broken. Whenever you there is a difference between single CPU and 1 and when running with changed or default priority, the application is broken. Grarantueed. Every time. 1000nds of cases have proven that.

>> When I run the product on single-processor system, its not giving the IVDEVNAM error.

That's just bad luck.
Do a $SHOW PROC/CONT on P1 and it might break.


>> 2) This problem occurs only during the startup of the product.

Of course, because there is a race condition in the startup as described.

Realizing that the MBX is create most of the time, I would suggest a simple retry mechanism, and not to add an eleborat handshake.

Just loop over all slaves.
If channel assigned, goto next
Else try assign.
If none left, done.
Else wait 10 ms and try again up to 10 times.

todo = SLAVES;
retry = 10
while (1) {
for (i=0; i chan = &slave[i]->chan;
if (!*chan) {
status = assign &chan ...
if (status & 1) {
todo--;
} else {
last_status = status;
last_slave = i;
}
}
}
if (!todo && retry--) break;
wait_a_while();
}
if (todo) {
printf STDERR "Failed to assign all channels"...
:
return
}


Good luck!
Hein.

[Hoff]

I'd use locks for process coordination. For instance, the controlling primary process seeks an exclusive lock the "FOO_I_AM_ALIVE" lock resource and all the servers queue an incompatible lock against this lock. If a server process is ever granted the lock, it exits. (The controller is gone, and the servers should go, too.)

I'd look to IP network connections, or maybe to ICC if this is a cluster. Mailboxes are pretty old design, and -- in general -- I'd not tend to write new code that ties you into OpenVMS platform interfaces if there's an existing and standard API that avoids such. This means something like IP. (ICC is most certainly a platform interface, but it lets you operate more easily within a cluster.)

And in general, I'd probably look for and look to use existing process management tools, rather than writing these anew. For instance, even inetd and cluster aliases and RPC calls or other such can deals with various of the aspects here for you. Web servers deal with server processes all the time, too, and there are various ways to use web-based servers.

Yeah, I know, you were probably told to use mailboxes by the lead designer or manager. (Oh, well.) There's a mailbox demo available on the Freeware here:
http://mvb.saic.com/freeware/freewarev80/hoffman_examples/
See the mbxdemo.* files.

Now if you're going to use these platform APIs, you'll want to use the DECw$Term_Port stuff to allow the created server processes to have a DECterm terminal device associated, or you'll want to start these processes with a WSA device. In either case, this because it makes it massively more easy to debug code when you can activate the created image with the debugger around, and walk through it. I'm here inferring that you're not using the debugger now, and this may or may not be a correct inference. Here's a demo:

http://mvb.saic.com/freeware/freewarev80/hoffman_examples/create_decterm.c

Stephen Hoffman
HoffmanLabs LLC

[#7]

One thing I can assure you there is no syncronization error.
Actually P1 will also create a mailbox (say MBA345), and it will do a sys$qiow to this mailbox.

Now when P2 process has created its mailbox (say MBA346) then it will write the coded value in P1 mailbox(MBA345), since P1 is continuosly reading MBA345 as soon P2 writes into it P1 will read the coded value and it will came to know that it has to establish a channel with mailbox (MBA346) created by P2.

Similarly, P1 process will establish channel with rest of the mailbox created by different process.

Regards,
ajadec

[Hoff]

There have been latent synchronization errors in code that has been heavily used (by all of us!) and particularly in code that has been running without incident for over twenty years. This in code written that was by some of the best engineers any of us here might know.

That the problem here occurs only on a multiprocessor system makes it almost certain that there one or more synchronization errors latent and lurking here.

Here is a list of some of the more common synchronization bugs that can exist:

http://h71000.www7.hp.com/wizard/wiz_1661.html

SMP and SMT are among the most common triggers for and very commonly expose latent synchronization bugs. SMP and SMT can and do exercise the application synchronization and related logic to a degree that uniprocessors and single-threaded applications can not even approach.

Fire up the debugger and/or integrate some debugging (or both), and go in for a look. And do take a look at the sequencing of the mailboxes here. And yes, the debugger and the introduction of integrated debugging can easily alter the behavior of a latent synchronization bug, or can entirely mask it.

[John Gillings]

ajaydec,

Well, I'm confused. I still don't understand how any of these processes know the mailbox name created by other processes. The default for LNM$TEMPORARY_MAILBOX is to place the logical name in LNM$JOB, which means it's only visible to processes in the same job tree. It's not clear in this case how the processes are related. If they're not subprocesses under the same master how are the names exchanged? Has LNM$TEMPORARY_MAILBOX been redefined?

This gets into a chicken and egg situation, you need a mailbox to communicate between processes, BUT you need to communicate the name of the mailbox.

Your symptom suggests the step of communicating the mailbox name is broken. It's not clear if you've taken the very obvious step of simply printing out the exact mailbox name you're attempting to open when $ASSIGN returns the IVDEVNAM? If it's via logical name, how do you know it's been defined? If it's a timing issue, try a short delay and retry the $ASSIGN - note this is NOT a recommended "fix", just a way to confirm diagnosis.

One common model for this type of situation is to have the master process (your P1) create a PERMANENT mailbox, with a system wide, well known logical name. The clients create a temporary mailbox then send a message to the master via the permanent mailbox, including the name of their temporary mailbox in the message. The master then opens a channel to the client mailbox and two way communication is established.

[#10]

John,

One common model for this type of situation is to have the master process (your P1) create a PERMANENT mailbox, with a system wide, well known logical name. The clients create a temporary mailbox then send a message to the master via the permanent mailbox, including the name of their temporary mailbox in the message. The master then opens a channel to the client mailbox and two way communication is established.

We are also doing the same, might be I am not able to explain it properly. The only difference is that we are not sending the name of temporary mailbox in the message, rather we have defined a logical "TEMP_MAILBOX_" which will contain the name of temporary mailbox created by process with process id .

So while calling sys$assign, P1 sends "TEMP_MAILBOX_" as one of its parameter and in return sometimes we are geting IVDEVNAM error.

Regards,
ajaydec

[Hein van den Heuvel]

>> One thing I can assure you there is no syncronization error.

And one thing I can assure you, and I'll put a decent wager on this if you are interested, is that there is!
There has to be! Otherwise you would not be asking the question right?

Your mindset to solve this problem has to be 'there is a communication / synchronization error here somewhere in this application code. You must assume VMS is doing the right thing, and that the program is doing something wrong somewhere. Maybe error handling, maybe logic, maybe both.

>> Actually P1 will also create a mailbox (say MBA345), and it will do a sys$qiow to this mailbox.
Now when P2 process has created its mailbox (say MBA346) then it will write the coded value in P1 mailbox(MBA345), since P1 is continuosly reading

Well duh! Now you tell us! :-).
That sounds a lot more serious, but not certain. For example, the iosb for the qiow read is not 'double booked' by accident (same variable used for different IO).

>> we are not sending the name of temporary mailbox in the message, rather we have defined a logical "TEMP_MAILBOX_" which will contain the name of temporary mailbox created by process with process id .

Why not send the mailbox name? Might as well!
Why go through the trouble of creating a logical name?
In what table?
Could that be failing?
Mind you.. you don't actually have to send anything special. Just 'Hi!' will do, as the IOSB after the read will contain the PID for the sender.

I would recommend an LNM trace of the process startup:
$ANALYZE/SYSTEM
SDA>LNM LOAD ! If not already loaded
SDA>LNM START TRACE
SDA>SPAWN "start application"
SDA>LNM STOP TRACE
SDA>SET OUTPUT LNM_SHOW_TRACE.LOG
SDA>LNM SHOW TRACE

Good luck!
Hein.

[#12]

Process P2-P10 will write to P1 mailbox as follows using sys$qiow:

status = SYS$QIOW( 0,channel,code,&iosb,0,0,buffer,buf_size,0,0,0,0 );

where:

channel is the channel no. assigned with the P1 mailbox.
code is IO$_WRITEVBLK | IO$M_NOW
buffer = ""
buf_size = 164.

Process P1 is reading the mailbox as follows:

qiow_status = sys$qiow(efn_flg, DECEDI$DS_CB_REMOTE.ds_mbx_channel_in, IO$_READVBLK,&iosb_blk,0,0,&dsr,sizeof(dsr),0,0,0,0);

where:

efn_flg value is obtain as follows:

LIB$GET_EF ( &efn_flg );

Regards,
ajaydec

[Colin Butcher]

This does sound like a design flaw in the application. It's a classic example of one of the problems that commonly occurs when moving code from a uniprocessor environment to a multiprocessor environment. The introduction of parallelism often exposes design flaws in the state machine. I've had it happen to me with code that "worked perfectly" for over 7 years, then promptly blew up when we tested it on a multiprocessor environment. I'd failed to correctly guard against a race condition that I'd not considered as the implicit assumption was that things would happen in the correct sequence. That's not always true in a multiprocessor environment.

Please look closely at the way in which you both synchronise and serialise access to the mailbox during the startup sequence. It's not enough to do one thing at once, your code also has to do the right thing in the right order. The lock manager is the mechanism you want to be using to implement the serialisation and synchronisation mechanisms. Don't "roll your own" with flag bits.

As for inter-process communication - if it's only a small amount of data to be passed around, consider using the extended lock value block. If it's a lot of data shared between the processes, consider using a global section and having sufficiently fine granularity of the data structures protected by locks so as to minimise wait states. Don't forget about termination mailboxes either.

Have fun.

Cheers, Colin (http://www.xdelta.co.uk).

[#14]

Hi All,

Thanks for your help and time, but still I am not able to get it right. I'll again try to explain the process in detail:

1) P1 process will start and it will create permanent mailbox and will read it as follows:

qiow_status = sys$qiow(efn_flg, DECEDI$DS_CB_REMOTE.ds_mbx_channel_in, IO$_READVBLK,&iosb_blk,0,0,&dsr,sizeof(dsr),0,0,0,0);

where:

efn_flg value is obtain as follows:

LIB$GET_EF ( &efn_flg );

2) P2-P10 process is being created and each process does the following:

i) Assign a channel to mailbox created by process P1 using sys$assign.
ii) Create a temporary mailbox.

status1 = SYS$CREMBX( 0, channel, msg_size, 0, 0, 0, &mbx_log_name );

where:
mbx_log_name is logical to temporary mailbox name
iii) pings/writes to the mailbox created by P1 as follows:
status = SYS$QIOW( 0,channel,code,&iosb,0,0,buffer,buf_size,0,0,0,0 );

where:

channel is the channel no. assigned with the P1 mailbox.
code is IO$_WRITEVBLK | IO$M_NOW
buffer = ""
buf_size = 164.
iv) After this waits for process P1 to write in its temporary mailbox as follows:
status = SYS$QIOW(efn_flag,io_channel_in,IO$_READVBLK,0,0,0,&P1_reply,sizeof(P1_reply),0,0,0,0);


3) Once Process P2-P10 writes in mailbox of process P1, process P1 will assign a channel to temporary mailbox of process P2-P10. Temporary mail box name will be defined in logical tmp_mbx_name_, where process_id is the process id of the process P2-P10.

Could anyone help me to know where, syncronization and parallelism is lacking because of which sometimes, I am getting IVDEVNAM error by process P1 when it is trying to assign a channel to temporary mailbox created by process P2-P10.

Regards,
ajaydec

[Robert Gezelter]

ajaydec,

I agree with Hein, Hoff, Colin, et al that this is a synchronization error. The IVDEVNAM error can occur for any number of reasons, not necessarily only the obvious ones.

Personally, I prefer to chase these problems analytically, not by gathering a lot of data, which, in the end, will not be helpful. However, if one wishes to exclude some possibilities, one can collect the names of the devices being used by the ASSIGN call in a separate array, and then look at the array with the debugger. DO NOT use printf statements, as the extra time has a good chance of disrupting the timing behavior.

- Bob Gezelter, http://www.rlgsc.com

[Guenther Froehlin]

So far I don't see a coding problem except that:

"iv) After this waits for process P1 to write in its temporary mailbox as follows:
status = SYS$QIOW(efn_flag,io_channel_in,IO$_READVBLK,0,0,0,&P1_reply,sizeof(P1_reply),0,0,0,0);"

...is not using an IOSB.

Here is what the book says about $ASSIGN and INVDEVNAM:

"No device name was specified, the logical name translation failed, or the device or mailbox name string contains invalid characters."

To get more information I would output/log the translation of the logical used in the $ASSIGN. My bet is that the transmission and/or handling of the logical name has a problem.

I don't see a synchronization problem.

/Guenther

[Hoff]

[[[One common model for this type of situation is to have the master process (your P1) create a PERMANENT mailbox, with a system wide, well known logical name. The clients create a temporary mailbox then send a message to the master via the permanent mailbox, including the name of their temporary mailbox in the message. The master then opens a channel to the client mailbox and two way communication is established.]]]

The uni-directional mailbox design here is certainly quite typical.

Conversely, permanent mailboxes can often serve as a way to trigger run-time bugs in my experience. Permanent mailboxes require cleanup, where temporary mailboxes do not.

I can't think of a recent case where I've chosen to use a permanent mailbox, and I have typically removed such usage out of code I'm maintaining as it can help better deal with failures and restarts.

Mailboxes themselves are not something I'd tend to use in new applications, save for specific cases. If you're willing to overtly tie into OpenVMS, I'd suggest ICC. If you're more interested in portability, I'd look to use middleware or IP (IPv4 or IPv6) sockets.

[[[We are also doing the same, might be I am not able to explain it properly. The only difference is that we are not sending the name of temporary mailbox in the message, rather we have defined a logical "TEMP_MAILBOX_" which will contain the name of temporary mailbox created by process with process id .]]]

I'd suggest not naming an object (a variable, file, mailbox, etc) for what it is (since any of us can look at same and figure that out), but to name the object for what it is used for, and for what particular application is using it.

Something akin to FOO_CLIENT_pid, for instance, identifies FOO as the application facility associated with the mailbox, and that the mailbox is a client mailbox. Or FOO_scsnode_CLIENT_pid, if you're working with a cluster-visible object. Use of the facility also avoids colliding with some other programmer which chose to use the logical name TEMP_MAILBOX_pid, too...

Opinion only, of course.

[Larry Bohan]

maybe consider setting things up such
the the temporary mailbox logicals (names)
are entered in a group logical table?

iirc, this was (or similar to) the VMS 3.x
behavior.

we had some old code (mid/late 1980's) that required this behavior, but there might well be some larger (newer) reasons why this would *not* be good idea in the general case.

$ DEFINE/TABLE=LNM$PROCESS_DIRECTORY -LNM$TEMPORARY_MAILBOX LNM$GROUP

..or (approximately):

l_sts = lib$set_logical(
$DESCR("LNM$TEMPORARY_MAILBOX"),
$DESCR2("LNM$GROUP"),
$DESCRL("LNM$PROCESS_DIRECTORY"),
0, 0);

[Jan van den Ende]

@Larry,

Just be aware that ANYBODY _WRITING_ anything in that mailbox now requires GRPNAM privilege!

Proost.

Have one on me.

jpe

[Hoff]

[[[Just be aware that ANYBODY _WRITING_ anything in that mailbox now requires GRPNAM privilege!]]]

Privileges are initially checked when an object (file, device, mailbox, logical name table, queue, etc) is access (or created), and not generally checked (again) at run-time as the channel is accessed. (Yes, there are specific operations that might involve extra checks -- tossing an IO$_DIAGNOSE function at the device, for instance -- but these are not typical).

And a programmer can set a group mailbox to any protection that might be required. (The associated logical name goes in the group table and grpnam can be (is) required there, but the mailbox itself has its own and separate protection.)

And if you're so inclined, you can reconfigure the group logical name table protection for a table, as these tables are also security objects and have ownership, protections and ACL capabilities.

Further, programmers will generally want to confirm the mailbox device ownership and the device protection are as expected, and any ACL that might be associated with the device. (Mailboxes are an ideal site for injecting messages into an environment; there can be security implications of a security-critical environment. This traffic can potentially require protection.)

[John Gillings]

re Jan:

>Just be aware that ANYBODY _WRITING_
>anything in that mailbox now requires
>GRPNAM privilege!

Not true. GRPNAM is required to CREATE the mailbox because part of the creation involves defining a logical name in LNM$GROUP. Once the mailbox exists, anyone in the group can *translate* the logical name without privilege. Assigning a channel to, reading or writing the mailbox is controlled by the security of the mailbox, which is completely independent of the LNM$TEMPORARY_MAILBOX logical name. There's a PROTMASK parameter to $CREMBX, and of course, a mailbox may have an ACL.

According to the documentation, the default protection of a mailbox comes from the "mailbox template", but I can't find one on a V8.3 system (see SHOW SECURITY/CLASS=SECURITY_CLASS *), nor does the command to change it seem to work:

$ SET SECURITY /CLASS=SECURITY_CLASS /PROFILE=TEMPLATE=MAILBOX /PROTECTION=(S:RWPL,O:RWPL,G,W) DEVICE

Regardless I still get mailboxes with W:RWPL access by default.

[John Gillings]

>"mailbox template", but I can't find one >on a V8.3 system

Found it...

$ show security/class=security device

DEVICE object of class SECURITY_CLASS
Owner: [SYSTEM]
Protection: (System: RW, Owner: RW, Group: R, World: R)
Access Control List:

Template: DISK
Owner: [SYSTEM]
Protection: (System: RWPL, Owner: RWPL, Group: R, World)
Access Control List:
...

Template: MAILBOX
Owner: [0,0]
Protection: (System: RWPL, Owner: RWPL, Group: RWPL, World: RWPL)
Access Control List:
...


but I still can't make it work!


Template: MAILBOX
Owner: [0,0]
Protection: (System: RWPL, Owner: RWPL, Group, World)
Access Control List:

(so group and world are no access)

but..

$ create/mailbox newmbx
$ show dev/full newmbx

Device MBA58676:, device type local memory mailbox, is online, record-oriented
device, shareable, mailbox device.

Error count 0 Operations completed 0
Owner process "" Owner UIC [OM_PROD]
Owner process ID 00000000 Dev Prot S:RWPL,O:RWPL,G:RWPL,W:RWPL
Reference count 1 Default buffer size 256


which doesn't match the protection in the template. Putting an explicit protection on the $CREMBX does as expected:

$ create/mailbox mbx2/prot=(s:rwpl,o:rwpl,g:rw,w)
$ show dev/full mbx2

Device MBA58680:, device type local memory mailbox, is online, record-oriented
device, shareable, mailbox device.

Error count 0 Operations completed 0
Owner process "" Owner UIC [OM_PROD]
Owner process ID 00000000 Dev Prot S:RWPL,O:RWPL,G:RW,W
Reference count 1 Default buffer size 256


The security templates are so well buried in the depths of the OS, I guess not enough people use them, and maybe they still have a few bugs?

[Hoff]

John: Make sure VMS$OBJECTS is shared across all nodes in a cluster; that if there are multiple system disks, that there is a definition (usually) in SYLOGICALS that points to one central copy of this file. (I've seen multiple copies of the file lead to weirdness.)

[John Gillings]

Hoff,

> shared across all nodes in a cluster

Remember you're talking to Mr "Let's hassle OpenVMS engineering about very poor built in management of cluster common environment files" ;-) I probably reported some of those "multiple copies of the file lead to weirdness". :-)

(besides, the node in question is standalone)

[Jan van den Ende]

John,

>>>
Not true. GRPNAM is required to CREATE the mailbox because part of the creation involves defining a logical name in LNM$GROUP.
<<<

Thank you! One is never too old to learn. This DOES give me the means to determine which privileges can be revoked from which users without breaking an application.

Proost.

Have one on me.

jpe