VMS error 2264 --%SYSTEM-W-MBFULL, mailbox is full

[Joseph Drozdz]

How can I see what's going on with the system mailbox.

%BCADOS-F-COMNDPROCFSPWNS, %DosCmdProc-F-Spawning Unable to spawn subprocess, VMS Status is 2264
%TRACE-F-TRACEBACK, symbolic stack dump follows

[Richard Whalen]

the "system" says that it is a (VMS) system level message. Your application is attempting to spawn a subprocess and during that process there is an attempt to put more data into a mailbox that spawn uses than the mailbox can hold.

[Hoff]

There are many, many, many mailboxes.

In cases such as this, the mailboxes tend to be application-specific. Accordingly, check with whomever is maintaining the DosCmdProc tool within the BCADOS environment.

Assumptions here would be that the subprocess isn't pulling off data off the mailbox fast enough, or too much data got stuffed into the mailbox, or the default mailbox size and default quota is too small or the application didn't specify or didn't specify sufficient of same when it created the mailbox, or some combination of these factors.

The stackdump itself may or may not point to the particular failure.

If you are the one charged with this code maintenance, there are tools such as the MBX$SDA extension for ANALYZE/SYSTEM that can be used to peek into the mailbox, and the code itself can certainly be instrumented to figure out what's happening. And this assumes that the mailbox input and the output and processes are operating appropriately. Depending on the context and the particular error, sometimes you can hack around this by increasing the DEF*MBX* system parameters. But if the subprocess is dead or is stuck, this increase won't help.

In any event, the best solution is to check with the organization maintaining DosCmdProc and BCADOS. Or to check within the source code. Or -- if you have the source code but are not in a position to deal with same -- to get somebody in to take a look at it, and at the stackdump (stackdumps tend to imply unhandled errors), and related run-time details.

Stephen Hoffman
HoffmanLabs LLC

[John Gillings]

Joseph,

When SPAWNing a subprocess, VMS uses a mailbox to send context from master to subprocess. By default, the global and local symbol tables are copied to the subprocess, as is the process logical name table.

Find the code, presumably LIB$SPAWN. Check for flags CLI$M_NOCLISYM and/or CLI$M_NOLOGNAM. If they're NOT specified, consider if the subprocess really needs symbols and logical name. If not, add the flags - that should significantly decrease the volume of data passed through the mailbox. You may see a performance improvement.

Check your process - have there been any additional symbols or logical names defined since this operation was working (assuming it has ever worked?) Look for excessively long logical names or symbols. Try deleting them. As Hoff suggested, you can increase the width and capacity of the mailbox with SYSGEN parameters DEFMBX* (BUT increasing these values may increase system wide consumption of non-paged pool, so be careful)

[Hein van den Heuvel]

I'm with my peers in that you are probably trying to push 10 pounds worth of logicals in a 4 pound bag. I'm kinda surprised VMS does not just wait for the child to suck out the description, but there you have it.
I'll see if I can find my old notes on this.

You really want to ask yourself whether the child needs any or all of those veriables.
Sometimes you can use JOB or application table stored logicals to lighten the load, sometimes only process logicals solve the needs.

btw.. while I do expect this 2264 to be decimal and thus a perfectly 'reasonable' (by some defintion of reason) MBFULL, is there any chance it is displayed as a hex number?
$ exit %x2264
%SYSTEM-F-INSSWAPSPACE, insufficient swap file space available

I think that message is retired. Help/mess does not have it. But it is a little bit in the right direction...

Cheers.
Hein.

[DECxchange]

Hello,
This looks a lot like you have an application program or process that is exiting because another process that is using that mailbox is not reading the mailbox.

It looks like the application program is not handling the exception properly and is exiting because the target mailbox is not being read by another program. You might want to investigate what application programs or processes should be running and if some of these are not running. There could be a number of application issues causing this.
You could do a show system and see what is running. If you know what should be running, this command will tell you if it is running or not. So I guess it depends on your knowledge of the application and what needs to be started/restarted.
I know that might not be very useful, but I think the bottom line is not all of your application programs are running for some reason.

Is this a VAX or Alpha? What version of VMS are you using? Has this happened before? And so on.

[Ian Miller.]

Are there lots of process logical names or DCL global symbols in the process that is doing the spawn?

This sort of data is passed to the spawned subprocess using a mailbox.

[Joseph Drozdz]

To answer a few questions asked I am currently running OPen VMS 7.3. There is no hex decimal in the error message. What can I do to just analyze this. Is there a command that I can start with to begin tracing this error code> Thanks all!

[Hoff]

[[[To answer a few questions asked I am currently running OPen VMS 7.3.]]]

I do tend to ask for that detail, but in this case the OpenVMS VAX or OpenVMS Alpha version is probably immediately relevant.

The OpenVMS code in this area has been consistent and very heavily tested for eons and for many, many sites; this looks to be an application-level error; some sort of unhandled exception lurking within the application source code.

There are an infinite number of triggers.

[[[What can I do to just analyze this.]]]

Do you have the source code? If so, you or somebody else will have to take a look at the context around the failure -- we're all guessing at the context and the application environment here; the source code and the run-time context is the source of authoritative information. And how the trigger can be identified, assuming the support organization for this package doesn't immediately recognize the error.

[[[Is there a command that I can start with to begin tracing this error code> Thanks all!]]]

The tool used here -- if you have the source code -- is the debugger. Probably not what you want, and only really easily useful if you have the source code and/or can perform a debug build.

If you don't have access to the source code, there's not really much you can do here to troubleshoot this beyond what's been suggested in the thread. Well, there are ways to peek into the processing, but they're cryptic and entirely non-trivial -- you're literally reverse-engineering the application and the run-time environment here. And that's not easy to describe.

As has been mentioned a couple of times, the most likely trigger is a subprocess that's too slow, or that has failed. But there can be any number of upstream triggers here.

If this environment has been here for a long time and there's no source code and there's no obvious trigger and no recent changes to the configuration and there's no formal support contract in place with the application vendor (if that's not you), try restarting the application and then try rebooting the box. These are big hammers, and may or may not cure the trigger. If the application has tipped over for cause, the reboot may or may not clear the trigger.

If this OpenVMS system has been operating for a while and has not seen recent modifications proximate to the failure, I'd expect you're going to need to get somebody to look at the source code. That may be you, or someone you designate. If there's no source code available, the next step is what's been suggested previously in this thread, followed by full-on reverse engineering (where such activity is permitted).

Debugging application errors without access to the source code is non-trivial.

Stephen Hoffman
HoffmanLabs LLC

[DECxchange]

Hello Joseph,

Is there any more to this error message you could share with us? It seems like this is not the full error message. Or there could be other error messages associated with this. Typically, you should be able to see an error that identifies the process or program name that is experiencing the symbolic stack dump.

In some cases, it will also give the line number of the program that is failing. If you have access to the source code (written in some language like C, Fortran, etc.), you can look through that and see if you can find the bug. You may have to investigate more than one program, since I'm assuming you are dealing with mailbox related system services (sys$crembx, sys$qio(w), etc.). The VMS SEARCH command is very useful in tracking down places in the application source where problems may exist.