Operating System - OpenVMS
1748185 Members
4165 Online
108759 Solutions
New Discussion юеВ

Re: Shared interlocked queues

 
Brian Reiter
Valued Contributor

Re: Shared interlocked queues

Hi Folks,

Well after Mondays plumbing fiasco, I can now try and answer your questions as best I can (although the forum software makes it ticky to see the response without resorting to notepad).

To be homest I was just curious as to how it could be done (and see if there're any examples about), the LIB$ and Programming Concepts manuals mention that it could be done but were a tad hazy about the specifics.

The situation I have is that under extreme loads the rate of messages coming in to the system can swamp the processes. All interprocess communication is done via mailboxes and these mailboxes filling up caused delays across the board and the eventual loss of messages. This siutation is meant to be rare but it is also transient and may only last a few minutes. The system will cope until the last minute or so.

The plan is to use ASTs and the queue routines to keep the mailboxes empty and feed the main process from the new queue. This (as long as memory allows) should allow the system to get over the processing hump without losing any data.

Many thanks for you help - interesting as always


Brian
Hein van den Heuvel
Honored Contributor

Re: Shared interlocked queues

Hmmm,

But in a sense mailboxes ARE an in memory shared queue managed by interlocked queue instructions.

So if you plan to keep using the mailbox as base communication and then add a layer to it, things may just get worse!

Maybe you can just create the troublesome mailboxes with much more quota mailbox and/or check stop waiting for the message to be consumed?

Did you check Bruce Ellis's writeup?
http://h71000.www7.hp.com/openvms/journal/v9/mailboxes.pdf

Now the QIO mechanism is not cheap (performance).

So if you were to replace the QIO with an alternative method then you may well come out ahead, but it may be a significant investment to get there.

Proof for the price of a QIO?
Check out the NULL device. Cheap right? No!
The 5MB example file below takes 10x longer to copy to NL: then to a disk file.
(1.3 Ghz RX2600, chached file, timings in deci-seconds)

Cheers,
Hein


$ creat/fdl="file; allo 10000; reco; size 10; form fix" fix.tmp
% set file/end fix.tmp
$ @time copy fix.tmp nl:
Dirio=512082 Bufio= 11 Kernel= 549 RMS= 297 DCL=0 User= 37 Elapsed= 941
$ @time copy fix.tmp tmp.tmp
Dirio= 167 Bufio= 16 Kernel= 2 RMS= 0 DCL=0 User= 0 Elapsed= 87
$ @time copy fix.tmp nl:
Dirio=512082 Bufio= 11 Kernel= 534 RMS= 315 DCL=0 User= 27 Elapsed= 938

Robert Gezelter
Honored Contributor

Re: Shared interlocked queues

Hein,

With all due respect, the timing comparison between NL and a disk is flawed.

NL is a record oriented device. The disk copy is done in multiblock mode. This is comparing apples and oranges.

What was the format of the data in the file?

- Bob Gezelter, http://www.rlgsc.com
Hein van den Heuvel
Honored Contributor

Re: Shared interlocked queues

>> the timing comparison between NL and a disk is flawed.

I was not comparing them.
I was just showing the price of a QIO (to the NL: device).
About 1 ms of kernel time per QIO !
A mailbox QIO takes fraction longer. See below.

>> NL is a record oriented device. The disk copy is done in multiblock mode. This is comparing apples and oranges.

Both fruits. Both copy all the data.
Some folks still believe using NL: speeds thinkgs up.
It might not... indeed because it is a record device.

>> What was the format of the data in the file?

The whole test was shown.
File as per FDL in the test: 512,000 records of 10 bytes.

Silly mailbox test below.
It just shows that adding a post-processor to a mailbox based ommunication method is not likely to address fundamental issues.

fwiw,

Hein


$ cre/mail my_mbx
$ spaw/nowait/proc=hein_mbx @time copy my_mbx: nl:
%DCL-S-SPAWNED, process HEIN_MBX spawned
$ @time copy fix.tmp my_mbx
Dirio= 82 Bufio=512014 Kernel= 634 RMS= 269 DCL=0 User= 46 Elapsed= 1908
$
Dirio=512001 Bufio=512011 Kernel=1142 RMS= 450 DCL=0 User= 35 Elapsed=n.a.


Hein van den Heuvel
Honored Contributor

Re: Shared interlocked queues

Ooops, too quickly.
Got my math wrong.
It didn't feel right, but needed to get back to work.

Dirio=512082 ... Kernel= 534 Deci-second = 5340 ms.

So that is about 100 QIOs per ms.
Or 10 micro-second per QIO.
Much better!
Sorry.

Hein


John Gillings
Honored Contributor

Re: Shared interlocked queues

Brian,

As Hein has pointed out, a mailbox is pretty much what you're talking about implementing. Indeed, if you can find the sources for the mailbox driver, I'd expect you'll find some excellent examples of how to use the INSQHI and REMQHI instructions ;-)

If you're having trouble dealing with spikes in load, make sure your mailboxes have plenty of headroom. See "bufquo" parameter for $CREMBX. This used to be limited to absurdly low values (64K?), but since circa V7.3 it's now 32 bit, limited only by process BYTLM and system NPAGEDYN.

If your $CREMBX doesn't specify a buffer quota, it inherits DEFMBXBUFQUO which defaults to 1K (yes, *K*). If you were going to just shovel the messages out of a mailbox and into your own mailbox, you may as well make room for them in the system mailbox and save yourself the work. The only caveat is the system allocates mailboxes from NPAGEDYN, so the resource isn't quite as cheap as pageable virtual memory.

That said. If you have a chain of processes that pass messages through mailboxes, as you've no doubt discovered, you don't want them synchronising with RWMBX (very expensive!).

The most obvious process design:

loop
$QIOW mailbox READVBLK into buffer
process buffer
endloop

can be a problem if processing potentially exceeds message interarrival time. You can move the spikes from the mailbox into local process virtual memory using a work queue design. It then becomes two threads, like this:


MailboxRead
$QIO mailbox READVBLK into buffer AST MailboxAST
End MailboxRead

MailboxAST
put buffer onto work queue
MailboxRead
$WAKE
End MailboxAST

MAIN
MailboxRead
Loop
$HIBER
$SETAST 0 ! block ASTs
remove buffer from work queue
$SETAST restore
If gotbuffer THEN process buffer
Endloop

Note there's no need for the work queue to be in shared memory and you don't need to use INSQHI/REMQHI, as you're using AST blocks to synchronise the threads. The AST thread can interrupt processing to add more buffers to the work queue.
A crucible of informative mistakes
John McL
Trusted Contributor

Re: Shared interlocked queues

I think there may have been an example of this in some VMS documentation many years ago. The example might have been about an airline reservation system. I'm thinking the documentation could have been around 1985-88 timeframe because I used the concept at a site that I worked at in late 1989.

The principles of that code are as follows:

Start by creating a mailbox, grabbing a bunch of buffers and putting them on the "free" list, then set a QIO read on the mailbox using one of the free buffers and with an AST routine.

The QIO AST routine puts the mailbox buffer onto an "active" list then sets the new QIO AST (using one of the "free" buffers) before exiting.

The main code takes the next buffer off the active list, processes it and then puts the used buffer onto the free list.

You'll see some similarity between the initial code and the AST code but because the AST code operates after that initial code (i.e. there's no chance of simultaneous access) there's no reason why the same routine - expand freelist if required, take buffer from freelist, set-up QIO with AST - can't be used for both.

I used LIB$INSQTI (NB. "tail") to handle putting buffers onto the two lists (and LIB$REMQHI to take them off) because I didn't want the main code halfway through putting a buffer onto the free list when the AST routine jumped in and wanted to take a buffer off that list or the other way around with the active list. (As John G says, the other way to do this is to have the main code disable ASTs while taking buffers off lists or putting them on.)
GuentherF
Trusted Contributor

Re: Shared interlocked queues

"...extreme loads...mailboxes filling up..."

Means your configuration has not enough umps to handle the burst. Find out which part of your configuration is the (current) bottleneck. CPU? I/O? Artifiial waits?

"...loss of messages."
Not caused by the OS's mailbox facility. Your application has a logic error to cause this.

/Guenther
Hoff
Honored Contributor

Re: Shared interlocked queues

This reeks of an underpowered and/or overloaded system, and a case where latent application bugs are revealed by the load.

Use DECset PCA to profile the application code, looking for wall-clock and processor usage. (I'd probably also look at bigger hunks of data; processing individual records from a mailbox or from a file is a slow technique.)

Look for synchronization and coding bugs. Omitting IOSBs or mishandling IOSBs and omitting return and IOSB status checks are a common trigger of these cases.

http://h71000.www7.hp.com/wizard/wiz_1661.html

Mechanisms that provide guaranteed message delivery can generate application-wide wedgies, too :-) and this when the slowest part of the application configuration is overrun. Your job: find the message loss (that's likely a bug), and find the slowest part of the application.
Richard J Maher
Trusted Contributor

Re: Shared interlocked queues

Hi Brian,

For those cases (and there are many) when the consumer just cannot keep up with the producer I/we have opted for the lightweight write-to-disk consumer that does nothing more than record/persist the phone-call/transaction/trade and maintain the last-available number in a lock-value-block.

The ultimate down-stream consumer can then read sequentially through this work-queue that can cater for the highest peaks and deepest troughs.

Mailboxes are very limited! Interactive users are fine but PABX or Switch traffic is too much. Horses for courses. Just make sure the event (trade, txn, call) is persistent (unlike the ASX :-) and everything else is lazyable.

Cheers Richard Maher