Operating System - OpenVMS
1748058 Members
5113 Online
108758 Solutions
New Discussion юеВ

Re: socket peek buffer sizes

 
Hoff
Honored Contributor

Re: socket peek buffer sizes

>The streams I am reading are well over 20,000 bytes long. If I issue a 50,000 byte QIOW read, I'd expect to block until all 50,000 bytes arrive. This is not the case. Similarly, the 64 byte read terminates after only 5 bytes have arrived. The driver decides to complete the I/O well before the requested number of bytes arrive. So what criteria does the driver use to complete the I/O? The driver must use some sort of "no-activity" timer, or perhaps it uses the size of a single transport that was received off the wire. If there is a timer, I am asking if there is any way to tune it.


Ah, this old chestnut.

As you quite correctly state, TCP provides a stream of data.

And you're presuming to treat that stream as a record device.

It's not, unfortunately.

UDP does give you something closer to the record-oriented model. (Depending on exactly what is going on, UDP multicasts or even raw Ethernet datagrams can be a pretty handy solution to some classes of application problems, but I digress.)

TCP is free to give you 50,000 separate I/Os of 1 byte each or the full 50,000 bytes in one shot. Or 49,999 hits of 1 byte each, and then 50,000 bytes containing the last byte and most of the next transfer. Or any combination between that.

With TCP-based application communications, you get to do the segmentation and window processing in your code. Attempts to use timers to segment TCP traffic into records will tend to combine with the intervening IP routers and switches and other application traffic to conspire to find edge cases in socket code, too.

I'd suggest looking for middleware. Socket-level programming is something like programming assembler. It's possible, functional, feasible and such, but it's usually easier to punt on that and to use available networking libraries and available middleware packages. Rolling your own is something that involves, well, dealing with TCP streams and buffers and such. And you probably have application and customer code to write, rather than all of the glue code involved with socket programming.

If you can't migrate to a middleware interface into the IP network (and there are certainly reasons why VMS application programmers might not find that feasible) then you'll have to deal with the sliding window yourself. It's common to see a 2x buffer (or more) for the reads, and to aim the I/Os at the next available byte in the buffer. Basically assembling the incoming data into the record structures.

Double-buffering a TCP stream gets a little more ugly (and can involve rather more buffer copies than might be pleasant to perform on a busy server), as you don't really know how much you're going to get in response to each read.
Elli M Barasch
Advisor

Re: socket peek buffer sizes

I get it, then. I'm seeing the actual transport as it arrives off the wire. My network partner must be segmenting into 1024 byte frames -- and in the case of the 'short' packet - 5 bytes. So I get what I get.

The driver does not wait; it just delivers as soon as something shows up, unless I provide the IO$M_LOCKBUF qualifier, in which case the I/O won't complete until I receive all the bytes I ask for or my partner closes the connection. Unfortunately, that's not an option for me.

Thanks for the help. I just wanted an explanation. My code works fine; I just thought I could tweak things a bit.
GuentherF
Trusted Contributor

Re: socket peek buffer sizes

With TCP/IP you typically include your own record header which indicates what is a record in your case. In the simplest form you just send a length value for the number of bytes to follow. Mostly you end up with a more sophisticated wrapper around your data to correctly identify a record and resynch to the true start of the next record.

Good ole DECnet made life much easier! A true record oriented protocol.

/Guenther
GuentherF
Trusted Contributor

Re: socket peek buffer sizes

Oh, forgot to mention. The amount of data you receive with one read has absolutely no relation to the size of data send in one call. So the 1024 bytes comes from the fact that the BGdrivers system buffer by default is 1024 bytes. If a nice burst of data is coming in that is what you get.

/Guenther
Elli M Barasch
Advisor

Re: socket peek buffer sizes

Guenther, Um. See my original question... How do I tweak that? :^)
GuentherF
Trusted Contributor

Re: socket peek buffer sizes

As I mentioned before:

To change the default internal read buffer size do a IO$_SETMODE with TCPIP$C_RCVBUF.

I haven't played much with that but I doubt you'll see a significant performance gain with larger RCVBUF values.

/Guenther
Elli M Barasch
Advisor

Re: socket peek buffer sizes

Sorry for the misunderstanding.
Elli M Barasch
Advisor

Re: socket peek buffer sizes

Guenther, turns out you're 100% correct. I raised the value to 4096 and saw no change in behavior. But when I lowered the value to 512 I saw the I/O's complete at 512 bytes each. This leads me to conclude that my partner is sending 1024 byte packets, and that's the best I'm going to do.

Thanks again.
GuentherF
Trusted Contributor

Re: socket peek buffer sizes

"Guenther, turns out you're 100%"

I never am. ;-)

/Guenther