socket peek buffer sizes

[Elli M Barasch]

When I issue a socket peek read (TCPIP$C_MSG_PEEK) via the QIOW interface, I notice that QIOW always returns with a maximum byte count of 1024. Is this adjustable?

TIA.

[Richard Whalen]

Can we see a code fragment so we know a bit more about the environment (TCP/UDP) and how the read is done?

[Elli M Barasch]

int peek_count = MIN(howmuch,50000);

#if 0
printf("peeking %d bytes | ",peek_count);
status = sys$qiow(0,conn_channel,IO$_READVBLK,&iosb,0,0,cursor, peek_count,0,TCPIP$C_MSG_PEEK,0,0);
EXIT_IF_BAD(status,iosb);
if (iosb.iosb$w_bcnt == 0)
break;
int count = iosb.iosb$w_bcnt;
printf("peeked %d and now reading\n",iosb.iosb$w_bcnt);
status = sys$qiow(0,conn_channel, IO$_READVBLK, &iosb,0,0, cursor,count,0, 0,0,0 );
EXIT_IF_BAD(status,iosb);
if (count != iosb.iosb$w_bcnt)
printf("count mismatch, expd/got = %d/%d\n",count,iosb.iosb$w_bcnt);
#else
printf("reading %d |",peek_count);
status = sys$qiow(0,conn_channel, IO$_READVBLK, &iosb,0,0, cursor,peek_count,0, 0,0,0 );
EXIT_IF_BAD(status,iosb);
printf("got %d\n",iosb.iosb$w_bcnt);
#endif

[GuentherF]

How much is 'howmuch'?

/Guenther

[John Gillings]

Elli,

It may help to see how you created the socket. Please include the values of any constants. If you can work out the device name, the output of SHOW DEVICE/FULL BGnnnn might also be interesting.

[Elli M Barasch]

howmuch varies from 64000 down to 1. The code never issues a read QIO of more than 50000 bytes (as per the MIN statement). howmuch is decremented by the iosb.iosb$w_bcnt on each iteration of the loop in which this code fragment executes.

The loop terminates when a certain string is found in the stream, or when howmuch = 0.

The code was written conditionally in two ways... One with a peek, one w/o a peek. The results are identical: the QIO completion size always maxes out at 1024 bytes.

Additionally, there is an 'interactive' part of this program that issues a "command" to the partner, prompting the partner to send a 5 byte response. Even though I issue a 64 byte read, I always get a 5 byte I/O completion. How does this work? Is there an inter-byte timeout that causes the driver to complete the I/O? IOW, why doesn't my 64 byte read block?

If I am correct in this "inter-byte timeout" assumption, is it possible that my network partner is pausing after it transmits every 1024 bytes? The delay might be long enough for the driver to then complete the I/O and force me to issue additional reads. I'd like to be able to increase this "inter-byte timeout" so that I can reduce the number of QIOs required.

[Elli M Barasch]

Kind of textbook socket set-up here...

conn_sockchar.prot = TCPIP$C_TCP;
conn_sockchar.type = TCPIP$C_STREAM;
conn_sockchar.af = TCPIP$C_AF_INET;

struct timeval tmo={15,0}; // 2 second timeout on any I/O

tmo_itemlist.length = sizeof(tmo);
tmo_itemlist.type = TCPIP$C_RCVTIMEO;
tmo_itemlist.address = &tmo;

sockopt_itemlist.length = sizeof(tmo_itemlist);
sockopt_itemlist.type = TCPIP$C_SOCKOPT;
sockopt_itemlist.address = &tmo_itemlist;

// whom are we connecting to?
CLEAR(serv_addr);
serv_addr.sin_family = TCPIP$C_AF_INET;
serv_addr.sin_port = htons(atoi(argv[2])); // second arg is port number

// set up socket type
status = sys$qiow(0, conn_channel, IO$_SETMODE, &iosb, 0, 0, &conn_sockchar, 0, 0, 0, 0, 0 );
EXIT_IF_BAD(status,iosb);

// set up socket timeout
status = sys$qiow(0, conn_channel, IO$_SETMODE, &iosb, 0, 0, 0, 0, 0, 0, &sockopt_itemlist, 0 );
EXIT_IF_BAD(status,iosb);

// connect to remote port
status = sys$qiow(0, conn_channel, IO$_ACCESS, &iosb, 0, 0, 0, 0, &serv_itemlist, 0, 0, 0 );
EXIT_IF_BAD(status,iosb);

// Send the prompt - a CR
status = sys$qiow(0,conn_channel, IO$_WRITEVBLK, &iosb, 0, 0, "\r", 1, 0, 0, 0, 0);
EXIT_IF_BAD(status,iosb);

[Elli M Barasch]

The comment above on the TMO set up is wrong. No matter what the time value is, the behavior is the same.

[GuentherF]

"Even though I issue a 64 byte read, I always get a 5 byte I/O completion"

You actually specified that you have a read buffer of 64 bytes. If there are only 5 bytes in the buffer then that's is what you get.

Typically with TCP/IP - a STREAM protocol - you have to read in a loop and extract your data. There is no gurantee on the amount of data you may get with a read.

To change the default internal read buffer size do a IO$_SETMODE with TCPIP$C_RCVBUF.

But no matter how large you make all the buffers (program's private or system buffer) the data stream typically is "hacked" into smaller pieces.

Btw. the timeout only works if NO data has been received so far. Once a byte or more is in the system buffer the QIO calls returns when either there is no more data in the system buffer or your program buffer has been filled.

/Guenther

[Elli M Barasch]

Guenther, I don't think you understood my question.

Issuing a QIO on a stream isn't the same as issuing a QIO on a disk device. There's no way to know if there's more in the stream coming, so the driver has to make a decision when to complete the I/O.

The streams I am reading are well over 20,000 bytes long. If I issue a 50,000 byte QIOW read, I'd expect to block until all 50,000 bytes arrive. This is not the case. Similarly, the 64 byte read terminates after only 5 bytes have arrived. The driver decides to complete the I/O well before the requested number of bytes arrive. So what criteria does the driver use to complete the I/O? The driver must use some sort of "no-activity" timer, or perhaps it uses the size of a single transport that was received off the wire. If there is a timer, I am asking if there is any way to tune it.

Thanks,
Elli

[Hoff]

>The streams I am reading are well over 20,000 bytes long. If I issue a 50,000 byte QIOW read, I'd expect to block until all 50,000 bytes arrive. This is not the case. Similarly, the 64 byte read terminates after only 5 bytes have arrived. The driver decides to complete the I/O well before the requested number of bytes arrive. So what criteria does the driver use to complete the I/O? The driver must use some sort of "no-activity" timer, or perhaps it uses the size of a single transport that was received off the wire. If there is a timer, I am asking if there is any way to tune it.


Ah, this old chestnut.

As you quite correctly state, TCP provides a stream of data.

And you're presuming to treat that stream as a record device.

It's not, unfortunately.

UDP does give you something closer to the record-oriented model. (Depending on exactly what is going on, UDP multicasts or even raw Ethernet datagrams can be a pretty handy solution to some classes of application problems, but I digress.)

TCP is free to give you 50,000 separate I/Os of 1 byte each or the full 50,000 bytes in one shot. Or 49,999 hits of 1 byte each, and then 50,000 bytes containing the last byte and most of the next transfer. Or any combination between that.

With TCP-based application communications, you get to do the segmentation and window processing in your code. Attempts to use timers to segment TCP traffic into records will tend to combine with the intervening IP routers and switches and other application traffic to conspire to find edge cases in socket code, too.

I'd suggest looking for middleware. Socket-level programming is something like programming assembler. It's possible, functional, feasible and such, but it's usually easier to punt on that and to use available networking libraries and available middleware packages. Rolling your own is something that involves, well, dealing with TCP streams and buffers and such. And you probably have application and customer code to write, rather than all of the glue code involved with socket programming.

If you can't migrate to a middleware interface into the IP network (and there are certainly reasons why VMS application programmers might not find that feasible) then you'll have to deal with the sliding window yourself. It's common to see a 2x buffer (or more) for the reads, and to aim the I/Os at the next available byte in the buffer. Basically assembling the incoming data into the record structures.

Double-buffering a TCP stream gets a little more ugly (and can involve rather more buffer copies than might be pleasant to perform on a busy server), as you don't really know how much you're going to get in response to each read.

[Elli M Barasch]

I get it, then. I'm seeing the actual transport as it arrives off the wire. My network partner must be segmenting into 1024 byte frames -- and in the case of the 'short' packet - 5 bytes. So I get what I get.

The driver does not wait; it just delivers as soon as something shows up, unless I provide the IO$M_LOCKBUF qualifier, in which case the I/O won't complete until I receive all the bytes I ask for or my partner closes the connection. Unfortunately, that's not an option for me.

Thanks for the help. I just wanted an explanation. My code works fine; I just thought I could tweak things a bit.

[GuentherF]

With TCP/IP you typically include your own record header which indicates what is a record in your case. In the simplest form you just send a length value for the number of bytes to follow. Mostly you end up with a more sophisticated wrapper around your data to correctly identify a record and resynch to the true start of the next record.

Good ole DECnet made life much easier! A true record oriented protocol.

/Guenther

[GuentherF]

Oh, forgot to mention. The amount of data you receive with one read has absolutely no relation to the size of data send in one call. So the 1024 bytes comes from the fact that the BGdrivers system buffer by default is 1024 bytes. If a nice burst of data is coming in that is what you get.

/Guenther

[Elli M Barasch]

Guenther, Um. See my original question... How do I tweak that? :^)

[GuentherF]

As I mentioned before:

To change the default internal read buffer size do a IO$_SETMODE with TCPIP$C_RCVBUF.

I haven't played much with that but I doubt you'll see a significant performance gain with larger RCVBUF values.

/Guenther

[Elli M Barasch]

Sorry for the misunderstanding.

[Elli M Barasch]

Guenther, turns out you're 100% correct. I raised the value to 4096 and saw no change in behavior. But when I lowered the value to 512 I saw the I/O's complete at 512 bytes each. This leads me to conclude that my partner is sending 1024 byte packets, and that's the best I'm going to do.

Thanks again.

[GuentherF]

"Guenther, turns out you're 100%"

I never am. ;-)

/Guenther