topic Re: TCPIP buffer problem? in Operating System - OpenVMS

TCPIP buffer problem?

Willem Grooters — Mon, 14 Jun 2004 05:19:06 GMT

I have a small 'problem' in TCPIP-programming:

A program on one machine sends out a request to another program using TCPIP and waits for a reply. That second program is started , proecesses the request and replies, using the same port, in one message of (estimated) 40K. After 32K has been read, the first program finds no more bytes available (number of bytes read seems to be 0).
Both programs use socket programming.

I checked the code of the requesting program, and apart from the first recv, the socket is set to 'blocking' to the program will wait for an answer. When a first recv is done, the socket is set to non-blocking, so no data would indeed mean 'Done'.
But it's not - there should be about 8K still to be retrieved...
The replying program sets the socket to 'non-blocking', before reading is (which is no problem since a message will exist). When data is sent, the program closes the connection and exits.

What could be done - preferably in the second program or in TCPIP parameters - to get the data over 32K insize the program?

Re: TCPIP buffer problem?

Antoniov. — Mon, 14 Jun 2004 05:44:57 GMT

Willem,
yes receive buffer is 32K.
You can change TCP system parameters with sysconfig.
Read this page for datils
http://h71000.www7.hp.com/doc/732final/6631/6631pro_004.html#subsys_sec

Antonio Vigliotti

Re: TCPIP buffer problem?

Willem Grooters — Mon, 14 Jun 2004 05:56:38 GMT

Antonio,

yes receive buffer is 32K.

I thought so. But this is a TCP connection, so I would expect (as is the specification) that if a message is bigger, the system will ASSURE complete and error free transmission.
But for some reaon, everyting in excess of 32K is lost

You can change TCP system parameters with sysconfig.

I know (thouigh thanks for the link, makes it easy to look up) but that would apply to ALL connections, wouldn't it? Since there are quite a lot of them, and for most (even ths one) even 32K is far to much, I would prefer to use another way (programmed, eventually) for those cases where a larger buffer is explicitly or eventually required.

Willem

Re: TCPIP buffer problem?

Brian Reiter — Mon, 14 Jun 2004 06:31:43 GMT

Willem,

Is it possible that the latter part of the data is being lost as a side-effect of the socket being closed? You could check any status from the recv, or use tcptrace to monitor the netowork traffic and ensure data is being sent (or not). A simplistic solution might be to modifiy the second program to add a small delay before closing the connection (assuming of course closing the socket is causing a problem).

cheers

Brian

Re: TCPIP buffer problem?

Willem Grooters — Mon, 14 Jun 2004 07:06:58 GMT

Antonio,

I checked the link you specified - I only can change the max size of any socketbuffer. I did check this on my (7.3-1) system and it's 100K (although the documentation specified a 128K default, it's that 100K).

Brian,

Is it possible that the latter part of the data is being lost as a side-effect of the socket being closed?

I guess it is, but AFAIK that would be a violation of 'the standard' where TCP will ensure proper, complete and 'in order' receeipt of a message. It fails on the second issue....

You could check any status from the recv,

It does. The program reads data in chunks of of 5000 bytes (or a few less) (why, I didn't figure out yet, I think the sending program is still busiy sending) until 32767 bytes are read - the last accepted chunk is < 3000 bytes. The next read gives me 0 bytes...

A simplistic solution might be to modifiy the second program to add a small delay before closing the connection (assuming of course closing the socket is causing a problem).

Simplistic indeed, since any delay is arbitary and bound to fail some time. I would rather be able to send a message and expect to get a return status from 'send' that tells me everthing has arrived and the socket can safely be closed, or that sending has timed-out.

Willem

Re: TCPIP buffer problem?

Jan van den Ende — Mon, 14 Jun 2004 08:45:46 GMT

Willem,

until 32767 bytes are read - the last accepted chunk is < 3000 bytes. The next read gives me 0 bytes...

That number gives me the creepes.
It is exactly 2^15 -1; sounds VERY much like the max value for a signed word!

Is there somewhere a counter of word size in your program? And if NOT, then, TCPIP engeneering: is there somewhere something like that in YOUR code?

jpe

Re: TCPIP buffer problem?

Willem Grooters — Mon, 14 Jun 2004 09:24:52 GMT

jan,

That number gives me the creepes.

Me too.
In my program, there ARE counters but these are all 32 bit.
I have thought of a limitation in the program itself: the buffer in the program itself is 20000 bytes, but since I DO read 32767 bytes, I don't think that is the problem either. There could be a problem with the BG-device - 32767 is a known limit in RMS. Does that give some hint? (I'm interested in "how" and "why" of this, but formeost - at the moment, is how to overcome this)

Willem

Re: TCPIP buffer problem?

Brian Reiter — Tue, 15 Jun 2004 01:55:36 GMT

Willem

We routinely send/receive data exceeding 32K bytes via the BG devices. Only difference is that we use the QIO interface - giving us a 64K limit. Could this be an issue with socket interface?

cheers

Brian

Re: TCPIP buffer problem?

Willem Grooters — Tue, 15 Jun 2004 04:00:40 GMT

Brian,

Rewrite is no option (timescale forbids).

Anyway, I may have found the reason.
The middleware - started as a service - takes over the socket from INETACP, sets buffersize (32768) and places it in a non-blocking mode. I also found send is NOT checked for status (I just learned the program has been ported from Unix - and this seems NOT to have changed...) so I have no idea of any errro found (buffer full or so)

I don't have much experience in TCPIP-programming yet, so if anyone could advise me on the next solution:

I could of course increase the buffer size but any size is arbitrary. I will run into the same issue some day....
So my consideration is as follows:
Since the program is started by INETACP as a service, there WILL be a message to receive, and I'm certain it's just a few 100 bytes, not 32K, nor will processing occur before the whole message is read. So I don'tsee why I would need to set it NONBLOCKING.
send will in that case wait if the buffer is full, until there is room for the rest of the message - no matter how long the reply will be.

Is this a right way of thinking, or do I overlook something?

Willem

Re: TCPIP buffer problem?

Martin Vorlaender — Tue, 15 Jun 2004 04:20:19 GMT

I also found send is NOT checked for status (I just learned the program has been ported from Unix - and this seems NOT to have changed...) so I have no idea of any errro found (buffer full or so)

That's the reason! The value returned by send() is the number of bytes written, so if you want to write more than 32k bytes, you need to loop through send calls until all of the buffer has been sent!

cu,
Martin

Re: TCPIP buffer problem?

Uwe Zessin — Tue, 15 Jun 2004 04:46:09 GMT

In that case I would _always_ check how many bytes have been sent. And I think you need to have the same logic on the receiving side.

TCP/IP works stream oriented, not 'record oriented'.

Re: TCPIP buffer problem?

Martin Vorlaender — Tue, 15 Jun 2004 05:40:39 GMT

To clarify my previous post: *ix socket programming should *always* run along the following lines:

int rc, sent = 0;
char *bufptr = buffer;

do {
rc = send(socket, bufptr, buffer_len-sent, flags);
if (rc < 0) {
/* error handling */
break;
}
sent += rc;
bufptr += rc;
while (sent < buffer_len);

cu,
Martin

Re: TCPIP buffer problem?

Antoniov. — Tue, 15 Jun 2004 07:12:36 GMT

Willem,
may be a type but you wrote "set buffer 32768"; if you are using a shot int, you have set a negative value!

Antonio Vigliotti

Re: TCPIP buffer problem?

Antoniov. — Tue, 15 Jun 2004 07:50:28 GMT

Willem,
typing
sysconfig -q inet tcp_recvspace
do see the magic number 32768?
If I understand this is the receive buffer for TCP/IP.

Antonio Vigliotti

Re: TCPIP buffer problem?

Brian Reiter — Tue, 15 Jun 2004 09:03:26 GMT

Willem,

Following on from Antonios suggestion. IF you're using TCPIP then try upping the send and receive queues.

SET CONFIGURATION PROTOCOL TCP/QUOTA=(send=xxxx,receive=xxxx)
SET PROTOCOL TCP/QUOTA=(send=xxxx,receive=xxxx

I'm not convinced that this will help with your problem - but its better than the default of 4096 (which still let us deal with large packets). Setting these values to 64100 seemed to give our system better performance.

cheers

Brian

Re: TCPIP buffer problem?

Willem Grooters — Tue, 15 Jun 2004 09:27:01 GMT

Antonio,

BPS2004A> tcpip sysconfig -q inet tcp_recvspace
inet:
tcp_recvspace = 61440
BPS2004A>

so that's Ok...(weird number, though)
That's on the system that holds both requestor and replier.

Brian, that answers your suggestion ;-)

I found 32768 as buffersize in the middleware code. As stated, I could increase the value but no doubt I would run into the same problem somewhere in the future. I'm currently waiting for some testdata, and after that I can start adapting the program.
(Required but mising knowledge: When is data transpoted? 'My' programs reads about 5000 bytes at a time, and the middleware writes straems of about 1700 in size. I don't get it that I seem to run om a limit of 32K - unless the read is actually done at socket close or buffer full...)

Re: TCPIP buffer problem?

Ian Miller. — Tue, 15 Jun 2004 09:51:02 GMT

Willem - remember TCP is a stream of bytes not record so writer and reader can write and read in different sizes

Re: TCPIP buffer problem?

Willem Grooters — Tue, 15 Jun 2004 10:13:27 GMT

Well, I got my data (sending >>32K).
Guess what: I CANNOT REPRODUCE THE PROBLEM on my development machine (that's where the buffersize within TCPIP seems to be big enough). AFAIK my system (7.3-1, TCPIP 5.3) hasn't been tuned.
I'll ask where it goes wrong on the other site. It might be another setting for TCPIP, quite likely an older VMS environment (7.1, TCPIP 5.0?) with different defaults.

Willem

Re: TCPIP buffer problem?

Willem Grooters — Wed, 16 Jun 2004 01:39:25 GMT

No luck.
The system that encounters the problem is like my developmentmachine....The only problem is that machine ahs much more users...

Re: TCPIP buffer problem?

Jan van den Ende — Wed, 16 Jun 2004 04:08:09 GMT

Willem,

"NO luck".

Meaning you cannot reproduce it, or you keep having it? :-(

Did you do anything that might have somehow changed your devellop system to not reproduce it?
And yes, your machine name prompt betrays that it will not be looked upon frindly if you shut THAT (or even only its IP) down!.

In _what_ way exactly is it like your development system? VMS & TCPIP versions, ECO's?

There HAVE been various issues with IP, each version a new surprise, but THIS one does not ring a bell (yet).

jpe