TCPIP buffer problem?

[Willem Grooters]

I have a small 'problem' in TCPIP-programming:

A program on one machine sends out a request to another program using TCPIP and waits for a reply. That second program is started , proecesses the request and replies, using the same port, in one message of (estimated) 40K. After 32K has been read, the first program finds no more bytes available (number of bytes read seems to be 0).
Both programs use socket programming.

I checked the code of the requesting program, and apart from the first recv, the socket is set to 'blocking' to the program will wait for an answer. When a first recv is done, the socket is set to non-blocking, so no data would indeed mean 'Done'.
But it's not - there should be about 8K still to be retrieved...
The replying program sets the socket to 'non-blocking', before reading is (which is no problem since a message will exist). When data is sent, the program closes the connection and exits.

What could be done - preferably in the second program or in TCPIP parameters - to get the data over 32K insize the program?

[Antoniov.]

Willem,
yes receive buffer is 32K.
You can change TCP system parameters with sysconfig.
Read this page for datils
http://h71000.www7.hp.com/doc/732final/6631/6631pro_004.html#subsys_sec

Antonio Vigliotti

[Willem Grooters]

Antonio,

yes receive buffer is 32K.


I thought so. But this is a TCP connection, so I would expect (as is the specification) that if a message is bigger, the system will ASSURE complete and error free transmission.
But for some reaon, everyting in excess of 32K is lost


You can change TCP system parameters with sysconfig.


I know (thouigh thanks for the link, makes it easy to look up) but that would apply to ALL connections, wouldn't it? Since there are quite a lot of them, and for most (even ths one) even 32K is far to much, I would prefer to use another way (programmed, eventually) for those cases where a larger buffer is explicitly or eventually required.

Willem

[Brian Reiter]

Willem,

Is it possible that the latter part of the data is being lost as a side-effect of the socket being closed? You could check any status from the recv, or use tcptrace to monitor the netowork traffic and ensure data is being sent (or not). A simplistic solution might be to modifiy the second program to add a small delay before closing the connection (assuming of course closing the socket is causing a problem).

cheers

Brian

[Willem Grooters]

Antonio,

I checked the link you specified - I only can change the max size of any socketbuffer. I did check this on my (7.3-1) system and it's 100K (although the documentation specified a 128K default, it's that 100K).

Brian,

Is it possible that the latter part of the data is being lost as a side-effect of the socket being closed?

I guess it is, but AFAIK that would be a violation of 'the standard' where TCP will ensure proper, complete and 'in order' receeipt of a message. It fails on the second issue....


You could check any status from the recv,


It does. The program reads data in chunks of of 5000 bytes (or a few less) (why, I didn't figure out yet, I think the sending program is still busiy sending) until 32767 bytes are read - the last accepted chunk is < 3000 bytes. The next read gives me 0 bytes...


A simplistic solution might be to modifiy the second program to add a small delay before closing the connection (assuming of course closing the socket is causing a problem).


Simplistic indeed, since any delay is arbitary and bound to fail some time. I would rather be able to send a message and expect to get a return status from 'send' that tells me everthing has arrived and the socket can safely be closed, or that sending has timed-out.

Willem

[Jan van den Ende]

Willem,


until 32767 bytes are read - the last accepted chunk is < 3000 bytes. The next read gives me 0 bytes...


That number gives me the creepes.
It is exactly 2^15 -1; sounds VERY much like the max value for a signed word!

Is there somewhere a counter of word size in your program? And if NOT, then, TCPIP engeneering: is there somewhere something like that in YOUR code?

jpe

[Willem Grooters]

jan,


That number gives me the creepes.


Me too.
In my program, there ARE counters but these are all 32 bit.
I have thought of a limitation in the program itself: the buffer in the program itself is 20000 bytes, but since I DO read 32767 bytes, I don't think that is the problem either. There could be a problem with the BG-device - 32767 is a known limit in RMS. Does that give some hint? (I'm interested in "how" and "why" of this, but formeost - at the moment, is how to overcome this)

Willem

[Brian Reiter]

Willem

We routinely send/receive data exceeding 32K bytes via the BG devices. Only difference is that we use the QIO interface - giving us a 64K limit. Could this be an issue with socket interface?

cheers

Brian

[Willem Grooters]

Brian,

Rewrite is no option (timescale forbids).

Anyway, I may have found the reason.
The middleware - started as a service - takes over the socket from INETACP, sets buffersize (32768) and places it in a non-blocking mode. I also found send is NOT checked for status (I just learned the program has been ported from Unix - and this seems NOT to have changed...) so I have no idea of any errro found (buffer full or so)

I don't have much experience in TCPIP-programming yet, so if anyone could advise me on the next solution:

I could of course increase the buffer size but any size is arbitrary. I will run into the same issue some day....
So my consideration is as follows:
Since the program is started by INETACP as a service, there WILL be a message to receive, and I'm certain it's just a few 100 bytes, not 32K, nor will processing occur before the whole message is read. So I don'tsee why I would need to set it NONBLOCKING.
send will in that case wait if the buffer is full, until there is room for the rest of the message - no matter how long the reply will be.

Is this a right way of thinking, or do I overlook something?

Willem

[Martin Vorlaender]

I also found send is NOT checked for status (I just learned the program has been ported from Unix - and this seems NOT to have changed...) so I have no idea of any errro found (buffer full or so)


That's the reason! The value returned by send() is the number of bytes written, so if you want to write more than 32k bytes, you need to loop through send calls until all of the buffer has been sent!

cu,
Martin

[Uwe Zessin]

In that case I would _always_ check how many bytes have been sent. And I think you need to have the same logic on the receiving side.

TCP/IP works stream oriented, not 'record oriented'.

[Martin Vorlaender · Accepted solution]

To clarify my previous post: *ix socket programming should *always* run along the following lines:


int rc, sent = 0;
char *bufptr = buffer;

do {
rc = send(socket, bufptr, buffer_len-sent, flags);
if (rc < 0) {
/* error handling */
break;
}
sent += rc;
bufptr += rc;
while (sent < buffer_len);


cu,
Martin

[Antoniov.]

Willem,
may be a type but you wrote "set buffer 32768"; if you are using a shot int, you have set a negative value!

Antonio Vigliotti

[Antoniov.]

Willem,
typing
sysconfig -q inet tcp_recvspace
do see the magic number 32768?
If I understand this is the receive buffer for TCP/IP.

Antonio Vigliotti

[Brian Reiter]

Willem,

Following on from Antonios suggestion. IF you're using TCPIP then try upping the send and receive queues.

SET CONFIGURATION PROTOCOL TCP/QUOTA=(send=xxxx,receive=xxxx)
SET PROTOCOL TCP/QUOTA=(send=xxxx,receive=xxxx

I'm not convinced that this will help with your problem - but its better than the default of 4096 (which still let us deal with large packets). Setting these values to 64100 seemed to give our system better performance.


cheers

Brian

[Willem Grooters]

Antonio,

BPS2004A> tcpip sysconfig -q inet tcp_recvspace
inet:
tcp_recvspace = 61440
BPS2004A>

so that's Ok...(weird number, though)
That's on the system that holds both requestor and replier.

Brian, that answers your suggestion ;-)

I found 32768 as buffersize in the middleware code. As stated, I could increase the value but no doubt I would run into the same problem somewhere in the future. I'm currently waiting for some testdata, and after that I can start adapting the program.
(Required but mising knowledge: When is data transpoted? 'My' programs reads about 5000 bytes at a time, and the middleware writes straems of about 1700 in size. I don't get it that I seem to run om a limit of 32K - unless the read is actually done at socket close or buffer full...)

[Ian Miller.]

Willem - remember TCP is a stream of bytes not record so writer and reader can write and read in different sizes

[Willem Grooters]

Well, I got my data (sending >>32K).
Guess what: I CANNOT REPRODUCE THE PROBLEM on my development machine (that's where the buffersize within TCPIP seems to be big enough). AFAIK my system (7.3-1, TCPIP 5.3) hasn't been tuned.
I'll ask where it goes wrong on the other site. It might be another setting for TCPIP, quite likely an older VMS environment (7.1, TCPIP 5.0?) with different defaults.

Willem

[Willem Grooters]

No luck.
The system that encounters the problem is like my developmentmachine....The only problem is that machine ahs much more users...

[Jan van den Ende]

Willem,

"NO luck".

Meaning you cannot reproduce it, or you keep having it? :-(

Did you do anything that might have somehow changed your devellop system to not reproduce it?
And yes, your machine name prompt betrays that it will not be looked upon frindly if you shut THAT (or even only its IP) down!.

In _what_ way exactly is it like your development system? VMS & TCPIP versions, ECO's?

There HAVE been various issues with IP, each version a new surprise, but THIS one does not ring a bell (yet).


jpe

[Willem Grooters]

NO LUCK = cannot reprocduce. That is: the problem does NOT occur on the development machine. If it were, I could dig into code to find out what caused the problem and solve it. Now I can change the program but leave testing to the customer....
Knowing how things go, I wonder if there have been patches installed - none, on 'my' system:

BPS2004A> prod show hist
----------------------------------- ----------- ----------- --------------------
PRODUCT KIT TYPE OPERATION DATE AND TIME
----------------------------------- ----------- ----------- --------------------
DEC AXPVMS VMS731_BACKUP V1.0 Patch Install 29-DEC-2003 15:15:50
DEC AXPVMS VMS731_BACKUP V1.0 Patch Install 29-DEC-2003 15:15:20
CPQ AXPVMS CSWS13_UPDATE V1.0 Patch Install 05-MAR-2003 14:53:33
CPQ AXPVMS CSWS V1.3 Full LP Install 05-MAR-2003 09:37:24
CPQ AXPVMS CDSA V1.0-2 Full LP Install 03-JAN-2003 11:33:16
DEC AXPVMS DWMOTIF V1.2-6 Full LP Install 03-JAN-2003 11:33:16
DEC AXPVMS OPENVMS V7.3-1 Platform Install 03-JAN-2003 11:33:16
DEC AXPVMS TCPIP V5.3-18 Full LP Install 03-JAN-2003 11:33:16
DEC AXPVMS VMS V7.3-1 Oper System Install 03-JAN-2003 11:33:16
----------------------------------- ----------- ----------- --------------------

9 items found
BPS2004A>

I don't think it's much better on the target system. I think I'll just have to sit and wait for the results.....

Martin, you get your points. I installed your solution and the program still works - on my system anyway.

[Willem Grooters]

Problem solved.

First of all: have status of send checked. It turned out that it's statis (that is: numbers of bytes written) was -1 abd errno EWOULDBLOCK - quite according the manual since the porgram set the socket to be non-blocking.
Well, that isn't needed in the first place. Although the same socket is used for reading the rquest and sending the rply back, it's just read - process - reply. During reply, no next message is to be received (since it's a one=shot TCPIP service and dies after it has done it's job). So I decided to remove this setting, so send would wait if the bufer was full.And now it does - in the end the whole 41K is received.

I just thought it correct to let you know
.
Willem

[Uwe Zessin]

Agreed, I think it is a special reward for those who have tried to help so they can see their effords were useful.

By the way: "I installed your solution and the program still works" - nice compliment ;-)

[Ian Miller.]

and the moral is 'always check status returns and report errors' :-)

I've seen too much code that does not and most of the time works but when something goes wrong its hard to find out why.

[Willem Grooters]

and report errors


Add: "in a meaningfull way". Just the number doesn't tell much. Even within layered products (like TCPIP) there's quite a lot of work to be done.
In this case: What's "35"? You need to get errno.h from the library (just forgot which one) to find out it's (the expected) EWOULDBLOCK, mentioned in the documentation.

(some point for advocacy...)