Incomplete buffer received through RECVFROM(..)

[Enrico Venturi]

Hello colleagues,
this is a strange behaviour occurring to me:
I have two process, one PERL and one executable, on the opposite ends of a socket conneection; I perform a syswrite () on the socket - PERL side, and I write 543 bytes; the result of the syswrite says "543 bytes have been sent to the socket"; on the exectutable receive side I perform a RECVFROM () and I specify "blocking call, 543 bytes to be received"; actually the recvfrom () exits but only 536 bytes have been received from the socket!! I cannot display the errno because the process doesn't catch it.
If the message is 535 bytes long then no problems occur!
Now, my question is:
is there any known socket parameter, network related, which manage the sent buffer size or something similar? ... is there any parameter manageable for instance trough ndd which set the maximum packet sent over the socket?
Do you have any idea to force the recvfrom () to actually receive the whole packet??

Thanks a lot
Enrico

[harry d brown jr]

Strip both the perl program and the C program down to their absolute minimum's and post the code here.

live free or die
harry d brown jr

[Enrico Venturi]

It's difficult to do because they are part of a more complex program ......

[harry d brown jr]

Are both processes on the same machine?

If not, are both machines Hp-ux servers?

To resolve a problem like this, one MUST break down the issue into manageable pieces, otherwise one will spend too much time on searching for ghosts.

live free or die
harry d brown jr

[Enrico Venturi]

see the draw.....

it seems like the IP network cut the packet ...
maybe some routing device's limiting the in-transit PDU's ....

There are no more pieces: PERL executes a syswrite (); compiled C executes a recvfrom().

It's impossible for me to isolate the code from the whole application, but the involved subroutines are only two.

regards
Enrico

[Enrico Venturi]

I forgot an important detail:
the application(s) works fine since a long time, over several HP UX servers, so I guess the reasons of the occurring trouble derive from the environment... maybe some network parameter has been tuned, changed, on the servers or on the routing IP nodes.
The problem is: what's the parameters I have to investigate??

[Laurent Menase]

Hi Enrico,

I think the first thing to do to use tusc/truss on your program.

Then you will see the exact sendto/receivefrom parameters.

you can also take a network trace on the 2 systems to look at what the frame looks like.
for that use 'tcpdump' or nettl.
with nettl to
nettl -tn all -e NS_LS_UDP -f /tmp/mytrace
once you have reproduced:
nettl -tf -e all
then format using a filter, or look at the
trace /tmp/mytrace.TRC00? ? is 0 or 1 with netfmt or with ethereal - it is the easiest way to look at a trace.
To continue, then take the trace at board level. - in 11.11 with a 100bt lan interface it is BTLAN in place of NS_LS_UDP.
or nettl -tn all -e all -f /tmp/mytrace.

[harry d brown jr]

Enrico,

WHats does

netstat -rn

return? Are any of the MTU sizes under 1500 ?

When I said "Strip both the perl program and the C program down to their absolute minimum's"

I meant you need to create a SIMPLE perl program that writes data to a socket and a C program that reads from that socket. Having THESE TWO programs you can perfrom other tests quite easily.

live free or die
harry d brown jr

[Jim Keeble]

If you are using udp (SOCK_DGRAM) sockets, then something is broken. If UDP, the recvfrom() should not return until the complete datagram has arrived.

But, this sounds like you are using SOCK_STREAM or tcp sockets. (See the socket() call that creates the socket). With tcp, the data stream is treated as a continuos stream of bytes, with nothing to "mark" the beginning and end of a message except the programmer's knowledge of what the data is.

The 536 number is the smallest size IP datagram "fragment" that routers are required to pass through without further fragmentation, so I suspect one of the systems or some IP router in between is fragmenting the 543 bytes into 2 fragments, 536 and the remainding 7.

You may be able to correct the issue by increasing the minimum size of the fragments in the path, but you may not even have access to the intermediate devices, for example, if your nodes are at opposite ends of the Internet.

The solution is to check the return value of the first recvfrom, and if only 536 bytes of data are returned, go back and recvfrom again. The two programs have to know where the end of the message is either by mutual agreement or by coding something in the datastream that the other side can decode.

From the recv/recfrom man page:

For message-based sockets such as SOCK_DGRAM, the entire message must be read in a single operation. If a message is too long to fit in the supplied buffer, the excess bytes are discarded. For stream-based sockets such as SOCK_STREAM, there is no concept of message boundaries. In this case, data is returned to the user as soon as it becomes available, and no data is discarded.

See /usr/lib/demos/networking/socket/serv.tcp.c for an example using 10 byte messages as.

[Enrico Venturi]

Hello Jim,
is this true even if we configure the recvfrom as "blocking until all the required bytes have been read"?

[Enrico Venturi]

Hello Jim,
the recvfrom() exits having received only 536 bytes over 739; the errno is set to 239 (ECONNECTIONREFUSED)

[Jim Keeble · Accepted solution]

Which is it, SOCK_DGRAM or SOCK_STREAM ?

It this the first data sent/received by the apps, or is the error returned sometime later ?

How do you set the recvfrom to block untill all the data arrives ? The MSG_WAITALL flag ?

Thanks.

[Enrico Venturi]

Jim
It's a SOCK_STREAM;
the flag MSG_WAITALL doesn't work because it's a HP UX socket, not an XOPEN one.

In anycase we now perform a loop, reading from the socket until the whole message has been received; in other words we read 536 bytes, then we read the remaining ones.

I sincerely thank you for your precious support.

Regards
Enrico

[Jim Keeble]

Ok, good.

The ECONNREFUSED threw me off a little, but know I see it's invalid because the return value wasn't -1.

Thanks .

[rick jones]

Strictly speaking, 536 bytes of data in a TCP segment is not that 576 is the smallest IP datagram. 576 happens to be the minimium limit for a _reassembly_ - that is to say that any IP stack must be able to reassemble IP datagrams of at least 576 bytes. MTU's can be much smaller.

Now, when TCP/routing does not know otherwise (explicit setting in route command or use of PMTU) it will use 536 bytes for the MSS to hopefully avoid fragmentation, but also to "know" that if the IP datagram carrying the TCP segment _is_ fragmented that the remote IP will be able to reassemble it.

As for MSG_WAITALL and all that, while it may be convenient to use, I would suggest that a robust application would _not_ rely on such mechanisms, and always keep in mind that TCP, being a bytestream protocol, does _NOT_ preserve message boundaries and is perfectly free to take that one 543 byte send at one end and turn it into 543 separate one byte receives at the other.

So, applications wishing to "preserve" message boundaries should (dare I say must) do so themselves. The simplest way, short of having a fixed message size, is to prepend each message with its length (in network byte order) and use that to determine how much more to recv out of the socket. If you know that a message will always be at least N bytes, then by all means post a recv of N bytes - perhaps it will then save the need of another recv call much of the time.

But the code should have logic in it to make multiple recv calls to get the entirety of a message.