TCP buffers and system freeze

[Nimesh Myanger]

Hi,

I have a problem which seems to be related to TCP/IP on my L2000 server (HP-UX 11.00), and would really appreciate any help.

I have a set of programs running on the server. The main program acts as a transaction handler that recieves data (as a string over a tcp port) from user applications (windows app). This data is forwarded to the necessary server program, and any data coming back from the server program is forwarded back to the user app.

The problem I am seeing is that when there are a lot of system changes, the user app receives these changes as soon as it connects to the main server program - which could sometimes involve a lot of tcp traffic. Now if the users are connecting over a WAN, and there is a bottleneck somewhere, then the main server program (the one that routes the tcp data back to the user app) just freezes. Nothing happens for a while (anything from 4 to 20 minutes), and then everything returns to normal (well, some tcp connections are reset - which I assume were the ones that were holding up the tcp traffic). I was able to replicate this on the test server by just having one client app connecting over a dial-up line into the LAN, and unplugging the line when data is coming down. After several kb of data has been sent down to the client socket, the main server program will then freeze for about 8 minutes, and then suddenly wake up and close the socket which the client app connected on.

My questions are:

- if there are several users connecting to the main server program over a tcp port (which the server program then assigns socket numbers to), why is it that the program itself freezes if it fails to deliver tcp packet to just one of the sockets?

- if there is a program running on the HP server that accepts socket connections, is it not able to keep the socket connections free from each other such that if one socket were to have its data buffer full, this would not affect other socket connections?

Do my questions make sense? Please let me know if anyone needs additional info.

Thanks,
NM

[harry d brown jr]

You need to have someone make changes to your application to test for this issue and resolve it gracefully.

live free or die
harry

[Trevor Dyson]

The application may be single threaded so it only processes requests from one socket connection at a time. If this is the case then it may wait for the socket to time out before processing data from other socket connnections.

Is the application in house or commercially available?

[Brian Hackley]

Nimesh,

I thought you might want to find out what the process is doing when it appears to be hung, since you are able to reproduce the problem. You could trace the system calls that your process is using and try to infer from that system call trace, what might be going on that causes the hangs, e.g. single threaded-ness or blocking I/O.

There is a very recent update to tusc (truss), available at the HP-UX Software archive and Porting Center. http://hpux.cs.utah.edu/hppd/hpux/Sysadmin/tusc-7.3/

Find the PID of the application that is hanging and tusc against it. Follow child processes with -f and select any other options you might think are appropriate. For example, if the PID is 1234:
/opt/tusc/bin/tusc -fpvo /tmp/output 1234

CTRL-C or kill the tusc job when you think you've collected unenough data and then have the developer of the application look thru the system call trace file you just collected.

Hope this helps,

-> Brian Hackley

[Nimesh Myanger]

Thanks all for your responses. I have access to the source code and makefiles. How do I tell from this that the programs are single or multithreaded?

Best regards,
NM