Tru64 Java Sockets Problem ...

[Bernhard von Gunten]

Hello all

We've got an interesting Tru64, Java, Socket problem.

Environment:

Compaq Tru64 UNIX V5.1B (Rev. 2650); Thu Nov 3 12:54:41 CET 2005
NHD7 with PK5 + 1148.00 1181.03 1254.00 1273.00 1283.00 1268.00 1271.00 1202.01

java version "1.4.2"
Java(TM) 2 Runtime Environment, Standard Edition
Fast VM (build 1.4.2-4.p8, native threads, mixed mode, 10/17/2005-01:44)

Apache Tomcat/5.0.28

Problem:

We're running a web application (Servlet) that is calling Webservices (remote in LAN). We're doing about 350'000 Soap Calls / Day. Under heavy load we're starting more than 25 calls / second. Important: Each soap call opens and closes a new Socket connection! No keep alive is set, because of the service we're calling is demanding this. No SSL involved. Calls are usually done between 0.2 second (small result) and 1.0 second (bigger result) and our application returns the result to the browser.

Anyway, this was performing pretty good for months, but lately we're discovering problems huge with Sockets. The creation of new connections to the remote Machine is stalling. TCP Dumps show that there is zero traffic to remote system while these phases. I've debugged & isolated the problem to the moment where Java is creating a new Socket Object. Stalls might last for 1 Second and up to 15 Seconds. While requests are stalling and waiting for a Socket, other existing or new Threads of the Java application (Servlet) are performing fine. But all Threads trying to call the remote webservices are of course having the same problem while stalls and are waiting. The load is not heavier than in the months before, BUT the problem occurs more often while there is some "load" on the system, about 1 - 2 occurances per minute.

My conclusion:

- Java thread stalls while creating a new Socket object.
- TCP dumps show no SYN or any other traffic to the remote system while the thread is stalling.
- So the problem has to be somehwere in the creation of the socket object while demanding a new socket from the os. Right ??

My questions:

- Are there possible restrictions, thresholds or anything else in Tru64 that might prevent us from creating new outgoing sockets for this java process?
- As I'm a tcp/ip Tru64 noob ... What are the most important and significant tcp/ip parameters in this area? How do I check these ?
- As I trust into the hardware and os, might it be a Java VM Problem ?
- Anything else ... "Carte Blanche" ;-)

Thanks for your support!

Greetings from Switzerland,

Bernhard von Gunten

[Mark Poeschl_2]

Check 'sysconfig -q socket' and 'sysconfig -q inet' for relevant OS tuning parameters. See man pages at 'man sys_attrs_inet' and 'man sys_attrs_socket' for descriptions of the parameters.

In particular, there are some read-only counters from 'sysconfig -q socket' that might give you a hint.

Our application is not Java-based but does sustain high volumes of connection requests during busy periods. We have 'sominconn' and 'somaxconn' both set to the maximum - 65535.

[Bernhard von Gunten]

Interessting new fact:

On the same system, especially while these phases of stalls a "lsof > /dev/null" sometimes also takes up to 15 seconds!?

Still, sysadmin told us, that the cpu / io was NOT under heavy load.

Any thoughts?