Why would this program take at times more than 20 sec??

[Meenakshi Sundaram]

Hi,
In the attached program am creating threads and accessing my local web server. Initially I was fetching the content, but now am just opening a connection and closing it.
And when I have 200 threads doing the same operation for 100 times, at times opening the socket takes more than 20 seconds.
I have also tried the same with multiple process. 17 process each having 10 threads. Even when I have multiple processes I was facing the same error.
I have increased tcp_conn_request_max till 4096. But of no help. I have also tried source quench param.
I am working on HP 11 on L2000 series.
But this thing is working fine on my Win 2000 !!!

Can somebody help me ???

[Ron Kinner]

Take a look at netstat -an while this is going on. You may be getting a buildup of sockets in various stages of closing. The closing of a TCP socket is not an instantaneous process. Even after the process is closed the socket is held open for some time to make sure that there are no further messages.

You can tune this timeout somewhat with NDD.

tcp_time_wait_interval

Also check your nslookup and make sure you are reading the address from /etc/hosts instead of asking a DNS for the address.

Ron

[rick jones]

i would cuation _against_ setting tcp_time_wait_interval in anything other than lab situations, and even then i would suggest that the better course of action if TIME_WAIT port reuse is an issue would be to increase the anonymous port space being used. that can be done via ndd.

TIME_WAIT is there as part of TCP's mechanisms to protect against silent data corruption.

better still is to have the application make explicit calls to bind() with port numbers in the range of 5000 to 65535, assuming the language in which the program is coded provides access to bind() and selecting port numbers. this is what SPECweb96 and SPECweb99 load generators do

[Manoj Kumar Sarangi]

Hi,

If the same thing is working with Windows NT it should work in HP.

So, I think the problem may be related to your web server. If you are running apache, pl. attach httpd.conf , if u are using older version, pl. also attch srm.conf.

Manoj

[Chris De Angelis]

Hi Meenakshi,

Again, as I replied in your post subject of (something like) "Multiple Threads in JVM using Sockets", I would say you need to adjust your kernel parameters. I refer you to the same page:

http://www.hp.com/products1/unix/java/infolibrary/prog_guide/java1_3/configuration.html

There you will see that the minimum and default value for the maximum number of threads per process on HP-UX 11 is only 64. Given that you say you have tried running this program with a parameter of 200 for the number of threads, I'm surprised you didn't see a hard error like "Out of Memory" from the JVM.

Try setting those kernel parameters to something more reasonable for your usage and see if that helps. And of course, make sure to keep a backup your current kernel and kernel configuration in /stand before trying this.

Regards,
Chris

[rick jones]

another good thing to check would be on the web server - if running HP-UX 11 the netstat statistics will provide a stat for connection requests dropped due to full queue. basically, if the apache listen socket overflows with queued connection requests, this stat will be incremented.

possible remedy is to increase the parameter apache passes for the backlog in the listen() call, and make sure that tcp_conn_request_max is at least that large. i say possible, because if the _sustained_ connection request rate is greater than apache can sustain, any queue, no matter how large, will overflow.

[Meenakshi Sundaram]

Hi All,
Thanks for all your suggestions. Now that I have lot of options to explore I would do the same and post the results.

Regards,
M Meenakshi Sundaram.

[Meenakshi Sundaram]

Hi All,
I have tried the options. But still at times it is taking more time to open socket. Should I look at tuning tcphashsz param. As Rick has pointed out this would be needed in case of request rates more than 100/200. Also how do I find the current value. The /stand/system doesnt have this param.

Regards,
M Meenakshi Sundaram

[Michael Tully]

Hi,

Try this:

# kmtune -l
or
Look for the value in /var/sam/boot.config
or
Look for the value in /usr/conf/master.d/core-hpux

[rick jones]

I doubt that tcphashsz would be indicated here. You might also look at the depth of the server's listen queue and the output of netstat -p tcp on that server - it may show connection dropss due to full queue, in which case you need to up tcp_conn_request_max (assuming the server is UX 11) and then have the server app call listen() with a larger backlog.

also, something in this space that "works" on NT may not work on Unix - NT apps/stack often use an abortive (really bad idea) close of sockets by default, which short-circuits the TIME_WAIT state. doubleplusungood.

this is java right? perhaps your remaining (few?) log connect times are coincident with java garbage collection?

[Rita Sekhon]

We have observed a similar problem when a process issues a connect re-using a port # for which there is an existing connection in TIME_WAIT.

Here is what the process does:
- gets a port # (e.g. 646)
- binds to the port (no addr in use error)
- calls connect. This request succeeds and the connection is established. The connection is to another process on the local system via inetd.
- close connection
- connecton goes into TIME_WAIT as shown by the netstat -a cmd

tcp 0 0 blaze.bpxx blaze.646 TIME_WAIT

- ....
- the above sequence is repeated. This time if the same port is used i.e. 646 and the previous connection is in TIME_WAIT, the connect hangs and times out after a minute or so. The errno is 238 (ETIMEDOUT).
During this time the netstat -a shows
tcp 0 0 blaze.bpxx blaze.646 TIME_WAIT
tcp 0 1 blaze.646 blaze.bpxx SYN_SENT
- if we retry connect w/ a diff port #, the connect succeeds.
- if the connect is issued for port # 646 (e.g.) and there is no previous connection for port# 646 in TIME_WAIT, the connect succeeds.

We had started inted w/ -l option. The logs msgs in syslog did not show any connect request implying that the connect that timed out never made it to inetd.

I think there is a problem in the transport layer in dealing w/ a new connection w/ the same source and destination address as a previous connection in TIME_WAIT.

This is a solid failure. We are running HP_UX B.11.00 on 9000/800 machine.

[rick jones]

using the same port number over and over again for a client is generally discouraged - among other reasons, the one you are encountering.

there are conditions whereby a connection in TIME_WAIT can indeed transition directly to ESTABLISHED, but in this day of people insisting on increasingly random initial sequence numbers that is becomming increasingly unlikely (not that it was very likely before)