I've searched through the forum and found that this is a common problem. Often the discussion ends with either an answer saying that it is safe to ignore OR to change some kernel parameters.
Can I confirm with everyone that is running iPlanet on HP-UX 11i that this problem is safe to ignore?
BTW, if it is safe to ignore, then how will I know if it is a real error or not?
Also look at the number of TCP connections to/from your machine using 'netstat -anf inet'. Look especially for the number of TIME_WAIT and FIN_WAIT connections. Perhaps lowering a some tcp timeouts could help too (tcp_fin_wait_2_timeout and tcp_time_wait_interval, for instance).
I had similar problems on a very popular site running iPlanet 4.1 and HP-UX 11.00 and lowering those variables helped keeping the system running, though we also tweaked some iPlanet settings.
Every problem has at least one solution. Only some solutions are harder to find.
i'm a triffle surprised that increasing tcp_conn_request_max helped get rid of the ENOBUFS since those are generated only by aborted connections that have already gotten onto the listen queue, and too short a tcp_conn_request_max (don't forget that the actual backlog will be the min of that and what the app passes in listen())
although I suppose if it means that some SYN's are ignored, increasing client connect() time, that would make it more likely that the client would terminate the connection before the server got around to calling accept().
another consideration in getting rid of those would be to run faster web server bits - Zeus comes to mind there, and you could even bring your NSAPI stuff along for the ride.
i suppose that if the 233 was a real error, the number of them relative to the number of connections established would be rather high.
there is no rest for the wicked yet the virtuous have no pillows
just did a check today and found that the changes we did for our kernel parameters din help much. the errors were lower yesterday as there were less users on the system.
elmar, you mentioned that tuning tcp_fin_wait_2_timeout and tcp_time_wait_interval could help. my values are 0 and 60000 respectively. what would be a good value to work on? can u share the tuning u did on your iPlanet?
rick, we are getting this error 64 times in 1 hour at peak times. the number of requests is about 3-5 per second. would u consider this a problem? for the moment we stuck with iPlanet 6.0 :-(
I'd just like to add that we have had a very similar error (not with iPlanet) on one of our heavily loaded machines here for some time now. Apart from it alerting annoyingly to ITO, it doesn't seem to have bothered anyone.
Never preceed any demonstration with anything more predictive than "watch this"
i suppose that 64 of those in one hour (3600 seconds) when the connection rate is 3 to 5 per second is low enough to ignore. that is 64 out of 10800 or more.
i personally would _not_ suggest altering tcp_time_wait_interval. I can see no way that would affect a web server's ability to accept a new connection. by the time a connection is in TIME_WAIT, there is no longer a reference (file descriptor) to it from the web server software, so it isn't consuming a file descriptor or anything, and if the file descriptor limits for the process are reached, accept() would start returning EMFILE (or ENFILE if the system-wide limit was reached)
tcp_fin_wait2_timeout is (IMO) simply a kludge to work-around broken firewalls and/or applications. it is an arbitrary timeout such that if no FIN is received from the remote in that length of time, the connection is aborted.
99 times out of ten, if the problem is that the remote end has simply vanished (because perhaps it was crappy client code written to use an abortive close) the tcp_keep_alive_detached_interval mechanism (started automagically when the app calls close() on a socket) should detect. It would only keep the connection alive if the other end of the connection were still there. at _that_ point, an application-level timeout is really what should be used as the tcp_fin_wait2_timeout is a global parm and it could nuke an otherwise useful connection.
i must say that I am really surprised that even iPlanet cannot keep-up with a 3 to 5 connection per second request rate. those connections must be doing something that can keep iPlanet occupied for a really long time. are you running any heavyweight in-process NSAPI code or something?
there is no rest for the wicked yet the virtuous have no pillows
thanks for your reply. do u know how often the error messages appear in your logs? i would like to compare with the rate the errors are appearing in my logs - just to be sure that I'm not the only one with this problem.
rick, thanks for your detailed reply. Its comforting to know that the errors might not be an issue. I guess I will not be changing those parameters then.
However, I just got hold of the Tuning Recommendations for IPlanet (NES) 4.x Server on HP-UX 11.0 document from some kernel. I've compared the recommendations and found that several parameters are not up to the recommended values, so we've decided to at least tune these parameters for the time being. They are:
nflocks eai02 value: 200 recommended value: 8192
ninode (change to 2x original value) ((NPROC+16+MAXUSERS)+32+(2*NPTY)) to (((NPROC+16+MAXUSERS+32+( 2*NPTY))*2)
nfile (change divisor for NPROC term to 2) From: (16*(NPROC+16+MAXUSERS)/10+32+2*(NPTY+NSTRPTY+NSTRTEL)) change to: (16*(NPROC+16+MAXUSERS)/2+32+2*(NPTY+NSTRPTY+NSTRTEL))
The frequency depends on the load on out machine. During heavy times we get it five to ten times a minute at other times we get it a few times in twenty minutes. We are taking this machine out of service soon so haven't really bothered to do much about it.
Never preceed any demonstration with anything more predictive than "watch this"
We recently retired an L1000 running iPlanet and VirtualVault. We saw these errors A LOT during heavy use, with the additional bonus of iPlanet dying at random intervals.
As I recall, we did increase values for ninodes, nfiles, semmni, and some other semaphore values. Matters improved a bit, but heavy traffic was still enough to bring iPlanet down.
We finally managed to move off the platform, but the underlying problem was still there.
A little knowledge is dangerous - none is absolutely terrifying!!!
those TCP settings aren't heinous, but I doubt they will speed-up iPlant much if at all.
going to one minute on the tcp_ip_abort_interval is a bit harsh in my mind - assuming the minimum retransmission timeout (RTO) is in effect then that means that there will be fewer retransmissions of a given lost segment before the transport aborts the connection - if you believe that there are "never" any "normal" times when the network is paused or really really busy for less than 10 minutes, but more than one I guess it is OK.
the keepalive setting is ok i guess.
there is no rest for the wicked yet the virtuous have no pillows
thanks for your replies. so it seems to me that most sites running iPlanet on HP-UX seems to be faced with this problem. :-( I guess I will monitor the server; if none of the users complain, I'll just leave it as is.
rick,
thanks alot for your useful commments on the settings. I'll pass on your comments to the sysadmin team.
just wanna clarify with you regarding the parameter tcp_ip_abort_interval. You mentioned that setting it to a lower value would mean fewer transmission when there is a lost segment.
I'm just curious - would this affect long running transactions? Coz occasionally, my webserver will connect to a middleware which sometimes takes a long time to complete. (sometimes 1-5 mins)
Would setting the parameter lower mean that my long running transactions time out easily?
The tcp_ip_abort_interval is how long TCP will wait for an ACK of a given segment from the remote TCP. It is not affected by application service times. So, if you have an application that takes 5 minutes to service a request, that is fine as far as tcp_ip_abort_interval is concerned. TCP will have ACKed the data long before that.
there is no rest for the wicked yet the virtuous have no pillows
