Command to find number of maxfiles and maxfiles_lim currently being used

[Jim Krol]

I have looked at sar, ems, and glance to see if any of these utilities would report how many of the maxfiles and maxfiles_lim are being used.

I am running several web servers on an HP L2000, 2 CPUs, 2Gb memory, HP-UX 11.00, 64-bit. We are using iPlanet-WebServer-Enterprise/4.1. One of the web servers running (ns-httpd) shows 40+ open files, 100+ threads. None of the other webservers have this many open files or threads.

I am also running about 4 other webservers and 3 proxy servers. On the above web server, I get the following message in the netscape error log:
"failure (thepid): Error accepting connection -5974, oserr=233 (Insufficient resources)".

We have talked to tech reps at iPlanet. They stated the error might indicate a "lack of file descriptors". As far as I know, this relates to maxfiles and maxfiles_lim.

So...how do I see what this current usage is of these kernel params. I know that I am NOT running out of resources for nproc, nfile, nflocks, ninode. Also, my max_thread_proc = 1024 and nkthreads = 2048. As far as I can tell, these parameters seem fine.

[Victor BERRIDGE]

Hi,
I would set maxfile quite large (>100)... just to be sure its not the same uid that is being connected several times and touches the limit...
For machine with java these are values I usually put:
maxfiles 2048
maxfiles_lim 2176

I wonder if its not more
shmmax max_thread_proc maxdsiz
maxssiz maxtsiz
parameters giving you trouble (lack of resource)

Just thoughts

All the best
Victor

[A. Clay Stephenson]

Hi Ted:

If you have lsof installed (and you should install it, if you don't) you can do a lsof and restruict by process or process group to list the open files and that is what the maxfiles_lim refers to.

You can also use Glance->Reports->Process List->Process Open Files.

[Steven Gillard_2 · Accepted solution]

In answer to your question, the tools lsof or glance can report the number of open files that a process has.

However, the error you've given does not indicate a problem with the number of open files - the iPlanet reps are wrong. The errno value given in the log of 233 is ENOBUFS or "no buffer space available". Note that this does not necessarily mean you are out of buffer space, but it occurs when the client resets the TCP connection before the servers accept() call is complete - something which can happen frequently especially when dealing with HTTP and browser clients. You can read the accept(2) man page to confirm the causes of ENOBUFS. There have been a couple of other threads on this recently.

As such, these messages can be safely ignored as they do not indicate any serious problem (as long as the application is not exiting after reoprting the error).

Regards,
Steve

[Jim Krol]

Thanks for the timely responses!

My maxfiles and maxfiles_lim is 2048. Shmmax = 201326592, max_thread_proc = 1024, maxdsiz = 268435456, maxssiz = 838608, maxtsiz = 201326592. Of hand, I would say these look good to me. Any suggestions?

Side note, NO other webserver error logs are giving me "Insufficient resource" errors. Also, if I am truly reaching a kernel resource limit, wouldn't I expect to see error messages in syslog or via dmesg?

I ran lsof -p and piped into wc...total 613. I looked in the GPM Process List and selected Process Open Files...total 37. Mmhh...why the discrepancy? Do I need to filter the lsof output more?

I did a man on accept, and as the last response states, ENOBUFS says "No buffer space available. The accept() cannot complete. The queued socket connect request is aborted". Now I haven't heard if the application is exiting or not (no user complaints), but, I don't really like just ignoring these errors (and BTW, excellent call...why didn't the iPlanet reps know this???). Some of the "Performance Tuning for iPlanet 4.0 server on HP-UX 11.00" states changing some of the TCP/IP parameters: tcp_time_wait_interval, tcp_conn_req_max, tcp_ip_abort_interval, tcp_keepalive_interval, tcp_rexmit_interval_initial, tcp_rexmit_interval_max, tcp_rexmit_interval_min, tcp_xmit_hiwater_def, tcp_recv_hiwater_def. I haven't applied any of these suggested tuned parameters. Should I? If so, which one would you recommend that might apply to the ENOBUFS?

[harry d brown jr]

Ted,

If you looked at the number of files opened in glance, did you do it per EACH process, because the "Glance->Reports->Process List->Process Open Files" is only good for the process highlighted.

Also, socket opens are file opens and count towards the number of files opened.

live free or die
harry

[Steven Gillard_2]

Actually, no matter what you tune, the ENOBUFS will not go away. On HPUX 11 they are no longer indicative of a problem, because they can be triggered by client behaviour which we have no control over.

Eg if a user hits the 'stop' button on a browser while the server is in the process of accept()ing the connection, the ENOBUFS error is returned.

So you really do have to ignore them! Our application only reports them if 10 in a row are received, otherwise they are not even logged.

Regards,
Steve

[Bill Hassell]

Just a note on maaxfiles and maxfile_lim: This is not a kernel resource but a protective fence. It's purpose is to prevent badly written programs from running away and using thousands of file handles. It is enforced on a per-process basis so 100 programs can each have up to 2048 files opened at the same time, for a total 204,800 files. Oops, better make sure that nfile is set to about 210,000.

The kernel parameter nfile is indeed a resource and limits the number of open files for the entire system. Use Glance or sar -v 1 to look at the nfile value to see if it is approaching 90% utilized. If so, then this is a true system-wide limitation and nfile should be increased accordingly.