how to monitor maxfiles on a server over a period of time.

[MRSG]

I have tried to look at the previous messages in the forum but was unable to find the solution to my query.

I want to monitor the kernel parameter maxfiles say over the next 7 days, to see whether it exceeds its set limit or not. We have had an error in the past saying too many open files (Error 24, which suggests it means too many open files per process). Maxfiles is set to 2048 right now.

Is there a way I can do that ? possibly by a shell script ?

[Kevin Wright]

sysdef |grep maxfiles

[Pete Randall]

Glance, in it's System Tables Report will give you current usage and maximum usage. Leave the server up for 7 days and just watch the high water mark with Glance. If you don't have Glance, you can install a free 60 day trial off the application CDs.


Pete

[Steve Steel]

Hi

Glance is best

sar -v (See man sar can help)

-v Report status of text, process, inode and file tables:



Steve Steel

[MRSG]

sysdef | grep maxfiles only tells what is the maxlimit. Does not say what is the current status.

System Table report in glance does not report maxfiles, though it does report nfiles.

sar -v also does not report maxfiles.

Any other possibilities ?

[Srinivas Thokala_1]

use sar -v 1 10 ( 10 is variable for number of repeats)

will show you present limit and tuned limit.
If they are close by then you can increase
nfiles parameters.

[MRSG]

Srinivas,

sar -v reports nfiles, but i want maxfiles to be reported.

[Bill Hassell]

maxfiles never changes value. It is a kernel parameter to limit bad programs from running away and using all the file descriptors in the system. You are correct that errno 24 is a per-process limit (maxfiles limit) and not errno 23 which is the system-wide limit of all files opened at the same time (nfile).

What you are asking is to identify how many files each process opens, but this isn't necessary. The program that crashes with errno 24 is the culprit. The first question is:

Why is this program trying to open 2,049 files? (seems like a programming error or poor design to me)

Now if this program can justifiably be allowed to open 3,000 files at the same time, then you need to change 2 kernel parameters:

maxfiles = 100
maxfiles_lim = 3000

Reducing maxfiles to 100 is a safety limit for bad scripts and programs that should never open more than 100 files at the same time. The second limit (maxfiles_lim) is the hard limit for any program. maxfiles is a default value but can be programatically increased with the setrlimit system call, or increased with the ulimit command (POSIX and csh shell only, not ksh). So for the truly strange program that needs 3,000 files open at the same time, change the current environment with ulimit and then run the program as in:

$ ulimit -Sn 3000
$ /opt/application/bin/open_lots_of_files

I recommend making maxfiles to default and not letting every user and program on the system mess around with massive numbers of files. The reason is that if all the file descriptors in the system are used, NO ONE CAN LOGIN including root! This has been a denial of service attack in Unix in the past. The -S option makes this a soft limit so for the current shell, the value may be lowered and raised as needed. Without -S, the value becomes the permanent upper limit for this shell. Note that if 20 copies of this program are run at the same time, you'll need nfile set to more than 60,000! And you'll also see a huge increase in system overhead while thousands of files or located or created in each process.

[Steven Gillard_2]

Attached is a C program I wrote that was extremely useful to us in tracking down a file descriptor leak. It uses the pstat*() syscalls to count the number of files open per process, keeping a sorted list based on this number.

By default it displays the top 20 processes every 20 seconds. You can change the number of processes reported and the interval with command line arguments.

To compile:
$ cc -o filecounts filecounts.c

And to report the top 2 processes every 10 seconds (data output is number of files, process id and command):
$ ./filecounts 2 10
================= Mon Jul 21 10:02:54 2003 =================
105 23717 /opt/bea/jdk131/jre/bin/../bin/PA_RISC2.0/native_threads/java -
47 1631 pmd -SOV_EVENT;t
================= Mon Jul 21 10:03:04 2003 =================
105 23717 /opt/bea/jdk131/jre/bin/../bin/PA_RISC2.0/native_threads/java -
47 1631 pmd -SOV_EVENT;t

Regards,
Steve

[James Randall · Accepted solution]

Glance has an adviser syntax script in /opt/perf/examples/adviser/num_open_files

Modify it to watch the offending process by name...and run as follows:

glance -j 60 -syntax /num_open_files -adviser_only
- This script will update every 60 seconds...and will not end until you -C out of it.

I've run into the same situation with MeasureWare's TTD process needing more than our configured MAXFILES value....and am looking at a fix using the Bill's "ulimit" solution above. (As I don't want all the processes the ability to go over the MAXFILES)

[Jeff Lightner_1]

I just wanted to add a note about the source Steve Gillard attached. It is good for finding highest usage. To compile for 64 bit (11.11 in my case) you have to do:
cc -D_PSTAT64 -o countfiles countfiles.c
If you don't it will compile without complaint but the object produces only timestampes. With the above compile flag it works as Steve stated.

P.S. Thanks to Steve for the source.