lsof only provides information about a single process. If you want a list of all files opened by all processes, you'll need to run it for every file, but I don't think that's what you want (a listing that is 12,000 lines long). There is no command to list the quantity of files open by each process. Now your current process count is 279 and average processes open 5-10 files each, so the current number of files works out to be about 43 files per process, a very high number.
Therefore, I would suspect that some application program is opening hundreds of files at the same time. To narrow down the choices, sort the processes by memory used:
UNIX95= ps -e -o vsz,pid,ruser,args|sort -rn|head -20
This sorts the processes by memory used. Now run lsof against the largest processes. That may identify the process(es) that use a large number of files. It's also possible that some processes are opening a large nunmber of network sockets. Run the command netstat to see if this is the case.
Now all of this is probably quite normal for the applications. You can set nfile to 50,000 or even 250,000. This is a parameter that makes room for a open files and if your applications need lots of files, this is a very normal task for the system administrator. On the other hand, if you have developers writing programs and they have made errors in coding, then a bad program might needlessly open a large number of files. In that case, you can limit these problem programs using maxfiles and maxfiles_lim. Set maxfiles to about 100 and maxfiles_lim to 2048, then see what programs fail to operate correctly. Then for those programs that are known to be working as designed, use the ulimit -Sn command in the shell to temporarily raise the limit for this one program (or programs).
nproc and nfile can be extremely large if needed, so adjust them as required once possible problem programs have been corrected.
Bill Hassell, sysadmin
