nfile parameter increasing

[Steven Richards]

Hi,

I'm looking for some help with regards to the nfile parameter on a HP-UX 11.11 server. I would like to understand what the "normal" behaviour is for this parameter.

The server in question had the nfile parameter set to 10000 until about a 45 days or so ago. Our alerting picked up that the usage was getting high and we raised the figure to 30000. Our capacity people were able to show from data collected that after a certain date that the figure started to increase steadily. Since increasing the figure to 30000 we have been keeping an eye on the output from 'sar -v' and noticed that after the reboot for the parameter change it started at around 3500 and has steadily increased over 45 days to 11500. So it appears it will eventually (although not for a while) hit 30000 if this behaviour continues, is this normal? My feeling is that some part of the application running on the server is misbehaving with regards to nfile.

Are there any tools for tracking what nfile is doing?

Hope this makes sense, many thanks in advance for your help.

Steve.

[Pete Randall · Accepted solution]

Steve,

The nfile parameter simply defines the number of slots available in the file descriptor table in the kernel. When you run out of available slots, then you can't open any more files.

Now, normally, when a process completes, it should close all it's open files, thus releasing the file descriptor table entries for re-use. If this number continues to increase, then you have some very poorly behaved applications that are not closing down properly!


Pete

[Steven Richards]

Hi Pete,

Thanks for the reply. What you have described confirms my fears that the applciation is poorly written and is not releasing it's open files once it has finished. The problem now is trying to locate what excatly is doing this.

I guess the next step is to log a call with HP and see if they have any tools so I can hand the app team some proof!?

Many thanks, Steve.

[MarkSyder]

If you know what the app is, you could run a ps -ef|grep appname and check for multiple occurrences. I have known this to cause a problem with nfile.

Mark Syder (like the drink but spelt different)

[Steven Richards]

Thanks Mark for the suggestion, but sadly it's not going to be as simple as that (wish it was!).

[Patrick Wallek]

If you don't have it, you need to download and install the utility lsof. This could help you pinpoint which process(es) are the culprits.

lsof (list open files) can list a ton of information about processes, tcp/ip connections, etc.

lsof can be obtained from:
http://hpux.connect.org.uk/hppd/hpux/Sysadmin/lsof-4.78/

[Steven Richards]

Hi Patrick,

I do believe we have lsof installed on our systems, although I have limited expertise in using it.

Many thanks for the suggestion.

Steve.

[Steven E. Protter]

Shalom,

Just type lsof and you'll see some common options and be able to track this.

You might also find glance gui high water mark display might give you a picture of how close your system is to limits during the day.

Commonly application issues can cause file handles to open and not be closed.

SEP

[A. Clay Stephenson]

One thing that you should be aware of with respect to the nfile tunable is that n open instances of the same file count as but one. This means that if 15 applications have the file "/etc/passwd" open, from the perspective of nfile the count is 1 not 15. Conversly, from the perspective of the per-process file limits (maxfiles, maxfiles_lim) each open file descriptor, whether or not the process has the same file open more than once, counts as 1.

It's difficult to say whether or not your application is behaving normally. Some applications are smart enough to examine the limits and tune their behavior accordingly because repeated opening and closing of a file are (relatively) expensive operations.

[tkc]

this link will give you more explanation about the nfile parameter :

http://docs.hp.com/en/939/KCParms/KCparam.Nfile.html


Other kernel parameters :

http://docs.hp.com/en/939/KCParms/KCparams.OverviewAll.html

[Steven Richards]

Hi Guys,

Thanks for the input much appreciated.

It would seem I have enough info to start trying to diagnose if something is behaving badly or not. Our best guess is that the a change was made to the App on the date when we start to see the steady increase begin. I'm going to ask the App support team to try and pin point what they think happenned at that point. Interestingly there also seems to be a big increase over month end, so also suspect that an end of month batch processing job maybe the culprit.

Cheers, Steve.

[A. Clay Stephenson]

One thing to note is that a single process could only get you into nfile trouble if maxfiles and maxfiles_lim were very large. Otherwise the maxfiles_lim per-process limit would come into play long before the system-wide nfile limit ever did. As soon as this rogue process terminated, all of its file descriptors would be freed. From your description, you may have processes which never terminate and that is what is actually eating your files.