NFILES & vmunix: file: table is full

[Robert A. Pierce]

Good morning.

I'm about to make some kernel changes to my HPUX 10.20 K9000 800-series server, and would like a sanity check on my ideas.

We recently changed our Progress database from single-volume to multi-volume. Unfortunately, this means that we run out of open files when less than half the users get on the system.

NFILES is 4208.

sar -v shows e.g.:
file-sz ov
2370/4208 311
3699/4208 0

lsof | wc -l shows
6643

If I increase maxusers from 800 to 1000, that would move nproc from 1228 to 8020 and NFILES from 4208 to 15385. That should be good.

I saw a warning, though, that ninode should be kept less than 4000 on HPUX 10.20 to prevent spinlock, so I would overwrite the default formula with 3500 or 4000. The current value is 2852.

Does that sound reasonable? I wonder also if I should increase nflocks up from 200.

Thanks in advance for your feedback, advice, & suggestions. Corrections & flames welcome.

Rob Pierce

[Bill Hassell · Accepted solution]

Unfortunately, the formula value for ninode has been out of date for several years. ninode is the cache for HFS filesystems (typically /stand) and should be changed to a fixed value, perhaps 500-1000. The note about ninode > 4000 is also obsolete (spinlock issue was resolved years ago) but since there is no reason to have ninode larger than a few hundred, it doesn't matter.

[Stefan Farrelly]

The spinlock problem you talk about is only a patch issue - as long as your patches are uptodate (especially for that bug) then no problem making nfiles much larger. Weve got many 10.20 K's fully loaded and very heavily used and we run nfile at 15,000 on each (and ninode at 10,000) and theyve been like that for a couple of years and weve never had a problem.

[Robert A. Pierce]

Thanks very much.

So, I don't need an ninode setting that high, but if I have the correct patch, I can have ninode settings much higher -- correct?

Which patch would I need to correct the ninode/spinlock problem?

Has anyone else had problems with the itrc search feature timing out?

Thanks,

Rob

[A. Clay Stephenson]

Hi Rob:

My preference is to throw away the formulae and simply assign the values directly; I will admit that this does require a bit of understanding of the relationships of the tunables. The value for ninode is a good case in point. I assume that your system's only hfs filesystem is /stand. Ninode only applies to hfs filesystems. You could easily reduce ninode to 800 or so (and that is very generous)
and conserve resources. You do need to increase nfile and you probably should increase nflocks to at least 512.

Probably your most important tunables are dbc_max_pct and dbc_min_pct; keep them reasonable and certainly dbc_max_pct should be below 25. My preference on 10.20 and 11.0 boxes is to use fixed buffer cache by setting bufpages to a non-zero value.

Because you are a database application, one very common problem is the use of the tuned parameter set for databases. It has a very serious flaw in that it incorrectly sets timeslice to 1 rather than 10.

[Ceesjan van Hattum]

The database asks more nfiles per user/session. So.. increasing MAXUSERS is a remedy to something that is not your problem. And indeed, maxusers is a basis for a lot of other variables, besides nfiles also ninode and nproc.
But you number of processes (nproc) does not increase.

As long as other variables then NFILES is not the problem, i would keep my hands off of these, as long as i will not be able to see a relationship.

So, NFILES is 16*(...). Why not increase this 16 to 20 ?
Databases might ask more NINODE, that i can believe.
Then indeed i would delete the formula and replace it by the hardcoded 3999.

In short: my advice is only to change that what is really needed: NFILES + NINODE, and not MAXUSERS.

Regards,
Ceesjan

[Sridhar Bhaskarla]

Hi Rob,

I am a strong favourite of "not" using formulas.

Unless you are using hfs file systems for your data, you don't need to increase ninode value.

With your new maxusers value, you are increasing nproc by 8-fold and nfiles by 4-fold approx. You may want to recheck if it is what you wanted.

Sure. nflocks are to be bumped up. You may just want to double it.

-Sri

[Stefan Farrelly]

All our 10.20 servers are running either the General Release patch bundle (XSW800GR1020) version B.10.20.40 or B.10.20.52.2 (March 2001) and neither have ever had a spinlock problem with our nfiles set really high. I think any GR bundle from 2000 on and youre fine.

Otherwise, you can go to the ITRC home page, then to individual patches, then select HP-UX, then S800 and 10.20 and search for keyword spinlock and it lists quite a few patches, all of which you could download and install, but I would think it much better to use a General Release bundle.

[Robert A. Pierce]

NINODE & HFS:

I have two HFS partitions in additions to /stand; since neither are involved in my current problem, I'll leave the ninode setting where it is. I assume it would be a good idea to convert the HFS partitions to VxFS?

dbc_min/max_pct: I don't want to do too much at once, since this is a production box. I'll start a new thread later about adjusting these settings.

NPROC & NFILES
Would it be better to increase them both equally? I want NFILES to be over 14,000, since we're bombing out at 4200 with just a few users. If I manually increase NFILES to 15,000 and nflocks to 512, what should I set NPROC to?

Thanks,

Rob

[Sajid_1]

hello,

this document will help you figure out the new values of parameters:
http://www.docs.hp.com/cgi-bin/otsearch/getfile?id=/hpux/onlinedocs/939/KCParms/KCparam.Nproc.html&searchterms=nproc&queryid=20020813-093107

[Robert A. Pierce]

Okay. I set nfile to 20,000 and nflock to 512 & rebooted, and now everyone can log in without crashing.

sar -v shows file-sz at about "5700/20010 0," much better than "4120/4208 316" of this morning.

Thanks all for your help.

Rob

[Randy Tarrier]

Rob,
One other thing you may want to look at:
Datafile size is important, each one needs its own handle per session. If your database has some default file sizes set too small for your growth rate, then you'll create an unneccesarily large number of datafiles, and burn up your open files.
Randy