Kernel Questions.

[Ian Green]

Hi There:

We had a LOAD/SOAK test and the following were the findings.
Environment:11.0 N-class. with march '02 patch level and 20 Gig memory.

when we hit 297 users the filetable was full and we had to change
nfile from 20000 to 40000
nproc from 4420 to 7220
ncallout from 4436 to 7236
maxusers from 550 to 900

I did not change ninode 'coz it was already at 9114. When we continued LOAD/SOAK testing, we hit 360 users before the following message
came up

sh:The fork function failed. Too many processes already exist.

my dmesg reported kthread: table is full.

However my sar -v reported only 3501/7220 on proc-sz.(nproc)
I am attaching the kmtune output. I think i should increase nkthread
and max_thread_proc.
what would be a good value in these circumstances? Also please
let me know how can i monitor the value of these two parameters?
The load average on the machine with
360 users was 22.00(output of w command).

Thanks
Ian.

[A. Clay Stephenson]

Your ninode setting could probably be reduced further as it only applies to HFS filesystems and I suspect that your only HFS filesyste is stand. It could actually be reduced to under 200.

There should be no need to increase max_thread_proc; that simply limits the number of threads that a single process is allowed to spawn. With nproc set at 7720, it only a few of these are multi-threaded, you could exceed your present setting of nkthread. The default setting sould be 15456 (nproc * 2) + 16 to a maximum of 30000. I might use the formula nkthread = (nproc * 1.5) + 16 = 11596. That should be a reasonable value that still serves as a reasonable 'fence' but allows a good number of multithreaded processes.

[Victor BERRIDGE]

Hi,
I compared your params with our N4000 I use to test big configs exept you number of users we are quite the same except for :
ksi_alloc_max 33792 - (NPROC*8)
max_thread_proc 3000
maxdsiz 0x40000000
maxdsiz_64bit 0x40000000
maxfiles 2048
maxfiles_lim 2200
maxssiz 0x8000000
maxssiz_64bit 0x40000000
maxswapchunks 4251
maxtsiz 0x40000000
maxtsiz_64bit 0x40000000
maxuprc 300
maxusers 512
shmmax 0x40000000
ncallout 7920 - (16+NKTHREAD)
nclist 8292 - (100+16*MAXUSERS)
ncsize 10104 - (NINODE+VX_NCSIZE)+(8*DNLC_HASH_LOCKS)
nkthread 7904 - (((NPROC*7)/4)+512)

These ajustments were made in order to run rdbms (oracle 8i), controlm server and BAS with JDK1.3.1

I agree with Clay for the rest...

All the best
Victor

[Ian Green]

Hi:

Thanks. so you think the following changes would be fine

nkthread: 12000 (from 7751)
ninode : 2000 (from 9114)(although my sar -v indicated 9114/9114 for ninode.

Also my sar -v output indicated that my file-sz
was 25280/40010 with 360 users. so i might as well change that also
nfile : 32000 (from 40000)

Any comments about the load average as reported in top and "w" command. the load average was around 22.50.

Please advise.

Thanks & Regards
Ian.

[Victor BERRIDGE]

Hi,
As Clay mentionned ninode is for HFS, so unless you are using some HFS files system except /stand, 2000-3000 should do fine, you havent said how many processors you have nor what you are trying to run so its difficult to say, the boxes Im responsible when they reach a load of 15 (uptime or top) are in a bad state...
For java apps:
max_thread_proc >3000
maxdsiz 0x40000000 - You have 10 times less...
maxdsiz_64bit 0x40000000



All the best
Victor

[Victor BERRIDGE]

I also noticed:
dbc_max_pct 3
dbc_min_pct 1

swapmem_on 1

You seem to have 4 GB of swap which explains the swapmem option but with 20 GB RAM I would increase dbc_max_pct to 10


all the best

Victor

[Mark Landin]

The ninode parameter is used for a couple of other things besides just HFS. It indirectly sizes the DNLC, an NFS client's rnode table, and the 'maxcnodes' kernal variable for CacheFS. While it's doubtful that it's an issue in this case, reducing the ninode parameter can have side effects on other systems that aren't that obvious. It's probably safer to set it way too big than way too small.

[Victor BERRIDGE]

Hi Mark,
True!
But then if you dont know What is been expected nor the configuration you would suggest on a "general purpose/rule"...

which would be for performance stay between 2000/4000 knowing that exceeding 3000 if unnecessary is just waiste of resource...
but 'ncsize' which controls the dynamic Name lookup Cache (DNLC) for all file system types uses ninode, we agree it depends on what you tasks you dedicate you box (sorry for my english - Im suffering of the heat and have abused of rose and grappa... In fact I should be in bed...).

Until we have more precisions, its difficult to evaluate on how to optimize such a box...
I am very proud (and having tuned) of a K360 2 procs-2GB running 7 oracle bases one never being stopped (24/24 web...) + 1 sybase and beeing a ftp server, with in peek more than 600 concurrent users in production without any failure for more than 3 years...

Its all a question of finding the good compromize and Im a bit sad that the gurus in performance/tuning have not yet snt any comments...
Although I agree with you Mark I remember having configured a box with parameters in excess "probably safer to set it way too big than way too small" that ever wanted to boot on such a kernel...


All the best
Victor

[Ian Green]

Hi Victor:

Thanks for your time. However i am not able to come to a solution. I am sorry i did not mention about the cpu's. We have 6 cpu @ 750 mhz. and 20 gig ram hosting mostly xmotif applications. I am not too sure about your recommendation on dbc params as the gurus have always maintained them around 300-400 megs. I am still waiting for some concrete answers about the parameters i mentioned(nkthread, ninode, nfile). May be the gurus will answer today. Let us hope.

Thanks
Ian.

[Victor BERRIDGE]

Hi Ian,
I would go and look if glance/gpm is installed on the box, if not - install a time-limited trial version, that you should find on your apps cds, or download from HP

You will be able to see far more what's going on, if you have bottlenecks etc...

Good luck

All the best
Victor

[Mark Landin]

Another performance tool for HP 9000's is SOS/9000 from Lund Performance (www.lund.com). It's a bit less expensive than Glance, I think, has an Oracle performance module (not sure how complete it is, though), and keeps performance logs and can plot historical trends. They also have a charting package for management charts, etc. You also get one free performance audit-and-advice per year on your support contract. Lund is very good to work with ... I've been a customer for 8 years now. Tell them I referred you. :)