max_tread_proc exceeded

[Dave Polshaw]

We recently observed that the vxfsd process was using 541 threads against to the max_thread_proc value of 512.

kmtune confirms that max_thread_proc is indeed 512 and the number of threads (541) was observed from glance.

An extract from the glance screen follows -
(Sorry - I cannot format the screen but the 541 figure was under the 'Thrd Count' header -

vxfsd 41 root 0.8 541 44.8

How can a process exceed the maxinum number of threads and how come vxfsd is using so many anyway?

Thanks in anticipation.

Dave

Cheers

Dave

[Kapil Jha]

Are you sure u looking at the right thing?
How do u loooking this in glance.

BR,
Kapil+

[Dave Polshaw]

On the main glance screen the 5th column along is the thread count. So I am led to believe!

[Dennis Handly]

>the 541 figure was under the 'Thrd Count'

That seems obvious enough.
Perhaps system demons aren't like you and me and can bypass these limits?

> On the main glance screen the 5th column along is the thread count.

In gpm, mine just happens to be column 8.
And my vxfsd now has 34 threads.

[Dennis Handly · Accepted solution]

You can also use my example program that uses pstat_getlwp(2) to count the number of threads for that PID:
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1296647

[Kapil Jha]

I am not sure Denice if it can increase on its own, I have never seen this kindda behaviour.

http://docs.hp.com/en/B3921-90010/max_thread_proc.5.html

this says threads can not exceed this limit
:(

BR,
Kapil+

[Dennis Handly]

>I have never seen this kindda behaviour.

Right, that's why double checking the number of threads would be helpful. I hope max_thread_proc wasn't decreased recently. That would leave the thread pigs orphaned.

[Dave Polshaw]

Hi. Thanks all for your input. Points on the way. We cannot, unfortunately, install or run any unsupported routines on these machines due to political circumstances but I have pointed out that there are routines which will, hopefully, be more accurate than the glance stats. The max_thread_proc has not been increased or decreased recently so I do not believe that is an issue. I think the nearest we can get to it is that the number of threads was high for the 5 second glance refresh window. I am not sure how glance calculates it stats but if, for instance, it was dropping old threads and starting new ones at the rate of 20 per second I could understand how the 5 second counter would show more than the maximum number for a short period of time. Does that make sense to anyone or can anyone provide any information to back up or refute the theory?

Thanks again

Dave

[Dennis Handly]

>Does that make sense to anyone?

I would think glance would call pstat_getlwp(2) once during the 5 seconds and shouldn't get too many threads.

I'm not sure if anyone asked but why do you care how many threads are used by vxfsd, do you have a real problem?

[Dave Polshaw]

Hi Dennis. You are right of course and, as long as it is working, it does not really matter but it was a question by a user who has access to glance. You know what they say about a little knowledge being dangerous! It is academic but I must say it intrigued me as to why the absolute limit has been exceeded, hence the question. Thanks for your time anyway.

[Michael Steele_2]

Hi

Determining the Number of Active JFS Inodes in Use

# vxfsstat -v / | grep inuse
vxi_icache_curino 128001 vxi_icache_inuseino 635

"...Note that the current number of inodes (128,001) is greater than the maximum number of inodes (128,000). This behavior is normal as there are few exceptions that allow allocating a few additional inodes in the cache..."

Page 18 - "Common Misconfigured HP-UX Resources"

http://docs.hp.com/en/5992-0732/5992-0732.pdf

(* Great doc BTW - All should download *)

[Michael Steele_2]

Sorry,

I think this is the important comment since inode and threads are interchangable here, also on page 18.

"...The inode cache is filled with 128,001 inodes but only 635 are in use. The remaining inodes are inactive, and if they remain inactive one of the vxfsd daemon threads will start freeing the inodes after a certain period of time...."

[Dennis Handly]

>Michael: I think this is the important comment since inode and threads are interchangeable here

I wouldn't think interchangeable since one thread should handle multiple inodes.
And you have two different policemen here, the kernel and Symantec. They may have different bribing policies. :-)

>the vxfsd daemon threads will start freeing the inodes

Thanks for letting us know one purpose of vxfsd.

[Michael Steele_2]

Dennis:

Thanks for pointing that out. I suppose I was trying to say the question was about threads but really links back to inodes. Sorry for the mistake. Thanks!

[Dave Polshaw]

I think it will just be one of those mysteries that make life such a treat:-)

Thanks allfor your input anyway,