1751738
Members
5762
Online
108781
Solutions
SOLVED
Hi Scott,
No, there isn't any artificial limit of 99 mounts on the NFS client. There must be a different kernel limit. If nkthread looks good then my guess is you've hit the "max_thread_proc" limit.
Can you please post the output of this command:
# kctune
I'd like to see how you have certain kernel tunables set.
Thanks,
Dave
No, there isn't any artificial limit of 99 mounts on the NFS client. There must be a different kernel limit. If nkthread looks good then my guess is you've hit the "max_thread_proc" limit.
Can you please post the output of this command:
# kctune
I'd like to see how you have certain kernel tunables set.
Thanks,
Dave
I work for HPE
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
Hi Scott,
Thanks for posting that output. Here's what I'm interested in:
max_thread_proc 1100 1100 Immed
nfs3_max_threads 16 16 Immed
Each NFS mount point will get a service thread pool of 16 threads. 99 mounts * 16 threads = 1584 threads total. Now, not every pool will have all 16 threads running at any given time. However, your max_thread_proc value is set at 1100, so as soon as the NFS client tries to create a new thread pool and the max_thread_proc limit is hit the mount fails.
You have a few options here.
o decrease nfs3_max_threads to a lower value
o increase max_thread_proc
o both
Once you allow more NFS mounts to occur you will likely need to increase the nkthread value to make sure you don't hit that limit next.
Regards,
Dave
Thanks for posting that output. Here's what I'm interested in:
max_thread_proc 1100 1100 Immed
nfs3_max_threads 16 16 Immed
Each NFS mount point will get a service thread pool of 16 threads. 99 mounts * 16 threads = 1584 threads total. Now, not every pool will have all 16 threads running at any given time. However, your max_thread_proc value is set at 1100, so as soon as the NFS client tries to create a new thread pool and the max_thread_proc limit is hit the mount fails.
You have a few options here.
o decrease nfs3_max_threads to a lower value
o increase max_thread_proc
o both
Once you allow more NFS mounts to occur you will likely need to increase the nkthread value to make sure you don't hit that limit next.
Regards,
Dave
I work for HPE
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
Hi Ravi,
When you say you have the same problem, you mean you're not able to mount NFS filesystems because of an exhausted kernel resource?
What OS is the NFS client system running? How many NFS mounts are you planning on using? What is the exact error message you see when you attempt to mount a filesystem and it fails? Are there any messages in either dmesg or /var/adm/syslog/syslog.log when this happens?
Regards,
Dave
When you say you have the same problem, you mean you're not able to mount NFS filesystems because of an exhausted kernel resource?
What OS is the NFS client system running? How many NFS mounts are you planning on using? What is the exact error message you see when you attempt to mount a filesystem and it fails? Are there any messages in either dmesg or /var/adm/syslog/syslog.log when this happens?
Regards,
Dave
I work for HPE
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
Hi Sujit,
> Also can the O/P of cat /etc/defaults/nfs
> be posted from the client so as to know
> which version of NFS is actually being
> affected? Is that NFS3 or NFS4.
The /etc/defaults/nfs file only exists on 11i v3, so if you have that file it answers my "which OS are you running" question. :)
Also, just because NFS v4 is enabled on this system that doesn't mean NFS v4 is being used. V4 only is used if *both* systems (client and server) support v4 and negotiate that at mount time. Even if both systems support NFS v4 you can always override that using the "vers=3" option.
To find out for certain which version of NFS is being used on a per-mount basis you should use the "nfsstat -m" command and check the output to see what version the mount points are using.
> In that case this kernel parameter can be
> checked;
> nfs4_max_threads 100 100 Immed
Be very careful tuning the nfs#_max_threads to something like 100. This parameter tells the NFS client how many threads it can use *for each individual mount point* - not how many threads to use system-wide for all NFS I/O on the client. In other words, if you have 100 NFS v4 mount points and you configure nfs4_max_threads to 100 you could conceivably cause 10,000 threads to perform NFS v4 operations. I've personally never seen a situation where that makes sense, but there may be some out there.
The default for all the nfs#_max_threads tunables is 8, meaning each NFS filesystem will have a dedicated pool of 8 worker threads for that specific filesystem. That typically works well for most workloads. On occasion I've seen situations where 16 or 32 gives better performance, but those are typically cases where the NFS client is a very large system and it uses few NFS mount points to do all its NFS activity. In those cases having a higher limit of worker threads may make sense.
Do you really need 100 dedicated worker threads for each filesystem (i.e. you've done comparisons with 8/16/32/64/100 threads and found 100 threads gives you the best performance and behavior)?
Obviously the higher you configure the limits for these worker thread pools the higher you'll need to increase nkthread and max_thread_proc. Again, these tunables only affect 11i v3 systems. Prior to 11i v3 we used a system-wide pool of biod daemons, so you would do all your sizing of the biod pool.
Regards,
Dave
> Also can the O/P of cat /etc/defaults/nfs
> be posted from the client so as to know
> which version of NFS is actually being
> affected? Is that NFS3 or NFS4.
The /etc/defaults/nfs file only exists on 11i v3, so if you have that file it answers my "which OS are you running" question. :)
Also, just because NFS v4 is enabled on this system that doesn't mean NFS v4 is being used. V4 only is used if *both* systems (client and server) support v4 and negotiate that at mount time. Even if both systems support NFS v4 you can always override that using the "vers=3" option.
To find out for certain which version of NFS is being used on a per-mount basis you should use the "nfsstat -m" command and check the output to see what version the mount points are using.
> In that case this kernel parameter can be
> checked;
> nfs4_max_threads 100 100 Immed
Be very careful tuning the nfs#_max_threads to something like 100. This parameter tells the NFS client how many threads it can use *for each individual mount point* - not how many threads to use system-wide for all NFS I/O on the client. In other words, if you have 100 NFS v4 mount points and you configure nfs4_max_threads to 100 you could conceivably cause 10,000 threads to perform NFS v4 operations. I've personally never seen a situation where that makes sense, but there may be some out there.
The default for all the nfs#_max_threads tunables is 8, meaning each NFS filesystem will have a dedicated pool of 8 worker threads for that specific filesystem. That typically works well for most workloads. On occasion I've seen situations where 16 or 32 gives better performance, but those are typically cases where the NFS client is a very large system and it uses few NFS mount points to do all its NFS activity. In those cases having a higher limit of worker threads may make sense.
Do you really need 100 dedicated worker threads for each filesystem (i.e. you've done comparisons with 8/16/32/64/100 threads and found 100 threads gives you the best performance and behavior)?
Obviously the higher you configure the limits for these worker thread pools the higher you'll need to increase nkthread and max_thread_proc. Again, these tunables only affect 11i v3 systems. Prior to 11i v3 we used a system-wide pool of biod daemons, so you would do all your sizing of the biod pool.
Regards,
Dave
I work for HPE
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]