How to increase the number of nfsd with minimal service interruption

[Oliver Hung_1]

Aug 10, 2004 08:45:33 GMT unassigned

--------------------------------------------------------------------------------
Hi all,

I suspect an insufficient nfsd issue in my production box as there is continual growth of udp socket overflows and the "system" CPU usage is abnormally high (> 50%)

I wonder if the # of nfsd can be increased without server reboot.

In a testing server, with MC/SG, 4 nfsds, I modified /etc/rc.config.d/nfsconf with "NUM_NFSD=16" and then issued "nfsd 16"

After a while, "rpcinfo -T udp 127.0.0.1 nfs" showed error RPC: Program not registered

I needed to reboot the server to resume nfs

Any help is appreciated

Thanks,
Oliver

--------------------------------------------

Dave Olker Aug 10, 2004 14:14:40 GMT unassigned

--------------------------------------------------------------------------------
Hi Oliver,

You may want to start a new thread with this question as it's not really related to Ralph's inquiry.

The UDP socket overflow counter is a system global value. In other words, ANY UDP socket that overflows will increment this counter, not just NFS. There are a couple of ways to determine whether NFS (i.e. port 2049) is the overflowing socket and if running more nfsds would help.

1) Collect a network trace and look for ICMP Source Quench packets indicating that port 2049 is requesting a source quench

2) Use the ndd tool to examine the ip_udp_status screen and look at the "overflows" column to see which UDP socket is logging the overflows:

# ndd -get /dev/udp ip_udp_status

Regarding your question of whether nfsds can be increased without server reboot, the answer is yes. You need to stop the existing nfsd daemons before adjusting the number of running daemons, but you can definitely do this without rebooting the server.

The easiest way is to adjust the desired number of daemons in the /etc/rc.config.d/nfsconf file, as you indicated, and then issue the commands:

# /sbin/init.d/nfs.server stop
# /sbin/init.d/nfs.server start

This will stop the running nfsds and start the new number based on the NUM_NFSD value in the nfsconf file.

Your clients may experience a temporary outage while the daemons are being killed and restarted, but this is nothing compared to the outage they'd experience if you reboot the server. ;)

Also, if you're running MC/SG and HA/NFS, you could migrate the NFS package to the alternate node and then adjust the number of nfsds on the node where the package isn't running and vice versa. That way the NFS clients shouldn't experience any downtime while you adjust the daemons.

Hope this helps,

Dave
---------------------------------------------

I post this discussion in a new thread


Thanks for your advices, Dave. They are very useful for my further investigation.

I tried the ndd command but the overflow column was missing in ip_upd_status output. Perhaps the ndd software version is not up-to-date. (I did get overflow info in another testing server)

In my testing environment, the NFS server is running client application with mounted directory from same server.

The nfs.server stop/start command worked but it left serveral stale NTP file handles in client application (I did not stop the client processes during the test) I haven't tried the MC/SG package switching method yet.

In other NFS performance tuning discussions, I read a suggestion of 4 nfsd per CPU.
I may increase the number of nfsd from 8 to 16 first for the 4-CPU HP-UX 11.0 server.

I'll post if any improvement observed.

[Ravi_8]

Hi,

nfsd is number of daemonms to start on NFS server. it's value is nothing to do with high CPU utilization.

to manipulate nfsd daemons to the value set in /etc/rc.config.d/nfsconf you no need to reboot the machine
#/sbin/init.d/nfs.server stop/start
will do the job.

As for your CPU utilization is concernred, use the tool "Glance plus" to find the process hogging the CPU

[Dave Olker]

Hi Oliver,

Regarding the ndd ip_udp_status column, that was a new column added at patch PHNE_23456 for HP-UX 11.0. You need to have this patch, or one that supersedes it (and any dependent patches) installed before the "overflow" column will appear. The current ARPA Transport patch for 11.0 is PHNE_29473, and it has been around for almost 6 months now. I'd recommend installing that patch, and its dependencies, to get the new ndd output.

When you say:
_______________________________

The nfs.server stop/start command worked but it left serveral stale NTP file handles in client application
_______________________________

Do you mean NFS file handles? I've never heard of an NTP file handle. Are you running NTP in such a way that it uses NFS for some of its resources?

I'd be surprised if you got any NFS stale file handles as a result of issuing these two commands, since ESTALE indicates that either a file does not exist or is not exported. While issuing the /sbin/init.d/nfs.server stop command will un-export the filesystems, by the time this is done the nfsd daemons should have been killed, and therefore unable to respond with an ESTALE error. Also, the /sbin/init.d/nfs.start script re-exports the filesystems, and it does so before starting the nfsd daemons. So the filesystems should be exported the entire time the daemons are running, which should avoid any ESTALE errors.

Regarding the "right" number of nfsds to run on the server, there is no magic answer. It depends on many factors.

First of all, do you know for a fact that your NFS clients are mounting filesystem via NFS/UDP and not TCP? The number of nfsd daemons only affects UDP traffic, not TCP. The easiest way to quickly determine whether the server is servicing UDP requests, TCP requests, or both, is to issue the command:

# nfsstat -rs

and look at the difference between "Connection oriented" vs "Connectionless oriented" requests. For example, my server shows:

# nfsstat -rs

Server rpc:
Connection oriented:
calls badcalls nullrecv
15373629 0 0
badlen xdrcall dupchecks
0 0 10030832
dupreqs
0
Connectionless oriented:
calls badcalls nullrecv
61822 0 0
badlen xdrcall dupchecks
0 0 0
dupreqs
0


My server is servicing both UDP and TCP requests, but the majority of requests are TCP requests (15373629 vs. 61822).

As far as sizing UDP daemons, my recommendation is to start 64 nfsds, again assuming your server is servicing NFS/UDP requests. There are reasons for my 64 daemon recommendation, and they are spelled out in detail in my NFS performance technical paper (available at http://docs.hp.com/hpux/onlinedocs/1435/NFSPerformanceTuninginHP-UX11.0and11iSystems.pdf) and in my book "Optimizing NFS Performance: Tuning and Troubleshooting NFS on HP-UX Systems".

I hope this helps,

Dave

[Oliver Hung_1]

Hi Dave,

Thanks again for your help.

You are correct. It's a typo in my post. I meant NFS stale handle. I wonder if the stale handle problem is due to my application using self-mounted directories (i.e. The NFS server exports a configuration file directory, and my testing application, running in same NFS server, mounts the directory using a shared IP address for MC/SG)

My production server is using NFS/UDP

Server rpc:
Connection oriented:
calls badcalls nullrecv
0 0 0
badlen xdrcall dupchecks
0 0 0
dupreqs
0
Connectionless oriented:
calls badcalls nullrecv
303791460 0 0
badlen xdrcall dupchecks
0 0 558321
dupreqs
459

I think my currect number of nfsd is way too small (8 vs recommended value of 64)

From your paper, the stream head buffer will be increased to (64 nfsds * 8KB) = 512KB
Do I need to tune other related kernel parameters?

My server is also hosting Sybase Database. I understand that Sybase may use high sys %CPU due to polling effect.

I hope by using the right number of nfsd I can lower the sys %CPU usage by say, 10%. My guess is that portion of existing system CPU overhead is on resolving outstanding NFS/UDP requests/errors

Thank you very much.

Regards,
Oliver

[Dave Olker]

Hi Oliver,

The fact that you're using a loopback NFS mount (i.e. client and server are the same system) should have no effect on the the semantics of the server returning an ESTALE to the client process - even if the client process is running on the same system. The ESTALE gets returned only if the file being referenced no longer exists or is not exported.

If your application is using temporary files and those files got deleted by another process, which may be likely since both client and server processes are running on the same system - so a local process could delete a file that the NFS client application is looking for, then this would explain the ESTALE.

If the ESTALE is returned because of a file not being exported then I'm concerned by that because in the "stop" logic the nfsds get killed before the filesystem are unexported, so theoretically the client shouldn't get a response from an nfsd while the filesystems are not exported during shutdown. The "start" logic exports the filesystems before starting the nfsds, so again the nfsds shouldn't be handling requests while the filesystems aren't exported. All of this stop/start logic is designed specifically to avoid cases where the nfsds would receive requests while the filesystems are not exported.

As far as tuning other kernel parameters, my technical paper covers all the applicable parameters for NFS servers. Since this system also runs Sybase, you should be sure that any kernel parameter tuning would not negatively impact Sybase.

Whenever you tune the kernel, you should always try to use the method: "make one change, measure, make another change, measure". That way you can see conclusively whether the kernel change you made had a positive or negative effect on the system and applications. If you try to make multiple kernel changes at once you run the risk of having some of the changes benefit the system while others hurt performance and you won't know which helped and which hurt.

Regards,

Dave