NFS server seems to hang when being shutdown

[Marc Ahrendt]

my NFS server hangs so long during a "shutdown -ry 0" at "/sbin/rc0.d/K900localmoun
t stop"

so to complete the reboot i have to resort to power-cycling the box! after the server comes back to run-level 3 i check the rc.log.old and see that umount is failing with "Device busy" for filesystem like /export/home, /export/p
rojects, etc...

why? ...knowing the two main NFS client servers were shutdown before i attempted to shutdown this server

[Sridhar Bhaskarla]

You need to unmount the NFS
file systems before shutting down the
NFS servers.

As the NFS server was down, all the NFS mounts
became stale and the server was not able to umount them successfully.

-Sridhar

[Vincent Fleming]

If the mount point that it's hanging on is an NFS mounted filesystem, and the NFS server is down, it's probably stuck trying to contact the NFS server to tell it that it wants to unmount the FS.

There are several things you can do about this. You can keep the NFS server up until you finish the shutown of the hanging system, or you can use soft mounts (see the -o soft option on mount_nfs), or ...uh... you could limit the number of retries on the FS (not really advisable, IMHO), or ...uh... hmm - I can't think of anything else offhand.

If it's a local FS, then there's a process somewhere that either has a file open on that FS, or some process' current directory is (PWD)is on the FS. Finding the offending process can be tedious. Look for daemons that keep log files (or other files) on NFS mounts; also for any users that may be logged in, or have background processes that are lingering.

Good luck!

[Marc Ahrendt]

thx sridhar/vincent

so what do i do if the NFS server is also a NFS client to itself? i have autmounter running and am using it to access the local file systems, and thus when the server shutdowns it runs "nfs.server stop" right way before even getting down to run-level 0. the reason i use automount on some of the local file systems of the NFS server is so that users can login as use it with their same enviroment that they use on the NFS clients. do i need to avoid that in order to shutdown the NFS server cleanly? the NFS server is the NIS master also, and the two NFS clients are also NIS slaves.

[Sridhar Bhaskarla]

Marc,

Even then you need to unmount the local file systems. Do 'bdf -t nfs' and unmount all the
nfs file systems manuall. Then you do an nfs.client stop. You can either do nfs.server stop or reboot the system.

However, the remote systems that are accessing this system as NFS server will hang until this system comes back up. To find out all the nfs clients to this system, you can use the command "showmount".

-Sri

[Byron Myers]

The shutdown scripts should umount these NFS file systems - any way, if you script this, then try running "fuser -k " before umounting. This will kill all processes using the NFS mount (on the system being shut down).

[Marc Ahrendt]

thx sridhar/byron

i will try both ideas whenever i am allowed to reboot this main production server again. i think u guys are on target ...i was even thinking of just removing the link at rc0.d!

[Sadayappan]

Hi Marc,

1. During your next attempt if nothing works, you can temporarily bypass this step by typing Cntrl + | (Control plus Pipe key) during the shutdown process. Sometime it expects twice. (You do not neet to wait for long). It will jump to next process.

2. If all the suggested tips are verified, please verify the PATCH level. There may be a PATCH required while I doubt this will be the cause.

-Sad.