NFS mount not going away

[Chandran Radhakrishnan]

Hello Everybody,

Have an issue in that a NFS mounted file system is no longer valid, as the NFS server was taken down. The clients are mounting the NFS filesystem as a soft mount thru /etc/fstab. Now on these servers, whenever we issue a bdf command, we get the below message: -
"NFS getattr failed for server xxx: RPC: Timed out", in other words bdf hangs for 5-10 minutes before responding. Have tried recreating the server xxx elsewhere, but guess on the client the pointers are to the first server xxx, so problem will not go away. Have tried stopping and starting the nfs daemons on the servers but problem stays. Other than rebooting the servers, is there a away to cleanup a "known hung NFS mount" - problem is compounded more by the fact that the servers in question are in production.

Regards
Radha Chandran

[Sanjay_6]

Hi,

Your only choice is to reboot the server (nfs client) to clear this issue. I do not think there is anything else that can be done.

Hope this helps.

Regds

[Alex Lavrov.]

NFS can be real trouble sometimes.
I agree that the only way to resolve it, it's to reboot the box.

I think if you'll do "bdf", it's gonna stuck too, so the faster you'll reboot the machine, the smaller are the chanses that something will go wrong on the system.

I had really big troubles with it in the past and from personal experience I can tell you that to leave a machine like this is VERY unhealthy.

[G. Vrijhoeven]

Hi,

If possible restart the nfs server ( or a new server that has the same ipadress) export the same exports and umount before taking the nfs server down.

Gideon

[eran maor]

Hi

i will not repeat the solution of reboot the server , but if this issue disturb you , you can define this nfs file system to be a soft mount and not a hard mount .

this will solve the problem of the timeout but there is also disadvantage of soft link againt hard links .

i m also sending you a nice pdf that will guide you on this problem

eran maor

[Charles Harris]

Hi,

I've had this problem before, and as everyone else has suggested, reboot's generally work. If your not in a position to do a reboot, I usually stop the nfs daemon, kill off any active processes attached to the mount using fuser, comment out the offending fstab entry, then use ndd disconnect_by_address (can't remember the real name sorry, it's in ndd -h unsupported) and remove all the hung tcp connections to the nfs mount by hand. To finish, stop and restart the nfs daemons and you *should* be ok. It's probably not a good solution but I've done it in emergency situations before and it's worked ok.

Good luck!

-=ChaZ=-

[Chandran Radhakrishnan]

Hi Charles,

I am trying the ndd command but hitting some roadblocks. The ip address of the nfs server is say x.y.z.q. From what I understand from ndd, the command syntax is to be

ndd -set /dev/tcp tcp_discon_by_addr x.y.z.q

the above gives me operation failed, invalid argument. Any idea what the correct syntax is?

Appreciate any information you can provide on this.

Thanks
RadhaC

[Charles Harris]

Hi, sorry for the very late reply, the NDD syntax needs the address and ports of both ends of the connection converting to hex.

It's a pain, I've got a script that does it I'll try to dig it out and attach!

Best regards,


ChaZ

[Charles Harris]

Hi,

Can't find my script, it's evaporated it's risky to automate it anyway.....

----snip----from---elsewhere------

The format is ndd -set /dev/tcp tcp_discon_by_addr "0a0a04081f42 a0a8c2f0527"

Where :

local ip=10.10.4.8 = 0x0a0a0408
local port=8002 = 0x1f42

remote ip = 10.10.140.47 = 0x0a0a8c2f
remote port = 1319 = 0x0527

---snip---------------------------

Hope this helps!

-=ChaZ=-