Command Hang ... NFS related issue

[subodhbagade]

Hi,

Need you valuable inputs on following case,

Having HP-UX server 11.31, facing some issue with command hang like bdf , df –k..

(1) While executing bdf it list all Non NFS FS but giving error for NFS on client

“NFS server 172.16.4.25 not responding still trying “

Note -NFS services are running on NFS server………

(2) During investigation, I noticed many FS are missing on NFS server itself due to which it is not mounted on NFS client but few are mounted NFS server but giving error of device busy on client ...

At same time not able to execute cmd mount –a …. It is hang


# mount -a
nfs mount: /sapcd: Device busy
NFS server 172.16.4.25 not responding still trying

I have exported as FS /sapcd from NFS server but still it is showing busy status on NFS client even not able to mount manually

Not sure why these commands are hang … pls suggest ..

Thanks in advance

Regards,
Subodh B

[Tingli]

Check the network between the client and the server.

[singh sanjeev]

try umounting the /sapcd and check what are the process keeping the FS busy

fusr -cu /sapcd

check also you should not be in that directory while executing the umount.

[Matti_Kurkela]

1.) As it says, the system is trying to communicate with the NFS server, but the server does not answer. That suggests a configuration or network problem of some kind.

2.) This looks more and more like a big problem in the NFS server. If the NFS server cannot see the filesystems, it cannot really serve them to any other systems either.

> # mount -a
> nfs mount: /sapcd: Device busy
> NFS server 172.16.4.25 not responding still trying

Did you run this command on the _NFS server?_
The NFS server should not attempt to NFS mount its own filesystems on itself, because that might lead to an endless loop.

Examine and fix the NFS server first.

If you don't know what to do, please show us the contents of /etc/fstab and /etc/dfs/dfstab (in HP-UX 11.31, /etc/exports is obsolete).

MK

[subodhbagade]

(1)
When I am trying to unmount the file system on client it giving me

# umount /sapcd
nfs umount: nfs_unmount: /sapcd: is busy
umount: return error 1.

# fuser -cu /sapcd
/sapcd: 16406c(root) 13087c(root)

(2) /sapcd file system exported from NFS server to other client and it is working there.

(3)Exported FS entry

# showmount â e
export list for tachyon6:
/var/opt/ignite/clients (everyone)
/sapcd graviton,quark,lepton6,neutron7,omicron
/usr/sap/trans (everyone)


(4) NFS entry in dfstab


/usr/sbin/share -F nfs -o rw=graviton:quark:lepton6:neutron7:omicron,ro= -d "none" /sapcd -

(5) on client , I can see its entry in df â k

even data are accessible

# df -k /sapcd
/sapcd (10.31.76.19:/sapcd ) : 30623040 total allocated Kb
12516819 free allocated Kb
18106221 used allocated Kb
60 % allocation used

But on bdf /sapcd

mount: 10.31.76.19:/sapcd is already mounted on /sapcd

(6) /etc/fstab

# more /etc/fstab
# System /etc/fstab file. Static information about the file systems
# See fstab(4) and sam(1M) for further details on configuring devices.
/dev/vg00/lvol3 / vxfs delaylog 0 1
/dev/vg00/lvol1 /stand vxfs tranflush 0 1
/dev/vg00/lvol5 /tmp vxfs delaylog 0 2
/dev/vg00/lvol4 /opt vxfs delaylog 0 2
/dev/vg00/lvol6 /usr vxfs delaylog 0 2
/dev/vg00/lvol7 /var vxfs delaylog 0 2
/dev/vg00/lvol8 /home vxfs delaylog 0 2
/dev/vg01/lvol /users vxfs defaults 0 2
/dev/vg01/lvol2 /usr/sap/DEV vxfs defaults 0 2
/dev/vg01/lvol3 /sapmnt/DEV vxfs defaults 0 2
/dev/vg01/lvol4 /oracle vxfs defaults 0 2
172.16.4.20:/printerPRD /printers nfs defaults 0 0
10.31.76.20:/sapmnt/DEV/global /sapmnt/DEV/global nfs rw,soft,bg,intr 0 0
10.31.76.20:/sapmnt/DEV/profile /sapmnt/DEV/profile nfs rw,soft,bg,intr 0 0
10.31.76.20:/users/avp /users/avp nfs rw,soft,bg,intr 0 0
10.31.76.19:/routes /routes nfs rw,soft,bg,intr 0 0
#10.31.76.19:/sapcd /sapcd nfs rw,soft,bg,intr 0 0
tachyon6:/sapcd /sapcd nfs defaults 0 0

10.31.76.19:/usr/etc/sudo /usr/etc/sudo nfs rw,soft,bg,intr 0 0



(7) # mount -aF nfs

nfs mount: 10.31.76.19:/usr/etc/sudo: Permission denied
mount: 10.31.76.19:/sapcd is already mounted on /sapcd
nfs mount: 10.31.76.19:/routes: Permission denied
mount: 10.31.76.20:/users/avp is already mounted on /users/avp
mount: 10.31.76.20:/sapmnt/DEV/profile is already mounted on /sapmnt/DEV/profile
mount: 10.31.76.20:/sapmnt/DEV/global is already mounted on /sapmnt/DEV/global
NFS server 172.16.4.25 not responding still trying

While below mwntion FS are already removed from NFS server.

:/usr/etc/sudo
/routes
:/users/avp
:/sapmnt/DEV/profile
sapmnt/DEV/global

Still issue with bdf , mount -a ...

Subodh

[singh sanjeev]

please hashed the entry of nfs file system which has been removed ,,,from /etc/fstab :

system is sending query for those FS

[Matti_Kurkela]

1.)

> # umount /sapcd
> nfs umount: nfs_unmount: /sapcd: is busy
> umount: return error 1.

> # fuser -cu /sapcd
> /sapcd: 16406c(root) 13087c(root)

"/sapcd: is busy" means there are still processes using the /sapcd filesystem on the client, so the OS thinks it's a bad idea to yank the filesystem out from under them.

Your fuser command just told you the PIDs of those processes. If you want to know more about them, run "ps -fp 16406,13087".

The letter "c" after the PID in the fuser output reveals that these processes have their current working directory set to somewhere within the /sapcd filesystem (i.e. these processes have executed the equivalent of "cd /sapcd" or "cd /sapcd/some/directory").

If these are shell sessions and you can still access them, simply use the command "cd /" on all of them before trying to unmount /sapcd.

If these sessions are hanging, try killing them with "kill -HUP". If that does not help, you may have to try "kill -KILL" on them.

In HP-UX 11.31, you could also use "umount -f /sapcd" to forcibly unmount the NFS filesystem even if it's still in use. If any processes on the client have unsaved data for that filesystem, that data will be lost.

2, 3, 4) thanks for information, this makes the situation clearer. Looks OK so far...

5.)
> even data are accessible

Perhaps some data is still in the local cache on the client, causing the illusion.

> But on bdf /sapcd

> mount: 10.31.76.19:/sapcd is already mounted on /sapcd

Do you mean that the bdf command actually tries to mount /sapcd??? This should not be happening, unless the standard HP-UX bdf command has been replaced with something else. Or did you just make a mistake in copy/pasting information to the forum?

6. and 7.)
When you run "mount -aF nfs", you're telling the system: "try to mount all NFS filesystems that are configured in /etc/fstab". So it does exactly what you asked.

As singh sanjeev said, you should comment out or remove from /etc/fstab the lines that refer to NFS filesystems that have been removed.

> NFS server 172.16.4.25 not responding still trying

This error message says that this client has problems communicating with the NFS server at IP address 172.16.4.25.

Try "rpcinfo -p 172.16.4.25" and "showmount -e 172.16.4.25" on this client host. Can the client get a list of RPC services from the server? Can the client communicate with the server to get a list of exported NFS filesystems?

MK

[subodhbagade]

Thanks for your inputs,

I have commented the entry in fstab of NFS client which are removed from NFS server, tested few things here

(1) fuser â cu /sapcd â ¦ No process was there
I have unmouted the /spcd
Stop the nfs.client , nfs.core on client
Restarted nfs.client , nfs.core

# Mount tachyon6:/sapcd /sapcd

o/p
mount: tachyon6:/sapcd is already mounted on /sapcd

I can see

Df â k /sapcd â ¦.. it is showing there but

Bdf /sapcd hang again

(2) #mount â v

tachyon6:/sapcd on /sapcd type nfs rsize=32768,wsize=32768,NFSv3,dev=400001f on Wed May 26 21:19:07 2010

# showmount â a ( no o/p)

(2) rpcinfo -p 172.16.4.25 ( this server even not pinging from client but not able mount

# Mount tachyon6:/sapcd /sapcd ( this is diff NFS server and FS are mounted on NFS server )


Please let me know why I am not able to mount / sapcd on client and bdf / df -k hang here â ¦.

[Dave Olker]

Something is not adding up.

> I have unmouted the /spcd
> # Mount tachyon6:/sapcd /sapcd
> mount: tachyon6:/sapcd is already mounted
> on /sapcd

If you've successfully unmounted /sapcd then why is the system reporting the filesystem still mounted?

The first thing we should do is get /sapcd unmounted from this client. What OS is the client running?

[subodhbagade]

NFS client is having 11.31 version same as server

I have unmounted the /sapcd successfully first and after that when i am trying to mount it again in that case it shows

mount: tachyon6:/sapcd is already mounted on /sapcd


df -k /sapcd show the the /sapcd but bdf -t nfs , df -k hang .. dnt know why ??


Do we really need to reboot the client to make this functional ..not sure but that would b d last option i blv...


Subodh