Command Hang ... NFS related issue

[subodhbagade]

Hi,

Need you valuable inputs on following case,

Having HP-UX server 11.31, facing some issue with command hang like bdf , df –k..

(1) While executing bdf it list all Non NFS FS but giving error for NFS on client

“NFS server 172.16.4.25 not responding still trying “

Note -NFS services are running on NFS server………

(2) During investigation, I noticed many FS are missing on NFS server itself due to which it is not mounted on NFS client but few are mounted NFS server but giving error of device busy on client ...

At same time not able to execute cmd mount –a …. It is hang


# mount -a
nfs mount: /sapcd: Device busy
NFS server 172.16.4.25 not responding still trying

I have exported as FS /sapcd from NFS server but still it is showing busy status on NFS client even not able to mount manually

Not sure why these commands are hang … pls suggest ..

Thanks in advance

Regards,
Subodh B

[Tingli]

Check the network between the client and the server.

[singh sanjeev]

try umounting the /sapcd and check what are the process keeping the FS busy

fusr -cu /sapcd

check also you should not be in that directory while executing the umount.

[Matti_Kurkela]

1.) As it says, the system is trying to communicate with the NFS server, but the server does not answer. That suggests a configuration or network problem of some kind.

2.) This looks more and more like a big problem in the NFS server. If the NFS server cannot see the filesystems, it cannot really serve them to any other systems either.

> # mount -a
> nfs mount: /sapcd: Device busy
> NFS server 172.16.4.25 not responding still trying

Did you run this command on the _NFS server?_
The NFS server should not attempt to NFS mount its own filesystems on itself, because that might lead to an endless loop.

Examine and fix the NFS server first.

If you don't know what to do, please show us the contents of /etc/fstab and /etc/dfs/dfstab (in HP-UX 11.31, /etc/exports is obsolete).

MK

[subodhbagade]

(1)
When I am trying to unmount the file system on client it giving me

# umount /sapcd
nfs umount: nfs_unmount: /sapcd: is busy
umount: return error 1.

# fuser -cu /sapcd
/sapcd: 16406c(root) 13087c(root)

(2) /sapcd file system exported from NFS server to other client and it is working there.

(3)Exported FS entry

# showmount â e
export list for tachyon6:
/var/opt/ignite/clients (everyone)
/sapcd graviton,quark,lepton6,neutron7,omicron
/usr/sap/trans (everyone)


(4) NFS entry in dfstab


/usr/sbin/share -F nfs -o rw=graviton:quark:lepton6:neutron7:omicron,ro= -d "none" /sapcd -

(5) on client , I can see its entry in df â k

even data are accessible

# df -k /sapcd
/sapcd (10.31.76.19:/sapcd ) : 30623040 total allocated Kb
12516819 free allocated Kb
18106221 used allocated Kb
60 % allocation used

But on bdf /sapcd

mount: 10.31.76.19:/sapcd is already mounted on /sapcd

(6) /etc/fstab

# more /etc/fstab
# System /etc/fstab file. Static information about the file systems
# See fstab(4) and sam(1M) for further details on configuring devices.
/dev/vg00/lvol3 / vxfs delaylog 0 1
/dev/vg00/lvol1 /stand vxfs tranflush 0 1
/dev/vg00/lvol5 /tmp vxfs delaylog 0 2
/dev/vg00/lvol4 /opt vxfs delaylog 0 2
/dev/vg00/lvol6 /usr vxfs delaylog 0 2
/dev/vg00/lvol7 /var vxfs delaylog 0 2
/dev/vg00/lvol8 /home vxfs delaylog 0 2
/dev/vg01/lvol /users vxfs defaults 0 2
/dev/vg01/lvol2 /usr/sap/DEV vxfs defaults 0 2
/dev/vg01/lvol3 /sapmnt/DEV vxfs defaults 0 2
/dev/vg01/lvol4 /oracle vxfs defaults 0 2
172.16.4.20:/printerPRD /printers nfs defaults 0 0
10.31.76.20:/sapmnt/DEV/global /sapmnt/DEV/global nfs rw,soft,bg,intr 0 0
10.31.76.20:/sapmnt/DEV/profile /sapmnt/DEV/profile nfs rw,soft,bg,intr 0 0
10.31.76.20:/users/avp /users/avp nfs rw,soft,bg,intr 0 0
10.31.76.19:/routes /routes nfs rw,soft,bg,intr 0 0
#10.31.76.19:/sapcd /sapcd nfs rw,soft,bg,intr 0 0
tachyon6:/sapcd /sapcd nfs defaults 0 0

10.31.76.19:/usr/etc/sudo /usr/etc/sudo nfs rw,soft,bg,intr 0 0



(7) # mount -aF nfs

nfs mount: 10.31.76.19:/usr/etc/sudo: Permission denied
mount: 10.31.76.19:/sapcd is already mounted on /sapcd
nfs mount: 10.31.76.19:/routes: Permission denied
mount: 10.31.76.20:/users/avp is already mounted on /users/avp
mount: 10.31.76.20:/sapmnt/DEV/profile is already mounted on /sapmnt/DEV/profile
mount: 10.31.76.20:/sapmnt/DEV/global is already mounted on /sapmnt/DEV/global
NFS server 172.16.4.25 not responding still trying

While below mwntion FS are already removed from NFS server.

:/usr/etc/sudo
/routes
:/users/avp
:/sapmnt/DEV/profile
sapmnt/DEV/global

Still issue with bdf , mount -a ...

Subodh

[singh sanjeev]

please hashed the entry of nfs file system which has been removed ,,,from /etc/fstab :

system is sending query for those FS

[Matti_Kurkela]

1.)

> # umount /sapcd
> nfs umount: nfs_unmount: /sapcd: is busy
> umount: return error 1.

> # fuser -cu /sapcd
> /sapcd: 16406c(root) 13087c(root)

"/sapcd: is busy" means there are still processes using the /sapcd filesystem on the client, so the OS thinks it's a bad idea to yank the filesystem out from under them.

Your fuser command just told you the PIDs of those processes. If you want to know more about them, run "ps -fp 16406,13087".

The letter "c" after the PID in the fuser output reveals that these processes have their current working directory set to somewhere within the /sapcd filesystem (i.e. these processes have executed the equivalent of "cd /sapcd" or "cd /sapcd/some/directory").

If these are shell sessions and you can still access them, simply use the command "cd /" on all of them before trying to unmount /sapcd.

If these sessions are hanging, try killing them with "kill -HUP". If that does not help, you may have to try "kill -KILL" on them.

In HP-UX 11.31, you could also use "umount -f /sapcd" to forcibly unmount the NFS filesystem even if it's still in use. If any processes on the client have unsaved data for that filesystem, that data will be lost.

2, 3, 4) thanks for information, this makes the situation clearer. Looks OK so far...

5.)
> even data are accessible

Perhaps some data is still in the local cache on the client, causing the illusion.

> But on bdf /sapcd

> mount: 10.31.76.19:/sapcd is already mounted on /sapcd

Do you mean that the bdf command actually tries to mount /sapcd??? This should not be happening, unless the standard HP-UX bdf command has been replaced with something else. Or did you just make a mistake in copy/pasting information to the forum?

6. and 7.)
When you run "mount -aF nfs", you're telling the system: "try to mount all NFS filesystems that are configured in /etc/fstab". So it does exactly what you asked.

As singh sanjeev said, you should comment out or remove from /etc/fstab the lines that refer to NFS filesystems that have been removed.

> NFS server 172.16.4.25 not responding still trying

This error message says that this client has problems communicating with the NFS server at IP address 172.16.4.25.

Try "rpcinfo -p 172.16.4.25" and "showmount -e 172.16.4.25" on this client host. Can the client get a list of RPC services from the server? Can the client communicate with the server to get a list of exported NFS filesystems?

MK

[subodhbagade]

Thanks for your inputs,

I have commented the entry in fstab of NFS client which are removed from NFS server, tested few things here

(1) fuser â cu /sapcd â ¦ No process was there
I have unmouted the /spcd
Stop the nfs.client , nfs.core on client
Restarted nfs.client , nfs.core

# Mount tachyon6:/sapcd /sapcd

o/p
mount: tachyon6:/sapcd is already mounted on /sapcd

I can see

Df â k /sapcd â ¦.. it is showing there but

Bdf /sapcd hang again

(2) #mount â v

tachyon6:/sapcd on /sapcd type nfs rsize=32768,wsize=32768,NFSv3,dev=400001f on Wed May 26 21:19:07 2010

# showmount â a ( no o/p)

(2) rpcinfo -p 172.16.4.25 ( this server even not pinging from client but not able mount

# Mount tachyon6:/sapcd /sapcd ( this is diff NFS server and FS are mounted on NFS server )


Please let me know why I am not able to mount / sapcd on client and bdf / df -k hang here â ¦.

[Dave Olker]

Something is not adding up.

> I have unmouted the /spcd
> # Mount tachyon6:/sapcd /sapcd
> mount: tachyon6:/sapcd is already mounted
> on /sapcd

If you've successfully unmounted /sapcd then why is the system reporting the filesystem still mounted?

The first thing we should do is get /sapcd unmounted from this client. What OS is the client running?

[subodhbagade]

NFS client is having 11.31 version same as server

I have unmounted the /sapcd successfully first and after that when i am trying to mount it again in that case it shows

mount: tachyon6:/sapcd is already mounted on /sapcd


df -k /sapcd show the the /sapcd but bdf -t nfs , df -k hang .. dnt know why ??


Do we really need to reboot the client to make this functional ..not sure but that would b d last option i blv...


Subodh

[Suraj K Sankari]

Hi,
Reboot is the last option.....

Suraj

[singh sanjeev]

i think you should share your File system from NFS server not via /etc/dfs/dfstab but via /etc/dfs/sharetab

#hash the entry in /etc/dfs/dfstab and add entry in /etc/dfs/sharetab


share the NFS file system via commandline manually ;

share -F nfs -o rw=graviton:quark:lepton6:neutron7:omicron /sapcd

if this work then same entry should be made in /etc/dfs/sharetab

and run shareall

before all above unmount the /sapcd from clients.

hope this work.

[subodhbagade]

I am having entry in /etc/dfs/dfstab on NFS server for many NFS client as

/usr/sbin/share -F nfs -o rw=graviton:quark:lepton6:neutron7:omicron,ro= -d "none" /sapcd -


/sapcd is properly mounted on other server but dont know why then it is not for omicron server

If it is then will it be work if i make in entry in /etc/dfs/sharetab ??

i have tested as you suggested but getting this error on NFS server now


share_nfs: share complete, however, failed to remove old sharetab entry, corrupt sharetab file


Still not able mount /sapcd on NFS client omicron ... bdf, df -k hang ..


Subodh.

[singh sanjeev]

please send us below output :

# more /etc/dfs/dfstab
#more /etc/dfs/sharetab

on client
showmount -e

If it is then will it be work if i make in entry in /etc/dfs/sharetab ??
answer :

i mean to share do not put nay entry related to sapcd in any dfstab or sharetab files..first we try to do it manually from commandline....if you succeed in these..then edit the sharetab file and run shareall to activate the shares...

in short instead of dfstab file you should use sharetab to export the filesystem from nfs server ....

test it and let us know the result...

[Dave Olker]

> I have unmounted the /sapcd successfully
> first and after that when i am trying to
> mount it again in that case it shows
>
> mount: tachyon6:/sapcd is already mounted
> on /sapcd

Are you sure the filesystem is actually unmounting? I'd like to see the following:

# cat /etc/mnttab
# umount /sapcd
# cat /etc/mnttab

That will show me if the system really is unmounting the filesystem. If it is unmounting and you're not able to re-mount it then this is likely something confused at the VFS layer. Only way to clear that up would be a reboot, but let's first verify whether the system thinks it's unmounting the filesystem or not.

Dave

[Laurent Menase]

you should try a "ping 172.16.4.25"

if answers it probably means that rpcbind is dead.

if not, connectivity pb to 172.16.4.25
check your network.