NFS Not responding

[Niraj Kumar Verma]

Hi,

I am getting a constant error of NFS not respondig the NFS server is EMC filer. The HP-UX OS is 11.11

any clue ...

# lanadmin
PPA Number = 0
Description = lan0 HP PCI 10/100Base-TX Core [100BASE-TX,FD,MANUAL,TT=1500]
Type (value) = ethernet-csmacd(6)
MTU Size = 1500
Speed = 100000000
Station Address = 0x306e488f6b
Administration Status (value) = up(1)
Operation Status (value) = up(1)
Last Change = 3823
Inbound Octets = 3667894503
Inbound Unicast Packets = 26185349
Inbound Non-Unicast Packets = 196983
Inbound Discards = 0
Inbound Errors = 47
Inbound Unknown Protocols = 8288
Outbound Octets = 3391045497
Outbound Unicast Packets = 26521762
Outbound Non-Unicast Packets = 37
Outbound Discards = 0
Outbound Errors = 0
Outbound Queue Length = 22
Specific = 655367

# nfststat

Server rpc:
Connection oriented:
calls badcalls nullrecv
1788 0 0
badlen xdrcall dupchecks
0 0 0
dupreqs
0
Connectionless oriented:
calls badcalls nullrecv
618 0 0
badlen xdrcall dupchecks
0 0 0
dupreqs
0

Server nfs:
calls badcalls
2406 0
Version 2: (0 calls)
null getattr setattr
0 0% 0 0% 0 0%
root lookup readlink
0 0% 0 0% 0 0%
read wrcache write
0 0% 0 0% 0 0%
create remove rename
0 0% 0 0% 0 0%
link symlink mkdir
0 0% 0 0% 0 0%
rmdir readdir statfs
0 0% 0 0% 0 0%
Version 3: (2406 calls)
null getattr setattr
618 25% 542 22% 0 0%
lookup access readlink
342 14% 183 7% 0 0%
read write create
4 0% 0 0% 0 0%
mkdir symlink mknod
0 0% 0 0% 0 0%
remove rmdir rename
0 0% 0 0% 0 0%
link readdir readdir+
0 0% 0 0% 0 0%
fsstat fsinfo pathconf
717 29% 0 0% 0 0%
commit
0 0%

Client rpc:
Connection oriented:
calls badcalls badxids
3713189 19831 4
timeouts newcreds badverfs
19812 0 0
timers cantconn nomem
0 0 0
interrupts
19
Connectionless oriented:
calls badcalls retrans
2469 0 0
badxids timeouts waits
0 0 0
newcreds badverfs timers
0 0 0
toobig nomem cantsend
0 0 0
bufulocks
0

Client nfs:
calls badcalls clgets
3693425 19 3693414
cltoomany
2653843
Version 2: (0 calls)
null getattr setattr
0 0% 0 0% 0 0%
root lookup readlink
0 0% 0 0% 0 0%
read wrcache write
0 0% 0 0% 0 0%
create remove rename
0 0% 0 0% 0 0%
link symlink mkdir
0 0% 0 0% 0 0%
rmdir readdir statfs
0 0% 0 0% 0 0%
Version 3: (3693426 calls)
null getattr setattr
0 0% 154537 4% 7364 0%
lookup access readlink
260104 7% 541023 14% 36 0%
read write create
2341391 63% 20012 0% 3861 0%
mkdir symlink mknod
182 0% 14 0% 0 0%
remove rmdir rename
717 0% 0 0% 111 0%
link readdir readdir+
32 0% 112 0% 321833 8%
fsstat fsinfo pathconf
32291 0% 112 0% 32 0%
commit
9662 0%

/etc/mnttab is clean no issues

Regards
-Niraj

[RAC_1]

Check if you can successfully ping/nslookup nfs server.
ping nfs_server
nslookup nfs_server_ip
nslookup nfs_server_name

Any more messages in dmesg, syslog.log??

[Niraj Kumar Verma]

Error:
-----
NFS server btc3x002 not responding still trying
NFS server btc3x002 not responding still trying


nslookup works fine

there is no errors in syslog

[Niraj Kumar Verma]

Could It be any patch problem ?

Regards

[Arunvijai_4]

Can you able to login to NFS server and check everything is up and running ? Are you able to mount from other client machines ?

-Arun

[Niraj Kumar Verma]

I tried mounting a exported file system from filer it seems to work as normal no issue ..

This particular HP server runs informix db and Java. is there any specific stuff required to run java on HP over NFS

Regards

[Arunvijai_4]

Java and NFS are unrelated, i dont think any Java process should be running for NFS. Can you check the patch level on your 11.11?

-Arun

[Niraj Kumar Verma]

root@ingbtc02$ uname -a
HP-UX ingbtc02 B.11.11 U 9000/785 2011991172 unlimited-user license

[Niraj Kumar Verma]

root@ingbtc02$ swlist -l file |grep nNFS server btc3x002 not responding still trying
fs
NFS.NFS-CLIENT: /sbin/init.d/nfs.client
NFS.NFS-CLIENT: /sbin/rc1.d/K570nfs.client
NFS.NFS-CLIENT: /sbin/rc2.d/S430nfs.client
NFS.NFS-CORE: /etc/rc.config.d/nfsconf
NFS.NFS-CORE: /sbin/fs/nfs
NFS.NFS-CORE: /sbin/fs/nfs/mount
NFS.NFS-CORE: /sbin/fs/nfs/umount
NFS.NFS-CORE: /sbin/init.d/nfs.core
NFS.NFS-CORE: /sbin/rc1.d/K600nfs.core
NFS.NFS-CORE: /sbin/rc2.d/S400nfs.core
NFS.NFS-CORE: /usr/bin/nfsstat
NFS.NFS-CORE: /usr/newconfig/etc/rc.config.d/nfsconf
NFS.NFS-CORE: /usr/sbin/nfsconfcol
NFS.NFS-CORE: /usr/sbin/nfskd
NFS.NFS-ENG-A-MAN: /usr/share/man/man1m.Z/nfsstat.1m
NFS.NFS-ENG-A-MAN: /usr/share/man/man1m.Z/rpc.pcnfsd.1m
NFS.NFS-ENG-A-MAN: /usr/share/man/man7.Z/nfs.7
NFS.NFS-ENG-A-MAN: /usr/share/man/man1m.Z/nfsd.1m
NFS.NFS-ENG-A-MAN: /usr/share/man/man1m.Z/pcnfsd.1m
NFS.NFS-JPN-E-MAN: /usr/share/man/ja_JP.eucJP/man1m.Z/nfsstat.1m
NFS.NFS-JPN-E-MAN: /usr/share/man/ja_JP.eucJP/man1m.Z/pcnfsd.1m
NFS.NFS-JPN-E-MAN: /usr/share/man/ja_JP.eucJP/man7.Z/nfs.7
NFS.NFS-JPN-E-MAN: /usr/share/man/ja_JP.eucJP/man1m.Z/nfsd.1m
NFS.NFS-JPN-E-MAN: /usr/share/man/ja_JP.eucJP/man1m.Z/rpc.pcnfsd.1m
NFS.NFS-JPN-S-MAN: /usr/share/man/ja_JP.SJIS/man1m.Z/nfsstat.1m
NFS.NFS-JPN-S-MAN: /usr/share/man/ja_JP.SJIS/man1m.Z/pcnfsd.1m
NFS.NFS-JPN-S-MAN: /usr/share/man/ja_JP.SJIS/man7.Z/nfs.7
NFS.NFS-JPN-S-MAN: /usr/share/man/ja_JP.SJIS/man1m.Z/nfsd.1m
NFS.NFS-JPN-S-MAN: /usr/share/man/ja_JP.SJIS/man1m.Z/rpc.pcnfsd.1m
NFS.NFS-KRN: /usr/conf/lib/libnfs.a
NFS.NFS-KRN: /usr/conf/master.d/nfs
NFS.NFS-PRG: /usr/include/nfs
NFS.NFS-PRG: /usr/include/nfs/nfs.h
NFS.NFS-SERVER: /sbin/init.d/nfs.server
NFS.NFS-SERVER: /sbin/rc2.d/K900nfs.server
NFS.NFS-SERVER: /sbin/rc3.d/S100nfs.server
NFS.NFS-SERVER: /usr/sbin/nfsd
NFS.NFS-SERVER: /usr/sbin/rpc.pcnfsd
OS-Core.CORE-ENG-A-MAN: /usr/share/man/man1m.Z/df_nfs.1m
OS-Core.CORE-ENG-A-MAN: /usr/share/man/man1m.Z/mount_nfs.1m
OS-Core.CORE-ENG-A-MAN: /usr/share/man/man1m.Z/umount_nfs.1m
OS-Core.CORE-JPN-E-MAN: /usr/share/man/ja_JP.eucJP/man1m.Z/mount_nfs.1m
OS-Core.CORE-JPN-E-MAN: /usr/share/man/ja_JP.eucJP/man1m.Z/df_nfs.1m
OS-Core.CORE-JPN-E-MAN: /usr/share/man/ja_JP.eucJP/man1m.Z/umount_nfs.1m
OS-Core.CORE-JPN-S-MAN: /usr/share/man/ja_JP.SJIS/man1m.Z/mount_nfs.1m
OS-Core.CORE-JPN-S-MAN: /usr/share/man/ja_JP.SJIS/man1m.Z/df_nfs.1m
OS-Core.CORE-JPN-S-MAN: /usr/share/man/ja_JP.SJIS/man1m.Z/umount_nfs.1m
OS-Core.CORE-KRN: /usr/conf/nfs
OS-Core.CORE-KRN: /usr/conf/nfs/fifo.h
OS-Core.CORE-KRN: /usr/conf/nfs/fifonode.h
OS-Core.CORE-KRN: /usr/conf/nfs/snode.h
OS-Core.CORE-KRN: /usr/conf/sys/nfs_conf.h
PHCO_29829.PAUX-ENG-A-MAN: /usr/share/man/man3.Z/confstr.3c
PHCO_30598.SAM: /usr/sam/lib/C/nfs_sa.ui
PHCO_30598.SAM: /usr/sam/lib/C/nfs_td.ui
# PHKL_25238 1.0 11.00 NFS nfsd deadlock
PHNE_30378.CORE-ENG-A-MAN: /usr/share/man/man1m.Z/mount_nfs.1m
PHNE_30378.CORE-ENG-A-MAN: /usr/share/man/man1m.Z/umount_nfs.1m
PHNE_30378.NFS-CLIENT: /sbin/init.d/nfs.client
PHNE_30378.NFS-CORE: /sbin/fs/nfs/mount
PHNE_30378.NFS-CORE: /sbin/fs/nfs/umount
PHNE_30378.NFS-CORE: /sbin/init.d/nfs.core
PHNE_30378.NFS-CORE: /usr/bin/nfsstat
PHNE_30378.NFS-CORE: /usr/newconfig/etc/rc.config.d/nfsconf
PHNE_30378.NFS-CORE: /usr/sbin/nfsconfcol
PHNE_30378.NFS-ENG-A-MAN: /usr/share/man/man1m.Z/nfsstat.1m
PHNE_30378.NFS-ENG-A-MAN: /usr/share/man/man1m.Z/nfsd.1m
PHNE_30378.NFS-KRN: /usr/conf/lib/libnfs.a
PHNE_30378.NFS-SERVER: /usr/sbin/nfsd
PHNE_30378.NFS-SERVER: /usr/sbin/rpc.pcnfsd
ProgSupport.C-INC: /usr/include/nfs
ProgSupport.C-INC: /usr/include/nfs/fifo.h
ProgSupport.C-INC: /usr/include/nfs/fifonode.h
ProgSupport.C-INC: /usr/include/nfs/snode.h
ProgSupport.PAUX-ENG-A-MAN: /usr/share/man/man3.Z/confstr.3c
ProgSupport.PAUX-JPN-E-MAN: /usr/share/man/ja_JP.eucJP/man3.Z/confstr.3c
ProgSupport.PAUX-JPN-S-MAN: /usr/share/man/ja_JP.SJIS/man3.Z/confstr.3c
SystemAdmin.SAM: /usr/sam/lib/C/nfs.ui
SystemAdmin.SAM: /usr/sam/lib/C/nfs_netg.ui
SystemAdmin.SAM: /usr/sam/lib/C/nfs_netg_sa.ui
SystemAdmin.SAM: /usr/sam/lib/C/nfs_sa.ui
SystemAdmin.SAM: /usr/sam/lib/C/nfs_td.ui
root@ingbtc02$ NFS server btc3x002 not re

[RAC_1]

Do you use manual NFS mount or automounter??
ping nfs_server
Does it return ok??

If you use automounter, send -usr2 signal to automounter and it will create a debug file in /var/adm. This should give you enough details to start troubleshooting.

[Niraj Kumar Verma]

unfortunately its manual mount using /etc/fstab

[RAC_1]

ping nfs_server

Does it return ok??

Can you mount this nfs share from some other client??

[Arunvijai_4]

Check the patch level of your server, Go to http://www1.itrc.hp.com/service/patch/search.do and type " NFS server" and check for the available patches, install the latest one.

-Arun

[KapilRaj · Accepted solution]

Niraj,

If you have this filesystem nfs mounted on another hpux box, it is worth checking whether u get the same error there as well. If u do not have this implemented yet , it is worth giving it a try atleast by a diffrent filesystem. How is your connections to the nfs server ? is it in the same subnet as the hpux box ? how is ur connections to the switch ( 1GB / FD etc ) , check whether it matches with the switch port settings.

Try ftp'ing a very large file over this interface and check if it gives an error in between , so here you would know whether it is NFS or your network who is troubling you.

Also check the network at the nfs server( i think it is a nas) whether the port related settings are fine or no. If you are performing heavy I/O in the nfs'd directory , then it is better to have them moved to a local filesystem as you will end up screwing up your network. NFS is a bad option for applications who write like hell.

Also check what nfs version does the server run. May be you will need to patch it to the latest (same applies to client as well).

Looking at the message (this is from experience , it may be wrong ) it looks like the problem is with your network or the nfs server (nas). But don;t take my word here, as i have not seen your network. Do your investigation and fix them. All the best to you to resolve this one. It takes a while to resolve such problems.

Regds,

Kaps

[Niraj Kumar Verma]

Hi Kapil,

Thanks for the descriptive info.

I have tried almost everything u suggested and finally restarted the NAS box. For time being it seems to be alright. I am observing it the now.


Regards
-Niraj