NFS shared mount point "ll " "ls" long wait

[Gary L]

Hi

There is a NFS shared mount point /repout on our rp4440 client server. The NFS server is a openvms box. Today, when I run ll and ls on /repout, LONG WAIT. Avg, I need took 3 min waiting, the file content could be listed. Openvms admin had checked ,there is no problems on its side. I have checked rp4440 box disk i/o and other parameters of performance, it was no different?
What's going on?
BTW,
I could not umount the NFS shared mount point, erro msg "Device busy"

How to fix it?
Any answers will be very appreaciate!!!

[A. Clay Stephenson · Accepted solution]

If you could not umount the file system then obviously some process is accessing files on the NFS mountpoint or it's CWD is in that file tree. If you have lsof insytalled, use it to find the process(es); if not, use fuser though it is a distant second choice. What you could be seeing is a process heavily using this filesystem and that is your fundamental problem. You can also use nfsstat to help diagnose your problems and to see if the file system is under heavy use.

[Gary L]

Hi SEP

After I run " fuser -cku /repout ", I could umount the NFS mount point /repout.
But after I re-mount it again, it is still LONG WAIT 3 min when i run ll /repout.

-Gary

[Aussan]

when you run the ll on /repout does it also show high cpu usage ?

run ll on the nfs directory and on a second window do top to see

[A. Clay Stephenson]

I'm not SEP but since you were able to umount, that indicates that no process is hammering NFS. The next thing that I would check is speed/duplex settings between your network card and it's corresponding switch port and also do the same on the VMS box. Somewhat surprisingly, a mismatched duplex setting will almost work well and would not be noticeable in a telnet session for example but would crawl in an NFS transfer or FTP session. I would also check hostname resolution on both ends of the connection.

If these don't help then it's time to use nfssat -c.

[Gary L]

Hi A. Clay Stephenson

I'm so so sorry about it. As you know, you and SEP have the same top level: 2-star Olympian, the graphic same.
Would you please tell me how to check the duplex setting on HP-UX, I just know how to check and tuning on redhat Linux:
# ethtool eth0
#ethtool -s eth0 speed 100 duplex full autoneg off
#mii-tool -F 100baseTx-FD eth1

thanks

[Gary L]

Hi Aussan

See you again.
Yes, the CPU usage is the top one when I run ll /repout, details are as follows:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.04 0.0% 0.0% 1.7% 98.3% 0.0% 0.0% 0.0% 0.0%
1 0.00 0.9% 0.0% 1.3% 97.8% 0.0% 0.0% 0.0% 0.0%
2 0.01 2.4% 0.2% 1.3% 96.1% 0.0% 0.0% 0.0% 0.0%
3 0.08 1.5% 0.0% 2.0% 96.5% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.03 1.3% 0.0% 1.5% 97.2% 0.0% 0.0% 0.0% 0.0%

Memory: 8923288K (5684640K) real, 9909344K (6447340K) virtual, 1519596K free Page# 1/14

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 pts/ta 25679 root 154 20 2980K 1312K sleep 0:01 1.59 1.57 ll
3 ? 1874 root 152 20 270M 51532K run 762:14 1.05 1.05 java

[A. Clay Stephenson]

Lanadmin is the command you need. You can run it interactively by simply executing "lanadmin" or you can also use the command line interface. Man lanadmin for details. Bear in mind that just check the NIC's is not enough; you must also check the matching port on your network switch.

You can also get an idea of a network problem by FTP'ing from one of these boxes to the other. Transfer a 1MB file or so; is the transfer rate about half of the raw network speed? (e.g. a 100Mbit/sec connection should be able to get or put about 5-6MByte/sec.)

[Gary L]

Hi Clay

I'll check them accourding as you suggestions.
BTW,
Although "ll /repout" long wait but after 3 min, all the files could be list well.


-Gary

[Aussan]

Clay has a good point

check the duplex setting you have

if they are not set properly they could cause the issue

[Gary L]

Thanks guys!

[Gary L]

Hi there

Below are the duplex and speed setting informations of HP-UX client and OpenVms server:

HP-UX:
======
#lanadmin -x 0
Speed = 100 Full-Duplex.
Autonegotiation = Off.

OpenVMS:
========
Device Characteristics EWB0:
Value Characteristic
----- --------------
Normal Controller mode
External Internal loopback mode
CSMA/CD Communication medium
64 Minimum receive buffers
128 Maximum receive buffers
Yes Full duplex enable
Yes Full duplex operational
TwistedPair Line media
100 Line speed (megabits/second)

[Gary L]

# nfsstat -cn

Client nfs:
calls badcalls clgets
3381654 53 3381654
cltoomany
0
Version 2: (3302668 calls)
null getattr setattr
0 0% 2910188 88% 2696 0%
root lookup readlink
0 0% 258212 7% 0 0%
read wrcache write
23245 0% 0 0% 1348 0%
create remove rename
1349 0% 1239 0% 1 0%
link symlink mkdir
0 0% 0 0% 0 0%
rmdir readdir statfs
0 0% 55125 1% 49265 1%
Version 3: (78986 calls)
null getattr setattr
0 0% 10395 13% 107 0%
lookup access readlink
676 0% 358 0% 84 0%
read write create
3 0% 54446 68% 112 0%
mkdir symlink mknod
24 0% 29 0% 0 0%
remove rmdir rename
15 0% 0 0% 0 0%
link readdir readdir+
0 0% 0 0% 270 0%
fsstat fsinfo pathconf
12099 15% 29 0% 4 0%
commit
335 0%

[A. Clay Stephenson]

At this point you have done half the checking for speed and duplex settings. You now need to check those corresponding on the switch ports. For example, your HP-UX box is hard-set to 100FD; the corresponding switch port must be hard-set to 100FD; both end MUST be hard set or both ends MUST be set to auto-negotiate. You cannot zig on one end and zag on the other.

[Gary L]

Thanks Clay

Thanks for your fast reply, one more quesiton: How to check the duplex setting of corresponding switch port?

thanks a lot

[A. Clay Stephenson]

That is highly vendor-specific. Go ask your Network Admin.

[Gary L]

sure!