NFS data corruption issue 11.31 Itanium

[Patrick Briggs]

So far in my testing here, it took about 60 seconds before the contents listed on the screen using a different 11.31 server than the one in discussed here. I don't know if 60 seconds is normal for a directory with 27K items in it, but our programmers seem to think it takes longer than it used to on the 11.11 PA-RISC servers.

It seems to take even more time if you do a "ll *" wildcard type search than if you do a plain ll without anything else on the command line. Once the command finally returns something, subsequent runs are fast for a while until it hasn't been done in some time, then it's back to slow for the 1st attempt.

All of our 11.31 servers have the same configuration as far as number of CPU's, memory, and OS patch level, with the exception of the ONC version.

Another question I have, the ONC version never cropped up when I ran a patch analysis on HP's ITRC website. Should ONC patches be covered as part of the normal patch bundle, or do I need to periodically check the web URL for the ONC package to see if there are updates and apply those instead?

[Dennis Handly]

>the ONC version never cropped up when I ran a patch analysis on HP's ITRC website. Should ONC patches be covered as part of the normal patch bundle

This is a new ONC version, not a patch, so it doesn't show up with a patch analysis. (Similar to new compiler versions.)

>do I need to periodically check the web URL for the ONC package to see if there are updates and apply those instead?

Yes, for now. I don't know if Bob has plans for swa and non-patches.

[Patrick Briggs]

Is NFS version 4 supported with this ONC and would it offer any performance benefits to my NetApp if I tried to use it?

[Dave Olker]

I took my 11.31 system running ONCplus B.11.31.08 and mounted a filesystem from my NetApp filer (F825c running OnTAP 7.1.3) and created two different directory structures: one with 27 sub-directories each with 1,000 files, and the other containing 27,000 files in a single directory.

I mounted the filesystem using default options:

# nfsstat -m
/netapp-1 from atcfiler1:/vol/vol1
Flags: vers=3,proto=tcp,sec=sys,hard,intr,link,symlink,acl,devs,rsize=32768,wsize=32768,retrans=5,timeo=600
Attr cache: acregmin=3,acregmax=60,acdirmin=30,acdirmax=60


I then timed how long it takes to use "ll" in each of these directory structures:

# timex ll -R 1000dirs | wc -l

real 1.48
user 0.25
sys 0.58

27107

# timex ll -R 27000dirs | wc -l

real 1.44
user 0.26
sys 0.41

27001


In both cases it only took 1.4 seconds to complete the operation. I'm not displaying the contents, merely passing the results to "wc -l" so there is no lag time waiting to display all 27,000 files on my screen.

So I agree, 60 seconds sounds like a long time to return 27,000 directory entries, but in my tests I'm getting very good response times so this doesn't appear to be a class problem with all 11.31 systems.


As for NFS v4, yes it is available on 11.31 systems. You can certainly try it and see if it provides better performance. I tried it on my systems and here are the numbers I got with NFS v4:

# timex ll -R 1000dirs | wc -l

real 15.76
user 0.25
sys 4.91

27107

# timex ll -R 27000dirs | wc -l

real 27.48
user 0.27
sys 0.66

27001


Looks like things take a lot longer with NFS v4 on my systems. Your mileage may vary.

Regards,

Dave

[Patrick Briggs]

Here is my mount options:

/gp05 from tec-storage2.tec.clinitech.net:/vol/ops/gp01
Flags: vers=3,proto=tcp,sec=sys,hard,intr,link,symlink,acl,devs,rsize=32768,wsize=32768,retrans=5,timeo=600
Attr cache: acregmin=3,acregmax=60,acdirmin=30,acdirmax=60

timex ll | wc -l

real 1:12.72
user 0.27
sys 0.34

28177

Also, I'm seeing this message in the syslog every once in a while...

Synchronous Page I/O error occurred while paging to/from NFS server tec-storage2.tec.clinitech.net
file system is /gp05

What could this error indicate? What sorts of things should I check?

[Dave Olker]

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&objectID=c01476881&prodTypeId=18964&prodSeriesId=3553037

[Ashish_33]

Patrick, Can you please let me know if the NFS performance problem you were facing has been resolved.
I also have an RX6600 server with HPUX 11.31 and i am also facing the NFS performance issue and the call is with HP since December 2009 and HP has still not been able to resolve it.

Thanks.

[Patrick Briggs]

After installing the ONCplus B.11.31.08 package, the major issues I had seemed to be solved. The only issue I've had lately is doing a LL command on a directory with a lot of files seems to take a really long time. One possible fix is to turn on bigendian hash directory on our NetApp NFS option, but it will cause all the mount points to go stale. I don't know if I want to do that right away. I'd have to find a time when I can do it when nothing is running on the system which is becoming more difficult with 24x7 shop.

[Dave Olker]

Hi Patrick,

An interesting test for your long "ll" times would be to mount the filesystem with the "readdir" option:

readdir Disable the READDIRPLUS functionality, which is
used by default on an NFS Version 3 mount point,
and use the NFS Version 2 READDIR functionality
instead. The performance of applications that
read huge directories over NFS will vary between
NFS Version 2 and NFS Version 3 depending on the
type of information that the applications need.
The find command will be faster using NFS Version
3 READDIRPLUS while the ls command will be faster
using NFS Version 2 READDIR. The readdir option
must be used on a case by case basis depending
upon the application. There is no effect on an
NFS Version 2 mount point.


If you can, give that a try and see if the ll times change for the better.

Regards,

Dave

[Ashish_33]

Hi Patrick/Dave,

Thanks for your reply.
My problem was resolved by using the following
I exported the filesystem on the NFS server with the ASYNC option.
On the NFS client i set the ketnel parameter nfs_enable_write_behind=1
and mounted the filesystem on the client side using the forcedirectio option.
I short i used the ASUNC mode on both the Server and the Client.