NFS data corruption issue 11.31 Itanium

[Patrick Briggs]

We just started using a new Itanium RX6600 with HP-UX 11.31 installed on it over to a NetApp FAS-3050c NAS device. We have an NFS mounted directory which is acting very strange. After a while, you can't do wildcard file searches with the ls command anymore. So if you had a collection of files that started with tr*, doing an ls tr* results in nothing found, but if you do a ls on a specific file, it will show up just fine. Unmounting the NFS directory and remounting it again fixes the problem, but then the problem comes back after a while.

This is a really bad issue for us because we have scripts which rely on being able to use wildcards to cat files together. This really smacks as some kind of BUG with NFS of some kind. We did not have any of this with our old 11.11 system on PA-RISC, so I don't think it's anything to do with our network.

The NFS mount options I'm using in FSTAB is:
tec-storage2.tec.clinitech.net:/vol/ops/gp01 /gp05 nfs rw,hard,vers=3,proto=tcp 0 0

[Michael Steele_2]

what versions of nfs on netapps and servers?

[Patrick Briggs]

They support version 3 and version 4. Right now I have version 4 support turned off. I also discovered I may not have the latest ONCplus patch bundle installed. The version I show loaded is B.11.31.06, but the version I see out on the web link is B.11.31.08. The web URL is https://software.hp.com/portal/swdepot/displayProductInfo.do?productNumber=ONCplus

The description of a possible bug that might be affecting me is:

Directory related operations on NFS client with ONCplus B.11.31.06 or B.11.31.07 installed and with file system mounted with read/write size greater than 8192 bytes, may result in system panic or data corruption.

Unfortunately, I can't reboot the server right now during the production day to try it, so I'm going to copy the NFS directory to local drive for now to avoid the data corruption.

[Dave Olker]

Hi Patrick,

Yes, the 11.31.08 ONCplus bundle will likely fix your problem. I was involved with that issue when it was first reported and the systems we used to reproduce it in-house were NetApp filers.

Regards,

Dave

[Patrick Briggs]

I have installed the new 11.31.08 bundle on our machine. The next issue we're experiencing is the performance of doing a directory listing seems really slow. So you could do an ll command, go away and have a cup of coffee, and it still isn't finished by the time you get back to your desk. This is the complaint I'm getting from my programmers who's home directory is stored on our NetApp filer. Is there reason you can think of for why this is slow? The server has two GB NIC's tied together in link aggregation, so I doubt it's because of a busy network.

[Dave Olker]

Please clarify for me - when did this directory listing performance issue start? Has it always been there? Did it get worse after installing 11.31.08? Does every NFS client see the same performance? Only certain NFS clients see this problem?

Dave

[Patrick Briggs]

Dave,

The directory performance issue started when we went to the new Itanium server running 11.31, even before I installed the .08 version of ONC. My programmer says it used to take a few moment to list the contents of the directory on the old 11.11 system on PA-RISC, but it's significantly worse on this new system.

Just to give you an idea, there are approximately 27,000 files inside that source code directory. So you could make an argument that it has a ton of files, but it didn't used to be as much of a problem with the other 11.11 box we moved off of.

So to summarize, not worse after .08 was installed. Only seems to be this client and no others. I do have two other 11.31 Itanium systems we're moving to in the near future, but has .02 ONC on them now and I need to update them.

Also, another observation I've seen is once you've gotten through the pain of doing a LL command on the directory, subsequent runnings of the command will be faster, but I have to guess it is because the contents are being cached. It will be slow again after the cache has aged out.

Patrick

[Dave Olker]

One point of clarification:

> Only seems to be this client and no
> others. I do have two other 11.31 Itanium
> systems we're moving to in the near
> future, but has .02 ONC on them now and I
> need to update them.

Are you saying these other two 11.31 systems do not show this problem? Or do all the 11.31 systems behave the same?

[Patrick Briggs]

There hasn't been much activity on the two other machines yet to be certain. They do have a much older version of ONC installed, .02, so they're definately going ot have problems until I update it.

[Dave Olker]

I understand they'll hit the known data corruption issue until they're updated. I'm focusing on the performance issue now. Do those two systems see the same performance to retrieve the 27K directory listings as the 3rd 11i v3 system running 11.31.08 or are they the same speed as the 11.11 systems?

I'm trying to understand if this is a "class" problem with all 11.31 systems in your environment or only with one specific system.