NFS and buffer cache

[Manfred Ferstl]

I have a proces that can run on both the nfs server and the nfs client.

The NFS server has a filesystem mounted from a local SAN disk and the nfs client has the same filesystem over NFS.

The process is indexing a large file (1GB) and createing an index file of 3.6 GB.

Whe the process is running on the NFS server, all the reads on the file are from the buffer cache, there are no physical reads from the disk.

When the process is running on the NFS client, all reads are "Remote Logical Reads" (seen in Glance). So all reads are going over the net. On the NFS server there are only logical reads, so no physical reads are happenening.

But why is the data no cached in the local buffer cache on the NFS client?

[Steven E. Protter]

Shalom,

Buffer cache is working only for local disk. What should actually happen is that the read through the net from the client, should get its data from the buffer cache of the server.

NFS client reads from the network. The buffer cache on the client caches disk, not network. Its pretty much the way nfs and buffer cache were designed.

If your network is not congested you should not experience a slowdown. The read on the server is from buffer cache and should be very fast.

SEP

[Dennis Handly]

Most likely because there is nothing in NFS that tells the buffer cache that the file has been changed? Perhaps if the file was mounted readonly? But that doesn't prevent the file from being changed on the server.

If performance is important, perhaps you should copy the whole file (periodically) to your client and then index it?

[Manfred Ferstl]

Hello,

i suspected the same but did not know for sure.
So there is no way to tell the client that it should not read the file over an over again from the net.

Well, the workarounds are obvious, but i cannot change the process. I'm not the process owner.

[Dave Olker]

Hi Manfred,

By default, if your NFS client system is reading data from the NFS server it will stage it in the client's buffer cache.

You can verify this yourself with a simple test. Find a small file (1-2 MB) in the NFS filesystem and use the following command to read it on the client:

# cat /nfs/file > /dev/null

Then zero out your nfsstat counters:

# nfsstat -z

Then read the same file again:

# cat /nfs/file > /dev/null

Then check your client-side nfsstat counters and see how many NFS read requests were used to read the file the second time. My guess is you'll find the client didn't issue a single NFS read the second time. This proves the client is caching the file in it's buffer cache.

Please try this simple test and let me know if it works. This will at least tell me if your NFS client is caching data under "normal" circumstances. Once we know this we can get to the bottom of why you're not seeing cached behavior when your specific application runs.

Regards,

Dave

[Manfred Ferstl]

Hi Dave,

i only had the possibility to perform your test now. The system was too busy before.

The test shows: NFS files are cached locally, they are normally not read again.

So what is different during the processing?
I noticed, that the file that gets indexed also changes its modification time during the proccess. The size does not change. So there are also NFS writes on this file.

Could it be, that modifiying a few buffers of the file invalidates all buffers in the cache, so that the entire file is read again from the server?

[Dave Olker]

Hi Manfred,

Now that we know your client is using buffer cache in a simple case, there are several reasons why it might not use buffer cache when your application runs.

Here are a few common ones:

1) Your application locks the file being read/written to

2) Your application uses mmap() to read/write the file

3) The buffer cache on your client is sized too small and eventually is exhausted and not used (This is why I had you test with a small file to make sure it would fit in your client's buffer cache).

From what you know about your application, do you know if it is doing any of these things? Does it lock the data file? Does it use mmap() against the data file? How large is your buffer cache sized on your NFS client vs. the physical memory in the client?

Regards,

Dave

[Manfred Ferstl]

Hi Dave,

well the Appliation is SAS from SAS Institute.

1. locks : I do not know how can I find this out?

2. mmap: I do not think so, lsof shows the filedescriptors for the data files as type REG, while other files (SAS libraries) are of type mem.

3. cache size: The cache should be large enough, it is 5.6GB. When i cat a file of 1 GB to /dev/null two times, the second cat does not generate any NFS reads.

So locking remains as an option.

Has this caching behaviour changed recently? We have recently replaced the NFS server by a newer machine, we suspect that the old NFS server (with older patches) did behave differently.
The client is the same.

[Dennis Handly]

>1. locks: I do not know how can I find this out?
>2. mmap: I do not think so, lsof shows the filedescriptors for the data files as type REG,

tusc should help. If you mmap a file, I don't think it has to stay open. But lsof may find some type of handle the kernel needs to keep track.