Any trick to determine heavy-io file on a filesystem that is being shared out via NFS?

[Adam Garsha]

We are having sporadic NFS problems with our HPC cluster, running RHEL 5.3. The issues appear to be centric to high-io wait, either at storage layer and/or network interface starvation. We are looking into storage/network optimizations with our hardware vendor, but we also seek any information that you may provide related to the following questions:

1.) Is there any recipe/method that you can think of to determine which files among a filesystem are recieving heavy read or write activity? Namely, if our NFS server is using ext3 for a shared filesystem, is there any tool or trick to determine which files are being written at a given time? Motivation: I am trying to figure out what is causing the heavy i/o spikes from the perspective of the NFS server and I can't figure out a way to narrow down the issue to a specific user nor file.

2.) Do you have any whitepapers or conference talks that speak of NFS optimizations? Any specific to HPC?

[Ivan Ferreira · Accepted solution]

1) I think that iotop can help you http://guichaz.free.fr/iotop/

2) There are a lot of documents, this for example is for HP-UX, but the concepts can be applied to any Unix/Linux platform

http://docs.hp.com/en/1435/NFSPerformanceTuninginHP-UX11.0and11iSystems.pdf

[Adam Garsha]

I'll swing around and give points later. Thanks. I'll add a question/clarification. Again the FS is shared out via NFS, so it wouldn't be a local process causing I/O, but instead nfs daemons doing all the work. I am wondering if there is a way to ask NFS and/or the FS, if it can tell me what files are being hit on the (ext3) FS. Maybe that is not possible so I'll add a question 3:

3.) It there a way to determine which tcp socket is causing the most network traffic? Hmm, maybe with netstat... I'll look, but am open to recipes. Motivation: if I can't tell from NFS server perspective, maybe I can use iotop on the client side, but to do so, I'll need to know which client or clients are the heavy hitters...

[Adam Garsha]

I'll answer my own question number 3 (but provide more information if you've got a better method). It looks like iptraf can give me a top-like view on socket activity so I could/can determine heavy hitting clients from the NFS server's perspective.

[Ivan Ferreira]

I'll swing around and give points later. Thanks. I'll add a question/clarification. >> Again the FS is shared out via NFS, so it wouldn't be a local process causing I/O, but instead nfs daemons doing all the work. I am wondering if there is a way to ask NFS and/or the FS, if it can tell me what files are being hit on the (ext3) FS. Maybe that is not possible so I'll add a question 3:

Yes, but you can use lsof for example to the pid that causes most I/O to identify the file is accessed.

3.) It there a way to determine which tcp socket is causing the most network traffic? Hmm, maybe with netstat... I'll look, but am open to recipes. Motivation: if I can't tell from NFS server perspective, maybe I can use iotop on the client side, but to do so, I'll need to know which client or clients are the heavy hitters..

Probably iptraf can help you.

[Adam Garsha]

using iptraf from nfs server, I can determine heavy-hitter client; after, I can use iotop on the heavy-hitter client(s) to acertain more granular source of i/o contention.