NFS IP over Infiniband issue

[Scott Giddens]

I tested the compatibility of HPUX 11i v2 NFS and WIndows (NFS) Services for UNIX in the lab and everything works great. But now in production I have a nastran application that attempts to read a file from an HPUX box to a Windows box. The nastran application hangs (no CPU load) and the wait state on the process is "Streams" accociated with NFS. Other applications seem to work fine with large files but I understand nastran "reads files backwards"

Does anyone have an idea what "Streams" are and what they have to do with NFS?

I am using Glance to monitor the performance but it just stops like it is waiting for me to answer a question or something. The process is a background process so it is not interative. Some things I have tried is to increase the number of nfsd and biod daemons on the client and the number of "worker processes" on the Windows server side.

This is an Infiniband fabric with IP over IB and the issue does not occur with a straight ethernet network the last time I tried it. As far as I can tell the Infiniband switch and HBCs work excellent together except for this one issue. The performance is great writing and transfering files over the fabric so I am a little confused.

The system is of course multihomed with the typical Infiniband approach with three network interfaces, one IB0IP, one GB LAN/WAN connection to users, and one GB backend maintance LAN for administration but has OpenFabrics Infiniband drivers on the Windows side of things. I have read that one of the reasons NFS fails is because of name resolution, not being able to reverse lookup name to IP and this may very well be the case on a multihomed system and an application that reads files backwards. I tried removing locking and dropped down to NFSv2 but still no luck.

So far my fstab for the NFS mount looks like this:

nas-srv1:/users /data nfs rsize=8192,wsize=8192,hard,nocto,rw,nosuid,vers=2 0 0

Any help would be much appreciated,
Scott

[Dave Olker]

Hi Scott,

I've never tried running NFS over IB so I don't have any specific experience to draw on, but my first suggestion would be to collect a trace of the NFS packets going across that link and see where it stops.

HP-UX uses a STREAMS-based networking stack, so Glance reporting that the process is blocked in STREAMS most likely means it's waiting on something from the network. The fact that it's making no progress, and likely taking no CPU cycles, tells me it's either sent a request and waiting for a response or received the response and doesn't know how to process it. That's why I'd be interested in a network trace.

I don't know if Wireshark can collect frames from an IP/IB link, but if so I'd suggest running Wireshark on both the HP-UX client and the Windows server and seeing if that tells you where the NFS traffic halted.

Aside from that you could try attaching to the blocked process with tusc and see if you can detect which system call it's waiting on. If the process is completely stopped you might have to try attaching to the process before it makes the NFS requests that cause it to block so you can see which system calls it was in the process of making when it got stopped.

Regards,

Dave

[Scott Giddens]

Interesting, does the kernel parameter 'streampipes' need to be enabled (streampipes=1) to use the hp streams-based network stack?

[Dave Olker]

Negative. STREAMS is the only option. It's enabled by default because...well...it's the only option. :)

Dave