topic Re: poor NFS performance when retrieving directories WITH attributes in Operating System - OpenVMS

poor NFS performance when retrieving directories WITH attributes

lxworks — Mon, 23 Jul 2007 08:05:57 GMT

hi, all.

this is the environment:
NFS Server is a NetApp FAS940 running DataOnTap 6.5.x. the NFS Client is an OpenVMS Alpha DS25 running OpenVMS 7.3-2, Patches not too old, TCPIP Version V5.4 - ECO 6 in a cluster with four other nodes, that may mount the same DNFS devices, but must not (I don´t know if NFS caching is clusterwide)

the DNFS is mounted as follows:

$ tcpip mount dnfs80: /host="mynetapp.mydomain.com" -
/path="/vol/vol2/vol3/vol4"/server_type=UNIX /CONVERT/ADF=CREATE -
/system /user=user /owner=[xxx,y] /uid=xxxx /gid=yyy

the share may contain up to 3000 directory entries in root level.

now the problem:
when issuing a "directory dnfs80:[000000]" without any qualifiers, the listing of these 3000 entries is very fast. (fresh mounted, I assume, there is no data in cache, but I´m not shure!?)

when adding some qualifiers as a directory /date/size/owner/prot, the same listing needs up to 20 Minutes (twenty!)

Any ideas or is this standard?

thanks&rgds
alex

Re: poor NFS performance when retrieving directories WITH attributes

Wim Van den Wyngaert — Mon, 23 Jul 2007 09:26:38 GMT

Is it also slow when you repeat the 2nd command again after it has completed ?

Wim

Re: poor NFS performance when retrieving directories WITH attributes

lxworks — Mon, 23 Jul 2007 09:32:36 GMT

the second run is faster sometimes until the point I reached with the first run, but the results are very different. In most cases a second complete listing is as slow as the first.

Re: poor NFS performance when retrieving directories WITH attributes

Hoff — Mon, 23 Jul 2007 09:35:50 GMT

This looks broken.

Confirm the NetApp FAS is current on its firmware, turn on whatever logging or sniffing might be available on either end in your network connection or via a protocol sniffer, confirm local client username process quotas are somewhere between adequate and generous, and then call HP support.

I'd also try the same sequence with something other than the NetApp FAS NFS server, to see if this is specific to the NetApp device or a connection with same, or if this is a generic problem with the NFS client.

If you have a testing environment, you could also see if one of the Process software IP stacks has a faster NFS.

Re: poor NFS performance when retrieving directories WITH attributes

lxworks — Mon, 23 Jul 2007 09:47:32 GMT

i tried it with a complete new FAS3020 running OnTap 7.2.2. was the same. we created a TCPDUMP and sent it to NetApp. IÂ´m actually installing the recent V5.6 ECO 1 on an IA64 OpenVMS 8.3 and hope I can gather more information and will finally open a call at HP. for a sniffer session I unfortunately have not the time and the resources actually.

Re: poor NFS performance when retrieving directories WITH attributes

Wim Van den Wyngaert — Mon, 23 Jul 2007 10:00:49 GMT

Can you setup the nfs device from a non-vms node and check if performance is bad too ?

Wim

Re: poor NFS performance when retrieving directories WITH attributes

lxworks — Mon, 23 Jul 2007 10:03:32 GMT

we have solaris and hpux and linux and all this stuff and the "ls -lisa" from these nodes is as fast as the "ls". 1 second for the complete listing including all the owner,size and time information.

Re: poor NFS performance when retrieving directories WITH attributes

Hoff — Mon, 23 Jul 2007 10:26:35 GMT

"...FAS3020 running OnTap 7.2.2..." versus the "...NetApp FAS940 running DataOnTap 6.5..."

It's quite typical and entirely common that a particular product vendor will use consistent code -- possibly with differences among available corrections and enhancements -- across its product line. The vendor builds or OEM-licenses the application code once, and re-uses it.

For purposes of bug isolation, testing with entirely different vendors is better. In this case, with whatever replaced Windows SFU, or with Linux, HP-UX, or Mac OS X Server, or...

And to keep debugging interesting, even entirely different vendors can sometimes use the same code base. In the case of NFS on OpenVMS and HP-UX, these do use different code bases, as the products themselves originated from DIGITAL and from HP; from different corporations.

Re: poor NFS performance when retrieving directories WITH attributes

Bojan Nemec — Mon, 23 Jul 2007 10:35:25 GMT

Alex,

Welcome to the VMS forum!

Because of how VMS directory works, I think that a comparable command on an UNIX node will be:

ls -lisa /vol/vol2/vol3/vol4
cp /vol/vol2/vol3/vol4/.\$ADS\$* /dev/null

Bojan

Re: poor NFS performance when retrieving directories WITH attributes

Wim Van den Wyngaert — Mon, 23 Jul 2007 10:36:53 GMT

Just did some playing and found that NFS is slow when working set is too small.
Check ucx help mount /acp to increase the settings (if you can, investigate what the process quotas are). If you have avail check and increase with this product.

Wim

Re: poor NFS performance when retrieving directories WITH attributes

Jan van den Ende — Mon, 23 Jul 2007 10:41:02 GMT

Alex,

to begin with: Welcome to the OpenVMS Forum.
>>>
the share may contain up to 3000 directory entries in root level.
<<<

Have you tried an unqualified DIR vs directory/date/size/owner/prot of the NFS mount anchor in VMS itself?
DIR only reads the INDEXFILE.SYS, where /DATE and /SIZE have to READ the file HEADER, and /OWN and /PROT have to use info from the header to tranlate it using the RIGHTSLIST.
And for a (directory-) file with 3000 entries, that DOES make a difference.
Although I would not expect THAT big.

Still, I am curious about the amount caused by the extra file access, and the amount caused by NFS.

fwiw,

Proost.

Have one on me.

jpe

Re: poor NFS performance when retrieving directories WITH attributes

Hein van den Heuvel — Tue, 24 Jul 2007 03:16:46 GMT

Are those files create from VMS and later reported on by VMS? When you did the Solaris comparision, did did you have it report on its 'own files' or on those VMS file?

How is the performance if you report on files created by a Unix box from the VMS system?

I suspect the /ADF option plays a significant roles. It forces the read of a 'helper' file for each file. This helper file holds the VMS file attributes which Unix does NOT have, for example something as 'simple' as a file creation date.

Unix has only a handfull (11 ? see fstat()) file attributes which are quickly retrieved from the directory (inode). VMS has many more attributes, but they are slow to retreive using an extra IO or two from the file headers in INDEXF.SYS on a VMS box, or from the .$ADF$file files on an NFS server.

Still 20 minutes seems excessive even if an extra file IO per file is needed

fwiw,
Hein.

Re: poor NFS performance when retrieving directories WITH attributes

lxworks — Tue, 24 Jul 2007 03:31:38 GMT

the files are contributed from various platforms, anyway, no other client show such a slow performance. for the acp-parameters, I think, the "mounter" and the "lister" have enough quota to satisfy the needs. maybe someone can add some hints...

WSdef: 131072
WSquo: 262144
WSextent: 393216

I'll next follow the adf issue, but I actually don't know, if its needed by anyone. as it is really productive, I have to take care :-(

Re: poor NFS performance when retrieving directories WITH attributes

Wim Van den Wyngaert — Tue, 24 Jul 2007 03:58:55 GMT

On VMS, also try to see what is consumed by the NFS client process (during the 20 minutes, IO's ? CPU ? PF's ?). show proc/con/id= will show the state. In wait states ?

You can try TCPTRACE to see who is delaying the communication.

Your quotas are fine. 20 minutes means about 3 files per sec. That is terribly slow.

Wim

Re: poor NFS performance when retrieving directories WITH attributes

Volker Halle — Tue, 24 Jul 2007 05:33:35 GMT

Alex,

tracing this is easy with tcpdump:

$ @SYS$MANAGER:TCPIP$DEFINE_COMMANDS
$ tcpdump host mynetapp and port nfs

Between 2 VMS systems, the first plain DIR command will exchange lots of 'lookup' NFS packets. When invoking the plain DIR command the second time, all it does is exchange a couple of 'readdir' messages. And that's much faster. For 2000 files, elapsed time drops from 1 minutes to less than 1 second on successive attempts.

The full DIR command always exchanges lots of 'getattr' NFS packets. When invoked a 2nd or 3rd time (without remounting the NFS device), it still is faster (about twice as fast). Elapsed time drops from 1 minute to about 30 seconds for successive attempts.

Tested with a directroy of 2107 files between Alpha with TCPIP V5.5 (NFS server) and rx2600 with TCPIP V5.5 (NFS client)

As some caching is certainly involved here (NFS server ? client ? file system ?), it makes predicting the performance a little bit hard.

Volker.

Re: poor NFS performance when retrieving directories WITH attributes

lxworks — Tue, 24 Jul 2007 06:14:08 GMT

I listed all environment data in my first message. the issue is the difference BETWEEN the NFS client on OpenVMS and the NFS client on other clients.

Re: poor NFS performance when retrieving directories WITH attributes

Volker Halle — Tue, 24 Jul 2007 06:32:58 GMT

Alex,

if you are interested in isolating the problem, you need to compare the data between the 'good' and the 'bad' performance case.

Comparing the tcpdump data, which can be easily collected on the VMS system - as I've shown - would be a first step.

An easily spottable difference may be in the no. of messages exchanged or in their timing/sequence.

As with all communication problems, you need a systematic approach to try to pin down the problem.

With the VMS to VMS case that I've tested, there is about a factor of 100 between the time for a subsequent DIR and a DIR/SIZ/...

Volker.

Re: poor NFS performance when retrieving directories WITH attributes

lxworks — Tue, 24 Jul 2007 07:01:09 GMT

for the tcpdump: I already did a tcpdump of one listing (with/without) yesterday, even on both platforms (NetApp & OpenVMS), the NetApp dump went to NetApp, but for the OpenVMS Call I wanted to wait for some opinions here.

As I am not able to read this data, I'll open a call at HP now, thanks to all who contributed their ideas.

brgds
alex

Re: poor NFS performance when retrieving directories WITH attributes

Wim Van den Wyngaert — Tue, 24 Jul 2007 07:33:23 GMT

You could try a remount with /cache_timeout=(dir:01:00,attr=1:00)

It seems that if changes are taking place the directory is reread every 30 secs. And file attributes remain in the cache for 15 secs. So, after 20 min you start again from scratch.

Wim

Re: poor NFS performance when retrieving directories WITH attributes

lxworks — Thu, 26 Jul 2007 00:35:32 GMT

I changed these values, too. no change. as it is very fast when doing this listing on a HPUX as NFS server I suspect this special OpenVMS/NetApp Connection as cause.