NFS across a WAN - moving files is slow

[RobertCarback]

We have a Linux server that mounts an HP-UX 11i filesystem via nfs over a WAN (Baltimore - Phila). The Linux server, every 2 minutes, does an 'ls' to get 50 of the names of the files that have been placed automatically in the nfs filesystem. The files are then moved (mv) to a local filesystem on the Linux server. The problem is that once a month, when there are around 1000 of these files (at a total of around 44MB -- 44K each) in the nfs filesystem, nfs is so slow that the 'ls' times out and gets 0 files instead of 50.

[Stuart Browne]

Do the files follow a given naming convention?

In any case, use 'ls -f' to do an un-sorted listing, or use 'find . -name ""'.

One of the reasons why 'ls' will take a long to list the files in a given directory is that it tries to 'sort' the contents into a nice order, thus needs to know *EVERY* filename in that directory.

Unf.. ;)

[Steve_160]

I'd suggest that the source server generate the list of 50 files and store it in a file with a unique filename.

The destination server could copy that list over, read it, move only those 50 files, then delete the list. When the source server finds the list missing--and the files on it, it could generate another list.

Alternately, you could incorporate a queue directory on the source server. Whenever the server finds less than 50 files in the active directory it would move a few more over from the queue.

[Al Licause]

Rather than try to find an application work around, why not try to analyze why the timeouts occur....i.e. might you possibly have a bad link or noise on the circuit, etc.

What is the speed of the WAN link ?

Are you using udp or tcp mounts for nfs ?

Do ftp copies run much faster ?

Have you actually established a base line operating norm for this link specifically for nfs ? i.e. what should your realistic expectations be in terms of throughput ?

Have you tried tunning the mount point by increasing timeout and retry values and/or lowering read and write buffer sizes ?

I suspect that if you can't get the ls results to work, you more than likely won't get the copies to work either so, you'd be best off trying to find get more information about the underlying cause.

[RobertCarback]

Please close this thread. I meant to put it through HP-UX, not Linux