NFS problem sharing many files

[Jens Ebert]

Szenario: HP-UX 11.11 Server exporting a directory holding 20000 files (or even more).
NFS-Client is Linux mounting this directory and doing an "ls". This takes about 30 minutes to complete (we have seen up to 90 minutes for more files). This is a mess.
HP told us to increase the number of nfsd processes on the NFS-server from 4 to 16 (it is a 4 processor system), this does not help. We see all the nfsd processes as top-cpu processes consuming almost all of the cpu time.
HP told us, that this is how NFS works and closed the case.
We tried the same using a HP-UX 11.11 client as well, the result is similar.
Any idea?
Does it make sense to try an smbmount?
Thank you, Jens

[Gopi Sekar]

Any network mounted filesystem containing that many huge number of files will have this problem. so i believe you will be facing the same issue with smbmount also.

you can try for some file grouping (grouping files in to a particular directory structure) so that at a time a directory can contain maximum of 1000 to 2000 files.

Regards,
Gopi

[C. Beerse_1]

Is it is only `ls` that takes that long or is it an `ls -l` which takes that long?

As far as I understand `ls` and filesystems, the `ls` 'only' reads the directory, sorts the names and spits that out This should be relatively quick. An `ls -l` does not only read the directory, but also each inode that come with each file for the info stored in there (all but the filename). Specially the latter wil be a lot. For 20000 files, it will be 20000 requests, which each have to be answered. On local storage, this info is cached info and hence relatively quick. With nfs, it has to be asked every time since it can change outside the systems knowledge.

For samba, the requiests are rougly equal, however, I don't know if there the inode info is combined with the directory info at the server side. Best to give it a try (and inform us)

[Jens Ebert]

It really is "ls" that takes that long (in fact, our java based applications does the listing java-like).
The problem seems to be based on the size of the directory entry itself: because of the huge number of files it contained at some time in the past, it was about 95MB. If I rmdir it and recreate it, all looks fine. BUT the problem will show up soon again, as our processes will put many files in it soon ...
I will try to find time to use cifs, but up to the comments I got from various sources up to now, that might no improve the situation.

[Zinky · Accepted solution]

This is a common problem amongst all Filesystems - UFS, VxFS, NFS. The fix is to reorg your storage so you do not have that great a number of files contained in one directory.

If it cannot be avoided, then application routines (and users) should simply avoid doing a directory listing in such filesystems. One possible appllication fix that we've tried in the past is to simply catalog and index the entries if such filesystems in a database. A web front-end (queriable) then takes place of the "ls" / listing front end.

[Jens Ebert]

We will redesign our application and switch to database usage instead of filesystem usage.