Nfs Slow

[gigiz]

Hi guys,
This is a scenario:
Server nfs with 350Gb file system exported to clients. This file system contains 550000 file of 1kb anyone.Nfs work fine also that
The problem is from ll from the client. If i do "ll command" in the nfs imported file system from the client it work in 9 minutes.
How Can i improve nfs performance for the "ll" from the client?

[Dennis Handly · Accepted solution]

I assume those 550,000 files aren't in the same directory?

You might want to look at Dave's replies about READDIR:
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1376095
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1081327

[gigiz]

unfortunately yes 550000 files in the same directory ... :-(

[Steven Schweda]

> unfortunately yes 550000 files in the same directory ... :-(

"Doctor, it hurts when I do this."

"Don't do that."

[gigiz]

Yes but i am the doctor !!

[Steven Schweda]

> Yes but i am the doctor !!

Physician, heal thyself.

Keeping many thousands of files in one
directory is simply asking for poor
performance (for many different OS and
file-system combinations).

There may be some directory cache parameters
which you could adjust in the NFS client or
server (about which I know nothing), but I'd
expect that a real solution to this problem
would involve re-designing your data storage
architecture to avoid having so many files in
one directory.

A few bad decisions can overcome a great deal
of computing power.

[gigiz]

>Physician, heal thyself
eh eh !! :-)

i have find this scenario from my customer.
I agree with you but i must improve the performance.
Help me

I tested the server file system and
the ll in server is very slow:
time ll
real 7:50.1
user 19.1
sys 1:22.9

I think that a problem is the file system and not nfs !!
any idea ?!!?

[Steven Schweda]

> I tested the server file system and
> the ll in server is very slow:
> [...]

Do you find this amazing? (I don't.)

> Keeping many thousands of files in one
> directory is simply asking for poor
> performance (for many different OS and
> file-system combinations).

I believe that that's still true.

> I think that a problem is the file system
> and not nfs !!
> any idea ?!!?

Only the same idea: "Don't do that."

> i have find this scenario from my customer.

Advise _him_ not to do that.


You're not the first person to observe this
effect. A Forum search for keywords like,
say:
many files directory slow
might find threads like these:

http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1331134
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1120029
[...]

[Dennis Handly]

Have you looked at Dave's READDIR threads? They may make your "ll"s faster?

[Dave Olker]

Have you looked at either of Dennis Handly's replies?

[gigiz]

Ok the last test :
In my nfs Server :


In my nfs Client :

The problem could be a file system exported from a Server and not NFS?
The file system in the nfs server are degraded from export or no ?
I still check READDIR and READDIR+?

[gigiz]

ops

last test on my server :
time ll|wc -l

real 6:39.5
user 18.6
sys 1:18.7
522511

last test from my client :
time ll|wc -l

real 13:17.6
user 18.6
sys 38.5
522487

[Elmar P. Kolkman]

Is pwgr running on the server ?
The problem is caused by collecting all info on the files and translating things like userid's and groupid's to the end-user-readable form with usernames and group names.

You might try something like 'ls -nl' to see if it's faster. If that is faster, tools like pwgr and nscd might help.

[gigiz]

Good idea ...
but :-( same result :

from a server
ls -nl|wc -l

real 6:32.2
user 18.1
sys 1:18.9
522709

[Elmar P. Kolkman]

Then you already have some caching daemon running...

But with 522000 files being questioned within 400 seconds, with date-conversion and all, your system isn't as slow as you suspect...

If you calculate the time per directory entry, it is comparable with the times I need on a local filesystem per file...
The only difference is the caching on local filesystems: when running the same time ll command again, the directory information is cached, resulting in a really quick response. Since NFS is not really cachable, that doesn't work. You might look at the ac* mount options, but I don't think they really help with this number of files.

If you really need it faster, and the storage is only used on this server, you might think about using different protocols for accessing the data, for instance iSCSI.

[Bill Hassell]

As mentioned before, 550k files is a terrible design but sometimes you are stuck with it. What is worse is that NFS makes this even slower. So there are two fixes:

1. DON'T use ll. Instead, create your own hash table or file index and look at the index to find the matches. Convenient? Probably not, but this is a side effect of a bad design.

2. Change your network to 10 Gbit links, switches and routers. Also, change the storage for the 550k files to a RAMdisk appliance. And if necessary, upgrade your computer to more and faster processors.

Either of these should improve performance. Did I mention that these choices are the result of a bad directory design?

[gigiz]

another test do similiar result between client and server:

SERVER
real 5:44.8
user 18.1
sys 1:14.9
521782

CLIENT
real 7:50.1
user 19.1
sys 1:22.9


uhhmm ?

[Orlando-Torres]

Hello Bill,

What would be a good directory design?

I'm facing a situation at customer site, where thousands of files in several directories.

Can you please advice me about NFS directory tree design for better performance.

Thank you.