topic Re: Nfs Slow in Operating System - HP-UX

Nfs Slow

gigiz — Tue, 10 Aug 2010 10:04:23 GMT

Hi guys,
This is a scenario:
Server nfs with 350Gb file system exported to clients. This file system contains 550000 file of 1kb anyone.Nfs work fine also that
The problem is from ll from the client. If i do "ll command" in the nfs imported file system from the client it work in 9 minutes.
How Can i improve nfs performance for the "ll" from the client?

Re: Nfs Slow

Dennis Handly — Tue, 10 Aug 2010 11:00:12 GMT

I assume those 550,000 files aren't in the same directory?

You might want to look at Dave's replies about READDIR:
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1376095
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1081327

Re: Nfs Slow

gigiz — Tue, 10 Aug 2010 11:06:49 GMT

unfortunately yes 550000 files in the same directory ... :-(

Re: Nfs Slow

Steven Schweda — Tue, 10 Aug 2010 12:11:44 GMT

> unfortunately yes 550000 files in the same directory ... :-(

"Doctor, it hurts when I do this."

"Don't do that."

Re: Nfs Slow

gigiz — Tue, 10 Aug 2010 12:29:23 GMT

Yes but i am the doctor !!

Re: Nfs Slow

Steven Schweda — Tue, 10 Aug 2010 12:53:54 GMT

> Yes but i am the doctor !!

Physician, heal thyself.

Keeping many thousands of files in one
directory is simply asking for poor
performance (for many different OS and
file-system combinations).

There may be some directory cache parameters
which you could adjust in the NFS client or
server (about which I know nothing), but I'd
expect that a real solution to this problem
would involve re-designing your data storage
architecture to avoid having so many files in
one directory.

A few bad decisions can overcome a great deal
of computing power.

Re: Nfs Slow

gigiz — Tue, 10 Aug 2010 14:23:55 GMT

>Physician, heal thyself
eh eh !! :-)

i have find this scenario from my customer.
I agree with you but i must improve the performance.
Help me

I tested the server file system and
the ll in server is very slow:
time ll
real 7:50.1
user 19.1
sys 1:22.9

I think that a problem is the file system and not nfs !!
any idea ?!!?

Re: Nfs Slow

Steven Schweda — Tue, 10 Aug 2010 17:07:54 GMT

> I tested the server file system and
> the ll in server is very slow:
> [...]

Do you find this amazing? (I don't.)

> Keeping many thousands of files in one
> directory is simply asking for poor
> performance (for many different OS and
> file-system combinations).

I believe that that's still true.

> I think that a problem is the file system
> and not nfs !!
> any idea ?!!?

Only the same idea: "Don't do that."

> i have find this scenario from my customer.

Advise _him_ not to do that.

You're not the first person to observe this
effect. A Forum search for keywords like,
say:
many files directory slow
might find threads like these:

http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1331134
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1120029
[...]

Re: Nfs Slow

Dennis Handly — Tue, 10 Aug 2010 22:14:33 GMT

Have you looked at Dave's READDIR threads? They may make your "ll"s faster?

Re: Nfs Slow

Dave Olker — Wed, 11 Aug 2010 04:03:08 GMT

Have you looked at either of Dennis Handly's replies?

Re: Nfs Slow

gigiz — Wed, 11 Aug 2010 07:31:15 GMT

Ok the last test :
In my nfs Server :

In my nfs Client :

The problem could be a file system exported from a Server and not NFS?
The file system in the nfs server are degraded from export or no ?
I still check READDIR and READDIR+?

Re: Nfs Slow

gigiz — Wed, 11 Aug 2010 08:18:25 GMT

ops

last test on my server :
time ll|wc -l

real 6:39.5
user 18.6
sys 1:18.7
522511

last test from my client :
time ll|wc -l

real 13:17.6
user 18.6
sys 38.5
522487

Re: Nfs Slow

Elmar P. Kolkman — Wed, 11 Aug 2010 08:39:28 GMT

Is pwgr running on the server ?
The problem is caused by collecting all info on the files and translating things like userid's and groupid's to the end-user-readable form with usernames and group names.

You might try something like 'ls -nl' to see if it's faster. If that is faster, tools like pwgr and nscd might help.

Re: Nfs Slow

gigiz — Wed, 11 Aug 2010 09:30:51 GMT

Good idea ...
but :-( same result :

from a server
ls -nl|wc -l

real 6:32.2
user 18.1
sys 1:18.9
522709

Re: Nfs Slow

Elmar P. Kolkman — Wed, 11 Aug 2010 10:24:16 GMT

Then you already have some caching daemon running...

But with 522000 files being questioned within 400 seconds, with date-conversion and all, your system isn't as slow as you suspect...

If you calculate the time per directory entry, it is comparable with the times I need on a local filesystem per file...
The only difference is the caching on local filesystems: when running the same time ll command again, the directory information is cached, resulting in a really quick response. Since NFS is not really cachable, that doesn't work. You might look at the ac* mount options, but I don't think they really help with this number of files.

If you really need it faster, and the storage is only used on this server, you might think about using different protocols for accessing the data, for instance iSCSI.

Re: Nfs Slow

Bill Hassell — Wed, 11 Aug 2010 10:55:57 GMT

As mentioned before, 550k files is a terrible design but sometimes you are stuck with it. What is worse is that NFS makes this even slower. So there are two fixes:

1. DON'T use ll. Instead, create your own hash table or file index and look at the index to find the matches. Convenient? Probably not, but this is a side effect of a bad design.

2. Change your network to 10 Gbit links, switches and routers. Also, change the storage for the 550k files to a RAMdisk appliance. And if necessary, upgrade your computer to more and faster processors.

Either of these should improve performance. Did I mention that these choices are the result of a bad directory design?

Re: Nfs Slow

gigiz — Wed, 11 Aug 2010 10:59:42 GMT

another test do similiar result between client and server:

SERVER
real 5:44.8
user 18.1
sys 1:14.9
521782

CLIENT
real 7:50.1
user 19.1
sys 1:22.9

uhhmm ?

Re: Nfs Slow

Orlando-Torres — Thu, 09 Nov 2017 15:36:22 GMT

Hello Bill,

What would be a good directory design?

I'm facing a situation at customer site, where thousands of files in several directories.

Can you please advice me about NFS directory tree design for better performance.

Thank you.