Would NFS be your choice?

[Jeff Carlin]

Weâ re setting up a big web project here. Firstly, weâ re running IBM WebSphere on a bunch of HP servers. All of the servers are fairly robust; in fact, most are over kill at the moment. The application we are deploying on WebSphere will need to access between 150,000 to 1,000,000 images ranging 8k to 15k in size. An average of 6-9 images will be displayed on a page in the 8k size and one image in the 15k size. These images will change every now and then, but not to the point I need to worry about it updating right after I have accessed it.

Since WebShere scales horizontally, I may be dealing with a lot of A500 web servers in time. The images need to be accessible on each web server. Would you replicate the images to each web server or would you NFS mount the images from a central location? Is there a better way altogether?

In test, we have set up NFS. The current problem is the page builds slowly. Of course, the NFS mount and the UNIX servers are being blamed for the slow down.

I have plenty of nfsdâ s running on the server and I am not seeing any problems on the client â no badcalls, no badxidsâ ¦

Tonight I will look into mounting the client side with actimeo=120 to cache a little longer and see if that helps.

Both clients and server are fairly idle (all stats) during 100 user tests. There is some activity â NFS calls and such, but nothing alarmingly high. But the performance still seems slow.

How can I prove, beyond a shadow of a doubt, that NFS is not the holdup?
Is NFS the best choice here?
Can I improve NFS perform

[Geoff Wild · Accepted solution]

This may help - or not - but a must read on NFS:

http://h21007.www2.hp.com/dspp/tech/tech_TechDocumentDetailPage_IDX/1,1701,952,00.html


Rgds...Geoff

[Chris Watkins_1]

The only way to prove, beyond a shadow of a doubt, that NFS isn't
the issue, is to get your baseline with the current configuration,
then copy the files locally, and run the tests again to compare.

Ultimately... that's the only way you'll get the point across,
whether it be for the good or bad. You may find that NFS is
indeed an issue, and then you can look into tuning or alternatives.

[Todd McDaniel_1]

First, I would check your various kernel parms to ensure none of them are too restrictive.


Then I would check your config file /etc/rc.config.d/nfsconf to make sure you have it setup for the type of use you are going to see.

This is off my one box which serves as NFS server for 20 boxes, we have rather low usage on this b/c it is an oracle CD which was copied to an NFS mount so it would be easier to access so we could free up the CD drive.

# NUM_NFSD: Number of NFS deamons (nfsd) to start on an NFS server. Four
# has been chosen as optimal.
# NUM_NFSIOD: Number of NFS BIO daemons (biod) to start on an NFS client.
# Four has been chosen as optimal.
# PCNFS_SERVER: 1 if this node is a server for PC-NFS requests. This
# variable controls the startup of the pcnfsd(1M) server.
# See Also: pcnfsd(1M).
#
NFS_CLIENT=1 # 0 to turn off client
NFS_SERVER=1
NUM_NFSD=16
NUM_NFSIOD=16
PCNFS_SERVER=0

I would also check your Nic card speeds on both ends on all servers that have NFS mounts. In my case with 11i it is /etc/rc.config.d/hpgelanconf... Force your card to the correct speed if it is not using it.

If you havent already built the VGs for the NFS mountpoint on the server side, I would stripe them, and for better reading with a small block size say 4MB since your data is very small.

[doug mielke]

I would absolutly use NFS over replication. We make heavy use of NFS mounted NAS filers for data and application storage. We have multiple high i/o databases running off of the NAS. NAS or any other centralised has a multitude of benefits vs. direct attached storage.

To dispell notions of NFS bottlenecks, time some benchmarks against some local storage. You can even load the systems with non-i/o functions to prove that NFS will perform even when system is busy.

Only caveat is that your network must be sound to avoid headaches

[John Dvorchak]

One thing to look at when you have a large number of NFS mounts to an 11.i server is run top and look at the sys cpu utilization. If it is hovering above about 4 to 5% then you should load patch PHNE_28568 which corrects this problem. It has to do with the lookup table size for requests.

I had a few servers here exhibiting a general sluggishness and HP suggest this patch which fixed the problem.

But you should still read the other respondents comments and tune your NFS accordingly. Many times performance issues are not fixed by just one thing. Tuning can be a series of steps where each one helps a little.


Good luck,

[Steven E. Protter]

The problem with NFS is that the data is transmitted unencrypted.

I would consider using CIFS/9000 instead. Samba is considered more secure and robust.

To improve NFS performance, install the latest version and all patches possible. Make sure the kernel is tuned properly.

Thats the best you can do.

SEP

[A. Clay Stephenson]

Obviously NFS will never be as fast as local files. Also, if you use a single NFS server then it had better be a very robust one because if you lose it, you've lost everything.

Let me say that I know nothing about WebSphere or the application you will be running but if you are using a filesystem as your database then I suspect performance is always going to be a problem -- local or NFS.
Directories were never intended to be used
as search engines.

If you can retrieve your images from a database that would be my first choice. The database should be redundant or running under ServiceGuard so that it is always available.

A hybrid design is also a possibility. The database might tell you to retrieve an image from a UNIX file starting at offset 560578 and reading 8182 bytes. In that model, you would pack many images into a single file (to limit directory searches) and use a database engine to store metadata.

The problem with keeping local copies of the data is synchronization. I would look into rdist; it is made to do just that.

[Wouter Jagers]

Should you have the diskspace available on all servers, you could consider using rsync to synchronize the content. It would greatly improve performance and depending on the changes (frequency, scale) you can run it more or less often. (http://rsync.samba.org/)

You can try and tune your NFS services, too, but in the end when you start accessing the central data from lots of servers, you'd better have real fast storage and a killer network..

regards
Wout

[Jeff Carlin]

Wow, and thanks everyone for the quick suggestions!

We are going to move 95,000 images over to one of the web servers and test local vs. nfs in a few minutes... I'll let everyone know the results...

[Jeff Carlin]

Well, there wasn't any measurable difference when the images were loaded locally. Looks like it's in WebSphere or the network...

[Mic V.]

How many files are in a single directory? One thing I've noticed (10.20 and 11.00) is that when you get to a certain number of entries in the directory, all accesses seem to slow dramatically.

I never pinned it down to a number, sorry. Probably somewhere between 5000 and 20000. I tried to get HP to talk about this, but never got beyond them admitting they knew about it. They couldn't quantify or solve it.

I then tried splitting data into multiple directories (less than 3000 in each) and performance was acceptable again.

Mic

[Jeff Carlin]

Yes, the files are all in the same directory. Listing the complete directory takes some time, but if you know the file name, it takes no more time to perform operations on the file.

[Mic V.]

Well...I don't know anything about WebSphere, so I can't say, but: we used a commercial application which "architecture" was to put all 30,000+ files in one directory. It was evident over time that the more files, the slower the client time to access the images (it was image data).

I don't have access to the setup any more, but it seemed to me that even my old-file-cleaner script, which does a stat, got veeeery slow with a large number of files.

The vendor claimed that no one had ever seen this problem before. I tend to believe it's more likely that they hadn't heard of it, since most of their customers did Solaris (and apparently it's not a problem on Solaris).

Anyway...would be interesting to prune a directory to the point where "ls" is no longer slow, then see if the app is still slow. Good luck. :)

Mic

[Todd Whitcher]

The White Paper that was poster earlier is a good guide to NFS performance and provides examples of using iozone. If you want to do some benchmarking you can download iozone here http://www.iozone.org/ its free. For NFS if you have a clean network use UDP if not test TCP. You have to tune some kernel params to use TCP, check the white paper.

Good Luck

Todd

[doug mielke]

I seem to remember reading about inodes, and that they use pointers to blocks to find data. There was a limit, after which another structure ( another inode? ) was created with pointers to additional blocks, up to 3, making it possible to have to examine 3 inodes to find your data. Since a directory has only 1 inode, it's somewhat related to block size. A trade off here could be raising block size, wasting a bit of space, but providing faster accesss.

Also, I've also read that long filenames (and long pathnames) in large directories can slow access.

[Chris Vail]

Following up on one of the oher posters:
30,000 entries in one directory?!?!? EEEEEEKKK!!!!! A directory is just a flat file. You should build a directory hierarchy instead: t'will be much faster and easier to maintain.

You could load these images into a SQL Database (Oracle, for example) and get much better access times and efficiencies than just putting them out into a directory as standalone files.

I suggest you re-architect this. As you have it now, t'will eventually work. But its like "socks on a rooster"--why would you want it to? Its too ugly for words.

Chris

[Jeff Carlin]

Well, we are going live with the images being served up via NFS. Performance is excellent. Even under load, the service time per image is 0.02 seconds. All of our current slow downs are because of database look up's that happen before the page is displayed.

Thanks again everyone for the excellent ideas and help!

[Bill Hassell]

NFS for primarily read-access is not a bad solution and as you've seen, the response time is quite acceptable. But before you go much further, be sure to get the NFS Performance book:

Optimizing NFS Performance: Tuning and Troubleshooting NFS on HP-UX Systems
by Dave Olker

The definitive reference on NFS for HP-UX.

[Jeff Schussele]

Hi Jeff,

NO

Rgds,
Jeff

[Jeff Carlin]

Thanks Bill! I have been using this guide. It is an excellent guide and probably why things are going so smoothly out the gate.