10TB NFS filesystem

[dmsmith32]

Hello

Please could someone advise me on the following.

I have been asked by the customer to create 7x10TB filesystems for use over NFS as non-backed up storage area. I would like to know if this is workable on my system (details below) I belive it should be but would like confirmation.

-70x1TB luns will be presented from an EVA8100
-Will use LVM to create 7x10TB striped logical
volumes. 10 luns to each logical volume.
-LVM volumes will be mounted on NFS server
-Filesystems will be contained within ServiceGuard and NFS mounted to NFS Clients


Versions.....

NFS server...

HPUX 11.31
LVM version 1 (16TB max Filesystem)
VxFS version 6
JFS Base VxFS System 4.1 (32TB max Filesystem)
ServiceGuard A.11.17.01
NFS network 1GB


NFS Client....

HPUX 11.23
VxFS version 5
JFS B.11.23 The Base VxFS File System
PHKL_33368 1.0 JFS3.5 direct I/O performance improvement


Obviously any more info you require please ask.

Thankyou in advance.

[Steven E. Protter]

Shalom,

Here is the limiting factor:
LVM version 1 (16TB max Filesystem)
VxFS version 6
JFS Base VxFS System 4.1 (32TB max Filesystem)

You will not be able to present one gigantic file system. You will need to deliver a series of file systems and manage them.

I like a system that creates soft links to the various file systems and permits you to move things around on the back end.

You can rebuild the soft links daily, changing them as you move files around.

SEP

[Dave Olker]

Steven,

Unless I'm mistaken, the proposal was sharing 7 separate filesystems each 10TB in size, not one 70TB filesystem.

Dave

[rick jones]

7, 10TB filesystems shared between just one NFS server and client? Even if it is multiple NFS clients, over just one GbE network?

Dave can speak to performance of NFS far better than I but it struck me as odd having 70 TB of filesystem(s) out there, and however much bandwidth there is in an EVA 8100, but then a soda straw of a network over which it would be accessed. I would have expected a 10Gbit/s network.

[dmsmith32]

Thanks for your replies..

I would like to expand on the system setup a bit more..

There are 3 NFS servers which run ServiceGuard.
The Filestystems on these 3 servers are contained in ServiceGuard packages.

Luns are presented from the EVA8100 over a 4Gbit network to these NFS servers.

These filesystems are then NFS mounted to the clients over a 1Gbit network.

There are approx 45 NFS clients, mainly HPUX and some Redhat

Server B has 74 fileystems upto 1.5TB in size, totaling 36TB

Server C has 172 fileystems upto 1.5TB in size, totaling 42TB

Server A is minimal

The new 10TB filesyetms will be split between server B (4 FS) and server C (3 FS).

My first question was.. Is it possible with the process described and versions of software to successfully create these 7x10TB luns? I believe it is.

The second question and my main concern is what impact do you see on the NFS environment by NFS mounting these large filesystems, taking into account what is already in this environment. Would you advise against it? Should I use smaller filesyetms e.g 5TB.


Once again any advice you can give would be greatly appreciated.

Best regards

[Dave Olker]

> My first question was.. Is it possible with
> the process described and versions of
> software to successfully create these 7x10TB
> luns? I believe it is.

I don't know of any limitation that would stop this from working. I've used NFS filesystems larger than 10TB without incident.

> The second question and my main concern is
> what impact do you see on the NFS
> environment by NFS mounting these large
> filesystems, taking into account what is
> already in this environment. Would you
> advise against it? Should I use smaller
> filesyetms e.g 5TB.

Others can chime in, but in my experience it's not necessarily the size of the filesystem that causes problems, it's the directory hierarchy on the filesystem that usually leads to performance problems.

In other words, if you're going to be building directories housing many thousands of files in a single folder then you can expect performance problems. You'll want to be careful about how you design the directory layout so that you don't end up with few massive directories where all the nfsd threads will contend against each other.

If you know you're going to have large directories on the server, I'd recommend you update your VxFS version to 5.0 and use the newer disk layouts for these new filesystems. One of the big performance improvements Veritas/Symantec added in 5.0 was large directory performance. The feature is described in the VxFS 5.0 Release Notes.

Hope this helps,

Dave

[dmsmith32]

Hi Dave

Thanks for your reply

In my config I stated that the NFS clients are on Vxfs version 5 and the NFS serevrs are on VxFS version 6.

So I guess I am covered as for as that is concerned.


Server
#fstyp -v /dev/vg00/lvol9
vxfs
version: 6

Client
#fstyp -v /dev/vg00/lvol3
vxfs
version: 5

[Duncan Edmonstone · Accepted solution]

You are confusing VxFS software versions with VxFS Disk Layout Versions (DLV). These are NOT the same.

You do not have VxFS 5.0 installed on your NFS server - you have version 4.1 (as you showed in your initial post). VxFS 4.1 supports DLVs up to v6 (the DLV is what you see from the fstyp command).

So the reccomndation to upgrade to VxFS 5.0 remains.

HTH

Duncan

[dmsmith32]

Hi Duncan

Thankyou very much for your quick response, much appriciated