16 TB filesystem size in xvfs 3.5 ?

[Richard Allen]

I was reading the HP-UX 11i v2 release notes when I came across a new feature there I've been waiting for for a very long time.
From the documentation:

New Version 5 Disk Layout - Version 5 enables the creation of file system sizes up to 16 terabytes. Files can be a maximum of 2 terabytes. File systems larger than 2 TB must be created on a VERITAS Volume Manager volume.

This 2TB maximum size of a filesystem has been a headache for me so I was very happy to see this untill I read that the Veritas volumemanager must be used for larger than 2TB filesystems.

I can easily make lvols larger than 2TB using the good old LVM so why does vxfs 3.5 care which LVM is used ?
The systems I need this for are a 3 node MC/SG cluster running NFS and samba. Migrating to another LVM scares me because I've never used it and I know the original one very wall.

[TwoProc]

A few issues:
vxvm isn't really Veritas - it's called jfs. Yes, this core code is more than likely a Veritas port (I've heard this, and the vxfs makes it look like it's really Veritas - but it's not the Veritas add-on product that you would go out and buy.
So, I don't know if that's the Veritas that you're talking about - or, are you speaking of plain old vxfs, or even the online-jfs product ( a higher level of same) or the actual Veritas File System manager that sold separately (and for quite a few bucks I might add).

I've a feeling that it's separately purchaseable Veritas File System that you're speaking of (reading your note above).

Now, assuming that it's Veritas File System that you're after - you'd have to learn a new volume manager anyway - at any size. Veritas File System and its administrative commands are truly different than the lvm commands that I think you're referring to.

I believe the note is saying that to get over 2TB, you'll have to use Veritas File Manager. It's not saying that you can't use it for less than 2TB. It's just that apparently, plain old lvm (using journaled file system or even online journaled file system - which is vxfs) isn't going to get you over the 2TB limit.

So, it looks like getting to Veritas File System at all - independent of how big the file system you're going to use, is going to cost you both in learning and $$$ for a license purchase.

[Steven E. Protter]

VxVM is a Veritas product.

It peforms the same function as LVM

I am not aware of any restrictions on LVM. If you want to go straight LVM you should lose no capabilities that were present under prior releases.

My first instinct is that note is wrong. I'll have to check into it more deeply. If you have time, go ahead and try to use LVM for a bigger fs and see what happens.

Install the latest LVM patches first.

SEP

[Pete Randall]

Whoa there - there seems to be a bit of confusion in the last post between file systems and volume managers.

There are two different file systems: HFS and VxFS or JFS.

There are also two different volume managers: LVM and VxVM.

What Richard is saying is that, in order to create 16 TB file *system*, it must be created on a VxVM *volume*.

I don't have any answer for Richard but I sincerely hope we can keep the discussion focused.


Pete

[Pete Randall]

Whoops - not confusion in Steve's post - make that "in the first post" instead.


Pte

[melvyn burnard]

I have not read this detail yet, but I believe what it is telling you is that to use a file system greater than 2TB, it must be created on a disk volume that is created/managed by VxVM, and not LVM.
LVM has a limitation of 2Tb I believe

[Robert-Jan Goossens]

Check this doc.

US
http://www2.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000076281216

Europe
http://www4.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000076281216

Document description: HP-UX11iV2: How large a file system is supported for JFS 3.5?
Document id: UNX11iV20409-45

[Richard Allen]

Not that Im the expert here but I've been reading the release notes quite extensively and my understanding of them is that in order to use VxFS (JFS) filesystems that are larger than 2TB I must use VxVM and not LVM.

In my environment I have a 3 node MC/SG cluster running NFS/Samba on many SAN boxes with storage getting close to 10TB, all using LVM and not VxVM.

Migrating to VxVM is almost out of the question.

I know LVM has no 2TB limit from experience. Using large Physical extents I've created lvols larger than 2TB and only had problems during newfs or "fsadm -F vxfs -b" (extending)

This leads me to belive that if VxFS (JFS) version 3.5 that comes with HP-UX 11.23 has support for 16TB, then it really should work on LVM also.

What I worry most about is the support side. If something should go wrong in the future would HP dismiss me because I would have an unsupported setup ?

[Geoff Wild]

According to this white paper - max lvol size is 2 TB:

http://docs.hp.com/en/6054/Limits_wp.htm

Rgds...Geoff

[Alzhy]

ITRC Forum management - with an ever increasing VxVM inquiries and posts - when are we gettijng a separate VXVM or Veritas category?

On the topic at hand:

Yes you will need VxVM as your volume manager if you need to create VxFS Filesystems on VOLUMES (LVOL in LVM lingo) that are more than 2TB in size. This is due to an LVM limitation:

http://docs.hp.com/en/6054/Limits_wp.htm

To correct an earlier post:

VxVM is the Veritas Equivalent of LVM (volume manager)

VxFS is EXACTLY the JFS (or HP-UX OJFS) product. HP oem's it from Veritas. It is a FILESYSTEM not a Volume Manager.

Note that Filesystems are INDEPENDENT of Volume Management. You can have VxFS or hsfs on top of simple disks, LVM LVOLS or VxVM VOLs.

[Robert-Jan Goossens]

http://docs.hp.com/en/5971-2383/5971-2383.pdf

vxfs 12 TB for 11i version 2

[Geoff Wild]

According to this doc, it looks like JFS 3.5 supports filesystems up to 32TB!

But there's a note stating "
VxFS disk version layout 5 (11.23) and HP OnlineJFS license is needed to create filesystems past 2TB.
Currently, HP supports filesystems <= 12 TB."

Rgds...Geoff

[Bill Hassell · Accepted solution]

The limitations for LVM have always been 2 Tb or less, depending on the patch level. Earlier versions of LVM would allow virtually unlimited volumes to be created because the LVM code was doing proper range checks. With a recent patch, the limits are now enforced. I think that multi-TB lvols could be created but the code was not known to be stable above specific limits.

LVM is actually an IBM product (which is why AIX looks similar for volume management). Volume managers and filesystms get easily confused. LVM and VxVM (by Veritas) are volume managers that can carve up disks into arbitrary sections (logical volumes). LVM and VxVM don't know anything about filesystems...indeed, an lvol may be a raw swap area, a raw data area for a database like Informix, or it may have a filesystem added to it.

There are two primary filesystems, HFS (typically used only on /stand) and VxFS, also known as JFS (Journaled FileSystem). Now to make things a bit more confusing, VxFS is also a Veritas product, as is Online JFS or Advanced JFS. So if you carve up the disks with VxVM and add a VxFS filesystem, then you are using two Veritas products. That's not too important as they are separate products. However, I believe that VxFS would format a very large lvol (multi-TB) but again, stability (read: thorough testing with all components) may not have been done.

So VxVM has been enhanced to allow 16Tb lvols that are known to be stable and VxFS (using the version 5 layout) will allow creation of 2Tb files. You can check the filesystem version for VxFS with fstyp as in:

fstyp -v /dev/vg00/lvol4 | grep version

(you have to use the lvol name, not the mount point)

[Alzhy]

VxVM 3.5/VxFS 3.5 IMHO were already very tightly integrated with HP-UX since 09/2002 - BUT - documentation was (and still is) confusing.

Hopefully with the advent of the Veritas Storage Foundation products on HP-UX 11i releases as a result of the HP-Veritas agreement - all documentation will be clear as well as support structures.

What is not clear as of now is whether Storage Foundation (4.0 or 4.1) will be released/integrated in HP-UX 11i v1.0 (PARISC) release OR if it will solely be available on version 2.0 and subsequent releases.

[Steven E. Protter]

I think there is enough confusion now that a practical test needs to be done.

Nelson, your query has a better chance of being listened to in the issues thread.

Someone with a big system and some time on their hands, how about a few tests?

SEP

[Steven E. Protter]

Actually, while I was typing my post, Bill cleared up my confusion. That kind of leaves our poster in a bind.

SEP

[melvyn burnard]

Perhaps this may answer your question:

http://docs.hp.com/en/6054/Limits_wp.htm

and 2TB is the maximum file system size supported on LVM.

Also see
http://docs.hp.com/en/5971-2383/5971-2383.pdf
particularly page 3, and the note 3 at the bottom of the page.
Also note this document was prior to the release of the September 2004 11iV2 Update release
And although it theoretically allows up to 32Gb, HP has only tested and certifed up to 16Gb

[Richard Allen]

So to summarise. There really IS a limit of 2TB in good 'ol LVM making Filesystems on LVM lvols actually impossible.

My misunderstanding stems from the fact that I've actually created lvols using LVM that are bigger than 2TB but as pr. the Limits document people have refered me to here there is a nasty comment there that really explains it all:

"[I/Os beyond 2TB will be rejected]"

Thus I must look into migrating to the VxVM volume manager in order to use larger filesystems than 2TB

I was really hoping this was unneccesary but apparently not.

Thank you all for your help :)

[TwoProc]

Yup - I did it - I had a brain f*rt and mixed up the idea of Veritas as a file manager with an lvm. Sorry all.
...Where's the danged recall button on this thing???