topic Re: Huge flat directories in HPUX 11i in Operating System - HP-UX

Huge flat directories in HPUX 11i

YLTan — Fri, 11 Jul 2003 08:59:54 GMT

I understand that HP-UX has troubles with *really* *big* directories that containing 100,000 files which is limit by the maximum arguments size that the shell will handle.

I heard that there are some patches released that fixed this outrageous problem.

Does anyone know what these patches number for HPUX 11i and if it is effective in fixing the problem.

Re: Huge flat directories in HPUX 11i

Stefan Farrelly — Fri, 11 Jul 2003 09:21:46 GMT

HPUX doesnt have a problem with really big directories. Its all a matter of tuning your filesystem if you do have really big dirs with a lot of files. There have been comments on this forum from customers with millions of files in a dir without problems.

Read these threads;

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xe87a42308663d611abdb0090277a778c,00.html

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xe87a42308663d611abdb0090277a778c,00.html

As long as you have the latest VXFS/JFS/LVM patches and take note of some tuning options as in the threads above you will be fine.

Re: Huge flat directories in HPUX 11i

Bill Hassell — Fri, 11 Jul 2003 10:54:38 GMT

Massive flat directories is a system design flaw, not an HP-UX problem. It's not an outrageous problem, it's a msunderstanding of how the shell works. For instance, consider these two commands:

ls
ls *

They seem to perform the same task, yet the first works just fine with millions of file names, but the second fails with an arg list too long message. The key is understanding what * means. It is NOT interpreted by the ls command! The shell sees the * and automatically translates * into the names of every visible file in the current directory and that is passed to the ls command. As you might expect, the resultant command line may be millions of characters long.

Now there was a patch to extend the maximum command line (known as ARG_MAX) from about 20,000 chars to over 2 million. One could argue that ARG_MAX should be billions, but the reality is that it is simply impractical to continue extending the maximum line length for a command. Besides, that's why the xargs command was created.

As every Unix system administrator has discovered, flat filesystems with thousands of files are a nightmare to manage and users must be taught how to work with this unusual configuration. Every task that requires looking at every filename (like ls and ll and find, etc) will take a long time to process, and most important: lots of directory (system) overhead which affects other programs on the system.

Re: Huge flat directories in HPUX 11i

Steven E. Protter — Fri, 11 Jul 2003 11:14:56 GMT

1) Keep up with any JFS/vxfs patches.

2) This is a management problem. There is a reason why there is a command mkdir. Its to make subdirectories. These huge directories should be reorganized in some logical way.

SEP