ls buffer size hp-ux 10.20

[jerry1]

Does anyone have a patch for "ls" that
fixes the buffer size when listing lots of
files?

/bin/ls: Arg list too long.

[John Poff · Accepted solution]

Hi,

I've never heard of a patch for that problem. Usually when I get that it is time to use 'find' and/or 'xargs'.

What were you trying to do with 'ls', and how many files do you have in that directory?

JP

[A. Clay Stephenson]

There is nothing wrong with ls. The problem is that the shell has a finite limit for the argument list. There was a patch. Look for "argmax" or "ncargs" -- something very close to that. The bad news is that you are going to have trouble find any 10.20 patches.

[A. Clay Stephenson]

You may already have the patch. Check to see if you have a tunable named "ARG_MAX" or something very close to that; that's your boy. The arg_limit was bumped up to a very large value in 11.x; in the good old days of UNIX it was 5Ki characters then it grew to 10Ki characters.

[OldSchool]

ARG_MAX is the name in the header files. Problem your running into is that when doing "ls *" the "*" is expanding out to a character string longer than ARG_MAX characters.

As noted above, "find" is your friend here.

First ran across this when porting from Data General UX (5 maybe) to HP-UX 9. DG had 2meg ARG_MAX and HP had the 5K(?). Custom app had 1800+ source files, some w/ very long names......it was ugly

[inventsekar_1]

hi jerry, ls is a command. commands usually not requires patches.

the kernel parameters affects the commands.
for example, "create_fastlinks" MAY affect the ln or ln -s commands.

[Bill Hassell]

As Clay mentions, this has nothing to do with ls. When you use ls like this:

ls /dir_with_10k_files/*

ls *NEVER* sees the "*". That character is preprocessed by the shell. It is replaced with all the filenames that match the *. Let's suppose there are 2 files named AB1 and AB2 and you want to use ls:

ls /dir_with_10k_files/AB*

You'll see the two files listed even though there are 10,000 files in the directory. What was passed to ls were the full pathnames of the 2 files. To better see this, use echo instead:

echo /dir_with_10k_files/AB*

Now try the same echo with all the files:

echo /dir_with_10k_files/*

and you'll see "arg list too long"

The shell only has a small buffer to expand the names. It used to be about 20k bytes and with a patch was increased to a couple of megs. NOTE: the buffer can never be made large enough! No matter how big the next patch might be, someone will create a massively large directory that will still exceed the latest buffer size and cause a problem.

So there are two steps:

First, do whatever you can to stop allowing massively large flat filesystems (directories with thousands of files). You are just beginning to se some of the difficulties in managing hundreds or thousands of files in one directory.

Second, start looking at xargs to filter long lists. xargs is like a small truck -- it picks up long lists and sends them out in small bundles.

[Mridul Shrivastava]

Unfortunately, you are experiencing a limit with the operating system that cannot be changed. The shell generates this message whenever exec() fails due to lack of memory for command arguments.

The file /usr/include/sys/param.h defines the variable NCARGS, which defines the size for the argument list. There is no way to change this value, because the OS has been compiled based on the current header file.

Due to a limitation in the OS, you only have two options at this point:

1. Break your ls search into smaller cross-sections.

2. Try an alternative searching method.

[A. Clay Stephenson]

I looked through some of my old notes and the 10.20 patch is PHKL_16751 (if this is a server) there is probably a different patch for workstations. In any event, you can download the patch from the archived patches and then you must enter "large_ncargs_enabled" in the /stand/system file and build a new kernel. Almost certainly you already have this patch installed as it was one of the more infamous 10.20 patches. This will expand the arggment list for the exec() system call to 2MiB. However, as already mentioned, a far better approach is to break you commands into manageable pieces using the xargs command.

[jerry1]

The limit is 2966 files of zero size.

# ls * |wc -l
2966

# touch onemore

# ls *

ksh: /usr/bin/ls: arg list too long

[A. Clay Stephenson]

You still don't get it. It has nothing to do with ls or the size of the files. It has everything to do with the size of the argument list. In this particular case, while the size of the files doesn't matter the length of the pathnames does. You should really plan on the argument list being no bigger than 5KiB because that is all that System V guarantees and if you stick to that limit, your scripts will run no matter the platform (although I don't know a single UNIX variant that is that small these days).

[James R. Ferguson]

Hi Jerry:

If you examine the patch notes for PHKL_16751 for 10.20 you will note:

/* begin_quote */

When this patch is installed the default environment size is 20478 bytes. To enable the system to use the larger environment size of 2048000 bytes, the following steps must be followed.

1. A new tunable called 'large_ncargs_enabled' must be defined in the sytem file in the following manner

large_ncargs_enabled 1

2. A new kernel must be built (using this system file) and the system rebooted.

/ *end_quote */

You can find the 10.x patch archive from here:

http://www1.itrc.hp.com/service/patch/mainPage.do

You can see the current value of ARG_MAX thusly:

# getconf ARG_MAX

Regards!

...JRF...