Trying to move (mv) large numbers of files

[Erik Olson]

I have had this problem for a while, but it is becoming more of a prob, as I am trying to clean house.
If my wildcard returns too many files, I get:
'arg list too long'
As in:
# mv ap* ..
ksh: /usr/bin/mv: arg list too long
This is probably a stupid question, but is there a parameter I can set to extend this 'arg list'
Thanks, Erik

[someone_4]

Hello
when you do
# mv ap* ..
are you actully using the .. at the end?
if you man on mv you can see it says:
mv cannot be used to perform the following operations:

+ Rename either the current working directory or its parent
directory using the . or .. notation.

+ Rename a directory to a new name identical to the name of a
file contained in the same parent directory.

try the full pathname instead of ..

richard

[Alan Wyskowski · Accepted solution]

I had a similar problem a while back ago. HP introduced a kernel parameter that increases the number of bytes of memory for the listing. I think it's something like nclist or something and it goes in your kernel. Once you get the files moved out of the directory, remove and remake the directory because the inode list is still so long. I would use find to get the files moved. I'll see if I can dig up those kernel parameters - what version of HPUX are you on.
find . -type f -print | xargs -i mv {} wherever

[Erik Olson]

I used a bad example...It does not seem to matter what command I use, I getthe same results.
Example of simple rm:

# rm ap*
ksh: /usr/bin/rm: arg list too long

[Soren Morton]

I have run into this before and I don't think that I ever found a fix. You might just try something like the following.

for i in `find . -name ap\*`
do
mv $i ..
done

[Erik Olson]

This worked to move or remove all files in the directory...Thanks!!
I still have the problem if I want to be selective, having to use a wildcard with text.
example 'rm ap*'
The find command gave the same error:
'arg list too long'
I'm on 10.20
Thanks, again
Erik

[someone_4]

Ok ..
lets try this .. I found this doc here does this help?


SYS ADM: Error when moving large group of files - arg list is too long.

Current Path Home
Score : 0
Document Type : EN
Date : 1998 Aug 16
Description : SYS ADM: Error when moving large group of files - arg list is too long.
Document Id : A5218911
Search String :

You may provide feedback on this document

View the printer friendly version of this document



--------------------------------------------------------------------------------

Problem Description

I am trying to move a group of files that are specified with
a wildcard. I am getting this error:

arg list is too long.

Why am I getting this message?

Configuration Info

Operating System - HP-UX
Version - 10.20
Hardware System - HP9000
Series - D350

Solution

The file names are about 45 characters, and you are trying to
move about 550 files.

500 files * (45 char / file) = 24750 characters

The ARG_MAX value is 20480.

Solve the problem by installing this patch:

PHKL_15457

The above patch has these dependencies:

PHCO_14842
PHCO_14990
PHNE_14770
PHKL_12340

Note: Patches can be superseded by subsequent versions;
be sure to load the current version.

[Bill Hassell]

No matter how large you can make the command line, there will be a possibility of exceeding the command line's maximum length. Here's what is happening:

rm ap*

does not pass the * character to rm! The shell sees the special * character and translates it to match the list of files so the rm program sees:

rm ap1 ap2 ap333 ap343 ap5 ap6 ... and so on

So if you have 5 million files and the average length of each file name is 10 characters, the the command line passed to rm must hold 55 million bytes. So the only way to process an unlimited number of arguments is to use xargs or find.

If you use find, remember that the shell will try to expand the * so it must be escaped as in:

find /some_directory -name ap\* -exec ll {} \;

In the above example, -exec ll will verify that the right filenames will be selected. Then replace ll with /usr/bin/rm. Do not use rm without /usr/bin just to make sure what command you are using.

[Suhas_3]

The lenght of command argument is ARG_MAX parameter ( in /usr/include/limits.h ). I see it is 20478 in my system. It is mentioned in the file that the parameter is configurable. I have no idea how can it be configured, but you can refer the file to see if oyu get any clue - following are extracts from the file -
START ------------------
/*
* The following limits are not actually invariant, but are configurable.
* The correct values can be determined using the sysconf() function.
* The default values are provided here because the constants are specified
* by several publications including XPG (X/Open Portability Guide) Issue 2
* and SVID (System V Interface Definition) Issue 2.
*/
# define ARG_MAX 20478 /* Maximum length of arguments for the
exec function in bytes, including
environment data */
END-----------------------------

[Erik Olson]

Thanks to all...
I will probably always run into this problem, as Bill has stated, there will always be a limit...
The escaped wildcard in the find or xargs will be my solution.
Thank, for this great forum.
God Bless America.
Erik