Moving Top 100 files matching a criteria specified in 'ls' command to a directory

[khalidM]

We have a folder (suppose Folder1) which has more than 250,000 files of format like "Filename_23Dec2009_update.log".

From this folder we need to move 2000 files to two other folders (suppose Folder_A and Folder_B)

here is the criteria,
--> I want to move all files which have 23Dec2009*.log appearing in their name.
--> In a single run of script, i want to move top 1000 files to Folder_A and bottom 1000 files to Folder_B
--> By Top & Bottom i mean the listing that will appear by "ls" command.


Any Help ?

reference thread : ?threadId=1378743

[Pete Randall]

Here is the URL of the thread being referred to:

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1378743


Pete

[James R. Ferguson]

Hi:

Here's a suggestion for you to build upon. Use 'ls -l' to insure that you only look at files. A line that begins with '-' is a regular file and thus a candidate. You can use 'grep' to filter these.

Use 'head' and 'tail' to filter the first and last files and using these subsets, move your files to the appropriate directory. The word "folder" is a Windows term. The correct term in UNIX is "directory".

Regards!

...JRF...

[Dennis Handly]

>We have a directory (suppose Folder1) which has more than 250,000

I assume you know by now that this is too many files in a directory?

>From this folder we need to move 2000 files

Are there more than 2000 files matching your pattern and some are to be left?

>JRF: Use 'ls -l' to insure that you only look at files.

Note: You only want to do that ll(1) once since you have too many stinkin' files:
ll | awk '/^-.*23Dec2009.*\.log/ { print $9 }' > file_list

[Steven Schweda]

> ll | awk [...]

Wouldn't "find -name" be simpler?

[James R. Ferguson]

Hi (again):

> Steven: Wouldn't "find -name" be simpler?

I might agree if we could assume that the source directory has no subordinate directories with files that would match the OP's criteria and that the output is sorted so the OP can snip n-files off the top and bottom.

Regards!

...JRF...

[Dennis Handly]

>Steven: Wouldn't "find -name" be simpler?

Only if you know how to use -prune and sort.

>JRF: that the output is sorted so the OP can snip n-files off the top and bottom.

The output won't be sorted and you would need to pipe it to sort(1).

[khalidM]

I got two ways to do it ... but when there are too many files matching the criteria, both of these commands reply me with "argument list too long" and the command aborts...
Once again i tell you that there are thousands of file i.g more than 40k

here are both
1.
ls -ltr /apps/metrica/Spool/gsm_alcatel_bss_b9/load/bad/*18Dec2009*.lif | head -1000 | nawk `{print $9}` | xargs -I -t mv {} /apps/metrica/Spool/gsm_alcatel_bss_b9/load/

2.
mv `ls -1 /apps/metrica/Spool/gsm_alcatel_bss_b9/load/bad/*17Dec2009*.lif | tail -1000` /apps/metrica/Spool/gsm_alcatel_bss_b9/load/


Now what ? :(

[Dennis Handly]

>but when there are too many files matching the criteria, both of these commands reply with "argument list too long"

We already assumed you would have that problem and my solution would handle the finding of the files.

>1. ls -ltr

If you want them in reverse time order, add that to my solution.
You probably can use just one mv since 1000 is small enough?
mv $(head -1000 file_list) /apps/metrica/Spool/gsm_alcatel_bss_b9/load/

[khalidM]

So,

what will be the final command that i will run... actually, i am unable to understand and neither i am a good scripter ;-(

[Dennis Handly]

>what will be the final command that I will run?

How many times will you do this? You still may want a script even if you do it once.

This first step will give you a list of all files for that pattern, sorted in descending order by time, possibly more than 2000:
ll -tr | awk '/^-.*23Dec2009.*\.log/ { print $9 }' > file_list

If you want more REs to add ".lif" (or was that a typo for .log?), you can add with:
/first/ || /second/

Then you need to do the moves. I'm not sure if 1000 is too many for the shell, you'll have to try it:
mv $(head -1000 file_list) FolderA
mv $(tail -1000 file_list) FolderB
Then cleanup: rm -f file_list

[khalidM]

The command returns no result and file is also empty .


grrrrrrrrrrrr

[Dennis Handly]

>The command returns no result and file is also empty.

The one with awk?
I'm pretty sure I got the ERE correct but I didn't have anything to test. You can toss the ERE and go back to the tried and true RE in grep:
ll -tr | grep '^-.*23Dec2009.*\.log' | awk '{ print $9 }' > file_list

[khalidM]

the system is HP-UX
and ll command not accepted.
neither there is bash hospital :(

[Dennis Handly]

>and ll command not accepted.

There should be /usr/bin/ll, provided you have /usr/bin in your PATH and you aren't in single user mode. Otherwise you can settle and use "ls -ltr".

>neither there is bash

Why would I want to use bash? :-)

[James R. Ferguson · Accepted solution]

Hi (again) Khalid:

> ls -ltr /apps/metrica/Spool/gsm_alcatel_bss_b9/load/bad/*18Dec2009*.lif

Getting an "argument list too long" from commands like this is expected. The shell is expanding the filenames before 'ls' can process the expanded argument list.

You can see this behavior by doing:

# cd /path
# echo *
# ls *
# set -f
# echo *
# ls *
# set +f #...the default restored

That said, now rearrange your pipe to snip out the files that match your criteria and then move them. As I originally said, you want to look at only files (not directories). Using the 'awk' Dennis suggested:

# cd /apps/metrica/Spool/gsm_alcatel_bss_b9/load/bad
# ls -l . | awk '/^-.*23Dec2009.*\.lif/ {print $9}' | head -1000 | | xargs -i -t mv {} /apps/metrica/Spool/gsm_alcatel_bss_b9/load

Notice that we used 'awk', not 'nawk' (which is a "new" 'awk' on some UNIX, but not necessary on HP-UX nor probably what you are running on.

We begin by changing into the source directory and doing the 'ls -l .' to search there. This will also mean that 'xargs' will run from this directory.

Notice that the "." in 'awk's regular expression means any character. Hence when we mean ".lif" we escape the dot with a backslash to say that the dot is really a dot only. The caret (^) anchors the "-" to the beginning of the line --- which is what we want to match regular files in the 'ls' output.

In the 'xargs' I changed to the lowercase '-i' for insertion. This is preferred.

Regards!

...JRF...

[Dennis Handly]

>JRF: In the xargs I changed to the lowercase '-i' for insertion. This is preferred.

If you want performance, you don't want to use -i or -l and do the mv(1) one at a time. You just want the default but you need to write a mv_reversed script.
mv_reversed:
#!/usr/bin/ksh
target=$1
shift
mv "$@" $target

Then:
... | xargs -t mv_reversed /apps/metrica/Spool/gsm_alcatel_bss_b9/load

[khalidM]

Thanks for your help guys...