moving files from a dir which contains a HUGE amount

[Christian Schulze]

We have a incoming-directory accessible via nfs and smb that holds tons of files from other systems.
Filename is in Format YYYYMMDDHHmmss.some_other_stuff
so its easy to get the oldes files.

Now I need to move the 1000 oldest files (not timestamp, but filename) from there to another directory, where another process is working on them.

this is how we managed it:

ls -1 $XML_DIR | head -n 1000 | xargs -i mv ${XML_DIR}/{} ${WORK_DIR}/{}

It works, we dont get any "arg list too long errors"
But this executes mv for every File and thus takes a long time.


the -i option is my only chance to get the:
mv source target


Any ideas?

[Steven E. Protter]

Use your initial methodology as follows.

ls -1 $XML_DIR | head -n 1000 > filelist

# ser variable destdirectory

while read -r filename
do
mv $filename $destdirectory
done < filelist

This will get around the arg list too long error

SEP
Israel

[Mel Burslan]

So, if I am getting this correctly, your process is working but taking a long time and you are looking for ways to shorten this time. Am I right ? If I am, I may have bad news for you. Unless you are moving files on the same filesystem, i.e., both source and destination firectories are on the same mountpoint, mv command will work as copy first, verify and delete fashion. If you want to move files quickly and can afford to have both directories on the same filesystem, it will speed up the process otherwise, you have to live with it.

[D Block 2]

Sounds like the 'WORK_DIR' is not part of the Share.

Can you make the 'WORK_DIR' as a sub-directory on the same nfs/smb share ? This way there should be a link/unlink rather than a data copy.

just my 2-cents.

[Christian Schulze]

Thanks SEP

But with the filelist I get executed a single mv for each file.

But if I would get a filelist-File containing 10 filenames in a line, separated with blanks,
then each mv would move 10 files.

So I have to generate a filelist to give the following arguments to the mv:

mv file1 file2 file3 file4 file5 destination_dir

But how to do this ?
ls -x ?
how does this behave, when executed in Background (eg. not from a Terminal)?


Christian

[Christian Schulze]

Sorry I forgot to mention that I am moving in the same filesystem ...

Christian

[Mel Burslan]

i=1
while [ $i -le 100 ]
do
MOVELIST=`ls -1 | head -10`
mv $MOVELIST $WORKDIR
(( i=$i+1 ))
done

this should move 10 files at a time for 100 times totaling the movement of 1000 files with the same logic.

Hope this helps

[Rick Garland]

What about using a find command and pipe that through cpio with passthru?

[Jean-Luc Oudart]

Well,

ususally I use -l option in xargs
option -t can be used to trace the command
man xargs for more information

I created a small script movethem :
mv $* ${TARGET_DIR}

then
ls -1 $XML_DIR | head -n 1000 | xargs -l10 movethem

If you take the -t option you will see how often then mv command (or movethem script) is called

Regards
Jean-Luc

[TwoProc]

I know that when I use xargs it *doesn't* pass just one file at a time. And I've watched it as proof by watching the ps command ("ps -ef | grep mv") in your case.

So, are you sure it's only moving one file at a time? Did you watch it with ps?

Let's say it is doing only one at time - it's probably b/c you're using the argument replacement {} brackets. Although I've not checked it out by watching it with ps - this makes sense b/c I can't figure out how it would do it otherwise using replacement brackets.

Now, based on what I know, it seems that we could make it work if the xargs portion of the command ran more like | xargs mymovecommand (and therefore didn't use the brackets).

So we create a little command file:
cat > /var/tmp/processmove
mv $* $WORKDIR
ctrl-d

make it executable
chmod u+x processmove

Now...

ls -1 $XML_DIR | head -n 1000 | xargs /var/tmp/processmove


OK, you should be moving many many files at once now.

P.S. I put the "processmove" command in a shell script b/c I haven't yet figured out to make xargs accept passing commands to shell functions. My first inclination would naturally be to define
processmove () { move $* $WORKDIR }
and let xargs call that. But, I've never gotten that to work. Maybe someone else knows how to make xargs run a shell function.

P.P.S. If you're moving to a different file system - this is really going to load up the ol' I/O schedule. Also, if you do let it move to different file systems this way - you may want to defrag (if you've got online jfs) as this will be 1000 heavily interleaved/intertwined chained files. If you are moving these files to the same filesystem the above command(s) will not have any effect on fragmentation.

[D Block 2]

how about this one...

can you create a number of 'moves' in the background and wait for there completion, then start up another set of 'moves' ?

my thinking is, the nfs share will be able to handle a number of threads concurrent, so why not try to run the jobs (only a few) in the background, until conpletion.

Then start up then set stage of jobs in the background, and then wait for completion.

Multitasking a few at a time, might do the trick for the CIFS server.

[Christian Schulze]

Hi, thanks to all

I managed it this way:

ls -1 DIR |head -n 1000 | sed "s:^\(.*\)$:DIR/\1:" | xargs -L 20 echo | sed "s:^\(.*\)$:\1 DIR2:" | xargs -l mv

this helped us a lot.

Points will follow


Christian

[Christian Schulze]

Thread closed

Thanks