Bash Shell Script help desired

[John Collier]

I have a task in front of me that I know I could simplify with a script, but I am now seeing just how little I actually understand about scripting.

Basic situation:

I have a directory full of subdirectories. Within these subdirectories there are typically about 500 files with several different extensions. I need to move all of the files in 150 directories to a single directory.

The directories containing the files that need to be moved all have the pattern "recup_dir.#" in common with # being a number between 1 and 150.

I know I can move the contents of a single directory by simply typing the command:

mv /media/disk/Sara/recup_dir.1/* /media/disk/Sara/Consolidate/

The for the next one I can type:

mv /media/disk/Sara/recup_dir.2/* /media/disk/Sara/Consolidate/

And I can keep this pattern up until I get all the way through recup_dir.150

Of course, this is a lot of typing and it will take much more time than I really want to take.

The shell script I tried to create was:

#!/bin/sh

a = 1

if $a < 150

then mv /media/disk/Sara/recup_dir.$a/* /media/disk/Sara/Consolidate/ ; $a=$a+1

if $a = 151 ; then

fi

When I try to run it, I get the following errors:

./consol.sh
./consol.sh: line 3: a: command not found
./consol.sh: line 11: syntax error near unexpected token `fi'
./consol.sh: line 11: `fi'

A little assistance in getting this hashed out to where I can accomplish what I want/need would be greatly appreciated.


On a foot note, any "BASH Scripting for Dummies" type references would also be appreciated. It seems like I could use some!

[Stuart Browne · Accepted solution]

There's the for-loop way (using the 'seq' command (easiest)), and then there's the while-loop way (using inbuilt stuff only). I'll show both:

-=-=-=-=-=-=- For loop -=-=-=-=-=-=-
#!/bin/bash
src=/media/disk/Sara/
DST=/media/disk/Sara/Consolidate/
for COUNT in $(seq 1 150)
do
mv ${SRC}/recup_dir.${COUNT}/* $DST
done
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-


-=-=-=-=-=-=- While loop -=-=-=-=-=-=-
#!/bin/bash
src=/media/disk/Sara/
DST=/media/disk/Sara/Consolidate/
COUNT=1
while [ $COUNT -lt 151 ]
do
mv ${SRC}/recup_dir.${COUNT}/* $DST
let $((COUNT++))
done
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Discussion:

The for-loop method creates a list of values with the 'seq' command (values 1 -> 150) in a subshell (the $() bit, could also use back-ticks (`)), and uses environment space to store it. It then selects them one-by-one based on the IFS (input-field-separator, usually space-tab-newline).

This method works beautifully for a relatively small set of values (up to 2-3KB of environment space). When you try putting too large a list into the environment space, you'll literally get an environment space error, and it just won't work.

But that's not to say it's a bad way to do things, just be aware of your situation.

Now, the while-loop. This particular method is possible due to the simple sequential nature of the directory names. It does simple arithmatic ($((VAR++)) -> c-style variable increment), with a simple less-than comparison. It's a bit longer because of only that.

As the comparison happens at the beginning, the variable needs to be set before the loop.

That's about it really. Looping in shell.. ;)

Now, as for beginners guide.. I'm not really sure of one. The one I always go back to is:

http://tldp.org/LDP/abs/html/

The advanced bash scripting guide. It does have some basics in it.

[John Collier]

Stuart,

I guess you just sit around on the weekends waiting for some poor slob like me to post so you can rush in and be the hero.

While this makes me worry about your personal life, I have to admit that I am glad you do!

I suppose I don't have to tell you that your solution worked like a charm. All of the files moved just as I wanted/needed them to do. (all 31,082 of them!)

I have to guess that you just saved me about 2 hours of manual labor on that one. I can't thank you enough.

As for the link you provided, I have already bookmarked it and I will try to use it to assist me in the future.

I do have one silly question, though. I have noticed that all of the people that provide links to scripting guides seem to provide those links to advanced type guides. I know that everybody had to start off crawling rather than running so there has to be some beginners references out there.

How did you get your basic knowledge to expand on? I'm sure you and most of the others weren't simply born knowing it.... were you??

[Stuart Browne]

Whilst I don't just sit around waiting for posts on the weekend, I do poke the computer every few hours and have a look-see.. I did once write a script which would check if there were any new forum posts (it would check every minute), and play a sound.. but that was when I really did have nothing better to do :) It's turned off currently.

I started learning DOS when I was about 13. From there, I learnt (from DOS manuals and examples) how to write batch scripts, how ERRORLEVEL's worked (read: exit states), and what you could do with the standard tool set.

This gave me a considerable head start into how shell scripting works, as many of the constructs and concepts are similar.

I started using and learning Linux when I was 19, over a decade ago now (wow, I'm getting old.. scary!).

I've always had the insatiable desire to figure out how something works. How does the machine start up? what does it do? where does it look? how did that installer work? why doesn't that startup routine work? etc. etc. etc..

This lead inevitably to taking my first linux box apart bit by bit, and putting it back together. That was a 468DX266 (it's in the cupboard here) who's hardrive finally burnt out after 5-6 years of constant operation. It was once a slackware box, but I changed so many things on it, it was it's own unique flavour by the time it died.

Some of the most important things I learnt from that experience is this:

'man' pages are the proveribal 'mothers milk'. -> man bash.. yay! WIN!

Learn through destruction! *mwahaha!* Break it... then Fix it. *ALWAYS* have a machine that you can muck around on, something on which it doesn't matter if you break it. This way, you can just get more learning experience from the break/fix cycle than you could ever get on a production 'working' machine which you cannot break.

Hrm.. how did this turn out to be a personal history lesson? Anyway :)

If you see something that looks like it does something funky, open it up! Shell scripts are just text files, so are easy to disect (unless they are decoded and executed on the fly (been there, done that, not too exciting)).

In shell scripts, use '-x'. Either as 'bash -x', or as 'set -x' in an existing shell. It's ugly as hell, spits out far too many things, but it tells you what it's doing. Very good for learning (and debugging!).

Don't be afraid of it. It's really not that scary :) No monsters in the cupboard here :)

But as with this original post, get a task. Solve it using a tool.

If you're anything like me, learning without direction is difficult; with an end goal however it is much simpler.

[John Collier]

I thought I did fairly well with the good old fashioned DOS batch files when I was a little bit (OK, a LOT) younger. I remember passing a 'test' in HS computer class by simply taking a 1.44 floppy and redirecting failed password attempts through several directories until it finally dead ended and locked the whole process, requiring a reboot to get out of. There was no winning combination and the final result was always a locked machine, but the teacher never figured that out and gave me an A on the assignment. Scared me to find out the 'teacher' was that far below the scale...

Anyway, I can't really explain why I am having such a hard time transferring the good old fashioned DOS knowledge to the BASH scripting. I have to admit that it is very frustrating to me. I never remember having this much of a challenge with any other task I have attempted to do around computers.

While I have pointed many people to "For Dummies" type references in the past I have never, until now that is, even thought of looking at one myself. Never needed it.

I like the man pages for many things, but for some reason I just don't get much out of them for the scripting thing. Seem to get lost in them, which is a bit scary. Not really used to that.

Quantum Physics seems to be more easily followed than this scripting thing for me.

Not used to asking for this level of help, but I do know, beyond a shadow of a doubt, that even this will pass and I will eventually conquer even this task. Right now, I just need to figure out how to take the first steps...

[emha_1]

Hi John,

seeing your profile and seeing your over thousand points in HPUX area, I would say you would be able to write a command that would copy everything you need in 5 letters :)

however, nobody can know everything...
Why to complicate it, when it can be solved in one line. Assuming there are not folders you need to skip (e.g. *151, *152...) I would simply use:
mv /media/disk/Sara/recup_dir.*/* /media/disk/Sara/Consolidate/

on some older platforms it could cause problems if you exceed max size of command line or environment space ('*' are first expanded then executed). there I would use (not so 'nice' but still one-line comand and still better than fighting with loop):
find /media/disk/Sara/recup_dir.*/* -exec mv {} /media/disk/Sara/Consolidate/ \;

it's written by head and not tested, so excuse some stupid bugs :)

emha.