UNIX Bourne Shell Scripting

Shell scripting involves chaining several UNIX commands together to accomplish a task. For example, you might run the 'date' command and then use today's date as part of a file name. I'll show you how to do this below.

Some of the tools of the trade are variables, backquotes and pipes. First we'll study these topics and also quickly review a few other UNIX topics.

• Topics covered: storing strings in variables
• Utilities covered: echo, expr

• To try the commands below start up a Bourne shell:

  /bin/sh
  

• A variable stores a string (try running these commands in a Bourne shell)

  name="John Doe"
  echo $name
  

• The quotes are required in the example above because the string contains a special character (the space)
• A variable may store a number

  num=137
  

• The shell stores this as a string even though it appears to be a number
• A few UNIX utilities will convert this string into a number to perform arithmetic

  expr $num + 3
  

• Try defining num as '7m8' and try the expr command again
• What happens when num is not a valid number?
• Now you may exit the Bourne shell with

  exit
  

• Topics covered: specifying the input or capturing the output of a command in a file
• Utilities covered: wc, sort

• The wc command counts the number of lines, words, and characters in a file

  wc /etc/passwd
  wc -l /etc/passwd
  

• You can save the output of wc (or any other command) with output redirection

  wc /etc/passwd > wc.file
  

• You can specify the input with input redirection

  wc < /etc/passwd
  

• Many UNIX commands allow you to specify the input file by name or by input redirection

  sort /etc/passwd
  sort < /etc/passwd
  

• You can also append lines to the end of an existing file with output redirection

  wc -l /etc/passwd >> wc.file
  

• Topics covered: using UNIX pipes
• Utilities covered: sort, cat, head

• Pipes are used for post-processing data
• One UNIX command prints results to the standard output (usually the screen), and another command reads that data and processes it

  sort /etc/passwd | head -5
  

• Notice that this pipe can be simplified

  cat /etc/passwd | head -5
  

• You could accomplish the same thing more efficiently with either of the two commands:

  head -5 /etc/passwd
  head -5 < /etc/passwd
  

• For example, this command displays all the files in the current directory sorted by file size

  ls -al | sort -n -r +4
  

• The command ls -al writes the file size in the fifth column, which is why we skip the first four columns using +4.
• The options -n and -r request a numeric sort (which is different than the normal alphabetic sort) in reverse order

• Topics covered: processing columnar data
• Utilities covered: awk

• The awk utility is used for processing columns of data
• A simple example shows how to extract column 5 (the file size) from the output of ls -l

  ls -l | awk '{print $5}'
  

• Cut and paste this line into a Bourne shell and you should see a column of file sizes, one per file in your current directory.
• A more complicated example shows how to sum the file sizes and print the result at the end of the awk run

  ls -al | awk '{sum = sum + $5} END {print sum}'
  

• In this example you should see printed just one number, which is the sum of the file sizes in the current directory.

• Topics covered: looping over files specified with wildcards
• Utilities covered: no new utilities

• File names are hard-coded inside the script
• What if you want to run the script but with different file names?
• To execute for loops on different files, the user has to know how to edit the script
• Not simple enough for general use by the masses
• Wouldn't it be useful if we could easily specify different file names for each execution of a script?

• Topics covered: specifying command-line arguments
• Utilities covered: no new utilities

• Command-line arguments follow the name of a command

  ls -l .cshrc /etc
  

• The command above has three command-line arguments

  -l	(an option that requests long directory listing)
  .cshrc	(a file name)
  /etc	(a directory name)
  

• An example with file name wildcards:

  wc *.sh
  

• How many command-line arguments were given to wc? It depends on how many files in the current directory match the pattern *.sh
• Use 'echo *.sh' to see them
• Most UNIX commands take command-line arguments. Your scripts may also have arguments

• Topics covered: testing conditions, executing commands conditionally
• Utilities covered: test (used by if to evaluate conditions)

• This will be covered on the whiteboard
• See Chapter 8 of the book

• Topics covered: search patterns for editors, grep, sed
• Utilities covered: no new utilities

• Zero or more characters: .*

  grep 'provided.*access' sdsc.txt
  sed -e 's/provided.*access/provided access/' sdsc.txt
  

• Search for text at beginning of line

  grep '^the' sdsc.txt
  

• Search for text at the end of line

  grep 'of$' sdsc.txt
  

• Asterisk means zero or more the the preceeding character

  a*     zero or more a's
  aa*    one or more a's
  aaa*   two or more a's
  

• Delete all spaces at the ends of lines

  sed -e 's/ *$//' sdsc.txt > sdsc.txt.new
  

• Turn each line into a shell comment

  sed -e 's/^/# /' sdsc.txt
  

• Attributes of pattern matching
• Greed: a regular expression will match the largest possible string
• Execute this command and see how big a string gets replaced by an underscore

  echo 'Big robot' | sed -e 's/i.*o/_/'
  

• Eagerness: a regular expression will find the first match if several are present in the line
• Execute this command and see whether 'big' or 'bag' is matched by the regular expression

  echo 'big bag' | sed -e 's/b.g/___/'
  

• Contrast with this command (notice the extra 'g')

  echo 'big bag' | sed -e 's/b.g/___/g'
  

• Explain what happens in the next example

  echo 'black dog' | sed -e 's/a*/_/'
  

• Hint: a* matches zero or more a's, and there are many places where zero a's appear
• Try the example above with the extra 'g'

  echo 'black dog' | sed -e 's/a*/_/g'
  

• Topics covered: clarify double meaning of asterisk in patterns
• Utilities covered: no new utilities

• Asterisk used in regular expressions for editors, grep, sed
• Different meaning in file name wildcards on command line and in find command and case statement (see below)

  regexp  wildcard  meaning
  
  .*      *         zero or more characters, any type
  .       ?         exactly one character, any type
  [aCg]   [aCg]     exactly one character, from list: aCg
  

• Regexps can be anchored to beginning/ending of line with ^ and $
• Wildcards automatically anchored to both extremes
• Can use wildcards un-anchored with asterisks

  ls *bub*
  

• Sequence of statements that can be called anywhere in script
• Used for
  • Good organization
  • Create re-usable sequences of commands

• Define a function

  echo_it () {
    echo "In function echo_it"
  }
  

• Use it like any other command

  echo_it
  

• Put these four lines in a script and execute it

• Functions can have command-line arguments

  echo_it () {
    echo "Argument 1 is $1"
    echo "Argument 2 is $2"
  }
  echo_it arg1 arg2
  

• When you execute the script above, you should see

  Argument 1 is arg1
  Argument 2 is arg2
  

• Create a script 'difference.sh' with the following lines:

  #!/bin/sh
  echo_it () {
  	echo Function argument 1 is $1
  }
  echo Script argument 1 is $1
  echo_it Barney
  

• Execute this script using

  sh difference.sh Fred
  

• Notice that '$1' is echoed twice with different values
• The function has separate command-line arguments from the script's

• Use functions to organize script

  read_inputs () { ... }
  compute_results () { ... }
  print_results () { ... }
  

• Main program very readable

  read_inputs
  compute_results
  print_results
  

• Can use a function in a pipe

  ls_sorter () {
    sort -n +4
  }
  ls -al | ls_sorter
  

• Function in pipe executed in new shell
• New variables forgotten when function exits

• Variables defined before calling script available to script

  func_y () {
    echo "A is $A"
    return 7
  }
  A='bub'
  func_y
  if [ $? -eq 7 ] ; then ...
  

• Try it: is a variable defined inside a function available to the main program?

• Functions are like separate scripts
• Both functions and scripts can:
• Use command-line arguments

  echo First arg is $1
  

• Operate in pipes

  echo "test string" | ls_sorter
  

• Return exit status

  func_y arg1 arg2
  if [ $? -ne 0 ] ...
  

• Common to store definitions of favorite functions in a file
• Then execute file with

  . file
  

• Period command executes file in current shell
• Compare to C shell's source command

• Data contained within script

  cat << END
  This script backs up the directory
  named as the first command-line argument,
  which in your case in $1.
  END
  

• Terminator string must begin in column one
• Variables and backquotes translated in data
• Turn off translation with \END

• Send e-mail to each of several users

  for name in login1 login2 login3
  do
    mailx -s 'hi there' $name << EOF
    Hi $name, meet me at the water
    fountain
  EOF
  done
  

• Use <<- to remove initial tabs automatically

• Can change Bourne shell's options at runtime
• Use set command inside script

  set -v
  set +v
  set -xv
  

• Toggle verbose mode on and off to reduce amount of debugging output

• Can change Bourne shell's options

  set -- word1 word2
  echo $1, $2
    word1, word2
  

• Double dash important!
• Word1 may begin with a dash, what if word1 is '-x'?
• Double dash says "even if first word begins with '-', do not treat it as an option to the shell

• Read a line from keyboard
• Echo words 3 and 5

  read var
  set -- $var
  echo $3 $5
  

• Best way to split a line into words

• Signals are small messages sent to a process
• Process interrupted to handle signal
• Possibilities for managing signal:
  • Terminate
  • Ignore
  • Perform a programmer-defined action

• Common signals are
  • SIGINTR sent to foreground process by ^C
  • SIGHUP sent when modem line gets hung up
  • SIGTERM sent by kill -9
• Signals have numeric equivalents

  2 SIGINTR
  9 SIGTERM
  

• Send a signal to a process

  kill -2 PID
  kill -INTR PID
  

• Handling Signals

  trap "echo Interrupted; exit 2" 2
  

• Ignoring Signals

  trap "" 2 3
  

• Restoring Default Handler

  trap 2
  

• See file

  /usr/include/sys/signal.h
  

• SIGUSR1, SIGUSR2 are for your use
• Send to a process with

  kill -USR1 PID
  

• Default action is to terminate process

• Script that catches USR1
• Echo message upon each signal

  trap 'echo USR1' 16
  while : ; do
    date
    sleep 3
  done
  

• Try it: does signal interrupt sleep?

• Common translations include
  • Splitting at spaces, obey quotes
  • $HOME -> /users/us/freddy
  • `command` -> output of command
  • I/O redirection
  • File name wildcard expansion
• Combinations of quotes and metacharacters confusing
• Resolve problems by understanding order of translations

• Try wildcards in echo command

  echo b*
  b budget bzzzzz
  

• b* translated by sh before echo runs
• When echo runs it sees

  echo b budget bzzzzz
  

• Echo command need not understand wildcards!

• Splits into words at spaces and tabs
• Divides commands at

  ; & | && || (...) {...}
  

• Echos command if -v
• Interprets quotes
• Performs variable substitution

• Performs command substitution
• Implements I/O redirection and removes redirection characters
• Divides command again according to IFS
• Expands file name wildcards
• Echos translated command if -x
• Executes command

• Delayed expansion for variable assignments

  VAR=b*
  echo $VAR
    b  b_file
  

• Wildcard re-expanded for each echo

• Variables translated before execution
• Can store command name in variable

  command="ls"
  $command
    file1 file2 dir1 dir2...
  

• Variables translated before I/O redirection

  tempfile="/tmp/scriptname_$$"
  ls -al > $tempfile
  

• Delayed expansion of wildcards in variable assignment
• Output of this echo command changes when directory contents change (* is re-evaluated each time the command is run)

  x=*
  echo $x
  

• Can view values stored in variables with

  set
  

• Try it: verify that the wildcard is stored in x without expansion

• Wildcards expanded after redirection (assuming file* matches exactly one file):

  cat < file*
    file*: No such file or directory
  

• Command in backquotes expanded fully (and before I/O redirection)

  cat < `echo file*`
    (contents of file sent to screen)
  

• Forces an extra evaluation of command

  eval cat \< file*
    (contents of matching file)
  

• Backslash delays translation of < until second translation

• Execute statements while a condition is true

  i=0
  while [ $i -lt 10 ]
  do
    echo I is $i
    i=`expr $i + 1`
  done
  

• Execute statements as long as a condition is false

  until grep "sort" dbase_log > /dev/null
  do
    sleep 10
  done
  echo "Database has been sorted"
  

• Example executes until grep is unsuccessful

• Can redirect output of a loop

  for f in *.c
  do
    wc -l $f
  done > loop.out
  

• Loop runs in separate shell
• New variables forgotten after loop
• Backgrounding OK, too

• Used in for, while, and until loops
• Skip remaining statements
• Return to top of loop

  for name in *
  do
    if [ ! -f $name ] ; then
      continue
    fi
    echo "Found file $name"
  done
  

• Example loops over files, skips directories

• Used in for, while, and until loops
• Skip remaining statements
• Exit loop

  for name in *
  do
    if [ ! -r $name ] ; then
      echo "Cannot read $name, quitting loop"
      break
    fi
    echo "Found file or directory $name"
  done
  

• Example loops over files and directories, quits if one is not readable

• Execute one of several blocks of commands

  case "string" in
  pattern1)
    commands ;;
  pattern2)
    commands ;;
  *) # Default case
    commands ;;
  esac
  

• Patterns specified with file name wildcards

  quit) ...
  qu*)   ...
  

• Read commands from keyboard and interpret
• Enter this script 'case.sh'

  echo Enter a command
  while read cmd
  do
    case "$cmd" in
      list) ls -al ;;
      freespace) df . ;;
      quit|Quit) break ;;
      *) echo "$cmd: No such command" ;;
    esac
  done
  echo "All done"
  

• When you run it, the script waits for you to type one of:

  list
  freespace
  quit
  Quit
  

• Try it: modify the example so any command beginning with characters "free" runs df

• Infinite loop with while

  while :
  do
    ...
  done
  

• : is no-op, always returns success status
• Must use break or exit inside loop for it to terminate

• Rsh command

  rsh hostname "commands"
  

• Runs commands on remote system
• Must have .rhosts set up
• Can specify different login name

  rsh -l name hostname "commands"
  

• Check who's logged on

  rsh spooky "finger"
  

• Run several remote commands

  rsh spooky "uname -a; time"
  

• Executes .cshrc on remote system
• Be sure to set path in .cshrc instead of .login

• May get "permission denied" error from rsh
• Fix this with ~/.rhosts on remote system
• Example: provide for remote shell from spunky to spooky

  spunky % rlogin spooky
  spooky % vi ~/.rhosts
  	(insert "spunky login-name")
  spooky % chmod 600 ~/.rhosts
  spooky % logout
  spunky % rsh spooky uname -a
  	spooky 5.5 sparc SUNW,Ultra-1
  

• May also rlogin without password: security problem!

• Standard output sent to local host

  rsh spooky finger > finger.spooky
  

• Standard input sent to remote host

  cat local-file | rsh spooky lpr -
  

• Get return status of rsh

  rsh mayer "uname -a"
  echo $?
  

• Returns 0 if rsh managed to connect to remote host
• Returns 1 otherwise
  • Invalid hostname
  • Permission denied

• What about exit status of remote command?
• Have to determine success or failure from stdout or stderr

• Use unique names to avoid clashes

  tempfile=$HOME/Weq_$$
  command > $tempfile
  

• $$ is PID of current shell
• Avoids conflict with concurrent executions of script
• Do not use /tmp!

• Wait for termination of background job

  command &
  pid=$!
  (other processing)
  wait $pid
  

• Allows overlap of two or more operations

• Provide control of collapsing of spaces and translation of variables
• Try it: run three examples
• No quotes (variables translated, spaces collapsed)

  echo Home:   $HOME
    Home: /users/us/freddy
  

• Double quotes (no collapsing)

  echo "Home:   $HOME"
    Home:   /users/us/freddy
  

• Single quotes (no translation or collapsing)

  echo 'Home:   $HOME'
    Home:   $HOME
  

• Try it: single quotes within double quotes

  echo "Home directory '$HOME' is full..."
  

• Characters with special meaning to shell

  " ' ` $ * [ ] ?
  ; > < & ( ) \
  

• Avoid special meaning with quoting

  echo 'You have $20'
  

• Backslash like single quotes
• Applies only to next character

  echo You have \$20
  

• Bad command line:

  grep dog.*cat file
  

• Shell tries to expand dot.*cat as file name wildcard
• Use quotes to avoid translation

  grep 'dog.*cat' file
  

• Single quotes OK in this case because we don't need variable translation

• Read name and search file for name

  read name
  grep "$name" dbase
  

• Single quotes not OK because we need variable translation

• Bad command line: search for dollar sign

  grep "Gimme.*$20" file
  

• Problem: shell translates variable $20
• Solution: use single quotes

  grep 'Gimme.*$20' file