Unix Shell Scripting - Best Practises

[Hunki]

What are the Best practises in Unix Shell Scripting . Lets build a list together and also if somebody can share a great shell scripting template.

Thanks,

[Dennis Handly · Accepted solution]

You may want to put this in the languages and scripting forum. They may have links already to refererence material.

1) Using set -u for uninit vars
2) Using proper quoting, handling string with blanks and other specials.
3) Using sh -n to check scripts.
4) Using set -x -v to debugging.
5) Using typeset -i

[Peter Godron]

Hi,
my best pratice advice is:
1. Comment your code
We have all come across a script we wrote 2 years ago, it's now going to take ages to work out again what the script did.

2. Establish the requirements
Analyse the problem propery, rather than going for a quick fix. Temporary solutions have a nasty habit of turning permanent.

3. Pick the right shell/tool
After point 2 decide what is the best implementation to get the job done.

In general the methods are the same for shell scripts as they are for traditional software languages.

And finally test-debug-test and test again !

[Steven E. Protter]

Shalom,

I created a stub years ago from which I started all shell scripts.

It contained prompts for documentation and contained a change log a the top.

It was very helpful when people used it.

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=51050

There are a lot of examples of best practices in this thread and it should be looked to for guidance.

SEP

[A. Clay Stephenson]

The best practice is to practice a whole lot.

1) Typeset everything.
2) Normal output goes to stdout; error/usage messages go to stderr.
3) Use {}'s around each and every variable.
4) Use X=$(commnd) rather than X=`command`.
5) No news is good news. Don't output stupid messages that say "all is well"; instead, produce output when things are bad.
6) Always return a valid exit status with 0 meaning OK.

Nowadays, if I were starting over, I would learn only enough shell to get by and concentrate on Perl.

[James R. Ferguson]

Hi Hunki:

Be efficient! Think about the number of processes and/or temporary files you are using!

> Don't use 'cat' to read a file and simply pipe the output to a 'read'. Instead do:

while read LINE
do
...
done < file

> Don't use 'grep' to find patterns and then pipe the output to 'awk' to extract a field. Use awk's pattern matching and do the operation with one process!

> Use pipes to glue the output of one process to the input stream of another instead of using temporary files.

> Use 'xargs' instead of '-exec' with 'find' *OR* use '-exec' with the "+" terminator instead of the ";" to achieve the same performance gain. See the 'find' manpages.

> Learn to use traps (signal handlers). This is a clean, handy way to automatically remove temporary files when a shell process terminates (abnormally or normally).

> Use shell built-ins instead of spawning new external processes. Use shell parameter substitution instead of 'basename' and 'dirname', for example.

Regards!

...JRF...

[Patrick Wallek]

My recommendations:

1) ALWAYS use the she-bang syntax on the first line of your script to declare what interpreter (sh, ksh, perl, etc) the script is to use.

2) Do NOT use back-ticks to execute commands. Instead use $() syntax. Like DATE=$(/usr/bin/date +%m%d%Y)

3) Document your script. Use lots of comments so whoever comes in later can figure out what you did.

4) Use the FULL path to executables so you always know what you are running (/usr/bin/date rather than just date).

5) Set your environment in the script. You may not always run the script interactively. You may run it via cron which has a sparse environment by default.

6) Use 'set -u' as mentioned above.

[Bill Hassell]

Another very important shell technique:

- NEVER use the existing $PATH value. Export PATH in your script so you know exactly where the shell will look for your commands. Start with:

export PATH=/usr/bin

and build on that only as necessary. This avoids a *LOT* of environment issues and improves security.

Also make sure that common shell and Unix commands in your script are not aliased. It's common to have 'nice' options aliased (like rm -i) so rm in your script suddenly goes interactive with: (y/n). Use unalias to stabilize commands you are using.

And a great shell scripting site:

http://www.shelldorado.com/

[Hunki]

Re-opened thru popular demand !

[Duncan Edmonstone]

My absolute most hated BAD practice in scripts is people doing:

su - user -c cmd

The use of 'su -' in scripts should be absolutely banned! This will always invoke /etc/profile and .profile and who knows what is in there that is inappropriate for a script. The least of it could be a bunch of terminal dependent code (result: you get lots of stty: not a typewriter messages), and the worst of it could be a DBA who implements a handy interactive menu in the .profile for the oracle user (result - none of your oracle scripts work or work on the wrong database).

The correct way to do this is:

export MY_ENV_VAR=xyz
su user -c cmd

for example:

export ORACLE_HOME=/oracle/10g
export ORACLE_SID=mydb
su oracle -c dbshut

One gotcha to watch for though is that some env vars are removed by su (such as HOME SHLIB_PATH, LD_LIBRARY_PATH - see SU_KEEP_ENV_VARS in security(4) for details) and may need to be reset, e.g.

export ORACLE_HOME=/oracle/10g
export ORACLE_SID=mydb
su oracle -c "export SHLIB_PATH=${ORACLE_HOME}/lib;dbshut"

HTH

Duncan

[Peter Nikitka]

Hi,

my additional recommendations:

1) Use default values for your data, but setup an option scanner (via getopts !) do modify these defaults: so you do not have to change the code, but just supply an option for such changes.

2) Implement a usage output for your scripts, which can be understood by its users.
I use a special comment tag for my usage output; here is my template for this:

#!/usr/bin/ksh
#@@
#@@ start_snav
#@@ initializing and starting SourceNavigator
#@@
#@@ options:
#@@ -f start with a really new configuration
#@@ -h help
#@@ -i don't start the sourcenavigator in the end
...snipped...

usage () { sed -e '/^#@@/!d' -e 's/^#@@//' $0
exit ${1:-1}
}

while getopts :kKifp:hc c
do
case $c in
i) sn_start=n;;
p) sn_proj=${OPTARG%%.proj} opt_p=y;;
...snipped...
?) usage;;
esac
done
shift $((OPTIND-1))

...

The corresponding usage output would read as

start_snav
initializing and starting SourceNavigator

options:
-f start with a really new configuration
-h help
-i don't start the sourcenavigator in the end
-k keep_ref: keep all cross reference files
-K keep_all: don't check the environment and keep all refs
-c generate a current versioninfo in proj.ccs
-p proj initialization with different project names


mfG Peter

[James R. Ferguson]

Hi Hunki:

What a shame that you have closed this thread! There are more contributions with excellent insights still coming!

...JRF...

[Hunki]

Its reopened on popular demand !

[James R. Ferguson]

Hi Hunki:

To add to Duncan's comments, in my opinion, there are better ways to define common environmental variables than by directly declaring them in the user's profile.

Create a separate file of application variables that you need and source (read)
that file when/whereever you need to do so.

This isolates common variables and constants in one place; simplfies maintenance; increases reusability and avoids replication errors if/as changes are made.

For the purposes of this discussion, maintaining a discrete file devoted to
environmental variable declaration, eliminates explicitly sourcing login profiles (as commonly done in 'crontabs') or implicitly sourcing them when running 'su - -c ' merely to obtain a variables and their values.

To 'source' a file, specify a dot character; a space; and the name of the file to source.

This technique can be used as the last operation in a login profile as well as within applicaton scripts during their startup logic.

If you elect, instead, to maintain your application specific variable within the
standard login profile, you can avoid the "not a typewriter" error messages by
encapsulating terminal queries involving 'stty' or 'tset' with:

...
if [ -t 0 ]; then
stty ...
fi

This logic tests to see if STDIN (file descriptor 0) is associated with a terminal. If it is, the process is interacive and can interact with a terminal. If the test fails, there is no associated terminal (perhaps because you 'cron'ed a process and sourced the '.profile' to gain your environmental varibles) and the meaningless 'stty' and 'tset' operations are skipped.

By the way, in the shell, if you want to declare a constant and keep it constant (!), declare it readonly:

# readonly PI=3.14159
# export PI

Within a POSIX shell script, you can use 'typeset -r' to accomplish the same task, too.

Regards!

...JRF...

[Sundar_7]

Hunki,

My style of scripting

* No Hard-coding !!, as much as possible

* Repeatitive codes are to be converted in to function !

* If the action is the same but the input differs, you know looping is the way to go ! I have seen people repeat the same code for different set of inputs over and over. This will stretch a 20 line script in to 100 line one

* Think in long-term. This is especially important for scripts that deals with some dynamic paramters, like the filesystem/logical volume names, volume group names etc to name a few.

When new FSes or volume groups added, we expect the exisiting scripts to handle it just fine without having to rewrite them all over.

* Run the scripts as a user with bare-minimum previleges. I have seen Oracle hot/cold backup scripts run as root, when I dont see any reason why it cannot be run as the oracle user.

It may just work fine except when the output filesystem is unavailable. If run as root, it is going to fill up the parent filesystem but whereas the job will fail if it is run as a normal user.

* Good revision control always helps.

[Heironimus]

At a minimum, always set (or unset, as appropriate) your umask, PATH, SHLIB_PATH, and LD_LIBRARY_PATH.

Don't use a fixed or predictable filename in /tmp or /var/tmp (or any other world-writable directories). Use mktemp, or at least create the filename with $$.

For scripts that run as root, don't use /tmp or /var/tmp at all. root shouldn't write to files in directories that other users can write to.

Use sudo, not su, even if you end up running "sudo -u somebody /usr/bin/sh -c 'blah'". sudo doesn't use the shell in /etc/passwd, so an app account doesn't even need a valid shell for the command to run.

Quote everything. Use single and double quotes in the appropriate places to make sure you know what is and isn't subject to variable interpolation.

Indent your code in a consistent manner. I won't argue styles because nobody can ever agree, just make sure you're consistent. You'll appreciate 4-8 space indents or tabs if you ever have to read it at 3:00am.

Try to stick to POSIX-standard stuff. That will make it easier if you move to a new platform or OS release.

If you start something that runs in the background (either a daemon or something you background with &) make sure you "cd /" first. If you background it with "&" you should probably nohup it too, and always redirect stdout and stderr to files or /dev/null.

[dirk dierickx]

for any language and shell scripting the rule is to check your input and error codes of commands before proceding to do something.

granted, this will in some cases increase the size of your script to something much larger then you first though of, but it is worth it.

[Kevin Nikiforuk]

So, along the lines of checking input and return codes, here's something that I've been struggling with for a long time. What are recommendations on checking on the return codes of piped commands.

E.G.
echo "hi there" | awk '{print $1}' | sed 's/h/H/g'

What happens if awk craps out? If you're going to some sort of check on the output of the pipe and it's now an empty string because awk didn't forward anything on, it could muck things up quite nicely, couldn't it?

For the longest while, I've tried doing things like:

( echo "hi there" ; echo $? > $ERRFILE ) |
( awk '{print $1}' ; echo $? >> $ERRFILE ) |
( sed 's/h/H/g' ; echo $? >> $ERRFILE )

after which I check $ERRFILE for error return codes. However, I've run into issues where $ERRFILE isn't necessarily being written to in the sequence of the pipes.

[OldSchool]

do a man on whichever shell your using

in posix shell

a || b

executes b only if a returned zero. there are other pipe operators depending upon what you want to accomplish

[Kevin Nikiforuk]

But I want to pipe a's result into b.

a || b will only execute a or b.

[Bill Hassell]

Another nifty technique is to turn on tracing by just setting a variable such as TRACEME. At the front of the script, you check if it is defined like this:

export TRACME=${TRACME:-NOTdefined}

This avoids any problems when you use set -u (always recommended to avoid spelling errors).

Then in the main script as well as all functions, put this line:

[[ $TRACEME = "NOTdefined" ]] || set -x

And that's it. Run the script without defining the TRACEME variable and it runs normally. Run the script and set TRACEME on the same line as in:

TRACEME=1 myscript

and now the script will trace it's own execution. This is really useful for old scripts that are having problems and everyone has forgotten how they work.