Unix Developer Guide - Please post your Tips.

[joe_91]

Hello Gurus:

I am in the process of Creating a Unix Developer's Guide possibly with examples to point out the dos and don't in the shell programming, etc. I am looking forward to Inputs from all gurus for this. Please contribute generously.

Thanks & Regards
Joe.

[Martin Johnson · Accepted solution]

Definately don't:

cd /tmp; rm -fR .*

:-)
Marty

[Rita C Workman]

For me....

Comments:
Do put remarks/comments in your script.
Do put your name as the author, so we know who to ask.
Do remember to comment/author/date anything you change & maybe why.

Common sense:
Do keep your scripts in a logical place..or maybe all in one place.
Do use a naming standard that makes sense as to what the script is. (Ex:start_oracle_payroll not franks_st)

Just a thought,
Rita

[Tom Maloy]

Use some kind of source code control.
Comment, comment, comment.
When using temporary files, use $$ (process ID) to make the file name unique.
Remove all temporary files when the script is done. Remember to do this even if the script exits early on an error condition.
Test for error conditions and produce a useful error message that will help resolve the problem.

Tom

[joe_91]

Tom:

Can you provide an example Please..

Thanks
Joe.

[Dave Chamberlin]

Keep the goal of solving the problem foremost. Don't kill yourself writing a script when another shell or language could solve the problem easier or be more maintainable. I have seen some scripts that solve a problem - but are very complex and difficult to understand - let alone maintain. I am sure that any problem can be solved with shell scripts - but some should not be. Be aware of differences in shells that you can use to your advantage to solve a problem. Don't be stuck with one shell, one syntax. To borrow from someone else - just my $.02 worth.

[Tom Maloy]

$$ is the shell variable that contains the process ID of the currently running process. So you can do

echo $$

and you will get the PID of the shell - you can also see this in the output of "ps".

TMP_DIR=/tmp
TMP_FILE=${TMP_DIR}/script$$
echo $TMP_FILE

should produce something like:

/tmp/script29803

This will be different each time the script is run, since it will be in a sub-shell with a unique PID.

So you won't write over your own temporary files, and won't have a conflict with someone else's temporary files.

Tom

[A. Clay Stephenson]

1) Rather than relying upon explicit rm statements to do cleanup of temp files, it is much better to handle that via a trap so that the files get removed even if the scripts receives an n'rupt.

e.g.

TDIR=${TMPDIR:-/var/tmp}
PID=${$}
T1=${TDIR}/T${PID}_1.txt
T2=${TDIR}/T${PID}_2.txt
T3=${TDIR}/T${PID}_3.txt

trap 'eval rm -f ${T1} ${T2} ${T3}' 0 1 2 3 15

xxxxxxxxxxx
xxxxxxxxxxx

Now the temp files will be removed if the script exits normally or receives signals 1, 2, 3, or 15.


2) It almost all cases readability is favored over efficiency. Very often I tend to write code in several clear steps that could be combined into fewer statements simply to make maintenance easier.

Now, just to make you mad. I would have new developers learn only the basic shell scripting techniques and then have then master
Perl. In the long run, they will be far more productive and there is then very little need to learn awk, sed, grep, tr ... . The cross-platform abilities of Perl make it my weapon of choice for most tasks these days. Even most of the stuff that I once had to do in C or C++, I can now usually do in Perl and it executes almost as fast.

[joe_91]

Thanks Clay. That was Brilliant!!! Thanks for the great response. I wish it goes on...

Thanks
Joe.

[Wodisch_1]

Hey Joe,

some points for scripts:
- always set the $PATH first
- never include the current directory
- always set the interpreter using the "#!/..." syntax in the very first line
- "unset" the shell-variables you don't need any more
- test for interactivity (ot the lack of) before using any "tty" command (like "stty", or "tabs", or such)
- always use "--" in "set" commands before appending the data
- always test for the exitance of files before overwriting/creating them (especially with ">" and ">>")
- always check the error/exit code

HTH,
Wodisch

[Thomas Schler_1]

Joe,

the header of my scripts are of nearly the same structure. This helps me in "re"learning my own scripts when I have to make corrections/modifications years after the first writing. I recommend to do so, too. The structure of the header is as follows:

1) Write a short text (just a few lines) that describes in very few words what your script is doing, on the top of the script.

2) Below the line "# HISTORY", write a short paragraph to explain what circumstances led to write this script. Explain why this script is needed.

3) Below the line "# USAGE", give the syntax how the script must be used. For that, use the syntax as is shown in nearly every command man page.

4) Below the line "# AUTHOR", write the name(s) of the author(s) and the date, when the script seems to be finished "the first time".

5) Below the line "# MODIFICATIONS", write date and name and a short description of >>every<< modification/correction made after the script went into production the first time.

6) Below the line "# RETURN VALUES", write all return values along with their meanings that can be given back by your script.

7) Below the line "# list of subroutines:", list all subroutines/functions you're using in your script along with a very short (half line) description of the subroutine/function.

8) Below the line "# list of important variables and their meanings:", list all important variables that are typical used as global variables along with a very short (half line) description of the variables.

9) Do use capital letters for all the important variables.

10) On the top of the main part of your script, set all default values to the variables you have to use. Define/initialize all the global variables on the top of the main part, even if a variable needs not to be initialized. A better solution is to put all these stuff into a separate subroutine.

11) Define one variable, e.g. $caller, that holds the name of your script, e.g. in perl this would be
($CALLER,@rem) = reverse split('/',$0); undef @rem;
or in ksh
caller=${caller##*/}

12) Explicitly, write error/warning messages into stderr, and normal information into stdout. Be strict, in doing so.

13) Every message you are printing to stdout must be preceded by "[$CALLER]: " or "ERROR [$CALLER]: ".

14) Be sure, your script never reaches undefined states. Very important!

15) In all cases when your are printing error message, be sure that even the most stupid assumed user can interprete the error message.

16) Each error message must explain (not just tell) the error. Avoid typical Microsoft error messages, like (in german) "Allgemeine SchuSSverletzung" or "This error should never occur." or "Division by zero"!! This would be a very bad style in error handling/messaging.

17) Use exit values. Use 0 for no errors occurred. Use a number greater than 0 if errors occurred. If possible, use different exit values for different possible error states. Use exit values, even if you today "definitely know, you would never interprete these exit values". Be sure, in the future, you'd like to do so, or one of your colleague will.

18) Always program the option "-h" or "-?" that will print the usage of the script.

19) Use the man page facility in Unix/Linux to document the usage of your script. While documenting, use the standard as you can see in ls(1), cp(1), man(1), and so on. (I am using self-written man pages of my own very often. Therefore, I and others do not have to look into the source code for possible options/arguments.)

I think, I forgot many other important things. But, other forum members will help you, too.

[Paula J Frazer-Campbell]

Joe

1. Anotate your scripts.
2. Use full path to system commands.

e.g. /usr/bin/rm not just rm.

3. When putting a rm in a script

WRONG
----------------
cd /usr/fred
/usr/bin/rm *
----------------

Right
---------------
/usr/bin/rm /usr/fred/*
---------------

If the cd fails then all sort of problems can occur.

4. Test fully each script before putting it near a live environment - we all know what a typo can do.


Just some ideas.

Paula

[fg_1]

Joe

Two definite DO's:

1)Make sure you document your scripts with as much commenting as possible, this will provide an invaluable training mechanism for your junior level admins just getting started in scripting.

2) Write shell scripts using a modular approach thus providing an easier path to following and understanding the logic.

Just my 2.5 cents worth.

Good luck in your book

FG.

[Joanne Keegan]

Hi Joe,

Something to consider - who will be running the script (root/other user) and what permissions should be set on it. If it is run by root, then it should be secured so that it can't be altered and run by another user - whether maliciously or not.

Regards,

Jo

[S.K. Chan]

I like to use a lot of functions in my script, it makes debugging and maintenance much easier and increase readability. For example ..
function whoareyou {
[[ `whoami` != root ]] && { print "Need superuser to run this."; exit 1; }
}
The function "whoareyou" can then be called later in the main program. You can even have different programmers code different functions, it makes distribution of work much easier.
..just my quick $0.02

[Martin Johnson]

For installations and upgrades: Create a detailed implemetation plan and a detailed test plan.

The implementation plan should include a check list to be checked off as the steps in the plan are implemented. (A check, date, and initials work fine). It should also contain the actual commands that are to be used.

If the steps in the implementation plan are incorrect, update the plan.

Save the plan for the next install/update.

HTH
Marty

[Bill Hassell]

- As mentioned, never use $PATH..always make your own to bypass Trojan Horses:

export PATH=/usr/bin

If you need commands from other directories, add them to the PATH just for this script.

- I like the idea of handling temp files (choose /var/temp ir $TEMP or $TMPDIR is not set). However, after creating about 10 tempfiles, remembering to purge them at the end or in a trap statement became unruly. So now I create a temp directory with $$ as part of the name, then throw whatever filenames I want into that directory and at the end, just recursively remove the one temp directory:

MYNAME=${0##*/}
TMPDIR="${TEMP:-/var/tmp}$MYNAME.$$"

cp LotsOFfiles $TMPDIR
..do stuff..
trap 'eval rm -rf $TMPDIR' 0 1 2 3 15

- And a real gem for script writers: www.shelldorado.com You can spend hours looking through the ideas.

[Michael Tully]

Here is a template I use:
Attached is a simple document on the explanation of RCS

################################################################################
# RCS Info: $Source: $
# $Revision: $
# $Date: $
#
##REMOVE This is a template for scripts. Edit the lines tagged
##REMOVE with "##REPLACE", with appropriate information, and then
##REMOVE develop the rest of the script.
##REMOVE Use RCS to track changes to your script.
##REMOVE See also TEMPLATE.config.
##REMOVE Lastly, remove these lines tagged with "##REMOVE".
##REMOVE
# Comments:
##REPLACE Description of script
#
# RCS log: $Log: $
#
################################################################################
#
# Set environment variables from the configuration file
#
. ${0}.config

rval=0 # set the default return value for this script

#
# FUNCTIONS
#

#
# Mail_Outcome -- mail log and exit
# To get the approriate responce from operators and/or
# Unix Sysadmins, start the subject of mail with either:
# SUCCESS:
# INFORMN:
# WARNING:
# FAILURE:
#
Mail_Outcome()
{
if [ $rval -eq 0 ]; then
mailx -s "SUCCESS: $(basename $0)" $MAIL_TO_SUCCESS < $LOGFILE
else
mailx -s "FAILURE: $(basename $0)" $MAIL_TO_FAILURE < $LOGFILE
fi
exit $rval
}

#
# Set_Return -- Check the exit value of a command run by this script
#
Set_Return()
{
# Check the exit value of a command run by this script. The exit code
# is echoed to the log file and if non-zero, the return value of this
# script is set to indicate failure.
return=$?
if [ $return -eq 0 ]
then echo "done (rc=$return)\n"
else
echo "ERROR (rc=$return)\n"
rval=1 # script had an error
fi
}

#
# Start_logging -- Open logfile and output "header" information.
#
Start_Logging ()
{
exec >$LOGFILE 2>&1 # Log change output from this script
echo $SEP1
echo "$(date) Starting $DESCRIPTION"
echo '$Source: $'
echo '$Revision: $'
echo "Script ${0} being run on host $(uname -n)"
echo '# '
echo '# KEY PARAMETERS'
echo '# --------------'
echo '# '
echo '# Logging Parameters'
echo "# LOGFILE: $LOGFILE"
echo '# '
##REPLACE Other key paramaters
echo ''
}

#
# Stop_Logging -- Finish off log and mail notification
#
Stop_Logging()
{
echo ""
echo "$(date) Finished $DESCRIPTION"
echo $SEP1
Mail_Outcome
}

#
# Usage -- displays the usage message and exits.
#
Usage()
{
printf "\nUsage: %s \n" `basename $0`
exit 1
}


#
# MAIN PROGRAM
#

##REPLACE Any set up stuff such as parsing paramaters.

Start_Logging

##REPLACE The guts of the script!

Stop_Logging

[H.Merijn Brand (procura]

Think portable. At the moment you might be working with HP-UX, but there will be a time that you are faced with AIX problems, or Solaris, or Linux, or ...

Do not use Vendor specific extensions to whatever utility - not even for the shell.

If you *do* need specific features for utilities (like +2Gb for tar) use only utilities that are freely available (GNU tar) and not vendor specific alterations that cannot be found on other systems.

Only script in scripting languages that are available on most systems, not on some

+ sh
+ ksh
+ csh
+ sed
+ awk
+ perl

- tcsh
- bash
- python
- tcl
- visual basic
- ruby

And as others have stated: Use comments, especially about what your prerequisitions, environment and other expected (and unexpected) side effects

[Deepak Extross]

1. Strange as it sounds, DON'T alias "rm" to "rm -i". I used to do this earlier, and I always expected a "rm" to ask if I'm sure. Till I got onto a production machine where the alias wasn't set, and the rest, as they say, is history :-)

2. To debug scripts, just stick in a "set -x". No need to "echo" anything!

3. Specify your shell on the first line of a script like so "#!/usr/bin/ksh". Scripts written to work with ksh wont work well with csh and vice versa.

4. If you're mucking around with terminal settings and your cursor disappears, dont panic. A "stty sane" should fix it.

5. To test a script that you're editing without having to exit from the vi screen, just do a ":!sh %"

6. Learn Perl! A look at some of the recent repsonses on this forum will show you that Perl is quicker, cleaner and more elegant than equivalent shell scripts. (I gotta do this myself, someday.)

[joe_91]

Thanks for all the great responses...A real good thread. What a great team here...

Thanks & Regards
Joe.

[Sajid_1]

Another quick reference tip, which will be useful always:
http://www.well.ox.ac.uk/~johnb/comp/unix/ksh.html#principle

[Mark Greene_1]

If coding in ksh or the posix shell, add these lines near the top for debugging purposes:

# next 2 lines for debugging only #
PS4='${0##*/}:$LINENO: '
typeset -ft $(typeset +f)

Then when you run ksh -x or sh -x, the name of the script and the line number will display before each line of code as it is executed.

HTH
mark

[joe_91]

This discussion is worth putting in the developer's guide.

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x0a4f0fe6d0f7d61190050090279cd0f9,00.html

Also Please help with some more tips

Thanks
Joe.

[Darrell Allen]

Don't assume simple commands execute successfully. Check the return code ($?) to verify commands succeeded.

For example:
cd /some_dir
rm *

If /some_dir doesn't exist or isn't mounted, you'll probably not like the results of the rm command.

Better method:
cd /some_dir
if [ $? -ne 0 ]
then
echo "Some error message"
exit
fi

I know this is a quite simple example but you get the point.

Darrell