d e meyers feb 02

Shell Programming Basics

CREATING A SCRIPT
Each UNIX shell (bash, csh, ksh, sh) has a specific programming (scripting) language. Scripts in this article use bash (a derivative of the bourne shell). Scripts must be ASCII text and written with a text editor such as emacs, vi, or pico. A shell script contains instructions like source code for programming languages such as C/C++, but scripts are not compiled into an executable. Scripts are interpreted and executed line by line by the shell. Scripts must have execution permission. The first line of a script controls which program will execute the script (in the example below it is bash). The syntax is #! followed by the full path of the program.
    #!/usr/bin/bash
Assume you have written and saved a script called myscript. Since the default permission set for text files is 644, you must explicitly make your script executable. This command will make myscript executable for you alone and no permission otherwise:
chmod 700 myscript 
Start your script by typing:
myscript 

or 

./myscript    (if you do not have a . in your PATH) 
DEBUGGING
Simple debugging can be done with echo. You can use echo to print specific variables around the place where you suspect the mistake. Scripts can be executed in debug mode by adding the following line at the beginning of the script:
set -x
COMMENTS
Comments in shell programming start with # and go until the end of the line.

VARIABLES
In shell programming all variables have the datatype string. You do not need to declare variables before use. The following statements will assign values to variables. The use of quotes mirrors the use of quotes at the command line:

myfirstname=samantha
myname="samantha spade"
myid=54321
To get the value back you just put a dollar sign in front of the variable:
#!/usr/bin/bash
# assign a value:
a="hello world"
# now print the content of "a":
echo "A is:"
echo $a
This script will print:
A is:
hello world
Use {} to separate a variable embedded in a literal:
num=2
echo "this is the $numnd"
echo "this is the ${num}nd"
The first statement will print "this is the " (not "this is the 2nd") because the shell searches for a variable called numnd which has no value.

From within a script you also have access to user-defined environment variables (variables set by the user in a script and exported for use) and shell variables (read-only variables set by the shell). These variables are always completely uppercase. Typical user-defined environment variables are TERM, PATH, EDITOR, USER. In a script type $varname, e.g. $PATH to access the variable. Shell variables are set by the shell at login. Examples are BASH_VERSION, HOME.Type 'man bash' for a complete listing of all shell variables (search for 'Shell Variables'. Type 'set' for a listing of all your current environment variables.

SHELL COMMANDS AND CONTROL STRUCTURES
A shell script can use any UNIX command that can be typed at the command line. Shell scripts also have access to shell control structures that will work only in a script. UNIX commands used in this document include:

COMMAND			PURPOSE
echo "some text"        write some text on your screen
grep 'pattern' file     earch for strings in a file
cut -b5-9 file.txt      get characters from positions 5 to 9 in file.txt
read var                read from stdin into a variable (var)
expr                    evaluate math expressions; e.g. expr 2 + 3
basename filename       strip the directory path
dirname filename        strip the file name
file filename           returns the file type of filename

Just as at the command line, the output of a UNIX command can also be used as a command line argument for another command or assigned to a variable. This is called command substitution and is denoted by the backquote or backtick (`).
# The quotes are backquotes (`)  not normal quotes ('):
tar -zcvf lastmod.tar.gz `find . -mtime -1 -type f -print`
or
today=`date | cut -f 1 -d " "`
echo $today

# evaluate the expression and store in result
result=`expr 2 + 3` 

The shell script language offers the same control structures that programming languages offer: conditions, loops, and breaks. Control structures only make sense in a script, not at the command line. The if statement tests if the condition is true (exit status is 0 for success). If the condition is true, the "then" part gets executed:
if ....; then
   ....
elif ....; then
   ....
else
   ....
fi
The test operator can be used to compare strings or test if a file exists, is readable etc... (type 'man test' for details). The "test" command is written as square brackets " [ ] ". Note that space is significant here: Make sure that you always have spaces around the brackets. Examples:
[ -f "somefile" ]  : Test if file exists.
[ -x "/bin/ls" ]   : Test if /bin/ls exists and is executable.
[ -n "$var" ]      : Test if the variable $var contains something
[ "$a" = "$b" ]    : Test if the variables "$a" and  "$b" are equal
Example:
#!/usr/bin/bash
if [ "$SHELL" = "/usr/bin/bash" ]; then
  echo "your login shell is the bash (bourne again shell)"
else
  echo "your login shell is not bash but $SHELL"
fi

if [ -f $HOME/.profile ]; then
  echo "your .profile exists"
else
  echo "your .profile does not exist"
fi
The AND (&&) and OR (||) operators can be used as shorthand if statements. You can do everything without the ANDs and ORs using just if-statements; sometimes the shortcuts AND and OR are more convenient. In the AND statement the right side is executed if the left is true; in the OR statement the right side is executed if the left is false. In the following, the echo statement will be executed if the file /etc/shadow exists.
[ -f "/etc/shadow" ] && echo "This computer uses shadow passwords"
In this script an error message is displayed and the script is exited if the mailfolder is NOT readable:
#!/usr/bin/bash
mailfolder=/var/spool/mail/james
[ -r "$mailfolder" ] || { echo "Can not read $mailfolder" ; exit 1; }
echo "$mailfolder has mail from:"
grep "^From " $mailfolder
A few notes about this code:

The case statement is a conditional control structure that can be used to match (using shell wildcards such as * and ?) a given string against a number of possibilities. The ';;' is required and marks the end of a case option.

case ... in
...) do something here;;
esac
Assume you have a gzipped file named lf.gz. The command:
file lf.gz

returns:

lf.gz: gzip compressed data, deflated, original filename,
last modified: Mon Aug 27 23:09:18 2001, os: Unix
This script called smartzip will uncompress bzip2, gzip and zip compressed files automatically :
#!/usr/bin/bash
ftype=`file "$1"`
case "$ftype" in
"$1: Zip archive"*)
    unzip "$1" ;;
"$1: gzip compressed"*)
    gunzip "$1" ;;
"$1: bzip2 compressed"*)
    bunzip2 "$1" ;;
*) error "File $1 can not be uncompressed with smartzip";;
esac
Note the special variable called $1. This variable contains the first of any number of command line arguments. See the section below for details.

Here is an example:

#!/usr/bin/bash
echo "What is your favourite OS?"
select var in "dUNIX" "Solaris" "Free BSD" "Other"; do
        break
done
echo "You have selected $var"
Here is what the script does:
What is your favourite OS?
1) dUNIX 
2) Solaris 
3) Free BSD
4) Other
#? 1
You have selected dUNIX
In the shell you have the while-loop, for-loop, and until-loop control structures:
while ...; do
 ....
done
The while-loop will run while the expression that we test for is true. The keyword "break" can be used to leave the loop at any point in time. The keyword "continue" forces the loop to continue with the next iteration and skips the rest of the loop body.

The for-loop takes a list of strings (strings separated by space) and assigns them to a variable:

for var in ....; do
  ....
done
E.g. the following will print the letters A to C on the screen:
#!/usr/bin/bash
for var in A B C ; do
  echo "var is $var"
done
ARITMETIC AND RELATIONAL OPERATORS
- +  unary minus and plus
  ! ~  logical and bitwise negation
  **   exponentiation
  * / %
       multiplication, division, remainder
  + -  addition, subtraction
  << >>
       left and right bitwise shifts
  <= >= < >
       comparison
  == !=
       equality and inequality
  &    bitwise AND
  ^    bitwise exclusive OR
  |    bitwise OR
  &&   logical AND
  ||   logical OR
  expr?expr:expr
       conditional evaluation
  = *= /= %= += -=
       assignment
arg1 OP arg2
       OP is one of -eq, -ne, -lt, -le, -gt, or -ge.  These arithmetic binary
       operators return true if arg1 is equal to, not equal to, less than,
       less than or equal to, greater than, or greater than or equal to arg2,
       respectively.  Arg1 and arg2 may be positive or negative integers.
COMMAND LINE ARGUMENTS
The script variables $* and $1, $2 ... $9 contain the arguments that the user specifies on the command line. $* is the complete command line, $1 is the first argument, $2 the second, etc. The special variable $# contains the number of arguments in $*; i.e., if there are 4 arguments $# is 4. The shift operator moves the arguments in $* one place to the left; i.e. if there are 3 arguments ($# = 3) the following occurs: $1 = $2, $2 = $3, the original value of $1 is lost, and $# = 2.

When the command line syntax of your script is complicated; i.e. you cannot guarantee the number or order of the arguments you will need a sophisticated way to parse the command line. One such method is to use a while loop combined with a case and shift statement.

#!/usr/bin/bash
# Funtion:     command line parser.
#
# USAGE EXAMPLE: parser [-m hello] filename [-h]
if [ $# -eq 0 ];
then
    echo "USAGE: parser [-m hello] filename [-h]"
fi

# while [ $# -ne 0 ]; do
while [ -n "$1" ]; do
case $1 in
    -h) echo "USAGE: parser [-m hello] filename [-h]";
        shift;;
    -m) message=$2;                # grab the next argument past -d
        echo "message: $message";  # display it
        shift; shift;;             # move past 2 command line arguments
    -*) echo "error: no such option $1. -h for help";
        exit 1;;
     *) filename=$1;
        echo "filename: $filename";
        shift;;
esac
done
If you run the script with:
parser -m hello somefile  -h
It produces
message: hello
filename: somefile 
USAGE: parser [-m hello] [filename] [-h]
The script loops through all arguments and matches them against the case statement. If it finds a matching one it sets a variable and shifts the command line by one or two and continues on. The script will work regardless of the order and number of arguments.

QUOTING
Before passing any arguments to a program the shell tries to expand wildcards and variables. To expand means that the wildcard (e.g. *) is replaced by the appropriate file names or that a variable is replaced by its value. To change this behaviour you can use quotes: Let's say we have a number of files in the current directory. Two of them are jpg-files, mail.jpg and tux.jpg.

#!/usr/bin/bash
echo *.jpg
This will print "mail.jpg tux.jpg".

Quotes (either single or double) will prevent this wildcard expansion:

#!/usr/bin/bash
echo "*.jpg"
echo '*.jpg'
This will print "*.jpg" twice.

Single quotes are most strict. They prevent even variable expansion. Double quotes prevent wildcard expansion but allow variable expansion:

#!/usr/bin/bash
echo $SHELL
echo "$SHELL"
echo '$SHELL'
This will print:
/usr/bin/bash
/usr/bin/bash
$SHELL
Finally there is the possibility to take the special meaning of any single character away by preceeding it with a backslash:
echo \*.jpg
echo \$SHELL
This will print:
*.jpg
$SHELL
HERE DOCUMENTS
Here documents are a nice way to send several lines of text to a command. It is quite useful to write a help text in a script without having to put echo in front of each line. A "Here document" starts with << followed by some string that must also appear at the end of the here document. Here is an example script, called ren, that renames multiple files and uses a here document for its help text:
#!/usr/bin/bash
# we have less than 3 arguments. Print the help text:
if [ $# -lt 3 ] ; then
cat << HELP
ren -- renames a number of files using sed regular expressions

USAGE: ren 'regexp' 'replacement' files...

EXAMPLE: rename all *.HTM files in *.html:
  ren 'HTM$' 'html' *.HTM

HELP
  exit 0
fi
OLD="$1"
NEW="$2"
# The shift command removes one argument from the list of
# command line arguments.
shift
shift
# $* contains now all the files:
for file in $*; do
    if [ -f "$file" ] ; then
      newfile=`echo "$file" | sed "s/${OLD}/${NEW}/g"`
      if [ -f "$newfile" ]; then
        echo "ERROR: $newfile exists already"
      else
        echo "renaming $file to $newfile ..."
        mv "$file" "$newfile"
      fi
    fi
done
The first if-statement tests if we have provided at least 3 command line parameters. (The special variable $# contains the number of arguments.) If not, the help text is sent to the command cat which in turn sends it to the screen. After printing the help text we exit the program. If there are 3 or more arguments we assign the first argument to the variable OLD and the second to the variable NEW. Next we shift the command line parameters twice to get the third argument into the first position of $*. With $* we enter the for loop. Each of the arguments in $* is now assigned one by one to the variable $file. Here we first test that the file really exists and then we construct the new file name by using find and replace with sed.

The backticks are used to assign the result to the variable newfile. Now we have all we need: The old file name and the new one. This is then used with the command mv to rename the files.

FUNCTIONS
As soon as you have a more complex program you will find that you use the same code in several places and also find it helpful to give it some structure. A function looks like this:

functionname()
{
 # inside the body $1 is the first argument given to the function
 # $2 the second ...
 body
}
You need to "declare" functions at the beginning of the script before you use them.

Here is a script called xtitlebar which you can use to change the name of a terminal window. If you have several of them open it is easier to find them. The script sends an escape sequence which is interpreted by the terminal and causes it to change the name in the titlebar. The script uses a function called help. As you can see the function is defined once and then used twice:

#!/usr/bin/bash
# vim: set sw=4 ts=4 et:

help()
{
    cat << HELP
xtitlebar -- change the name of an xterm, gnome-terminal or kde konsole

USAGE: xtitlebar [-h] "string_for_titlebar"

OPTIONS: -h help text

EXAMPLE: xtitlebar "cvs"

HELP
    exit 0
}

# in case of error or if -h is given we call the function help:
[ -z "$1" ] && help
[ "$1" = "-h" ] && help

# send the escape sequence to change the xterm titelbar:
echo -e "\033]0;$1\007"
#
It's a good habit to have extensive comments inside your scripts.

EXAMPLES
The following script b2d converts a binary number (e.g 1101) into its decimal equivalent. It is an example that shows that you can do simple mathematics with expr:

#!/usr/bin/bash
# vim: set sw=4 ts=4 et:
help()
{
  cat << HELP
b2h -- convert binary to decimal

USAGE: b2h [-h] binarynum

OPTIONS: -h help text

EXAMPLE: b2h 111010
will return 58
HELP
  exit 0
}

error()
{
    # print an error and exit
    echo "$1"
    exit 1
}

lastchar()
{
    # return the last character of a string in $rval
    if [ -z "$1" ]; then
        # empty string
        rval=""
        return
    fi
    # wc puts some space behind the output this is why we need sed:
    numofchar=`echo -n "$1" | wc -c | sed 's/ //g' `
    # now cut out the last char
    rval=`echo -n "$1" | cut -b $numofchar`
}

chop()
{
    # remove the last character in string and return it in $rval
    if [ -z "$1" ]; then
        # empty string
        rval=""
        return
    fi
    # wc puts some space behind the output this is why we need sed:
    numofchar=`echo -n "$1" | wc -c | sed 's/ //g' `
    if [ "$numofchar" = "1" ]; then
        # only one char in string
        rval=""
        return
    fi
    numofcharminus1=`expr $numofchar "-" 1`
    # now cut all but the last char:
    rval=`echo -n "$1" | cut -b 0-${numofcharminus1}`
}


while [ -n "$1" ]; do
case $1 in
    -h) help;shift 1;; # function help is called
    --) shift;break;; # end of options
    -*) error "error: no such option $1. -h for help";;
    *)  break;;
esac
done

# The main program
sum=0
weight=1
# one arg must be given:
[ -z "$1" ] && help
binnum="$1"
binnumorig="$1"

while [ -n "$binnum" ]; do
    lastchar "$binnum"
    if [ "$rval" = "1" ]; then
        sum=`expr "$weight" "+" "$sum"`
    fi
    # remove the last position in $binnum
    chop "$binnum"
    binnum="$rval"
    weight=`expr "$weight" "*" 2`
done

echo "binary $binnumorig is decimal $sum"
#
The algorithm used in this script takes the decimal weight (1,2,4,8,16,..) of each digit starting from the right most digit and adds it to the sum if the digit is a 1. Thus "10" is:
0 * 1 + 1 * 2 = 2 
To get the digits from the string we use the function lastchar. This uses wc -c to count the number of characters in the string and then cut to cut out the last character. The chop function has the same logic but removes the last character, that is it cuts out everything from the beginning to the character before the last one.

Perhaps you are one of those who save all outgoing mail to a file. After a couple of months this file becomes rather big and it makes the access slow if you load it into your mail program. The following script rotatefile can help you. It renames the mailfolder, let's call it outmail, to outmail.1 if there was already an outmail.1 then it becomes outmail.2 etc...

#!/usr/bin/bash
# vim: set sw=4 ts=4 et:
ver="0.1"
help()
{
    cat << HELP
rotatefile -- rotate the file name

USAGE: rotatefile [-h]  filename

OPTIONS: -h help text

EXAMPLE: rotatefile out
This will e.g rename out.2 to out.3, out.1 to out.2, out to out.1
and create an empty out-file

The max number is 10

version $ver
HELP
    exit 0
}

error()
{
    echo "$1"
    exit 1
}
while [ -n "$1" ]; do
case $1 in
    -h) help;shift 1;;
    --) break;;
    -*) echo "error: no such option $1. -h for help";exit 1;;
    *)  break;;
esac
done

# input check:
if [ -z "$1" ] ; then
 error "ERROR: you must specify a file, use -h for help"
fi
filen="$1"
# rename any .1 , .2 etc file:
for n in  9 8 7 6 5 4 3 2 1; do
    if [ -f "$filen.$n" ]; then
        p=`expr $n + 1`
        echo "mv $filen.$n $filen.$p"
        mv $filen.$n $filen.$p
    fi
done
# rename the original file:
if [ -f "$filen" ]; then
    echo "mv $filen $filen.1"
    mv $filen $filen.1
fi
echo touch $filen
touch $filen
How does the program work? After checking that the user provided a filename we go into a for loop counting from 9 to 1. File 9 is now renamed to 10, file 8 to 9 and so on. After the loop we rename the original file to 1 and create an empty file with the name of the original file.