COMP2100/2500 Lecture 16: Shell Programming I · 2016. 5. 9. · Lecture 16: Shell Programming I Summary An introduction to writing shell scripts using bash. Aims Explain what the

[ANU] [DCS] [COMP2100/2500] [Description] [Schedule] [Lectures] [Labs][Homework] [Assignments] [COMP2500] [Assessment] [PSP] [Java] [Reading]

[Help]

COMP2100/2500Lecture 16: Shell Programming

I

Summary

An introduction to writing shell scripts using bash.

Aims

Explain what the shell is, what scripting languages are, and whatthey are good for.

Introduce the basic structures and commands of bash, includingfilters, pipes, redirection, expansion, special variables and returncodes.

History lesson

Not so long ago, if you wanted to run a program on a computer, youwould have to submit a job, in the form of a stack of punched cards.That stack contained a card for each line of your program, a card foreach line of input data, and a number of cards that described how torun the program and access other devices attached to the computer.The instructions on those control cards were written using thecomputer's job control language, or JCL for short. Indeed, JCL is alsothe name of IBM's job control language. Here's an example (takenfrom Appendix 2 of IBM 360 Assembler Language Programming by

Lecture 16 - COMP2100/2500 - DCS - ANU file:///home/ericm/courses/sc/2005/www/lec-bash-...

1 of 21 01/05/15 09:37

Gopal K. Kapur, published by Wiley in 1970). Each line corresponds toa separate punched card.

// JOB PRACTICE// ASSGN SYS012,X'182'// ASSGN SYS014,X'01F'// ASSGN SYS010,X'181'// OPTION LINK// EXEC ASSEMBLY

The set of cards for the program goes here

/*// EXEC LNKEDT// EXEC

The set of cards for the data goes here

/*/&

The job control cards specified how to compile the program, whichdevices to use and how they should be configured (e.g. the recordingdensity of the magnetic tape in the tape drive), and where in the stackof cards the program and data start and finish. More advanced jobcontrol languages included commands for specifying the name of theuser running the job, accounting information ((computer) time ismoney!), the priority of the job, and so on.

With the rise of so-called on-line (i.e. interactive) systems, the jobcontrol language did not disappear, but the commands could now beentered directly into the computer by the user.

Today's shells are the descendants of the early job control languages.They incorporate many of the features we have discussed above, butalso provide higher-level structuring facilities that we are used to inconventional imperative programming languages (such as Java).Indeed, we will be interested in the shell as a programming language.

The Shell

Here is a simplified hierarchical view of a computer system and itsusers.


2 of 21 01/05/15 09:37

Hardware is roughly characterized as those parts of the systemwhich have an immediate physical reality.

Memory

Processors

Disks

Network cabling

The Operating System is low-level system software that isresponsible for controlling the hardware and providing facilities toapplication programs.

Organizes disks into file-systems


3 of 21 01/05/15 09:37

Runs programs on processors

Allocates memory to processes

The Shell conveys user requests to the OS.

Selects programs to run

Selects input for programs

Collects output from programs

In some systems, the shell is just one component of the operatingsystem (and thus can control devices and processes directly).However, in Unix, the shell has no such special privileges, and has thestatus of a normal application program.

What is a shell?

In the schema shown, the shell is the program that acts as the user'sinterface to the operating system. When the user logs in to thecomputer system, they begin interacting with the shell program inorder to start (and stop) application programs.

For most users these days, and for most of you so far, that rolehas been performed by the facilities of the graphical desktop.

Unix predates graphical desktops, so in Unix ‘shell' refers to aprogram that interprets typed user commands.

For expert users, the shell is a far faster and more powerful wayto interact with the operating system than the graphicalinterface.

One of your tasks this semester is to move from being an ordinaryuser of the computer system to becoming an expert user; a necessarycondition for becoming an effective software developer. You will needto become proficient at using the shell.

What is a Shell Script?

A shell reads and executes user commands.


4 of 21 01/05/15 09:37

A sequence of commands can be saved in a file for reuse.

Such files are called shell scripts.

They are really programs in the shell language.

Which Shell?

“The nice thing about standards is that there are so many ofthem to choose from.”

Grace Murray Hopper, as quoted in the Unix HatersHandbook, p.10

There have been many different shell programs written for Unixsystems, and most are available for you to try. They include:

The Bourne Shell (sh) -- The first UNIX shell, written by StephenBourne.

1.

The C Shell (csh) -- A replacement shell with syntax like that ofthe C language, written by Bill Joy between 1978 and 1980. Joyalso created vi and, much later, worked on the design of Java.

2.

The T C Shell (tcsh) -- A modernised C shell with command andfile completion. (This is the default shell on student accounts.)Bill Joy's source code was the starting point; around sixty peoplecontributed code in the years 1980-1998.

3.

The Bourne Again Shell (bash) -- A modernised Bourne shellincorporating many of the best features of sh and tcsh. Written byBrian Fox and Chet Ramey starting in 1989; the latest versiondates from 2002. This is the standard shell on many Unixsystems.

4.

Others such as ksh, zsh, . . . .5.

In this course we will focus on bash because:

Bourne shell is the traditional Unix scripting language, and bash isthe most widely installed version of a Bourne-style shell.

Bash is more expressive as a scripting language than csh or tcsh.


5 of 21 01/05/15 09:37

It's freely available (on Unix and Windows).

Both csh and tcsh can also be used for scripting purposes. It's good tobe able to read both styles. Remember that your default login shell istcsh, so to try out some of the things shown here on a command line,you will need to start a copy of bash.

Our First Shell Script

It's traditional for the first program in any new language to be onewhich just writes the string “Hello world!” on the screen. So here it isin bash.

First you must open a shell window. You can do this from thegraphical interface by selecting the icon that looks like a terminaland a shell from the panel at the bottom of the screen.

1.

Create a file called hello in your bin directory. (From here on, stuffthat you type is shown in bold.

[comp2100@partch]$ cd ~/bin[comp2100@partch]$ emacs hello &

The shell has a notion of current directory. When you open a newshell window, the current directory will be your home directory.Its path is something like /students/u1234567/. The cd commandchanges the current directory. It moves you around the directorytree. The tilde symbol ~ is a shorthand way of writing the path toyour home directory. So cd ~/bin means “Move to the binsub-directory of my home directory.”

Most of you have usually started the Emacs editor by double-clicking on a file in the File Manager, or by selecting theappropriate menu item. You can also start it from the shell bytyping its name emacs. (All lower case.) What comes after that isthe name of the file you want to edit. Emacs is smart enough tocheck whether that file exists, and create it if it doesn't. Theampersand character ‘&' at the end of the line tells the shell torun this command in the background. This means it doesn't waitfor the command to finish before giving you a prompt for anothercommand — which is just as well, because editing sessions oftenlast a long time.

2.

Type the following two lines into Emacs.3.


6 of 21 01/05/15 09:37

#!/bin/bash

echo Hello world!

Now save your shell script to the disk. You all know how to dothis using the mouse and the Emacs menus. But if you're doing ita lot, that gets too slow. You need to start learning somekeyboard command shortcuts. The shortcut for “Save the currentbuffer” is Control-X Control-S, which is usually written C-x C-s.

Make the file executable.

[comp2100@partch]$ chmod +x hello

Every Unix file has permissions associated with it, whichdetermine who is allowed to do what to it. The main permissionsare read, write and execute, and they are set separately for eachof three categories: the user who owns it, other users in the samegroup, and other users not in the same group. If you type thecommand ls -l into the shell, you will see a long listing of thecontents of the current directory, including all these permissions.The default for new files depends on the system, but it willcertainly have read and write permission for the user, and it willnot have execute permission for anyone. The chmod command(short for “Change Mode”) modifies these permissions. You canonly do it to files you own.

See the manual page for this command by typing man chmod. (Theman command is extremely useful, although you'll probably getmore information than you ever wanted. Learning to read manpages is another important task for you for this semester.)

4.

Run your script by typing

[comp2100@partch]$ ./helloHello world![comp2100@partch]$

5.

You can also run the bash shell and interact with it just like we havebeen interacting with tcsh so far.

Basic Shell Commands

Built in commands.


7 of 21 01/05/15 09:37

Some commands are a part of the shell.

echo just copies its arguments to the output.

cd changes the current directory.

read waits for input.

test performs comparisons and checks file types.

External commands.

For any command which is not built in, the shell will search for aprogram with that name to run. This allows any program to be used asa command in a shell script.

ls lists the contents of the current directory.

grep searches files for a regular expression.

sort sorts the lines of a file into order according to differentcriteria.

cat copies the contents of a file to the standard output.

Commands are separated by new lines or a semi-colon ;. For examplethe single input line

cd ..; ls

moves up to the parent directory and then lists its contents.

Filters and Pipes

Many commands read user input, and produce output. Suchcommands are called filters.


8 of 21 01/05/15 09:37

We can pipe the output of one program to be the input of another. Thesyntax is

program1 | program2


9 of 21 01/05/15 09:37

Examples of Pipes

Counting files (and directories).

If ls lists files in a directory, and wc -w counts words of input, then

ls | wc -w

counts the files in a directory!

Counting all your files:

The command ls -R lists the files in a directory, and all sub-directories.For example, here is part of the output produced by running it in mycomp2100 directory:

[barnes@partch comp2100]$ ls -R.:assignmentsbinindex.src.html


10 of 21 01/05/15 09:37

labslecturesmiscschedulewww

./assignments:

./bin:build...

and so on for several screens... You can see that the assignmentsdirectory was empty, but that the bin directory had a file orsubdirectory in it called build.

So you can count all your files (and directories) with:

ls -R ~ | grep -v ':$' | wc -l

How does that work? The output of ls -R is piped into grep. The -voption of grep says select all those lines which do not match theregular expression. The regular expression :$ matches all lines whichend with a colon. The dollar sign matches the end of a line. So theresult of

ls -R ~ | grep -v ':$'

is just a list of all the file and directory names, one per line, starting atyour home directory and going through all sub-directories. Piping thatinto wc -l (or wc -w) counts them.

Umm, except that the above is plain wrong. The output from ls alsoincludes a lot of blank lines, and these are not stripped out by the callto grep. The quick way to fix this is to add another grep to the pipeline:

ls -R ~ | grep -v ':$' | grep -v '^$' | wc -l

Here, ^ stands for the beginning of a line, so the pattern ^$ means ablank line.

File Redirection

So far I've been a bit vague about “input” and “output”. Now it's timeto fix that. Every Unix program takes its input from a stream calledstandard input — abbreviated as stdin — and sends its output to astream called standard output — abbreviated as stdout. I'm not going


11 of 21 01/05/15 09:37

to go into detail about what a stream is, but you should know thatstdin is usually connected (indirectly) to the keyboard and stdout isusually connected (indirectly) to the terminal screen. But they don'thave to be:

The output of a command can be redirected to go to a file:

command > file

The input of a command can be redirected to come from a file:

command < file

Redirection and piping can be combined, for example:

spell < lec-bash-1.src.html | sort -u > typos.txt

will find all the spelling mistakes in this lecture, sort themalphabetically (discarding duplicates), and store them in the filetypos.txt.

Sometimes we don't care about the output of a command. There is aspecial file called /dev/null where such output can be sent. It is adigital black hole.

Variables

Assignment:

Variables do not need to be declared before use! This is one of themajor differences between bash (and other scripting languages) andJava. This is one of the reasons that scripting languages are good for“quick and dirty” jobs.

Assignment has the form

variable=value

The assigned value must be a single word, or it must be quoted. Forexample,

bash$ x=hello

is OK, but you need to type

bash$ y="hello world"


12 of 21 01/05/15 09:37

The other big difference is that spaces matter here (unlike in Java).There must be no spaces before or after the = sign.

Note that in bash, as in csh, the set of shell variables is not the same asthe set of environment variables. However, in bash (unlike in csh) thereis a very strong link between these two sets. All environment variablescan be used just like shell variables, but not all shell variables arepassed on as environment variables to programs invoked by theshell — only those that have been explicitly exported.

Expansion

So what are variables good for? So far, our scripts have just been abunch of commands that we could have typed interactively, stored in afile to be run in “batch mode”. But scripts can be much more thanthat. Before a line of a script is executed, the shell performs all sortsof transformations — known as expansions — on it.

Variable expansion:

Before a command is executed, any instances of ${variable} or $variableare replaced by the value of variable. For example,

bash$ y="hello world"bash$ echo yybash$ echo $yhello worldbash$ echo $yly possessionspossessionsbash$ echo ${y}ly possessionshello worldly possessions

In the last example there, you need the braces, otherwise bash doesn'tknow where the variable name ends. This is what happened in thesecond-last example: bash looked for a variable called yly and couldn'tfind one. Unlike Java, it doesn't care if you use a variable you haven'tdeclared; it just happily treats it as the empty string.

This is something to watch out for: if you make a spelling mistake in avariable name, Java will give you an error message, and you'll find itthe first time you try to compile, but bash will happily run yourprogram, giving possibly quite bizarre results. This is one of thereasons that languages like Java are superior to scripting languagesfor large programs.


13 of 21 01/05/15 09:37

Expression expansion:

Before a command is executed, any instances of $[expression] arereplaced by the value of expression.

For example,

bash$ echo 2 + 32 + 3bash$ echo $[2 + 3]5bash$ echo 2 + 3 = $[2 + 3]2 + 3 = 5

This is another common surprise for Java programmers new to shellscripting. Expressions are only evaluated when you tell the shell toevaluate them.

Command expansion:

Before a command is executed, any instances of $(command) or `command`are replaced by the output of command. (In the second form that's thebackquote character, usually found at the top left of the keyboard.)

For example,

bash$ echo This directory has $(ls | wc -w) filesThis directory has 10 files

This is incredibly useful, but it can also lead to seriously cryptic codeif overused.

Pathname expansion:

Any instances of “shell style” regular expressions (words with ‘*’, ‘?’,and ‘[...]’) are replaced by possible matches before the command isexecuted.

For example,

bash$ echo *

functions just like ls. (Actually, that's not true; the spacing of theoutput is different.) We'll see more uses for this later.

Special Variables


14 of 21 01/05/15 09:37

A number of special environment variables have values when the shellstarts:

${USER} The login name of the user.${HOST} The name of the computer.

You can see the complete list by running the command env.

More special variables describe the parameters passed to the shellscript:

${#} The number of parameters.${0} The name of this shell script.${1} The first parameter.${*} A list of all the parameters.

Remember that ${0} can be expressed more simply as $0. Indeed, it'susually written the latter way.

The parameters passed to a script (or to any command you invoke in ashell script or interactively at the command line) are the things youtype on the same line after the name of the command. The shellbreaks them up by looking for spaces (unless you put somethinginside quote marks). For example, suppose the script params is:

#!/bin/bash

echo \${#} = ${#}echo \${0} = ${0}echo \${1} = ${1}echo \${2} = ${2}echo \${*} = ${*}

then we can type

bash$ params first second third${#} = 3${0} = params${1} = first${2} = second${*} = first second third

Notice another new thing here: if we want to use a special characterjust as itself, without its special meaning in shell language, we can“escape” it by putting a backslash before it. That's how I got it to print${#}.


15 of 21 01/05/15 09:37

Return Codes

All commands return a number to the operating system.

Most commands use this number to indicate success or failure. Areturn code of 0 means success. Anything else means failure.

There are some exceptions. An important one is the diff program.The line

bash$ diff file1 file2

prints the differences between file1 and file2. It returns 0 for nodifferences, 1 if there were differences, and 2 if there was anerror.

The special variable ${?} is the return code of the last command.

For example

bash$ diff file1 file2 > /dev/null; echo ${?}

Will print 0 if the files are the same, and 1 if they differ.

Control Structures

Bash has a full range of control structures: loops, conditionals andsubroutines. The way it handles tests for conditions is a little differentthan what you're used to in Java.

While loops

while first-command-listdo second-command-listdone

This repeatedly executes both command lists while the last commandof the first list returns an exit code of 0. That is:

Execute first-command-list.1.

If the exit code from the last command was zero, then continue,otherwise stop.

2.

Execute second-command-list.3.


16 of 21 01/05/15 09:37

Go back to step 1.4.

For example:

#!/bin/bash

while lpq | grep ${USER} > /dev/nulldo sleep 10doneecho All your print jobs have finished.

Conditionals

if command-listthencommand-list

elif command-listthencommand-list

...elsecommand-list

fi

This is pretty similar to the if-then-else-end construction in Java, exceptthat the condition for choosing the “then” part or not is the returncode of the last command in the command list between if and then.

if diff ${file1} ${file2} > /dev/nullthen echo Files are identical.else echo Files differ (or there's an error).fi

For Loops

These are very useful for scripts which have to process many files, ordo the same thing for a whole list of arguments.

for variable in listdocommand-list

done

Repeatedly execute command-list, with variable taking successive valuesfrom list.

For example:

for file in *.txt


17 of 21 01/05/15 09:37

do echo ${file} has $(cat ${file} | wc -w) words.done

This will find all the text files (assumed to have the extension .txt) inthe current directory and count the number of words in them.

An Example Application: BirthdayReminders

Suppose I record my friends' birthdays in the file .birthdays in my homedirectory:

Charles Manson:11/12Zsa Zsa Gabor:06/02William H. Gates III:28/10

This reminder script will tell me whose birthday it is today.

#!/bin/bash

today=$(date +%d/%m)lines=$(grep -n ${today} ~/.birthdays | cut -d: -f1)for x in ${lines}do echo -n Today is echo -n $(cut -d: -f1 ~/.birthdays | head -${x} | tail -1) echo \'s birthday!done

How does this work?

The first line runs the date command, and saves its output in thevariable today. The argument to the date command tells it to printthe date as a two-digit day-of-the-month number, followed by aslash, followed by a two-digit month-in-the-year number. This isthe format I used for the birthday dates in the .birthdays file. (Andas usual you're likely to run into trouble with American-stylebirthdays.)

1.

The next line produces the line numbers within the file of all thepeople whose birthdays match today's date. (There might bemore than one. This would be a little easier if we didn't have toallow for that possibility.) The grep command selects all lines from.birthdays which contain the string we just stored in ${today} andprecedes each by its line number and a colon. The cut command

2.


18 of 21 01/05/15 09:37

takes divides its input up into fields using the delimiter characterspecified. Here that is the colon character; that's what the -d:option means. The -f1 option tells the cut command to onlyoutput the first of the fields it has divided each line into. So theoutput of the whole thing in $(...) is a list of the line numbers ofthe people whose birthday is today.

The loop performs one iteration for each number in the list${lines}. For each of these it first prints ‘Today is ’ and doesn'tmove to the next line. (That's what the -n option for echo does.)

3.

In the next line, the cut command takes the file ~/.birthdays andthrows away everything after and including the first colon oneach line. So it just keeps the names, and throws away the dates.The result of this is passed to the head command, which copies thefirst few lines of a file to its output and throws away the rest.How many? Well if there's no option, it's 10, but if you put anumber there (with a minus sign before) then it takes exactlythat many lines. So head -${x} after variable substitution is goingto take all the lines up to and including the current value of $x.Finally the tail command throws away all but a few lines at theend of its input. With the option tail -1 it produces only the lastline, which is the one we want, with the name of one of thepeople whose birthday is today.

4.

The last line isn't very interesting. The only thing to watch out foris that the apostrophe had to be escaped, because otherwise bashwould think it was the beginning of a string.

5.

The next thing is to put a line into your .login file to run this scriptevery time you log in. Any ideas?

What Characterises a Scripting Language

Variables do not need to be declared before they are used. (Somescripting languages do allow you to declare variables and thenonly use variables you have previously declared. Some don'tallow you to declare variables even if you wanted to.)

There are not many basic data types. In some languages (such asshells) all data is represented as strings.

Literal strings do not need to be enclosed in special markers


19 of 21 01/05/15 09:37

(quotes). Almost everything else does.

There are few features for structuring programs. Well, maybe:some scripting languages are more expressive than others.

Programs are interpreted rather than compiled. Well, maybe:some scripting languages (e.g. Perl) come with a compiler,although you don't have to use it.

Other scripting languages include:

awk

tcl

Perl

Python

PHP

The last three are large and popular programming languages whichhave at least partly outgrown their role as tools for writing quick-and-dirty solutions to small problems. Both now incorporate someobject-oriented features, while retaining some of the convenience ofshell scripts. However they don't offer features such as design bycontract, type checking, and so on.

When Would You Use a ScriptingLanguage?

When the task

is not logically complex

requires a great deal of string and text manipulation

can be accomplished largely by running existing programs if youcan control their options, input and output

requires the manipulation of many files.

Scripting languages — particularly Perl and PHP — are now usedextensively to generate dynamic web pages.


20 of 21 01/05/15 09:37

[ANU] [DCS] [COMP2100/2500] [Description] [Schedule] [Lectures] [Labs][Homework] [Assignments] [COMP2500] [Assessment] [PSP] [Java] [Reading]

[Help]

Copyright © 2005, Jim Grundy & Ian Barnes & Richard Walker, The Australian National UniversityVersion 2005.2, Monday, 4 April 2005, 14:49:53 +1000

Feedback & Queries to [email protected]


21 of 21 01/05/15 09:37

Documents

COMP2100/2500 Lecture 16: Shell Programming I · 2016. 5. 9. · Lecture 16: Shell Programming I Summary An introduction to writing shell scripts using bash. Aims Explain what the