Getting started with shell scripting

Thomas Roblitzthomas.roblitz@usit.uio.no

RCS Course Week, October 22-25, 2012

Outline

I What is shell scripting ?I InvocationI VariablesI Basic control flow (if-then-else-fi)

Poll

I Who did alreadyI use a shell?

I write a shell script?I write a script (R, MATLAB, Bioportal, . . . )?

Poll

I Who did alreadyI use a shell?I write a shell script?

I write a script (R, MATLAB, Bioportal, . . . )?

Poll

I Who did alreadyI use a shell?I write a shell script?I write a script (R, MATLAB, Bioportal, . . . )?

What is shell scripting?

I A way to store, edit & execute shell commandsI store⇒ avoid re-typing again and againI edit⇒ easily change commandsI execute⇒ execute commands (interactive or batch mode)

⇒ Simplifies the repetition of long & complex commandsequences.

What is shell scripting used for?1

I A wrapper for a programI aka SLURM fileI prepare job input data, run job, check results, copy results⇒ May help in avoiding (unnoticed) job failures.

I To manage research environmentI create data/jobs layout (e.g., for parameter studies)I perform post-processing of results⇒ May help in reproducing results if necessary.

1on Abel

Simple script: Edit simple.sh

I use your favourite (ASCII) text editor: vim, emacs, . . .

#!/bin/bash

WHO=All

if [ $# -gt 0 ]then

WHO=$1fi

DATE=$(date)echo "Welcome ’$WHO’ to RCS Course Week\n$DATE"

Simple script: Running

> sh simple.shWelcome ’All’ to RCS Course WeekMon Oct 22 21:48:10 CEST 2012

> sh simple.sh AbelWelcome ’Abel’ to RCS Course WeekMon Oct 22 21:48:50 CEST 2012

Step-by-step

I hashbangI invocation & argumentsI variablesI if-then-else-fiI tests

Hashbang2

I 1st line in each shell scriptI hashbang #!

I command interpreter, e.g. /bin/bash

#!/bin/bash

I How about perl, /bin/perl ?

> which perl/usr/bin/perl

which shows the full path of (shell) commands(see man which)

2aka shebang, pound-bang, hash-pling, . . .

Invocation – Explicit

I use shell interpreter explicitly

> sh simple.sh

I Why sh and not bash ?

> which sh/bin/sh

> file /bin/sh/bin/sh: symbolic link to ‘bash’

⇒ sh is a symbolic link to /bin/bash (the default shell in aLinux system)

Invocation – Implicit

I Run script without sh ...?

> ./simple.sh-bash: simple.sh: Permission denied

I Check

> ls -l simple.sh-rw-r--r-- 1 thomarob users 117 Oct 22 18:04 simple.sh

I Fix

> chmod u+x simple.sh> ls -l simple.sh-rwxr--r-- 1 thomarob users 117 Oct 22 18:04 simple.sh

Invocation – Scientists have no timeI How about saving two more keystrokes?

> simple.sh-bash: simple.sh: command not found

I Check

> which simple.sh/usr/bin/which: no simple.sh in (/hpc/bin:...

which searches the directories in the environment variable PATH

> PATH=$PATH:$PWD> simple.shWelcome ’All’ to RCS Course WeekMon Oct 22 21:48:10 CEST 2012

I Alternative: use absolute path, i.e., /path/simple.sh

Arguments – Welcome ’everybody’

I Passing arguments

> arguments.sh everybodyWelcome ’everybody’ to RCS Course Week

I Simplified script

#!/bin/bash

echo "Welcome ’$1’ to RCS Course Week"

I $1 is a positional parameterI positional parameters are set when the shell is invokedI set with everybody in the invocation above

Arguments – Welcome everybody

I Passing multiple arguments

> argumentsX.sh bash cshWelcome ’bash’ (1) to RCS Course WeekWelcome ’csh’ (2) to RCS Course WeekWelcome ’’ (3) to RCS Course Week

I Enhanced script

#!/bin/bashecho "Welcome ’$1’ (1) to RCS Course Week"echo "Welcome ’$2’ (2) to RCS Course Week"echo "Welcome ’$3’ (3) to RCS Course Week"

I $n is a positional parameterI unused positional parameters are empty

Variables

I Use (also Substitution)$VARIABLENAME

I no need to declare before use (default: empty string)I AssignmentVARIABLENAME=any (un)quoted string

I VARIABLENAME is of characters a-z, A-Z, 0-9, _, . . .I no space between VARIABLENAME and = and valueI quoting (" or ’) needed if value contains spaces

I Note the differences (with/out $)I untyped (think: everything is a string)

Variables – Examples

I Script

#!/bin/bashABEL=AbelABLE="Abel is able"echo "Welcome ’$ABEL’ (ABEL) to RCS Course Week"echo "Welcome ’$ABLE’ (ABLE) to RCS Course Week"echo "Welcome ’$UNSET’ (UNSET) to RCS Course Week"

I Output

Welcome ’Abel’ (ABEL) to RCS Course WeekWelcome ’Abel is able’ (ABLE) to RCS Course WeekWelcome ’’ (UNSET) to RCS Course Week

I Experiment with quotingI remove double quotes in assignment of ABLEI remove single and/or double quotes in echo for UNSET

Variables – Environment (1)

I Many useful variables defined in the environmentI General (run env after logging in), e.g.

PWD current path/directoryHOME home directoryPATH searched for executables/scripts

SHELL the current shellI Can you (now)3 explain the following command?

> PATH=$PATH:$PWD

3See example on slide 14.

Variables – Environment (2)

I Job (run env after qlogin ...), e.g.

SLURM_JOBID job idSCRATCH directory for in/out data on /work

SLURM_JOB_NODELIST list of nodes allocated to the jobSLURM_TASKS_PER_NODE number of tasks per node

SLURM_MEM_PER_CPU memory per core

Status

I You should now understand the highlighted parts.

#!/bin/bash

WHO=All

if [ $# -gt 0 ]thenWHO=$1

fi

DATE=$(date)echo "Welcome ’$WHO’ to RCS Course Week\n$DATE"

Variables – Command substitution

I Suppose we want to use the same date multiple times

#!/bin/bashdate +"Script starts at %c"sleep 3 # do sthdate +"Script started at %c"date +"Script ends at %c"

Script starts at Tue 23 Oct 2012 09:02:46 AM CESTScript started at Tue 23 Oct 2012 09:02:49 AM CESTScript ends at Tue 23 Oct 2012 09:02:49 AM CEST

I need to save the start time in a variable

Variables – Command substitution(2)

I SyntaxVARIABLENAME=$(command with arguments)

I Variable is substituted with the command’s output.

#!/bin/bashSTART=$(date +%c)echo "Script starts at $START"sleep 3 # do sthecho "Script started at $START"date +"Script ends at %c"

Script starts at Tue 23 Oct 2012 09:10:38 AM CESTScript started at Tue 23 Oct 2012 09:10:38 AM CESTScript ends at Tue 23 Oct 2012 09:10:41 AM CEST

I Command substitution is a powerful tool. Try it !

If-then-else – Example

I Example

#!/bin/bashDATE=$(date +%s)START=$(date -d "Tue 23 Oct 2012 01:15:00 PM CEST" +%s)if [ $DATE -gt $START ]thenecho "RCS course already started"

elseecho "RCS course not yet started"

fi

RCS course not yet started

If-then-else – Syntax

I if [ conditional_expression ]thenstatements

elsestatements

fi

I else branch is optionalI [ conditional_expression ] same astest conditional_expression

I Note! Must not remove spaces around [ and ]

Tests – Conditional expressions

I For details see man test

I Examples

expression evaluates to true if

! EXP EXP is falseEXP1 -a EXP2 both EXP1/2 are trueEXP1 -o EXP2 either EXP1/2 is true

STRING1 = STRING2 strings are equalSTRING1 != STRING2 strings are not equal

INT1 -eq INT2 integers are equalINT1 -gt INT2 INT1 ≥ INT2

-e FILE FILE exists-x FILE FILE exists incl. execute permission

Putting it all together

#!/bin/bash

WHO=All

# $# is the number of argumentsif [ $# -gt 0 ]then

WHO=$1fi

DATE=$(date)echo "Welcome ’$WHO’ to RCS Course Week\n$DATE"

Become a shell scripting user/expert

I You just learned the ver basics: hashbang, invocation,arguments, variables, if-then-else & tests

I 2nd course this week:

Intermediate shell scripting

Andreas Buzh Skau

Wednesday, Oct 24, 12:15

I read man pages: man bash, man test

I search internet for tutorialsI study examples and scripts of co-workersI Most important! Practice, practice, practice.

Happy scripting