Bash Shell Programming - VAIOLabs · •Bourne Again SHell Bash comapatable with 'sh' (Linux default). •The C shell from UCB is a shell with a “C-like” syntax, it is not compatible

Bash Shell Programming

Presented ByAlex M. Schapelle

Copyright Alex M. Schapelle , VAIOLabs LTD, www.MobiusDevTeam.com, www.vaiolabs.com, License: GPLv3

The Bash Shell Environment•The process environment.•Use of shell variables.•Common variables.•The search path.•The shell prompt.•Profiles and environment.


Command Line scanning• All commands end with a <CR> character.• The shell responds to <CR> by scanning (interpreting) the line, which

involves:• Identifying words, according to white spaces.• Interpreting wildcards to generate filenames.• Removing any escape (quote) characters.• Substituting variables and parameters.• Substituting embedded commands.• Whilst scanning and interpreting the line, the shell continues to recreate it.• Once the scan is completed, the shell will attempt to identify the command

to be executed.Copyright Alex M. Schapelle , VAIOLabs LTD, www.MobiusDevTeam.com, www.vaiolabs.com, License: GPLv3

Parent and Child Processes•A process is an instance of a program being executed.•Every process is assigned a unique id number (PID).•The shell creates a child process for every external

command (program on disk).•A child process is often referred to as a sub-shell.

Bash Shell

Sub-Shell ls cmd

Parent Process

Child Process


Bash Shell Environment• Each process has its own environment used to store variables (also aliases and

functions).• Variables are labels used to hold information.

• Changes to an environment are local to the process.• Defined shell variables are not automatically visible to other processes.• Programs inherit part of the parent environment (marked for export).

• The export command declares a variable as ‘global’.• Child processes will be able to use exported variables.

• $ export variable_name.• Child process cannot pass environmental changes to its parent.


Defining Shell Variables• Variables are defined using assignment of the format:• Variable names must begin with a letter.• Must be exported to be seen by a child.• Bash shell permits variables to be defined and exported at the same time.• Substituted on a command line using a $name or ${name} , name=value• Use the unset command to delete variables.

name=value

$ user=Alex; export user

$ echo $hi ${user}Hello Alex

$ unset hi

$ echo $hi ${user}Alex


Handling Variables: Example•How to prepare the environment of your vi session by

defining and exporting two variables:

•Variable substitutiontakes place inside double quotes.

$ TERM=bash #define the terminal type you are using$ EXINIT= "set show number" #set desirable VI options$ export TERM EXINIT #make those variables global$ vim script.sh #vim inherits TERM and EXINIT from the environment of

current shell

$ HI="Hello"$ echo "$HI World"

hello world$ echo '$HI world'

$HI world


Standard Shell Variables• Shell has a number of standard variables:

• HOME home directory.• PWD current working directory.• OLDPWD previous working directory.• PATH search path.• PS1, PS2 prompt strings.• HISTFILE history-file name.• HISTSIZE number of lines in history file.• ENV bash-shell environment file.• TERM defines the terminal type, vi will fall over without it.• EDITOR default editor (used by some commands if VISUAL not set).

• Other useful variables:• LOGNAME your login name.• SHELL the login shell.• EXINIT ex / vi initialisation commands.• PAGER a preferred pager for man command ( more , less or pg ).


Search Path•The shell uses variable PATH to locate commands:

• The shell searches each directory until program found.• Order directories for efficient searching.• The current directory must be included explicitly.• If not, the current directory is not searched.

•Typical PATH setting: PATH=/bin:/usr/bin:.•The type command indicates where a program can be

found in the search path.


Modifying PATH: Example•Say you want to append local bin directory to search

path

$ echo $PATH /usr/bin:/bin:/usr/local/sbin:/usr/sbin:/home/Alex/.local/bin$ PATH=$PATH:/src/app/bin$ echo $PATH /usr/bin:/bin:/usr/local/sbin:/usr/sbin:/home/Alex/.local/bin:/src/app/bin

Was substituted by the $PATH variable value Was added to $PATH value when assigning the new value


Shell Prompts: Examples•- PS1 used as a primary (ordinary) shell prompt.•- PS2 used as secondary (continuation) shell prompt.•- PS4 used to force the shell to show result of line scan.


Bash startup scripts


look in /etc/init.d for examples: e.g network.

Environment Files• Environment files are read by your shell at login time.

• Usually these files are: /etc/profile, $HOME/.profile, and usually a third file with your shell’s unique definitions.

• The /etc/profile contains:• Common shell environment for all users.• Selected commands to be executed for all users at login time.

• The $HOME/.profile contains:• Environmental definitions (new or modified), customised by the user.• Commands to be executed at login time, selected by the user.

• The $HOME/.bashrc , Bash shell specific start-up file• Shell identifies its name by looking at the value of ENV variable.• ENV is usually set to $HOME/.bashrc.• contains Bash shell specifics, such as aliases, or commands unique to it, for example: set -o vi.


Environment Files: Example


• look up for bashrc file and in it lttle bit.

Startup files- bash•The bash shell reads two files when the user logs in:•/etc/profile - this is a system-wide startup file read for

every user.$HOME/.bash_profile - this is found in the user’s home directory.$HOME/.bashrc - for each subsequent shell (subshell).$HOME/.bash_logout - when a login shell exits.


Exploring the Environment• Use the set command to manipulate current environment.• set on its own will show all known variables.• set -o will display current shell options and settings.• set -o option turns the given option on.• set +o option turns the given option off.• Use the env command to list global variables.• Variables that have been marked for export by the parent.

$ set -o vi # set vi as command line editor$ set -o allexport # automatically export all new variables$ set -o ignoreeof # disallow ^D as log off special key$ set +o ignoreeof # restore ^D as log off key

Directory Shortcuts•The home directory is used to store configuration files.

• Known as environmental variable HOME,~ as a shortcut.•The current and previous working directories also have

bash-shell variables and shortcuts.

$ cd ~ --> $ cd $HOME$ cd - --> $ cd $OLDPWD$ ls ~ --> $ ls $HOME$ ls ~user --> $ ls $HOME #home of user$ ls ~- --> $ ls $OLDPWD$ ls ~+ --> $ ls $PWD


Aliases•A Bash alias is a keyboard shortcut.


Summary• The shell uses variables to configure options.• Variables are defined by commands of the form.

• name=value• Variable substitution is invoked by the special.

• character $name.• Variables must be exported to the environment to be available to other

programs.• At login time, commands are read from two files .

(/etc/profile and $HOME/.profile)• The shell variable ENV defines an environment file read by all bash shells.

(usually $HOME/.bashrc)• Command set allows to examine and manipulate shell’s environment.


IntroTo Shell Scripting

•The shell is a command interpreter.•Insulating layer between the operating system kernel

and the user.•powerful programming language.•A shell program, called a script.


Why Shell Scripting ?•Shell scripting is essential knowledge for:

• Power Users.• DevOps.• Sysadms.• SW/HW Developers.


Writing Shell Scripts•Simple scripts.•Positional parameters.•Command exit status.•Selection commands.•Looping commands.•Interactive input.


What shells are available?• Under Linux, the shell is an ordinary program. There are many available:

• Bourne SHell sh The Original.• C SHell csh 'C' language compatable.• Korn SHell ksh backwards compatable with 'sh'.• Bourne Again SHell Bash comapatable with 'sh' (Linux default).

• The C shell from UCB is a shell with a “C-like” syntax, it is not compatible with the Bourne shell.

• The Bourne Again Shell (bash) is a public domain program from the Free Software Foundation.

• The Visual shell is no longer shipped.


Reasons to use the ‘bash’ shell• Many high-level language constructs (if, while, select, etc.).• Supports powerful tests on files and directories.• Supports integer arithmetic.• Development of programs more rapid than a comparable language like C.• Interactive command history mechanism emulating either of the two most

popular Linux editors supports functions.• Advanced pattern matching constructs and ability to parse complicated strings.• Available on all modern versions of Linux, UNIX, and with other operating

systems, too and because we are stuck with it :-).


Shell Scripts•Shell programs are called scripts.

• Same as batch or command files in other systems.•Scripts are text files containing sequence of commands.

• Each line is equivalent to a command entered interactively.• Shell scripts execute in a separate sub-shell.

•Script files must have execute permission.• Normally, scripts are in the search path.• Otherwise, full pathname must be specified.

cat >laecho "Full List List of files"ls -la

^D (ctrl+D)$chmod +x la$ ./la


Example 1#!/usr/bin/env bash #alternative header for bash

##########################################developer: br0k3ngl255#purpose: log clean up#suggestions : run as root#date :01.03.2017#########################################LOG_DIR=/var/log

cd $LOG_DIR

cat /dev/null > messagescat /dev/null > wtmp

echo "Logs cleaned up."exit

#commads executed manuallycd /var/logcat /dev/null > messagescat /dev/null > wtmpecho "Logs cleaned up."


Ex1•Please go to exercise tab, at LPI_LABS and execute

execise 1 in LPI102_Labs.


Variable Substitution•The name of a variable is a placeholder for its value, the

data it holds. Referencing (retrieving) its value is called variable substitution.

$ var=123$ echo var-> var$ echo $var->123


Running a Shell Script•Enter command (script) filename on the

command line $ ls.•Start a new shell and specify the script

filename as argument.•Use the ‘dot’ command:

• Which means: ask the current shell to read the command file.

• Often used for start-up files after modifications.

• No sub-shell created, thus environment shared.

$ ls

$ sh ls

$ . ./ls

$ vi ./profile...

$ . ./.profile


Using # in Shell Scripts• Use # character to include comments in

scripts:• Shell scripts are programs and should be

annotated with comments.• Shell comment starts with # and extends to

the end of the line.• Use # character as part of script

‘header’.• A sub-shell is used to run a script.• The Bourne shell is used by default,

unless the header says otherwise.• Headers allow to integrate scripts

written in different shells.

$ more ladate#full listing filesls -la


type•Used to find out about commands the shell executes.

$ type cdcd is a shell built in$ type man man is hashed (/usr/bin/man)$ type if if is a shell keyword keyword$ type -type man file$type -path man /usr/bin/man$ type -all manything manything is aliased to ‘echo “Hi”’manything is a functionmanything (){ echo “Hi” }manything is ./manything


Positional parameters• The shell assigns arguments to variables called “positional parameters” according to

their position.• The first argument is assigned to $1, the second to $2, the third to $3, etc.• $0 contains the name of the program/script currently executing.• A bug in the original Bourne shell does not allow argumentsbeyond 10 to be accessed cleanly.• Unfortunately, this bug has been preserved for backwards compatibility but you can

use {} to assign it.

$ myargsecho $0 $1 $9 $10 $11$ one two three myargs one two three four four five six seven eight nine ten ten eleven eleven./myargsone nine one0 one1


Collective arguments•Arguments may be referred to collectively by two other

special variables set by the shell, $*, and $@.•They are exactly the same except when used between

quotation marks.•A count of the number of arguments is placed in $#.

$cat myargsecho $#echo $*echo $@# ./myargs one two three four4one two three fourone two three four


The shift built-in•Due to the problems dealing with the ninth argument

and beyond, programmers use the shift command•The shell shifts the positional parameters left by n,

these parameters being lost. The default is 1.$ cat myargsecho $#: $0 $*shiftecho $#: $0 $*shift 3 echo $#: $0 $*$ ./myargs one two three four4: myargs one two three four 3: myargs two three four 0: myargs


Reading User Input• Data can be provided by the user, using the read

command:

• Reads one line from standard input.• Line is split into arguments.• First argument is assigned to first variable (data1), second

argument to second variable(data2 ) and so on.

• Left-over arguments are assigned to last variable.• Prompt for input to read using echo or by -r option

$ cat tmp_input.shecho -n "Please enter your terminal type: "

read TMPecho $TMP


read data1 data2 ...

Command Exit Status• All Unix commands return an exit status

• Exit status informs the parent whether command succeeded or failed• Exit status available in ? variable

• Exit status is always a number• 0 means success• Non-zero value means failure• Meaning of success/failure depends on the command

• Use exit command in scripts to return a status to their calling program or script$who |grep mobius

$ echo $?1$


Flow Control in Shell Scripts• Flow control can be managed with if-then-fi statement

combination.• That statement uses exit status of a command to decide what to

do next:• if command(s) only if the last command succeeded --> then statement

goes to next set of commands in block command(s) --> run commands listed here:

• fi closes the statement(this one is one most important)

$ cat day_check.sh#!/usr/bin/env bashif date +%a |grep Frithen

echo "fill in the timesheet"fi$


The Test Command•The test command checks variable and file attributes:

• Does not produce any output, we only use it for its exit status.• Returns exit status 0 (succeeds) when condition is true.• Often used in conjunction with the if-then-fi statement.

•Bash shell uses [ ] compound for test command :• Preferred way of specifying tests.• More powerful tests can be performed.

•Testing for length of variable value.• test -z "$var" or [ -z "$var" ]• test -n "$var" or [ -n "$var" ]

•Testing variables for values.• test "$var" = value or [ "$var" = value ]• test "$var" != value or [ "$var" != value ]


Test : Examples•Hints for specifying correct

syntax:•Always put spaces around all

arguments to [ ]•Always enclose variables with

double quotes

if [ -z "$TERM" ];then echo "Terminal error"fi

if [ "$#" != 2 ] ;then echo "please provide 2 args"fi

if [ $"PWD" == $"HOME" ];then echo "you are at $HOME"fi


More Test Conditions•The test command can be used to check file attributes:

• [ -r filename ] file exists and is readable• [ -w filename ] file exists and is writable• [ -x filename ] file exists and is executable• [ -f filename ] file exists and is writable

if [ -r $HOME/.bashrc ];then echo "bashrc exists and is readale"

fi


More Test Conditions• [ -e filename ] file exists• [ -d filename ] file exists and is directory• [ 5 -gt 10 ] 5 is greeater then 10• [ 5 -lt 10 ] 5 is less then 10• [ 5 -ge 10 ] 5 is greeater or equal to 10• [ 5 -le 10 ] 5 is lesser or equal to10• [ 5 -gt 10 ] 5 is greeater then 10• [ 5 -eq 10 ] 5 is equal to 10


Combining Test Conditions• Bash also has logical operators: AND (&&), OR (||) and NOT(!)• && - can check 2 conditions but will check 2nd only if 1st successds.

• || - can check 2 conditions but will check 2nd only if 1st will NOT successds

if [ -e "$HOME"/testfile ] && [ -d $"HOME"/testfile ];thenecho "file exists and is a directory"

fi

if [ -e "$HOME"/testfile ] || [ -d $"HOME"/testfile ];thenecho "file exists and is a directory"

fi


• The conditional expression is meant as the modern variant of the test command. Since it is not a builtin command, Bash doesn't need to apply the normal commandline parsing rules like recognizing && as command list operator.

Conditional Expression: The Safer “Test” Command substitution

( <EXPRESSION> ) Used to group expressions, to influence precedence of operators<EXPRESSION1> && <EXPRESSION2> TRUE if <EXPRESSION1>and<EXPRESSION2> are TRUE (do not use -a!)<EXPRESSION1> || <EXPRESSION2> TRUE if <EXPRESSION1>or<EXPRESSION2> is TRUE (do not use -o!)<STRING> == <PATTERN> <STRING> is checked against the pattern <PATTERN> -

TRUE on a match, but note, quoting the pattern forces a literal comparison.<STRING> = <PATTERN> equivalent to the == operator<STRING> != <PATTERN> <STRING> is checked against the pattern <PATTERN> - TRUE on no match<STRING> =~ <ERE> <STRING> is checked against the extended regular expression <ERE> - TRUE on a match


Conditional Expression • When the == and != operators are used, the string to the right of the operator is considered a

pattern and matched according to the rules of pattern matching. If the shell option nocasematch is enabled, the match is performed without regard to the case of alphabetic characters.

• Any part of the pattern may be quoted to force it to be matched as a literal string.• When the operators < and > are used (string collation order), the test happens using the current

locale when the compat level is greater than "40".

Do not use the test-typical operators -a and -o for AND and OR, they are not known to the conditional expression. Instead, use the operators && and ||.

( <EXPRESSION> ) ! <EXPRESSION> <EXPRESSION1> && <EXPRESSION2> <EXPRESSION1> || <EXPRESSION2>


Conditinal Expression: Word splitting• Word splitting and pathname expansion are not performed in the expression

you give. That means, a variable containing spaces can be used without quoting:

• Compare that to the classic test command, where word splitting is done (because it's a normal command, not something special):

sentence="Be liberal in what you accept, and conservative in what you send"checkme="Be liberal in what you accept, and conservative in what you send"if [[ $sentence == $checkme ]]; then echo "Matched...!"else echo "Sorry, no match :-("fi

sentence="Be liberal in what you accept, and conservative in what you send"checkme="Be liberal in what you accept, and conservative in what you send"if [ “$sentence” == “$checkme” ]; then echo "Matched...!"else echo "Sorry, no match :-("fi


Conditional Expression: Regular Expression Matching• Using the operator =~, the left hand side operand is matched against the extended regular expression (ERE) on the

right hand side. This is consistent with matching against patterns: Every quoted part of the regular expression is taken literally, even if it contains regular expression special characters.

• Best practise is to put the regular expression to match against into a variable. This is to avoid shell parsing errors on otherwise valid regular expressions.

• The interpretation of quoted regular expression special characters can be influenced by setting the compat31 and compat32 shell options (compat* in general)

• .

REGEX="^[[:upper:]]{2}[[:lower:]]*$"# Test 1STRING=Helloif [[ $STRING =~ $REGEX ]]; then echo "Match."else echo "No match."fi# ==> "No match."

REGEX="^[[:upper:]]{2}[[:lower:]]*$"# Test 2STRING=HElloif [[ $STRING =~ $REGEX ]]; then echo "Match."else echo "No match."fi# ==> "Match."


Case StatementThe case-statement can execute commands based on a pattern matching decision. The word <WORD> is matched against every pattern <PATTERNn> and on a match, the associated list <LISTn> is executed. Every commandlist is terminated by ;;. This rule is optional for the very last command list:

47Copyright Alex M. Schapelle , VAIOLabs LTD, www.MobiusDevTeam.com, www.vaiolabs.com, License: GPLv3

printf '%s ' 'Which fruit do you like most?'read -${BASH_VERSION+e}r fruit

case $fruit in apple) echo 'Mmmmh... I like those!' ;; banana) echo 'Hm, a bit awry, no?' ;; orange|tangerine) echo $'Eeeks! I don\'t like those!\nGo away!' exit 1 ;; *) echo "Unknown fruit - sure it isn't toxic?"esac

Looping • For every word in <WORDS>, one iteration of the loop is performed and the variable <NAME> is set to the current

word. If no "in <WORDS>" is present to give an own word-list, then the positional parameters ("$@") are used (the arguments to the script or function). In this case (and only in this case), the semicolon between the variable name and the do is optional.

• If you use the loop-variable inside the for-loop and it can contain spaces, you need to quote it, since normal word-splitting procedures apply.


for x in 1 2 3{ echo $x}

#!/usr/bin/env bashecho "Directories in $PWD"for file in *;do

if [[ -d $file ]];ls -ld $file

fidone

Looping : C Style for loop• Nested for loops means loop within loop. They are useful for when you want to repeat

something serveral times for several things. For example, create a shell script called nestedfor.sh and run it:


#!/usr/bin/env bash# A shell script to print each number five times.for (( i = 1; i <= 5; i++ )) ### Outer for loop ###do

for (( j = 1 ; j <= 5; j++ )) ### Inner for loop ### do echo -n "$i " done

echo "" #### print the new line ###done

Looping : While• The while-loop is relatively simple in what it does: it executes the command list <LIST1> and if the

exit code of it was 0 (TRUE) it executes <LIST2>. This happens again and again until <LIST1> returns FALSE.

• Here is example for us to run:


#!/usr/bin/env bash # set n to 1n=1

# continue until $n equals 5while [ $n -le 5 ]do

echo "Welcome $n times."n=$(( n+1 )) # increments $n

done

Shell FunctionsA function is a subroutine a "black box" that performs a specified task

function func_name(){some commands

}

orfunction func_name {

some commands}

or func_name(){

some commands}


Arrays• An array is a parameter that holds mappings from keys to values. Arrays are used to store

a collection of parameters into a parameter. Arrays (in any programming language) are a useful and common composite data structure, and one of the most important scripting features in Bash and other shells.

• The following explicitly give variables array attributes, making them arrays:


ARRAY=() Declares an indexed array ARRAY and initializes it to be empty. This can also be used to empty an existing array.ARRAY[0]= Generally sets the first element of an indexed array. If no array ARRAY existed before, it is created.declare -a ARRAY Declares an indexed array ARRAY. An existing array is not initialized.declare -A ARRAY Declares an associative array ARRAY. This is the one and only way to create associative arrays.

Arrays: Storing Values

• Storing values in arrays is quite as simple as storing values in normal variables.


ARRAY[N]=VALUE Sets the element N of the indexed array ARRAY to VALUE. N can be any valid arithmetic expression.ARRAY[STRING]=VALUE Sets the element indexed by STRING of the associative array ARRAY.(ARRAY=()), the array will be set to an empty array. This method obviously does not use explicit indexes. ARRAY=([S1]=E1 [S2]=E2 …) Individual mass-setting for associative arrays. The named indexes (here: S1 and S2) are strings.ARRAY+=(E1 E2 …) Append to ARRAY.ARRAY=("${ANOTHER_ARRAY[@]}") Copy ANOTHER_ARRAY to ARRAY, copying each element.

As of now, arrays can't be exported.

Arrays: Getting Values

• For clarification: When you use the subscripts @ or * for mass-expanding, then the behaviour is exactly what it is for $@ and $* when mass-expanding the positional parameters. You should read this article to understand what's going on.


${ARRAY[N]} Expands to the value of the index N in the indexed array ARRAY. If N is a negative number, it's treated as the offset from the maximum assigned index (can't be used for assignment) - 1${ARRAY[S]} Expands to the value of the index S in the associative array ARRAY.

Arrays: Destruction

• The unset builtin command is used to destroy (unset) arrays or individual elements of arrays.


unset -v ARRAYunset -v ARRAY[@] Destroys a complete arrayunset -v ARRAY[*] Destroys a complete arrayunset -v ARRAY[N] Destroys the array element at index Nunset -v ARRAY[STRING] Destroys the array element of the associative array at index STRING

Documents

Bash Shell Programming - VAIOLabs · •Bourne Again SHell Bash comapatable with 'sh' (Linux default). •The C shell from UCB is a shell with a “C-like” syntax, it is not compatible