Shell Scripting

A brief HOW TO guide of UNIX/Linux

Shell Commands scripting

Prepared by:

Eng. Mohamed Samy Ali

1

Shell scripting Index

Index 1.Introduction…………………………………………………2 2.A review on UNIX commands………………………..4 3.Environment variables………………………………….9 4.Pipelining and input/output redirection…………10 5.Shell scripting……………………………………………12

5.1.The 'if' command.………………………………….14 5.2.The 'case' command……………………………..16 5.3.Loops…………………………………………………..18 5.4.Arrays………………………………………………….19 5.5.functions………………………………………………20

6.Debugger………………………………………………….21

2

Shell scripting Introduction 1. Introduction: Shell scripting is about automating the repetitive procedures and simplifying the inter-connection between different UNIX/Linux programs/utilities. Basically to start shell scripting, there are two practices to be covered

1. How to use the different UNIX/Linux systems utilities. 2. Programming techniques based on sequence, selection and

iteration. Shell scripting is a programming-like environment; it is an interpreted sequence of commands with the presence of control structure. Shell commands are built-in command in UNIX/Linux shells, but what are shells? Actually shells are defined as the user interface to the system; it interprets the user commands and passes them to system to be executed.

From the block diagram above it can be noticed that "Shell" is directly above the kernel, but due to the fact that a shell itself is a compiled binary program, it exists on the same level among the other UNIX/Linux programs and utilities. So what gives a shell its advantages?

Shell

3

Shell scripting Introduction A shell's built-in commands are faster in execution than the other binaries in the system, because it does not call an external binary (program); therefore it implicitly performs an embedded procedure executing the desired functionality. When a shell executes a binary (program) it creates a new process in which the binary runs, this is also called forking. Forking means creating a new sub process (child process) in which the original process is called parent.

4

Shell scripting A review on UNIX commands 2. A review on UNIX commands:

1. cd • Changes the current directory • cd mydir Browses into "mydir". • cd.. Browses up one level. • cd - Browses to previous directory (like hitting back in a

browser). • cd Browses to the user home directory.

2. pwd • Print working directory. Displays the current absolute

path.

3. ls • List current directory contents. • ls /home/user1 Lists contents of directory

"/home/user1". • ls –l Lists with full details. • Ls –lR /home/user1 Lists recursively inside all directories

beneath /home/user1.

4. mkdir • Make directory, creates new directory. • mkdir /home/user1/mydir Creates "mydir". • mkdir –p /home/user1/dir/dir2 Creates hierarchy.

5. touch • Creates an empty file. • touch newfile Creates empty file named "newfile".

6. rmdir • Removes an empty directory • rmdir mydir Removes "mydir".

7. rm • Removes a file • rm myfile Removes "myfile". • rm –r mydir Removes a non-empty directory.

5

Shell scripting A review on UNIX commands 8. cp

• Copies a file from source to destination • cp myfile1 /home/user2/ Copies "myfile1" to

"/home/user2/". • cp myfile1 myfile2 Creates copy in the same

directory • cp –R mydir mydir2 Copies the content and

hierarchy from "mydir" to "mydir2".

9. mv • Moves a file or directory and can also rename. • mv myfile /home/user2 Moves "myfile" from current

directory to "/home/user2". • mv myfile newfile Renames "myfile" to "newfile".

10. cat • Displays the content of a file on the standard output. • cat myfile Displays the content on screen • cat –n myfile Displays the content on screen with

lines numbered

11. more • Displays the content of a file on the standard output but with

paging enabled.

12. head • Displays the first 10 lines of a file. • head myfile • head –n 5 myfile Displays first 5 lines.

13. tail • Displays the last 10 lines of a file • tail myfile • tail –n 5 myfile Displays the last 5 lines. • tail +5 myfile Displays the file and skip first 5 lines • tail –f myfile Loops forever reading the end of

file.

6

Shell scripting A review on UNIX commands 14. echo

• Echoes some text on standard output. • echo "hi" Writes "hi" on screen. • echo $HOME Writes the full path of user's home

directory.

15. df • Disk format, displays the current disk formatting, used, and

free space.

16. du • Disk usage by certain file or directory. If directory is

encountered it displays in a nested mode. • du –s Displays disk usage in a summarized manner

17. grep • Searches for a word or pattern inside a file and returns the

line/lines in which the pattern occurred. • grep user1 /etc/passwd Returns all lines with "user1" in

"file /etc/passwd". • grep –v user1 /etc/passwd Returns all files that DO NOT

contain the pattern. • grep –i user1 /etc/passwd 'grep' using case insensitive

pattern. • grep –w use /etc/passwd Returns the exact match of

"use". • grep –e user1 –e user2 /etc/passwd 'grep' for

"user1" and "user2". • With –e also regular expressions can be applied.

18. cut • Cuts a file in a vertical file like extracting columns out of file. • cut –d":" -f3 /etc/passwd '-d' denotes delimiter used

and '–f' field number. • cut –c1-8 Cuts first 8 characters.

19. sort • Output to standard output a sorted file.

7

Shell scripting A review on UNIX commands • sort –k2 –t; /etc/passwd Sorts "/etc/passwd" based on

key field '-k' number 2 and separator '-t'. • sort –u –k2 –t; /etc/passwd Sorts output with no

duplicate keys. • Add –i to indicate the sorting on numerical key. Default

manner to sort according to ASCII precedence.

20. uniq • Output a file to standard output with no successive repeated

lines. • Duplicates can still occur in file, but not successively. • uniq /etc/passwd • it is preferred to use sort –u.

21. join • Joins 2 files on a common column.

22. paste • paste file1 file2 Vertically concatenates two files

and prints them on screen.

• paste –s file file2 Concatenates two files and prints them on screen serialized.

23. find • Searches for a file in a certain path. • find <path> -<option> filename • find /home -name myfile • find /home –iname myfile Like 'name' but case insensitive. • find /home –atime <days> Files accessed "<days>" days

ago. • find /home –mtime <days> Files modified "<days>" days

ago.

24. which • Locates the paths of executables existing in $PATH variable. • Try to echo $PATH and see the result.

8

Shell scripting A review on UNIX commands 25. diff

• Compares two files and prints the differences. • diff file1 file2

26. gzip • gzip mybigfile Compression utility.

27. bzip • bzip mybigfile Compresses with rate 1:5.

28. gunzip • gunzip mybigfile.gz Decompresses "mybigfile.gz".

29. bunzip • bunzip mybigfile.bz Like 'gunzip'.

30. tar • tar cvf myfile.tar myfolder Performs archive for a

folder and its contents into "myfile.tar". • tar xvf myfile.tar Restores the original folder and

hierarchy beneath it. • c compress, x extract, v verbose, f target file name. • tar tf myfile.tar Displays the content of tar file. • tar was originally made to deal with tape drives, but the 'f'

argument is used to indicate dealing with normal files on file system.

9

Shell scripting Environment variables 3. Environment variables: These are variables stored in the current shell's session. Those variables are strings by default but can also be declared as integers.

• samy@localhost>X=5 This is an assignment of the character "5" (treated as string) into "X".

• to view the value of X we type samy@localhost>echo $X The "$" denotes the presence of variable and the echo commands prints it on screen.

• We can perform something like samy@localhost>NAME=Mohamed Or samy@localhost>LASTNAME="Samy" Both give the same value when echoed.

• samy@localhost>echo First name is $NAME and last name is $LASTNAME

• Try to echo $LOGNAME, $HOME, $SHELL, $PWD, $TERM, $PS1, $PS2,

$PS3.

10

Shell scripting Pipelining and input/output redirection 4. Pipelining and input/output redirection: UNIX and LINUX systems offer ways to redirect input and output from among processes. For example: samy@localhost>ls –l /root | wc –l Notice the '|' (pipeline). So what can a pipeline do? Simply it takes the output of the preceding command as an input to the following command. In the previous example 'ls –l' lists all the content of "/root" and the pipeline takes the output as input to 'wc –l' which counts the number of lines. As a result this will print the number of lines output from 'ls –l' which will happen to be the number of files in "/root". The previous example showed how to redirect input/output among processes, now we will redirect input/output between a command and a file. Consider the following: Example1: samy@localhost>head myfile >myfile2 Example2: samy@localhost>line <myfile2 In example 1 the 'head' command will return the first 10 lines of "myfile". But instead of printing them on the current shell window, they will be printed into "myfile2", this is due to the output redirection character '>'. In example 2 the 'line' command is a command that normally prints only the first line of a multi line text, but it does not take a file as an argument, it takes a redirected text from a file. Another example about the input redirection: samy@localhost>mailx root <myfile2 This will email the content of "myfile2" to the user "root". Last form of redirection is the `` command. In sophisticated scripts any command enclosed between `` will be first executed and then replaced with its value. For example: samy@localhost>x=`line <myfile` samy@localhost>echo $x By executing the previous commands you will find that the value of "x" became the output of 'line' command. Thus the 'line' command with first executed and its result then was assigned to the variable named "x". Consider "myfile" has the following written inside One

11

Shell scripting Pipelining and input/output redirection Two Three Four And then we perform the following command: samy@localhost>touch `cat myfile` If you then make an 'ls' you will find the presence of new four empty files named "One", "Two", "Three", and "Four", this is due to the fact that 'cat' returns the file content to the 'touch' command, which then created empty files with the arguments passed. NB: The '>' refers to output redirection to a file rather than the standard output (STDOUT). STDOUT is defined the current output facility, which we miss-call it the monitor, it is rather to be considered the current active shell session in which the command was executed, this is the valid definition unless output was redirected. The '<' refers to input redirection to input from a file rather than the standard input (STDIN), which is usually the keyboard or any other capable peripheral device unless input was redirected.

12

Shell scripting Shell scripting 5. Shell scripting: The shell provides some built-in capabilities which are somehow like programming languages as it can provide control structure and iteration. Through the following part we are going to see how to utilize the previously mentioned utilities to automate tasks and script for shell. Let's start with a "Hello world" script. You will need a text editor under UNIX which you can use. #!/usr/bin/bash echo "Hello World" samy@localhost>chmod 777 hello.sh Allows the script to be executable samy@localhost>bash hello.sh Executes the script The first line '#!/usr/bin/bash' indicates the path to sub-shell in which the commands will be executed, thus it must be the absolute path to the desired shell. You can type 'which bash' to get the full path to "bash". Let's try simple program that takes two numbers as arguments and prints their sum.

1. #!usr/bin/bash 2. let a=$1 3. let b=$2 4. let c=0 5. ((c=$a+$b)) 6. echo $c

7. samy@localhost>chmod 777 sum.sh 8. samy@localhost>bash sum.sh 2 3

In line 8 the script is called and passed arguments "2" and "3". The shell by script has some built-in variables by default like $0…$n. $0 Script name. $1…$n Argument number n. $* All passed arguments. $@ All passed arguments. $# Number of arguments. $? Exit status or return value of last executed command. $$ Current shell's process id.

13

Shell scripting Shell scripting So according to the above list the values "2, 3" passed will be assigned the shell variables $1 and $2 which are then assigned to "a, b". In lines 2, 3 & 4 the keyword 'let' declares the variables as integer variables to be allowed to perform numerical operations like the one in line 5. In line 5 the '(())' denotes the enclosure of numerical operations so shell can discriminate them from string operations, which is the default data type for shells to deal with. Key word 'let' can be replaced by 'typeset –i x', and the symbols '(())' can be replaced by 'c=`expr $a + $b`'. NB: Notice that there are no spaces to the right or the left of the '=' and there are spaces one to the right and one to the left of the '+' as shell scripting is sensitive to spaces. Try to do the following: samy@localhost>state=cal samy@localhost>echo $state samy@localhost>echo $sateifornia samy@localhost>echo ${state}ifornia samy@localhost>echo state$state You can also assign values by reading them from the user by using the keyword 'read'. 1. #!usr/bin/bash 2. let a=0 3. let b=0 4. let c=0 5. echo "please enter first value" 6. read a #read and assign value to a 7. echo "please enter second value" 8. read b #read and assign value to b 9. ((c=$a+$b)) 10. echo "the value is $c" 11. samy@localhost>chmod 777 sum.sh 12. samy@localhost>bash sum.sh 2 3

14

Shell scripting Shell scripting 5.1. The 'if' command: The 'if' statement here does not check for the correctness of the condition, it rather checks for the return of the condition, where a "0" value means success. Therefore the 'if' statement needs the 'test' command first to check for the return of the condition and then the 'if' takes a decision. The 'test' keyword can be replaced by '[ ]'.

1. #!usr/bin/bash 2. name="" #initializing variables with default

value 3. let degree=0 4. echo "Please enter your name" #reading the values 5. read name 6. echo "Please enter degree" 7. read degree 8. if [ $name == mohamed ] 9. then 10. echo "welcome back Mohamed" 11. else 12. echo "have a good day $name" 13. fi

14. if [ $degree –eq 100 ] 15. then

16. echo "excellent"

17. elif [ $degree –lt 100 –a $degree –gt 60 ]

18. then

19. echo "very good"

20. elif [ $degree –lt 60 –o $degree –eq 60 ]

21. then

22. echo "u need to work harder"

23. fi

15

Shell scripting Shell scripting 24. samy@localhost>chmod 777 iftest.sh 25. samy@localhost>bash iftest.sh 2 3

In line no.8 the 'test' command (substituted by '[ ]') checks for the validity of the logical operator used and returns valid or invalid by setting its own exist status. The 'if' command then checks for the 'test''s exist status, in which a non zero exit value means false. You can try the following on shell prompt, and notice the difference in output. samy@localhost>x=5 samy@localhost>test $x == 2 samy@localhost>echo $? samy@localhost>test $x == 5 samy@localhost>echo $? So in line no.8 the 'if' command checks for the value of $?. Notice that 'if' must be followed by "then" in next line to indicate the start of the 'if' block. The block is terminated with keyword 'fi' but it can have and 'else' written in the same block: if [ $var == $val ] then <cmd> else <cmd> fi However if we need to use nested if-else statements, instead of using 'else'we use keyword 'elif' followed by condition and followed by 'then' on next line: if [ $var == $val ] then <cmd> elif [ $var == $val2 ] then <cmd> elif [ $var == $val3 ] then <cmd> else <cmd> fi

16

Shell scripting Shell scripting Unfortunately the manner of comparison of strings ( $var == mystring ) does not work with integers. The reason is that there is no overloading in shell scripting, and since string operations are the most frequently used, therefore the normal (C-like) operators are reserved for the string operations. Follows a list for operations supported within the 'test' or '[ ]' commands. '-eq' equals for integers '-lt' less than for integers '-gt' greater than for integers '-ne' not equal for integers '-a' equals to AND operator '-o' equals to OR operator '-f' true when type is file '-d' true when type is directory '-e' true when file exists '-s' true when file exists with a non-zero length '-z' true when string is non-zero length The 'test' command also supports regular expressions, for example if you want to test for user pressed "Y" or "y" or even "Yes" or "yes" you can type the following: if [[ $answer == [Yy]* ]] Notice the spaces around "== "and

after '[[' and before ']]' then … fi Use 'man test' for more information. 5.2. The 'case' command: The 'case' command is used to select based on predefined set of expectations for user input. If the user input does not belong to the predefined expectations, there is a default value to be used.

1. #!usr/bin/bash 2. name="" Initializing variables with default value 3. echo "Enter your name" 4. read name

17

Shell scripting Shell scripting

5. case $name in 6. mohamed) 7. echo "Welcome Mohamed" 8. ;; 9. samy) 10. echo "Welcome Samy" 11. ;; 12. *) 13. echo "Do I know you?" 14. ;; 15. esac

In line 5 the beginning of 'case' command, it checks for the value entered and stored in "$name" against the cases set. The case '*' refers to anything else use this. Notice that the 'case' command ends with 'esac'. Another neat way to prompt user for choices is using the 'select' command. The 'select' command is used to display a menu of choices to the user in which he is prompted to select a choice.

1. #!usr/bin/bash 2. name="" #initializing variables with

default value 3. select name in Mohamed Samy 4. do 5. case $name in 6. Mohamed) 7. echo "Welcome Mohamed"; 8. break;; 9. Samy) 10. echo "Welcome Samy"; 11. break;; 12. *) 13. echo "Do I know you?";; 14. esac 15. done

Notice that the 'select' command creates something like a loop prompting the user for a choice therefore in lines 8, 11 the 'break' keyword is used to terminate the loop. At lines 13 and 14 the 'break' keyword is not used, therefore it will keep prompting for a known selection.

18

Notice that the "case" command is not by necessary to be used with "select" command, the "if" command can also be used. Shell scripting Shell scripting 5.3. Loops: Shell scripting supports 3 kinds of loops, while loop, until loop and for loop. The 'for' loop in shell scripting is different from the normal for loops in functionality, which will be explained further. The 'until' loop is out this guide's scoop. The 'while' loop loops on a condition similar to the 'if' command's condition and the logic is enclosed between 'do…done' statements.

1. #!/usr/local/bin/bash 2. let i=0 #initializing variables 3. let max=10 4. while [ $i -lt $max ] #while loop condition 5. do #start of loop 6. echo $i 7. ((i=$i+1)) 8. done #end of loop

The while loop checks for the value of "I" against the value of "max", while "I" is less than "max" (line 4), it keeps printing the value of "I" and then increments the value of "I" by one (line 7). The 'for' loop loops on the output of a command, in which each output line represents a value passed for the 'for'. 'for' loops make use of the '``' redirections. For example:

1. #!/usr/local/bin/bash 2. user="" Initializing variable 3. for user in `cat /etc/passwd` First 'cat' the file

"/etc/passwd", each line returned represents

4. do A value for the "user" variable 5. echo $user Prints the value of "user" which an entire

line of the file 6. done

Now replace line number 3 by the following for user in `cat /etc/passwd | cut -f1 –d":"`

The purpose of the previous line is to show that between the `` you can write any combination of commands.

19

Shell scripting Shell scripting 5.4. Arrays: Just like any other programming language, shell scripting supports the usage of arrays. Those arrays are limited to 1024 location per array. Arrays do not require previous declaration or initialization. Consider the following example, in which we wish to fill an array with the usernames of the system.

1. #!/usr/local/bin/bash 2. userArray[0]="" Initializing variable as a convention (optional) 3. let i=0 4. for user in `cut -f1 –d":" /etc/passwd` 5. do A value for the "user" variable 6. userArray[$i]=$user Prints the value of "user" which an entire

line of the file 7. ((i=$i+1)) 8. done 9. let i=0 10. while [ $i -lt ${#userArray[*]} ] 11. do 12. echo ${userArray[$i]} Special notation to print the array 13. ((i=$i+1)) 14. done

In line 2 an optional initialization of the place number zero in the array, this is just a convention to denote that "userArray" is an array. In line 3 variable "i" is declared as integer to increment with the array. In line 4 a for loop reads the values returned from the 'cut' command on "/etc/passwd" returning the usernames used on the system. In line 6 the value of the array at place of index $i is assigned by the current value of $user. In line 7 the value of $i is incremented by one. In line 9 reset the value of i to zero. In line 10 a while loop with condition value of $i is less than size of array is used to output the values of the array in a reverse manner. In line 12 the value of $i is decremented by 1. In line 12 the values are output using the echo command. The format ${var} is a special representation of the variable the same as $var but the first is more appropriate to use with arrays. Note that ${userArray [*]} or ${userArray [@]} [prints all the elements stored in the array. Also ${#userArray [*]} prints the number of elements in array.

20

Shell scripting Shell scripting 5.5. Functions One important feature that must exist normally in programming languages is modularization (use of functions) thus shell scripting supports the use of functions. Functions aids code portability and minimize the overhead of re-coding and is more organized that the normal copy-paste procedure. Functions are treated as sub-shell scripts inside the main shell; they are executed in a sub shell from the parent (main) shell executed. In shell scripts functions must be defined before they are called, thus functions are better written before the main logic of the function and after the variable declaration. This makes all variables global so no need to work around returns and tracking exist status of functions. Functions are written as follows.

1. #!/usr/local/bin/bash 2. let value=5 #variables are being global here 3. let in=0 4. let correct=0 5. function guess #function start here 6. { 7. if [ $in –eq $value ] #variable in is set from the main logic 8. then

a. correct=1 9. else

a. correct=0 10. fi 11. } #function end here 12. #start of main logic 13. echo "Try to guess the number I store." 14. Read in 15. guess #calling function 16. if [ $correct –eq 0 ] 17. then 18. echo "Wrong guess" 19. else 20. echo "Correct" 21. fi

21

Shell scripting Debugger 6. Debugger

To debug shell scripts, simply surround the piece of code you want to debug with "set –x" at the beginning of block and "set +x" at the end. #!/usr/local/bin/bash let value=5 #variables are being global here let in=0 let correct=0 function guess #function start here { set –x #start of debug if [ $in –eq $value ] #variable in is set from the main logic then correct=1 else correct=0 fi set +x #end of debug } #function end here #start of main logic echo "Try to guess the number I store." Read in guess #calling function if [ $correct –eq 0 ] then echo "Wrong guess" else echo "Correct" fi