Upload
ngonga
View
226
Download
3
Embed Size (px)
Citation preview
1
Some announcements
• Labtest II next Tuesday (Aug 11) …y ( g )
• Plan: a review lecture on Aug 18, same time, same location− To be confirmed,
− I will keep you updated
Plan for last week, today and next week
• Unix
− Introduction (file system, process, security …)( y , p , y )
− Utilities
− Shell functionalities (common)
− Bourn shell and shell script
2
Last timeOverview of UNIX
StructureFil t File systems
Processes Pipes Shells (briefly) UNIX Philosophy UNIX’s History ….
Utilities Basic cat, rmdir ….. Advanced grep, sort, cut, find, …..
Unix System Structure
user c programsscripts
shell & utilities
kernel
scripts
lsksh
gccfind
open()fork()exec()
hardware
3
UNIX Shell (briefly)• Shell is the (command line) user interface to the operating system
− between you and the raw Unix os
• Functionality:• Functionality:
− Execute other programs
− Manage files, processes
• Interactively
− When you log in, you interactively use the shell
• Scripting
− A set of shell commands that constitute an executable program: a script
datacalecho hello
Files and Processes
• A file is a collection of data that is usually stored on disk, although some files are stored on tape.
• When a program is started, it is loaded from disk into RAM. When a program is running, it is called a process.
RAM DISK
inter-processcommunication
4
Absolute and Relative Pathnames• A process may also unambiguously specify a file by
using a pathname relative to its current working directory.
• UNIX file system supports the following special fieldsthat may be used when supplying a relative pathname:
Field Meaning
. current directory
.. parent directory Same in DOS
cd ..
choose wisely cd../../../../../../../../asssignment1 ???
Relative Pathnames
• Relative Pathnames (from /home/huiwang)
readme.txt or ./readme.txt
../tim/readme.txt
../../bin/readme.txt
/
readme.txt
readme.txt
binhome
timhuiwang
readme.txt
5
Unix Security
• Processes and files have an owner and may be protected against unauthorized access.
• A set of users can form a group. A user can be a member of multiple groups.− You are in group: ugrad, submit, labtest
− use groups to check
• A special user (id 0, name root) has complete control.
• Each user has a primary (default) group.− ugrad (undergrad)
File Permissions (Security)• File permissions are the basis for file security. They are given in
three clusters. In the example, the permission settings are “rw-r--r--”:
1 - rw- r-- r-- 1 glass cs 213 Jan 31 00:12 heart final1 rw r r 1 glass cs 213 Jan 31 00:12 heart.final
User (owner) Group Others
rw- r-- r-- clusters
10
Each cluster of three letters has the same format:
Read permission Write permission Execute permissionr w x
6
File and Directory Permissions UNIX provides a way to protect files based on users and groups.
Three sets of permissions:
permissions for owner
permissions for group
permissions for other
Three types of permissions:
read process may read contents of file
write process may write contents of file
execute process may execute file (used for running script)
Same types and sets of permissions as for files apply to directories:
read process may read the directory contents (i.e., list files)
write process may add/remove files in the directory
execute process may open files in directory or subdirectories ???
Changing File Permissions: examples
Requirement Change parameters
Add group write permission g+w
Remove user read and write permission u-rw
Add execute permission for user, group, a+xand others. Give the group read permission only. g=r
Add writer permission for user, and u+w, g-rremove group read permission.
12
7
100010001
Processes
Ending a program
When a process ends, there is a return code (exit status) p , ( )associated with the process outcome
This is a non-negative integer 0 means success
anything larger than 0 represent various kinds of failure
The return value is passed to the parent process
8
Communication• Processes can communicate using a
number of means:− passing arguments, environment− read/write regular filesread/write regular files− exit values− inter-process communication with
shared queues, memory and semaphores− signals− pipes− sockets: different machines
A i i di d d t h l th t ll t• A pipe is a one-way medium-speed data channel that allows two processes on the same machine to talk.
• If the processes are on different machines connected by a network, then a mechanism called a “socket” may be used instead. A socketis a two-way high-speed data channel.
Pipeline
Th i thi b t i li i th t bl• The nice thing about pipelines is that many problems can be solved by such an arrangement of processes.
• Each process in the pipeline performs a set of operationsupon the data and then passes the results on to the next process for further processing.
Process 1 Process 2 Process 3
A B Cwho | sort
who | wc -l
who | sort | wc -l
9
Pipeline Example
abacus
abacus
abacus
abacus
3 abacus
1 bottlesort uniqbottle
abacus
abacus
bottle
1 bottlesort uniq
sort file uniq terminal
U W T
wc -l
S
sort xFile123 | uniq | wc -l
Pipe-Equivalent Communication Using a File
Could we use a file instead of a pipe? YES.
Run first program, save output into file
sort xFile123 | uniq | wc -l
Run second program, using file as input ….
sort file122 > tmp; uniq tmp > tmp2; wc –l tmp2
Disadvantages:
process 1 process 2 process 3
Disadvantages:
Unnecessary use of the disk Slower Can take up a lot of space
Makes no use of multi-tasking
Pipe is very similar, but does not involve the external device
all mechanisms stay in the realm of the operating system
10
Unix Utilities
• Standard UNIX comes complete with at least 200 small utility programs, usually including:
− shells,
− editors,
− a C compiler,
− matching with regular expressions, e.g., grep
− searching, e.g., find,
− a sorting utility, e.g. sortg y g
− software development tools,
− text-processing tools, etc. awk, sed
Basic utilities Introduces the following utilities, listed in in groups:
Generalmanclearcaldatemail
Directoryls -ldtSrmkdirrmdircdpwd
Filecatmoreheadtailcp -r
File printlplprlprmlpqlpstat
20
(quota –v)p
mvrm -rchmodwc -lcchgrpchown
11
Utilities II – advanced utilitiesIntroduces utilities for power users, listed in
alphabetical order:
t datawkbiffcmpcryptdiffdump
odtartimetrulcompresscpio
grepgunzipperlsedsortsuumount
egrepgziplnmount
croncrontabfgrepfind
uncompressuniqwhoami
gREp , eREep RE: regular expression
Pattern Maning Example
c Non-special, matches itself 'tom'
\c Turn off special meaning '\$'
$ End of line 'ab$'$ End of line ab$
^ Start of line ^ab
. Any single character '..nodes'
[…] Any single character in [] '[tT]he'
[^…] Any single character not in [] '[^tT]he'
R* Zero or more occurrences of R 'e*'
R+ One or more occ rrences of R (egrep) 'e+'
Don’t get confused
R+ One or more occurrences of R (egrep) 'e+'
R? Zero or one occurrences of R (egrep) 'e?'
R1R2 R1 followed by R2 '[st][fe]'
R1|R2 R1 or R2 (egrep) 'the|The'
22Don’t get confused with UNIX
metacharacter (file name wildcards)ls file*.c *.java cp file?.sh .
12
We introduce about thirty useful utilities.
Section Utilities
Filtering files egrep fgrep grep uniq
Introduces utilities for power users, grouped into logical sets
Utilities II – advanced utilities
Filtering files egrep, fgrep, grep, uniqSorting files sort Extracting fields cutComparing files cmp, diff Archiving files tar, cpio, dump Searching for files find Scheduling commands at, cron, crontabProgrammable text processing awk, perlHard and soft links lnSwitching users sugChecking for mail biff Transforming files compress, crypt, gunzip, gzip,
sed, tr, ul, uncompressLooking at raw file contents odMounting file systems mount, umountIdentifying shells whoamiDocument preparation nroff, spell, style, troffTiming execution of commands time23
Removing Duplicate Lines: uniq
The uniq utility displays a file with all of its identical adjacentlines replaced by a single occurrence of the repeated line.
Here’s an example of the use of the uniq utility: Here s an example of the use of the uniq utility:
$ cat animals # look at the test file. cat snake monkey snake dolphin elephant dolphin elephant
t l h t
$ uniq animals # filter out duplicate adjacent lines.
cat snake monkey snake
goat elephant pig pigpig pigmonkey pig pig pig
24
ydolphin elephant goat elephant pig pigmonkey pig pig pig
13
Removing Duplicate Lines: uniq
The uniq utility displays a file with all of its identical adjacentlines replaced by a single occurrence of the repeated line.
Here’s an example of the use of the uniq utility: Here s an example of the use of the uniq utility:
$ cat animals # look at the test file. cat snake monkey snake dolphin elephant dolphin elephant
t l h t
$ uniq -c animals # filter out duplicate adjacent lines.
1 cat snake 1 monkey snake
goat elephant pig pigpig pigmonkey pig pig pig
25
y2 dolphin elephant 1 goat elephant 2 pig pig1 monkey pig 1 pig pig
How about un-adjacent lines? sort and then uniq
Sort
• sorts a file in ascending or descending order based on one or more fields.one or more fields.
• Individual fields are ordered lexicographically, which means that corresponding characters are compared based on their ASCII values.
-t: separator is : instead of blank tab-r descending instead of ascending-r descending instead of ascending
-n numeric sort-f ignore case-M month sort (3 letter month abbreviation)
14
Sort Examples
sh-3.00$ cat ftoSortJohn Smith 1222 26 Apr 1956Tony Jones 1012 20 Mar 1950Tony Jones 1012 20 Mar 1950John Duncan 1111 20 Jan 1966Larry Jones 1223 20 Dec 1946Lisa Sue 1222 15 Jul 1980
sh-3.00$ sort ftoSortJohn Duncan 1111 20 Jan 1966John Smith 1222 26 Apr 1956
sh-3.00$ sort –r ftoSortTony Jones 1012 20 Mar 1950Lisa Sue 1222 15 Jul 1980John Smith 1222 26 Apr 1956
Larry Jones 1223 20 Dec 1946Lisa Sue 1222 15 Jul 1980Tony Jones 1012 20 Mar 1950
Lisa Sue 1222 15 Jul 1980Larry Jones 1223 20 Dec 1946John Smith 1222 26 Apr 1956John Duncan 1111 20 Jan 1966
sh-3.00$ sort -k3 –n ftoSort # +2 -3 start 0Tony Jones 1012 20 Mar 1950John Duncan 1111 20 Jan 1966John Duncan 1111 20 Jan 1966John Smith 1222 26 Apr 1956Lisa Sue 1222 15 Jul 1980Larry Jones 1223 20 Dec 1946
sh-3.00$ sort -k3 -k4 -n ftoSort # +2 -3 +3 -4Tony Jones 1012 20 Mar 1950Tony Jones 1012 20 Mar 1950John Duncan 1111 20 Jan 1966Lisa Sue 1222 15 Jul 1980John Smith 1222 26 Apr 1956Larry Jones 1223 20 Dec 1946
28
15
Sort Examples
sh-3.00$ sort -k5 ftoSort # +4 -5 John Smith 1222 26 Apr 1956L J 1223 20 D 1946Larry Jones 1223 20 Dec 1946John Duncan 1111 20 Jan 1966Lisa Sue 1222 15 Jul 1980Tony Jones 1012 20 Mar 1950
sh-3.00$ sort -k5 -M ftoSort # +4 -5 John Duncan 1111 20 Jan 1966
LexicographicallyMonths not sorted correctly
John Duncan 1111 20 Jan 1966Tony Jones 1012 20 Mar 1950John Smith 1222 26 Apr 1956Lisa Sue 1222 15 Jul 1980Larry Jones 1223 20 Dec 1946
-M enablesmonths to be sorted correctly
sort + uniq
• uniq is a little limited but we can combine it with sortsort | uniq -csort | uniq c
• counts number of times line appears in file
• output would now be:3 abacus
1 bottle
30
16
sort + uniq
• To understand:
abacus
abacus
bottle
abacus
abacus
abacus
abacus
bottlesort uniq
31
abacus bottle
Comparing Files: cmp, diff There are two utilities that allow you to compare the contents of two files:
cmp, which finds the first byte that differs between two files diff, which displays all of the differences and similarities between two files
Testing for sameness: cmp Testing for sameness: cmp
The cmp utility determines whether two files are the same.
$ cat lady1 # look at the first test file. Lady of the night, I hold you close to me, And all those loving words you say are right.
$ cat lady2 # look at the second test file$ cat lady2 # look at the second test file. Lady of the night, I hold you close to me, And everything you say to me is right.
$ cmp lady1 lady2 # files differ. lady1 lady2 differ: char 48, line 3$ _32
17
File Differences: diff
The diff utility compares two files and displays a list of editing changes that would convert the first file into the second file.
$ diff lady1 lady2 # compare lady1 and lady2.
3c3
< And all those loving words you say are right.
…
> And everything you say to me is right.
$ _
33
cut deal with fields (columns)• Used to split lines of a file
• A line is split into fields• A line is split into fields
• Fields are separated by delimiters
• A common case where a delimiter is a space:− Default is tab, (not “ ”) need to set it for blank
− cut -f3 -d” ”
• hello there world
34
field
delimiter
18
sh-3.00$ cat ftoSort # assuming tab as delimiterJohn Smith 1222 26 Apr 1956Tony Jones 1012 20 Mar 1950John Duncan 1111 20 Jan 1966Larry Jones 1223 20 Dec 1946Lisa Sue 1222 15 Jul 1980
sh-3.00$ cut -f 1 ftoSortJohnTonyJohnLarryLisa
sh-3.00$ cut -f 1,3 ftoSortJohn 1222Tony 101John 1111Larry 1223Lisa 1222
indigo 399 % cut -f 1-3 ftoSortJohn Smith 1222Tony Jones 101John Duncan 1111Larry Jones 1223Lisa Sue 1222
find Utility
• find pathList expression
• finds files starting at pathList• finds files starting at pathList
• finds files descending from there
• Allows you to perform certain actions − e.g., deleting the files
− e.g., copying , mv, rm …..
find . –name “*.c” -exec mv {} {}.bak \;
find . –name “*.class” -exec rm {} \;
“Find all the c files and make a backup of them/rename to xx.bak“
“Find all the Java class files and delete them”
19
Find Utility
• -mtime count true if the file has been modified within count days
• -atime count− true if the file has been accessed within count days
• -ctime count− true if the contents of the file have been modified within
count days or any of its file attributes have been modified• -exec command
t if th it d 0 f ti th d− true if the exit code = 0 from executing the command. command must be terminated by \; If {} is specified as a command line argument it is replaced by
the file name currently matched
Find Examples• $ find / -name x.c
− searches for file x.c in the entire file system
• $ find . -mtime 14 − lists files modified in the last 14 days
• $ find . -name '*.bak' -ls -exec rm # “*.bak”− search for all back files
$ fi d ' ? ' {} {} # “ ? ”• $ find . -name 'a?.c' -exec cp {} {}.# “a?.c”− Find all a?.c and then cp it to a?.c.bak
a1.c a1.c.bak a2.c a2.c.bak a3.c a3.c.bak
20
Find Examples• $ find / -name x.c
− searches for file x.c in the entire file system
• $ find . -mtime 14 -ls− lists files modified in the last 14 days
• $ find . -name '*.bak' -exec rm {} \;− ls and then remove all files that end with .bak
• $ find . -name ‘a2.c’ -exec cp {} a2.c.bak \;
$ fi d ‘ ? ’ {} {} b k \• $ find . -name ‘a?.c’ -exec mv {} {}.bak \;− Find ax.c and then rename it to ax.c.bak
• $ find . -name ‘*.c’ -exec cp {} {}.2015Su \;− Find x.c and then cp it to x.c.2015Su
Transforming Files: tr tr copies the standard input to the standard output with
substitution or deletion of selected characters input characters from string1 are replaced with the
corresponding characters in string2 (the last character in t i 2 i d f ddi if )string2 is used for padding if necessary)
tr –cds string1 string2
the –d option results in deletion from the input string of all characters specified in string1
the –s option condenses all repeated characters in the input string to a single character
with the –c option, string1 is substituted with its complement; i.e., all characters that are not in the original string1
40
21
Transforming Files with tr: Examples
$ tr xyz abcthis is a x world y zthis is a a world b cthis is a a world b c^D
$ cat aFilea lot of space
$tr alo xyz < aFile # transfer a to x, l to y# and o to z#
x yzt zf spxce
41
Transforming Files with tr: Examples
# transfer all lower case character to upper case$ tr [a-z] [A-Z] [:lower:] [:upper:]A LOT OF SPACE
# transfer all spaces into dots$ tr " " .a...lot.....of......space
# transfer all spaces into dots and compact the# dots
$ tr " " . | tr -s "."a.lot.of.space
$ tr 0-9 “*” < input.txt #replace digits with *
42
22
We introduce about thirty useful utilities.
section Utilities
Filtering files egrep fgrep grep uniq
Introduces utilities for power users, grouped into logical sets
Utilities II – advanced utilities
Filtering files egrep, fgrep, grep, uniqSorting files sort Extract fields cutComparing files cmp, diff Archiving files tar, cpio, dump Searching for files find Scheduling commands at, cron, crontabProgrammable text processing awk, perlHard and soft links lnSwitching users su gChecking for mail biff Transforming files compress, crypt, gunzip, gzip,
sed, tr, ul, uncompressLooking at raw file contents odMounting file systems mount, umountIdentifying shells whoamiDocument preparation nroff, spell, style, troffTiming execution of commands time
Utilities II – advanced utilities
grep/egrep
sort
grep -w –i ^[Tt]he file123
sort –t: -k4 -r -n/M file
Regular Expression
sort
uniq
cut
awk
tr
/
uniq -c
cut -d” ” -f2,3
awk –F”:” ‘{print $1,$2}’
tr a-z A-Z < filename
Start column 1, default delimiter:
blank/tab
Start column 1Default delimiter: tab
cmp/diff
ln
find
cmp/diff
ln
find . –name “*.c” -exec
cp {} {}.bak \;
Start column 1Default delimiter blank/tab
23
Plan for last week, today and next week
• Unix
− Introduction (file system, process, security …)( y , p , y )
− Utilities
− Shell functionalities (common)
− Bourn shell and shell script
UNIX Shells
Ch 4 Unix shells “UNIX for Programmers and Users”
Third Edition, Prentice-Hall, GRAHAM GLASS, KING ABLES
24
INTRODUCTION
A shell is a program that is an interface between a user and A shell is a program that is an interface between a user and the raw operating system.
It makes basic facilities such as multitasking and piping easyto use, and it adds useful file-specific features such as wildcardsand I/O redirection.
There are four common shells in use:
· the Korn shell · the C shell · the Bourne shell· the Bash shell (Bourne Again Shell)
SHELL FUNCTIONALITY
- This part describes the common core of functionality thatall four shells provide.
- The relationship among the four shells:
Common Common
Bourne shell
Korn shellC shell
Bourne Again Shell
Commoncore
Commoncore
25
SHELL FUNCTIONALITY
- A hierarchy diagram to illustrate the features shared by the four shells
Shell functions
Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing subsitution
Local Environment Conditional Unconditional
Displaying Information : echo
The built-in echo command displays its arguments to standard output and works like this:
Shell Command: echo {arg}*
echo is a built-in shell command that displays all of itsarguments to standard output.
By default, it appends a new line to the output. t fi use –n to fix
sh-3.00$ echo hello world # echo “hello world”hello worldsh-3.00$ echo -n hellohellosh-3.00$
26
SHELL OPERATIONS
Commands range from simple utility invocations like:
$ ls$ ls
to complex-looking pipeline sequences like:
$ cat compact123 | sort | uniq | wc -l
- a command with a backslash(\) character, and the shell will allow you to continue the command on the next line:
$ echo this is a very long shell command and needs to \be extended with the line-continuation character. Note \that a single command may be extended for several lines.
$ _
METACHARACTERSSome characters are processed specially by a shell and
are known as metacharacters.
All four shells share a core set of common metacharacters, whose meanings are as follow:
Symbol Meaning
> Output redirection; writes standard output to a file.>> Output redirection; appends standard output to a file.< Input redirection; reads standard input from a file. * File substitution wildcard;* File-substitution wildcard;
matches zero or more characters.? File-substitution wildcard;
matches any single character.[…] File-substitution wildcard;
matches any character between the brackets.
CSE1020 lab tour. Don’t confuse with RE
27
Symbol Meaning
`command` Command substitution; replaced by the output from
Shell functions
Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing substitution
Local Environment Conditional Unconditional
command Command substitution; replaced by the output fromcommand.
$ Variable substitution. Expands the value of a variable.
& Runs a command in the background. | Pipe symbol; sends the output of one process to the
input of another. ; Used to sequence commands. % echo hello; wc lyrics
|| Conditional execution;executes a command if the previous one fails.
&& Conditional execution; executes a command if the previous one succeeds.
(…) Groups commands.
# All characters that follow up to a new line are ignoredby the shell and program(i.e., used for a comment)
\ Prevents special interpretation of the next character.<<tok Input redirection; reads standard input from script up to tok.
- When you enter a command,the shell scans it for metacharacters and processes them specially.
When all metacharacters have been processed,the command is finally executed.
To turn off the special meaning of a metacharacter, precede it by a backslash(\) character. # Also “ ” ‘ ‘ (later)
Here’s an example:
$ echo hi > file # store output of echo in “file”. $ cat file # look at the contents of “file”. hi hi $ echo hi \> file # inhibit > metacharacter. hi > file # > is treated like other characters. $ cat file # look at the file again. Not writtenhi
$ echo 3 \* 4 = 12$ echo 3 + 2 = 5 # this is a comment
28
- When you enter a command,the shell scans it for metacharacters and processes them specially.
When all metacharacters have been processed,the command is finally executed.
To turn off the special meaning of a metacharacter, precede it by a backslash(\) character. # Also “ ” ‘ ‘ (later)
Here’s an example:
$ echo hi > file # store output of echo in “file”. $ cat file # look at the contents of “file”. hi hi $ echo hi \> file # inhibit > metacharacter. hi > file # > is treated like other characters. $ cat file # look at the file again. Not writtenhi
$ echo 3 \* 4 = 12$ echo 3 + 2 = 5 # this is a comment
Redirection > >> < <<
The shell redirection facility allows you to:
Shell functions
Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing subsitution
Local Envirnment Conditional Unconditional
1) store the output of a process to a file (output redirection) 2) use the contents of a file as input to a process (input redirection)
Output redirection
To redirect output, use either the “>” or “>>” metacharacters.
The sequence
$ a.out > fileName
$ echo “new line” > fileName$ echo “new line” >> fileName
sends the standard output of command to the file with name fileName.
Difference?
29
• Input Redirection
Input redirection is useful because it allows you to prepare a processinput beforehand and store it in a file for later use.
To redirect input, use either the ‘<‘ or ‘<<‘ metacharacters.
The sequence
$ a.out < fileName
executes command using the contents of the file fileNameas its standard input.
If the file doesn’t exist or doesn’t have read permission, an error occurs.
• FILENAME SUBSTITUTION ( WILDCARDS )
All shells support a wildcard facility that allows you to select files
Shell functions
Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing subsitution
Local Envirnment Conditional Unconditional
- All shells support a wildcard facility that allows you to select files that satisfy a particular name pattern from the file system.
- The wildcards and their meanings are as follows: ls *.c cp a?.c
Wildcard Meaning
* Matches any string, including the empty string.
? Matches any single character.
[..] Matches any one of the characters between the brackets.A range of characters may be specified by separating
a pair of characters by a hyphen. [a-d] [0-9]
Don’t confuse with Regulation Expression grep a*b file123.* grep a?.c file123?
30
- Prevent the shell from processing the wildcards in a string by surrounding the string with single quotes(apostrophes) or doublequotes.
-A backslash(/) character in a filename must be matched explicitly A backslash(/) character in a filename must be matched explicitly. - Here are some examples of wildcards in action:
$ ls *.c # list any text ending in “.c”. a.c b.c cc.c
$ ls ?.c # list text for which one character is followed by “.c”.a.c b.c
$ cp /cs/dept/course/2014-15/S/2031/xFile? .
$ cp /cs/dept/course/2014-15/S/2031/xFile* .
$ cp /cs/dept/course/2014-15/S/2031/xFile[12] .
$ ls [ac]* # list any string beginning with “a” or “c”. a.c abcd a3.pdf cc.c ce3.doc
$ ls [A-Za-z]* # list any string beginning with a letter. a.c b.c cc.c
$ ls dir*/*.c # list all files ending in “.c” files in “dir*”# directories ( that is, in any directories beginning
with “dir” ). dir1/d.c dir2/g.c
$ ls */*.c # list all files ending in “.c” in any subdirectory. dir1/d c dir2/g cdir1/d.c dir2/g.c
$ ls *2/?.? ?.? ---> list all files with extensions in “2*” directoriesand current directory.
a.c b.c dir2/f.d dir2/g.c$ _
31
• PIPES
Shell functions
Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing subsitution
Local Envirnment Conditional Unconditional
- Shells allow you to use the standard output of one processas the standard input of another process by connecting the processes
together using the pipe(|) metacharacter.
- The sequence
$ command1 | command2
th t d d t t f d1 t “fl th h” t causes the standard output of command1 to “flow through” to the standard input of command2.
- Any number of commands may be connected by pipes.
A sequence of commands changed together in this wayis called a pipeline.
$ head -4 /etc/passwdroot:eJ2S10rVe8mCg:0:1:Operator:/:/bin/cshnobody:*:65534:65534::/:daemon:*:1:1::/:sys:*:2:2::/:/bin/csh
Or, awk -F: ‘{ print $1 }’ sys:*:2:2::/:/bin/csh
$ cat /etc/passwd | cut –d “:” -f 1 | sort audit bin daemon glass ls Pipe cut Pipe sort Terminalingresnews news nobody rootsync sys tim$ cat /etc/passwd | grep –w Wang | wc -l ???
Lets do it
32
Pipeline Example
abacus
abacus
abacus
abacus3 abacus
bottle
abacus
abacus
bottle
1 bottlesort uniq
sort file uniq terminal
U W T
wc -l
S
sort xFile123 | uniq | wc –l
COMMAND SUBSTITUTION used very very … heavily in script!
A command surrounded by grave accents (‘) - back quote - is executed,
Shell functions
Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background CommandCommands Processing substitution
Local Envirnment Conditional Unconditional
A command surrounded by grave accents ( ) back quote is executed,and its standard output is inserted in the command’s place in the entirecommand line. Any new lines in the output are replaced by spaces.
For example:
$ echo the date today is ‘date`, right?the date today is Sun Jul 26 08:57:44 EDT 2015, right? $ _
$ who ---> look at the output of who. posey ttyp0 Jan 22 15:31 ( blackfoot:0.0 ) glass ttyp3 Feb 3 00:41 ( bridge05.utdalla ) huynh ttyp5 Jan 10 10:39 ( atlas.utdallas.e )
$ echo there are ‘who | wc -l` users on the systemthere are 3 users on the system
$echo there are ‘wc –l classlist` students in the classthere are 71,students in the class
$echo there are ‘cat classlist | grep –w Wang` students with name Wanthere are 1,students with name Wang
$x=`wc –l classlist` # x get value 54 (talk later)
33
COMMAND SUBSTITUTION used very very … heavily in script!
A command surrounded by grave accents (‘) - back quote - is executed,
Shell functions
Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background CommandCommands Processing substitution
Local Envirnment Conditional Unconditional
A command surrounded by grave accents ( ) back quote is executed,and its standard output is inserted in the command’s place in the entirecommand line. Any new lines in the output are replaced by spaces.
For example:
$ echo the date today is ‘date`, right?the date today is Sun Jul 26 08:57:44 EDT 2015, right? $ _
$ who ---> look at the output of who. posey ttyp0 Jan 22 15:31 ( blackfoot:0.0 ) glass ttyp3 Feb 3 00:41 ( bridge05.utdalla ) huynh ttyp5 Jan 10 10:39 ( atlas.utdallas.e )
$ echo there are ‘who | wc -l` users on the systemthere are 3 users on the system
$echo there are ‘cat calsslist | wc –l` students in the classthere are ,44 students in the class
$echo has ‘cat classlist | grep –w Wang | wc -l`students with name Wanghas 1,students with name Wang
$x=`wc –l 2031classlist` # x get value 44 (talk later)
• SEQUENCES ;
If you enter a series of simple commands or pipelines separated bysemicolons the shell will execute them in sequence from left to right
Shell functions
Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing substitution
Local EnvirnmentConditional Unconditional
semicolons, the shell will execute them in sequence, from left to right.
This facility is useful for type-ahead(and think-ahead) addicts who liketo specify an entire sequence of actions at once.
Here’s an example:
$ date; pwd; ls # execute three commands in sequence. Mon Feb 2 00:11:10 CST 1998 /home/glass/wild a.c b.c cc.c dir1 dir2 $ _
$ date > date.txt; ls; pwd > pwd.txt a.c b.c cc.c date.txt dir1 dir2
- Each command in a sequence may be individually I/O redirected as well:
34
• Conditional Sequences && ||
Every UNIX process terminates with an exit value
Shell functions
Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing substitution
Local Envirnment Conditional Unconditional
- Every UNIX process terminates with an exit value. By convention, an exit value of 0 means that the process completed successfully, and a nonzero exit value indicates failure. (opposite to C)
- All built-in shell commands return a value of >=1 if they fail. You may construct sequences that make use of this exit value:
1) For a series of commands separated by “&&” tokens, the next command is executed only if the previous command returns
an exit code of 0. i.e., successful
2) For a series of commands separated by “||” tokens, the next command is executed only if the previous command returns
a non-zero exit code. fails
- For example, if gcc compiles a program without fatal errors,it creates an executable program called “a.out” and returns an exitcode of 0; otherwise, it returns a non-zero exit code.
$ gcc myprog.c && a.out # if gcc successful, do a.out
$ gcc myprog.c || echo “compilation failed.”
$ grep huiwang classlist && echo “found in the list”
return 0 if match, return 1 otherwise
35
• Background Processing &
Shell functions
Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing substitution
Local Environment Conditional Unconditional
g g
- If you follow a simple command, pipeline, sequence of pipelines, or group of commands by the “&” metacharacter, a subshell iscreated to execute the commands as a background process
- The background process runs concurrently with the parent shell and does not take control of the keyboard.
- Background processing is therefore very useful for performing severaltasks simultaneously, as long as the background tasks do not requireinput from the keyboard.
• SHELL PROGRAMS: SCRIPTS
- Any series of shell commands may be stored inside a regular text file
Shell functions
Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing substitution
Local Environment Conditional Unconditional
Any series of shell commands may be stored inside a regular text filefor later execution.
A file that contains shell commands is called a script. batch file in Windows
Before you can run a script, you must give it execute permission by using the chmod utility.
chmod u+x filename echo hello world
To run it, you need only to type its name.
- Scripts are useful for storing commonly used sequences of commands,and they range in complexity from simple one-liners to fully blown
programs.
date
36
• SHELL PROGRAMS: SCRIPTS
one for the Bash shell and the other for the Korn shell.
$ cat > script.sh # create the bash script. #! /bin/sh# This is a sample sh script # This is a sample sh script. echo “hello world”echo -n “the date today is “ # in sh, -n omits new linedate # output today’s date.^D # end of input.
$ chmod u+x script.sh # make the scripts executable.
$ script.sh # execute the shell script. hello worldThe date today is Sun Feb 1 19:50:00 CST 2004
• VARIABLES substitution $
A h ll ki d f i bl
Shell functions
Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing substitution
Local Environment Conditional Unconditional
- A shell supports two kinds of variables: local and environment variables.
local: user defined, positional
Both kinds of variables hold data in a string format. Both kinds of variables hold data in a string format.
the child shell gets a copy of its parent shell’s environment variables, but not its local variables.
Environment variables are therefore used for transmitting useful information between parent shells and their children.
E h ll h f d fi d i i bl h
37
• Environment VARIABLES (for your reference)
- Here is a list of the predefined environment variables that are common to all shells:
Name Meaning
$HOME the full pathname of your home directory
$PATH a list of directories to search for commands
$MAIL the full pathname of your mailbox
$USER your username
$SHELL the full pathname of your login shell
$TERM the type of your terminal
-several common built-in local variables that have special meanings:
Name Meaning
Built-in local variables (must know!)
$$ The process ID of the shell.
$0 The name of the shell script ( if applicable ).
$1..$9 $n refers to the nth command line argument( if applicable ).
$* A list of all the command-line arguments.
$ myscript we are the world$0 $1 $2 $3 $4
$*
38
#! /bin/sh
echo the name of this script is $0echo the first argument is $1echo a list of all the arguments is $*echo the script places the date into a temporary file called $1.$$ date > $1.$$ # redirect the output of date.
$ script.sh paul ringo george john # execute the script. the name of this script is script.shthe first argument is paul
$ $$ pls $1.$$ # list the file. rm $1.$$ # remove the file.x=5echo the value of variable x is $x
$ variable substitution
the first argument is paula list of all the arguments is paul ringo george john the script places the date into a temporary file called paul.2431paul.2431value of variable x is 5.$ _
• QUOTING
There are often times when you want to inhibit the shell’s wildcard-substitution * ? [], variable-substitution $, and/or command-substitution ` ` mechanisms.
The shell’s quoting system allows you to do just that.
- Here’s the way that it works:
1) Single quotes(‘ ’) inhibits wildcard substitution,variable substitution, and command substitution.
2) Double quotes(“ ”) inhibits wildcard substitution only.
3) When quotes are nested, it’s only the outer quotes that haveany effect
39
• QUOTING
- The following example illustrates the difference between the two different kinds of quotes:
$ echo 3 * 4 = 12 # remember, * is a wildcard. 3 a.c b b.c c.c 4 = 12
$ echo “3 * 4 = 12” # double quotes inhibit wildcards. 3 * 4 = 12
$ echo ‘3 * 4 = 12’ # single quotes inhibit wildcards. 3 * 4 = 12
another way?$ echo 3 \* 4 = 12 # backslash inhibit a metacharacter3 * 4 = 12
$ name=Graham
- By using single quotes(apostrophes) around the text,we inhibit all wildcarding and variable and commandsubstitutions:
$ name=Graham # assign value to name variable$ name=Graham # assign value to name variable
$ echo ‘3 * 4 = 12, my name is $name - date is `date`’3 * 4 = 12, my name is $name - date is ‘date’$ _
- By using double quotes around the text, we inhibit wildcarding,but allow variable and command substitutions:
inhibitedinhibitedinhibited
$ echo “3 * 4 = 12, my name is $name - date is `date`”3 * 4 = 12, my name is Graham - date is Mon Feb 2 23:14:56 CST $ -
inhibited inhibitedinterpreted
40
- TERMINATION AND EXIT CODES
Every UNIX process terminates with an exit value.
By convention an exit value of 0 means that the processBy convention, an exit value of 0 means that the processcompleted successful, and a nonzero exit value indicates failure.
All built-in commands return an exit value of 0 if they succeedreturn an non-zero is they fail.
In the Bash, Bourne and Korn shells, the special shell variable$? always contains the value of the previous command’s$? always contains the value of the previous command sexit code.
In the C shell, the $status variable holds the exit code.
date, ls, grep, find ….. Every command has a exit code (return status)Look for man
Matching 0 No matching 1 No such file 2
-In the following example, the date utility succeeded, whereasthe gcc utilities failed:
$ date # date succeeds. Mon Feb 2 22:13:38 CST 1998 $ echo $? # display its exit value $ echo $? # display its exit value. 0 # indicates success.
$ gcc prog.c # compile a nonexistent program. cpp: Unable to open source file ‘prog.c’.
$ echo $? # display its exit value. 1 # indicates failure.
41
-In the following example, the grep utility succeeded or failed:
$ grep Wang 2031classlist$ echo $? # display its exit value. 0 # indicates success
Matching 0 No matching 1 No such file 2
0 # indicates success.
$ grep Hui 2031classlist$ echo $?1 # indicates failure (not matching).
$grep Wang 2031classlistXgrep: classlistX: No such file or directory$ echo $?2 # indicates failure (not such a file).
Used in scriptingLook for manman grep | grep –w “exit”
Try it
- Any script that you write should always explicitly return an exitcode.
To terminate a script, use the built-in exit command, which works as follows: which works as follows:
Shell Command: exit number
exit terminates the shell and returns the exit value numberto its parent process.
If number is omitted, the exit value of the previous commandis used.
42
SummaryShell functions
Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing subsitution
< >>>
* ? [ ] | ;&
` `$
• Covered core shell functionality− Built-in commands − Redirection < > >>− Wildcards * ? [ ] filename substitution
Local Environment Conditional Unconditional
&& ||$0 $1-9 $*
Wildcards ? [ ] filename substitution− Pipes |− Background processing &− Command substitution ` `− Variables $ variable substitution− Quotes “ ” ‘ ’