View
51
Download
0
Category
Tags:
Preview:
Citation preview
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
System Administration
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
Key Knowledge Areas
Redirecting standard input, standard output and standard error. Pipe the output of one command to the input of another command. Use the output of one command as arguments to another command. Send output to both stdout and a file.
Unix Commands
Use streams, pipes and redirects
Terms and Utilities
tee xargs
2
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
Process text streams using filters
Rundown of commands
commands Overview
3
tee - send output to standard output and files
xargs - expand input to command line arguments
< - redirect standard input from a file
<< - redirect standard input from text in a script
> - send standard output to a file
>> - append standard output to a file
| - pipe a programs's standard output to standard input of another
` ` - capture the output of a command and use it in a script
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
Use streams, pipes and redirects
bash shell, and the other shells employed by Linux allow the user to specify what should happen to the input and output of programs that run.
Input and output redirection
4
shell creates new processes as follows:
1.shell reads the commands to be executed
2.shell opens the files which will be needed (open(2) for files, pipe(2) for streams)
3.shell creates the new processes which will be executed (fork(2) each process)
4.Each new process is connected to the appropriate plumbing (dup2(2) each redirection)
5.Control of processes is passed to programs the user asked for (execve(2) each command)
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
Use streams, pipes and redirects
Input and output redirection
5
Redirect with pipes (|) are processed 1st, other redirect (> and <) are processed left to right.
Redirection Effect of redirection
command1 | command2 Output from command1 is input for command2
command < file Command reads input from a file
command > file Output of command goes to file
command 2> file Errors from the command go to a file
command >> file Output of a command is added to a file
command > file 2>&1 Output and errors go to a filecommand >& filecommand &> file
$ ls > file$ cat < file$ ls -la | grep file$ cat /etc/passwd | sed 's/:.*//'$ badcommandorfilename 2> errorlog$ grep pop3 /etc/* 2>/dev/null$ grep pop3 /etc/* 2>&1 | less$ less < errorlog$ less errorlog
Ex:
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
Use streams, pipes and redirects
Input and output
6
Normal Flow of Data:
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
Use streams, pipes and redirects
Output Redirects
7
Simple output redirection. Creates/overwrites file.
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
Use streams, pipes and redirects
Output Redirects
8
stderr output redirection. Creates/overwrites file.
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
Use streams, pipes and redirects
Output Redirects
9
Combined out & err redirection. Creates/overwrites files.File names must be different!
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
Use streams, pipes and redirects
Output Redirects
10
Combined out & err redirection. Creates/overwrites files.Only one file name, used for both output streams
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
Standard output redirection (>, >>)
[jack@foo jack]$ uptime5:09pm up 8:26, 6 users, load average: 0.22, 0.20, 0.20
Ex:
Standard output is the terminal, by default. The following redirections are possible:
11
Use streams, pipes and redirects
No redirection
Redirection to a file (“>” means “to”). If the file already exists, it is overwritten.
[jack@foo jack]$ uptime > Uptime-file[jack@foo jack]$ ls -l Uptime-file-rw-rw-r-- 1 jack jack 62 Apr 8 17:09 Uptime-file[jack@foo jack]$ cat Uptime-file5:09pm up 8:26, 6 users, load average: 0.13, 0.18, 0.19
Ex:
Redirection appending to a file (“>>” means “to the end of”)Ex:[jack@foo jack]$ uptime >> Uptime-file[jack@foo jack]$ cat Uptime-file5:09pm up 8:26, 6 users, load average: 0.13, 0.18, 0.195:10pm up 8:27, 6 users, load average: 0.24, 0.21, 0.20
Redirecting to a program. Here the output of uptime is being piped as the input to wc.Ex:jack@foo jack]$ uptime | wc1 10 62
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
Standard error redirection (2>, 2>>, 2>&1)
By default, Linux processes produce two error streams. Regular output is sent to stdout (standard output “1”) and errors are sent to stderr (standard error “2”).
12
Use streams, pipes and redirects
Ex: [jack@foo jack]$ tar cf sysconfig.tar /etc/sysconfigtar: Removing leading `/' from member namestar: /etc/sysconfig/rhn/systemid: Cannot open: Permission deniedtar: /etc/sysconfig/rhn/up2date-keyring.gpg: Cannot open: Permissiondeniedtar: /etc/sysconfig/iptables: Cannot open: Permission deniedtar: /etc/sysconfig/static-routes: Cannot open: Permission deniedtar: Error exit delayed from previous errors
Output sent to stderr:•Error messages (command 2>/dev/null)•Warning messages (command 2>> warning.log)•Debugging and progress indicators (command 2>&1 | less )
If stderr is not redirected, error output is displayed on the terminal.
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
Standard error redirection (2>, 2>>, 2>&1)
(command 2>/dev/null) sending the errors to a file. It is quite common to send error text to /dev/null
13
Use streams, pipes and redirects
Ex:[jack@foo jack]$ tar cf sysconfig.tar /etc/sysconfig 2> /dev/null[jack@foo jack]$ tar cf sysconfig.tar /etc/sysconfig 2> warning.log
append the errors to an existing file ...Ex:[jack@foo jack]$ tar cf pam.tar /etc/pam.d/tar: Removing leading `/' from member namestar: /etc/pam.d/sshd: Cannot open: Permission deniedtar: Error exit delayed from previous errors[jack@foo jack]$ tar cf pam.tar /etc/pam.d/ 2>>warning.log[jack@foo jack]$ cat warning.logtar: Removing leading `/' from member namestar: /etc/sysconfig/rhn/systemid: Cannot open: Permission deniedtar: /etc/sysconfig/rhn/up2date-keyring.gpg: Cannot open: Permission deniedtar: /etc/sysconfig/iptables: Cannot open: Permission deniedtar: /etc/sysconfig/static-routes: Cannot open: Permission deniedtar: Error exit delayed from previous errorstar: Removing leading `/' from member namestar: /etc/pam.d/sshd: Cannot open: Permission deniedtar: Error exit delayed from previous errors
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
Standard error redirection (2>, 2>>, 2>&1)
The notation 2>&1 means “stderr(2) to copy of stdout(1)”. The following all send standard error output to the same destination as standard output:• command >file 2>&1• command 2>file 1>&2• command &> file• command >& file (this is not the preferred method)• command 2>&1 | anothercommand
14
Use streams, pipes and redirects
discard the output and the errors from the commands, sending them to /dev/null.Ex:[jack@foo jack]$ tar vcf pam.tar /etc/sysconfig >/dev/null 2>&1[jack@foo jack]$ nosuchcommandorfilename 2>/dev/null 1>&2[jack@foo jack]$ ls -la /root 2>/dev/null 1>&2[jack@foo jack]$ updatedb >& /dev/null &[jack@foo jack]$ killall -HUP inetd sendmail &> /dev/null[jack@foo jack]$ killall -HUP inetd sendmail &> /dev/null
commands send standard error output to the same destination standard output (a program).Ex:[jack@foo jack]$ bash -x /etc/init.d/network restart 2>&1 | less[jack@foo jack]$ tar -c / 2>&1 | less[jack@foo jack]$ strace mailq 2>&1 | less
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
Use streams, pipes and redirects
Output Redirects Summury
15
> filecapture stdout to file; overwrites; > is equivalent to 1>2> filecapture stderr to file; overwrites> file 2> file2capture stdout to file; capture stderr to file2; overwrites
>> filecapture stdout to file; appends; >> is equivalent to 1>>2>> filecapture stderr to file; appends>> file 2>> file2capture stdout to file; capture stderr to file2; appends
> file 2>&1capture stdout to file; capture stderr to file; overwrites>> file 2>&1capture stdout to file; capture stderr to file; appends
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
Use streams, pipes and redirects
Input Redirects
16
Simple input redirection
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
Use streams, pipes and redirects
Input Redirects
17
• Input redirection isn’t common anymore - now cmds can handle their own file I/O
• Input and output redirection can be combined:cat < who.all > cat.who.allcat < who.all 2> cat.who.all.errcat < who.all > cat.who.all.all 2>&1
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
Standard input redirection (<, <<EOF, |)
[jack@foo jack]$ wcHello. I typed this line. I typed and typed. I wondered, as I typed, how many words I was typing. Tis wondrous that as I typed, bash piped.<Ctrl+D>4 28 143
Ex:
Standard input is the terminal, by default. The following redirections are possible:
18
Use streams, pipes and redirects
Read input from the terminal:
Read input from file with <. standard input anonymous: wc does not know file name which it is reading from.
[jack@foo jack]$ wc < /etc/passwd46 62 2010
Ex:
Read input from a program using the pipe redirection. Here cat is piping its output to wc.Ex:[jack@foo jack]$ cat /etc/passwd | wc46 62 2010
Read input from script or cmdline. Operator << is traditionally followed by EOF, but any letters can be used.When a line is found which consists only of EOF, that's the end of the input. This is called a “here document”.
Ex:
[jack@foo jack]$ wc << EOF> Hello> There are> Four lines> I think> EOF4 7 35
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
Use streams, pipes and redirects
Pipes
19
The output of who is piped into the input of wc -lThis produces a count of the current user sessions
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
Command pipelines (|)
Pipeline – Ability to join input and output of processes together.To set up a pipe use pipe symbol | between cmds that generates output. Generally, data piped is text, but its possible to join far too many processes together.
20
Use streams, pipes and redirects
Counting how many unique words appear in the fortune cookie database.Ex:$ ls /usr/share/games/fortune/* | grep -v '\.' | xargs cat |
sed 's/[^a-zA-Z]\+/ /g; s/^ *// ' | tr 'A-Z ' 'a-z\n' | sort | uniq | wc -l
28234
ls lists files in fortune cookie directory. grep v removes files that have a dot in the name. xargs runs cat with all of the file names on its command line. cat prints the contents of the files. sed edits the stream replacing anything that is not AZ and az with a space, then removes leading spaces. tr converts everything to lowercase. sort sorts the words in alphabetical order.uniq removes the duplicates. wc says how many lines there were in its input.
Pipes can be chained as needed, and can also be combined with redirection:who | fgrep bob | wc -l > bob.sessions
It’s even possible to intermix pipes and redirection! Just keep streams straight:who 2> who.errors | fgrep bob 2>&1 | wc -l
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
tee – divert a copy to a file
tee is like a teepiece for pipes (or T piece). tee takes one argument, a filename, and will feed all input from stdin to file, simultaneously feeding output to stdout. tee forks its input stream, sending one copy to file on disk, and another copy to stdout By default tee is like redirection > and overwrites contents in the files specified. To apend use tee -a.
21
Use streams, pipes and redirects
tee can save the results of a command in a file while view them on the terminal.Ex:
# ifconfig | tee current-netconf# route | tee current-routes
tee catches the output of grep in resultsfile, and wc counts it.Ex:
$ ls -lR /etc | grep host | tee resultsfile | wc -l$ ls -lR /etc | grep host | tee /dev/tty | wc -l
tee -a does not overwrite current contents of a file. In scripts is useful for appending results to log files.Ex:
$ rpm -hiv ppp-2.4.1-7.i386.rpm 2>&1 | tee -a install-log networklog
Recording results of loading a kernel module by reading kernel message buffer with dmesg. The results are also displayed on the terminal.
Ex:
# date | tee -a log-file# dmesg -c > /dev/null# modprobe ppp_generic | tee -a log-file# dmesg -c | tee -a log-file
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
Command substitution – $(command) and `command`
Command substitution - grab standard output of a command and put it on the command line of another command.
22
Use streams, pipes and redirects
grep and cut commands conspire here to find the name server which is in use.This is then entered into an environment variable.
Ex:
$ cat /etc/resolv.confsearch example.comnameserver 192.168.44.2$ grep nameserver /etc/resolv.confnameserver 192.168.44.2$ grep nameserver /etc/resolv.conf | cut -d ' ' -f 2192.168.44.2$ NAMESERVER=`grep nameserver /etc/resolv.conf | cut -d ' ' -f 2`$ echo $NAMESERVER192.168.44.2
Forms `command` and $(command) are equivalent. Quote: `…` is backward quote –difficult to distinguish from regular quote (apostrophe), so some prefer the form $(...)
Rather than using /usr/bin/ftp we use `which ftp`Ex:$ rpm -qif `which ftp`$ basename `which ftp`$ dirname `which ftp`
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
xargs
xargs - expands its standard input to be arguments to the command it is given.
23
Use streams, pipes and redirects
generate a list of files with grep, and then read files with less. The actual command run is less file1 file2 ...Ex:grep -l hello /etc/* | xargs lessgrep -l root /etc/* 2>/dev/null | xargs wc
The following commands are equivalent:Ex:echo one two three | xargs rmrm ` echo one two three `rm one two three
xargs is often used to handle a list of file names generated by another command.
xargs is frequently used with find. find finds the files, and xargs starts up command that handles the file.
find regular files that have not been accessed for more than 30 days, and removes themEx:find /tmp -type f -atime +30 | xargs rmfind /tmp -type f -atime +30 -print0 | xargs -0 rm
Core
Lin
ux fo
r Re
d H
at a
nd F
edor
a le
arni
ng u
nder
GN
U F
ree
Doc
umen
tatio
n Li
cens
e -
Copy
left
(c) A
cáci
o O
livei
ra 2
012
Ev
eryo
ne
is p
erm
itte
d to
co
py
and
dis
trib
ute
verb
atim
co
pie
s o
f th
is li
cen
se d
ocu
me
nt,
cha
ngin
g is
allo
wed
Fim de sessão
24
Recommended