UNIX Toolbox Philosophy

1. Create powerful command line tools– Each does one thing– Each does it very well

2. Design these tools to work well together3. The tools handle streams using pipes

– Filters are commands whose inputs and outputs are stream

– Job is to transform the input– input is either from stdin or a file

4. The tools are normally C programs

Bash Command Processing1. Read a command2. Transform the command

• Process history substitutions• Process alias substitutions• Process special character (break sequences) • Process variable substitutions• Process command substitutions• Process pathname expansions

3. Execute the Command4. Back to step 1 to get next command

Note: transformations occur in the order stated above

History Command• UNIX records all of the commands executed• Use the up arrow to step through previously

entered commands; the down arrow steps forward• $HISTFILE contains the name of the hidden file

with a log of previously entered commands• Ctrl R <keyword> finds the previous command

entered with <keyword>• The history command

– displays all previous commands starting with its # identifer.

– !# will execute the command with the # identifier– history > commands creates a file with a record of

past executed commands

Bash History• Turn on History command: export HISTSIZE=20• Clear history list: history -c • History list: >history• Execute nth history command: >history

>!3• Execute last command: >grep Bush grades

>!! | awk '{print $4}'• Use arguments from the last command: echo !*• Use last argument of last command: >cp prog.c

new.c>ls !$

• Use the first argument of last command: >ls –l>echo !^

• Navigate around history commands: use arrow keys

Note: argument is anything following the command name

The script command

• The script command is handy

>script OR >script scriptFile

• It logs all your input and system output to a file

• If you do not specify a file, the log goes to a file called typescript

• Type >exit to terminate the logging

Pipes• Definition: The facility to direct the output of one

command to the input of the next (cmd1 | cmd2)• The pipe character: |• Example: ls -R ~ | grep a* | sort | uniq

List all unique file names starting with a in alphabetic order

• Example: ls –al | less

Directory listing enabling us to page through the result

Streams and Redirection• C programs have default stream sources or sinks

– 0 or stdin: standard input (keyboard) – 1 or stdout: standard output (monitor)– 2 or stderr: error output (often same as stdout)

• Redirection: alter the default source or sink• Examples

– Redirect stdout to a file: ls ~ > fileName– Redirect a file list and append to a file: ls ~ >> filename– Redirect input from a file instead of stdin: sort < file_list.txt– Redirect error messages: grep harley ~ 2> file – Redirect both stdout and stderr: grep harley ~ &> file– Redirect stdout, stderr: ls –r / 1>out 2>err– Redirect stdout, stderr – ignore errors: ls –r / 1>out 2>dev/null– Redirect, append stdout and stderr: grep harley ~ &>>file– Create an empty file: >file– Type into an empty file: cat

Stream: a flow of data from source to sink

Shell Command Parsing

• The shell transforms commands before they are executed

• Examples– >echo $TERM replaces the variable with its

contents– >ls x* replaces the * with a list of files (in this

case starting with x)

Pathname Expansions• Any string of 0 or more length: *

• Any single character: ?

• Any single character specified: [abc]

• Examples:– ls cs367/*.c– ls cs367*c– ls newfile???– ls out[12]

Notes: Filenames must not have an initial period or slashes

ls Examples using Wildcards

• * means any character sequence– ls cs367/*.c, ls 367*c, ls newFile*

• ? Means any single character– ls newFile???

• [characters] means any of the characters– ls newfile[12]

• [char-char] means any within range– ls newfile[0-9][0-9][0-9]

• [:class:] means files matchin predefined class– ls [:alpha:], ls [:digit:], [:lower:]

More Command Line Expansions• Arithmetic expansion: use command line as a

calculator (cumbersome syntax)

echo $( (2+3) )echo $( ( $( (5**3) ) % 2) )

• Brace expansion: simple loop of names

echo Front-{A,B,C}-backOutput: Front-A-back Front-B-back Front-C-backecho a..cOutput: a b c

• Command substitution: command output becomes an argument to another command

echo $(pwd) or echo`pwd`Output: /home/SOU/harveyd/cs367

File Permissions

• Ls –l format:

-rw-r--r-- 1 owner group 213 Aug 26 16:31 README

• The line contains 1. - for a regular file or d for a directory2. permission string (user, group, and other (world))3. Number of links to file or directory4. the file size in decimal right-justified in a 13-byte field; 5. modification data6. file name

• Permission groups: u=user, g=group, o=other (world), a=all• Permission types:

read [r (4)], write/delete [w (2)], execute [x (1)]

Change File Mode (chmod)

• Examples: – Write for user and group, read for world

>chmod ug+w,o+r trak – All permissions for user, read and execute for

group and world: >chmod u+rwx,go+rx file.cgi

• Remove execute permission to user, read, and world: >chmod a-x fileName– An easier approach is to the use octal bits for (ogw)

– For example, read,write, execute for owner, read,write for group and read only for world >chmod 765 fileName

Purpose: Change file permissions

Concatenating files

• Method one (cp command)>cp file1 file2 file3 destination

• Method two (cat command with redirection)cat jnk.txt grades >both.txt

Note: cat is the command to type the contents of a file

• Method three (concatenation redirection)– cat jnk.txt >both.txt– cat grades >>both.txt– cat >abc allows keyboard entry into a file called abc

Parts of files: head and tail

• Display the first five lines of a file: head –n 5

• Display the last three lines of a file:

tail –n 3 file

• Copy first file lines of file to newFile:head –n 5 file >newFile

Inhibit Browser Command Line Expansion

• Problem: we might want to override special characters and have UNIX interpret them literally

• Example>echo $HOST ~ echoes the host computer plus the

home directory name

>echo \$HOST \~ echoes the literal string

>echo '$HOST ~' echoes everything in quotes literally

>echo "$HOST ~" echoes everything except for \ and $

The tar command

• Problem: want to copy a bunch of files at once• Solution: tar glues a series of files together• Note: this is not compression• tar commands

– Glue files together: >tar –cf tarfile fileList– Verbose output: tar -cvf tarFile fileList– List files in a tarFile: tar -tf tarFile– Extract files from tarFile: >tar -xvf tarFile

• Note: The dash is not needed in the above commands

The compress utilities• Compression

– greatly reduces the size of the files being transferred– Caution: Compressing a compressed file can make it bigger

• Commands– Compress a file and create a .gz extension (gzip <file>)– Uncompress a file (gunzip <file>.gz or gzip –d <file>)– Uncompress compressed tar archive (tar zxvf <file>.tar.gz)

Or > tar zxvf fileName.tar.gz

• Syncing directories (if rsync is installed)– Make sure that second directory is “synced” with the first– rsync -av --delete <first> <second>

Note: bzip2/bunzip2 work like gzip/gunzip, with another compression algorithmNote: zip/unzip are used to compress and un-compress as done in Windows

Managing Processes• Display processes and pid number

>ps displays all of your processes>ps a removes "only yourself" restriction>ps u display in a user friendly format>ps au combines the above options>ps –l displays your processes in long format

• Kill a process – Force without graceful termination (kill -9 <pid>)– Graceful termination (kill <pid> or killall <name>)

• Monitor running processes (top)

Note: the above ps options should not be preceded with a dash (-)

Printing and transferring• Printing

>lpr filename

>lpr –Pprinter filename

>lpq (see what is in the print queue)

>lpstat (status of printers)

• The sftp program allows secure file transfers>sftp hostname

Then enter commands on the remote system like cd, ls, put, get, quit

Foreground, Background, Suspend

• Concepts– A single foreground process can run within the shell– Many background processes can run outside the shell– A suspended process stops running till it is restarted

• Commands– suspend process and move to background: <ctrl>z– track background processes: >jobs– resume last suspended process in background: >bg– resume nth background process: >bg %n– move last suspended process to foreground: >fg– move nth background process to foreground: fg %n– run a command in the background >sort file &

Simple Text Processing

• grep ‘regular expression’ file…• Simple examples (Search file for the string abc)

– Display all matching lines: grep abc file– Display all, ignoring case: grep -i abc file– Count lines that match: grep –c abc file– Prefix with line numbers: grep –n abc file

• Count lines, words, characters, bytes– Show count of words, characters, bytes: wc file– Show only word count: wc -w file– Show number of lines: wc -l file– Show only byte count: wc -c file– Show character count: wc -m file

More Later

Editors: History• ed: a line editor (Ken Thompson)• ex: An improved version of ed (Bill Joy)• sed: stream editor, can apply editing commands from

the command line using streams• vi: Screen oriented text editor (We will use this one)

– Standard UNIX editor– available on all UNIX systems– smaller and faster than emacs– minimal keystrokes to do tasks, hard to learn

• emacs: (Richard Stallman)– powerful, cumbersome, part of the GNU IDE

• Others: these work more like those on windows, but they are not guaranteed to be on every UNIX system

vi editor

• Enter and exit the program– To enter: >vi fileName– To exit: colon and then wq (write

and quit) or q! (quit without write)

• Modes– insert mode: input text into a file– command mode: keyboard

command key– last line mode: longer commands

command

input

last-line

: or / esc

esccommandkey

File Pointer

• All commands are relative to a single file pointer• Central to using vi is navigating around the file• Navigation commands:

– arrows work as you would expect on most vi– Go to line 1: 1G – Go to line 50: 50G– Go forward one word: w– Go to the end of the line: $– Go to the beginning of the line:^

Delete, Cut, Copy, Paste• delete one word: dw• delete 4 words: 4dw or d4w• delete (cut) 50 lines 50dd• yank (copy the current line: yy• yank (copy) 50 lines: 50yy• yank (copy) into one of 26 buffers buffer c: "cy• paste cut/copied text after the current line: P• paste cut/copied text before the current line: p• paste from buffer c: "cp• delete one character: x• delete 50 characters: 50x

More vi Commands

• Undo command after a mistake: u• Undo all recent commands: U• join this line and the next: j• joint four lines: 4j• replace a single character: r• Save and exit: ZZ• Change the tab stops: set tabstop=4• Entering unix commands: sh

vi Insert Mode Examples

• change rest of a line until <esc>: c$• Change a word until <esc>: cw• Change line until <esc>: cc• Insert before file pointer until you <esc>: i• Append after file pointer until <esc>: a• Insert below until <esc>: o• Insert above until <esc>: O• Replace until <esc>: R

vi Last-Line examples

• Search for the word cat: /cat• Search backwords for cat: ?cat• Replace all this line: cat by dog: s/cat/dog/g• Replace once: cat by dog: s/cat/dog/• Replace once on every line: %s/cat/dog/• Replace all every line: %s/cat/dog/g• Save the file: w• Save the file as: w fileName• Save and quit: wq• Merge a file into the current place: r fileName• Display line numbers: set number• Don't display line numbers: set nonumber