View
225
Download
0
Category
Preview:
Citation preview
Lecture 1: Introduction, Basic UNIX
Advanced Programming Techniques
Unix Programming Environment1
Objective: To introduce students to the basic features of Unix and the Unix Philosophy (collection of combinable tools and environment that supports their use) Basic commands File system Shell Filters (wc, grep, sort, awk) File redirection, Pipes
Operating Systems
An Operating System controls (manages) hardware and software. provides support for peripherals such
as keyboard, mouse, screen, disk drives, …
software applications use the OS to communicate with peripherals.
The OS typically manages (starts, stops, pauses, etc) applications.
Kernel (OS) Interacts directly with the hardware through device drivers Provides sets of services to programs, insulating these programs from the underlying hardware Manages memory, controls access, maintains file system, handles interrupts, allocates resources of the computer Programs interact with the kernel through system calls
Structure of the UNIX system
Applications
Shell
\Kernel (OS)
Hardware
There are many standard applications:
• file system commands
• text editors
• compilers
• text processing
Unix and Users
Most flavors of Unix (there are many) provide the same set of applications to support humans (commands and shells).Although these user interface programs are not part of the OS directly, they are standardized enough that learning your way around one flavor of Unix is enough.
Flavors of Unix
There are many versions of Unix that are used by lots of people: SysV (from AT&T) BSD (from Berkeley) Solaris (Sun) IRIX (SGI) AIX (IBM) LINUX (free software)
Logging In
To log in to a Unix machine you can either: sit at the console (the computer itself) access remotely, via SSH, e.g.
The system prompts you for your username and password.Usernames and passwords are case sensitive!
CS Dept. Accounts
See http://www.cs.drexel.edu/~kschmidt/Ref/csLogin.htmlAll CS machines (that you have access to) running Linux tux machines – the farm you may
connect to from anywheretux.cs.drexel.edu
lab machines – any of the desktops you may sit at in the lab, classrooms
Not administered by Drexel IRT
Username
(typically) a sequence of alphanumeric characters of length no more than 8.the primary identifying attribute of your account.(usually) used as an email addressthe name of your home directory is usually related to your username.
Passworda password is a secret string that only the user knows (not even the system knows!)When you enter your password the system encrypts it and compares to a stored string.passwords should have at least 6 charactersIt's a good idea to mix case, include numbers and/or special characters (don't use anything that appears in a dictionary!)
User's Home Directory
The user’s personal directory. E.g., All home (users') directories (on tux)
are in /home/home/kschmidt/home/vzaychik
Where all your files go (hopefully organised into subdirectories)Mounted from a file server – available (seemlessly) on *any* department machine you log into
Home Directory
Your current directory when you log incd (by itself) takes you homeLocation of many startup and customization files. E.g.:.vimrc .bashrc .bash_profile .forward .pl
an .mozilla/ .elm/ .logout
Files and File Names
A file is a basic unit of storage (usually storage on a disk).Every file has a name.Filenames are case-sensitive!Unix file names can contain any characters (although some make it difficult to access the file) except the null character and the slash (/).Unix file names can be long! how long depends on your specific flavor of Unix
Directories
A directory is a special kind of file - Unix uses a directory to hold information about other files.We often think of a directory as a container that holds other files (or directories).A directory is the same idea as a folder on Windows.
More about File Names
Every file has a name (at least one)Each file in the same directory must have a unique name.Files that are in different directories can have the same name.Files that start with a '.' are, by default, hidden by many utilities
The Filesystem (eg)
/
bin etc home/ tmp usr
hollid2 scully bin etc
netprog unix X ls who
Unix Filesystem
The filesystem is a hierarchical system of organizing files and directories.
The top level in the hierarchy is called the "root" and holds all files and directories in the filesystem.
The name of the root directory is /
Pathnames
The pathname of a file includes the file name and the name of the directory that holds the file, and the name of the directory that holds the directory that holds the file, and the name of the … up to the rootThe pathname of every file in a given filesystem is unique.
Pathnames (cont.)
To create a pathname you start at the root (so you start with "/"), then follow the path down the hierarchy (including each directory name) and you end with the filename.In between every directory name you put a "/".
Pathname Examples
/
bin/ etc/ home/ tmp/ usr/
Hollid2/ scully/ bin/ local/
netprog unix/ X ls who
/usr/bin/lsSyllabus
/home/hollid2/unix/Syllabus
Absolute Pathnames
The pathnames described in the previous slides start at the root.These pathnames are called "absolute pathnames".Special absolute: ~kschmidt/ – /home/kschmidt (for
users’ home directories only) ~/ – Your home directory (so, relative
to login, $USER)
Relative PathnamesPrefixed w/the current directory, $PWDSo, relative to the current working directory$ cd /home/hollid2$ pwd/home/hollid2$ ls unix/Syllabusunix/Syllabus$ ls Xls: X: No such file or directory$ ls /home/scully/X/home/scully/X
Special Relative paths…
. – The current directory .. – The parent directory$ pwd
/home/holid2
$ ls ./netprog
./netprog
$ ls ../scully
X
Disk vs. FilesystemThe entire hierarchy can actually include many disk drives. some directories can be on other computers
/
bin etc users tmp usr
hollid2 scully
Commands – Basic Syntax
Shell expects the first token (for now) to be a command. Subsequent tokens are argumentsArguments that start with a – are called options (generally, Posixly)
ls -o /home/kschmidt /home/kschmidt/Public
– -o provides a long listing– The final 2 tokens are directories
to be listed
Commands for Traversing Filesystem
ls – lists contents of a directory -a – all files -l – long listing
pwd – print working (current) directorycd – change directory w/out argument, takes you home
man Pages
To get information about anything that's been properly installed, use man:man lsman catman man
– You can do a keyword search:man -k keyword
Linux boxes also have info pages
The ls command
The ls command displays the names of the named filesGive it the name of a directory as a command line argument to list all the (unhidden) files in the directory.By itself, it lists the current working directory
Command Line Options
We can modify the output format of the ls program with a command line option.The ls command support a bunch of options: -l long format (include file times, owner and
permissions). Compare to -o -a all (shows hidden* files as well as regular
files) -F include special char to indicate file types. -C place into columns
*hidden files have names that start with "."
cd – change directory
The cd command can change the current working directory:cd change directory
The general form is:cd [directoryname]
By itself, returns you to your home directory
Viewing files
cat – concatenate, send to stdout. View contents of text filesless, more – paging utilities (hit ‘h’ for help, 'q' to quit)od – octal dump. For viewing raw data in octal, hex, control chars, etc.
Copying, removing, linking
rm – remove filerm ~/tmp/download
mv – move (rename) filemv old.file ../otherDir/new.name
cp – copy filecp someDir/file someDir/file.copy
ln – create hard (inode) or soft (symbolic) links to a file
Commands for directories
mkdir make directoryrmdir remove directoryDirectories can also be moved or renamed (mv), and copied (cp –r)
Commands for Archiving
tar – Tape Archive makes a large file from many files
gzip, gunzip compression utility
tar on Linux does gzip compression with the z option:$ tar czf 571back.tgz CS571
$ tar xzf assn1.tgz
File attributes
Every file has some attributes: Access Times:
when the file was createdwhen the file was last changedwhen the file was last read
Size Owners (user and group) Permissions Type – directory, link, regular file, etc. ACLs – access control lists (not today)
File Time Attributes
Time Attributes: when the file was last changed ls -l sort by modification time ls -lt
File Owners
Each file is owned by a user.You can find out the username of the file's owner with the -l or -o option to ls:
[jjohnson@ws44 winter]$ ls -l
total 24
drwxr-xr-x 7 jjohnson users 80 Jan 3 2005 cs265/
-rw------- 1 jjohnson users 8258 Jan 3 2005 cs265.html
-rw-r--r-- 1 jjohnson users 8261 Jan 3 2005 cs265.html~
ls -l
$ ls -l foo
-rw-rw---- 1 hollingd grads 13 Jan 10 23:05 foo
permissionsowner group
size
time
name
File Permissions
Each file has a set of permissions that control who can mess with the file.There are three types of permissions:
read abbreviated r write abbreviated w execute abbreviated xThere are 3 sets of permissions:
1. user2. group3. other (the world, everybody else)
ls -l and permissions
-rwxrwxrwx User Group Others
Type of file:- – plain filed – directorys – symbolic link(others)
rwx
Files: r - allowed to read. w - allowed to write x - allowed to execute
Directories: r - allowed to see the names of the
contents w - allowed to add and remove files. x - allowed to “enter” the directory
Changing Permissions
The chmod command changes the permissions associated with a file or directory.There are a number of forms of chmod, this is the simplest:
chmod mode file
chmod – numeric modesConsider permission for each set of users (user, group, other) as a 3-bit # r – 4 w – 2 x – 1
A permission (mode) for all 3 classes is a 3-digit octal # 755 – rwxr-xr-x 644 – rw-r—r-- 700 – rwx------
chmod - examples$ chmod 700 CS571$ ls –o Personaldrwx------ 10 kschmidt 4096 Dec 19 2004 CS571/
$ chmod 755 public_html$ chmod 644 public_html/index.html$ ls –ao public_htmldrwxr-xr-x 16 kschmidt 4096 Jan 8 10:15 .drwx--x--x 92 kschmidt 8192 Jan 8 13:36 ..-rw-r--r-- 5 kschmidt 151 Nov 16 19:18 index.html
$ chmod 644 .plan$ ls –o .plan-rw-r--r-- 5 kschmidt 151 Nov 16 19:18 .plan
chmod – symbolic modes
Can be used to set, add, or remove permissionsMode has the following form:
[ugoa][+-=][rwx] u – user g – group o – other a – all + add permission - remove permission
= set permission
chmod examples
$ ls -al foo
-rwxrwx--x 1 hollingd grads foo
$ chmod g-wx foo
$ ls -al foo
-rwxr----x 1 hollingd grads foo
$ chmod u-r .
$ ls
ls: .: Permission denied
Shell as a user interface
A shell is a command interpreter, an interface between a human (or another program) and the OS runs a program, perhaps the ls
program. allows you to edit a command line. can establish alternative sources of
input and destinations for output for programs.
Is, itself, just another program
Bourne-again Shell (bash)
We’ll teach bash in this courseExtension of the Bourne Shell (sh)Contains many of the Korn Shell (ksh) extensionsYou may use the shell of your choice (tcsh, zsh, etc.), but that’s on you.
Session StartupOnce you log in, your shell will be started and it will display a prompt. (for our examples, we will use $ as
the prompt. It is not part of the input)
When the shell is started it looks in your home directory for some customization files. You can change the shell prompt,
your PATH, and a bunch of other things by creating customization files.
Customization
Each shell supports some customization. User prompt Where to find mail Shortcuts
The customization takes place in startup files – files that are read by the shell when it starts up
Startup filessh,ksh:
/etc/profile (system defaults) ~/.profile
bash:
~/.bash_profile
~/.bashrc
~/.bash_logout
csh:
~/.cshrc
~/.login
~/.logout
Incorrect login
You will receive the “Password:” prompt even if you type an incorrect or nonexistent login name– Can you guess why?Nothing will happen while you type your password. It's fine
Entering Commands
The shell prints a prompt and waits for you to type in a command.The first token on the line is taken to be a command (for now). Come in 2 flavors: shell builtin - commands that the shell
interprets directly. External programs (utilities) –
standalone programs on disk (directories in your $PATH are searched, in order)
Interpreting a Command - type
When a command is seen, the shell:1. Checks for aliases2. Checks for user-defined functions3. Looks for a builtin4. Checks directories in $PATH for a utility
Use Bash’s type builtin to see what the shell is using:kschmidt@ws60 kschmidt> type echoecho is a shell builtinkschmidt@ws60 kschmidt> type chmodchmod is /bin/chmod
Command Options and Arguments
standardized command syntax (applies to most commands):
command option(s) arguments
options modify the way in which a command works, often single letters prefixed with a dash (can be sometimes combined after a single dash
Getting helpmanual – original Unix help (flat, single page)$ man bashinfo – 2-d system, emacs-like navigation$ info bashUse the help builtin for help on Bash builtin commandsThe resource frame on the class pageInternet – google, wikipedia
The linux documentation project (http://www.tldp.org/)Safari online Friends, group-mates, and others
Some simple commandsdate – print current datewho – print who is currently logged infinger user – more information about userls -ao – lists (long) all files in a directorydu -sh – disk usage summary, human readablequota
Logging off
exit builtin (command) Exits the shell If it is the login (top-level) shell, then
it disconnects you
A shell is just another program that is running. Can recursively invoke shellsPlease don’t just disconnect w/out exiting
Standard I/OWhen you enter a command the shell creates a subshell to run the process or script.The shell establishes 3 I/O channels: Standard Input (0) – keyboard Standard Output (1) – screen Standard Error (2) – screen
These streams my be redirected to/from a file, or even another command
Programs and Standard I/O
ProgramProgramStandard Input
(STDIN)Standard Output
(STDOUT)
Standard Error(STDERR)
Terminating Standard Input
If standard input is your keyboard, you can type stuff in that goes to a program.To signal the end of input press Ctrl-D (^D), the EOF signal, on a line by itself, this closes the input stream.The shell is a program that reads from standard input.What happens when you give the shell ^D? (See the bash set command, ignoreeof)
Shell metacharactersSome characters have special meaning to the shell. These are just a few: I/O redirection< > |
wildcards* ? [ ]
others& ; $ ! \ ( ) space tab newline
These must be escaped or quoted to inhibit special behavior
Wildcards* – matches 0 or more characters? – matches exactly 1 character[<list>] – matches any single character in <list>
E.g.ls *.cc – list all C++ source files in directory
ls a* – list all files that start w/’a’ls a*.jpeg – list all JPEGs that start w/’a’ls * - (make sure you have a subdirectory, and try it)
Wildcards (more examples)
ls file?
- matches file1, file2, but not file nor file22
ls file?.*.DEL
- matches file1.h.DEL, file9.cc.DEL, file3..DEL but not file8.DEL nor file.html.DELThese are not regular expressions!
Wildcards - classes[abc] matches any of the enclosed
charactersls T[eE][sS][tT].doc
[a-z] matches any character in a rangels [a-zA-Z]*
[!abc…] matches any character except those listed.
ls [!0-9]*
Shell Variables
bash uses shell variables to store informationShell variables are used to affect the behavior of the shell, and many other programsWe can access these variables: set new values for some to customize
the shell. find out the value of some to help
accomplish a task.
Setting/Viewing VariablesTo assign (in sh, ksh, bash):VAR=someStringOTHER_VAR=“I have whitespace” Note, no whitespace around the ‘=‘!
To view (dereference) a variable:$ echo $VARsomeString$ echo $OTHER_VARI have whitespace
Shell maintains some variables
Some common ones:PATH – list of directories shell searches
for non-shell commandsPS1 – Primary promptUSER – user's login nameHOME – user’s home directoryPWD – current working directory
Other Useful OnesSHELL – the login shellTERM – the type of terminal
interfaceHISTFILE – where your command
history is savedEDITOR – holds user's preferred
editorHOSTNAME – machine's hostnameSHELLOPTS – status of various shell
options (see Bash's set built-in)
Displaying Shell Variables
Prefix the name of a shell variable with "$".The echo command will do:$ echo $HOME
$ echo $PATH
You can use these variables on any command line:
$ ls -al $HOME
Setting Shell Variables
You can change the value of a shell variable with an assignment command (this is a shell builtin command):
HOME=/etc
PATH=/usr/bin:/usr/etc:/sbin
NEWVAR="blah blah blah"
set command (shell builtin)
The set command with no args prints out a list of all the shell variables.Some bash options noclobber – won't let re-direct
overwrite an existing file ignoreeof – Shell won't exit on ctrl-D vi – use vi-style interface (by default,
bash uses emacs-style key bindings) -n – dry-run (just parse, but don't
execute). Handy for debugging scripts
Quoting – escape character, \
Use the backslash to inhibit the special meaning of the following character:$ echo $USER
kschmidt
$ echo \$USER
$USER
$ echo a\\b
a\b
Quoting – double quotes
Double quotes inhibit all behavior except variable substitution, command substitution, and the escape, \$ echo “$USER is $USER”
kschmidt is kschmidt
$ echo “\$USER is $USER”
$USER is kschmidt
$ echo “I said, \”Wait a moment\””
I said, “Wait a moment”
Quoting – single quotes
Single quotes inhibit nearly all special behaviorMay not contain a single quote$ echo ‘I said “Wait!”’
I said “Wait!”
$ echo ‘My name is $USER’
My name is $USER
$ mv rambleOnByLedZeppelin ‘ramble on – led zeppelin’
Input Redirection
The shell can read stdin from places other than your keyboard, like, a disk file, or even output from another command A file (the contents of the file are fed
to a program as if you typed it). A pipe (the output of another program
is fed as input as if you typed it).
Output Redirection
The shell can attach things other than your screen to standard output (or stderr). A file (the output of a program is
stored in file). A pipe (the output of a program is fed
as input to another program).
Redirecting stdout
Use “>” after a command (and its arguments) to send output to a file:ls > lsout if lsout previously existed it will be
truncated (gone), unless noclobber is set (see bash)
Redirecting stdin
To tell the shell to get standard input from a file, use the “<“ character:
sort < nums– Sort the lines in the file nums ,
sends the result to stdout.
You can do both!
sort < nums > sortednums
tr a-z A-Z < letter > rudeletter
Appending Output
Use >> to append append output to a file:ls /etc >> foo
ls /usr >> foo
Easy way to concatenate files:cat rest_of_file >> my_file
Redirecting stderrstderr is file descriptor 2, so:
gcc buggy.c 2> error.log
grep ‘[Vv]era’ *.html > log 2> errorlog
To send both to the same place (stdout is file descriptor 1):
find . -name 'core*' > core.lis 2>&1
Orfind . -name 'core*' 2> core.lis
Pipes – connecting processes
A pipe is a holder for a stream of data.A pipe can be used to hold the output of one program and feed it to the input of another.
prog1prog1 prog2prog2STDOUT STDIN
Asking for a pipe
Separate 2 commands with the “|” character.The shell does all the work!
ls -1 | sort
ls -1 | sort > sortedlist
ls -1 | sort | head > top.ten
The Unix Philosophy
Stringing small utilities together with pipes and redirection to accomplish non-trivial tasks easilyE.g., find the 3 largest subdirectories:$ du –s * | sort –nr | head -n3120180 Files
22652 Zaychik
9472 tweedledee.tgz
filtersPrograms that read some input (but don’t change it), perform a simple transformation on it, and write some output (to stdout)Some common filters… wc – word count (line count, character
count) tr – translate grep, egrep – search files using regular
expressions sort – sorts files by line (lexically or
numerically) cut – select portions of a line uniq – Removes identical adjacent lines head, tail – displays first (last) n lines of a
file
pipes and combining filters
Connect the output of one command to the input of another command to obtain a composition of filters
who | wc -lls | sort -fls -s | sort -nls -l | sort -nr -k4ls -l | grep ‘^d’
Process ControlProcesses are run in a subshell (by default)Subshells inherit exported variablesEach process is has an ID (pid) and a parent (ppid)Use the ps utility to look at some processes:$ ps
PID TTY TIME CMD 350 pts/4 00:00:00 bash22251 pts/4 00:00:00 vim22300 pts/4 00:00:00 ps
Process Control (cont.)
Use the –f option for a long listing:$ ps –f
UID PID PPID C STIME TTY TIME CMD
kschmidt 350 349 0 10:06 pts/4 00:00:00 -bash
kschmidt 22251 350 0 17:32 pts/4 00:00:00 vim myHomework
kschmidt 22437 350 0 17:36 pts/4 00:00:00 ps -f
Use the –e option to see more processes (all of them).$ ps –e | grep xmms
29940 pts/0 00:33:47 xmms
Killing a process (not usually nice)
The kill command sends a signal to a process (the given pid)By default, sends TERM (terminate), which asks the process to finish, so that it may do clean-upuse -9 to send a KILL (won’t be ignored), but no cleanupMy mp3 player hangs once in while:$ kill -9 29940
Job Control
The shell allows you to manage jobs place jobs in the background move a job to the foreground suspend a job kill a job
Background jobsIf you follow a command line with "&", the shell will run the job in the background. you don't need to wait for the job to
complete, you can type in a new command right away.
you can have a bunch of jobs running at once.
you can do all this within a single terminal (window).
ls -lR > saved_ls &
Listing jobs
The command jobs will list all background jobs:
> jobs
[1] Running ls -lR > saved_ls &
>
The shell assigns a number to each job (this one is job number 1).
Suspending and Resuming the Foreground Job
You can suspend the foreground job by pressing ^Z (Ctrl-Z).
Suspend means the job is stopped, but not dead. The job will show up in the jobs output.
You give fg a job number (as reported by the jobs command) preceeded by a %.
Without an argument, fg brings the last job forward
$ jobs[1] Stopped ls -lR > saved_ls &$ fg %1ls -lR > saved_ls
Placing a suspended job in the background
If it’s in the foreground, suspend itUse bg, just as you did fg, to let a suspended job continue in the background:$ bg %3
Killing a job
Kill may also take a job number or even a job name, introduced by %:$ find . –name core\* -print > corefiles &
$ firefox&
$ jobs
[1]+ Running find . –name …
[2]+ Running firefox
$ kill %2
EditorsA text editor is used to create and modify text files.The most commonly used editors in the Unix community: vi (vim on Linux)
$ vimtutor emac
$ emacs Then, hit ctrl-h t (that’s control-h, followed by ‘t’)
You must learn at least one of these editors
Recommended