Unix Essentials V2.0

8/8/2019 Unix Essentials V2.0

1/45

13 December 2010 TCS Internal

Unix EssentialsCourse @ ILP, TCS Limited


2/45

13 December 2010TCS Internal 2

Course Objective

After the course you should

Have a fundamental understanding of the Unix architecture

Be able to use Unix as a platform for application development


3/45


History of unix

Ken thompson of AT&T ,BELL laboratories originally designed unix os in late 1960swhich evolved from timesharing OS called multics.

AT & T licenses source code for low cost

Originally written in assembly language but with the development of c programming

language in 1973,it was then written in c. Two versions of unix emerged are AT&T Unix

and BSD Unix


4/45


5/45


Unix Principles

Multiuser and Multitasking

Everything is a file

Small, single purpose programs

Configuration data stored in text

Ability to chain programs together to perform complex tasks


6/45


UNIX Architecture

Shell and kernel together make UNIX system work.

H/W

user

Shells

interact

with users

KERNEL

interacts

with H/W

Database

Packages

Other Application

& System Software

Compilers

Utilities orcommands

user

user

user

user


7/45


The UNIX Kernel

Kernel is a collection of programs mostly written in C, that runs directly on the hardware -so parts of kernel must be customized to each systems hardware features.

Kernel is directly loaded into memory when the system is booted. Low level jobs are itsbasic services:

System Initialization

Process/Memory/File/I-O Management

Programming Interface

Communication Facilities


8/45


The UNIX Process

A process in UNIX is a program in execution with definite life-time and well-definedhierarchy.

User Space - Process execution on user data structure that represents an applicationand is manipulated by its own functions (user mode execution).

Kernel Space - Process execution on systems data structures that represent theresources (memory, files, devices) and supported by the kernel routines. Dependingupon resource requirement and availability, processs states are controlled by executingthe kernel routines accessed through system calls (system mode execution).


9/45


Environment around the UNIX Process When a process is created, UNIX opens 3 streams stdin/stdout/stderrfor basic communication with

respect to the process control terminal. In addition, it knows the current working directory forperforming file I/O.

Each login shellmaintains a description of the environment as a table of pointers to strings.

A global shell environment pointer called environ is maintained by UNIX kernel and it can be used bya process to access its own table

The shell does not directly use this table, but it creates a child process and calls exec() system call toexecute a command program that uses the table inherited from the shell parent.

TERM=vt100USER=ashokLOGNAME=ashokHOME=/usr/ashokPATH=.:/bin:/usr/ahok/bin

extern

char

**environ

A UNIX process


10/45


Environmental Variables (EVs)

Children inherit the entire execution environment from the parent. Some examples of theenvironment variables are the USER, LOGNAME, HOME, PATH, PS1, PS2, TERM MAIL,etc.

The HOMEVariable

It specifies an associated directory with every user in a UNIX system. If the HOMEvariable for the user Sita contains/usr/sita/stores, every time Sita logs in, she is taken tothe directory stores.

The variable HOME is referenced the same way:

$ echo ${HOME}


11/45


Environmental Variables (EVs) ...

The PATHVariablecontains a list of all full path-names (separated by a colon) of directories that are to besearched for an executable program. For example, the command$PATH=.:/usr/bin:/bin specifies directories to be searched for any executable fileor a command file (current directory, /usr/bin and /bin, in that order).

The PS1 VariableThe system prompt may be changed by setting the value of this variable to the desiredprompt:$ PS1=Hello>Hello> #can be changed only at the UNIX command line, not within a shell script.

The PS2Variable: prompt string for continued command line (default > ).


12/45



The LOGNAMEVariablecontains users login name. Its contents cannot be changed by the user, but can bedisplayed:echo ${LOGNAME}

The TERMVariableNames the kind of terminal you are using; setting it helps to manage your screen moreeffectively, say,$ TERM=vt100

The PWD VariableThe current working directory can be displayed:echo ${PWD}In fact the whole environment table can be displayed.

$IFS: String of characters which are used as word separators in command line ( space,tab, newline chars).


13/45



The MAIL Variable

Names the standard file where your mail is kept Variables for AbbreviationIt is useful if frequently referred names are abbreviated: d=/very/long/directory/name

The .profile FileSome of above variables like HOME and LOGNAME are set automatically each time theuser logs in. The others, however, have to be set. The .profile is used for this purpose asit is executed as soon the user logs in. A sample .profile file would look like:PATH=.:/bin:/usr/binexport HOME PATH PS1 MAIL


14/45


Any Questions ?


15/45


Utilities and Applications

UNIX utilities or commands are collection of about 200 programs (ed, sed,awk, vi, grep, make, link, debug, etc.) that service day-to-day processingrequirements. They are invoked through the shell, which is itself anotherutility.

More than a 1000 UNIX-based application programs like DBMS, word

processors, language processors, accounting software, etc. are available fromindependent vendors


16/45


The UNIX Shell

It is the interpreter of UNIX to decipher and execute commands - user interface tothe kernel for isolating the user from the knowledge of kernel functions.

Maintains interactive dialogue with the user and is capable of Input/OutputRedirection. Can do background processing so that time-consuming, non-interactivetasks can proceed side by side.

Makes it possible to connect a couple of commands with one command taking inputfrom another (pipes). The commands that can be connected in this way are called

filters, because they filter or manipulate data in different ways. Pipes and filters formthe central feature of the UNIX philosophy. Shell variables can control the behavior of shell as well as other programs by storing

data in them.


17/45


The UNIX Shell ...

Shell includes programming language features which can be used to build shell scripts forperforming complex operations.

Shell scripts are frequently used sequence of shell commands stored in a fi le; file-name can beused later to execute the stored sequence.

Types of Shells - Bourne Shell - original command interpreter developed at AT&T by Stephen R.Bourne; fastest official shell distributed with UNIX systems (executable filename sh)

C Shell - developed by William Joy and others at UCB; gets its name from C due to syntaxresemblance of its programming language (executable file name csh)

Korn Shell - developed by David Korn, combines best features of both shells, not popular(executable file name ksh)

Restricted Shell - restricted version of Bourne shell, typically used for guest logins and in secureinstallations (executable file name).


18/45


Shell variables

Data in shell scripts and environment settings stored in variables

Conventionally all upper-case

Shell variables exist only in current shell instance

Environment variables passed to subshells

Shell variables can be exported into environment


19/45


Shell features

Command-Line expansion/ substitution

Command Completion

Command-Line Editing

Aliases

Shell Functions Command Recall

History search

Job Control


20/45


Getting HelpGetting Help

Check the manual pages!

man man - help on the man commandman who - help with the who command

info ls

ls --help


21/45


Some Shell Commands

Command Function pwd print working directory cd change directory mkdir make directory rmdir remove directory ls list contents of directory cat display contents of files cp copy one file to another rm remove files mv change name of a file/directory

or move file from one directory to another date display system date who display logged-in users names echo display string; used for prompt ln create another link to a file


22/45


The UNIX File System (UFS)

It is a hierarchical collection of 3 types of files: ordinary, directory and special files (device, pipe, fifo,socket).

A UNIX file is featureless because it is simply an array of bytes. Dominant file type in UNIX is the text file. System related files are also stored in text form. Separate device can be added by creating a file for it.

Root is the supremo and is represented by the /. Every subdirectory must have a parent. File names can be up to 14 characters long; can contain both upper and lower case alphabets, digits,

a dot, hyphen (-), underscore (_) anywhere; should not have a blank or tab; are case-sensitive. Path names are a sequence of directory names separated by /. They are used to access files. Absolute pathname - file location is determined with respect to the root. Relative pathname - file location is determined with respect to the current directory.


23/45


The UNIX File System

/bin

/dev/etc

/mnt

bin shetcgamesdictincludelib

mansbinsharetmp

root ( / )

vi/awk

cat/cutchmod

date/sort

echo

ed/sed

grep/tarkill/pwd

ln/ping

mv/ps/ls

rm/rmdir

Though the UFS looks hierarchical, it is actually a directed acyclic

graph because files can be shared.

audiobeep

cpu

fd0 to

fd7

ram

ttyxx

aliaseshosts

init.d

mail

mknod

motdnews

passwd

uucp

/usr

/home/tmp

/var/sbin/proc

lost+foundcrashftplibmail

opttmpspool

cron

faxnewsmailmqueueuucp

/lib ntpuser1to 4

lost+foundntpusr


24/45


Important Directories in Linux File System

/home It holds users home directories. In other UNIX systems, this can be /usr directory./bin It holds many of the basic Linux programs; bin stands forbinaries, files that are executable./usr It holds many user-oriented directories:

bin It holds user-oriented Linux programs.sbin It holds system administration files.spool It has several subdirectories:

. mail holds mail files

. spool holds files to be printed

. uucp holds files copied between Linux machines.

docs various documents including useful Linux infoman man pages accessed by typing the man games the fun stuff!

/sbin It holds system files that are usually run automatically./etc It and its subdirectories hold many of Linux config files./dev It holds device files.All info sent to /dev/null is thrown into trash. Your terminal is one of the /dev/tty

files.


25/45


File Access Permissions (FAP)

FAP refer to permissions associated with a file with respect to the following:

File Owner, Group Owner, Other Users.

Read r Allows displaying /copying

Write w Allows editing/deleting

Execute x Allows execution of a file

The protection on a file is referred to as its file mode on UNIX systems. The long version ofls command also displays FAP in addition to user and group ownership. The first characterindicates the file type, a hyphen for a plain file, and a dfor a directory; rest of the bytesmay be rwxrwxrwx or a - for r/w/x if that particular permission does not apply.


26/45


File Access Permissions (FAP) ...

Meaning of 0123456789 FAP bytes of a directory:

No read does not allow list or remove directory.

No write does not allow copy files to directory, remove files from directory, rename files indirectory, make a subdirectory, remove a subdirectory, move files to/from directory.

No execute does not allow display from within directory, change to directory,display a file in dir, copy a file to/from dir.

Byte 0 is b (block), c (character) or p (device) for special files.


27/45


Any Questions ?


28/45


Filters and Pipes

Filter is a program that takes its input from standard input file, processes it, and sends its output to

standard output file (simple filters: head, tail, cut, paste, sort, uniq, etc; advanced filters: grep, pg, wc, tr,awk). The grep filter(global search for regular expressions) searches files for a particular pattern. The format of

grep commandis:grep [options] reg-expr filenamesOptions:-n prints each line matching the pattern with line no.-c prints only the count of lines that match the pattern.-v prints out all those lines that do not match the pattern.Specifying grep regular expressions

\c Backslash turns off any special meaning of character c.^ Pattern following it must occur at the beginning of line.$ Pattern preceding it must occur at the end of line.. Matches any single character[ ] Matches any one of chars in ; ranges like a-z are legal[^] Matches any single char not in ; ranges are legal.r1r2 Concatenation of regular expressions: r1 followed by r2


29/45


Filters and Pipes ...

pgdisplays a large file, a screen-full at a time. wccounts number of lines, words, characters tris used to translate one set of characters to another; does not accept a filename as an

argument but only the redirected file input. cut(vertical file slit) - extracts only the desired columns from the output of certain

commands.paste (vertical file paste) - what you cutwith previous cmd can be pasted back with thepaste cmd. But while catpastes more than one file horizontally,paste does it vertically.

pipes (|)combine filters and commands such that standard output of one can be sent asstandard input to another.

Advantage of pipes: UNIX tools (commands) can be combined rather than be rewritten.


30/45


Filters and Pipes - awk

Pattern Scanning and Processing Language

$ awk program filenames whereprogram is:

pattern { action }

pattern { action }

# By Aho, Weinberger, Kernighan

awkreads the input in the filenames one line at a time. Each line is compared with each

pattern in order; for eachpattern that matches the line, the corresponding action isperformed.Awkdoes not alter its input files.

Patterns can be regular expressions exactly as in egrep, or C-like complicated conditions.


31/45


awk ...

Example 1

$ awk /regular expression/ { print } filenames

$ awk /regular expression/ filenames Both print every line that matches the regular expression

Example 2

$ awk { print } filenames

does exactly what catdoes, albeit more slowly.

Example 3

$ who | awk { print $1, $5 }prints names of logged-in users and time of login.Awksplits each input line automaticallyinto fields strings) separated by blanks or tab.


32/45


sed

Stream editor

Reads a file or stream of data, writes out the data, performing search and replaceinstructions

Uses regular expressions in search string


33/45


Using sed

Sed addresses

sed s/TCS/Tata Consultancy Services Ltd/g company.txt

Sed 1,50s/TCS/Tata Consultancy Services Ltd/g company.txt

Sed s/[Tt]cs/TCS/g company.txt

Multiple sed instructions


34/45


Any Questions ?


35/45


The viEditor

It is a visual editor used to enter and edit text files

Invoking viwith/without filename puts it in command mode:

$ vi []

vicommand mode syntax:

[]

viworks in three different modes:

y Input - where any key is entered as texty Command - where keys are used as commands to act on text

y ex- where ex commands can be entered in last line to act on text


36/45


Relationship between 3 Modes

Command

Mode

Input

Mode

exMode


37/45


Input Mode Commands

Command Actiona Appends text to right of cursorA Appends text at end of linei Inserts text to left of cursoro Inserts blank line below + inserts textO Inserts blank line above + inserts textrx Replace current character with charx

Rtext Replaces character with texts Replaces character under cursor with

many charactersS Replaces entire line20i- followed by or enters

20 hyphens (-) in one line


38/45


Save and Exit from vi

Command Action

ZZ saves file only ifvistarted with a filenameSave and Exit Commands ofexMode

:w save file and remain in edit mode

:wq save file and quit edit mode

:w write buffer to filename

:q quit editing mode when no changes

are made

:q! quit editing mode but after

abandoning changes

:x save file and quit editing mode


39/45


Cursor Movement Commands

Command Actionh or backspace Moves Cursor Leftlor spacebar Moves Cursor Right

j Moves Cursor Downk Moves Cursor UpnG Go to line numbern$ Moves cursor to end of line

w Moves cursor to next wordb Moves cursor back to previous worde Moves cursor forward to end of word

Note: W, B, E perform functions similar to w, b, e except that punctuation isskipped)


40/45


Paging and Scrolling Commands

Command Action

^d Scroll down half screen^u Scroll up half screen^for Full Page forward (12 lines by^b or Full Page backward default)^l Redraw Screen[[ Scroll to beginning of document

]] Scroll to end of document{ Scroll to previous paragraph

} Scroll to next paragraph(or) Scroll to previous or next sentence


41/45


Search and Repeat-Search Commands

Command Action/pat Search forward for patternpat?pat Search backward for patternpatn Repeat search in previous search

direction (no repeat factor)N same as n but in opposite directionfxortx Move cursor forward to or before the

first occurrence of characterxin current lineFxorTx Same as above but move backward

(repeat factor works with all f/F and t/T); or, Repeat last character search in same

or opposite direction (by f/F/t/T) onlyin current line (repeat factor works)


42/45


Deletion and Modification Commands

Command Action

dw Delete Current Worddd Delete Current Lined$orD Delete from cursor to end of linecw Change Current Wordcc Change Current Linec$orC Change from cursor to end of line

x Delete character under cursorX Delete character before cursor

J Join current, next lines (also nJ).(dot) Repeat last editing instructionu Undo single last changeU Restore all changes to line since

cursor moved to it


43/45


Commands to Move or Copy Lines

[][n]ddCut (delete) n lines starting from current line

[][n]yyCopy n lines starting from current line

[]pPut yanked text after current cursor position

[]PPut yanked text before current cursor position

Note: Named-buffer is useful for copying an area from one file into another. Open

some files simultaneously by vi. Mark an area in one file by ddoryy; move toanother file (by :next) to paste (byp orP) the named area; then say :rewind toreturn to the parent file.


44/45


Customizing vi

vican be customized as per users requirements using ex-mode commands.exCommands Action:set all Display all setoptions; those pre-fixed with no are not operative:set autoindent(aifor short)

Extremely useful to programmers forindentation of lines

:set number(nu for short)Displays all lines with numbers which are not preserved on

saving file:set nonumber(nonu for short) - Reverses number setting:set tabstop=6(ts for short)

Changes default tab setting (8 spaces):set ignorecase (icfor short)

ignores case while pattern matching


45/45


Customizing vi ...

:set showmatch (sm for short)

Helps locate matching brackets when ) or } is entered in input mode; beeps when no match found to alertfor correction

:set autowrite (awfor short)

Writes current file automatically whenever switching files with :n and escaping to shell with :sh

:set showmode - Displays a message when vigoes to input mode

:next(n for short) Moves to next file opened in vi

:rewind Comes back to parent file

All sets can be stored in .exrcfile used by vifor its startup instructions.

Equivalently an environment variable, EXINIT can be assigned to store the settings:

$EXINIT=set nu tabstop=6 ignorecase

It is better to make this assignment in the .profile so that it is available for all sessions.

Documents

Unix Essentials V2.0