Introduction to Linux on Iceberg• What are UNIX and Linux?• Getting Connected• Basic UNIX/Linux Commands• Help!• Working with Directories• Running Programs• Editors• The Shell• Exercise

History of UNIX/Linux?.

• Unix operating system was developed around 1969 in the Bell Labs

• Originally written using C• Around 1990 Linus Torvalds of Helsinki University

started off a freely available academic version of Unix

• Linux is the Antidote to a microsoft dominated future

What is UNIX/Linux ?

• Multi-Tasking O/S• Multi-User O/S• Available on a range of Computers

Which UNIX?

• SunOS Sun Microsystems• IRIX Silicon Graphics• HP-UX Hewlett Packard• Linux For IBM PC compatibles

There are a number of certification bodies with published standards and test suites to ensure quality of products. Such as;

• Posix: Portable Operating System Interface ( IEEE standard)• COE: The Common Operating Environment (COE) Platform

certification.

UNIX Internals (Simplified)

• Kernel– System Internals

• Shell– Command Interpreter– Programming language

• File System• Process Management

UNIX Shells

• sh Bourne Shell (Original Shell) (Steven Bourne of AT&T)• bash Bourne Again Shell (GNU Improved Bourne Shell)• csh C-Shell (C-like Syntax)(Bill Joy of Univ. of California)• ksh Korn-Shell (Bourne+some C-shell)(David Korn of AT&T)• tcsh Turbo C-Shell (More User Friendly C-Shell).

You can switch from one shell to another by just typing the name of the shell. exit return you back to previous shell.

Linux and GNU• Linux is an implementation of Unix• Linux/Unix operating system is written in ‘C’• Linux is not part of the GNU project but uses the same licensing

agreements• Many of the linux utilities and tools are taken from the GNU

project. • There are many flavours of linux distributions. The mix of the

kernel (linux) with the utilities (GNU and other) and the installation procedure determine the flavour. Some of these are;– Fedora– SuSE– Redhat– Debian – Mandrake– Knoppix

Terminal Emulation SoftwareUsed in Sheffield Univ.

• ssh : Secure Shell Client (no XWindows)• Exceed : ( Xwindows support and also secure shell)

Graphical UNIX via Exceed

• X Concepts• Using the Exceed Software• Starting an xterm session• Starting a ssh session

Basic X Concepts• X Server runs on local machine

– PC Exceed, Cygwin, Xming

– UNIX WorkstationIncluded in OS

– Apple Mac Exodus

• X Client runs on remote machine– Graphical Application

• xterm

• xcalc

• Modelling and visualisation packages etc.

ssh on Managed/XP workstationsSecure-shell client can be accessed via;

Start Unix_ConnectivityExceed3Dssh.

When the program initialises you will be asked to enter hostname (default is iceberg) , username and password.

ssh on Managed/XP workstationsWhen ssh starts you will be presented with the login panel as shown below; Default host name is iceberg.shef.ac.uk and can be changed if you want to login somewhere else.

You may next be asked if you wish to save the public host-key into a local-database. This is not a crucial question and you may answer YES or NO.

Also if asked to provide host response just press ENTER.

Exceed on the stand-alone PCsStarting an Exceed xterm session

Start Exceed ->Xstart and fill in the panel as shown belowFill in host,username,p/w,command

and click on the run icon to start xterm

Exceed on the stand-alone PCsAn Exceed xterm session

Multiple ssh or xterm shells

There are no limits to the number of ssh or xterm windows one can start simultaneously by methods described in the previous slides. You may also start extra xterm windows from the host by simply typing xterm &

On titania and iceberg we also have a local command named Xterm that starts up an xterm window with nicer to use parameters. On iceberg we strongly recommend that you use Qsh rather than xterm or Xterm command so as to make use of a free worker node. Qsh will act like Xterm but will make use of a worker node.

Typing exit will terminate an xterm or ssh session neatly.

This will also close the xterm window but not the ssh window. ssh windows can be closed via the file exit menu.

Starting gnome

Gnome was the default graphical desktop interface that was installed on iceberg before delivery. We currently do not allow the use of gnome on the head-node ‘iceberg’ for efficiency reasons. You could however use the gnome panel while working on one of the worker nodes ( via qsh ). This provides the user with access to all the gnome commands available via the red-hat menus. To do this type

qsh to start an interactive session

and in that session type

/usr/bin/gnome-panel &

This is the nearest, one would get to using the common-desktop environment as was provided on titania but it is not worth it.

Some basic rules

• Unix is case sensitive.• Commands are in lower case.• Backspace and/or Del Keys correct typing errors.

If the terminal parameters are not correctly set; try Ctrl+H

• Ctrl+C Aborts a program or command.• You can use the arrow keys to recall previous

commands, optionally edit and execute them.

Key combinations in bashCtrl+A Move cursor to beginning of line

Ctrl+C <Break> Stop running program

Ctrl+D <Logout>

Ctrl+E Move cursor to end of line

Ctrl+H Delete character to the left

Ctrl+Z Suspend the program

Arrow left/right Move pointer left/right to insert chars etc.

Arrow up/down Recall previous commands so as to edit them andsubmit them by pressing ENTER

Tab

Tab tab

Command or filename completion. Auto-complete the command line (to save typing)

If tab does not work tab again to show possibilities

passwords

• In normal linux environment the passwd command can be used to change the user passwords. However, because we manage passwords centrally this command will not work on iceberg.

• If you wish to change your iceberg password you will have to do this via a web interface at the following URL:

http://www.shef.ac.uk/cics/password

Filenames• Filenames can comprise of:

a-z, A-Z alphabetic characters

0-9 digits

.-_+ special characters

mon+tue_01.06-03-96

• Wildcards when referencing files* any character or sequence of characters

? any single character

Format of Unix commands

• command [option ...] [filename ...]

eg: ls

ls -l tutorial

more tutorial

List Directory

• ls list directory• ls -l list directory in long format• ls -a list all (inc. hidden) files

-rw------ l course01 57 Oct 18 11:05 hello.c

Number ofbytes in file

AccessPermissions

Date and time last modified

Displaying contents of a text_file• more filename

This command will start listing the contents of filename on screen and pause after a screenfull of data. While pausing, use the following characters to control the output.

Spacebar next screenful

n Spacebar : next n lines

Enter next line

b back one screen

n b : back n screen’s full

q quit

? or h list commands

where n is a whole number

Displaying contents of a text_file… continued

• cat [options] filename [filename … ]

This command will output the contents of filename[s] to standard-output ( normally screen) without pausing.

Following options are useful;

-v display non-printing characters

-n display with lines numbered on the left

• tail [-n] filename

This command lists the last 10 lines of a text file.

If a number is specified (.eg. -20 ) lists the last n (i.e 20) lines

Copying files

• Copy files (optionally directories)

cp fromfile tofile

Some of the useful options are:

-R or –r : Recursive copy ‘fromfile’ is a directory so the entire directory and its contents are copied.

e.g. cp –r mydir newdir

-p : preserve. Preserves all attributes of the file ,such as access rights and creation date.

• Copy and concatenate files by using cat

Cat command concatenates contents of list of files and directs the output to standard output (normally screen). When used with redirection ‘>’ it can be used to join files together.

e.g. cat file1 file2 file3 > new_big_file

Renaming and deleting files• mv :This command will move a file or directory to a new location. It can thus be

used to rename files/directories as well as change their locations in the global directory structure.

Syntax:mv source destination

Example:mv myfile mynewfile mv myfile subdirectory/myfilemv mysubdir mynewsubdir

• rm : This command will delete a file (optionally a directory if used with –r option).Syntax: rm object_to_deleteExample:

rm myfile rm –r mydirectory

Finding files and information about them• find :Finds a file in the directory hierarchy

find root_dir –name filename –print

Example :

find . –name “myprog.*” -print

Note that wild-character containing string must be enclosed in quotas• which : Shows in which directory a command is located.

Syntax: which command_name

• file: Can be used to see what type of data a particular file contains. For example, script , program, library, executable binary etc…

Syntax: find command_name

Searching in files

• grep string file

This command finds and prints out the lines in the file(s) containing the specified string

string = word or phrase

file = file or list of files (wild_card can be used)

Note: We strongly advise that the string is quoted.

Examples: grep ‘Green Man’ england.dat

grep ‘Zodiac’ t*.dat

grep ‘Zone[a-z]’ security.fil

Directory Structure

Home directory of user cs4un1 : /home1/cs/cs4un1

usr

/ (root)

home1

cs me ph

cs4un2cs4un1

When you log in you are positioned in your home directory.The environment variable $HOME is also set to contain this directory name.

Working with Directories• pwd print working directory• cd change directory

cd move to home directory

cd .. move up one level

cd mydir move into a subdirectory

cd /var/adm move to an absolute directory

• mkdir directory_name create a new directory

• rmdir directory_name delete an empty directory

Managing your filestore• Each user has an allocation of filestore (Disk Space)• Monitor filestore usage with the ‘quota’ command

EG: quota Disk quotas for username (uid 20002)Filesystem usage quota limit grace files quota limit timeleft

/home1 1200 2000 4000 153 0 0

/data 411 99600 100000 13 0 0

/scratch 0 99600 100000 0 0 0

Note: quota –s will display quota in sensible units.

• The quota value can be exceeded for 7 days

quota < usage < limit for 7 days

• The limit value can not be exceeded at all.

usage < limit always

• If usage > quota for 7 days account is expired and files can not be created

Manual Pages and Info systemIt is possible that there are more than one manual pages with the

same name. This is because the manual pages contain

information about the system library routines as well as the usual

unix commands. Manual pages for commands are contained in

section 1 whereas the system library call related items are in

section 3. To access manual pages within a specific section

simply specify the section_number as the first parameter.

Example: man 3 random

• Note that man – k topic will list the relevant manual pages and

their section numbers.

• The –S parameter can also be specified.

Example: man files –S 1:2:4

Manual Pages and Info system

Man: Manual pages : Give text-based help on usage.

Manual pages are grouped into sections (1,2,3,4 ..).

There is usually one manual page per command which is

located in one of the directories defined by the MANPATH

environment variable.

To access the Manual page for a command just type;

• man command

To get a list of manual pages that contain a ‘word’ type;

• man – k topic

Info : Similar to man but can scroll with cursor keys and link to other information.

Using whatis and apropos commands• whatis displays the short description of a system command ‘if

you know the command name’ .

• apropos searches the same ‘whatis-database` and reports back the names of all the commands that a key-word occurs.

Example usage: Suppose we wish to create a temporary file with a unique name but do not know if there is a command for doing so. You could try :

apropos temporary or apropos file or apropos create

This will give ‘whatis’ output for a list of commands which contain the keyword. You can now use the man command for potentially useful commands to get the full details of how to use them.

man mktemp

Running programs on iceberg• Iceberg is the gateway to the cluster of worker nodes and the

only one where direct logging in is allowed. • Iceberg’s main purpose is to allow access to the worker

nodes but NOT to run cpu intensive programs. • All cpu intensive computations must be performed on the

worker nodes. This is achieved by the qsh command for the interactive jobs and qsub command for the batch jobs.

• Once you log into iceberg, taking advantage of the power of a worker node for interactive work is done simply by typing qsh and working in the new shell window that is opened. This what appears to be a trivial task has would in fact have queried all the worker nodes for you and started a session on the least loaded worker in the cluster.

• The next set of slides assume that you are already working on one of the worker nodes (qsh session).

Running programs• Two modes of operation foreground and background

– Foreground Interact with program via keyboard/screen

– Background No connection with keyboard/screen

Submit to backbround by Appending ‘&’

EG: myprog >& myfile &

The symbols ‘>&’ redirect output and any errors to the file myfile

Although the above method of running jobs on the background is feasible, the prio we recommend that you submit your background into the batch queue via the qsub command.

Redirection

Most unix commands are not aware of the source of their input or the destination of their output. They simply read/write from/to stdin/stdout. The shell takes care of these issues.

• Standard Input (default=>keyboard) • Standard Output (default=>screen)• Redirection symbols <,>,>> can be used to specify

files as the source/destination of the read/write operations to override the above defaults.

Redirection continued …Most unix commands are not aware of the source of their input or

the destination of their output. They simply read/write from/to stdin/stdout. The shell takes care of these issues.

• To redirect the output to a file use the ‘>‘ symbol.Example: ls -l > dirlist

• The ‘>‘ symbol should be used with care as it may over-write an existing file. ‘>>’ symbol can be used instead if the output should be appended to the end of an existing file rather than over-writing it. Example: ls -l >> logfile

• If nothing is directed to a file then a zero size file is created, or if the file already existed then the contents of the file is removed Example: > afile

• The file /dev/null is a special symbol to indicate a ‘black-hole’Example : ls –l > /dev/null

Redirection continued …

If any program expects any of it’s input from the standard-input-channel , i.e. the keyboard, it can also read the same information from a file by redirection.

• To read input from a file use the ‘<‘ symbol.Example: write cs1xyz < message.fil

Here any text input {write} program expects from the keyboard will be simply read from a file named message.fil.

Each end-of-line will be treated as an <ENTER> on the keyboard.

Piping

• Feeding the output of one command into the input of another command

• The symbol ‘|’ is called a pipecommand | command

• eg: ls -al | more

ls -la | grep Nov

Examples of re-direction and piping

• ls –l | grep ‘Jun’• ls –l | grep `Jun` > june_files• ls –l | grep `Jun` | cut –c 57-80 > june_files• cut –c 1-10 < test_files• aspell –l < message.txt > report.txt• grep fluent < news.dat • Grep fluent < news.dat | cut –c 1-72 • (grep fluent < news.dat ) | cut –c 1-72 • (grep fluent < news.dat ) | cut –c 1-72 > fluent.news

Foreground Program Control

• Kill a program Ctrl C• Stop a program Ctrl Z

Note a stopped program still exists in the system

and hence can be re-started.

Program control within current shell• jobs Lists jobs (programs)• bg %job_id Place a job in the background• fg %job_id Return a job to the foreground• stop %job_id Stop a job• kill %job_id Kill a jobProcess_id can be used in place of %job_id for more

definitive way of identification.jobs[1] + Running time.sh > outstop %1[1] + Stopped (signal)time.sh > outbg %1[1] + time.sh > out &kill %1Terminated

Program control using ‘ps’ and ‘kill’• ps Report process status

ps -f -u username UID PID PPID C STIME TTY TIME CMD

username 24816 24585 0 16:23:04 pts/50 0:00 sleep 2

username 20169 19956 0 16:05:45 pts/50 0:01 -csh

username 24585 20169 0 16:35:07 pts/50 0:00 /bin/sh time.sh

• kill Terminate process

– Find its process ID (PID) using the ps command

– Kill the process using the kill commandkill 24585

– Sometimes kill on its own does not work so trykill -KILL 24585 or kill -9 pid

Program Control using top

• top is a separate utility and not part of the OStop -Uusername

– k kill program (process) k PID, k -KILL PID

– h help

– q quit top

EditorsThere are a number of editors for Linux platforms most are not as easy to use

as the Windows based editors. Our recommendation is; Use nedit if you are using XWindows ‘e.g. Exceed’, cygwin.Use vi or vim if you have a text terminal ‘telnet’.

• pico Easy ‘graphical’ , rather basic.• nedit Easy ‘graphical’ , good.• vi, vim Require some knowledge, vim is like vi but

more and includes help• emacs Have faithful following, good once learned.

Vi Text Editor

• Very powerful• Available on all versions of UNIX• No in-built help• Man page gives detailed help

• Vi has three modes– Command– Text entry– Last line

Vi Text Editor

• Very powerful• Available on all versions of UNIX• No in-built help• Man page gives detailed help

• Vi has three modes– Command– Text entry– Last line

T-Shell - Filename Completion

• Complete a partially typed filename– Operation

• Type enough characters to uniquely identify the name• Press the ‘Tab’ key (for C – shell use ‘Esc’ key)• If the response is a ‘bleep’ press Ctrl-d to list possible matches

– Setup• Add the following to your .cshrc file• set filec

T-Shell -Repeating Previous Commands

• Operation– history List previous commands

– !! re-run last command

– !n re-run the nth command

– !str last command starting with str

eg: !vi

• Setup– Add the following to your .cshrc file

– set history=40

T-Shell - Command Aliasing

• Definitions– alias ll ' ls -lF '– alias h ' history 24 '– alias dir ' ls '

• Setup– Add alias definitions to your .cshrc file

UNIX Utilities

• tar• compress• sed• awk

Exercise Tutorial• Turn on PC• Login to your PC Managed XP account

login digley/mep98xxxpassword:

• Login to iceberg

– Start XSession or telnet titania

Login using the provided username and password