Introduction to Linux - McGill University - McGill... · Introduction to Linux - February 5, 2015 6/64 Bash Bash is a shell: Gives access to file systems Runs commands as a specific

Introduction to LinuxFebruary 5, 2015

Pier-Luc St-Onge, Calcul Québec – McGill HPC

[email protected]

Introduction to Linux - February 5, 2015 2/64

Schedule and Outline

● 9:00-10:00 The Terminal● 10:00-11:00 File Systems

● 10:20-10:30 Break

● 11:00-12:00 Running Commands● 12:00-13:00 Lunch

● 13:00-14:00 Processes● 14:00-14:20 How to Compile

● 14:20-14:30 Break

● 14:30-14:45 Remote Connections● 14:45-15:00 Graphical User Interfaces in Linux


Short History of Linux

● Reference:● http://www.cs.cmu.edu/~awb/linux.history.html

● Project : started in 1991 by Linus Torvalds● Context :

● Unix vs DOS● New Intel 386 architecture

● Linux kernel:● Open source● Under the GNU GPL

● Linux used in high-performance computing clusters

http://www.cs.cmu.edu/~awb/linux.history.html


Outline

● The Terminal● File Systems● Running Commands● Processes● How to Compile● Remote Connections● Graphical User Interfaces in Linux


The Terminal

● Bash● How to Exit● Previous Commands in Bash● Auto-completion in Bash● User Manuals● Help Topics● Environment Variables (1-2)● Exercises


Bash

● Bash is a shell:● Gives access to file systems● Runs commands as a specific user● Has per-session environment variables

● Started from a terminal, a parent shell or SSH.● In Ubuntu, search for application “Terminal”.

● The Bourne-Again shell (Bash) is the default shell on Guillimin and on many other linux systems.

● Other shells : Bourne shell (sh), C shell (csh), Tenex C shell (tcsh), Korn shell (ksh), etc.


How to Exit

● Type “exit” and press Enter● The “exit” command will go to the history

● Or use Ctrl+D● Bash receives commands from the STDIN stream● Ctrl+D closes the STDIN stream● Bash will exit itself as soon as STDIN is closed● This exit will not be logged in the history

● If Bash was started by:● A terminal : the terminal will close as well● A parent shell : the user will return to the parent shell● SSH : the connection will be closed


Previous Commands in Bash

● To get all the logged previous commands, type:

history● To get a previously executed command, use the Up

arrow key until you find it● You may search for a command in the history:

● Use Ctrl+R● Start typing parts of the command you are looking for● Use the Left or Right arrow keys to select and edit the

command (if needed)● Use Ctrl+C to cancel


Auto-completion in Bash

● The first word in a command is usually an executable:

<executable>● The other words (or arguments) may be options or short or

long paths to files or folders:

<executable> a b option option2

<executable> ab <filename>● Knowing that, Bash can auto-complete what you are typing

or give you some suggestions● Use the Tab key to trigger the auto-completion function


User Manuals

● Some programs have one or many manual pages

man <programname>● Most of the man pages are also on the Web● In the man page, you can search for a keyword:

● Type “/”● Type your keyword and press Enter● All instances of the keyword will be highlighted. Man will go to

the first instance. Press the “n” key to go to the next one.

● To quit the man page, press the “q” key● Man pages are displayed (piped) through a program called 'less'


Help Topics

● Some programs provide additional help topics on the command line:

executable help

executable h

executable help


Environment Variables

● To print a list of the current environment variables:

env● Notable variables:

● HOSTNAME: computer's name● PWD: current directory● SHELL: default shell (Bash)● LANG: session language, text encoding


Environment Variables (2)

● Other notable variables:● LOGNAME, USER: session user name● GROUP_ID: groups of current user● HOME: home directory● PATH: paths to executable files● LIBRARY_PATH: paths to libraries (mainly for compilation)

● LD_LIBRARY_PATH: paths to libraries (mainly for execution)


Exercises

● Use the auto-completion to reach the following executable and print the help topics:

/usr/bin/svn

● /u<Tab>b<Tab>...

● Connect to Guillimin and get the host name:

ssh guillimin.hpc.mcgill.ca l class##

env● What is Python? Read its manual page:

man python


Outline



File Systems

● Where am I?● Structure of Directories● Mounted File Systems● Files and File Attributes (1-4)● Changing File Attributes● Directories and Directory Attributes (1-2)● Changing Directory Attributes● Symbolic and Hard Links● Exercises (1-2)


Where am I?

● Print working directory:

pwd● List the files and folders in the current directory (# for comments):

ls # Same as 'ls *' where '*' could be anything

ls a● Hidden files and folders start with a dot “.”● The “.” folder is the current directory● The “..” folder is the parent directory● To change the directory (use “;” between commands on one line):

cd <directory_name_or_full_path>

cd .. ; cd # parent, previous


Structure of Directories

● / : root directory of everything in Linux

● /home : contains all home directories

● /tmp : temporary files and folders

● /bin, /usr/bin, /usr/local/bin, etc. :

● see the PATH variable (directories for executable files)

● /dev : devices and virtual devices

● /dev/null, /dev/random, /dev/urandom, /dev/zero

● /proc : processes information

● /proc/meminfo, /proc/cpuinfo


Mounted File Systems

● Directories are mount points

/media/USBDRIVE● Print the mounted file systems:

mount● Usage of the mounted file systems:

df h● Your personal quota (if any):

quota # or myquota on Guillimin● Disk usage of a file or a directory:

du sh <file_or_directory>


Files and File Attributes

● Create a file or change its time stamps:

touch <filename>● Show the attributes of a file:

stat <filename>

ls l <filename>● Rename or copy a file

mv <filename> <newname> # No confirmation

cp a <filename> <newname> # Preserves all● Remove a file permanently (no trash/recycle bin):

rm <filename> # No confirmation


Files and File Attributes (2)

● Current language (LANG):

locale● Edit a new or an existing text file (nano, emacs, vi):

nano w <filename>● Keyboard shortcuts in nano:

● Ctrl+X : Exit● Ctrl+O : Save● Ctrl+K : Cut the current line. Repeat for consecutive lines● Ctrl+U : Paste the multiple lines



● Get the type of file (text encoding, image format, ...):

file <filename>● View a text file:

less <filename>– Press “q” to quit

more <filename>

cat <filename>● Word count and line count:

wc <filename> # All statistics

wc w <filename> # Word count only

wc l <filename> # Line count only



● See the default file attributes:

ls l # rwrwr username group

umask # 0002

id # uid=... gid=... groups=...● Permissions:

● : regular file

● rw : user has read and write access, no exec.

● rw : group has read and write access, no exec.

● r : other users have read access only


Changing File Attributes

● Changing the permissions:

chmod u+x,or <filename>

chmod 760 <filename> # rwxrw760 is an octal number : 421 42

chmod 640 <filename> # rwr● Changing the ownership:

chown <username> <filename>

chown <username>:<group> <filename>

chgrp <group> <filename>


Directories and Directory Attributes

● Go to your home directory and list its attributes

cd ~

cd

ls ld . # drwx username group

ls la

● To access inside a directory, the execute “x” permission must be enable.

● Hidden directories start with a “.”


Directories and Dir. Attributes (2)

● Create a directory:

mkdir <dirname> # mkdir p very/long/new/path● Show the attributes of the directory:

stat <dirname> ; ls ld <dirname>● Rename or copy a directory

mv <dirname> <newname>

cp r <dirname> <newname> # cp pr OR cp a● Move a file or a directory into another directory:

mv <file_or_dir> <dirname>/[<newname>]● Remove a directory

rmdir <emptydir>

rm r <dirname>


Changing Directory Attributes

● Remove other accesses recursively:

chmod R orwx <dirname>● Add the group sticky bit recursively:

chmod R g+w <dirname>find <dirname> type d exec chmod g+s {} \;

● Changing the ownership recursively:

chown R <username> <dirname>

chown R <username>:<group> <dirname>

chgrp R <group> <dirname>


Symbolic and hard links

● Create a symbolic link (files and directories):ln s <original_file_or_dir> <linkname>

ls l● Create a hard link (files only):

ln <original_file> <linkname>

ls li # inode_ID ... num_links● Remove a link:

rm <linkname>


Exercises

● Create a text file. Write something in it.● Create a directory. Move the file into the directory.● Block all other users access (rwx) to this directory.● Create a second directory. Move the first directory in this

second directory.● Create a symbolic link to the first directory.● Create a hard link to the text file. Modify the file from the

new hard link.● Print the content of “both” files.● Remove everything created in these exercises.


Exercises (2)

● Import a compressed archive from Internet:

mkdir www; cd wwwwget http://www.hpc.mcgill.ca/downloads/ intro_linux/introlinux201409.tar.gz

● Extract the content:

tar xvf introlinux201409.tar.gz● Create another tar archive:

tar cvf archive.tar introlinux● Compress the new archive:

gzip archive.tar


Outline



Running commands

● Which, Where?● What Time?● Hello World!● Filtering (1-2)● Bash Scripts● The .bashrc file● Exercises


Which, Where?

● Which command is this?

which <executable>● Where is my file? Where is this program?

locate <file>● Find a file in a directory:

find <dirname> name <file_or_dir>


What Time?

● The current date and time:

date● Time to go to sleep?

time p sleep 60● Want to stop a command running?

Ctrl+C


Hello World!

● Say hello and world (two arguments):

echo "Hello" $(echo n "World")● Pipe between output and input of two processes:

echo "Hello World" | less● Output redirection to a file (if the file exists, its previous content

will be lost):

echo "Hello World" > file.txt● Output redirection appended to the end of a file:

date >> file.txt● Input from a file:

less < file.txt


Filtering

● Basic regular expressions / searching in files:

cat file.txt | grep i hello # Case insensitive

grep n i hello file.txt # Show line number

grep some_string file1 file2 file3● Extended regular expressions:

grep E "^k?[hH]el+o.{2}o.*$" file.txt● ^: beginning of line, $: end of line

● ?: 0 or 1 occurrence, *: 0 or many occurrences

● +: 1 or many occurrences, {n}: exactly n occurrences

● .: any character

● [azAZ09]: list of characters (matches only one)


Filtering (2)

● A .csv file (introlinux/examples/):

Animal,Color,Age

cat,black,10

dog,black,5

dog,brown,8● Show all dogs:

awk F, '/dog/ {print $3,$2}' file.csv

grep dog file.csv | awk F, '{print $3,$2}'


Bash Scripts

● Consider script.sh:

#!/bin/bash

echo ""

date

ls a

● Make script.sh executable:

chmod +x script.sh

● Run script.sh:

./script.sh > myfiles.txt


The .bashrc file

● Located in your home directory:

cat ~/.bashrc● This is a Bash script that is executed at the

beginning of each Bash session● Used to set environment variables and aliases● Example of an alias:

alias rm='rm i' # Adds confirmation


Exercises

● Use the .bashrc file, echo, hostname, date and output redirection to log all new Bash sessions in ~/.mysessions with the following format:

New session on <hostname>: <date>● Open and close Bash a few times.● Print only the date of the last 4 sessions (use tail n, the separator would be “: “ for awk).


Outline



Processes

● Running Processes● How to Kill Processes● Processes in Background● Introduction to Threads● Exercises


Running Processes

● Listing the running processes in your session:

ps● Listing all the running processes on the server:

ps ef● The init process (PID = 1)● The parent PID (PPID)● Monitoring the running processes (“q” to quit):

top


How to Kill Processes

● Sending signals to a process:

kill <number> PID

kill s SIGNAL_NAME PID● To list the different signals:

kill l

● The default termination signal is SIGTERM (15):

kill PID● The termination signal from a terminal (like Ctrl+C):

kill s SIGINT PID # or kill 2 PID● To force the process to stop now:

kill s SIGKILL PID # or kill 9 PID


Processes in Background

● Processes unattached to a terminal : they are running in background

● To start a process directly in background (nohup is optional):

nohup command arg1 ... > outputfile &

jobs● To resume in foreground:

fg● To suspend a foreground process (SIGSTOP):

Ctrl+Z

● To resume in background (SIGCONT):

bg● To wait for all background processes to terminate:

wait


Introduction to Threads

● One or many threads in a process● CPU usage may be greater than 100%

● Shared memory● False sharing

● Synchronization concepts:● Barrier: synchronisation● Mutex: one and only one thread● Semaphores: wait and release


Exercises

● Start a secondary bash session in your primary session● Start the following command

dd if=/dev/urandom of=/dev/null● Put it in the background

● Suspend the command● Resume in the background

● While running, look at the CPU usage● Quit the secondary bash session

● Get the new PPID of dd● Kill the dd process with SIGTERM


Outline



How to Compile

● C with OpenMP Code● C with MPI Code● C++ Code● Makefiles● Perl Scripts


C with OpenMP Code

● Code example

main.c● Compilation

gcc o exec_ser main.c

gcc o exec_omp main.c fopenmp● Execution

time p ./exec_ser

time p ./exec_omp


C with MPI Code

● Code example

main.mpi.c● Compilation

module add ifort_icc # On Guillimin

module add openmpi

mpicc o exec_mpi main.mpi.c● Execution

time p mpiexec n 2 ./exec_mpi


C++ Code

● Code example

main.c● Compilation

g++ o exec_cpp main.c● Execution

time p ./exec_cpp


Makefiles

● A Makefile file

● Variables (set: VAR=..., use: $(VAR))

target: dependencies

commands● Regular expressions

%.o: %.cpp● Special variables ($@, $<, $+)

● The make command

make f <makefile>

make # if Makefile or makefile

make <target>


Perl Script

● Code example

main.pl (rwxrx)

● Code validation

perl c main.pl● Execution

perl main.pl

./main.pl


Outline



Remote Connections

● Using SSH● Using SCP● Using SSH Keys


Using SSH

● From a terminal (Mac OS X, Linux):

ssh <username>@<server.name.ca>

ssh <server.name.ca> l <username>● (Accept the certificate) Enter the password

● Connect to a different port number:

ssh <username>@<server.name.ca> p <num>● From PuTTy (Windows, Linux):

● Host Name: <server.name.ca>● Click on “Open” (Accept the certificate)

● Login as: <username>, Enter the password


Using SCP

● Secure copy with SSH protocol● From a terminal (Mac OS X, Linux)

scp filename <user>@<server>:/full/path/

scp filename <user>@<server>:directory/

scp filename <user>@<server>:newname

scp pr dirname <user>@<server>:

scp * <user>@<server>:directory/● From WinSCP (Windows)

● Server name, credentials, drag & drop● Beware of the text copy mode (\n vs \r\n)


Using SSH Keys

● Having access with or without a passphrase:sshkeygen

Generating public/private rsa key pair.

Enter file in which to save the key (~/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

● Copy the public key to a remote site (id_rsa.pub):scp .ssh/id_rsa.pub <user>@<server>:

ssh <user>@<server> \

"cat id_rsa.pub >> .ssh/authorized_keys"


Outline



Graphical User Interfaces in Linux

● Desktop Environments and Shell Interfaces● The X Server


Desktop Environments andShell Interfaces

● Gnome● http://www.gnome.org/gnome-3/● https://unity.ubuntu.com/about/

● KDE● http://userbase.kde.org/An_introduction_to_KDE

● Linux Mint Cinnamon● http://cinnamon.linuxmint.com/

● LXDE● http://lxde.org/image_galleries/screenshots

● MATE (Gnome 2 fork)● http://mate-desktop.org/about/

● Xfce● http://www.xfce.org/about/screenshots

http://www.gnome.org/gnome-3/

https://unity.ubuntu.com/about/

http://userbase.kde.org/An_introduction_to_KDE

http://cinnamon.linuxmint.com/

http://lxde.org/image_galleries/screenshots

http://mate-desktop.org/about/

http://www.xfce.org/about/screenshots


The X Server

● Decoupling the application and the rendering engine● X Client: OS, application, remote server● X Server: keyboard, mouse, screen

● Using SSH

ssh X <user>@<server.name.ca>● X Server for Windows : Xming

● http://sourceforge.net/projects/xming/

● Open remote application

command args... &

http://sourceforge.net/projects/xming/


Conclusion

● Try Linux in a virtual machine (Virtual Box)● Many distributions

● http://en.wikipedia.org/wiki/List_of_Linux_distributions

● Arch Linux, CentOS, Debian, Fedora, Gentoo, Linux Mint, Ubuntu, etc.

● Web pages are friendlier than Man pages

● Questions?● [email protected]

http://en.wikipedia.org/wiki/List_of_Linux_distributions

Documents

Introduction to Linux - McGill University - McGill... · Introduction to Linux - February 5, 2015 6/64 Bash Bash is a shell: Gives access to file systems Runs commands as a specific