whoami: Grégory Mounié - ENSIMAG · PDF fileUnix® and Linux® System Administration Handbook,Fourth ... • ed (brotherof sed) ... • ZFS(Sun/Oracle)allowsplentyofnicethings

Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability

SAaNS: Introduction to Unix

Grégory Mounié

2014-2015

1 / 75


Outline

Presentation of the lecture

History of unix systems

Base concepts

Linux installation

CLI

Configuration

Distribution and Scalability

2 / 75


Presentation of the lecture

History of unix systems

Base concepts

Linux installation

CLI

Configuration

Distribution and Scalability

3 / 75


whoami: Grégory Mounié

• [email protected]• Associate professor at Ensimag• Office: Building of Inria Montbonnot (11 km of the campus)• Current teaching: mainly Operating Systems; Networks anddistributed systems; Python;

• Former teaching: Networks, Java, Ada, Operational research,OpenGL

• Research: Scheduling for High Performance Computing(HPC), or the art to put cubes in boxes

3 / 75


Content of the course

• OS basics and OS administration in 3 hours• Network basics and Network administration in 3 hours• Security in 3 hours• Perl in (3 + 3 hours)

4 / 75


Bibliography

Main referenceUnix® and Linux® System Administration Handbook, FourthEdition, Video Enhanced Edition.Evi Nemeth; Garth Snyder; Trent R. Hein; Ben Whaley

Basics

OSModern Operating Systems, Tanenbaum

NetworkComputer Networks, Tanenbaum

Network and system programmingUnix programming, Stevens (a bit old but still the reference formany low levels and raw stuff)

5 / 75


Read the man, Luke !

• Man pages• GNU Texinfo (Html with automatic index)• Package-specific doc

• typically in /usr/share/doc/• RFC and Internet documents

• beware of version mismatch

6 / 75


Challenges

• People with sufficent background may skip the basics part ofthe lecture and do the challenge instead.

Challenge for the OS lectureYou are employee of X and you will go to Paques Island to sellsome Y. The major provider of Y is an US society. You will crossthe US border with commercial documents in your laptop.Choose a way to protect your documents, implement it and presentit to others lecturers (10 min at the end of the lecture)

7 / 75


Duties of the system administrator

• Account provisioning• Adding and removing hardware• Performing backups• Installing and upgrading software• Monitoring the system• Troubleshooting• Maintaining local documentation• Vigilantly monitoring security• Fire fighting

Difficult pointEach point is simple. Complexity come from the cross-referencesbetween points.

8 / 75


Origin of UNIX

in the beginning was multics (65-68)

• failure ⇒ Bell decides to stop making operating systems• Kernigham and Ritchie et al. develop a small operating systemof their own

• by derision they call it UNICS (69)• and create a programming language (C) to recode it

• property of Bell Labs• restrained diffusion when successful

9 / 75


History of UNIX

• sources was given (Unix V6) to users (device drivers)

Berkeley University (76)

• first version of BSD• large diffusion in universities

Parallel evolution, in 1983

• 4.3BSD (SunOS)• Unix System V

10 / 75


History of UNIX

• no free (as in speech) version• development of internet:

• ⇒ ideal conditions for the development of free unices:• strong demand• skilled peoples• means enabling common development

11 / 75


MINIX and Linux

Philosophy

• Minix (85) written by Prof. A.Tannenbaum for his courses

• internet ⇒ large diffusion andimprovements

• but restriction on user rights• Linux written by a student LinusTorvalds

• support of recent architectures• GPL licence

12 / 75


Unixes Today

1. Linux and its embedded variants:• Android, Firefox OS, Sailfish OS (ex-Mer, ex-Meego), Ubuntu

Touch, Tizen (ex-Meego), WebOS, Plasma Active• Desktop distributions: Debian, Fedora, etc.

2. BSD family:• Darwin (MacOS X, IOS), NetBSD, FreeBSD, OpenBSD

3. A given number of (vanishing) proprietary unixes :• Solaris, AIX, HP/UX

4. other exotic unixes: Minix, Hurd, Haiku (BeOS clone) . . .

13 / 75


Free software

• free as in speech not as in beer !• based on open standards• different licences : BSD, GNU Public Licence, . . .• defending users rights :

1. The freedom to run the program, for any purpose2. The freedom to study how the program works, and adapt it to

your needs.• Access to the source code is a precondition for this.

3. The freedom to redistribute copies4. The freedom to redistribute modified copies.

14 / 75


Free software central point

Freedom 2The freedom to improve the program, andrelease your improvements to the public.Access to the source code is a preconditionfor this.

15 / 75


Free software today

GNU/Linux increasingly popular

• Huge hidden base: Android Smartphone, xDSL box, router,rice cooker,

• Desktop user groups in almost all countries• in France : user groups in every major city

• huge amount of different projects with public on-going sourcecode

• http://github.com• http://gitorious.org• http://freshmeat.net• http://sourceforge.net

16 / 75


Computer are fast, stupid and huge but we have to workwith

Computer are:• stupid [G. Berry]:: only know the 4 operations in base 2 anddata movement

fast do 1 operation while a photon travels few centimetershuge The number of bits in a disk is similar to the solar mass of our

galaxy

Computer science challengeHow to build a non-stupid result using numerous simple operations?

Computer science 4 pillars [G. Doweq]

1. Algorithmic :: Express human idea in computer logic2. Data :: data structure and storage to store human ideas

efficiently for the computer logic3. Compilation :: programming languages to explain what to do

to the computer (fast, correct, easy to understand, etc. )4. Hardware :: How to build a computer

17 / 75


Are we using 40 years old tools ???

Yes some tools worked on dot-matrix printer before screenage:

• ed (brother of sed) line-by-line text editor is stillinstalled in MacOS X, and vi is installed inbusybox used by many embedded systems(Android phones, DSLboxes, linux-base networkrouter)

No git and vlc are 10 years old

18 / 75


Does Unix 40 year history means. . .

• we are using 40 year old tools ?

• Most concepts are present in Multics• Some tools are still the same• however, many tools evolved• time filters things : only the mostvaluable things are kept over time

• ⇒ unix has some simple and elegantconcepts

19 / 75

http://github.com

http://gitorious.org

http://freshmeat.net

http://sourceforge.net


Unix concepts

composition and K.I.S.S • lots of small and efficient tools• easy composition of different tools to obtainpowerful effects

modularity Unix =• a kernel handling low level tasks• a set of applications

(almost) everything is a file • sockets, devices, pipes, . . .• uniform way of handling all resources

20 / 75


Everything is a file

• Why ?

In computers, all devices are about IO

• reading data from a device• writing data to a device

• File interface for all device ⇒ Fewer system calls

21 / 75


System Calls

22 / 75


Windows vs Linux http server

23 / 75


Multitasking

• an executing program is called a process• several processes execute simultaneously

• sharing of memory (code and data)• sharing of CPU resources• sharing of peripheral devices

Warningprocess != program. A process is a program in execution

24 / 75


Controlling process

Process attributes

• PID unique process ID number• PPID, UID, GID, priority

Process control

• ps, kill, nice• strace, gdb• /proc filesystem

25 / 75


Multi-user

• several users on the same machine• notion of user accounts• home directory, ID number (UID), password, name, . . .

• each file is owned by one of the users• processes are owned by the user launching them• several processes belonging to different users coexist in thesame system

26 / 75


Rights

• Users have credentials to manipulate the files (everything is afile)Basic model user/group/other: chmod, setuidAdvanced model ACL

Basic security ruleLimit the credentials to the minimum (It is a lot of work)The system may refuse some actions even with administrativerights (security, DRM, containment)

27 / 75


Memory handling

• two types of memory :• physical memory : the one really existing inside the machine• virtual memory : the address space used by processes

• less physical than virtual• if physical becomes overused ⇒ swapping on disk• laziness : memory is physically reserved at the first r/w

• hardware translation from virtual to physical (MMU)• software management of hardware translation (Page Table)

28 / 75


Filesystem

• logical organization (tree) of physical space• dependent on the OS• numerous formats• DOS/Windows 98 (FAT16, FAT32), Windows NT and later

(NTFS), Linux (ext2, ext3, ext4, btrfs, reiserfs, . . . ), . . .• CDROM, DVD• . . .

• a partition has to be mounted to become accessible

Example (mount)mount (1) to manage mounted partitionsAllow to use multiple partitions and hardwares in a single tree.

29 / 75


Low level: filesystem is a block array

• Hardware space is a large block array• Every normal file data are stored in a particular sequence ofblock

• The sequence of block of a file is described in a structure(inode) stored in a (fixed size) table, with the file metadata(UID, access right . . . save the file name)

• A directory is just a file (a single block or a sequence) storingpairs of (file name, inode number)

Example (stat)stat (1) / stat (2) to get information on a particular file

30 / 75


Medium Level: filesystem is an hardware independant

• All filesystems expose the same interface thus similar conceptas the inheritance of object programming may be applied:

• Programs and systems use filesytem through the same genericinterface (VFS)

• Filesystems differences are hidden from common users• Matroska stack for network filesystems: NFS expose VFS

interface and access (remotely) the data through VFS)

31 / 75


High level: Main directories

standard rules

/bin /sbin base system programs/usr/bin programs

/usr/share data files for programs in /usr/bin/usr/share/doc program and system documentation

/home user home directories/var /var/log log files, changing system files

/tmp temporary files/etc configuration files

/usr/local/ locally compiled or installed file

32 / 75


Special files

• Directories• Character (eg. mouse) and Block (HD, SSD) device files• Sockets• Named pipes• Symbolic link

33 / 75


Kernel

• processes handling• creation, destruction, time-sharing

• virtual memory handling• allocation, freeing, swap• one process == one memory zone

• peripheral devices accesses• keyboard, mouse, hard-disk/SSD, network, . . .

• process communication and synchronization• energy saving

34 / 75


Linux distributions

• Distribution = a set of programs distributed with the kernel• base tools for OS configuration and maintenance• user applications

• Distribution = hierarchy of directories• file positions• configuration files• one (loose) norm : FHS (Filesystem Hierarchy Standard)

35 / 75


Main Desktop Linux distributions

• Debian (Ubuntu, Mint)• Redhat (Fedora, RHEL, CentOS, OpenSuse)• Mageïa (ex Mandrake, ex Mandriva)• Oracle Linux (ex Netware)• Gentoo• Arch linux• Slackware There are 300+ others http://distrowatch.com.

36 / 75


Counter-example: Android

• The kernel is the same• The set of program is very different:

• JVM (Dalvik): most programs are Java program• C library (Bionic, not the full blend GNU Library)• Some other ad-hoc daemons

37 / 75


Hardware handling

two types of hardware :

1. handled by the kernel• serial port, disk controllers, keyboard, . . .

2. handled partially by applications (user-space drivers)• graphical card, some USB gadgets, . . .

golden rules for System Administration

• no material too recent• lots of online documentation to read first

38 / 75


Different steps

• preparation• boot• disk partitioning• files installations• base system configuration• accounts creations

39 / 75

http://distrowatch.com


Preparation

• check if the disk can be erased• else re-partitioning• boot on: CD/DVD, USB key, network (eg. PXE), harddisk/SSD, floppy disk

• UEFI may stop you from booting anything but signed kernel• GPT is much modern alternative than standard PC partitioning• Raid (hard/soft) and LVM helps a lot but should be used withcaution

40 / 75


Disk partitioning

• swap partition• / : main system partition• /boot : eventually• /home : users files• /var : changing system files• /tmp : temporary files

Why so many partitions ?Containing problems in case of filled-up partitions (eg. /var/log/)

41 / 75


Files installations

• choice of type of installation• desktop• workstation• server• scientific computing• graphic station• media center• . . .

• more fine tune by manual packages selection

42 / 75


Why so many filesystem choices

The needs change with time: storage increase with time, or changetechnology.

• Ext4 is still the standard for desktop and phone• ZFS (Sun/Oracle) allows plenty of nice things (integrity,storage pool, snapshot, copy-on-write, deduplication) but isnot part of the standard linux kernel and distribution (FUSE)

• Btrfs is still not ready yet, but should arrive very soon

43 / 75


Packages

Definition (Package)application / library / set of applications

• add or removes packages instead of files• goal : ease administration• handling of dependencies between packages (The maindifference with GooglePlay, AppleStore, etc.)

44 / 75


Different packages systems

• DEB : debian, ubuntu, mint. . .• RPM : redhat, mandriva, suse, . . .• TAR : just an archive file without dependencies• some conversion tools between packages (alien)

45 / 75


User accounts

• Role-based: at least two categories of accounts• administrators : root• user accounts

• possibilities to delegate root powers to standard users (setuid,sudo)

Warningthe root account should only be used for administration

46 / 75


Account creations

• creating a new user account• username, home directory, password• all users are referenced in the /etc/passwd file• encrypted passwords are to be found in the /etc/shadow file

• adduser command (useradd: low level)

47 / 75


Advanced account management

PAM: Pluggable Authentication Modules (Debian) libpam has 66modules. Most are different ways to authenticate auser (password, ssh keys, kerberos, two-factor, ldap,mysql, geoIP, etc.)

Kerberos third party cryptographic authenticationACL Access control lists fine grain chmod on filesroot account strong password, indirect login (su/sudo), lost root

passwordPOSIX capabilities program/shell may withdraw some rightsLDAP and Active Directory centralizing account management

48 / 75


Configuration files

• configuration files are text files• easy to modify using any text editor• global configuration files in /etc• local configuration files hidden in home directory• one file for every application

49 / 75


Access rights

• each file has an owner :• initially, the file creator

• rights are associated to each file• reading, writing, execution rights

• for the owner• for the members of the group• for all other users

50 / 75


Groups

• each user belongs to a one or more groups• possibility to share files among all users of a group• security and flexibility• the list of groups is in /etc/group• the root user can create new groups

51 / 75


Mounting points

• under DOS/WINDOWS : different peripheral with different filesystems

• a: c: d: . . .• under linux : only one root directory (/)

• adding new devices with the mount command

• configuration file : /etc/fstab

52 / 75


Command line

• text based interface to the system• one main disadvantage : less intuitive than mouse interface• many advantages :

• scripts to automate actions• very useful for distant administration (ssh)• same interface everywhere• powerful for mass operations (e.g. renaming 200 files)• very quick : automatic completion• completion allows to avoid typing mistakes

53 / 75


Base commands

ls list files in directorycat display a filerm remove a filecd change directory

mkdir make new directorymv move a filecp copy a file

grep filter file contentxargs take input as argument of the following command

54 / 75


Common and useful commands for scripts

cat concatenate files content (many UUOC)cut select columns

find find a fileseq enumerate an interval (for)

sort sortuniq remove consecutive duplicatetee save pipe in a filewc word count

head and tail first lines and last lines of a filebasename remove suffix in a name

55 / 75


Script internal commands

• Basic control flow:• branch (if-then-else)• loops• arrays• regular expressions• matching• literal and special character

• The syntax varies with the interpreters• sh/bash/dash, csh/tcsh, ksh/zsh

56 / 75


Example

Simple command, complex interactionFind the number of line of the 5 largest files in the current directoryand its sub-directoriesfind . -type f -exec ls -l ’{}’ ’;’ | sort -k 5 |tail -5 | xargs wc -l

57 / 75


Documentation

• all commands are programs• manual pages : man

• other doc in /usr/share/doc :

• programs documentation• HOWTO files

58 / 75


Devices

• all devices are accessible as files in the /dev directory• access rights to devices == standard files rights• special files created using the mknod command• standard commands (cat, . . . ) working !• example : /dev/mouse

59 / 75


Common devices

• /dev/printer• /dev/modem• /dev/mouse• /dev/dsp• /dev/cdrom• /dev/fb0

60 / 75


Disks

• IDE (ATA) disks• /dev/hda : primary master• /dev/hdb : primary slave

• SCSI and SATA disks• /dev/sda, /dev/sdb, . . .• first partition on first SCSI : /dev/sda1

• SDcard disks (Android)• /dev/block/mmcblk0

61 / 75


Configuration files

• all configuration in text files• allow easy automatic modifications using scripts• almost all files in /etc• there exists some graphical frontends and global managementsystems

WarningConfiguration files vary from distribution to distribution as dofrontends.

62 / 75


Main configuration files

• /etc/fstab : all mount points• /etc/modules.conf : kernel modules configuration• /etc/X11/xorg.conf : X11 configuration file• /etc/cups/ : all config files for CUPS service• /etc/resolv.conf : DNS config (often automatically generated)• . . .

63 / 75


Services

• service : functionality offered by the machine• program running in the background : daemon• examples :

• printing• email• web server• ssh• ftp server• . . .

64 / 75


Services

Cron configuration

• one file containing a list of jobs : /etc/crontab• man crontab

mm hh dd MMM DDD tâche > log

65 / 75


At

• delayed execution• at time action• useful for use of machines during night• examples :

• at 10pm wget http://linuxfr.org• at 2am tomorrow launch_computation

66 / 75


Anacron

• cron works only if the machine is always on• in other cases anacron is preferable• asynchronous cron• execution if not done recently

67 / 75


Kernel

• drivers can be included in the kernel• or separated in modules

• only loading useful drivers• dynamic loading / unloading at runtime

• auto-detection by hotplug

68 / 75


Modules

• some commands :• insmod : load a module• rmmod : remove a module• lsmod : display all loaded modules• modprobe : load a module with dependencies

69 / 75


Bootloader

• program enabling OS choice• if not secured : security weakness• several bootloaders :

• LILO• /etc/lilo.conf

• GRUB• /boot/grub/menu.lst

70 / 75


How to manage a large set of machine ?

• Managing 10 machines the same way as a single machine istedious

• Managing 1000 machines required other tools• Managing 100000 required other tools than previous one

71 / 75


Authenticating on a large set of machine

LDAPRemote BD often dedicated to directory of people and theiradministrative right. Easy to interconnect with local PAM service

NISOld school (No black hat). trivial to set up but no security

ScalabilityUsing a large set of machine on a single central service may becomea bottleneck.

72 / 75


Executing any command on a large set of machine

command-line is your friend

• scriptable: easy to repeat a sequence of operation• remote access: easy do do it remotely using ssh

Parallel ssh• parallel launchers exist. Use them !

• taktuk/kanif, dsh, pdsh, dish, capistrano, pssh, clustershell

• related to the scalability of authentication

73 / 75


Copy data at a large scale

BittorrentLarge scale distribution of file is quite useful !

74 / 75


Centralized configuration management

The configuration of the "client" host may be managed by amaster:

• puppet (http://www.puppetlabs.com) (ruby)• chef (http://wiki.opscode.com/display/chef) (ruby/git)• ansible (http://ansible.com) (python)• salt (http://saltstack.org/) (python)

75 / 75

http://www.puppetlabs.com

http://wiki.opscode.com/display/chef

http://ansible.com

http://saltstack.org/

Documents

whoami: Grégory Mounié - ENSIMAG · PDF fileUnix® and Linux® System Administration Handbook,Fourth ... • ed (brotherof sed) ... • ZFS(Sun/Oracle)allowsplentyofnicethings