“Operating Systems” Introduction to the Course and to Operating Systems

Otto J. Anshus

OS Course History• Inspirations

– LurOS ‘78 • Stein Krogdahl, OS course, Dept. of Computer Science, University of Tromsø, 1978 • Paper, no metal, but detailed, and small enough to be understandable and raise my interest

– Mymux (Mycron Multiplexer) ‘79 • Stein Gjessing (spec, 1979), Otto Anshus (dumbed down spec & implementation, adding multicomputer, 1980), Peter Jensen

(initialization code, 1980), Sigurd Sjursen (advisor for Anshus, debugging interrupt SW & HW, 1980), • Used in OS course, Dept. of Computer Science, UiTø (1981-83) • Technical

– Mycron 1 (64KILObyte RAM, no disk, 16 bit address space, Intel 8080/Zilog 80) • Hoare monitors, multiple computers (3 computers, UART, 300 bits/sec, transparent process and monitor location,

process and monitor migration between machines) • POS (Project Operating System), a.k.a. TeachOS, a.k.a. LearnOS

– 1994: Otto Anshus, Tore Larsen, first light (working) code by Åge Kvalnes (& Brian Vinter), OS course, Dept. of computer science, UiTø, 1994-1998, laptops w/Intel 486/Pentium

– 1998: Princeton University, USA, Kai Li, adopts the projects (and adding P6: File System), Pentium desktop PCs – 1999: Tromsø & Princeton: Common code platform (Kai Li, O. Anshus@1998/99@PU) – 2001: Oslo: Vera Goebel, Thomas Plageman, Otto Anshus – 2006: Auburn – 2007: Yale (Yale is now (2013’ish) on their own multi-core OS) – 2008: Humboldt-Universität zu Berlin

Approaches to Teaching Operating Systems

• Paper: read about it, do exercises on paper (Read & Forget?) • Smaller exercises using existing operating systems (See &

Remember?) • Modifications to existing systems (See & Remember?)

– Emulator • NachOS

– ”Metal”, bare machine • Unix, BSD, Linux, … • Original Minix & Latest Minix

• Do your own (Do & Understand?) – Overcome the barrier, dive into the system – Gain confidence: you have the power instead of only SW, OS and

computer vendors

Approaches to Teaching Operating Systems

• Paper: read about it, do exercises on paper (Read & Forget?) • Smaller exercises using existing operating systems (See &

Remember?) • Modifications to existing systems (See & Remember?)

– Emulator • NachOS

– ”Metal”, bare machine • Unix, BSD, Linux, … • Original Minix & Latest Minix

• Do your own (Do & Understand?) – Overcome the barrier, dive into the system – Gain confidence: you have the power instead of only SW, OS and

computer vendors

We’ll do our own

Course Approach• You will do a very small and simplified operating system

– But with most of the significant fundamentals • However, we don’t do:

– windows manager and desktop, multi-touch human-computer interfaces, multi-core

• You/we will do it in steps. For each step: – We: define what your OS should achieve for this step

• You: Say what? Chaos. – We: provide you with a starting point (pre code files)

• You: Despair. I hate you. – You: contemplate a design and present a design report indicating design issues, discussions,

and decisions. The design report is presented, discussed, and reviewed by staff/TA’s • You: Frustration. Irritation.

– You: develop, implement, and debug a working prototype (post code files) • You: cries will be heard late at night in the hallways. All is hopeless as well as useless. • Do Not Cheat: Discussions are OK, but no copying of code, please. You learn by trying & making

mistakes. Besides, we may find out if you cheat and it’ll be embarrassing for both you and us.

• You: By the end of the semester: King of the Hill (Konge på Haugen) – You: Life is beautiful. Happy singing in the hallways. It is OK not to hate us.

Course Approach• You will do a very small and simplified operating system

– But with most of the significant fundamentals • However, we don’t do:

– windows manager and desktop, multi-touch human-computer interfaces, multi-core

• You/we will do it in steps. For each step: – We: define what your OS should achieve for this step

• You: Say what? Chaos. – We: provide you with a starting point (pre code files)

• You: Despair. I hate you. – You: contemplate a design and present a design report indicating design issues, discussions,

and decisions. The design report is presented, discussed, and reviewed by staff/TA’s • You: Frustration. Irritation.

– You: develop, implement, and debug a working prototype (post code files) • You: cries will be heard late at night in the hallways. All is hopeless as well as useless. • Do Not Cheat: Discussions are OK, but no copying of code, please. You learn by trying & making

mistakes. Besides, we may find out if you cheat and it’ll be embarrassing for both you and us.

• You: By the end of the semester: King of the Hill (Konge på Haugen) – You: Life is beautiful. Happy singing in the hallways. It is OK not to hate us.

“Can I do it myyyy way?”: No. We know that our way, “dumb and boring” as it surely is, is a path to a doable solution, and we can give you better help more efficiently if you humor us. Besides, the TAs will hate you if you give them your strangely brilliant code organized in your own strangely brilliant ways, and demand that they help you figure out what you have done so brilliantly strangely.

Why Bother?• OS is a key part of a computer system

– it makes our lives better (and worse) – it is “magic” and we want to understand how – it has “power” and we want to have the power

• OS is complex – “... understand the system at all abstraction levels” (Yale Patt) – “The devil is in the details” (https://books.google.com/ngrams) – How many instructions and procedures does a keystroke invoke? – What happens when your running application program (let’s call it an application level

process) dereferences a null pointer? (run-time error, immediate program crash, something else)

– Real technical insight vs. dogmatism (“amazing stuff”)

• Tradeoffs between performance and functionality, division of labor between HW and SW

• Combine language, hardware, data structures, algorithms, money, art, luck, and more.

• Great Pretender/Magician (implemented by software between applications and hardware) – Make finite resources appear (more) as infinite resources

• Would be great if we had an infinite number of CPUs, size of memory, and network latency & bandwidth

– Invent and implement abstractions: Make resources look & behave to applications like the OS wants them to do

• Resource Manager – Give resources to applications – Take resources from applications – Protection and Security

• Control Freak – Must never (ever!) loose control of the hardware

But what is an Operating System?

• Great Pretender/Magician (implemented by software between applications and hardware) – Make finite resources appear (more) as infinite resources

• Would be great if we had an infinite number of CPUs, size of memory, and network latency & bandwidth

– Invent and implement abstractions: Make resources look & behave to applications like the OS wants them to do

• Resource Manager – Give resources to applications – Take resources from applications – Protection and Security

• Control Freak – Must never (ever!) loose control of the hardware

But what is an Operating System?

Hang on, say again: “between” … ?

How To Deal with Complexity (a.k.a. Best Advice You Will Ever Get)

?

7

How To Deal with Complexity (a.k.a. Best Advice You Will Ever Get)

?(a.k.a. Scream and Leap)

7

Do Early - Fail Early

8

Opera&ng)Systems)TeachOS(

(OS)(

(ESXI)(

Opera&ng)Systems)

Cell)OS)

Xbox)360))Dashboard)

Wii)U)System)So>ware)

“Old”Opera+ngSystems:Mul&cs,IBMOS/360,IBMVM/370,MSDOS,Unix’es,Mach,L4,Exokernel(Aegisandothers)

New(er)Opera+ngSystems:Xen,Barrelfish(ETHZurich),Corey/TheLibraryOS(MIT),Tessela&on(Berkley),L4-based,others

https://en.

wikipedia.o

rg/wiki/Lis

t_of_opera

ting_syste

ms

Source Lines of Code (SLOC)

• Today is like in the late 60s. OS's are large – Small OS: 100K lines – Real OS is huge and very expensive to build

• big OS: 10M lines (and more), 100-1000 people years • Win/NT: 8 years, 1000s of people

• OS Comparisons • But ~90% are device drivers • We don’t do device drivers (well, just a few)

– Project 5 (“post files”) • ~6500 lines • Gradually built up & some we give you

SLOC Linux 2.6.18

• Architecture specific: 817,880 • x86 specific: 55, 463 • Drivers: 2,365,256 • Common: 1,800,587 • Typical config: ~2,000,000 • Minimized and hardened: ~500,000 • http://www.informationisbeautiful.net/visualizations/million-lines-of-code/

10

Typical Unix OS “Structure”

Application

Libraries

Portable OS code

Machine-dependent OS code

•Low-level system initialization and bootstrap

•Fault, trap, interrupt and exception handling

•Memory management: hardware address translation

•Low-level kernel/user-mode process context switching

•I/O device driver and device initialization code

C

Assembler

System Call Interface

• Easier to do

•Performance

User LevelKernel Level

Low-level architecture dependent programming (we’ll stick with only one architecture in only one configuration)

Yet Another Reason Why It Matters to Study Operating Systems Somebody Must Do The

Operating Systems of Tomorrow• Moore’s Law

– 1965: #transistors/inch2 doubles/12 months 1975: #transistors/inch2 doubles/24 months (in “performance” every 18 months)

• Increasing Density - Increasing Trouble? – Just cranking frequency: chip gets too hot

• 0.5 micron: hot plate, 0.1 micron: nuclear reactor

– Why not just increase per CPU or core functionality? • not very much pay-off (instruction level parallelism: ILP)

• or chip becomes complicated (thread-level parallelism: TLP)

– We need architectures with multi and many (simpler, possibly specialized) cores (?) • 28 single core PC’s vs. one 112 core GPU doing Mandelbrot: Who produces output fastest?

– GPU wins, no contest: 190x faster

12

The World may actually need You

New Architectures - New OS Approaches Needed

• CPU vs. GPU – Victor W. Lee, Changkyu Kim, Jatin Chhugani, Michael Deisher, Daehyun Kim, Anthony D. Nguyen, Nadathur Satish, Mikhail

Smelyanskiy, Srinivas Chennupaty, Per Hammarlund, Ronak Singhal, and Pradeep Dubey. Debunking the 100X GPU vs. CPU myth: an evaluation of throughput computing on CPU and GPU. In Proceedings of the 37th annual international symposium on Computer architecture (ISCA '10). ACM, New York, NY, USA, 451-460. DOI=http://dx.doi.org/10.1145/1815961.1816021

• Data-centers and cloud – Jitsu: Just-In-Time Summoning of Unikernels, Anil Madhavapeddy, Thomas Leonard, Magnus

Skjegstad, Thomas Gazagnaire, and David Sheets, University of Cambridge; Dave Scott, Citrix Systems UK Ltd.; Richard Mortier, Amir Chaudhry, and Balraj Singh, University of Cambridge; Jon Ludlam, Citrix Systems UK Ltd.; Jon Crowcroft and Ian Leslie, University of Cambridge

• https://www.usenix.org/conference/nsdi15/technical-sessions/presentation/madhavapeddy

• This paper is included in the Proceedings of the 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’15). May 4–6, 2015 • Oakland, CA, USA

13

Is it challenging to write an OS?• Yes

– Don’t panic, you’ll manage employing your own efforts, and the assistance of fellow students, Teaching Assistants (TAs), (and perhaps even professors :))

• Must – understand a few new abstractions – figure out how a computer works in some detail

• This is serious stuff so don’t sweat it – enjoy the crashes (frustration is anyhow unavoidable) – destructive testing works (what can possibly go wrong?)

Remember Best Advice Ever

From Tony Hoare’s Biography http://research.microsoft.com

• First, Success – He led a team in the design and delivery of the first

commercial compiler for the programming language Algol 60. (1960)

• Second, Failure – He then led a larger team on a disastrous project to

implement an operating system • Then, Consequence

– His research goal was to understand why operating systems were so much more difficult than compilers, and to see if advances in programming theory and languages could help with the problems of concurrency. (Queens Univ. 1968)

Why OS is not Trivial• Obvious Idea: KISS-OS

– Keep It Simple, Stupid-OS: Do one “thing” at a time • Obvious drawback: It is simply stupid

– Inefficient: If the single thing is waiting for something, whole computer sits idle – Costly: Need multiple computers per user to run multiple applications

• Obvious Improvement – Run more than one thing “at the same time” (interleaved or overlapped), when one thing is delayed, switch to

another thing to do • Obvious Potential Problems

– Performance: N users, M things per computer • Each thing can experience a slow-down of about N*M (it depends on many things, including

number of CPUs, cores, and if the thing can benefit from multiple CPUs & cores) • What if N & M becomes (too) large (not enough memory, long time until a thing gets to do things)?

– Protection: (against evil/friendly users and things) • a thing runs an infinite loop: processor starvation of other things unless the OS can get control back

again • a thing starts to randomly access memory: Can destroy other things’ things, unless the OS prevents it

• Obvious lesson learned – OS must carefully share resources while protecting resources, users and applications (from each

other) – This can become complicated, and complicated can be the devil

Why OS is not Trivial• Obvious Idea: KISS-OS

– Keep It Simple, Stupid-OS: Do one “thing” at a time • Obvious drawback: It is simply stupid

– Inefficient: If the single thing is waiting for something, whole computer sits idle – Costly: Need multiple computers per user to run multiple applications

• Obvious Improvement – Run more than one thing “at the same time” (interleaved or overlapped), when one thing is delayed, switch to

another thing to do • Obvious Potential Problems

– Performance: N users, M things per computer • Each thing can experience a slow-down of about N*M (it depends on many things, including

number of CPUs, cores, and if the thing can benefit from multiple CPUs & cores) • What if N & M becomes (too) large (not enough memory, long time until a thing gets to do things)?

– Protection: (against evil/friendly users and things) • a thing runs an infinite loop: processor starvation of other things unless the OS can get control back

again • a thing starts to randomly access memory: Can destroy other things’ things, unless the OS prevents it

• Obvious lesson learned – OS must carefully share resources while protecting resources, users and applications (from each

other) – This can become complicated, and complicated can be the devil

By the way, we will soon call the implementation of a thing for a process (and then refine it further with threads)

How To Deal with Complexity

17

How To Deal with Complexity

17

I don’t know

How To Deal with Complexity

17

I don’t know

I do believe that the Best Advice You’ll Ever Get is to

Do early, fail early, and learn how to make things work

Some of What You Will Learn• Operating System Structure & basic abstractions

– structures, processes, threads, and system calls • Synchronization

– locks, mutex, semaphores, monitors • Processor

– time slices, scheduling • Virtual memory

– address spaces, demand paging • Communication and I/O subsystems

– device drivers, inter process and inter thread communication, networking

• Storage systems – disks and file systems

CPU

ChipsetMemory

I/O bus

CPU.

NetworkKeyboard

ROM

USB Mem Stick

A Simple Computer

CPU

ChipsetMemoryI/O bus

CPU. . .

NetworkKeyboard

ROM

USB Mem Stick

More on the OS Projects• 6 projects

– From boot to a useful OS kernel w/demand paging – We hand out templates (pre files), but never the finished

source (post files) • Lectures and Projects are (somewhat) synchronized • 2-3 weeks/project • Design Review • Linux, C, assembler • Close to the computer,

– but emulator (Vmware/Bochs/Virtual PC) is useful to reduce the number of reboots

– Or have an extra PC to try your code on

The Projects• P1: Bootup

– Bootblock, createimage, boot first ”kernel”

• P2: Non-preemptive kernel – Non-preemptive scheduling, simple syscalls, simple locks

• P3: Preemptive kernel – Preemptive scheduling, syscalls, interrupts, timer, Mesa style monitor (practical version of

the original Hoare monitor), semaphores (Dijkstra) • P4: Interprocess communication and driver

– P3 functionality+keyboard interrupt & driver, message passing, simple memory management, user level shell

• P5: Virtual memory – P4 + demand paging memory management

• P6: File system

Platform• PC with Intel Pentium or better • USB memory stick as “hard disc” for the OS we are developing

(previously floppy drive - I’m still at the floppy drive…)

• Develop code for the OS (we know the projects will work on the version installed for you in the

lab/PC room) – Linux – Language C (gcc) and assembler (gas from gnu)

• To boot OS and see what happens – PC emulators/virtual machines on top of Linux/Windows/OS X

• VMware (projects may work) • Bochs (projects should work)

– The projects must at least work on the bare PCs in the lab

Literature• Modern Operating Systems, by Andrew (Andy) Tanenbaum, Prentice-Hall • Information given on the course web pages. • We will also provide additional readings/notes/documents. Please, check the

syllabus/course web pages • All lectures, lecture notes, precept notes and topics notes • All projects • IA-32 Intel Architecture Software Developer's Manual, Volume 3: System

Programming Guide • Other books that may help you are:

– Protected Mode Software Architecture, by Tom Shanley, MindShare, Inc. 1996. This book rehashes several on-line manuals by Intel

– Undocumented PC, 2nd Edition, by Frank Van Gilluwe, Addison-Wesley Developers Press, 1997

– The C Programming Language, Brian W. Kerningham, Dennis M. Ritchie

The Best Advice etc.

24

?

The Best Advice etc.

24

?Do Early - Fail Early