Microkernels: The veterans of OS design

MicroMicrokernelskernelsThe veterans of OS design

Jakub Jermář[email protected]

@jjermar

Microkernels: The veterans of OS design, Jakub Jermář, Avast TechTalk, April 8, 2016 2

A Bit of Historical ContextA Bit of Historical Context



Per Brinch HansenRegnecentralen

The Nucleus of a Multiprogramming System (RC 4000)

1969

extensibility,message passing,

protected components





1969



Hubert ZimmermanINRIA

Chorus1979

Unix personality, multiserver,distributed systems,

“IPC”, “server”, “micro-kernel”





1969




Chorus1979



Richard RashidCarnegie Mellon University

Mach, Mach 3.01985, 1990

Unix too complicated already,4.3BSD UNIX server,distributed systems,

capability-based IPC, new VM,threads





1969




Chorus1979



Richard RashidCarnegie Mellon University

Mach, Mach 3.01985, 1990

Unix too complicated already,4.3BSD UNIX server,distributed systems,

capability-based IPC, new VM,threads

Jochen LiedtkeGMD

L41993, 1995

Improving IPC by Kernel Design,minimality principle,

fast IPC, small cache footprint,non-portability


MicrokernelMicrokernel

Hardware

Microkernel



Hardware

Microkernel

Minimality: mechanisms for address space and process management and communication



Hardware

Microkernel


Minimality: basic scheduling policies



Hardware

Microkernel


Minimality: basic scheduling policies

Minimality: mechanisms for dealing with devices


Single-server systemSingle-server system

Hardware

Microkernel

ServerLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs



Hardware

Microkernel

ServerLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

Reliability: not very helpful, the single server is just another SPOF



Hardware

Microkernel

ServerLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs


Extensibility: something extra in addition to the OS personality



Hardware

Microkernel

ServerLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs



Portability: confine most of machine dependencies in the microkernel



Hardware

Microkernel

ServerLibs

ServerLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ServerLibs

ServerLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs




Reusability: multiple OS personalities



Hardware

Microkernel

ServerLibs

ServerLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ServerLibs

ServerLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs




Reusability: multiple OS personalities

Virtualization: multiple isolated OS personalities running at the same time


Mach + UNIX serverMach + UNIX server

Hardware

Mach 3.0Device drivers

4.3BSD ServerLibs

compressLibs | TEL

sedLibs | TEL

gccLibs | TEL

ApplicationLibs | TEL

ApplicationLibs | TELTEL



Hardware


4.3BSD ServerLibs

compressLibs | TEL

sedLibs | TEL

gccLibs | TEL



UNIX syscall



Hardware


4.3BSD ServerLibs

compressLibs | TEL

sedLibs | TEL

gccLibs | TEL



UNIX syscall syscall redirection



Hardware


4.3BSD ServerLibs

compressLibs | TEL

sedLibs | TEL

gccLibs | TEL



UNIX syscall syscall redirection IPC send syscall



Hardware


4.3BSD ServerLibs

compressLibs | TEL

sedLibs | TEL

gccLibs | TEL



UNIX syscall syscall redirection IPC send syscall IPC receive wakeup



Hardware


4.3BSD ServerLibs

compressLibs | TEL

sedLibs | TEL

gccLibs | TEL






Hardware


4.3BSD ServerLibs

compressLibs | TEL

sedLibs | TEL

gccLibs | TEL






Hardware


4.3BSD ServerLibs

compressLibs | TEL

sedLibs | TEL

gccLibs | TEL



UNIX syscall syscall redirection IPC send syscall IPC receive wakeup restore context


Single-server performance ISingle-server performance I

● Mach 3.0 and Ultrix are binary compatible!● Interesting opportunity for comparison

– Chen, J., Bershad, B., N.: The Impact of Software Structure and Policy on CPU and Memory System Performance, 1993

● Execution traces → MIPS R3000 memory simulator → interesting data

● Mach has bigger MCPI than Ultrix

● IPC responsible for a small portion of overall system overhead [IC]








– Liedtke, J.: On µ-Kernel Construction, 1995



U

M

U

M

U

M

U

M

U

M

U

M

U

M

se

de

gre

py

acc

gc

cc

om

pre

ssa

be

spr

es

so

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

0.06

0.08

0.01

0.01

0.03

0.03

0.3

0.3

0.15

0.16

0.2

0.15

0.03

0.03

0.17

0.42

0.02

0.07

0.04

0.1

0.13

0.39

0.1

0.26

0.23

0.38

0.01

0.04

Baseline MCPI for Ultrix and Mach

(From Figure 3 on p. 8)

system cache miss MCPI other MCPI








– Liedtke, J.: On µ-Kernel Construction, 1995● Mach has bigger cache-miss-induced MCPI than Ultrix

● Most of these cache misses are capacity misses

● Mach has relatively more capacity misses than Ultrix

● It is the system cache footprint which matters


MkLinuxMkLinux

Hardware

OSF Mach 3.0Device drivers

Linux 2.0 ServerLibs

compressLibs

sedLibs

gccLibs

ApplicationLibs

ApplicationLibs


LL44LinuxLinux

Hardware

L4

Linux 2.0 ServerLibs | Device drivers

compressLibs | TEL

sedLibs | TEL

gccLibs | TEL




Single-server performance IISingle-server performance II

● MkLinux and L4Linux both use(d) Linux (2.0)!● Differ in the underlying microkernel

– OSF Mach 3.0 vs. L4

● Interesting opportunity for comparison– Härtig, H. et al.: The Performance of µ-Kernel-Based Systems, 1997

– Example: Linux build

● L4: 6.3% overhead● Mach: 27.1% overhead

Linux L4Linux L4Linux (T) MkLinux (K) MkLinux (U)0

100

200

300

400

500

600

700

476 476 476 476 476

30 33 79 129

Real time for compiling the Linux Server (133 MHz Pentium)

(from Figure 7 on p. 9)

Overhead

Baseline

Tim

e (

s)



● Pentium not ripe for microkernels– INT instruction for doing syscalls

– Untagged TLB

– 8/16+8/16KB L1 cache




– Untagged TLB

– 8/16+8/16KB L1 cache

● L4– Liedtke, J.: Improving IPC by Kernel Design, 1993

– Liedtke, J.: On µ-Kernel Construction, 1995● synchronous IPC, syscall symmetry: send & wait, reply & receive next, direct process switch

● direct message copy (long IPC), arguments in registers (short IPC)

● use segmentation to avoid TLB flushes

● non-portable, cache-friendly hand-coded implementation in assembly

● minimality principle




– Untagged TLB

– 8/16+8/16KB L1 cache







– Elphinstone, K., Heiser, G.: From L3 to seL4: What Have We Learnt in 20 Years of L4 Microkernels?, 2013



● Pentium not ripe for microkernels– INT instruction for doing syscalls [SYSENTER]

– Untagged TLB [PCIDs, global pages, multiple page sizes]

– 8/16+8/16KB L1 cache [larger, deeper caches]







– Elphinstone, K., Heiser, G.: From L3 to seL4: What Have We Learnt in 20 Years of L4 Microkernels?, 2013


Multiserver systemMultiserver system

Hardware

Microkernel

ServerLibs

ServerLibs

ServerLibs

ServerLibs

ServerLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs



Hardware

Microkernel

ServerLibs

ServerLibs

ServerLibs

ServerLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

Fault tolerance: some servers can crash now

Oops!



Hardware

Microkernel

ServerLibs

ServerLibs

ServerLibs

ServerLibs

ServerLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

Fault tolerance: some servers can crash now and be restarted



Hardware

Microkernel

ServerLibs

ServerLibs

ServerLibs

ServerLibs

ServerLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs


Verification: assume correctness for servers, formally verify their interactions



Hardware

Microkernel

ServerLibs

ServerLibs

ServerLibs

ServerLibs

ServerLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs



Verification: individual servers themselves are easier to check



Hardware

Microkernel

ServerLibs

ServerLibs

ServerLibs

ServerLibs

ServerLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs




Polymorphism: competing implementations in different languages



Hardware

Microkernel

Server v2.0Libs

ServerLibs

ServerLibs

ServerLibs

ServerLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs




Polymorphism: competing implementations in different languages, live updates



Hardware

Microkernel

Ext4fsLibs

TMPFSLibs

FATLibs

VFSLibs

ServerLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs




Polymorphism: competing implementations in different languages, live updates

Polymorphism: multiple implementations of the same IPC protocol


Microkernel spectrumsMicrokernel spectrums

Licenses

GPL, BSD, MIT, proprietary

Language

assembly, C, C++, SPARK, Rust

Modularity

single-server – decomposed to protocol level

Compatibility

legacy vs. clean-slate design

Components

reuse vs. reimplementation

Aims

user freedom, reliability, security

Purity

Portability

Specialization

embedded vs. general purpose


The ZOOThe ZOO


GenodeGenode

IA-32 / AMD64 / ARM / RISC-V / Muen

Base-HW / Codezero / L4/Fiasco / Fiasco.OC /L4Ka::Pistachio / Linux / NOVA / OKL4 / seL4

InitLibs

CoreLibs

TimerLibs

NitpickerLibs

ScoutLibs

FBLibs

● http://genode.org,GPLv2 or commercial license

● Operating system framework for composition of sophisticated systems

● Components also from Linux, DDE, rump kernels, OpenBSD, iPXE, FUSE

● Scenarios: what components will be included

● Recursive structure, resource trading

LaunchpadLibs

liquid_fbLibs

NouxVFS | Libs

bashLibs

http://genode.org/


HelenOSHelenOS

● http://helenos.org, BSD

● Clean-slate design, reimplementation of components

● Modularity: fully decomposed networking stack

● Portability: wide range of supported architectures

● Functionality: USB, sound, networking, file systems, drivers, GUI

AMD64 / ARM / IA-32 / IA-64 / MIPS / PowerPC / SPARC V8 / SPARC V9

SPARTAN

ata_bdLibs

vfsLibs

nsLibs

ethipLibs

e1000Libs

inetsrvLibs

tcpLibs

locsrvLibs

devmanLibs

ext4fsLibs

inetsrvLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

http://helenos.org/


HurdHurd

IA-32 / Xen PV

GNU MachLinux drivers*

authLibs

storeiolibstore

netddeLibs

pfinetLibs

ext2fslibdiskfs

procLibs

ApplicationLibs

ApplicationLibs

ApplicationLibs

mplayerlibrumpuser

● http://hurd.gnu.org, GPLv2

● Debian GNU/Hurd, ~80% packages (~34000)

● Freedom #0 (from sysadmin and failures, and to experiment)

● Translators interpose IPC of services exposed in the filesystem

● Support legacy GNU/Linux applications

● Components also from Linux, DDE, rump kernels

http://hurd.gnu.org/


MINIX 3MINIX 3

IA-32, ARM

MINIX kernel

VFSLibs

MFSLibs

e1000Libs

INET / lwIPLibs

RSLibs

DSLibs

VMLibs

SCHEDLibs

ApplicationLibs

AHCILibs

● http://minix3.org, BSD

● NetBSD userland, thousands packages

● Focus on reliability features: resurrection service, live update

● Components also from lwIP, DDE, rump kernels

MIBLibs

PMLibs

ProcFSLibs

isofsLibs

ext2Libs

http://minix3.org/


Microkernels.infoMicrokernels.info


In ConclusionIn Conclusion

● Monolithic OSes move functionality to userspace (FUSE, UIO, UMDF)

● Modern Intel CPU more than 1000x faster than the original Pentium

● Modern processors better suited for microkernels

● People have chosen high level programming languages over performance

● Why not choose the added value of microkernels over performance too?

● Microkernels will increasingly use high level programming languages

● Microkernels often function as hypervisors

● Microkernels will hopefully become more ubiquitous


Q&Awww.microkernel.info

Thank you!Thank you!

Technology

Microkernels: The veterans of OS design