Virtualization

Yiying Zhang

Acknowledgement: slides adapted/extended from Steve Gribble (Washington) (https://courses.cs.washington.edu/courses/cse490h/08au/lectures/cse490_virtualization.pdf)

Virtualization• Adding another level of indirection to run OSes

on an abstraction of hardware

• Virtual Machine (Guest OS)– OS that runs on virtualized hardware resources– Managed by another software (VMM/Hypervisor)

• Virtual Machine Monitor (Hypervisor)– The software that creates and manages the execution of

virtual machines– Runs on bare-metal hardware

Regular Machine Stack

Virtualization

Outline• History of virtualization

• Core virtualization technologies

• Different forms of virtualization

Virtualization – a Brief History• Invented by IBM in 1960’s (System/360):

– Sharing resources on expensive mainframes– CP: a “control program” that created and managed virtual

machines – CMS: the “Cambridge monitor system” -- a lightweight,

single-user OS

• 1970’s - 1990’s:– Cheap hardware and multiprocess OS became popular– Motivation for virtualization became unclear – Virtualization became unpopular

Virtualization – a Brief History• VMware co-founded by Mendel Rosenblum and Diane

Green in 1998 – commercialized ideas incubated in Stanford DISCO project,

which originated from Stanford FLASH Multiprocessor– Brought virtualization to PC computers

• Their initial market was software developers – often need to develop and test software on multiple OSs

(windows, linux, …) – using multiple PCs is very inconvenient – instead, run multiple OSs simultaneously in separate VMs

Server Consolidation• Big companies (datacenters)

– operate many services: mail servers, file servers, Web servers, search services

– want to run at most one service per machine (administrative best practices)

– leads to low utilization, lots of machines, high power bills, administrative hassles

• Instead, run one service per virtual machine – and consolidate many VMs per physical machine – leads to better utilization, easier management

• Much larger market when cloud computing started

Other Benefits• Resource management is simplified

– Application can be started from preconfigured VM images / appliances

– Virtualization layer / hypervisor permits resource allocations to be varied dynamically

– VMs can be migrated without application down-time

Outline• History of virtualization

• Core virtualization technologies

• Different forms of virtualization

Regular Machine Stack

What is an OS?• A piece of software that manages and virtualizes hardware for

applications– An indirection layer between applications and hardware– Provides a high-level interface to applications, – while interact with hardware devices with low-level interfaces– Runs privileged instructions to interact with hardware devices

• Applications– Can only execute unprivileged instructions– Perform system calls or faults to “trap” into OS– OS protect applications from each other (to some extent) (e.g., address space)

• Another practical definition of OS:– A giant interrupt handler (hardware interrupts, software faults, system calls)

Virtualization

Protection Rings• Only Ring 0 can execute privileged instructions• More privileged rings can access memory of less privileged ones• Calling across rings can only happen with hardware enforcement

Image Source: https://commons.wikimedia.org/wiki/File:CPU_ring_scheme.svg

Trap-and-Emulate• Cannot allow guest OSes to directly manipulate

hardware with “sensitive” instructions

• Hand off sensitive operations to the hypervisor• Hypervisor emulates the effect of these operations on

virtual hardware provided to the guest OS

• Performance implications– Almost no overhead for non-sensitive instructions– Large overhead for sensitive instructions

Goldberg (1974) two classes of instructions– privileged instructions: those that trap when in user mode– sensitive instructions: those that modify or depends on hardware configs

System Call without Virtualization

System Calls with Virtualization

Virtualizing Memory• Extra level of memory addressing

Virtualizing Memory• TLB miss flow with software-managed TLB

Logistics• Attendance tracked from next week

• Sign up for discussion lead before the end of this weekend (Sunday midnight)

• Form your group and your selection of project by next Wed (and email TA your group info)

• Project proposal due by 10/18 midnight

Outline• History of virtualization

• Core virtualization technologies

• Different forms of virtualization

X86 Difficulties• Not all sensitive instructions are privileged,

i.e., non-virtualizable processor– These instructions do not trap and behave differently in

kernel and user mode

• Hardware-managed TLB– Hardware does page table walk on each TLB miss– Hypervisor doesn’t have chance to intercept on TLB

misses

Possible Solutions • Emulate

– Interpret each instruction, super slow (e.g., Virtual PC on Mac)

• Binary translation– Rewrite non-virtualizable instructions (e.g., VMware)

• Para-virtualization– Modify guest OS to avoid non-virtualizable instructions (e.g., Xen)

• Change hardware– Add new CPU mode, extend page table, and other hardware

assistance (e.g., Intel VT-x, EPT, VT-d, AMD-V)

Reading: The Evolution of an x86 Virtual Machine Monitor

Binary Translation• Hypervisor dynamically rewrites non-virtualizable

instructions to invoke hypervisor

• Pros – No need to modify guest OS (or applications)– Performance is decent since majority of the instructions still run at

close-to-native speed

• Cons– Implementing hypervisor can get tricky– Performance is not as good as paravirtualization or hardware-

assisted virtualization

Working with Hardware-Managed TLB

• Guest OS manages its own page tables– Virtual to physical page mappings– Virtualized CR3– Hardware does not know these

• Hypervisor manages shadow page tables– Virtual to machine page mappings– The actual CR3 and page tables that hardware sees– Hypervisor marks the memory that stores guest page tables

as write-protected (guest changing its page tables will trap to hypervisor)

Working with Hardware-Managed TLB

Image source: https://rayanfam.com/topics/hypervisor-from-scratch-part-4/

Para-virtualization• Full virtualization (no guest OS modification)

– Tricky and has performance overhead

• Para-virtualization: modified guest OS– Guest OS works with hypervisor (i.e., knows that it is a

VM) and has some exposure to hardware– e.g., guest OS informs hypervisor of page table changes– e.g., guest OS directly calls hypervisor on system calls

(hypercalls)– Better performance, but need to modify guest OSes

Hardware-Assisted Virtualization: Intel VT-x

• VMX Transitions– Between VMX root and non-

root operations– VM Entry: transition into non-

root– VM Exit: transition into VMX

root

• VMCS (VM Control Structure)

– Manages VMX transitions– Guest- and host-states saved

and loaded during transitions

Image source: https://www.anandtech.com/show/2480/9

(Ring -1)

Hardware-Assisted Virtualization:Extended Page Table (EPT)

Hardware-Assisted IO Virtualization: SR-IOV (Single-Root I/O Virtualization)

Image source: https://www.ramcity.com.au/data-storage/internal-ssd/pcie-aic-enterprise/MZPLL1T6HEHP-00003

Image source: https://www.juniper.net/documentation/en_US/junos/topics/concept/disaggregated-junos-sr-iov.html

Type 1 and Type 2 Hypervisor• Type 1: Hypervisor runs directly on hardware

– Examples: VMware ESX, Xen, Microsoft Hyper-V

• Type 2: Hypervisor runs on hardware– Examples: VMware Workstation, QEMU, Microsoft Virtual PC, Virtual Box

Hardware

Hypervisor

Guest OS Guest OS

Type 1

Hardware

Guest OS

Hypervisor

Guest OS

OS

Type 2

KVM and QEMU

Xen

Source: https://wiki.xen.org/wiki/Xen_Project_Software_Overview