Machine Virtualization:Efficient Hypervisors, Stealthy Malware

Muli Ben-Yehuda

Technion &Hypervisor Technologies and Consulting Ltd

Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 1 / 21

Background: x86 machine virtualization

Running multiple different unmodified operating systemsEach in an isolated virtual machineSimultaneouslyOn the x86 architectureMany uses: live migration, record & replay, testing, . . . , securityFoundation of IaaS cloud computingUsed nearly everywhere

Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 2 / 21

x86 virtualization primer

How does it work?Popek and Goldberg’s virtualization model [Popek74]: Trap andemulatePrivileged instructions trap to the hypervisorHypervisor emulates their behaviorWithout hardware supportWith hardware support

Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 3 / 21

What is a rootkit?

First you take control. How?Then you hide to avoid detection and maintain control. How?Usual methods are ugly and intrusive: easy to detect!Can rootkit authors do better?

Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 4 / 21

Hypervisor-level rootkits

Hypervisors have full control over the hardwareHypervisors can trap any operating system eventCode can enter hypervisor-mode at any timeBluepill: run the rootkit as the hypervisor

Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 5 / 21

Bluepill: a hypervisor level rootkit [Rutkowska06]

Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 6 / 21

Recursive Bluepill

Bluepill installs itself on the flyBluepill is now the hypervisorReminder: x86 only supports one hypervisor in hardwareSo how can you bluepill bluepill?

Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 7 / 21

The Turtles project: Nested x86 Virtualization

Efficient nested virtualization for Intel x86 based on KVMRuns multiple guest hypervisors and VMs

“The Turtles Project: Design and Implementation of Nested Virtualization”, [Ben-Yehuda10]

Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 8 / 21

What is the Turtles project? (cont’)

Nested VMX virtualization for nested CPU virtualizationMulti-dimensional paging for nested MMU virtualizationMulti-level device assignment for nested I/O virtualizationMicro-optimizations to make it go fast

+ + =

Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 9 / 21

Theory of nested CPU virtualization

Trap and emulate[PopekGoldberg74] ⇒ it’s all about the trapsSingle-level (x86) vs. multi-level (e.g., z/VM)Single level ⇒ one hypervisor, many guestsTurtles approach: L0 multiplexes the hardware between L1 and L2,running both as guests of L0—without either being aware of it(Scheme generalized for n levels; Our focus is n=2)

Hardware

Host Hypervisor

Guest

Hardware

Host Hypervisor

Multiplexed on a single level Multiple logical levels

L0

L1

L2

L1

GuestL2

GuestL2

L0

GuestL2L2

Guest Hypervisor

GuestHypervisor GuestGuest

Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 10 / 21

Detecting hypervisor-based rootkits

Bluepill authors claim “undetectable”“Compatibility is Not Transparency: VMM Detection Myths andRealities” [Garfinkel07]Hardware discrepanciesResource-sharing attacksTiming attacks: PCI register access, page-faults on MMIO access,cpuid timing vs. nopsCan you trust time?

Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 11 / 21

The Dual Role of a Hypervisor

Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 12 / 21

Background: interrupts

IDTIDTR

Limit

Address

IDT Entry

IDT Entry

…

IDT Entry

Vector 1

Vector n

Vector 2

InterruptDescriptorTable

IDTRegister

Interrupt handlers

I/O devices raise interruptsCPU temporarily stops the currently executing codeCPU jumps to a pre-specified interrupt handler

Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 13 / 21

Interrupts as an Attack Vector

Follow the White Rabbit [Rutkowska11]Tell the device to generate “interesting” interruptsAttack: fool the CPU into SIPIAttack: syscall/hypercall injectionIn interrupt-based attacks an untrusted guest generates maliciousinterrupts which are handled in host modeProtect: handle interrupts in guest—not host—modeServe: bare-metal performance!

Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 14 / 21

ELI: Exitless Interrupts

bare-metal

Baseline

guest

hypervisor

(time)

ELI delivery

guest

hypervisor

ELIdelivery & completion

guest

hypervisor

PhysicalInterrupt

Interrupt Completion

(a)

(b)

(c)

Interrupt Injection

Interrupt Completion

(d)

ELI: direct interrupts for unmodified, untrusted guests

“ELI: Bare-Metal Performance for I/O Virtualization”, Gordon12

Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 15 / 21

ELI: delivery

ShadowIDT

Hypervisor

ShadowIDT

InterruptHandler

AssignedInterrupt

PhysicalInterrupt

Non-assignedInterrupt(#NP/#GP exit)

ELIDelivery

GuestIDT

VM

IDT Entry

IDT Entry

…

IDT Entry

P=0

P=1

P=0

Handler

#NP

#NP

IDT Entry#GP

IDTRLimit

All interrupts are delivered directly to the guestHost and other guests’ interrupts are bounced back to the host. . . without the guest being aware of it

Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 16 / 21

ELI: signaling completion

Guests signal interrupt completions by writing to the LocalAdvance Programmable Interrupt Controller (LAPIC)End-of-Interrupt (EOI) registerOld LAPIC: hypervisor traps load/stores to LAPIC pagex2APIC: hypervisor can trap specific registers

Signaling completion without trapping requires x2APICELI gives the guest direct access only to the EOI register

Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 17 / 21

ELI: threat model

Threats: malicious guests might try to:keep interrupts disabledsignal invalid completionsconsume other guests or host interrupts

Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 18 / 21

ELI: protection

VMX preemption timer to force exits instead of timer interruptsIgnore spurious EOIsProtect critical interrupts by:

Delivering them to a non-ELI core if availableRedirecting them as NMIs→unconditional exitUse IDTR limit to force #GP exits on critical interrupts

Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 19 / 21

Conclusions

Machine virtualization be used for good, or evilHow do you protect and serve?Happy hacking!

Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 20 / 21

Questions?

muli@cs.technion.ac.ilmulix@hypervisorconsulting.com

Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 21 / 21