CSE543 - Computer and Network Security Module: Virtualizationtrj1/cse543-f13/slides/cse543-virtualization.pdf · CSE543 - Computer and Network Security Page Virtual Machines • Instead

CSE543 - Introduction to Computer and Network Security Page

CSE543 - Computer andNetwork Security

Module: Virtualization

Professor Trent Jaeger

1

Friday, November 15, 13

CSE543 - Computer and Network Security Page

Operating System Quandary• Q: What is the primary goal of system security?• OS enables multiple users/programs to share resources

on a physical device‣ OS’s now have millions of lines of code

‣ Access control policies of OS become complex• E.g., SELinux

• What can we say about security?

2



Virtual Machines• Instead of using system software

to control sharing, use system software to enable isolation

• Virtualization‣ “a technique for hiding the physical

characteristics of computing resources from the way in which others systems, applications, and end users interact with those resources”

• Virtual Machines‣ Single physical resource can

appear as multiple logical resources

3



Virtual Machine • Full system simulation

‣ CPU can be simulated

• Paravirtualization (Xen)‣ VM has a special API

‣ Requires OS changes

• Native virtualization (VMWare)‣ Simulate enough HW to run OS

‣ OS is for same CPU

• Application virtualization (JVM)‣ Application API

4



Virtual Machine Types• Type I

‣ Lowest layer of software is VMM

‣ E.g., Xen, VAX VMM, etc.

• Type II ‣ Runs on a host operating system

‣ E.g., VMWare, JVM, etc.

• Q: What are the trust model issues with Type II compared to Type I?

5Penn State Systems and Internet Infrastructure Security Lab Page

Virtual Machine Monitor Approaches

Hardware!

Host OS!

VMM!

Guest OS 1! Guest OS 2!

App! App!

Hardware!

Host OS! VMM!


App! App!

Hardware!

VMM!


App! App!

Type 2 VMM! Type 1 VMM!Hybrid VMM!

JVM!CLR!

VMware Workstation!

MS Virtual Server!KVM!

VMware ESX!Xen!

MS Hyper-V!



How Can VMs Improve Security?• Isolation

‣ Separate two applications to run in two VMs

• Specialize‣ Run a hardened, specialized kernel for some applications

• Isolate groups of VMs‣ Like a VLAN

• Better IDS from outside the VM‣ VM Introspection

• Control data release to VMs‣ TCB can decide whether to release data to a new VM

• And more...

6



VAX VMM Security Kernel• A1 assured system that enforces MLS

‣ Based on an assured virtual machine monitor (VMM)• AKA hypervisor

7

Ultrix OS VMS OS

VMM Security Kernel

VMS OS

Applications(Top Secret)

Applications(Secret)

Applications(Unclassified)

MemoryDevice

DiskDevice

PrintDevice

DisplayDevice ...



VAX VMM Goals• Provide isolation boundaries between VMs

‣ On real (at the time) hardware

• Keep OS and apps working without modification (much...)• Enforce MLS access control among VMs• Enable practical management of security at coarse

granularity ‣ E.g., VMs, not processes

• Full formal assurance!

• A general-purpose foundation for security!

8



Provide Isolation• Why would this be difficult?‣ Real hardware (at that time) was designed for a

single OS

• DEC Hardware, VAX/VMS OS‣ UNIX-ish system

• Fundamental problem‣ Some HW instructions are sensitive, but not

privileged• Sensitive: May impact security - lots on VAX• Privileged: Must run in ring 0

• What property should a solution meet?9



VAX VMM Access Control• Want to enforce MLS

‣ What kind of protection system do we need?

10



VAX VMM Access Control• Subjects and objects

‣ Coarse-grained access control possible • VMs are subjects (as are users)

• Disk partitions (volumes) are objects‣ Also devices, security kernel files, virtualized resources

• Lattice policies for secrecy and integrity‣ Bell-LaPadula for secrecy

‣ Biba for integrity

• Privileges for special operations‣ E.g., administrative operations

• Discretionary access controls‣ Have them, but limited within MAC

11



Keep OS/Apps Running• Why would this be difficult?‣ OS thinks it has control of ...

12



Keep OS/Apps Running• Why would this be difficult?‣ OS thinks it has control of I/O processing

• But, now the VMM manages hardware resources‣ What happens when OS performs device I/O?‣ Sensitive instruction trapped? Then, no problem

• But if these are just memory accesses...

• This depends a lot on the hardware design‣ IBM VM/370 used sensitive instructions for I/O‣ VAX and Intel x86 used separate I/O memory

• What can be done?

13



Paravirtualization• Modify OS to be aware of virtualization for I/O‣ Mentioned this at start

• Also done for Xen/x86 and VMware initially

14



VAX VMM Security Kernel• A1 assured virtual machine system• Virtualization‣ Provide isolation

• Sensitive instructions must be virtualized (i.e., require privilege)

• Access to sensitive data must be virtualized (ditto)

‣ MLS• Mandatory protection of VMs, volumes

‣ I/O Processing• Paravirtualization• Special driver interface (all in VMM security kernel)

15



VAX VMM Challenges• Q: Why was the project cancelled?• Manual formal assurance has impacts

‣ New Drivers? In VMM... Need to assure again

‣ Limited functionality• No Ethernet

‣ 48K LOC (bigger than Multics)• 11K of assembly

• Does modern virtualization support in hardware solve these problems?‣ Modern hardware

‣ Modern hypervisors

‣ Modern assurance

16



MAC for Modern VMMs• Xen, VMware, etc. provide ‣ Isolation and I/O: sensitive instructions are made

privileged‣ What about enforcing flexible MAC policies?

• This is something that VAXVMM did...

17Penn State Systems and Internet Infrastructure Security Lab Page

IOMMU Role In System

Application!

Application!

System !Software!

RAM!

Peripheral!

Peripheral!

Peripheral!

Application!

MMU!

IOM

MU!

control



Future Virtualization• Modern Hardware‣ Native Virtualization Support‣ IOMMU

• Modern Hypervisors‣ Xen is 300K+ LOC‣ MAC enforcement is becoming complex

• Xen Security Modules

• Modern Assurance‣ Some advances, but small (seL4)‣ 10K LOC is max that has been assured

18



Xen• Paravirtualized Hypervisor• Privileged VM

19

VM: DomU VM: DomU

Xen Hypervisor

Guest OS’ Guest OS’

Partitioned Resources

DeviceRequestsDom 0

Host OS’Drivers

VM Services



MAC for Modern VMMs• Xen, VMware, etc. provide ‣ Isolation and I/O: sensitive instructions are made

privileged‣ What about enforcing flexible MAC policies?

• This is something that VAXVMM did...

20



NetTop• Isolated networks of VMs• Alternative to “air gap” security

21

VM: Secret VM: Public

SELinux Host OS


VMWareMLS

VM: Secret VM: Public

SELinux Host OS


VMWareMLS



Intrusion Detection w/ VMs• Can virtualization help in detecting an intrusion?

• Network intrusion detection‣ Can only track packets to and from host

‣ Cannot see what is running on the host

• Host intrusion detection‣ Can see processes on host

‣ But adversary can see HIDS too!

‣ Stuxnet took advantage of that

22



Intrusion Detection w/ VMs• Garfinkel and Rosenblum paper (NDSS 2003)• Premise: Use VMM to enable introspection of one

VM from another‣ For antivirus or host intrusion detection

• Leverages 3 properties of VMM‣ Isolation: protect from target‣ Inspection: can see target’s memory‣ Interposition: can intercept privileged instructions

• Can then “checkpoint” target VM‣ What is the checkpoint algorithm in terms of above 3?

23



Intrusion Detection w/ VMs

24

Policy Framework

MetadataGuest OS

OS Interface Library

Guest OS

Guest Apps

Virtual Machine

Virtual Machine Monitor

callback orResponse

Policy Modules

Query Response

Hardware State

Config File

Policy Engine

IDS

Monitored Host

Command

Figure 1. A High-Level View of our VMI-Based IDS Architecture: On the right is the virtual machine (VM) thatruns the host being monitored. On the left is the VMI-based IDS with its major components: the OS interfacelibrary that provides an OS-level view of the VM by interpreting the hardware state exported by the VMM, the policyengine consisting of a common framework for building policies, and policy modules that implement specific intrusiondetection policies. The virtual machine monitor provides a substrate that isolates the IDS from the monitored VM andallows the IDS to inspect the state of the VM. The VMM also allows the IDS to interpose on interactions between theguest OS/guest applications and the virtual hardware.

INSPECTION COMMANDS are used to directly examineVM state such as memory and register contents, and I/Odevices’ flags.MONITOR COMMANDS are used to sense when certain

machine events occur and request notification through anevent delivery mechanism. For example, it is possible fora VMI to get notified when a certain range of memorychanges, a privileged register changes, or a device statechange occurs (e.g. Ethernet interface address is changed).ADMINISTRATIVE COMMANDS allow the VMI IDS to

control the execution of a VM. This interface allows theVMI IDS to suspend a VM’s execution, resume a sus-pended VM, checkpoint the VM, and reboot the VM.These commands are primarily useful for bootstrappingthe system and for automating response to a compromise.A VMI IDS is only given administrative control over theVM that it is monitoring.The VMM can reply to commands synchronously

(e.g. when the value of a register is queried) or asyn-chronously (e.g. to notify the VMI IDS that there has beena change to a portion of memory).

4.3 The VMI IDS

The VMI IDS is responsible for implementing intrusiondetection policies by analyzing machine state and ma-

chine events through the VMM interface. The VMI IDSis divided into two parts, the OS interface library and thepolicy engine. The OS interface library’s job is to providean OS-level view of the virtual machine’s state in orderto facilitate easy policy development and implementation.The policy engine’s job is purely to execute IDS policiesby using the OS interface library and the VMM interface.

4.3.1 The OS Interface Library

VMMs manage state strictly at the hardware level, butprefer to reason about intrusion detection in terms of OS-level semantics. Consider a situation where we want todetect tampering with our sshd process by periodicallyperforming integrity checks on its code segment. A VMMcan provide us access to any page of physical memory ordisk block in a virtual machine, but discovering the con-tents of sshd’s code segment requires answering queriesabout machine state in the context of the OS running inthe VM: “where in virtual memory does sshd’s code seg-ment reside?”, “what part of the code segment is in mem-ory?”, and “what part is out on disk?”We need to provide some means of interpreting low-

level machine state from the VMM in terms of the higher-level OS structures. We would like to write the code todo this once and provide a common interface to it, instead



Introspection Challenges• Can you find what you are looking for?

‣ OS’s are complex and have important dynamic data• Lots of function pointers (data, but not really dynamic)

‣ Semantic gap gets larger when you want to inspect apps

• Can you monitor everything you need to?‣ Need to create at critical times

‣ Use privileged commands, hardware watchpoints, debuggers, or voluntary hooks (like paravirtualization)

‣ Too many interrupts impedes performance

• Can you protect yourself from adversary?‣ Adversary could try to compromise IDS from VM

‣ Adversary could try to compromise VMM from VM or IDS

25



Virtual Machine Threats• How does the insertion of a virtual machine layer

change the threats against the system?

26



Virtual Machine Rootkit• Rootkit

‣ Malicious software installed by an attacker on a system

‣ Enable it to run on each boot

• OS Rootkits‣ Kernel module, signal handler, ...

‣ When the kernel is booted, the module is installed and intercepts user process requests, interrupts, etc.

‣ E.g., keylogger

• VM Rootkit‣ Research project from Michigan and Microsoft

‣ If security service runs in VM, then a rootkit in VMM can evade security

27



Java Virtual Machine• Interpret Java bytecodes

‣ Machine specification defined by bytecode

‣ On all architectures, run same bytecodes• Write once, run anywhere

• Can run multiple programs w/i JVM simultaneously‣ Different ‘classloaders’ can result in different protection

domains

• How do we enforce access control?

28



Java Security Architecture• Java 1.0: Applets and Applications• Java 1.1: Signed code (trusted remote -- think

Authenticode)

29



Java Security Architecture

• Java 1.2: Flexible access control, included in Java 2

30



Stack Inspection• Authorize based on protection domains on the stack

‣ Union of all sources• All must have permission

31



Do Privileged• doPrivileged terminates backtrace • Like setuid, with similar risks

32



Take Away• VM systems focus on isolation between VMs

‣ Useful for coarse-grained security

• VM systems sometimes provide MAC to allow controlled interaction‣ Same kind of policies as for OS, coarse-grained objects (VMs)

• Can use for VM introspection‣ Watch out for VMM rootkits...

33


Documents

CSE543 - Computer and Network Security Module: Virtualizationtrj1/cse543-f13/slides/cse543-virtualization.pdf · CSE543 - Computer and Network Security Page Virtual Machines • Instead