Cloud Computing Infrastructure Security

ACNS 2012 1

Computer Science

Cloud Computing Infrastructure Security

Peng NingWith Ahmed Azab, Xiaolan Zhang, Wu Zhou, Xuxian Jiang, and Zhi Wang.

June 29, 2012

Supported by the US NSF under grant # 0910767, the US ARO under the grant # W911NF-08-1-0105, and IBM under Open Collaboration Research (OCR) Awards.

Computer Science ACNS 2012 2

Outline

• Background• Security threats to cloud computing• Security of cloud computing infrastructure

– Driven by a new security architecture for cloud computing– Hypervisor-based security services– Offline VM image security services– Hypervisor integrity services– Isolation that can bypass hypervisor control

• Conclusion

June 29, 2012


What is Cloud Computing

• Wikipedia– Cloud computing is a paradigm of computing in which

dynamically scalable and often virtualized resources are provided as a service over the Internet

– Users need not have knowledge of, expertise in, or control over the technology infrastructure in the "cloud" that supports them

– Reduction in hardware, operational and administrative costs

• Virtualization is a key to cloud computing– Scalability– Ease of use– Affordable pricing

June 29, 2012


Example: Amazon Elastic Compute Cloud (EC2)

June 29, 2012

EC 2 Management ConsoleStart an InstanceSet up the InstanceLaunch the InstanceVerify the Instance


Security Threats in Cloud Computing

External threats

Guest-to-guest threats

Guest-to-cloud threats

Cloud-to-guest threats

June 29, 2012


Cloud Computing Infrastructure Security

• Our proposal – A security architecture for compute clouds

• Focus on Infrastructure as a Service (IaaS)– Addition of security architecture components

• Hypervisor-based security services• Offline VM image security services• Hypervisor integrity services• Isolation mechanisms that can bypass the hypervisor

June 29, 2012


A Typical Compute Cloud

June 29, 2012


Virtualization-based Runtime Security Services

June 29, 2012

HIMA [ACSAC ’09]HookSafe [CCS ’09]

Computer Science ACNS 2012

Example Service: HIMA• HIMA: Hypervisor based Integrity Measurement Agent• Validation of VMs with runtime guarantees

– Measure the VM OS and applications loaded into guest VMs– Actively monitor all guest events that could change measured

applications• Time of Check to Time of Use (TOCTTOU) consistency

• Prototypes– Initial implementation works for Xen (para-virtualization)– Ported to support KVM (hardware assisted virtualization)

June 29, 2012 9

Ahmed M. Azab, Peng Ning, Emre C. Sezer, and Xiaolan Zhang, "HIMA: A Hypervisor Based Integrity Measurement Agent," in Proceedings of ACSAC 2009, December 2009.


VM Image Security Services

June 29, 2012

Nüwa—offline patching [ACSAC ’10]Offline virus scanning [CCSW ’09]


Example Service: Nüwa（女娲）• Nüwa – Offline Patching of VM Images• Motivation

– Dormant VM images usually contain vulnerabilities– Offline patching service

• A desirable security service in compute clouds• Technical challenge

– Current patching system: Designed for running systems– Pre- and post-processing scripts– Examples: Stop/start daemons; Conditional updates

June 29, 2012 11


Nüwa Approach

• Two phases– Phase 1: Automatic script rewriting – Phase 2 (Leftovers): Resort to online updates– Our research focus is on script rewriting

• Variations– Standalone Nüwa: Offline patching of individual VM

images in emulated environments– Mirage-based Nüwa: Batched offline patching using the

Mirage VM image library

June 29, 2012

Wu Zhou, Peng Ning, Xiaolan Zhang, Glenn Ammons, Vasanth Bala, Ruowen Wang, "Always Up-to-date -- Scalable Offline Patching of VM Images in a Compute Cloud," in Proceedings of ACSAC 2010, December 2010.


Some Evaluation Results

• Standalone Nüwa– Base VM image

• 64-bit Ubuntu 8.04; 406 patches (collected on 10/26/2009)• 402 patches can be applied offline (99%)

– Failure cases• Mono-gac package and three others that depend on it

• Mirage-based Nüwa– 100 VM images based on 32-bit Ubuntu 8.04

• Using 100 randomly selected subsets of basic Ubuntu tasks– Top 8 security updates from Ubuntu Security Notices

• Ranked by Ubuntu popularity contest• All data collected on 01/18/2010

June 29, 2012 13


Some Evaluation Results (Cont’d)

• Performance gain by standalone Nüwa– About 4 times speedup

June 29, 2012 14

bash dbus libfreetype6 libpng12-0 libxml2 perl samba-common

udev Average0

20

40

60

80

100

120

16.5 14.9 18.6 17.8 16.626.2 26.8 23.1 21.3

70.2 72.6 75.1 75.480.2

97.3 101.9

88.2 89.0

Offline Online

Tim

e (in

sec

onds

)

* “Average” refers to the average of all 402 packages.


Some Evaluation Results (Cont’d)

• Additional speedup by Mirage-based Nüwa– Another 2 – 10 times

June 29, 2012 15

10 20 30 40 50 60 70 80 90 1000

2

4

6

8

10

12

Speedup of Mirage-based batch patching over one-by-one patching

dbusdashlibdbuslibfreetype6Libpng12-0libglib2.0-0libxml2udev

Number of images patched

Spee

dup


Hypervisor Integrity Services

June 29, 2012

HyperSentry [CCS ’10]


Example Service: HyperSentry

• Why HyperSentry?– Hypervisor is the highest privileged software– Compromise of hypervisor compromise of the system– Hypervisors cannot be blindly trusted

• Example #1: Xen owning trilogy [BlackHat 2008]• Example #2: VM Ware ESX 3.x

– 6/18/12: 67 Secunia advisories; 562 vulnerabilities; 7% Secunia advisories not patched

– Hypervisor's code base is growing More vulnerabilities?

June 29, 2012


HyperSentry

• HyperSentry– Stealthy and in-context measurement of hypervisor integrity

• Challenges– A fundamental problem

• How to measure the integrity of the highest privileged software?– Hypervisor has full control of the software system (most of

the time)• Scrubbing attacks• Tampering with the measurement agent• Tampering with the measurement results

– Relying on a higher privileged software goes back to the same problem

June 29, 2012

Computer Science

The HyperSentry Approach

• HyperSentry– A generic framework to stealthily measure the integrity of a

hypervisor in its context• Key ideas

– Allow the measurement software to gain the highest privilege temporarily

– Measurement is triggered stealthily • Scrubbing attacks

– Isolate measurement results from the hypervisor

June 29, 2012 ACNS 2012 19

Ahmed M. Azab, Peng Ning, Zhi Wang, Xuxian Jiang, Xiaolan Zhang, Nathan C. Skalsky, "HyperSentry: Enabling Stealthy In-context Measurement of Hypervisor Integrity,” in ACM CCS 2010, October 2010.


Case Study: Verifying the Integrity of Xen• Integrity measurement

– Code: SHA-1 hash of Xen's code; Control flow verification– Date: Detect unauthorized sharing of physical pages across guest VMs

• Performance (on IBM HS21 XM blade server)– End-to-end execution time: 35ms– Periodical measurement: Every 8s: 2.4%; every 16s: 1.3%

June 29, 2012


Isolated Execution Bypassing Hypervisor Control

June 29, 2012

SICE[CCS ’11]

Computer Science 22

VM VM VM

SICE: Strongly Isolated Computing Environment

June 29, 2012 ACNS 2012

Legacy Host(Hypervisor/OS)

Hardware

Virtualized Platform

VM VMVM/

WorkloadLegacy Host

(Hypervisor/OS)

Hardware

Virtualized Platform with SICE

SICE

Ahmed Azab, Peng Ning, Xiaolan Zhang, “SICE: A Hardware-Level Strongly Isolated Computing Environment for x86 Multi-core Platforms,” in Proceedings of ACM CCS 2011, October 2011.

Computer Science 23

Foundation of SICE

• System Management Mode (SMM)– x86 operating mode for system management functions– Single entry point: System Management Interrupt (SMI)– SMRAM: Isolated from the rest of system

• Not accessible by the system software (e.g., hypervisor)– AMD processors implementation

• Resizing the SMRAM at runtime• Separate SMRAM range for each CPU core

– Main challenges• SMM has escalated privileges• The CPU runs slowly and has limited functionality

June 29, 2012 ACNS 2012

Computer Science 24

Foundation of SICE (Cont’d)

• Trusted/Secure boot– Building the trust chain during system initialization

• C-RTM BIOS Boot loader Initial hypervisor/OS image

– Secure hardware extensions (e.g., the TPM)• Seals and authenticates the measurement output

– Main challenge• Trust cannot be sustained due to potential runtime attacks

June 29, 2012 ACNS 2012

Computer Science

• Implementation requirements1. Hardware: AMD processors2. BIOS: Load the SMI handler3. Legacy host: Provide the communication channel

Hardware

Legacy Host

SICE Architecture

25June 29, 2012 ACNS 2012

VM VM

Virtualized Platform with SICE

SMRAM

CPU, TPM, BIOS, etc…DMA Devices

Hypervisor/OS

APP

Isolated Environment

Security Manager (hypervisor)

Isolated Workload (VM)

SMI Handler(SICE) Trusted

Untrusted

Isolated

• Secure boot• Consists of 300 SLOC to:1. Maintain the isolation2. Initialize and attest to the

isolated environment

Provided by the userSpecial hypervisor:• Confines the isolated workload• Trusted by the host

Communication channel:•Providing hardware services


SICE Operations

• Operating modes– Time-sharing Mode (an intermediate step)– Multi-core Mode

• Remote attestation– The initial image of the isolated environment– Secure communication with remote users

June 29, 2012

Computer Science 27

Legacy Host

Time-sharing Mode

• Initialization– Secure boot– SMI to load the isolated workload

• Execution environment switching– SMI to trigger the isolated environment– Changing the saved CPU state– Changing the SMRAM memory range– Fresh CPU start in the new environment– SMI to return to the legacy host

• Termination

ACNS 2012

Hardware

Legacy Host SMRAMIsolated Env.

Workload Image

Security Manager

Image

BIOS/TPM

SMRAM

SMI Handler(SICE)

RunningWorkloadSecurity Manager

(Hypervisor)

SMISMI

June 29, 2012

Computer Science 28

Multi-core Mode

• Concurrent sharing of the hardware– Good utilization– One or more CPU cores are assigned to either

• The isolated environments• The legacy host

• Main challenges– Event isolation– Memory isolation

June 29, 2012 ACNS 2012

Computer Science 29

• General multi-core processor architecture

Core n

Multi-core Mode (Cont’d)

June 29, 2012 ACNS 2012

Memory Control Hub (North Bridge):• Configuration registers• L2 Cache• I/O registers

Core 1

Core 0:• Registers• MSRs • L-APIC• L1 Caches

Core n

Memory Control Hub (North Bridge):• Configuration registers• L2 Cache• I/O registers

Core 1

Core 0:• Registers• MSRs• L-APIC• L1 Caches

AMD processors:• Define the SMRAM

• SMM_BASE• SMM_MASK

Computer Science

Hardware

• Initialization– Secure boot– Loading the isolated workload

Multi-core Mode Operations

June 29, 2012 ACNS 2012 30

Legacy Host SMRAM

Security Manager

Workload Image

SMISMI

Handler(SICE)

Computer Science

Isolated CPU Core

SMRAMSMRAM Legacy Host

• Running the isolated environment: The isolated core– Changing saved CPU state

• E.g., page tables, interrupt descriptor, instruction & stack pointers– Changing the SMRAM memory range (password stored in SMRAM)– Fresh CPU start in the isolated environment

Hardware


June 29, 2012 ACNS 2012 31

Isolated Env.

Security Manager

Workload Image

SMRAM

SMI Handler(SICE)

Host CPU Core

Legacy Host

Security Manager

Isolated Workload

BIOS/TPM

SMI Handler (SICE)

Security Manager (Hypervisor)

Workload Image

Running IsolatedWorkload (VM)

Computer Science

Isolated CPU Core

SMRAM Legacy Host

• Running the isolated environment: The host core– Return to the legacy host

• No environment switching necessary

Hardware


June 29, 2012 ACNS 2012 32

Isolated Env. SMRAM

SMI Handler(SICE)

Host CPU Core

Legacy Host

Security Manager

Isolated Workload

BIOS/TPM

SMI Handler (SICE)

Security Manager (Hypervisor)

Workload Image

Running IsolatedWorkload (VM)

Computer Science 33

Multi-core Mode Event Isolation

• Event isolation – Prevent the legacy host and the isolated workloads send

events to each other– Events between cores: Inter-Processor Interrupts (IPI)

• Two types of IPIs– Maskable IPIs

• Can be blocked by recipient core’s APIC– Non-maskable IPIs

• Can be controlled by Global Interrupt Flag (GIF)• Clear GIF to ignore or hold all IPIs

June 29, 2012 ACNS 2012

Computer Science 34

Multi-core Mode Event Isolation (Cont’d)

• Protecting the host core– The security manager runs as a thin hypervisor

• Prevents the isolated workload from privileged hardware access

• Protecting the isolated core– The security manager

• Clear GIF• All IPIs are disabled

– The isolated workload• Set the GIF and re-enable maskable IPIs• IPIs will cause a VM exit, which are examined by the security

manager for processing

June 29, 2012 ACNS 2012

Computer Science 35

Registers:•cr3MSRs:•SMM_Mask•SMM_Address

Multi-core Mode Memory Isolation

June 29, 2012 ACNS 2012

Host Core(s) Physical Memory

• Memory Isolation– Assigning different memory ranges to different CPU cores

• Memory double-view technique• Each CPU core has its own SMRAM

Registers:•Nested cr3MSRs:•SMM_Mask•SMM_Address

Isolated Core(s)

Security Manager

Isolated Workload

Set by the SMI handler

Set by the security manager

Shared Memory

SMI Handler

Computer Science 36

The SICE TCB

• The isolated environment– Hardware + BIOS + SMI handler (300 SLOC)

• The legacy host– Hardware + BIOS + SMI handler + The security manager– Similar to micro-hypervisor approaches

June 29, 2012 ACNS 2012

Computer Science 37

SICE Prototype & Evaluation• IBM LS 22 blade server• Modifying Qemu/KVM to support a SICE isolated Linux guest

– No disk emulation: RAM disk image• Time needed for context switching: 46 µs• Evaluation results

June 29, 2012 ACNS 2012


Conclusion

• Infrastructure security of cloud computing– Necessary for new research– Better security protection for cloud workloads

• Security architecture for cloud computing– Hypervisor-based runtime security services– VM image security services– Hypervisor integrity services– Isolated execution bypassing hypervisor control– Not necessarily complete

• Hopefully a guidance/framework for innovative ideas

• Stay relevant!!!

June 29, 2012


Questions?

Thank You!

June 29, 2012

Documents

Cloud Computing Infrastructure Security