02.05.2006

Havard Bjerke

CERN Openlab

Virtualization experience with Xen

Overview

VM technologyOpenlab I

HistoryXen/ia64Deployment in the LCG testbed

Openlab IINew hardwareETICS, SmartfrogVirtualization in batch subsystem

Vision

Virtualization Technologies ­ Hosted

Microsoft Virtualization ServerUsed at CERN for consolidation

Runs MS Windows, Linux (SLC3, SLC4)Non­negligible CPU overhead: every ~ 3rd cycle wasted

HostedFreeAPI to control VMM and VMs32­bit, single CPU only

VMWareHosted: GSX

Non­negligible CPU overhead

Non­hosted: ESXLimited hardware support

Virtualization Technologies ­ Non­hosted

XenParavirtualization, non­hostedClose to native performanceSupports only paravirtualized OSs unless hardware­virtualized platform64­bit supportSMP supportOpen source, GPL

02.05.2006

Havard Bjerke

CERN Openlab

Openlab I

CPU performance benchmarks (Rune)

CPU: Dual Xeon 2.4 GhzBenchmark: test40forSPECOS: SLC3

Cluster benchmarks (with Rune)

Lower bandwidth and higher latency in guest domains.Aggregate bandwidth of multiple domains scales, but not latency.

Xen/ia64

X86 virtualization unfriendlyNon­faulting privilege sensitive instructions

IA64 a little more friendlyThree non­faulting privilege sensitive instructionsTagged TLB / Region registers

No need to flush the TLB when switching domainEasier to virtualize physical memory

No more segmentation, real mode, protected modeRedundant hypercalls

EFI – easier to virtualize than BIOS

Optimized paravirtualizationLinux is a moving target ­> minimize changes in the guest Linux kernelInstead: trap faulting instructions

Physical machine

LCG Deployment

Xen 2.0.7Proof of concept GRID­in­a­boxComplete LCG 2.6 installation

Computing Element (CE)Storage Element (SE)Worker Node (WN)

Successful deployment in LCG testbedCurrently used in test gridTests passed as good as native nodes

IssuesAutomatic software updates cause /lib/tls to be restoredSupport for 2.4 kernels dropped

Possible applicationsServer consolidation (GRID­in­a­box)Security (VO­box)Customizable environments (Openlab II)Availability, management flexibility (Live­migration)

LCG Deployment

02.05.2006

Havard Bjerke

CERN Openlab

Openlab IIFocused effort with Intel

New Hardware

MontecitoIA64VTiParavirtualization – OK

No support for FPSWA yet

Dempseyx86 EM64TVTxParavirtualization ­ OK

Smartfrog, ETICS

ETICS (Xavier)Testing environment

SmartfrogUtility computingProvide a single configuration file

MemoryHD capacitySoftware configuration...

Deploy a complete site – clean up afterwards

PBSResource scheduling independent from queues

LSFOne queue per VO

BLAHPCommon interface to batch subsystemsLet LSF do scheduling decisions based on time constraints

Virtualization in Batch Subsystems

time: 9hVO: CMS

Queue AVO = CMS

Queue Bt < 10h

?

Virtualization in Batch Subsystems

RequirementsCustomized execution environmentsIsolated execution environmentsDynamic resource management

Three goals over three phases1. Selection of static execution environments2. Dynamic instantiation of execution environments – on­demand3. Dynamic configuration of execution environments – VM factory

Virtualization in Batch Subsystems

Client/CE Routingqueue

Executionqueue

[max 1 job]

Executionqueue

[max 2 jobs]

One-way

WNSLC3WN

SLC4

Two-way

WNSLC3WN

SLC4

1. Static execution environments

Virtualization in Batch Subsystems

Client/CE Routingqueue

Executionqueue

[max 1 job]

Executionqueue

[max 2 jobs]

One-way

Two-way

2. Dynamic instantiation

WNSLC3WN

SLC4

WNSLC3WN

SLC4

Virtualization in Batch Subsystems

Client/CE Routingqueue

Executionqueue

[max 1 job]

Executionqueue

[max 2 jobs]

One-way

Strawberry

Two-way

Vanilla

Chocolate

3. Dynamic configuration

Pistachio

Vision

Close to native performanceWithout hardware support

XenVMWare ESX

With hardware supportMany to come

VM tech agnosticCasatt's XVM

User­supplied or ­specified execution environmentsDomain migration

Flexible resource managementHigh availability

Vision

Execution environment characteristicsIsolatedSecure

XenSESVM hardware extensions

Clean

02.05.2006

Havard Bjerke

CERN Openlab

Questions?Questions?

More info:More info:http://openlab-mu-internal.web.cern.ch/openlab-mu-internal/openlab-II_Projects/http://openlab-mu-internal.web.cern.ch/openlab-mu-internal/openlab-II_Projects/

Platform_Competence_Centre/Virtualization/Virtualization.aspPlatform_Competence_Centre/Virtualization/Virtualization.asp