Lars Kurth - svlug. · PDF fileProduct Companies, Open Source & Commercial Distros (e.g. Huawei, Debian) Service/Cloud Providers (e.g. Alibaba) ... ABIs for PV Sound, PV Display &

Presentations on www.slideshare.net/xen_com_mgr/presentations

Lars KurthCommunity Manager, Xen Project

Chairman, Xen Project Advisory Board

Director, Open Source, Citrix lars_kurth

Cloud1. Virtual Machine Introspection (with demo)

2. Vulnerability Management In Xen Project

3. Live Patching (with demo)

Embedded & Automotive5. Why Virtualize Embedded Systems

8. Additional Security Properties of Xen

9. Xen Project In Security Applications

10. Xen Project in Embedded and Automotive (with demo)

12. Conclusion

Bonus Material: Embedded & Automotive (Slide 49)6. Hypervisor Architectures on ARM

7. PV Drivers and Protocols for Embedded Use-Cases

11. Schedulers and interrupt latency

Bonus Material: Security (Slide 67)4. Assessing a FOSS Project’s Security Record

www.xenproject.org

www.xenserver.org

A new way to protect against malware

Developed by Zentific, Citrix, Bitdefender, Intel and others

VM3

Guest OS

App

VMn

Guest OS

App

VM2

Guest OS

App

Dom0

Dom0 Kernel

Drivers Agent(s) Agent(s) Agent(s)

Installed in-guest agents, e.g. anti-virus software,

VM disk & memory scanner, network monitor, etc.

Can be disabled by rootkits

Several

VM3 VMnVM2Dom0

Dom0 Kernel

Drivers

VM3

Guest OS

App

VMn

Guest OS

App

VM2

Guest OS

App

Security

Appliance

VM1

IntrospectionEngine

Protected area

authentication mechanism to protect the IF

Uses HW extensions to monitor memory (e.g. Intel EPT) Low Intrusion

Register rules with Xen to trap on and inspect suspicious activities

(e.g. execution of memory on the dynamic heap)

All malware need an attack technique to gain a footholdAttack techniques exploit specific software bugs/vulnerability

Most exploits use one of a small set of attack techniques Buffer Overflows, Heap Sprays, Code Injection, API Hooking, …

Because VMI protects against attack techniquesIt can protect against entirely new malware

Verified to block these advanced attacks in real-timeAPT28, Energetic Bear, DarkHotel, Epic Turla, Regin, ZeuS, Dyreza, EternalBlue… solely by relying on VMI

WannaCry/EternalBlue blocked in real installations1

1 businessinsights.bitdefender.com/hypervisor-introspection-defeated-enternalblue-a-priori

Rootkits Exploit 0-days in Operating Systems/System SoftwareCan disable agent based security solutions (mask their own existence)

VMI solutions operate from outside the VMThus, it cannot be disabled using traditional attack vectors

BUT:VMI is not a replacement, for traditional security solutionsIt is an extra tool that can be used to increase protection

Pratap Sankar @ Flickr

Documentationwiki.xenproject.org/wiki/Virtual_Machine_Introspection

Products

Bitdefender HVIXenServer

www.bitdefender.com

Protection & Remedial

Monitoring & Admin

Citrix Ready

Zentific ZazenXen & XenServer & …

www.zentific.com

Protection & Remedial

Monitoring & Admin

Forensics & Data gathering

Malware analysis

AIS IntrovirtXenServer

www.ainfosec.com


https://www.youtube.com/watch?v=qpQPBvOniUU

Result of several community consultations

R: Vulnerability reported to [email protected]

P: Vulnerability pre-disclosed on [email protected]

R P

Fixing Security Bugs:

Dedicated security team =

security experts from within

the Xen Project Community

Security Team:

Triage

Creation of fix/patches

Validation of fix/patches

Assignment of CVE

Issue description and risk analysis

AR P

Fix their systems/software:

Eligible Xen Project Users

are informed under embargo

of the vulnerability

Eligible Users = Pre-disclosure list members:

Product Companies, Open Source & Commercial Distros (e.g. Huawei, Debian)

Service/Cloud Providers (e.g. Alibaba)

Large Private Downstream (e.g. Google)

Allowed to share information via

xen-security-issues- [email protected]



A: Vulnerability announced on [email protected] & xenbits.xen.org/xsa

A XR P

General Publication:

Information about

vulnerability is made public

Everyone else:

Patches their systems either through

security updates from distros/products or

builds them from source.

Users of service/cloud providers will

not be impacted




A XR P




Product Vendors:

Create and test live patches

Product customers:

Apply live patches here

Service Providers:

Create, test and deploy live patches

Users of service/cloud providers will

not be impacted

Security Team:

Can a Livepatch can be created?

No? If possible, re-write fix/patches

A tale of close collaboration withinthe Xen Project Community

const char *xen_extra_version(void)

{

return XEN_EXTRAVERSION;

}

push %rbp

mov %rsp,%rbp

lea 0x16698b(%rip),%rax

leaveq

retq

const char *xen_extra_version(void)

{

return “Hello World”;

}

push %rbp

mov %rsp,%rbp

lea 0x29333b(%rip),%rax

leaveq

Retq

Replacing compiled functions with new code, encoded in an ELF file called

payload, while the hypervisor is running without impacting running guests.

Design: xenbits.xenproject.org/docs/unstable/misc/livepatch.html

The exact source tree used to

build the running Xen instance.

The .config from the original

build of Xen.

A build-id onto which the

livepatch will be applied.

A source patch.

livepatch-

build-

tools

The exact same

compilation toolchain

used to build the

running Xen.

Livepatch

payload

Supports stacking of different payloads; payloads depend on build-id

Functionality:

list: lists loaded and applied live patches

upload: load & verify a live patch

unload: unload a live patch

apply: apply a live patch

revert: un-apply a live patch Xen 4.8.1

XSA 213

XSA 214

XSA 215

Depends on

build-id of 4.8.1

Depends on

build-id of XSA 213

Depends on

build-id of XSA 214

Target

Dom0 &

Guest

Linux

Kernel

Hypervisor

Technology

Kernel

Live Patching

kPatch

(RedHat)

Xen

LivePatch

kSplice

(Oracle)

kGraft

(SUSE)

Function +

Data

✔

✔

✔ Xen 4.7

✔

✔

Inline f()

patching

✗

✗

✗ Future

✔

✗

Data

Structures

✗

✔ via hooks

✔

✔

✗

Xen 4.8

via hooks

XenServer

LivePatch

Integrates

different solutions

into a single user

experience

For Dom0

(CentOS)

For Xen

Source patches

+ other

build artifacts

Hot Fixes containPer valid patch level: a Xen or Dom0 Live Patch

Matching RPMs for most recent patch level In case of a reboot or for Xen/Dom0 not capable of Live Patching

Extensive Verification and Validation:

The process of patching a live hypervisor or kernel

is not an easy task. What happens is a little bit like open

heart surgery. The patient is the hypervisor and/or Dom0

itself, and precision and care are needed to get things right.

One wrong move and it is game over.

Live

Patch

Live

Patch

Live

Patch

buildfor each

patch level

package

Hot Fix

LPsRPMs

RPMRPMRPM

buildfor most recent

patch level

Hot Fix

LPsRPMs

Publication

SigningValidation

Verification

Q&A

(livepatch-build

or kpatch-build)

(iso)

XAPI

Toolstack

Hot Fix

LPsRPM

downloadHot Fix

LPsRPM

Dom0

Dom0 Kernel(CentOS)

Hypervisor

XenCenter

or xe

Initiates

host update

SysAdmin

Running System

instance that supports

live patching

Disk

updates(such that

after reboot

the patches

are applied)

works out

correct LP

& updates

(using native

live patching

tools or APIs)


xenbits.xenproject.org/people/larsk/LCC17 - Build LivePatch.mp4xenbits.xenproject.org/people/larsk/LCC17 - Apply LivePatch.mov


Xen Project LivePatch Specification & Statusxenbits.xenproject.org/docs/unstable/misc/livepatch.htmlwiki.xenproject.org/wiki/LivePatch

Xen Project LivePatch Presentations & Videosxenbits.xenproject.org/people/larsk/FOSDEM17-LivePatch.pdf (Short)people/larsk/XPDS16-LivePatch.pdf (Long)

Xen Project LivePatch Videosfosdem.org/2017/schedule/event/iaas_livepatxen/

XenServerxenserver.org

ConsolidationReduce cost, size, weight, power consumption

and heat emission (already an issue today in cars)

CAN bus scalability / Architecture complexity

Reduce development costs: platform independence

Security and SafetySeparate safety critical apps from general apps

Safety Certification of the Hypervisor

Embedded Requirements (Bonus)Minimal IRQ latency

Low or 0 scheduling overhead

Drivers for special I/O devices

Flexible architecture

System Partitioning Sandboxing drivers & system components

Fine-grain control of VM capabilities

Enables multi-layered security approach

Other Security FeaturesTrusted Execution Environment (TEE)

Virtual Machine Introspection, alt2pm

Live Patching

Guest Kernel

Xen Project Hypervisor

Driver Domain Guest OS*: Linux, BSD, MiniOS, unikernel, …

DiskController

Guest Kernel*

Storage Domain

Disk Driver

Guest Kernel*

Network Domain

Network Driver

NetworkController

BlockFront Driver BlockBack Driver

Dom0

Kernel

Application

NetFront Driver NetBack Driver

Attack Surface Reduction

Similar to Linux Security Modules/SELinux

Same policy syntax as SELinux

Different types, roles, users and attributes

Same tools for policy compilation / verification (checkpolicy)

VM

hypervisor domain(self) domain(other) memory (grant, mmu, shadow)

inter-VM communicationpassthroughsecurity config

Fine-grained policy, controlling

which hypervisor functionality is

accessible to this (class of) VM

Effect: limit what an exploit in

this VM could do


Crucible:Defensestarlab.io

Xen Project based virtualization

platform for technology protection,

cyber-hardening, and system integrity

for aerospace & defense systems

Documentationwiki.xenproject.org/wiki/Dom0_Disaggregationwiki.xenproject.org/wiki/Xen_Security_Modules_:_XSM-FLASK

Products & Projects

Qubes OSwww.qubes-os.org

Secure OS

OpenXTwww.openxt.org

FOSS Platform for security research,

security applications and embedded

appliance integration building on

Xen & OpenEmbedded

User defined App VMs for individual

apps or groups of apps

USB

Service

Domain

Banking

Domain

Personal

Domain

Firewall VMenforces network

policies

Network

Domain

Dom0

Secure UI and

sysadmin domain


AISainfosec.com

BAE Systems

baesystems.com

Galoisgalois.com

Maintain FreeRTOS Xen Port

Developed and maintain HalVM

Dornerworksdornerworks.com/xen

Consulting

Xen Embedded Distros

Xen for Xilinx Zynq

Xen for NXP i.MX 8

ARLX HypervisorDO-178 (EAL6+), IEC 62304, ISO 26262

MILS EAL

FACE, VICTORY, ARINC 653

Starlabstarlab.io

Crucible and Crucible:Defense

Xen embedded hypervisorIn progress: DO-178, MILS EAL

Uses a minimal Dom0 using

MiniOS, disaggregation and

XSM/FLASK

Precedents of military grade certification for Xen based systems

www.slideshare.net/xen_com_mgr/art-certification & www.youtube.com/watch?v=UyW5ul_1ct0

xenbits.xenproject.org/people/larsk/XPDS14 - Xen and the Art of Certification.pdf

www.linux.com/news/xen-project/2017/2/how-shrink-attack-surfaces-hypervisor


LG ElectronicsDemobit.do/lg-xen-demo-2016

Bosch Car GmbHContributions10 smaller features in 2016

Perseus (stealth)Founded by Xen maintainerbit.do/perseus-2017

Demo at IAA 2017 in Frankfurt, Germany

ADITJoint venture Bosch & DENSOLittle known at this stage

GlobalLogicProduct: Nautilusbit.do/gl-nautilus

First product in production

expected in Q1 2018

Supports:HW: Renesas R-Car Gen2 & Gen3,

TI Jacinto6, Intel Apollo Lake, Qualcomm

410C, Sinlinx A33

Guests: Linux up to 4.9 Android M, N,

N-Car QNX, ThreadX, FreeRTOS

PV Drivers for: GPU, Audio, HW

accelerated Video codecs, DRM, …

Contributions:27 smaller features from 2013 to 2016

EPAMDemoNext slide

Interesting Features:Container based telematics applications

running in a Xen VM that can be

downloaded from a cloud service

Ongoing Contributions:ABIs for PV Sound, PV Display & PV DRM

Leading development of co-processor

sharing framework


xenbits.xenproject.org/people/larsk/

LCC17 - The Internet of Transportation[1080P].MP4

AWS

Dom0 - Control DomD – HW Drivers & Cluster

Wayland/Weston

OpenGL ES

Linux Kernel with GPU and other HW Drivers

ALSA wPV_ALSAS_BE

DomU Fusion

Container mgmt tool

Linux Kernel w/oHW Drivers

Minimal rootfswith systems

library

Telematics simulation Agent (Acceleration, Braking, Corning, GPS)

DomU – Linux IVI

MW Frameworks

PVDISPLAY

Linux Kernel with GPU and w/o other HW Drivers

PVEVENTS

PVSOUND

IVI Simulation App Trusted Apps

TrustZone

Hypervisor

R-Car H3 Platform

OP-TEE OS

TZ monitor

Driver Behavior Based Insurance Backend

Telematics Simulation Agent ver 2.0

Telematics Simulation Agent ver 1.0

Monitoring Dashboard

Wayland BE(Events/Display)

Cluster Simulation AppDom0 Services

Minimal rootfs

Linux Kernel w/o HW Drivers

Containers

Picture by Lars Kurth

Xen Project & Security in Cloud:

Only Hypervisor with VMIProtection from new classes of malwareSeveral security companies working with XenServer

Live PatchingDisruption free application of vulnerabilitiesUsed by several cloud providersUsed best in commercial products, e.g. XenServer

Industry Leading Vulnerability ProcessIncludes QEMU and Kernel XSAs


Xen Project in Embedded & Automotive:

Extremely Flexible and VersatileProven in many different marketsEasy to port to new environmentsEasy to develop new PV drivers (see bonus 2)Highly customizable

Security and ResilienceIsolation, Partitioning, Security Features

SafetyExamples of Military Grade CertificationBUT: looking at ways to make this easier and cheaper

Challenges still being addressedStandardization of more I/O devices via PV protocolsStandardization of GPU and co-processor sharingRTOS or other minimal OS as Dom0Testing of embedded Hardware by the project


www.slideshare.net/xen_com_mgr/presentations



EL0/PL0 least privileged mode used for applications (user mode)

EL1/PL1 privileged mode used for running kernels such as the Linux kernel

EL2/PL2 This has a higher level of privilege and can be used to run a hypervisor which takes control

of the system and can host multiple "guest" operating systems

EL2

EL1

EL0

Guest Kernel

Guest Kernel

Guest

Userspace

Guest

Userspace

Host

Userspace

Host Kernel + Hypervisor Native DDs

Type 2 with VHE/ARMv8.1 (e.g. KVM)

Guest Kernel

Guest Kernel

Guest

Userspace

Guest

Userspace

Guest

Userspace

Guest Kernel

Hypervisor

Traditional Embedded Type 1 Hypervisor

Native DDs

EL1

EL2

EL0

Guest Kernel

Guest Kernel

Guest

Userspace

Guest

Userspace

Host Kernel + Hypervisor Native DDs

Type 2 with VHE/ARMv8.1 (e.g. KVM)

Guest Kernel

Guest Kernel

Host

Userspace

Guest

Userspace

Guest

Userspace

Hypervisor

Guest Kernel

Native DDs

Traditional Embedded Type 1 Hypervisor

Host

Userspace

EL2

EL1

EL0


Guest Kernel

Guest Kernel

Guest

Userspace

Guest

UserspaceStrong Isolation Device Drivers run in EL1,

not EL2

Protected Address Spaces:

Grant tables

Trusted Computing

Base (TCB)

Dom0

Kernel

Native DDs

Dom0

Userspace

Toolstack

EL1

EL2

EL0


Guest Kernel

Guest Kernel

Guest

Userspace

Guest

Userspace

Control PlaneServer: sysadmin

Embedded: config/setup, system

health monitoring (watchdog),

maintenance, SW updates, …

Dom0

Kernel

Native DDs

Dom0

Userspace

Toolstack

Dom0

Kernel

Guest Kernel

HWI/O

Native Driver


*Back Driver *Front Driver

Application

Existingnet, block, consolekeyboard, mouse, USBframebuffer, GPU sharing*

New in Xen 4.99pfs (share a filesystem between VMs)Pvcalls (forward POSIX calls across VMs)multitouch, sound, display, DRM

Developing New OnesEasy to write (GPL and BSD samples)Kernel and User Space

*) A number of different approaches by different vendors in different market

segments are being deployed, which are PV-like, but not strictly a PV

protocol

Xen supports several different schedulers with different properties.

Xen supports several different schedulers with different properties.

Regular VM

scheduler (Credit)Hard real-time

(ARINC653)

Dedicated to 1 VCPU via pinning and Null scheduler

no scheduler overheads

Soft real-time

(RTDS)

Scheduler Use-cases Today Future plans

Credit General Purpose Supported

Default

Supported

Optional

Credit 2 General Purpose

Optimized for lower latency, higher VM density

Supported Default

RTDS Soft & Firm Real-time

Multicore

Embedded, Automotive, Graphics & Gaming in

the Cloud, Low Latency Workloads

Experimental

Better XL support

<1μs granularity

Supported

Hardening

Optimization

ARINC 653 Hard Real-time

Single core

Avionics, Drones, Medical

Supported

Compile time

Null Hard Real-time Experimental Supported

vCPU 0

pCPU 0

vCPU 1

pCPU 1

irq 109

virq 109

IRQ injection

Always on the CPU running the vCPU

vCPU 0

pCPU 0

irq 109

virq 109

vCPU 1

pCPU 1

IF

vIRQ target changes or vCPU is moved

THEN

vIRQ is moved immediately

virq 109

vCPU 0

pCPU 0

vCPU 1

pCPU 1

irq 109

virq 109

IRQs always shadow the vIRQ

minimizes latency

Xilinx ZynqMP board

(four Cortex A53 cores, GICv2)

WARM_MAX (excluding the first 3 interrupts): <2000ns

Without Null scheduler

See blog.xenproject.org/2017/03/20/xen-on-arm-

interrupt-latency/

Developer Portal: bit.do/xen-devsXen on ARM whitepaper: bit.do/xenarm-whiteXen on ARM wiki: bit.do/xenarm-wiki

Port Xen to a new SOC: bit.do/xenarm-portingAdd Xen support Xen to your OS: bit.do/xenarm-os

Device Passthrough presentation: bit.do/xenarm-ptOE meta-virtualization Xen recipe: bit.do/xenmetaOpenXT (Xen + OpenEmbedded): openxt.orgXenbedded presentation: bit.do/xenbedded

Monthly ARM Community Call: bit.do/xenarm-call


Security Process Number of Vulnerabilities Media CoverageOther Considerations

A XR P

Responsible Disclosure: fix critical systems/software before publication

R: Vulnerability reported to security@...

P: Vulnerability pre-disclosed to eligible users

A: Vulnerability announced publicly

F: Fix available

Full Disclosure, immediate (no-fix): public disclosure without a fix

A XR F

A XR

Full Disclosure, post-fix: public disclosure with a fix

F

F

4) New members: http://seclists.org/oss-sec/2017/q2/6385) http://www.openwall.com/lists/oss-security or devel list6) https://wiki.qemu.org/index.php/SecurityProcess

Only handles x86 KVM bugs (no ARM or other bugs)

1) Is the CVE severity used to handle vulnerabilities differently?2) Days embargoed (information is classified)3) D = Distros/Products, S = Public Service, P = Private Downstream4) http://oss-security.openwall.org/wiki/mailing-lists/distros

Responsible only

Days 2 Who? 3FOSS Project Bug Severity 1 Process Type

14-19 D 4

Linux Kernel via

OSS-security distros 4 ≥ Medium – Critical Responsible Disclosure

14-19 D 4

QEMU (KVM) via

QEMU Security

Process 6≥ Medium – Critical

≤ Low

Responsible Disclosure

Full Disclosure, no-fix

3-5 D, S, POpenStack OSSA

OpenStack OSSN

≥ Medium – Critical

≤ Low


Full Disclosure, post-fix

Xen HypervisorIncludes Linux & QEMU

vulnerabilities in supported Xen

configurations

Low – Critical Responsible Disclosure 14 D, S, P

14-19

Linux Kernel via

OSS-security distros 4

OSS-security 5≥ Medium – Critical

≤ Low


Full Disclosure, no-fix

Impacts how long a user (aka you) is at riskIs my distro/vendor on a pre-disclosure list?

A surprisingly large number of distros are not: including a few on Linux.com’s Best Linux Distros of 2016 list

Impacts cloud / service providers As a user, are security issues fixed before public disclosure?

Low Severity vulnerabilities can still be High Risk Temporal and Environmental CVSS scores are not covered by CVE databases (neither cvedetails.com or nvd.nist.gov)

Vulnerabilities can be chained together, making the combo High Severity(e.g. Hot Potato used 3 old unpatched vulnerabilities to gain root access)

cvedetails.com (bit.do/guide-cvedetails)

Easy to use interface for vulnerability data

Data from several sources

Browsable by vendor, product, version, type, date…

Vulnerability statistics, trends, reports

BUT: rigid➜ getting data outside pre-defined vendor/product categories is near-impossible

vulners.com (good guides on slideshare.net – search for vulners)

In many ways more accurate and flexible than cvedetails

type:cve AND (description:kvm OR description:qemu) AND published: [2012 TO *] ➜307

type:cve AND (description:xen) AND published: [2012 TO *] ➜ 245

…

Works best when used through its API (in particular if you want to visualize the data)

Vulnerability data from vulners.com

0

50

100

150

200

250

300

350

2012 2013 2014 2015 2016 2017

Linux Kernel

KVM w. QEMU

Xen Project

*) Data up to Sept 4th, 2017

169

517

222

197

129

120

51

7

63

143

34

5

Legend: CVSS

Score Distribution

Low

Medium

High

Critical

*

Data covering September 2016 – September 2017: from vulners.com, mention.com and theregister.com

Clips covering

vulnerabilities

(US only)

% of Vulnerability

Stories on

The Register155370 314

7283

0

1000

2000

3000

4000

5000

6000

7000

8000

Xen KVM QEMU Linux

0

50

100

150

200

250

300

350

400

450

Xen KVM QEMU Linux

Number of

Vulnerabilities

0%

5%

10%

15%

20%

25%

30%

35%

40%

Xen KVM QEMU Linux

33% of Xen stories

cover Vulnerabilities

2.5%

16%

6.5%

Does the Project Look for Vulnerabilities?

Approach to Quality and Testing: e.g. Fuzzing, Audits of components

Does the project award Bug Bounties (e.g. NetBSD)?

Do vendors supporting the project offer Bug Bounties?

Infrastructure related to VulnerabilitiesTransparency: How well are processes documented

Vulnerability Testing: XTF (in Xen)

Vulnerability Tooling: XSATool, XSAMatch (in Xen)

Documents

Lars Kurth - svlug. · PDF fileProduct Companies, Open Source & Commercial Distros (e.g. Huawei, Debian) Service/Cloud Providers (e.g. Alibaba) ... ABIs for PV Sound, PV Display &