SBSA UART Emulation in Xen on 64-bit ARM Server Processor

Bhupinder Thakur

Principal Engineer

Qualcomm Datacenter Technologies

Xen Developer and Design Summit

July 11-13 2017, Budapest Hungary

▪ Working as a Linaro assignee from Qualcomm Datacenter Technologies

▪ Working on VM spec compliance for ARM based servers

▪ First open source contribution

▪ SBSA UART emulation in Xen

About The Speaker

Challenges in implementing ARM as a new ISA

▪ ARM processors are built into a large variety of systems

▪ Feature requirements vary from chip to chip

▪ This variability in the architecture impacts the software system development

▪ x86 relies on PC standards

▪ Challenge: How to provide software portability across ARMsystems ?

Enabling ARM as a First Class Architecture

▪ SBSA (Server Base System Architecture)

▪ It defines minimum requirements from the hardware such as CPU, PCIe, IOMMU, UART, timers and interrupt

▪ SBBR (Server Base Boot Requirements)

▪ Firmware must be UEFI/ACPI compliant

▪ UEFI for server booting

▪ ACPI for describing the hardware resources that are installed

▪ Any compliant software is guaranteed to run on any SBSA/SBBR compliant hardware

▪ The specifications can be downloaded from

▪ http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.den0029/index.html

Challenges in Enabling VM System for ARM Processors

▪ SBSA/SBBR specs were defined for the physical systems

▪ Similar standardization was required for the virtual systems▪ How should the VM bring up the guest OS ?

▪ How should the guest OS discover the devices ?

▪ What are mandatory peripherals to be supported by the hypervisor ?

▪ Linaro published a specification to standardize the requirements from the hypervisor/OS/firmware ▪ http://www.linaro.org/app/resources/WhitePaper/VMSystemSpecificationForARM-v2.0.pdf

▪ Virtual Firmware

▪ UEFI compliant virtual firmware

▪ UEFI binary must be executed as the first software by the VM which then executes the EFI application

▪ VM Image Format

▪ The raw disk format must be partitioned with a GUID table

▪ The bootable software must reside in a EFI partition and must be an EFI application

▪ Hardware Description

▪ The virtual UEFI firmware must implement ACPI and DT tables

VM System Specification for ARM Processors

▪ Mandatory Hypervisor Peripherals

▪ SBSA UART

▪ Support for both GICv2 and GICv3 (Generic Interrupt Controller)

▪ Virtual timer and counter system register must be available to the VM

▪ Virtual bus that supports block and network devices hotplug

▪ The guest OS

▪ must include console driver for SBSA UART

▪ must include support for both GICv2 and GICv3

▪ strongly recommended to include support for block, network, console, balloon, virtio and Xen PV drivers

VM System Specification for ARM Processors

SBSA UART

▪ A simple UART which offers basic facility for software bring up

▪ It is a subset of ARM PL011 UART

▪ Supports only a subset of registers to handle data, status and interrupts

▪ No support for features such as DMA, hardware flow control etc.

▪ It is defined in Appendix B of the SBSA specification

Benefits of Emulating SBSA UART in Xen

▪ Compliant with the VM specification

▪ The guest OS can access the console even if it does not include PV console driver

▪ Can be used for debugging if PV console is not working for some reason

Options Available for Emulating SBSA UART

▪ Emulating it in Dom0 using QEMU

▪ Emulating it in the Xen hypervisor

Challenges in Emulating SBSA UART in Xen

▪ Xen/ARM does not use QEMU

▪ Typically devices are not emulated in Xen/ARM (GIC is an exception)

Current Design for Emulating SBSA UART

App App

Kernel

DomU

XencosoleApp

Kernel

Dom0

EL0

EL1

EL2SBSA UART Emulation Logic

▪ Emulate SBSA UART in the Xen hypervisor

Current Design for Emulating SBSA UART

App App

Kernel

DomU

XencosoleApp

Kernel

Dom0

EL0

EL1

EL2SBSA UART Emulation Logic

▪ Emulate SBSA UART in the Xen hypervisor

▪ Reuse the PV console backend to minimize the changes in Dom0

Current Design for Emulating SBSA UART

App App

Kernel

DomU

Xenco

soleApp

Kernel

Dom0

Xen

EL0

EL1

EL2

SBSA

UART

Emulation

Logic

▪ Emulate SBSA UART in the Xen hypervisor

▪ Reuse the PV console backend to minimize the changes in Dom0

▪ Future: Move emulation to Xen non-privileged (EL0) mode

Xen

DomUDom0

User I/O

Events

IN

OUT

PV Console in Xen

Implemenation of SBSA UART Emulation in

Xen

DomUDom0

User I/OMMIO Traps

IN ring buffer

OUT ring buffer

Device Tree

SBSA UART

emulation

IRQ

Events

DOMCTL

▪ Changes in the Xen hypervisor

▪ MMIO trap handling

▪ IRQ injection

▪ Event handling

Xen

Implementation of SBSA UART Emulation in

Xen

DomUDom0

User I/OMMIO Traps

IN ring buffer

OUT ring buffer

Device Tree

SBSA UART

emulation

IRQ

Events

DOMCTL

▪ Changes in the toolstack

▪ New DOMCTL API to enable emulation in

Xen

▪ New SBSA UART node added in the

xenstore

▪ New SBSA UART node added to the

device treeXen

Implementation of SBSA UART Emulation in

Xen

Device Tree ▪ Changes in Xenconsole

▪ Multiple console support added

▪ New option “vuart” added in DomU.cfg

file to enable sbsa uart for a domain

▪ xl console command enhanced to allow

the user to connect to vuart

DomUDom0

User I/OMMIO Traps

IN ring buffer

OUT ring buffer

SBSA UART

emulation

IRQ

Events

DOMCTLXen

Using SBSA UART

Sample DomU.cfg file

vuart = “sbsa_uart”

extra = “console=ttyAMA0 …”

Start the domain with SBSA UART enabled

xl create domU.cfg

Connect to SBSA UART

xl console –t vuart <domain name>

TBD Items

▪ SBSA UART support in UEFI firmware▪ Currently UEFI firmware uses PV console for input/output

▪ Add support for SBSA UART

▪ How should the guest OS select the default console ?▪ Currently PV console is the default console

▪ Add SBSA UART support in ACPI

Upstream Status

▪ The SBSA UART patch set is under review on the xendev mailing list

▪ It has gone through five revisions

▪ The changes have matured over these revisions

▪ Hopefully it should be included in 4.10

▪ The code is available @ https://git@git.linaro.org:/people/bhupinder.thakur/xen.git branch: vpl011_v5

Thank You!

Xenconsole Changes

Handle_ring_read()

DomU

Xen

PV Ring buffer

write and event

raised

PL011 ring buffer

write and event

raised

Buffer_append()PV buffer

SBSA UART

buffer

Handle_tty_write()

Write to PV

tty client

Handle_tty_read()

DomU

Xen

PV Ring buffer write and

event raised

SBSA UART Ring

buffer write and event

raised

Read from PV tty

client

Read from PV tty

client

PV console logs SBSA UART

console logs

console_create_ring()

console_create_tty()

create_console_log()

console_cleanup()

…

Write to

SBSA UART

tty client