Embed Size (px)
Citation preview
Porting Xen on ARM to a new SOC
Julien Grall <[email protected]>
Xen Developper Summit 2016
© ARM 2016
Xen Architecture
2 © ARM 2016
Virtualization
ARMv7 and ARMv8Provides virtualization for
TimerInterrupt ControllerPage Table
3 © ARM 2016
Virtualization - 2
4 © ARM 2016
Xen on ARM
5 © ARM 2016
Dom0
First guest to start
Known as the hardware domainNearly all devices are assigned to DOM0
Serial, IOMMU, Timer and GIC are used by XenSome devices can be blacklisted by Xen
DOM0 kernel should discover devices via ACPI or Device Tree
6 © ARM 2016
Groundwork
7 © ARM 2016
Preparation before porting
Before starting to port Xen, some groundwork needs to be done:
Check the hardware support
Having the firmware/bootloader to boot the image at EL2
Having an OS supporting the targeted platform
8 © ARM 2016
Hardware
ARMv7 and ARMv8 processor with virtualization extension
General Interrupt Controller (GIC) v2 or later
9 © ARM 2016
Firmware and bootloader
The firmware or bootloader must drop into EL2 (hypervisor) before starting Xen.Some vendors locked down the firmware/bootloader to drop into Non-secure EL1 (kernel mode).
Hypervisor Call instruction (HVC) must be enabled.It can be done by setting SCR EL3.HCE (AArch64) or SCR.HCE (AArch32) to 1.
10 © ARM 2016
Firmware tables
Xen supports the below firmware tables out-of-box:Device Tree
https://www.devicetree.org/
ACPI 6.0 and onwardshttp://www.uefi.org/acpi/specsUEFI onlyTechnical preview in Xen 4.7DOM0 with ACPI support has been merged for Linux 4.8
11 © ARM 2016
DOM0 kernel
Before adding Xen in the equation, it is highly recommended to get the kernel booting natively
DOM0 support is upstreamed in LinuxAdding support to any other kernel is easy
See http://www.slideshare.net/xen_com_mgr/bsdcan-2015-how-to-port-your-bsd
12 © ARM 2016
DOM0 kernel - Linux
DOM0 support has been added in Linux 3.8
It is recommended to use the latest release when possible
Minimal list of options to enable:
CONFIG_XEN_DOM0=yCONFIG_XEN=yCONFIG_XEN_BLKDEV_BACKEND=yCONFIG_XEN_NETDEV_BACKEND=yCONFIG_HVC_XEN=yCONFIG_XEN_BACKEND=yCONFIG_XENFS=yCONFIG_XEN_SYS_HYPERVISOR=y
13 © ARM 2016
Porting Xen
14 © ARM 2016
A single binary to rule them all
A single Xen binary can
be loaded via different methods (e.g multiboot, UEFI).
boot on multiple hardware.
15 © ARM 2016
It is recommended to use the latestversion of Xen when porting to a new
SOC.
16 © ARM 2016
Early debugging with Xen
Xen provides early printk to debug crash before the UART driver is initialized.
Only available when CONFIG DEBUG=y
Xen will not be portable, intented only for developmentUART selected on the build command line with
CONFIG EARL PRINTK=machCONFIG EARL PRINTK= INC>, BASE ADDRESS>, OTHER OPTIONS>
More details onhttp://xenbits.xen.org/docs/unstable/misc/arm/early-printk.txt
17 © ARM 2016
Early debugging with Xen - 2
Major UARTs supported: pl011, 8250,...8250: CONFIG EARL PRINTK=8250, BASE ADDRESS>, REG SHIFT>
REG SHIFT> is the left-shift to apply to register offsets within the uart (optional).
pl011: CONFIG EARL PRINTK=pl011, BASE ADDRESS>, BAUD RATE>BAUD RATE is optional. We recommend to let the bootloader setting the baud rate.
18 © ARM 2016
Getting the firmware to load Xen and DOM0
The firmware needs to load in memory Xen, DOM0 kernel and potentially others modules (e.ginitramfs, XSM...). There are 3 methods to do it:
Multiboot
UEFI
GRUB via UEFI (work in progress)
19 © ARM 2016
Loading Xen and DOM0 using multiboot
Multiboot is a protocol based on Device Tree.
It is used to describe where the kernel, initramfs... reside in memory.An example to generate multiboot nodes with U-Boot can be found on the wiki.
https://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions#Boot_Modules
20 © ARM 2016
Loading Xen and DOM0 via UEFI
On AArch64, Xen is built as an EFI application.A configuration file is used to describe:
The command lineThe binaries to load (device tree, kernel, initramfs...) in memory.
The configure file could be passed to the EFI application using the parameter -cfg=myxen.cfg
https://xenbits.xenproject.org/docs/unstable/misc/efi.html
21 © ARM 2016
Loading Xen and DOM0 via UEFI - Example
Configuration file example for UEFI:
[global]default=model
[model]options=console=dtuart conswitch=x dom0_max_vcpus=2 dtuart=serial0kernel=vmlinuz console=hvc0 earlycon=pl011,0xf2a00000 root=/dev/ram1 rootwaitramdisk=initrd.imgdtb=model.dtb
22 © ARM 2016
Troubleshooting - Xen is not entering in EL2
Xen will panic when it is not entered in EL2 with the following message:
For AArch32:
- Xen must be entered in Hyp mode -- Please update the bootloader -
For AArch64:
- Xen must be entered in NS EL2 mode -- Please update the bootloader -
23 © ARM 2016
What to do if Xen is not entered in EL2?
Even if the hardware supports virtualization extensions, the firmware/bootloader may be configuredto enter the kernel/hypervisor in EL1.
Find a version which dropped in EL2.
Find the source code and modify it to enter the hypervisor in EL2.
24 © ARM 2016
Platform specific code
In most of the case, platform specific code is not necessary.
Hooks in the core code is provided specific initialization is required.
Platform code resides in xen/arch/arm/platforms
25 © ARM 2016
Platform specific code - 2
List of callbacks available:/* Platform initialization */int (*init)(void);int (*init_time)(void);int (*smp_init)(void);int (*cpu_up)(int cpu);/* Specific mapping for dom0 */int (*specific_mapping)(struct domain *d);/* Platform reset */void (*reset)(void);/* Platform power-off */void (*poweroff)(void);/** Platform blacklist devices* List of devices which must not pass-through to a guest*/const struct dt_device_match *blacklist_dev;
26 © ARM 2016
UART support
Xen has multiple UART drivers (pl011, 8250,...).
They can be found in xen/drivers/char.
The UART used by Xen will not be available for DOM0.A virtual UART will shadow the real one.
Useful if the kernel use early printkVery basic: Only write is supported
The UART configuration can be read from:the parameter dtuart=cfgstdout-path in the device treeThe SCPR table in ACPI
27 © ARM 2016
Debugging DOM0 kernel
Xen consoleSwitch from DOM0 console to Xen console via CTLR-a three timesUseful key
0 Dump Dom0 vCPUsq Domains informatione Event channel informationR Reboot the machine
28 © ARM 2016
Using Xen debugging facilities in the kernel
Use of hvc 0xFFXX
Supported when Xen is compiled with debug=y
Requires to modify the kernel
0xFFEX0xFFFD0xFFFE0xFFFF
Print the register rX/xXPrint the program counterPrint the character stored in r0/x0Dump the state of the vCPU
29 © ARM 2016
SMP support
Xen is able to bring up secondary processors via different protocols:Power State Coordination Interface (PSCI)
This is the recommended protocol to bring up CPU.PSCI 0.1, 0.2 and 1.0 supportedIt can be used for rebooting the platform (PSCI >= 0.2).
Spin table (AArch64 only)Platform specific bringup (AArch32 only)
This protocol should be avoided in favor of PSCI.It can be implemented with the callcack smp init and cpu up.
30 © ARM 2016
Upstreaming
Even if your platform does not require platform specific code, it is recommended to
Document the step to boot Xen on the wiki.
Testing new release of Xen.
31 © ARM 2016
Questions?
33 © ARM 2016
The trademarks featured in this presentation are registered and/or unregistered trademarks of ARM limited
(or its subsidiaries) in the EU and/or elsewhere. All rights reserved. All other marks featured may be
trademarks of their respective owners.
Copyright © 2016 ARM Limited
© ARM 2016
