©2014 GlobalLogic Inc.

GPU Virtualization in Embedded Automotive Solutions

Sergiy Kibrik

GlobalLogic Embedded TechTalkJuly 22, 2015

2

Why One Wants to Virtualize?Why One Wants to Virtualize?

VSVS

3

ARM implemented virtualization support on some of ARM v7 processorsARM implemented virtualization support on some of ARM v7 processors

4

Key Virtualization TargetsKey Virtualization Targets

• isolationisolation• parallelismparallelism• safetysafety• robustnessrobustness• performanceperformance

5

Common GPU Virtualization ApproachesCommon GPU Virtualization Approaches

OpenGL/DirectX/OpenCL etcOpenGL/DirectX/OpenCL etc API RemotingAPI Remoting

match:match: isolation, parallelism, safety isolation, parallelism, safety

no match: no match: robustness, performancerobustness, performanceon embedded systemson embedded systems

Vendors: Xen (VMGL), Chromium,Vendors: Xen (VMGL), Chromium, Android Emluator,Android Emluator, presumably MentorGraphics & RedBendpresumably MentorGraphics & RedBend

6

Common GPU Virtualization ApproachesCommon GPU Virtualization Approaches

Driver SplitDriver Split

Vendors: VMWare SVGA2Vendors: VMWare SVGA2

match:match: isolation, parallelism, safety isolation, parallelism, safety

no match: no match: robustness, performancerobustness, performanceon embedded systems on embedded systems + high cost+ high cost

7

Do It The Hard WayDo It The Hard WayTMTM

Full GPU Emulation!Full GPU Emulation!

Registers & VRAM Registers & VRAM virtualizedvirtualized

Every single one Every single one access is trapped & access is trapped & emulatedemulated

8

Common GPU Virtualization ApproachesCommon GPU Virtualization Approaches

Direct pass-through:Direct pass-through:

match: match: safety, performance,safety, performance,robustnessrobustness

no match: no match: isolation, parallelismisolation, parallelism

Vendors: NVIDIA GRID vGPUVendors: NVIDIA GRID vGPU

9

And the Right Approach :)And the Right Approach :)

match: match: isolation, parallelism,isolation, parallelism,safety, robustnesssafety, robustness

not quite match: not quite match: performanceperformance

Mediated pass-throughMediated pass-through

10

A bit of GPU Pipeline…A bit of GPU Pipeline…

11

……and what this stuff actually doesand what this stuff actually does

12

Real World Example: GeForce 8800 Real World Example: GeForce 8800 (2006)(2006)

13

Embedded Case: PowerVR Graphics PipelineEmbedded Case: PowerVR Graphics Pipeline

• tile-based renderingtile-based rendering• hidden surface removalhidden surface removal

14

PowerVR SGX5xx HardwarePowerVR SGX5xx Hardware

TI OMAP5770/DRA7/Jacinto6:TI OMAP5770/DRA7/Jacinto6:•2 SGX544 cores2 SGX544 cores•4 USSE processors each4 USSE processors each•4 pipes & 16 threads each4 pipes & 16 threads each•128 parallel threads128 parallel threads•runs OS: uKernelruns OS: uKernel

15

Key Virtualization Techniques: MMU Pass-ThroughKey Virtualization Techniques: MMU Pass-Through

• trap all MMU accessestrap all MMU accesses• introduce shadow PD/PTintroduce shadow PD/PT• hypercall to Xen for translationhypercall to Xen for translation

16

Key Virtualization Techniques: control hijackingKey Virtualization Techniques: control hijacking

• un-map IO registers for all un-map IO registers for all domainsdomains

• trap each register accesstrap each register access• if domain is active ‒ repeat if domain is active ‒ repeat

actionaction• remember action otherwiseremember action otherwise

17

Key Virtualization Techniques: Global (firmware) context switchKey Virtualization Techniques: Global (firmware) context switch

• we can inject commandwe can inject command• uKernel has power commanduKernel has power command• forward IRQ to active domain onlyforward IRQ to active domain only

18

Key Virtualization Techniques: VGPU SchedulingKey Virtualization Techniques: VGPU Scheduling

• GPU tasks non-preemptibleGPU tasks non-preemptible• long context switch timelong context switch time• GPU scheduled separatelyGPU scheduled separately• simple FIFO scheduler for nowsimple FIFO scheduler for now

19

20

Similar Approach: XenGTSimilar Approach: XenGT

• our inspirationour inspiration• has some kind of independent channelshas some kind of independent channels• Intel onlyIntel only• heavily abstractedheavily abstracted

21

Some Performance IllustrationSome Performance Illustration

TestTest RateRate

GL Benchmark EgyptGL Benchmark Egypt 22.5 fps22.5 fps

NenaMark2NenaMark2 27.3 fps27.3 fps

22

Further Improvements:Further Improvements:• virtualization support in uKernel (VE)virtualization support in uKernel (VE)• smart GPU schedulersmart GPU scheduler

Materials:Materials:• Kun Tian , Yaozu Dong , David Cowperthwaite, A full GPU virtualization solution with mediated pass-Kun Tian , Yaozu Dong , David Cowperthwaite, A full GPU virtualization solution with mediated pass-

through, Proceedings of the 2014 USENIX conference on USENIX Annual Technical Conference, June through, Proceedings of the 2014 USENIX conference on USENIX Annual Technical Conference, June 19-20, 2014, Philadelphia, PA19-20, 2014, Philadelphia, PA

• Yusuke Suzuki, Shinpei Kato, Hiroshi Yamada, and Kenji Kono. 2014. Yusuke Suzuki, Shinpei Kato, Hiroshi Yamada, and Kenji Kono. 2014. GPUvm: why not virtualizing GPUs GPUvm: why not virtualizing GPUs at the hypervisor?. at the hypervisor?. In In Proceedings of the 2014 USENIX conference on USENIX Annual Technical Proceedings of the 2014 USENIX conference on USENIX Annual Technical ConferenceConference (USENIX ATC'14), Garth Gibson and Nickolai Zeldovich (Eds.). (USENIX ATC'14), Garth Gibson and Nickolai Zeldovich (Eds.). USENIX Association, USENIX Association, Berkeley, CA, USA, 109-120. Berkeley, CA, USA, 109-120.

• Micah Dowty and Jeremy Sugerman. 2008. GPU virtualization on VMware's hosted I/O architecture. In Micah Dowty and Jeremy Sugerman. 2008. GPU virtualization on VMware's hosted I/O architecture. In Proceedings of the First conference on I/O virtualizationProceedings of the First conference on I/O virtualization (WIOV'08). (WIOV'08). USENIX Association, Berkeley, CA, USENIX Association, Berkeley, CA, USA, 7-7USA, 7-7

• NVIDIA, “Fermi: The First Complete GPU Computing Architecture”, 2009NVIDIA, “Fermi: The First Complete GPU Computing Architecture”, 2009• J. Nickolls and D. Kirk, ‘‘Graphics and Computing GPUs,’’ Computer Organization and Design: The J. Nickolls and D. Kirk, ‘‘Graphics and Computing GPUs,’’ Computer Organization and Design: The

Hardware/Software Interface, D.A. Patterson and J.L. Hennessy, 4th ed., Morgan Kaufmann, 2009, pp. Hardware/Software Interface, D.A. Patterson and J.L. Hennessy, 4th ed., Morgan Kaufmann, 2009, pp. A2-A77.A2-A77.

• Imagination, “PowerVR Series5 Architecture Guide for Developers”, 2014Imagination, “PowerVR Series5 Architecture Guide for Developers”, 2014• Imagination, “PowerVR Hardware Architecture Overview for Developers”, 2015Imagination, “PowerVR Hardware Architecture Overview for Developers”, 2015• H.AndrésLagar-Cavilla, “VMGL: VMM-Independent Graphics Acceleration”, undatedH.AndrésLagar-Cavilla, “VMGL: VMM-Independent Graphics Acceleration”, undated• OpenSynergy, “Shared GPU using Virtualization”, undatedOpenSynergy, “Shared GPU using Virtualization”, undated

©2014 GlobalLogic Inc.

Thank You