Paris ACM Standards Tutorial Nov 2013

© Copyright Khronos Group 2013 - Page 1

Open Standards for Today’s Gaming Industry

Erik Noreke VP of Business Development, Khronos Group


The Topics for this Afternoon • Why do we need open standards for gaming and animation?

- What makes successful standards?

• Graphics Acceleration Standards

- Khronos acceleration API standards for low power and high performance gaming and

animation

• Bringing accelerated graphics into the Browser

- WebGL – revolutionary access to the GPU in standard browsers

• 3D Content Standards

- How to make content re-usable without stifling innovation


Why Do We Need Standards? • Standards are interoperability interfaces so that compelling user experiences

can be created inexpensively to build mass markets

- Don’t slow growth with functionality fragmentation that adds no value

• E.g. Wireless and IO standards

- GSM/EDGE, UMTS/HSPA, LTE, IEEE 802.11, Bluetooth, USB …

Standards drive mobile market

growth by expanding device

capabilities


Standards in the Real World

Vendor differences adding no value -

fragmentation is slowing growth – clear

goals emerge for a standard

REFINE BY COMMITTEE

Industry agrees on what to standardize

– cooperative refinement from multiple

viewpoints creates a robust solution

A good standard enables

implementation innovation

Darwinian industry is still

experimenting with what works

and what doesn’t

DESIGN BY COMMITTEE

Experimentation and design by

committee can be

slow and unfocused

A bad standard stifles innovation

and causes commoditization

Right time to Standardize?

Every successful open standard has a de facto proprietary

competitor and is open to competitive evolution

Ecosystems seem to work best when both are

healthy and evolving


Busting Some Standardization Myths • “Standards are slow to develop”

- Time to productive ecosystems is the key …

… rather than minimizing time to a proprietary specification

- OpenCL 1.0 took just 6 months – intensive cooperation

• “If I particpate in standards I ‘lose’ my IP”

- Good IP Framework fully protects Members IP and the specification

- Members agree not to assert IP claims against other Members or Adopters

- License grant is VERY narrow – but protects the specification in practice

• “Standards are boring”

- A good standard is the industry coming together to solve real issues


Khronos Connects Software to Silicon

ROYALTY-FREE, OPEN STANDARD APIs for

advanced hardware acceleration

Low level silicon to software interfaces needed on every platform

Graphics, video, audio, compute,

vision, sensor and camera processing

Defines the forward looking roadmap for

the silicon community

Shipping on billions of devices across

multiple operating systems

Rigorous conformance tests for

cross-vendor consistency

Khronos is OPEN for any company to

join and participate

Acceleration APIs BY the Industry

FOR the Industry


Courtesy of PC Gaming Alliance

www.pcgamingalliance.org


OpenCL – Heterogeneous Computing

• Native framework for programming diverse

parallel computing resources

- CPU, GPU, DSP – as well as hardware blocks(!)

• Powerful, low-level flexibility

- Foundational access to compute resources for

higher-level engines, frameworks and languages

• Embedded profile

- No need for a separate “ES” spec

- Reduces precision requirements

A cross-platform, cross-vendor standard for

harnessing all the compute resources in an SOC

OpenCL

Kernel

Code

OpenCL

Kernel

Code

OpenCL

Kernel

Code

OpenCL

Kernel

Code

GPU

DSP

One code tree can be executed on

CPUs, GPUs, DSPs and hardware.

Dynamically interrogate system load

and load balance work across

available processors

CPU

CPU HW


OpenCL Overview • C Platform Layer API

- Query, select and initialize compute devices

• Kernel Language Specification

- Subset of ISO C99 with language extensions

- Well-defined numerical accuracy - IEEE 754 rounding with specified max error

- Rich set of built-in functions: cross, dot, sin, cos, pow, log …

• C Runtime API

- Runtime or build-time compilation of kernels

- Execute compute kernels across multiple devices

• Memory management is explicit

- Application must move data from

host global local and back

- Implementations can optimize data movement

in unified memory systems


OpenCL SPIR 1.2 Provisional released!

OpenCL Roadmap

OpenCL 2.0

Significant enhancements to memory and execution models to

expose emerging hardware capabilities and provide increased

flexibility, functionality and performance to developers

OpenCL SPIR (Standard Parallel Intermediate Representation)

LLVM-based, low-level Intermediate Representation for IP Protection and as

target back-end for alternative high-level languages

OpenCL HLM (High Level Model)

High-level programming model, unifying host and device execution environments through

language syntax for increased usability and broader optimization opportunities

OpenCL 2.0 Provisional released!


Mobile OpenCL Shipping • Android ICD extension released in latest extension specification

- OpenCL implementations can be discovered and loaded as a shared object

• Multiple implementations shipping in Android NDK

- ARM, Imagination, Vivante, Qualcomm, Samsung …


OpenGL 3D API Family Tree

OpenGL ES 1.0

OpenGL ES 1.1 OpenGL ES 2.0 OpenGL ES 3.0

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

OpenGL 1.5 OpenGL 2.0 OpenGL 4.3 OpenGL 2.1

OpenGL 3.0

OpenGL 3.1

OpenGL 3.2

OpenGL 3.3

OpenGL 4.0

OpenGL 4.1

OpenGL 4.2

2002

OpenGL 1.3

ES-Next

GL-Next

OpenGL ES 2.0

Content OpenGL ES 1.1

Content

OpenGL ES 3.0

Content

ES3 is backward compatible

so new features can be

added incrementally Fixed function

3D Pipeline

Programmable vertex

and fragment shaders

WebGL 1.0

OpenGL 4.4 is a

superset of DX11

WebGL-Next

Desktop 3D

Mobile 3D

OpenGL 4.4


OpenGL ES 3.0 Highlights • Better looking, faster performing games and apps – at lower power

- Incorporates proven features from OpenGL 3.3 / 4.x

- 32-bit integers and floats in shader programs

- NPOT, 3D textures, depth textures, texture arrays

- Multiple Render Targets for deferred rendering, Occlusion Queries

- Instanced Rendering, Transform Feedback …

• Make life better for the programmer

- Tighter requirements for supported features to reduce implementation variability

• Backward compatible with OpenGL ES 2.0

- OpenGL ES 2.0 apps continue to run unmodified

• Standardized Texture Compression

- #1 developer request!


DirectX 11.1

2004 2006 2008 2009 2010 2005 2007 2011

Accelerating OpenGL Innovation

DirectX 10.1

OpenGL 2.0 OpenGL 2.1 OpenGL 3.0

OpenGL 3.1

DirectX 9.0c DirectX 10.0 DirectX 11

OpenGL 3.2

OpenGL 3.3/4.0

OpenGL 4.1

Bringing state-of-the-art functionality to cross-platform graphics

2012

OpenGL 4.2

OpenGL 4.4

2013

OpenGL 4.3


OpenGL 4.3 Compute Shaders • Execute algorithmically general-purpose GLSL shaders

- Can operate on uniforms, images and textures

• Process graphics data in the context of the graphics pipeline

- Easier than interoperating with a compute API IF processing ‘close to the pixel’

• Standard part of all OpenGL 4.3 implementations

- Matches DX11 DirectCompute functionality

Physics AI Simulation Ray Tracing Imaging Global Illumination


OpenGL Conformance Test Suite released!

Conformance submissions are required for GL 4.4 implementations encouraged for earlier driver versions

Shared codebase with OpenGL ES 3.0 CTS additional desktop-specific tests

Core profile functionality

Enhancements underway to add more coverage


OpenCL and OpenGL Compute Shaders • OpenGL compute shaders and OpenCL support distinctly different use cases

- OpenCL provides a significantly more powerful and complete compute solution

Enhanced 3D

Graphics apps

“Shaders++”

Pure compute

apps touching

no pixels

Compute Shaders

1. Full ANSI C programming of

heterogeneous CPUs and GPUs

2. Utilize multiple processors

3. Precisely defined IEEE accuracy

1. Fine grain compute operations

inside OpenGL

2. GLSL Shading Language

3. Execute on single GPU only

Imaging

Video

Physics

AI


Active Gamers by Platform




Visual Sensor Revolution • Single sensor RGB cameras are just the start of the mobile visual revolution

- IR sensors – LEAP Motion, eye-trackers

• Multi-sensors: Stereo pairs -> Plenoptic array -> Depth cameras

- Stereo pair can enable object scaling and enhanced depth extraction

- Plenoptic Field processing needs FFTs and ray-casting

• Hybrid visual sensing solutions

- Different sensors mixed for different distances and lighting conditions

• GPUs today – more dedicated ISPs tomorrow?

Dual Camera LG Electronics

Plenoptic Array Pelican imaging

Capri Structured Light 3D Camera PrimeSense


OpenVX • Vision Hardware Acceleration Layer

- Enables hardware vendors to implement

accelerated imaging and vision algorithms

- For use by high-level libraries or apps

• Focus on enabling real-time vision

- On mobile and embedded systems

• Diversity of efficient implementations

- From programmable processors, through

GPUs to dedicated hardware pipelines

Open source sample

implementation

Hardware vendor

implementations

OpenCV open

source library

Other higher-level

CV libraries

Application

Dedicated hardware can help make vision

processing performant and low-power enough

for pervasive ‘always-on’ use


OpenVX - Power Efficient Vision Acceleration • Create vision processing graph for power and performance efficiency

- Each Node can be implemented in software or accelerated hardware

- Nodes may be fused by the implementation to eliminate memory transfers

• EGLStreams can provide data and event interop with other APIs

- BUT use of other Khronos APIs are not mandated

• VXU Utility Library provides efficient access to single nodes

- Open source implementation – easy way to start using OpenVX

OpenVX Node

OpenVX Node

OpenVX Node

OpenVX Node

Heterogeneous

Processing

Native

Camera

Control


OpenVX and OpenCV are Complementary

Governance Open Source

Community Driven No formal specification

Formal specification and full conformance tests

Implemented by hardware vendors

Scope Very wide

1000s of functions of imaging and vision Multiple camera APIs/interfaces

Tight focus on hardware accelerated functions for mobile vision Use external camera API

Conformance No Conformance testing

Every vendor implements different subset Full conformance test suite / process

Reliable acceleration platform

Use Case Rapid prototyping Production deployment

Efficiency Memory-based architecture

Each operation reads and writes memory Sub-optimal power / performance

Graph-based execution Optimized nodes and data transfer

Highly efficient


Typical Imaging Pipeline • Pre- and Post-processing can be done on CPU, GPU, DSP…

• ISP controls camera via 3A algorithms

Auto Exposure (AE), Auto White Balance (AWB), Auto Focus (AF)

• ISP may be a separate chip or within Application Processor

Pre-processing Image Signal Processor

(ISP)

Post-

processing

CMOS sensor

Color Filter Array

Lens

Bayer RGB/YUV

App

Lens, sensor, aperture control 3A

Need for advanced

camera control API!


Camera Control API Goals • Provide functional portability for advanced camera applications

- Reduce extreme fragmentation for ISVs wanting more than point and shoot

• Generate image bursts with parameterized camera control and ISP control

- For downstream processing by flexible combination of CPU, GPU and DSP

• Control multiple sensors with multi-sensor synch and alignment

- Stereo pairs, Plenoptic arrays, Depth Cameras

• Enable system-wide sensor time-stamping

- Synchronize MEMS and image sensor samples

• This functionality is not available on any current platform APIs

- Make this API align with future platform direction for easy adoption


Camera API Design Philosophy • C-language API starting from proven designs

- e.g. FCAM, Android camera platform

• Design alignment with widely used hardware standards

- e.g. MIPI CSI

• Focus on mobile, power-limited devices

- But do not preclude other use cases such as automotive, surveillance, DSLR…

• Minimize overlap and maximize interoperability with other Khronos APIs

- But other Khronos APIs are not required

• Provide support for vendor-specific extensions

Apr13

Jul13

Group charter approved

4Q13

Provisional specification

1Q14

First draft specification

2Q14

Sample implementation and

tests

3Q14

Specification ratification


Low Power Environment Scanning • Many sensor use cases would consume too much power to be running 24/7

- Environment aware use cases have to be very low power

• ‘Scanners’ - very low power, always on, detect things in the environment

- Trigger the next level of processing capability

ARM 7 1 MIP and accelerometers can

detect someone in the vicinity

DSP Low power activation of camera

to detect someone in field of view

GPU GPU acceleration for precision

gesture processing


Sensor Industry Fragmentation …


StreamInput Sensor Fusion Stack

OS Sensor OS APIs (E.g. Android SensorManager or

iOS CoreMotion)

Low-level native API defines access to

fused sensor data stream and context-awareness

…

Applications

Sensor Sensor

Sensor

Hub Sensor

Hub

StreamInput implementations

compete on sensor stream quality,

reduced power consumption,

environment triggering and context

detection – enabling sensor

subsystem vendors to increased

ADDED VALUE

Middleware (E.g. Augmented Reality engines,

gaming engines)

Platforms can provide

increased access to

improved sensor data stream

– driving faster, deeper

sensor usage by applications

Middleware engines need platform-

portable access to native, low-level

sensor data stream

Mobile or embedded

platforms without sensor

fusion APIs can provide

direct application access

to StreamInput

Hardware transport

interfaces are defined

by each system, e.g.

IIO or HID sensor


Khronos APIs for Virtual Reality

Advanced Camera Control and stream

generation

3D Rendering and Video

Composition

On GPU

Audio

Rendering

Application

on CPUs, GPUs

and DSPs

Sensor

Fusion

Feature

Tracking

MEMS

Sensors

Camera Control

API

EGLStream Stream frames between APIs

Precision timestamps

on all sensor samples

VR needs not just advanced sensor processing, vision

acceleration, computation and rendering - but also for

all these subsystems to work efficiently together


Worldwide Game Users - 2012

0

100

200

300

400

500

N.America Europe Asia Other

WORLDWIDE GAME USERS BY REGION AND PLATFORM (IN MILLIONS)

Console PC Dedicated Handheld Mobile




Leveraging Proven Native APIs into HTML5 • Khronos and W3C liaison

- Leverage proven native API investments into the Web

- Fast API development and deployment

- Designed by the hardware community

- Familiar foundation reduces developer learning curve

Native APIs shipping

or Khronos working group

JavaScript API shipping,

acceleration being developed

or work underway

WebVX? Vision

Processing

WebCAM(!) Camera

control and

video

processing

Possible future

JavaScript APIs or

acceleration

WebStream? Sensor Fusion

Native

JavaScript Canvas

Path Rendering

Camera

Control

HTML


Content

JavaScript, HTML, CSS, ...

WebGL Implementation Anatomy

JavaScript Middleware

HTML5

JavaScript CSS

Browser provides WebGL functionality

alongside other HTML5 technologies

- no plug-in required

OS Provided Drivers. WebGL on Windows

can use Google Angle to create conformant

OpenGL ES 2.0 over DX9

OpenGL ES 2.0 OpenGL

DX9/Angle

Content downloaded from the Web.

Middleware can make WebGL accessible to

non-expert 3D programmers

http://code.google.com/p/angleproject/


WebGL Availability in Browsers

- Microsoft – “where you have IE11, you have WebGL – turned on by default and working all the time” - Microsoft - WebGL also enabled for Windows applications - web app framework and web view - Apple - WebGL must be explicitly turned on MAC Safari and only exposed on iOS for iAds - Chrome OS - WebGL is the only cross-platform API to program the GPU - Google IO announcement - Chrome on Android will soon launch with WebGL

Much WebGL content uses three.js library:

http://threejs.org/

http://threejs.org/

http://threejs.org/


C/C++

SDK Dalvik (Java)

Objective C C#

DirectX

HTML/CSS HTML/CSS HTML/CSS

Cross-OS Portability

HTML5 provides cross

platform portability. GPU

accessibility through

WebGL available soon on

~90% mobile systems

Preferred development

environments not

designed for portability

Native code is portable-

but apps must cope with

different available APIs

and libraries


WebGL First Wave Application Categories • Maps and Navigation

• Modeling Tools and Repositories

• Games

• 3D Printing

• Visualization

• Music Videos and Promotion

• Education

• Photo Editors

• Music Visualizers

• Vision/Video Processing


Google Maps • All rendering (2D and 3D) in Google Maps uses WebGL


WebCL – Parallel Computing for the Web • JavaScript bindings to OpenCL APIs

- Enables initiation of Kernels written in OpenCL C within the browser

• Bindings stay close to the OpenCL standard

- Maximum flexibility to provide a foundation for higher-level middleware

- Minimal language modifications for 100% security and app portability - E.g. Mapping of CL memory objects into host memory space is not supported

• API definition underway – public draft released - https://cvs.khronos.org/svn/repos/registry/trunk/public/webcl/spec/latest/index.html

• Compelling use cases

- Physics engines for WebGL games

- Image and video editing in browser River Trail

JavaScript binding to OpenCL APIs to enable initiation of OpenCL C kernels

Data-parallel language extensions to JavaScript

for offload at run-time

https://cvs.khronos.org/svn/repos/registry/trunk/public/webcl/spec/latest/index.html




3D Needs a Transmission Format! • Compression and streaming of 3D assets becoming essential

- Mobile and connected devices need access to increasingly large asset databases

• 3D is the last media type to define a compressed format

- 3D is more complex – diverse asset types and use cases

• Needs to be royalty-free

- Avoid an ‘internet video codec war’ scenario

• Eventually enable hardware implementations of successful codecs

- High-performance and low power – but pragmatic adoption strategy is key

Audio Video Images 3D

MP3 H.264 JPEG ? !

An effective and widely adopted codec ignites previously

unimagined opportunities for a media type


glTF Goals • Binary file format for efficient transmission for 3D assets

- Reduce network bandwidth and minimize client processing overhead

• Run-time neutral - DO NOT IMPLY OR MANDATE ANY RUN-TIME BEHAVIOR

- Can be used by any app or run-time – usually WebGL accelerated

• Scalable to handle compression and streaming

- Though baseline format does not include compression

• ‘Direct load efficiency’ for WebGL

- Little or NO processing to drop glTF data into WebGL client

• Carry conditioned data from any authoring format

- Prototyping and optimizing efficient handling of COLLADA assets

A standards-based

content pipeline for

rich native and Web 3D

applications Playback Authoring


COLLADA and glTF Open Source Ecosystem

Tool Interop

Three.js glTF Importer. Rest3D initiative

COLLADA2GLTF

Translator

OpenCOLLADA

Importer/Exporter

and COLLADA

Conformance Tests

On GitHUB

Pervasive WebGL deployment

Other

authoring

formats

Web-based Tools

https://github.com/KhronosGroup/glTF

https://github.com/KhronosGroup/OpenCOLLADA

https://github.com/KhronosGroup/COLLADA-CTS

https://github.com/KhronosGroup/glTF

https://github.com/KhronosGroup/OpenCOLLADA





Conclusion • Gaming and Animation is a complex and expanding application domain and

multiple standards across multiple domains are needed to enable the market

• Advances in SOC silicon processing and associated APIs are about to enable

Generated Reality to truly meet user expectations on all platforms

• Now is a great time to get involved with the standards initiatives

that effect your business

• www.khronos.org

• [email protected]

http://www.khronos.org/

mailto:[email protected]

Technology

Paris ACM Standards Tutorial Nov 2013