Camera BOF - khronos.org BOF SIGGRAPH 2013 © Copyright Khronos Group 2013 - Page 2 ... Imaging systems that are situationally aware (Camera, Mobile, PC) • Face Detection/Track

© Copyright Khronos Group 2013 - Page 1

Camera BOF SIGGRAPH 2013


Cameras are Everywhere

Interactive Systems that respond

to user actions

(PC, Gaming, Mobile)

• Motion/Gesture tracking and

recognition

• Body tracking

Environmental Imaging systems that

are situationally aware

(Camera, Mobile, PC)

• Face Detection/Track

• Gesture tracking

• Object tracking

Ubiquitous “Cameras Everywhere”

Distributed Systems

(Mobile, Camera, DIY-SOHO)

• Point and shoot

• HDR

• Surveillance

http://www.trendbird.co.kr/attach/1/1360703205.png

http://images.google.com/imgres?imgurl=http://i.ehow.com/images/GlobalPhoto/Articles/4518947/cell-phone-camera-400x400-93k-main_Full.jpg&imgrefurl=http://www.ehow.com/how_4861892_better-photos-cell-phone-camera.html&usg=__5kkBssCijxKk-D1GbZs6-bePTe0=&h=336&w=336&sz=21&hl=en&start=40&sig2=5lv2ruZxqWssiWRfIqGvLw&itbs=1&tbnid=7o44lminxL17AM:&tbnh=119&tbnw=119&prev=/images?q=cell+phone+camera+user&gbv=2&ndsp=21&hl=en&sa=N&start=21&ei=R9ZdS_W8CpfGtAPUpsmZAg


Computer Imaging Evolution

Spatial

(Volumetric)

Gesture

AR

Face Detect

Face Track

Presence

Colorimetry

Brightness

Web Cam

Smart

Camera

True Color, Brightness

Compensation, Exposure control

User Identity

Access Control

Augmented Information

3D Imaging

Interactive

Services

http://images.google.com/imgres?imgurl=http://thetopblog.org/wp-content/uploads/images/080516-sdi.jpg&imgrefurl=http://thetopblog.org/2008/05/samsung-sdi-have-a-prototype-of-the-laptop-with-the-12-inch-amoled-display/&usg=__pNr14l-cc15pNS8mYZLTSl1xbOc=&h=384&w=450&sz=27&hl=en&start=100&sig2=ntFaZj5riOfxyvtJt_rzIA&tbnid=9Jc5aP3xL5MenM:&tbnh=108&tbnw=127&ei=ZAK4Sa2gDJKMsQObuqnzAQ&prev=/images?q=laptop+screen&gbv=2&ndsp=21&hl=en&sa=N&start=84

http://images.google.com/imgres?imgurl=http://demo.pittpatt.com/front_page/webcam_tracker.jpg&imgrefurl=http://demo.pittpatt.com/&usg=__RTnfKxYYPQSMuJdCnF97xAmCGWw=&h=398&w=300&sz=48&hl=en&start=23&sig2=bwpWCm2EkoO71NkOYzfQzQ&tbnid=opRKx5rSWU0qzM:&tbnh=124&tbnw=93&prev=/images?q=face+tracking&gbv=2&ndsp=21&hl=en&sa=N&start=21&ei=xplfSsSsIo_GlAfU5tWdDg

http://images.google.com/imgres?imgurl=http://francisanderson.files.wordpress.com/2008/12/mini_augmented_reality.jpg&imgrefurl=http://francisanderson.wordpress.com/2008/12/&usg=__TzNYAEJtomu4BY5gaXsgh5bcmhk=&h=383&w=520&sz=49&hl=en&start=12&sig2=VvNW3gnBcB5-MQ25RDX0sQ&tbnid=62hRl-vpmrrKrM:&tbnh=96&tbnw=131&prev=/images?q=Augmented+reality&gbv=2&hl=en&ei=oppfStKwLsfRlAe4zejQCQ

http://images.google.com/imgres?imgurl=http://www.infotech.oulu.fi/Annual/1999/pics/VIRGINhand.jpg&imgrefurl=http://www.infotech.oulu.fi/Annual/1999/VIRGIN.html&usg=__u1hGDACqOga3lqRDyO2hFbDPR24=&h=864&w=1152&sz=142&hl=en&start=9&sig2=w43CXOJhzOFc3u1dwE9CJg&tbnid=oLW98k1i_4GupM:&tbnh=112&tbnw=150&prev=/images?q=gesture+recognition&gbv=2&hl=en&ei=aJxfSumlHMuwlAeB2YDJCQ

http://images.google.com/imgres?imgurl=http://www.biometricvisions.com/images/face1.jpg&imgrefurl=http://www.biometricvisions.com/technology/technology.htm&usg=__BIoDwbLtqXEoUpWsZCMvZOOWLjI=&h=400&w=297&sz=43&hl=en&start=1&sig2=q9q-i5e8sIFFSLJdBeTpcg&tbnid=Czm4wl0WEIP_bM:&tbnh=124&tbnw=92&prev=/images?q=face+identification&gbv=2&hl=en&ei=H55fSt3KBNTKlAeVz6HRCQ

http://images.google.com/imgres?imgurl=http://blog.360dgrs.nl/wp-content/uploads/3dbaseballjpg.jpeg&imgrefurl=http://blog.360dgrs.nl/archives/category/new-media&usg=__CYVQ7l76QssqkKhXmmLLvt0-qFQ=&h=384&w=504&sz=55&hl=en&start=16&sig2=OvYjoOPtUUwBWl2yOuxzaw&tbnid=G21QBzaH6y-XwM:&tbnh=99&tbnw=130&prev=/images?q=augmented+reality+games&gbv=2&hl=en&ei=sZ9fSvrQGcy3lAeDqdjUCQ

http://images.google.com/imgres?imgurl=http://news.cnet.com/i/bto/20080218/shockwave_540x283.bmp&imgrefurl=http://news.cnet.com/8301-10784_3-9871126-7.html&usg=__rpEiSCaqw-4HRCfk9rPt5LOe_ks=&h=283&w=540&sz=448&hl=en&start=61&sig2=f_2BfImI7t_y8ng-uhZhYg&tbnid=lg_kEIZSMX3EMM:&tbnh=69&tbnw=132&prev=/images?q=3d+imaging&gbv=2&ndsp=21&hl=en&sa=N&start=42&ei=i6FfStOTCZHtlAes1sziCQ

http://images.google.com/imgres?imgurl=http://www.informationkiosk.co.nz/images/main_top_image.jpg&imgrefurl=http://www.informationkiosk.co.nz/&usg=__Q9aRCF-SZ4AFAagRDbAgIBggXb0=&h=370&w=385&sz=23&hl=en&start=17&sig2=y3YtvT8UT2odlv82sAUb1g&tbnid=n-PM7EHsrua9BM:&tbnh=118&tbnw=123&prev=/images?q=touch+kiosk&gbv=2&hl=en&ei=F8dfSqPIOpCxsAaQtKSjBw


Mobile Compute Driving Imaging Use Cases • Requires significant computing over large data sets

Augmented Reality

Face, Body and Gesture Tracking

Computational Photography

3D Scene/Object Reconstruction

Time


How Imaging Sensors Work

http://www.photoaxe.com

Bayer GRBG pattern

• 50% green

• 25% red and blue

Bayer CFA is one type

of pattern


Typical Imaging Pipeline

• Pre-processing is non-existent in basic use-cases

• Pre- and Post-processing can be done on CPU, GPU, DSP…

• ISP controls camera via 3A algorithms

Auto Exposure (AE), Auto White Balance (AWB), Auto Focus (AF)

Pre-processing Image Signal Processor

(ISP)

Post-

processing

CMOS sensor

Color Filter Array

Lens

Bayer RGB

YUV

App

Lens, sensor, aperture control


Typical Image Signal Processor (ISP)

• Simplified Pipeline for illustration purpose

• Non programmable

• Some (re-entrant) pipeline may accept YUV input too

• May be part of a SoC or a separate processor

Lens correction

Bad pixel correction White Balance

Noise

reduction Demosaic

Color

correction Tone mapping Sharpening

Gamma

correction

RGB to YUV Scaler RGB YUV

Bayer

3A

Statistics

• Statistics

- For AF, AE, AWB

- For adaptation over time of

pipeline’s filters


Pipelined Sensor Model • Traditional one-shot sensor model

- Need to know which parameters were used

- reset pipeline between shots slow

• Viewfinding / video mode:

- Pipelined, high frame rate

- Settings changes take effect later

• Need new model for Computational

Photography

- Need parameterized SEQUENCE of images

to feed advanced algorithms

• Real image sensors are pipelined

- While one frame exposing

- Next one is being prepared

- Previous one is being read out


High Dynamic Range (HDR) • HDR works by combining differing exposures into the same image

• A variety of methods for HDR, based on application

- Multiple frame HDR (requires frame memory)

- Interlace HDR

- Multiple Zone HDR

• HDR requires

- Precise control over camera parameters (exposure)

- Fast capture and processing of multiple images

- Note: with interlace HDR, only 1 image is needed

Short

exposure Long

exposure

Optional mid

exposure

HDR processing


Image stitching, panoramic images

• Made with


Image stitching, panoramic images

• Made with

• Requires processing of multiple images

• Requires position / geometry

information

• Requires control of camera (e.g. AE lock)


Typical Burst Sequence Applications


Some Existing APIs • FCAM (2010) Stanford/Nokia, open source

- Capture of stream of camera images with precision control - A pipeline that converts requests into image stream

- All parameters packed into the requests - no visible state

- Programmer has full control over sensor settings for each frame in stream

- Control over focus and flash - No hidden daemon running

- Control ISP - Can access supplemental

statistics from ISP if available

• Android New Camera HAL (2013)

- Extends these concepts


Camera, Sensor Fusion and Vision Acceleration


Increasing Use of Imaging Sensors D

iffe

renti

ati

on O

pport

unit

y

Time

Photography Input = 2D Camera

Processors = ISP + CPU Product = Static Images

Computational Photography Input = MEMS + 2D Camera

Processors = ISP + CPU + GPU Product = Real Time Images

We are here

Perceptual Imaging Input = MEMS + Depth Camera

Processors = ISP + CPU + GPU + DSP Product = Real Time Extracted Information

Perceptual Imaging 1. Uses the full array of mobile sensors 2. to extract information in real-time 3. about the user and environment

4. to generate enhanced user interactions


Low Power Environment Scanning • Many sensor use cases would consume too much power to be running 24/7

- Environment aware use cases have to be very low power

• ‘Scanners’ - very low power, always on, detect things in the environment

- Trigger the next level of processing capability

ARM 7 1 MIP and accelerometers can

detect someone in the vicinity

DSP Low power activation of camera

to detect someone in field of view

GPU GPU acceleration for precision

gesture processing


StreamInput Sensor Fusion Stack

OS Sensor OS APIs (E.g. Android SensorManager or

iOS CoreMotion)

Low-level native API defines access to

fused sensor data stream and context-awareness

…

Applications

Sensor Sensor

Sensor

Hub Sensor

Hub

StreamInput implementations

compete on sensor stream quality,

reduced power consumption,

environment triggering and context

detection – enabling sensor

subsystem vendors to increased

ADDED VALUE

Middleware (E.g. Augmented Reality engines,

gaming engines)

Platforms can provide

increased access to

improved sensor data stream

– driving faster, deeper

sensor usage by applications

Middleware engines need platform-

portable access to native, low-level

sensor data stream

Mobile or embedded

platforms without sensor

fusion APIs can provide

direct application access

to StreamInput

Hardware transport

interfaces are defined

by each system, e.g.

IIO or HID sensor


Visual Sensor Revolution • Single sensor RGB cameras are just the start of the mobile visual revolution

- IR sensors – LEAP Motion, eye-trackers

• Multi-sensors: Stereo pairs -> Plenoptic array -> Depth cameras

- Even stereo pair can enable object scaling and enhanced depth extraction

- Plenoptic Field processing needs FFTs and ray-casting

• Hybrid visual sensing solutions

- Different sensors mixed for different distances and lighting conditions

• GPUs today – more dedicated ISPs tomorrow?

Stereo Camera LG Electronics

Plenoptic Array Pelican imaging

Capri Structured Light 3D Camera PrimeSense


OpenVX • Vision Hardware Acceleration Layer

- Enables hardware vendors to implement

accelerated imaging and vision algorithms

- For use by high-level libraries or apps

• Focus on enabling real-time vision

- On mobile and embedded systems

• Diversity of efficient implementations

- From programmable processors, through

GPUs to dedicated hardware pipelines

Open source sample

implementation

Hardware vendor

implementations

OpenCV open

source library

Other higher-level

CV libraries

Application

Dedicated hardware can help make vision

processing performant and low-power enough

for pervasive ‘always-on’ use


OpenVX Power Efficiency • OpenVX Graph for power and performance efficiency

- Each Node can be implemented in software or accelerated hardware

- Nodes may be fused by the implementation

- Eliminates transferring the image to and from memory

• EGLStreams can provide data and event interop with other APIs

- BUT use of other Khronos APIs are not mandated

• VXU Utility Library provides efficient access to single nodes

- Open source implementation – easy way to start using OpenVX

OpenVX Node

OpenVX Node

OpenVX Node

OpenVX Node

Heterogeneous

Processing

Native

Camera

Control


OpenVX and OpenCV are Complementary

Governance Open Source

Community Driven No formal specification

Formal specification and full conformance tests

Implemented by hardware vendors

Scope Very wide

1000s of functions of imaging and vision Multiple camera APIs/interfaces

Tight focus on hardware accelerated functions for mobile vision Use external camera API

Conformance No Conformance testing

Every vendor implements different subset Full conformance test suite / process

Reliable acceleration platform

Use Case Rapid prototyping Production deployment

Efficiency Memory-based architecture

Each operation reads and writes memory Sub-optimal power / performance

Graph-based execution Optimized nodes and data transfer

Highly efficient


Khronos APIs for Augmented Reality

Advanced Camera Control and stream

generation

3D Rendering and Video

Composition

On GPU

Audio

Rendering

Application

on CPUs and

GPUs

MEMS

Sensor Fusion

Feature

Tracking and SLAM

MEMS

Sensors

Camera Control

API

EGLStream Stream frames between APIs

Precision timestamps

on all sensor samples


Advanced Camera Control Requirements • Generate image bursts with parameterized control

- For processing by flexible combination of CPU, GPU, ISP and DSP

• Control of new wave sensors – multi-sensor synch and alignment

- Stereo pairs, Plenoptic arrays

- Depth Cameras: Time-of-flight and Structured light

• Enable system-wide sensor time-stamping

- Synchronize MEMS and image sensor samples

• Enable data routing with and to Khronos ecosystem APIs

- EGLStreams for efficient routing

- OpenVX, OpenCL, OpenGL etc. for processing


Camera Control API


Camera Control API Goals • Provide functional portability for advanced camera applications

- Reduce extreme fragmentation for ISVs wanting more than point and shoot

• Generate image bursts with parameterized camera control and ISP control

- For downstream processing by flexible combination of CPU, GPU and DSP

• Control multiple sensors with multi-sensor synch and alignment

- Stereo pairs, Plenoptic arrays, Depth Cameras

• Enable system-wide sensor time-stamping

- Synchronize MEMS and image sensor samples

• This functionality is not available on any current platform APIs

- Make this API align with future platform direction for easy adoption


Camera Control API

Pre-ISP

Processing (Bayer Space)

ISP

Post-ISP

Processing (YUV Space)

Application

Camera Control API - Controls flash, lens and sensor

- Controls ISP

Compute/Image/Vision

Acceleration APIs

Sensor Lens

Flash

Downstream

Processing


Camera API Design Philosophy • C-language API starting from proven designs

- e.g. FCAM, Android camera platform

• Design alignment with widely used hardware standards

- e.g. MIPI CSI

• Focus on mobile, power-limited devices

- But do not preclude other use cases such as automotive, DSLR…

• Minimize overlap and maximize interoperability with other Khronos APIs

- But other Khronos APIs are not required

• Provide support for vendor-specific extensions

Apr13

Jul13

Group charter approved

4Q13

Provisional specification

1Q14

First draft specification

2Q14

Sample implementation and

tests

3Q14

Specification ratification


Participating Companies

True Multi-vendor

initiative including

sensor, processor and

platform vendors

http://www.google.com/

http://www.amd.com/

http://www.fujitsu.com/

http://www.antixlabs.com/

http://www.marvell.com/index.jsp


Khronos Camera Control Working Group • Royalty free API for portable access to advanced mobile camera functionality

- Reduce fragmentation and encourage more advanced camera applications

• Control for the new wave of sensors to enable advanced imaging and vision

- Multiple sensors, depth cameras, synchronized sensors

• Provide sophisticated camera functionality not available on today’s platforms

- But work to enable easy adoption by platform vendors

• Eager to contribute? Join Khronos Camera WG!

- Detailed design is just starting – great time for your input and requirements!

- http://www.khronos.org/camera

• Mikaël Bourges-Sévenier

- [email protected]

http://www.khronos.org/camera

mailto:[email protected]


Camera BOF SIGGRAPH 2013

Documents

Camera BOF - khronos.org BOF SIGGRAPH 2013 © Copyright Khronos Group 2013 - Page 2 ... Imaging systems that are situationally aware (Camera, Mobile, PC) • Face Detection/Track