User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D...

Mantis Vision MV4D

A bit about me

USER GENERATED 3D CONTENT CREATION

Mantis Vision MV4D

Mobile Depth Sensing Methods What are they good for?

Mantis Vision MV4D

FACT: 3D is here. And it’s going mobile.

Mantis Vision MV4D

Apps are next. Content Creation is the key.

Mantis Vision MV4D

3D Use Cases

On desk AR

Indoor navigation -> Augmentation – Modeling

Hand\body tracking\gestures

• Facial tracking – Avatar

• Human \ objects modeling and printing

Computational photography

3D videos-captured events to share in the social media

Robotics vision

New killer apps – Not on the list

Mantis Vision MV4D

Each depth sensing technology has its own set of

LIMITS, BENEFITS & VALUES.

Knowing these is the key to a winning mobile experience.

I hope today’s talk can help...

Mantis Vision MV4D

3D SENSING

TECHNOLOGIES

THE PLAYERS

TIME OF FLIGHT TRIANGULATION

PASSIVE ACTIVE

Mantis Vision MV4D

TIME OF FLIGHT – PRINCIPLES OF OPERATION

•Time of Flight (TOF) – Direct – as its name implies (3DV) •(TOF) – Phase detection of modulated light source (PMD, SoftKinetics, KinectOne) 2.5m back and forth from an object takes 16.67ns…

Mantis Vision MV4D

TIME OF FLIGHT – SAMPLE DATA SHOTS

oExplanary videos (30 sec)

oSome sample data shots:

•Fast •Robust •High-Noise

State of the Art TOF Doesn’t meet mobile

constraints (size\power\cost)

Mantis Vision MV4D

TIME OF FLIGHT – PROS AND CONS

Pro:

Direct measurement per pixel

Low computation and latency

Ideal for gesture\Human interactions

Cons:

High amount of depth noise ~ centimeter scale

Biased depth results due to object reflectivity, ambient

light, edges

Absolute dimensions are not preserved well

• Phase shift calculation requires multiple, very short exposure integration within the

duration of a single modulated cycle (20-130MHz).

• This means large pixels (>10um) which limits sensor resolution.

Mantis Vision MV4D

TRIANGULATION– PRINCIPLES OF OPERATION

Requires: Baseline (A) Correspondence (B) Localization (C)

Mantis Vision MV4D

PASSIVE TRIANGULATION – CORRESPONDENCE CHALLENGE

• Correspondence issues are the main challenge as nothing ensures distinctive features across images.

*Images by Pelican Imaging

Mantis Vision MV4D

PASSIVE TRIANGULATION METHODS

• Stereo cameras

• Multi aperture

• Shape from multiple images

• Pros: Passive • Cons: Texture dependent

• Pros: Multi-view Robustness Computational Photography (Refocus) • Cons: Texture dependent • and very small baseline = low depth

accuracies

• Pros: Large virtual baseline = High depth accuracies Use standard back camera

• Cons: Static only • Cloud only processing

Complicated capture process texture dependent

Mantis Vision MV4D

SOLVING THE CORRESPONDENCE PROBLEM

Coded light source

Instead of looking for correspondence in a featureless image …

Create your own by replacing one camera with a coded light source

Mantis Vision MV4D

CODED LIGHT CHALLENGE - INDEXING

How to index projected features in space ? •Time multiplexed – limited motion •Unique cluster of points – Kinect360 •Dense bi-dimensional epipolar code-

Mantis Vision

Mantis Vision MV4D

Active Triangulation – Kinect360 vs. MV4D

KINECT MV4D Robust Correspondence, High Fill Factor, High Code Capacity Robust Correspondence, Low Fill Factor

Mantis Vision MV4D

Mantis Vision MV4D

Mantis Vision MV4D

Mantis Vision MV4D

Mantis Vision MV4D

Mantis Vision MV4D

Mantis Vision MV4D

Mantis Vision MV4D

Mantis Vision MV4D

280 critical features in selected region…Vs.

Mantis Vision MV4D

40… That’s a x7 difference

Mantis Vision MV4D

Where does it matter?

Mantis Vision MV4D

The noisier the depth data, the greater the need to rely on heavy

de-noising\averaging

Yan Cui, Derek Chan, Sebastian Thrun and Christian Theobalt. See http://ai.stanford.edu/~schuon/ for details.

How does TOF compare?

Mantis Vision MV4D

AVERAGING HAS ITS LIMITS

o Works only for static scenes

o Reduces actual frame rate\speed for clean surfaces (x 5-20 depending on

quality of data)

o Loss of small features, as a result of amount of depth noise (STD)

Mantis Vision MV4D

DEPTH NOISE VS. DISTANCE

We wanted to put both passive and active methods on same graph, but…

0

200

400

600

800

1000

1200

1400

1600

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

Pocket

F5-Pro

Kinect

Pelican

Mantis Vision MV4D

DEPTH NOISE (STD) VS. DISTANCE

Therefore…

0

5

10

15

20

25

30

35

40

45

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

Stan

dar

d d

evi

atio

n in

mm

Distance (m)

Depth noise (mm) Vs distance (m)

Pocket

F5-Pro

Kinect

-- -- -- -- --

TOF –

MOBILE

PASSIVE

STEREO

SHAPE

FROM

MOTION

MULTI

APERTURE

KINECT

MV4D

NUI

INDOOR

AR

LARGE

SCALE

DESK AR

SMALL

SCALE

MODELIN

G

UGC 3D

STATIC

UGC 3D

DYNAMIC

FACIAL

TRACKIN

G

SUMMARY

TABLE

(by app and

technology)

POOR MID EXCELLENT

Mantis Vision MV4D

Thank You gur@mv4d.com