Embed Size (px)
Citation preview
Mantis Vision MV4D
A bit about me
USER GENERATED 3D CONTENT CREATION
Mantis Vision MV4D
Mobile Depth Sensing Methods What are they good for?
Mantis Vision MV4D
FACT: 3D is here. And it’s going mobile.
Mantis Vision MV4D
Apps are next. Content Creation is the key.
Mantis Vision MV4D
3D Use Cases
On desk AR
Indoor navigation -> Augmentation – Modeling
Hand\body tracking\gestures
• Facial tracking – Avatar
• Human \ objects modeling and printing
Computational photography
3D videos-captured events to share in the social media
Robotics vision
New killer apps – Not on the list
Mantis Vision MV4D
Each depth sensing technology has its own set of
LIMITS, BENEFITS & VALUES.
Knowing these is the key to a winning mobile experience.
I hope today’s talk can help...
Mantis Vision MV4D
3D SENSING
TECHNOLOGIES
THE PLAYERS
TIME OF FLIGHT TRIANGULATION
PASSIVE ACTIVE
Mantis Vision MV4D
TIME OF FLIGHT – PRINCIPLES OF OPERATION
•Time of Flight (TOF) – Direct – as its name implies (3DV) •(TOF) – Phase detection of modulated light source (PMD, SoftKinetics, KinectOne) 2.5m back and forth from an object takes 16.67ns…
Mantis Vision MV4D
TIME OF FLIGHT – SAMPLE DATA SHOTS
oExplanary videos (30 sec)
oSome sample data shots:
•Fast •Robust •High-Noise
State of the Art TOF Doesn’t meet mobile
constraints (size\power\cost)
Mantis Vision MV4D
TIME OF FLIGHT – PROS AND CONS
Pro:
Direct measurement per pixel
Low computation and latency
Ideal for gesture\Human interactions
Cons:
High amount of depth noise ~ centimeter scale
Biased depth results due to object reflectivity, ambient
light, edges
Absolute dimensions are not preserved well
• Phase shift calculation requires multiple, very short exposure integration within the
duration of a single modulated cycle (20-130MHz).
• This means large pixels (>10um) which limits sensor resolution.
Mantis Vision MV4D
TRIANGULATION– PRINCIPLES OF OPERATION
Requires: Baseline (A) Correspondence (B) Localization (C)
Mantis Vision MV4D
PASSIVE TRIANGULATION – CORRESPONDENCE CHALLENGE
• Correspondence issues are the main challenge as nothing ensures distinctive features across images.
*Images by Pelican Imaging
Mantis Vision MV4D
PASSIVE TRIANGULATION METHODS
• Stereo cameras
• Multi aperture
• Shape from multiple images
• Pros: Passive • Cons: Texture dependent
• Pros: Multi-view Robustness Computational Photography (Refocus) • Cons: Texture dependent • and very small baseline = low depth
accuracies
• Pros: Large virtual baseline = High depth accuracies Use standard back camera
• Cons: Static only • Cloud only processing
Complicated capture process texture dependent
Mantis Vision MV4D
SOLVING THE CORRESPONDENCE PROBLEM
Coded light source
Instead of looking for correspondence in a featureless image …
Create your own by replacing one camera with a coded light source
Mantis Vision MV4D
CODED LIGHT CHALLENGE - INDEXING
How to index projected features in space ? •Time multiplexed – limited motion •Unique cluster of points – Kinect360 •Dense bi-dimensional epipolar code-
Mantis Vision
Mantis Vision MV4D
Active Triangulation – Kinect360 vs. MV4D
KINECT MV4D Robust Correspondence, High Fill Factor, High Code Capacity Robust Correspondence, Low Fill Factor
Mantis Vision MV4D
Mantis Vision MV4D
Mantis Vision MV4D
Mantis Vision MV4D
Mantis Vision MV4D
Mantis Vision MV4D
Mantis Vision MV4D
Mantis Vision MV4D
Mantis Vision MV4D
280 critical features in selected region…Vs.
Mantis Vision MV4D
40… That’s a x7 difference
Mantis Vision MV4D
Where does it matter?
Mantis Vision MV4D
The noisier the depth data, the greater the need to rely on heavy
de-noising\averaging
Yan Cui, Derek Chan, Sebastian Thrun and Christian Theobalt. See http://ai.stanford.edu/~schuon/ for details.
How does TOF compare?
Mantis Vision MV4D
AVERAGING HAS ITS LIMITS
o Works only for static scenes
o Reduces actual frame rate\speed for clean surfaces (x 5-20 depending on
quality of data)
o Loss of small features, as a result of amount of depth noise (STD)
Mantis Vision MV4D
DEPTH NOISE VS. DISTANCE
We wanted to put both passive and active methods on same graph, but…
0
200
400
600
800
1000
1200
1400
1600
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
F5-Pro
Kinect
Pelican
Mantis Vision MV4D
DEPTH NOISE (STD) VS. DISTANCE
Therefore…
0
5
10
15
20
25
30
35
40
45
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Stan
dar
d d
evi
atio
n in
mm
Distance (m)
Depth noise (mm) Vs distance (m)
F5-Pro
Kinect
-- -- -- -- --
TOF –
MOBILE
PASSIVE
STEREO
SHAPE
FROM
MOTION
MULTI
APERTURE
KINECT
MV4D
NUI
INDOOR
AR
LARGE
SCALE
DESK AR
SMALL
SCALE
MODELIN
G
UGC 3D
STATIC
UGC 3D
DYNAMIC
FACIAL
TRACKIN
G
SUMMARY
TABLE
(by app and
technology)
POOR MID EXCELLENT
