25
Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab & Brown University Armando de la Re Vega October 14th 2011

Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Embed Size (px)

Citation preview

Page 1: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light FieldsMattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh RaskarMIT Media Lab & Brown University

Armando de la Re Vega

October 14th 2011

Page 2: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Introduction

•Transform a LCD into a display that supports▫2D multitouch▫Unencumbered 3D gestures

•Inspired by LCD embedded optical sensors.

•Exploit the spatial light modulation capability of LCD to allow imaging without interfering with display functionality.2

Page 3: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Introduction

•Using light sensors to detect multiple points in contact with the surface of LCD.

•Sharp Co. and Planar Systems inc. LCD with arrays of optical sensors interlaced within the pixel grid.

•Touch is determined from the spatial position of occluded sensors that receive less light.

•Touch, but not gestures.3

Page 4: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Introduction

•This paper describes how to modify LCDs to allow capture and display.

•Captures the angle and intensity of light entering a co-located sensor array.

•Enables the detection of gestures.

4

Page 5: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

BiDirectional Screen

•Sensor array located slightly behind the spatial light modulating layer of a conventional LCD.

•Two modes:▫Display mode: backlight and liquid crystal

spatial light modulator function normal.▫Capture mode: backlight disabled and light

modulator displays an array of pinholes or tiled broad banded code.

•Two applications: touch+gestures interaction and a light gun mode for interaction.

5

Page 6: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Contributions• Thin depth sensing LCDs

▫Support on 2D multitouch and 3D gestures.▫Alternates displayed image and optical mask.▫Maximize display and capture frame rate using

optimally light-efficient mask patterns.• Lensless Light Field Capture.

▫Lensless light field camera, composed of optical sensor array and a spatial light modulator.

▫Evaluation of pinhole arrays and tile broad band masks.

• Unencumbered 3D interaction.▫Novel interaction scenarios to recognize on- and off-

screen gestures.6

Page 7: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Benefits and Limitation

•Ability to capture multiple orthographic images.

•Potentially thin device.•Not blocking the backlight or portions of

the display.

•Separates LCD layers.•Uses a pair of cameras increasing the

device dimensions.•Will reduce the native frame rate.•Reduce of contrast by external

illumination.7

Page 8: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Design Goals

•Capture 3D enable depth and lighting aware interaction.

•Prevent image capture from interfering with image display.

•Support walk-up interaction.•Achieve these goals with a portable, thin

form facto device.

8

Page 9: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Comparison of Design Alternatives

•Capacitive, Resistive or Acoustic Modalities.▫Are effective for multitouch but not 3D

gestures.▫Some capacitives detect approaching, but

not their distance.▫This technologies not support lightning

aware interaction.▫Optical sensing does.

9

Page 10: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Comparison of Design Alternatives

•Cameras behind, To the Side, or In Front the Display.▫Behind interferes with backlighting,

casting shadows and brightness variations.▫In front or to the side, risks being occluded

by users.▫In the bezel, increase the display thickness

and suffer from user self occlusion.

10

Page 11: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Comparison of Design Alternatives

•Photo detector Arrays.▫Array located behind the LCD.▫Not suffer from user self occlusion.▫Detector layer can be extremely thin and

optically transparent.▫Requires a small gap between the spatial

light modulating and lighting planes.▫This gap allows to measure the angle of

incident light and intensity.

11

Page 12: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Comparison of Design Alternatives

•Camera Arrays.▫A dense camera array is similar to Photo

detector array.▫But they must be synchronized and

assembled, increasing the engineering complexity.

▫The sensors and lenses required by each cam, introduce backlighting non uniformity.

12

Page 13: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Designing a Thin-Depth Sensing LCD• LCD Components

▫ Backlight: cold cathode fluorescent lamp or array of LEDs, a light guide, a rear reflecting surface, a diffuser and several brightness enhancing films.

▫ Spatial light modulator: a pair of crossed linear polarizers and a layer of liquid crystal molecules.

13

Page 14: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Designing a Thin-Depth Sensing LCD

•Hardware Design▫Remove light, light guide, reflector,

brightness enhancing films and final diffuser.

▫Use spatial light modulator to display masks.

▫A coded image equivalent to the mask is formed on the diffuser that cameras can photograph.

▫Additional array of LEDs behind the diffuser.

14

Page 15: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Designing a Thin-Depth Sensing LCD

•Optical design with Pinhole Arrays▫18 LCD pixels between each.▫0.2% of incident light reaches the diffuser.▫Extremely bright external light.

15

Page 16: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Designing a Thin-Depth Sensing LCD

•Optical design with Tiled Broadband Masks▫19x19 LCD pixels.▫50% of incident light reaches the diffuser.▫Allows external light to be dimmed by a

180 factor.

16

Page 17: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Multi-view Processing• Heterodyne decoding method of Veeraraghavan et al.

(2007).▫ http://www.merl.com/papers/docs/TR2007-115.pdf

• Focus method of Nayar and Nakagawa (1994).▫ http://www1.cs.columbia.edu/CAVE/publications/pdfs/Nayar_PAMI94.pdf

• Methods for synthetic aperture photography by Vaish et al.(2006).▫ http://graphics.stanford.edu/papers/sap-recons/final.pdf

• Synthetically focus at a distance, computationally efficient approach of Ng (2005).▫ http://graphics.stanford.edu/papers/fourierphoto/fourierphoto-600dpi.pdf

17

Page 18: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Interaction Modes

•Multitouch and 3D Interaction▫Supports on screen multitouch and off

screen gestures.▫Real time depth map: Allows 3D tracking of

objects in front of display.

18

Page 19: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Interaction Modes

•Lighting Sensitive Interaction▫Altering the light striking the screen.▫Model lightning application allows

interactive relighting of virtual scenes.

19

Page 20: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Performance

•Implementation LCD▫Sceptre X20WG NagaII 20.1” LCD

Spatial light separated from backlight and front diffuser polarizer.

Spatial light mounted backwards. Backlight diffuser placed 2.5cm behind and

with a linear polyvinyl alcohol-iodine (PVA) filter.

16 Luxeon Endor Rebel cool white LEDs.

20

Page 21: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Performance

•Implementation LCD▫Point Grey Flea2 cameras 1m behind the

diffuser (1280x960 8 bit grayscale image 7fps).

▫Intel Xeon 8 Core 2.66GHz, 4GB RAM▫NVIDIA Quadro FX 570.▫External halogen lamps with tiled-MURA.▫Pinhole mask requires additional halogen

lamp above the region in front of display.

21

Page 22: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Performance

•Limitations▫Lower limit on the pixel sizes in LCD and sensor.▫Limit in the maximum angular and spatial

resolution.▫Optimized for real time interaction, rather than

high resolution photography.▫Frame rate limited to 7.5fps, video cameras and

transfer rate.▫External lightning is required in image capture.▫Reduce of display contrast.▫Objects close can be occluded from ambient

light.22

Page 23: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Discussion and Future Directions

•Capable of dynamically updating the mask.

•Should be scaled to provide photographic quality images.

•Higher frame rates should allow flicker-free viewing and more accurate tracking.

•Could allow to track multiple users.23

Page 24: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Conclusions

•Inspire the inclusion of features to light sensing displays.

•Including an array of low resolution cameras, will increase the angular resolution directly facilities unencumbered 3D interaction with thin displays.

24

Page 25: Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light Fields Mattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh Raskar MIT Media Lab

Thanks

25