Upload
odalys-garcia
View
220
Download
0
Tags:
Embed Size (px)
Citation preview
Bidi Screen: A thin, Depth-Sensing LCF for 3D Interaction using Light FieldsMattew Hirsh, Douglas Lanman, Henry Holtzman, Ramesh RaskarMIT Media Lab & Brown University
Armando de la Re Vega
October 14th 2011
Introduction
•Transform a LCD into a display that supports▫2D multitouch▫Unencumbered 3D gestures
•Inspired by LCD embedded optical sensors.
•Exploit the spatial light modulation capability of LCD to allow imaging without interfering with display functionality.2
Introduction
•Using light sensors to detect multiple points in contact with the surface of LCD.
•Sharp Co. and Planar Systems inc. LCD with arrays of optical sensors interlaced within the pixel grid.
•Touch is determined from the spatial position of occluded sensors that receive less light.
•Touch, but not gestures.3
Introduction
•This paper describes how to modify LCDs to allow capture and display.
•Captures the angle and intensity of light entering a co-located sensor array.
•Enables the detection of gestures.
4
BiDirectional Screen
•Sensor array located slightly behind the spatial light modulating layer of a conventional LCD.
•Two modes:▫Display mode: backlight and liquid crystal
spatial light modulator function normal.▫Capture mode: backlight disabled and light
modulator displays an array of pinholes or tiled broad banded code.
•Two applications: touch+gestures interaction and a light gun mode for interaction.
5
Contributions• Thin depth sensing LCDs
▫Support on 2D multitouch and 3D gestures.▫Alternates displayed image and optical mask.▫Maximize display and capture frame rate using
optimally light-efficient mask patterns.• Lensless Light Field Capture.
▫Lensless light field camera, composed of optical sensor array and a spatial light modulator.
▫Evaluation of pinhole arrays and tile broad band masks.
• Unencumbered 3D interaction.▫Novel interaction scenarios to recognize on- and off-
screen gestures.6
Benefits and Limitation
•Ability to capture multiple orthographic images.
•Potentially thin device.•Not blocking the backlight or portions of
the display.
•Separates LCD layers.•Uses a pair of cameras increasing the
device dimensions.•Will reduce the native frame rate.•Reduce of contrast by external
illumination.7
Design Goals
•Capture 3D enable depth and lighting aware interaction.
•Prevent image capture from interfering with image display.
•Support walk-up interaction.•Achieve these goals with a portable, thin
form facto device.
8
Comparison of Design Alternatives
•Capacitive, Resistive or Acoustic Modalities.▫Are effective for multitouch but not 3D
gestures.▫Some capacitives detect approaching, but
not their distance.▫This technologies not support lightning
aware interaction.▫Optical sensing does.
9
Comparison of Design Alternatives
•Cameras behind, To the Side, or In Front the Display.▫Behind interferes with backlighting,
casting shadows and brightness variations.▫In front or to the side, risks being occluded
by users.▫In the bezel, increase the display thickness
and suffer from user self occlusion.
10
Comparison of Design Alternatives
•Photo detector Arrays.▫Array located behind the LCD.▫Not suffer from user self occlusion.▫Detector layer can be extremely thin and
optically transparent.▫Requires a small gap between the spatial
light modulating and lighting planes.▫This gap allows to measure the angle of
incident light and intensity.
11
Comparison of Design Alternatives
•Camera Arrays.▫A dense camera array is similar to Photo
detector array.▫But they must be synchronized and
assembled, increasing the engineering complexity.
▫The sensors and lenses required by each cam, introduce backlighting non uniformity.
12
Designing a Thin-Depth Sensing LCD• LCD Components
▫ Backlight: cold cathode fluorescent lamp or array of LEDs, a light guide, a rear reflecting surface, a diffuser and several brightness enhancing films.
▫ Spatial light modulator: a pair of crossed linear polarizers and a layer of liquid crystal molecules.
13
Designing a Thin-Depth Sensing LCD
•Hardware Design▫Remove light, light guide, reflector,
brightness enhancing films and final diffuser.
▫Use spatial light modulator to display masks.
▫A coded image equivalent to the mask is formed on the diffuser that cameras can photograph.
▫Additional array of LEDs behind the diffuser.
14
Designing a Thin-Depth Sensing LCD
•Optical design with Pinhole Arrays▫18 LCD pixels between each.▫0.2% of incident light reaches the diffuser.▫Extremely bright external light.
15
Designing a Thin-Depth Sensing LCD
•Optical design with Tiled Broadband Masks▫19x19 LCD pixels.▫50% of incident light reaches the diffuser.▫Allows external light to be dimmed by a
180 factor.
16
Multi-view Processing• Heterodyne decoding method of Veeraraghavan et al.
(2007).▫ http://www.merl.com/papers/docs/TR2007-115.pdf
• Focus method of Nayar and Nakagawa (1994).▫ http://www1.cs.columbia.edu/CAVE/publications/pdfs/Nayar_PAMI94.pdf
• Methods for synthetic aperture photography by Vaish et al.(2006).▫ http://graphics.stanford.edu/papers/sap-recons/final.pdf
• Synthetically focus at a distance, computationally efficient approach of Ng (2005).▫ http://graphics.stanford.edu/papers/fourierphoto/fourierphoto-600dpi.pdf
17
Interaction Modes
•Multitouch and 3D Interaction▫Supports on screen multitouch and off
screen gestures.▫Real time depth map: Allows 3D tracking of
objects in front of display.
18
Interaction Modes
•Lighting Sensitive Interaction▫Altering the light striking the screen.▫Model lightning application allows
interactive relighting of virtual scenes.
19
Performance
•Implementation LCD▫Sceptre X20WG NagaII 20.1” LCD
Spatial light separated from backlight and front diffuser polarizer.
Spatial light mounted backwards. Backlight diffuser placed 2.5cm behind and
with a linear polyvinyl alcohol-iodine (PVA) filter.
16 Luxeon Endor Rebel cool white LEDs.
20
Performance
•Implementation LCD▫Point Grey Flea2 cameras 1m behind the
diffuser (1280x960 8 bit grayscale image 7fps).
▫Intel Xeon 8 Core 2.66GHz, 4GB RAM▫NVIDIA Quadro FX 570.▫External halogen lamps with tiled-MURA.▫Pinhole mask requires additional halogen
lamp above the region in front of display.
21
Performance
•Limitations▫Lower limit on the pixel sizes in LCD and sensor.▫Limit in the maximum angular and spatial
resolution.▫Optimized for real time interaction, rather than
high resolution photography.▫Frame rate limited to 7.5fps, video cameras and
transfer rate.▫External lightning is required in image capture.▫Reduce of display contrast.▫Objects close can be occluded from ambient
light.22
Discussion and Future Directions
•Capable of dynamically updating the mask.
•Should be scaled to provide photographic quality images.
•Higher frame rates should allow flicker-free viewing and more accurate tracking.
•Could allow to track multiple users.23
Conclusions
•Inspire the inclusion of features to light sensing displays.
•Including an array of low resolution cameras, will increase the angular resolution directly facilities unencumbered 3D interaction with thin displays.
24
Thanks
25