OmniTouch: Wearable Multitouch Interaction Everywhere

Chris Harrison (1 & 2)

Hrvoje Benko (1)

Andrew D. Wilson (1)

Affliations: 1. Microsoft Research

2. Carnegie Mellon University

Objectives

• Developed a novel wearable depth-sensing and projection system that enables interactive multitouch applications on everyday surfaces. The paper shows that anything one can do on today’s mobile devices, they could do in the palm of their hand.

• The developed system allows the wearer to use their hands, arms and legs as graphical, interactive surfaces.

• It also allows the users interact on appropriate surfaces from the environment to expand the interactive area.

• OmniTouch provides a X-Y plane (2D interface) as in a touchscreen mobile phone display with clicking or hovering of fingers are detected for interactions.

Project Demo

Hardware Setup

The lone hardware module consisted of a Microsoft Kinect (PrimeSense Depth Camera), Microvision ShowWX+ laser pico-projector, and a computer tethering the Kinect and pico-projector.

Hardware Capabilities

• Resolution: 320 x 240• Speed: 30 FPS• Error on depth sensing: 5mm for an object that is

20cm far on Z-axis

PrimeSense depth camera

• Wide angle• Focus free projection of graphical elements regardless

depth

Microvision ShowWX+ laser pico-projector

Multitouch Finger Tracking

• Finger SegmentationA. Depth derivative in the X-Y

plane using slidingB. Derivative of the depth mapC. Finger slices overlaid in blueD. Path finding and tip

estimation

Close up example of a candidate finger slice

Multitouch Finger Tracking (cont’d)

• Finger Click Detection• Midpoint of the active fingers

are computed• Flood fill towards the fingertip• This operation is performed on

the depth map

Hovering Clicked

On demand projected interfaces

Due to the dynamic nature of the objects of real world, the displays must be projected in such a way as to account for their host surface’s position and orientation in 3D space.

Surface segmentation and tracking• Distinct surfaces are segmented by performing

a 3D connected components operation on the depth map. Surfaces smaller than hand size are discarded.

• Object orientation is computed on a 3D co-ordinate system.

• Lock points are calculated taking the centroid of the object on which the display is being projected. This method is chosen for stability.

Characteristics of the projected interfaces

• OmniTouch can use a surface’s lock point and orientation to provide an interface that tracks with a surface.

• When a projecting on a large table, the interface will still be hand-sized.

One Size Fits All

• The authors performed surface classification among a set of five common surfaces (hand, arm, pad, wall, and table) by considering a variety of features derived from each surface’s depth image.

• Unfortunately, this classification-driven approach suffers from scalability issues, since it is simply not possible to build a classifier for every conceivable surface.

Classification-Driven Placement

• So that users can drag and replace the entire projection.

User-Specified Placement

Context-awareness of projection surface

• The surface tracking procedure generates a variety of metadata, such as whether a surface is public or private, organic or flat, big or small, and vertically or horizontally oriented – all of which are exposed to applications wishing to be more context sensitive.

User study

The four surfaces we tested and user click distributions. 95% confidence ellipses shown in green. Axis units in mm.

System accuracy

Finger click detection• Click receiving: 96.5%• Click segmentation:

98.9%

1Finger click spatial accuracy• Finger click spatial

precision: 95% (Button diameter, next slide)

2

Button diameter

Button diameter necessary to encompass 95% of touches. Error bars denote standard deviation across

all trials.Button diameter needed to encompass 95% of

touches. Error bars show standard deviation across trials.

My opinions

• Novel idea. Well written manuscript. Strong background work. Stable implementation and excellent demonstration.

• Achieved accuracies advocate for feasibility on consumer level applications.

Positives:

• Application receives a lot of ‘WOW’s. But this particular technology didn’t go far in the consumer market of gadgets.

• Ranges of applications are limited to typical smartphone applications, which is not that bad. However, the authors should have presented one application regarding multimedia presentation. For example, this class presentation could have been better using OmniTouch.

Negatives:

Follow up works

Questions?