Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training

Paul Varcholik, Joseph J. LaViola Jr., Denise NicholsonInstitute for Simulation and TrainingUniversity of Central Florida

pvarchol@ist.ucf.edu, jjl@eecs.ucf.edu, dnichols@ist.ucf.edu

TACTUS: A Hardware and Software Framework for Research in Multi-Touch Interaction

Multi-Touch: Introduction

Framework Motivation Primary multi-touch components

1. Physical interaction surface2. Software system for collecting and

interpreting points of contact

Present a barrier to entry for researchers focused on higher-level interface issues or application development

Multi-Touch Origins (pre-2000) Keyboards Touchscreens Pen-based computing 1984 – Bob Boie (Bell Labs) perhaps the first

multi-touch screen 1985 – Bill Buxton, Multi-Touch Tablet 1990 – Sensor Frame, Paul McAvinney (CMU)

Optical Sensor 3 fingers (some trouble with ambiguous finger positions)

1998 – Fingerworks

Multi-Touch Origins (2000 to present) 2001 – Paul Deitz – Mitsubishi DiamondTouch 2003 – Jazz Mutant 2004 – Andy Wilson – TouchLight Oct. 2005 – Jeff Han – FTIR, UIST Paper Feb. 2006 – Jeff Han – TED Talk 2006 – Jeff Han – Perceptive Pixel (CNN Magic Wall) Spring 2007 – Microsoft Surface Summer 2007 – Apple iPhone 2008 – Open-Source Community 2009 – Windows 7

Framework RequirementsEssential Component DescriptionMulti-Touch Surface Commercial-off-the-shelf hardware, requiring

near zero pressure to detect an interaction point

Software Hit-Testing Ability to determine the presence and location of each point of surface contact; supporting at least four concurrent users

Software Point Tracking Identifying a continuous point of contact and reporting its velocity and duration

Framework RequirementsSecondary Component Description

Application Programming Interface (API)

A software system upon which multiple multi-touch applications can be developed

Multi-Platform Support The ability to access multi-touch interaction data from different computing environments (e.g. languages, OS’s, etc.)

Reconfiguration Modifying the software system without recompilation

Software Service Allowing multiple applications to access multi-touch interaction data simultaneously, including over a computer network

Presentation-Layer Independence

Isolating the multi-touch interaction data from the system to graphically present such data, allowing any GUI to be employed when developing multi-touch applications

Mouse Emulation Support for controlling traditional ‘Window, Icon, Menu, Pointing Device’ interaction through a multi-touch surface

Tangible Interfaces The ability to detect and interact with physical devices placed on or near the multi-touch surface

Customizable Gesture System Support for training arbitrary multi-touch gestures and mapping them to software events

Shape Detection e.g. Finger, blob, hand

Multi-Touch Hardware Frustrated Total Internal Reflection (FTIR)

1. Optical waveguide2. Supporting structure (e.g. cabinet)3. IR sensing camera4. Projector and diffuser5. IR emission source6. Computer

Multi-Touch Efforts: Hardware

Software Framework Image processing Presentation layer Multi-platform communication Pen/writing style interaction (Ink) Gesture recognition

Image Processing Pipeline

Webcam RotateFlip Background Filter

Threshold

Scaling

Blob Detection GrayScale

Grayscale Image 8bpp

Binary Image 1bpp

Calibration Point Tracking

ProcessedFrame

Camera Frame24bpp

Image Processing

Raw Camera Frame

Background Filtered Threshold Processed

Points

Multi-Touch Efforts: Software

Mouse EmulatorFunction Gesture DescriptionLeft Click Quick tap on the surface with one finger.

Left Click (alternate) While holding down a finger, tap another finger to the left side of the first.

Drag Perform a Left Click (alternate) but do not release the left side press. Drag both fingers to the destination and release.

Right Click While holding down a finger, tap another finger to the right side of the first.

Double Click Tap two fingers at the same time.

Mouse Wheel Scroll While holding down a finger, drag another finger vertically and to the right side of the first. Dragging up scrolls the mouse wheel up and vice versa.

Alt-Tab While not a mouse command, the Alt-Tab command is a useful Windows feature that switches between applications. To perform an Alt-Tab, hold down a finger and drag another finger horizontally above the first. Dragging to the left moves backward through the list of active applications and dragging to the right moves forward.

Multi-Touch Efforts: Video Montage

Multi-Touch Efforts: Additional Videos Demo Explorer Improved Calibration Multi-Touch Starcraft Mouse Emulator Ink Demonstration Particle System Demonstration SurfaceSimon SurfaceCommand

Conclusions The Framework:

In development since Fall 2007Open-source availability since Spring 2008Now in its fourth major releaseAnswers most of the specified requirementsSuccessfully employed by UCF and colleagues

from Brown UniversityDownloaded extensively by the open-source

communityProvides a robust, extensible platform for

research in multi-touch interaction

Questions?

Paul Varcholik, Joseph J. LaViola Jr., Denise NicholsonInstitute for Simulation and TrainingUniversity of Central Florida

pvarchol@ist.ucf.edu, jjl@eecs.ucf.edu, dnichols@ist.ucf.edu

TACTUS: A Hardware and Software Framework for Research in Multi-Touch Interaction

Resources: Commercial Platforms Microsoft Surface Mitsubishi DiamondTouch Apple iPhone & iPod Touch Perceptive Pixel N-trig MultiTouch OY (Finland) Jazz Mutant HP TouchSmart Dell Latitude XT2 TacTable

Resources: Open-Source Platforms TouchLib (NUI Group) tBeta (NUI Group) reacTIVision Bespoke Multi-Touch Framework Sparsh UI (Iowa State) BBTouch Touche`

Resources: Online NUI Group TouchKit by NOR_/D Multi-Touch Systems that I Have Known

and Loved – Bill Buxton, Microsoft Research

Jeff Han’s TED Talk Bespoke Software