© 2014 Autodesk

Hollywood Has a Lot to Answer For: Making Gestural Input Work for the Real World

Ian Hooper Principal UX Designer

UXPA 2015

John Schrag UX Architect, Media & Entertainment Division

© 2014 Autodesk

© 2014 Autodesk

Part One: A Brief History of Gestural Interaction

© 2014 Autodesk

What do we mean by gestures?

A gesture is a motion of the body that contains information. Waving goodbye is a gesture. Pressing a key on a keyboard is not a gesture because the motion of a finger on its way to hitting a key is neither observed nor significant. All that matters is which key was pressed.

“

”

Kurtenbach and Hulteen (1990):

© 2014 Autodesk

§  Human to Human Interaction: §  Music Conducting §  Dance, Theatre & Magic §  Sign Language & informal salutations and signaling

§  Early Electronic Systems §  Musical instruments (Theremin, Sackbut) §  Sensors (The Clapper, Photodetectors)

Historical Context

© 2014 Autodesk

A long road from theory to practice

Direct Manipulation of Graphical Objects

1990 1970 1960 1950 1980

The Mouse

Windows

Research

Commercialization

2000 2010

Gesture Recognition

derived from: Brad A. Myers (1998). A brief history of human-computer interaction technology. Interactions, vol 5(2), pp. 44-54

© 2014 Autodesk

1981: Xerox Star 8010 §  Original Price: $16,595 §  Professional workstation

was the inspiration for the Macintosh and all Windows / Icons / Mouse / Pointer (WIMP) interfaces

Image: digibarn.com

© 2014 Autodesk

1983: HP-150 §  Original Price: $2795 §  Early personal computer

had infrared touch-screen capability

Image: columbia.edu

© 2014 Autodesk

1984: Apple Macintosh

§  Original Price: $2,495 §  Popularized WIMP §  Introduced the ‘drag’

gesture in place of a dedicated mouse button

Image: oldcomputers.net/

© 2014 Autodesk

§  Games

Outside the mainstream

Dataglove (Fifth Dimension Technologies)

§  Niche

Phantom Omni (SensAble Technologies)

Xbox Kinect (Microsoft)

Playstation Move (Sony) Wii Remote (Nintendo)

© 2014 Autodesk

1998: Dance Dance Revolution (Konami)

2001: Black & White (Lionhead Studios)

2003: Eye Toy (Sony)

§  Games

Outside the mainstream

© 2014 Autodesk

§  Art

Outside the mainstream

1985: Video Place (Myron Krueger)

Projector

Camera

Projector Screen

Back-lit screen

2000: Body Language (Nathaniel Stern)

© 2014 Autodesk

The Renaissance

Image: Apple

§  2007: iPhone §  2010: iPad §  2012: Cintiq

Image: Wacom

© 2014 Autodesk

§  The four best-known players in mid-air gestural input devices right now are: §  Built-in cameras §  Kinect camera §  LEAP Motion device §  Thalmic Labs’ Myo device

A Look at the Hardware

© 2014 Autodesk

Built-in Camera

§  Cheap and ubiquitous §  Open-source libraries for

extracting gesture §  Slower (image processing

in software) §  Bad depth sense §  Error prone

Image: Dell

© 2014 Autodesk

Kinect Camera

§  Reads full body or upper body and creates a 3d skeleton

§  Close up: includes gaze direction, head position, fingers

§  Further back: no fingers

Image: Microsoft

© 2014 Autodesk

LEAP Motion Device

§  Provides full hand skeletal model

§  Has a limited volume – hard to see its limits

§  Occlusion is a problem

Image: LEAP

© 2014 Autodesk

Thalmic Labs Myo

§  Senses hand pose using myoelectrics

§  Tracks rotation & accelleration

§  No occlusion problem §  Usable anywhere §  Only handles 5 poses §  Requires training

Image: Thalmic Labs

© 2014 Autodesk

Case Study: Curating Multi-touch Interaction Across Form-Factors

© 2014 Autodesk

§  Started in January 2010. §  Wanted to create a common, good standard for using

multi-touch for 3d Navigation. §  Wanted to avoid everyone doing it differently

The History

© 2014 Autodesk

§  The interactions should be easy to learn and retain. §  They should fit in naturally with existing standard 2d

interactions. §  They should be suitable for common and key multitouch

hardware setups. §  Blah blah other stuff

Our research goals

© 2014 Autodesk

First Test (July 2010)

§  Mac Laptop running Mudbox

§  Wacom Intuos 3 tablet §  iPhone providing a

multitouch surface §  Also some “soft buttons”

on the phone

© 2014 Autodesk

The Gestures

© 2014 Autodesk

§  Navigation Gestures intuitable? - almost §  Learnable? – Yes! §  Comfortable? – Hard to gesture on that tiny surface §  Soft buttons? – Bad idea; users are looking at the

screen, and must shift focus to use buttons. This breaks their flow.

First Test Results

© 2014 Autodesk

Second Test (Nov 2010)

§  Introducing the iPad! §  Tested gesture retention §  Compared navigation

speed to mouse+hotkey

© 2014 Autodesk

§  Gestures learnable? – Yes! §  Retained after 7-14 days? – Yes! §  Comfortable? – Yes, on the iPad §  Preferred? – Yes… except for fixable technical

problems (lag, etc) §  Faster? – Yes

Second Test Results

© 2014 Autodesk

Third Test (April 2012)

§  Goal was to try the new engine with many different kinds of people

§  Try it on new form factor – 27” PPI multi-touch monitor with stylus

§  First test with on-screen touch

© 2014 Autodesk

§  Experience was surprisingly consistent for people of all sizes/shapes/genders/etc.

§  Users kept trying to touch interface buttons, even when they knew it didn’t work.

§  Camera tilt and zoom interactions felt wrong, unlike in earlier tests – Huh?

Third test results

© 2014 Autodesk

This feels right

© 2014 Autodesk

Same interaction on a touch screen feels wrong

© 2014 Autodesk

Obvious finger/screen mapping

© 2014 Autodesk

§  Try separate behaviours for touchscreens and touch-tablets.

§  Used 27” PPI touch/stylus monitor, vs large monitor with Wacom Intuos 5 tablet.

§  Test new 3, 4, and 5-finger gestures (Mudbox specific).

Fourth Test (July 2012)

© 2014 Autodesk

§  Users perceived the different zoom/roll behaviours on different devices to be the same

§  4 and 5 finger gestures were awkward on the small tablet, but fine on the big screen.

Fourth Test Results

© 2014 Autodesk

§  How do you map tablet touches to multiple 3d views?

Related Considerations

© 2014 Autodesk

§  Just another non-display multitouch tablet like the Intuos, so we used the same behaviour.

§  WRONG.

Adding Magic Trackpad Support

Image: Apple

© 2014 Autodesk

§  Even though this is not a screen device, it controls a persistent pointer on the screen.

§  The entire surface is mounted over a big button, so that you can tap it (lightly) or click it (harder) like a mouse.

How is the Magic Trackpad different?

© 2014 Autodesk

Magic Mouse

§  All the problems of the magic trackpad, and more!

§  Tiny touch area §  Cursor moved by touch

OR by moving mouse §  Frequent false touch

detection

© 2014 Autodesk

§  Testing with users is critical. §  Obvious device classification is not enough; dig for the

salient differences. §  Sometimes you have to do things differently to make

things feel the same across form-factors.

Conclusions

© 2014 Autodesk

Part Three: What’s Wrong with Mid-Air Gestures?

© 2014 Autodesk

Mid-Air Gesture Control is Really Cool

§  It’s been around for ages §  It’s completely natural §  The demos look

awesome §  Why don’t we use it

everywhere? Image: Minority Report (2002) Twentieth Centurey Fox

© 2014 Autodesk

§  Cost of the equipment? §  Availability of hardware? §  To hard to install? §  Too computationally intensive? §  No killer application? §  The user experience sucks?

Why we don’t use mid-air gestures everywhere

© 2014 Autodesk

§  Cost of the equipment? §  Availability of hardware? §  To hard to install? §  Too computationally intensive? §  No killer application? §  The user experience sucks?

Why we don’t use mid-air gestures everywhere

© 2014 Autodesk

§  It looks awesome in demos. §  BUT in practice, it’s often:

§  Disappointing §  Hard to control §  No better (and often worse) than existing interactions

Primary Reason

© 2014 Autodesk

§  Awesome YouTube Demo!

Google Earth + Leap Motion

© 2014 Autodesk

“A lot of potential, but it needs much more development” “It’s much more intuitive than clicking with a mouse… but it’s not actually doing what I want.”

Maya + Leap Motion

Image: Digital Arts Online

© 2014 Autodesk

§  Gorilla arm §  Gesture delineation & intentionality §  No tactile feedback §  Context and mapping §  Internationalization

Why Is Mid-Air Gestural Interaction Hard?

© 2014 Autodesk

Gorilla Arm

Photo by Richard Ruggiero, public domain

§  Holding your arms up to interact is quickly tiring.

Diagram from Austin, Gilbert (1806): Chiromania; or a Treatise on Rhetorical Delivery

Seriously?

© 2014 Autodesk

Gesture Delineation & Intentionality

Photo by: Biswarup Ganguly, used under Creative Commons license (CC)

© 2014 Autodesk

Context and Mapping

Image: Bolt, Richard A. “Put That There”: Voice and Gesture at the graphics interface. SIGGRAPH 1980.

© 2014 Autodesk

No tactile feedback

Image: Syntact ultrasonic mid-air haptic feedback device by Ultrasonic-Audio.com

© 2014 Autodesk

Internationalization

Frames from the short film Hand Gestures, by Jeff Werner, Vancouver, Canada

© 2014 Autodesk

Doomed? No.

© 2014 Autodesk

How do we know?

© 2014 Autodesk

1. Because Kinect Games are pretty Awesome

Frame from Kinect Star Wars

© 2014 Autodesk

2. Because music conductors don’t suffer from Gorilla Arm

Photo courtesy of John Schrag

© 2014 Autodesk

“Consumed Endurance” – metric for Gorilla arm

Diagram from Consumed Endurance: A Metric to Quantify Arm Fatigue of Mid-Air Interactions. Juan David Hincapié-Ramos, Xiang Guo, Paymahn Moghadasian, Pourang Irani,University of Manitoba

© 2014 Autodesk

§  We do it with voice, eye contact, facial expression, repetition.

§  This is an area of active research, both in HCI and linguistics (ASL)

3. Because we can all delineate gestures

© 2014 Autodesk

§  The question isn’t: “is mid-air gestural interaction good”

§  The question is: “What is it good for?”

4. Because we understand this:

© 2014 Autodesk

§  Is there a killer app for mid-air gestural input? (aside from Games)

The Killer App?

© 2014 Autodesk

The Killer App: VR and AR

Image: technologyreview.com

Video: ign.com

© 2014 Autodesk

Case Study: Mid-Air Gestures with the Myo Device

© 2014 Autodesk

§  Test the suitability of the Myo device for VR/AR navigation applications.

§  Come up with useful design guidelines and mental models to help us design better gestural interactions with this class of device

§  Improve our own shared gesture-interpretation library for other Autodesk developers.

Our research goals

© 2014 Autodesk

§  “Gorilla Arm”… no use of upper arm §  To delineate… use hand poses §  Occlusion… not a problem with this device §  Context… not absolute mapping needed §  Tactile feedback… not an issue for this task

Designing the Gestural Interaction

© 2014 Autodesk

First Test

§  PC running Stingray game engine

§  Thalmic Labs Myo arm band

§  Calibrated Myo §  Users received

training

© 2014 Autodesk

§  Continuous pose detection is problematic on the Myo §  Holding poses continuously strains the hand §  Non-mnemonic gestures were poorly recalled §  Instinctive movement often overrode training §  Gesture success depends on personal physiology

First Test Results

© 2014 Autodesk

Second Test (informal)

§  Changed steering to more natural mapping

§  Removed need to hold hand poses

§  Added speed control §  Various small fixes

© 2014 Autodesk

§  Gestures are deep-rooted: adapt to your user’s movement patterns where possible.

§  User testing is still crucial – test with different physiologies.

§  Remember to measure physical comfort when testing – people may not mention it unless asked.

§  Simplify your problem domain if necessary §  Don’t give up right away

Conclusions

© 2014 Autodesk

The Real World is Not Magic

Video: https://www.youtube.com/channel/UC0bM0PxqixYjYY7ypt5mqGw

§  Gestural control can seem like magic… IF it is designed well

§  When you want to get something done… cool is irrelevant

© 2014 Autodesk

John Schrag @jvschrag

Ian Hooper +IanHooperUX

Autodesk is a registered trademark of Autodesk, Inc., and/or its subsidiaries and/or affiliates in the USA and/or other countries. All other brand names, product names, or trademarks belong to their respective holders. Autodesk reserves the right to alter product and services offerings, and specifications and pricing at any time without notice, and is not responsible for typographical or graphical errors that may appear in this document.

© 2014 Autodesk. All rights reserved.