Advanced Interaction Techniques

Carlos Duarte 2015/2016

Gestures

What is a gesture?

In this context, any physical movement that can be sensed and responded by a digital system without the aid of a traditional input device such as keyboard or mouse.

History• Sketchpad used light-pen gestures

(1963) • Teitelman in 1964 developed the first

trainable gesture recogniser • A very early demonstration of gesture

recognition was Tom Ellis' GRAIL system on the RAND tablet (1964, ARPA funded).

• A gesture-based text editor using proof-reading symbols was developed at CMU by Michael Coleman in 1969.

• 1972 – PLATO IV Touch Screen Terminal. Computer-based Education Research Laboratory, University of Illinois. 16-by-16 grid infrared touch panel

• Gesture recognition has been used in commercial CAD systems since the 1970s, and came to universal notice with the Apple Newton in 1992

History1983 – VIDEOPLACE

• Myron Krueger

• Real time image processing

• Background subtraction and edge detection to create a silhouette of the user

• Introduce Pinch gesture

History1991 – Digital Desk

• Pierre Wellner

• Project a digital surface onto a physical desk

• Use optical and acoustic techniques to sense hands/fingers

• Demonstrate multi-touch concepts such as two finger scaling and translation

History

1992 – Simon

• First smart phone

• Touch-screen

History2006 – Jeff Han

• Frustrated Total Internal Reflection

• Multi-touch touch-screen

• Created Perceptive Pixel

• http://www.youtube.com/watch?v=5JcSu7h-I40

History

2007 – iPhone

• Soft touch-based interface

• Very smooth interaction

• Multi-touch capability to a limited degree

History

2007 – Microsoft Surface

• Interactive table surface

• Capable of sensing multiple fingers and hands

• Capable of identifying various objects and their position on the surface

Characteristics

Types of gestures• Touch

• Single touch

• Multi touch

• Free-form

• Body (hand, head, …) gesture

• Mouse, glove, …

Styles of gestures• Deictic - the gestures of pointing to objects in the

environment

• Manipulative - direct relationship between the movement of the hand and the object to be manipulated

• Symbolic - system of gestures that use a dynamic or static dictionary

• Gesticulation - most natural ways of communicating, frequently used in multimodal interfaces

Gestural interfaces• Pros

• More flexible

• Less visible hardware

• More “natural”

• More fun

• Cons

• Heavy data input

• Relies heavily on the visual

• Can be inappropriate for context

• More physically demanding

Good characteristics• Discoverable (visibility

and affordable)

• Trustworthy

• Responsive

• Appropriate

• Meaningful / Adequate

• Smart

• Clever

• Playful

• Pleasurable

• Good

Attributes• Presence

• Position / Locale

• Duration

• Motion / Speed

• Pressure (if touch)

• Size / Form

• Orientation

• Objects / Context

• Number of touch /recognised points

• Participants

• Sequence

Technology

Sensors• Pressure

• Light

• Proximity

• Acoustic

• Tilt

• Motion

• Orientation

The (really) free

The (not so) free

TouchscreensResistive

• Change in electrical field is noted and the coordinates of the point of contact are calculated

TouchscreensCapacitive

• A layer that stores electrical charge is placed in the monitor

• Some charge is transferred to the user

TouchscreensSurface acoustic wave systems

• Ultrasonic waves pass over the touchscreen panel

• When the panel is touched, a portion of wave is absorbed

TouchscreensInfrared grid

• Finger interrupts infrared light grid • FTIR • RDI • LLP • DSI

TouchscreensInfrared grid

• FTIR - Frustrated Total Internal Reflection • Infrared light is shined into the

side of an acrylic panel • Light is trapped inside the

acrylic by internal reflection • When a finger touches the

acrylic surface this light is “frustrated”

• Picked up by an infrared camera

TouchscreensInfrared grid

• RDI - Rear Diffused Illumination • Infrared light is shined at the

screen from below the touch surface

• Diffuser is placed on top or bottom of the touch surface

• When a finger touches the surface it reflects more light than diffuser

• Picked up by an infrared camera

TouchscreensInfrared grid

• LLP - Laser Light Plane • Infrared light from a laser(s) is

shined just above the surface • Laser plane of light is about

1mm tick • When a finger touches it, it will

hit the tip of the finger which will register as a IR blob

• Picked up by an infrared camera

TouchscreensInfrared grid

• DSI - Diffused Surface Illumination • Uses a special acrylic to

distribute the IR evenly across the surface

• This acrylic uses small particles that are inside the material, acting like thousands of small mirrors

• When a finger touches the surface it reflects more light

• Picked up by an infrared camera

Humans

Ergonomics of motion• Avoid hyperextension or extreme stretches

• Avoid repetition

• Utilize relaxed, neutral positions

• Avoid staying in a static position

• No “Gorilla Arm”

Arms, hands and fingers

Hands and fingers• Fingernails: Blessing and curse • Fake fingernails: evil • Finger oil • Fingerprints • (Left) Handedness • Wrist support • Gloves • Inaccurate (when compared to a cursor) • Attached to a hand (Screen Coverage)

Screen Coverage• Avoid putting essential features or information like a

label below an interface element that can be touched, as it may become hidden by the user’s own hand

Screen Coverage• Avoid putting essential features or information like a

label below an interface element that can be touched, as it may become hidden by the user’s own hand

Screen Coverage• Activity Zones for Touchscreen Tablets and Phone

• Dan Saffer • http://www.kickerstudio.com/blog/2011/01/

activity-zones-for-touchscreen-tablets-and-phones/

Touch Target Tips

Iceberg tips Adaptive targets