designing easily learnable eyes-free interaction

Kevin Li University of California, San Diego

can we make easily learnable eyes-free interfaces by leveraging existing learned associations?

touchtouch

mapping eyes free interactionmapping eyes free interaction

humanobject

hearinghearing

music speech sounds

Tapping and Rubbing

Tapping and Rubbing

Simulating buttons

Simulating buttons

PeopleTonesPeopleTones

Mapping language to tactile

Mapping language to tactile

stimulus class

sense

example project

PeopleTonesPeopleTones

lessons for application designerslessons for application designers

why eyes free?

PCs…PC screens have the users’ undivided attention design for the visual channel

environment

screen-less device

can’t seescreen

visual impairment

social factors

easily learnable?

binary is easy

Vibratese Language - Communicate via vibrations

Map alphabet and digits to 5 vibrators

user with 65 hours training -> 90% accuracy, 38wpm

System capable of delivering at 67wpm

anything can be learned

vibratese language

• Ok, so let’s just translate all text from visual to auditory– Easily learnable– High bandwidth

• Wait, that sounds familiar

auditory feedback

Please listen carefully as ouroptions have changed…

blindSightevaluation

interfaces

Smartphone 2003 (sighted)BlindSight (eyes-free) vs.vs.

task

while “driving”idle

(1) schedule appointments and (2) add contacts

0 1 2 3 4 5 6 7 8

Was not missing information

Knew position in the menu

Knew what day/time I was at

Felt in control of the conversation

Better for setting meeting times

Prefer if driving and talking

Prefer Overall

blindSight Smartphone

Overall preference

results

1. brevity is good, but use in moderationclarification of navigation overrides brevity

2. predictable/modeless user interface is key

3. auditory feedback goes a long way even during phone call(disclaimer: need to study how it interferes with activities… driving)

lessons

• Requires direct interaction with device

• Not usable in all scenarios– Loud environments (concerts)– Social environments– Driving and talking on the phone

auditory feedback doesn’t always work

what about touch?

replace visual applications…

… new applications?

human-to-humaninteraction

Tapping andRubbing

Can we create tactile stimuli that feels like tapping or rubbing?

Will people associate it with what they already associate with tapping and rubbing in human-human communication?

research questions

rubbing head(under user’s hand)

soundTouch

voice coil motor

hammer

Tapping

Rubbing

audio out

voice coil motor amplifierhead

soundTouch

cardboard

battery(metal)

eraser

IBM track pointer (sandpaper texture)

foam

IBM track pointer(rubber)

what feels like a finger?

arm

voice coil motor

head

a b

arm motionhead

b

applications

in-car navigation

Task 1: Distinguish

Given two taps of different

strength, which one feels stronger?

Given two taps of different frequency, which one feels faster?

error %for distinguish amplitude

1 2 3 4 5 6 7 1 2 2

2 0 10 0

3 2 6 17 4

4 6 17 15 6

5 14 6 29 8

6 4 13 25

7 2 21

Level of first stimulus

Leve

l of s

econ

d sti

mul

us

1 2 3 4 5 6 7 1 0 4

2 4 4 0

3 0 8 3 0

4 4 17 4 4

5 4 4 8 4

6 0 0 17

7 8 0

1 2 3 4 5 6 7 1 0 4

2 8 15 4

3 0 0 13 8

4 0 0 13 13

5 4 13 46 29

6 8 13 63

7 8 17

error %for distinguish frequency

Level of first stimulus

Level of first stimulus

Leve

l of s

econ

d sti

mul

us

Leve

l of s

econ

d sti

mul

us

ConstantDuration

ConstantNumber

Task 2: Identify

Given a tap rate its strength on a scale of 1-7.

Given a tap rate its speedon a scale of 1-7.

0

1

2

3

4

5

6

7

8

1 2 3 4 5 6 7

Use

r Rep

orte

d Le

vel

Stimulus Level

Tapping - Amplitude Identification

01234567

1 2 3 4 5 6 7

Use

r Rep

orte

d Le

vel

Stimulus Level

Tapping - ConstantNumber Identification

01234567

1 2 3 4 5 6 7

Use

r Rep

orte

d Le

vel

Stimulus Level

Tapping - ConstantDuration Identification

questionnaire

How would you describe the tactile sensations you just experienced to someone who has not experienced them?

Which aspects of the experience felt natural and which aspects did not?

If your phone could generate these types of sensations, what would you like to use them for?

Tapping has human quality– 13 of 16 used the word ‘tap” in their description– 12 of 16 volunteered it had a human quality

Fast and slow are perceptually different– 12 participants mentioned harder taps don’t feel natural– 5 said fast ones don’t feel natural

Good for alerts– 6 volunteered that single taps would be good for silent

environments– 7 volunteered they would be useful when vibrations can’t

be felt (walking around)

results

Taps have a number of characteristics that make them desirable– Quiet– Strong

Rubbing is more subtle– Useful for in-the-hand scenarios

Number of taps and rubs is key element– Sometimes, this has pre-learned meaning– Limits number of viable distinct icons

design implications

mapping musicto vibrations

PeopleTones:Buddy Proximity Notification

only two states, nearby and far away

when a buddy is near, play their song

if phone is in vibrate mode, vibrate the equivalent

PeopleTones

measuring vibrations

0 5 10 15 20 25 30 35 40 45 50

Time (ms)

Vo

lts

(V

)0

1

.5

3.0

4.5

6.

0

0 5 10 15 20 25 30 35 40 45 50

Time (ms)

Vo

lts

(V

)0

1

.5

3.0

4.5

6.

0

0 5 10 15 20 25 30 35 40 45 50

Time (ms)

Vo

lts

(V

)0

1

.5

3.0

4.5

6.

0

0 5 10 15 20 25 30 35 40 45 50

Time (ms)

Vo

lts

(V

)0

1

.5

3.0

4.5

6.

0

Duty Cycle = 100%

Duty Cycle = 60%

Duty Cycle = 18%

Duty Cycle = 37%

generating vibrations

capturing essence of music

just using beat doesn’t always work

mapping lyrics doesn’t work well

remove noise

isolate 6.6kHz to 17.6kHz components using 8th order Butterworth Filter

use amplitude threshold, to keep only components greater than the average

takerunning sum

take running sum of absolute value, generate 1 value every 20ms

this keeps length consistent

exaggeratefeatures

compose output from previous step with power function

Axn , x is sample, A and n are constants,

10<=A<15, 1<=n<=2

field study

3 groups of friends, 2 weeks

response to the cue

response to the cue’s information

lessons

cues in the wild should be music

higher comprehension rate when users select their own cues

obtrusiveness of music cues was not a concern

mapping music to vibration was most successful for people who knew the songs well

semantic association is key

mapping physical objects to tactile feedback

problem

tactile feedback is always the same, but visual and motor has directional aspect

information is lost in the conversion

• Add state information to tactile feedback– Hover state– Moved to the left/right/up/down

• Where do we put it?– Under the button– Make the sides move– Tapping?– solenoid

solution

mapping speech to touch

researchquestions

what are relevant characteristics of speech when mapping to tactile?

how do users naturally perceive these to be mapped?

pilot study

5 common phrases from text messaging literature

20 vibration sequences

Which phrase does this vibration feel like?

5 phrases

Hello.

Goodbye.

Where are you?

Are you busy?

I miss you.

lessons

intonation is important

syllables should match number of pulses

duration should match (roughly)

potential applications

learning to sign

augmented sms messages

messaging backchannel

thesissummary

touchtouch

mapping eyes free interactionmapping eyes free interaction

humanobject

hearinghearing

music speech sounds

Tapping and Rubbing

Tapping and Rubbing

Simulating buttons

Simulating buttons

PeopleTonesPeopleTones

Mapping language to tactile

Mapping language to tactile

stimulus class

sense

example project

PeopleTonesPeopleTones

lessons for application designerslessons for application designers

timeline

Jan Apr Jun Sept Dec

2008

Ubicomp Doctoral colloquium

Paper Deadlines

Projects

UISTDoctoral symposium

CHI Doctoral consortium

CHI

Doctoral Symposiums

Mapping language to

tactile

Mapping language to

tactile

Simulating buttons

Simulating buttons

2009

Paper Deadlines

Projects

Jan Apr Jun Sept Dec

Mapping language to tactilePart 2: Field Deployment

Mapping language to tactilePart 2: Field Deployment

CHIUIST

Simulating buttons

Simulating buttons Find a jobFind a job

2010

Paper Deadlines

Projects

Jan Apr Jun Sept Dec

Dissertation

Find a jobFind a job

Formalize music mapping

designing easily learnable eyes-free interaction

Kevin Li University of California, San Diego