Interaction DevicesInteraction Devices

Human Computer InteractionHuman Computer Interaction

CIS 6930/4930CIS 6930/4930

Interaction PerformanceInteraction Performance► 60s vs. Today60s vs. Today

PerformancePerformance► Hz -> GHzHz -> GHz

MemoryMemory► k -> GBk -> GB

StorageStorage► k -> TBk -> TB

InputInput► punch cards -> punch cards -> ► Keyboards, Pens, tablets, mobile Keyboards, Pens, tablets, mobile

phones, mice, cameras, web camsphones, mice, cameras, web cams OutputOutput

► 10 character/sec ->10 character/sec ->► Megapixel displays, HD capture Megapixel displays, HD capture

and display, color laser, surround and display, color laser, surround sound, force feedback, VRsound, force feedback, VR

► Substantial bandwidth Substantial bandwidth increase!increase!

Interaction PerformanceInteraction Performance► Future?Future?

Gestural inputGestural input Two-handed inputTwo-handed input 3D/6D I/O3D/6D I/O Others: voice, wearable, Others: voice, wearable,

whole body, eye trackers, whole body, eye trackers, data gloves, haptics, force data gloves, haptics, force feedbackfeedback

Engineering research!Engineering research! Entire companies created Entire companies created

around one single technologyaround one single technology► Current trend: Current trend:

Multimodal (using car Multimodal (using car navigation via buttons navigation via buttons oror voice)voice)

Helps disabled (esp. those w/ Helps disabled (esp. those w/ different levels of disability)different levels of disability)

Keyboard and KeypadsKeyboard and Keypads

► QWERTY keyboards been QWERTY keyboards been around for a long timearound for a long time (1870s – Christopher (1870s – Christopher

Sholes)Sholes) Cons: Cons: Not easy to learnNot easy to learn Pros: Pros: FamiliarityFamiliarity Stats:Stats:

► Beginners: 1 keystroke per Beginners: 1 keystroke per secsec

► Average office worker: 5 Average office worker: 5 keystrokes (50 wpm)keystrokes (50 wpm)

► Experts: 15 keystrokes per Experts: 15 keystrokes per sec (150 wpm)sec (150 wpm)

► Is it possible to do Is it possible to do better? better?

Keyboard and KeypadsKeyboard and Keypads► Look at the piano for Look at the piano for

possible inspirationpossible inspiration► Court reporter keyboards Court reporter keyboards

(one keypress = multiple (one keypress = multiple letters or a word) letters or a word) 300 wpm, requires 300 wpm, requires

extensive training and useextensive training and use► How important is:How important is:

AccuracyAccuracy TrainingTraining

► Keyboard properties that Keyboard properties that mattermatter Size Size AdjustabilityAdjustability

► Reduces RSI, better Reduces RSI, better performance and comfortperformance and comfort

Mobile phone keyboards, Mobile phone keyboards, blackberry devices, etc. blackberry devices, etc.

Keyboard LayoutsKeyboard Layouts► QWERTYQWERTY

Frequently used pairs far apartFrequently used pairs far apart Fewer typewriter jamsFewer typewriter jams Electronic approaches don’t jam.. why Electronic approaches don’t jam.. why

use it?use it?► DVOARK (1920s)DVOARK (1920s)

150 wpm->200 wpm150 wpm->200 wpm Reducing errorsReducing errors Takes about one week to switchTakes about one week to switch Stops most from tryingStops most from trying

► ABCDE – styleABCDE – style Easier for non-typistsEasier for non-typists Studies show no improvement vs. Studies show no improvement vs.

QWERTYQWERTY► Number padsNumber pads

What’s in the top row? What’s in the top row? Look at phones (slight faster), then look Look at phones (slight faster), then look

at calculators, keypadsat calculators, keypads► Those for disabledThose for disabled

Split keyboardsSplit keyboards KeyBowl’s orbiTouchKeyBowl’s orbiTouch Eyetrackers, miceEyetrackers, mice Dasher - 2d motion with word predictionDasher - 2d motion with word prediction

KeysKeys

► Current keyboards have Current keyboards have been extensively testedbeen extensively tested SizeSize ShapeShape Required forceRequired force SpacingSpacing

► Speed vs. error rates for Speed vs. error rates for majority of usersmajority of users

► Distinctive click gives Distinctive click gives audio feedbackaudio feedback Why membrane keyboards Why membrane keyboards

are slow (Atari 400?)are slow (Atari 400?)► Environment hazards Environment hazards

might necessitate might necessitate ► Usually speed is not a Usually speed is not a

factorfactor

Keys GuidelinesKeys Guidelines► Special keys should be Special keys should be

denoteddenoted► State keys (such as caps, etc.) State keys (such as caps, etc.)

should have easily noted should have easily noted statesstates

► Special curves or dots for Special curves or dots for home keys for touch typistshome keys for touch typists

► Inverted T Inverted T Cursor movement Cursor movement keys are important (though keys are important (though cross is easier for novices)cross is easier for novices)

► Auto-repeat featureAuto-repeat feature Improves performanceImproves performance But only if repeat is But only if repeat is

customizable (motor customizable (motor impaired, young, old)impaired, young, old)

► Two thinking points:Two thinking points: Why are home keys fastest to Why are home keys fastest to

type?type? Why are certain keys larger? Why are certain keys larger?

(Enter, Shift, Space bar)(Enter, Shift, Space bar)

► This is called This is called Fitt’s LawFitt’s Law

Keypads for small devicesKeypads for small devices► PDAs, Cellphones, Game consolesPDAs, Cellphones, Game consoles► Fold out keyboardsFold out keyboards► Virtual keyboardVirtual keyboard► Cloth keyboards (ElekSen)Cloth keyboards (ElekSen)► Haptic feedback?Haptic feedback?► Mobile phonesMobile phones

Combine static keys with dynamic Combine static keys with dynamic soft keyssoft keys

Multi-tap a key to get to a characterMulti-tap a key to get to a character Study: Study: Predictive techniques Predictive techniques

greatly improve performancegreatly improve performance Ex. LetterWise = 20 wpm vs 15 wpm Ex. LetterWise = 20 wpm vs 15 wpm

multitapmultitap► Draw keyboard on screen and tap Draw keyboard on screen and tap

w/ penw/ pen Speed: 20 to 30 wpm (Sears ’93)Speed: 20 to 30 wpm (Sears ’93)

► Handwriting recognition (still hard)Handwriting recognition (still hard) Subset: Graffiti2 (uses Subset: Graffiti2 (uses unistrokesunistrokes))

Pointing DevicesPointing Devices► Direct manipulation needs some pointing Direct manipulation needs some pointing

devicedevice► Factors:Factors:

Size of deviceSize of device AccuracyAccuracy DimensionalityDimensionality

► Interaction Tasks:Interaction Tasks: Select – menu selection, from a listSelect – menu selection, from a list Position – 1D, 2D, 3D (ex. paint)Position – 1D, 2D, 3D (ex. paint) Orientation – Control orientation or provide Orientation – Control orientation or provide

direct 3D orientation inputdirect 3D orientation input Path – Multiple poses are recorded Path – Multiple poses are recorded

► ex. to draw a lineex. to draw a line Quantify – control widgets that affect Quantify – control widgets that affect

variablesvariables Text – move textText – move text

► Faster w/ less error than keyboardFaster w/ less error than keyboard► Two types (Box 9.1)Two types (Box 9.1)

Direct control Direct control – device is on the screen – device is on the screen surface (touchscreen, stylus)surface (touchscreen, stylus)

Indirect control Indirect control – mouse, trackball, – mouse, trackball, joystick, touchpadjoystick, touchpad

Direct-control pointingDirect-control pointing► First device – lightpenFirst device – lightpen

Point to a place on screen and press Point to a place on screen and press a buttona button

Pros: Pros: ► Easy to understand and useEasy to understand and use► Very fast for some operations (e.g. Very fast for some operations (e.g.

drawing)drawing) Cons: Cons:

► Hand gets tired fast! Hand gets tired fast! ► Hand and pen blocks view of screenHand and pen blocks view of screen► FragileFragile

► Evolved into the touchscreen Evolved into the touchscreen Pros: Pros: Very robust, no moving partsVery robust, no moving parts Cons: Cons: Depending on app, accuracy Depending on app, accuracy

could be an issue could be an issue ► 1600x1600 res with acoustic wave1600x1600 res with acoustic wave

Must be careful about software Must be careful about software design for selection (land-on design for selection (land-on strategy).strategy).► If you don’t show a cursor of where If you don’t show a cursor of where

you are selecting, users get confusedyou are selecting, users get confused User confidence is improved with a User confidence is improved with a

good lift-off strategygood lift-off strategy

Direct-control pointingDirect-control pointing

► Primarily for novice Primarily for novice users or large user users or large user basebase

► Case study: Disney Case study: Disney WorldWorld

►Need to consider Need to consider those who are: those who are: disabled, illiterate, disabled, illiterate, hard of hearing, hard of hearing, errors in usage (two errors in usage (two touch points), etc.touch points), etc.

Indirect-Control PointingIndirect-Control Pointing► Pros: Pros:

Reduces hand-fatigue Reduces hand-fatigue Reduces obscuration problemsReduces obscuration problems

► Cons: Cons: Increases cognitive load Increases cognitive load Spatial ability comes more into playSpatial ability comes more into play

► MouseMouse Pros:Pros:

► FamiliarityFamiliarity► Wide availabilityWide availability► Low costLow cost► Easy to useEasy to use► AccurateAccurate

Cons:Cons:► Time to grab mouseTime to grab mouse► Desk spaceDesk space► Encumbrance (wire), dirtEncumbrance (wire), dirt► Long motions aren’t easy or obvious (pick up and replace)Long motions aren’t easy or obvious (pick up and replace)

Consider, weight, size, style, # of buttons, force feedbackConsider, weight, size, style, # of buttons, force feedback

Indirect-Control PointingIndirect-Control Pointing► TrackballTrackball

Pros:Pros:►Small physical footprintSmall physical footprint►Good for kiosksGood for kiosks

► JoystickJoystick Easy to use, lots of buttonsEasy to use, lots of buttons Good for tracking (guide or Good for tracking (guide or

follow an on screen object)follow an on screen object) Does it map well to your Does it map well to your

app?app?► TouchpointTouchpoint

Pressure-sensitive ‘nubbin’ Pressure-sensitive ‘nubbin’ on laptopson laptops

Keep fingers on the home Keep fingers on the home positionposition

Indirect-Control PointingIndirect-Control Pointing

► TouchpadTouchpad Laptop mouse deviceLaptop mouse device Lack of moving Lack of moving

parts, and low profileparts, and low profile Accuracy, esp. those Accuracy, esp. those

w/ motor disabilitiesw/ motor disabilities►Graphics TabletGraphics Tablet

Screen shotScreen shot comfortcomfort good for cad, artistsgood for cad, artists Limited data entryLimited data entry

Comparing pointing devicesComparing pointing devices

► Direct pointingDirect pointing Study: Faster but less accurate than indirect (Haller ’84)Study: Faster but less accurate than indirect (Haller ’84)

► Lots of studies confirm mouse is best for most tasks Lots of studies confirm mouse is best for most tasks for speed and accuracyfor speed and accuracy

► Trackpoint < Trackballs & Touchpads < MouseTrackpoint < Trackballs & Touchpads < Mouse► Short distances – cursor keys are betterShort distances – cursor keys are better► Disabled prefer joysticks and trackballs Disabled prefer joysticks and trackballs

If force application is a problem, then touch sensitive is If force application is a problem, then touch sensitive is preferredpreferred

Vision impaired have problems with most pointing devices Vision impaired have problems with most pointing devices ► Use multimodal approach or customizable cursorsUse multimodal approach or customizable cursors► Read Vanderheiden ’04 for a case studyRead Vanderheiden ’04 for a case study

► Designers should smooth out trajectoriesDesigners should smooth out trajectories► Large targets reduce time and frustrationLarge targets reduce time and frustration

ExampleExample►Five fastest places to click on for a Five fastest places to click on for a

right-handed user?right-handed user?

ExampleExample►What affects time?What affects time?

Fitts’s LawFitts’s Law► Paul Fitts (1954) developed a model of human Paul Fitts (1954) developed a model of human

hand movementhand movement► Used to predict time to point at an objectUsed to predict time to point at an object► What are the factors to determine the time to What are the factors to determine the time to

point to an object?point to an object? D – distance to targetD – distance to target W – size of targetW – size of target

► Just from your own experience, is this function Just from your own experience, is this function linear?linear? No, since if Target A is No, since if Target A is DD distance and Target B is distance and Target B is

2D2D distance, it doesn’t take twice as long distance, it doesn’t take twice as long What about target size? Not linear there eitherWhat about target size? Not linear there either

► T = a + b logT = a + b log22(D/W + 1)(D/W + 1) T = mean timeT = mean time a = time to start/stop in seconds a = time to start/stop in seconds

(empirically measured per device)(empirically measured per device) b = inherent speed of the device b = inherent speed of the device

(empirically measured per device)(empirically measured per device) Ex. a = 300 ms, b = 200 ms/bit, D = 14 cm, Ex. a = 300 ms, b = 200 ms/bit, D = 14 cm,

W = 2 cmW = 2 cm► Ans: 300 + 200 logAns: 300 + 200 log22(14/2 + 1) = 900 ms(14/2 + 1) = 900 ms

Really a slope-intercept modelReally a slope-intercept model

Fitts’s LawFitts’s Law► T = a + b logT = a + b log22(D/W + 1)(D/W + 1)

T = mean timeT = mean time a = time to start/stop in seconds (empirically measured a = time to start/stop in seconds (empirically measured

per device)per device) b = inherent speed of the device (empirically measured b = inherent speed of the device (empirically measured

per device) [time/bit or ms/bit]per device) [time/bit or ms/bit] Ex. a = 300 ms, b = 200 ms/bit, D = 14 cm, W = 2 cmEx. a = 300 ms, b = 200 ms/bit, D = 14 cm, W = 2 cm

► Ans: 300 + 200 logAns: 300 + 200 log22(14/2 + 1) = 900 ms(14/2 + 1) = 900 ms Question: If I wanted to half the pointing time (on average), how Question: If I wanted to half the pointing time (on average), how

much do I change the size?much do I change the size?► Proven to provide good timings for most age groupsProven to provide good timings for most age groups► Newer versions taken into account Newer versions taken into account

Direction (we are faster horizontally than vertically)Direction (we are faster horizontally than vertically) Device weightDevice weight Target shapeTarget shape Arm position (resting or midair)Arm position (resting or midair) 2D and 3D (Zhai ’96)2D and 3D (Zhai ’96)

ExamplesExamples

►T = a + b logT = a + b log22(D/W + 1)(D/W + 1)

a=300, b= 200, X, W = 10

0

100

200

300

400

500

600

700

800

Distance (cm)

Tim

e (

ms

)

ExamplesExamples

►T = a + b logT = a + b log22(D/W + 1)(D/W + 1)a=300, b= 200, D=30, X

0

200

400

600

800

1000

1200

1400

Distance (cm)

Tim

e (

ms

)

ExamplesExamplesBLUE a=300, b=200, D=15, W=[1-30]

PURPLE a=300, b=200, D=[1-30], W=15

0

200

400

600

800

1000

1200

Variable

Tim

e (

ms)

Fitts’s LawFitts’s Law►T = a + b logT = a + b log22(D/W + 1)(D/W + 1)

T = mean timeT = mean time a = time to start/stop in seconds a = time to start/stop in seconds

(empirically measured per device)(empirically measured per device) b = inherent speed of the device b = inherent speed of the device

(empirically measured per device) (empirically measured per device) [time/bit or ms/bit][time/bit or ms/bit]

First part is device characteristicsFirst part is device characteristics Second part is target difficultySecond part is target difficulty

Very Successfully StudiedVery Successfully Studied

► Applies toApplies to Feet, eye gaze, head mounted sightsFeet, eye gaze, head mounted sights Many types of input devicesMany types of input devices Physical environments (underwater!)Physical environments (underwater!) User populations (even retarded and User populations (even retarded and

drugged)drugged) Drag & Drop and Point & ClickDrag & Drop and Point & Click

► LimitationsLimitations DimensionalityDimensionality Software accelerated pointer motionSoftware accelerated pointer motion TrainingTraining Trajectory Tasks (Accot-Zhai Steering Trajectory Tasks (Accot-Zhai Steering

Law is a good predictor and joins Law is a good predictor and joins Fitt’s Law)Fitt’s Law)

Decision Making (Hick’s Law)Decision Making (Hick’s Law)

Very Successfully StudiedVery Successfully Studied

► Results (what does it say Results (what does it say about)about) Buttons and widget size?Buttons and widget size? Edges?Edges? Popup vs. pull-down menusPopup vs. pull-down menus Pie vs. Linear menusPie vs. Linear menus iPhone/web pages (real borders) iPhone/web pages (real borders)

vs. monitor+mouse (virtual vs. monitor+mouse (virtual borders)borders)

► Interesting readings:Interesting readings: http://particletree.com/features/vihttp://particletree.com/features/vi

sualizing-fittss-law/sualizing-fittss-law/ http://www.asktog.com/columns/http://www.asktog.com/columns/

022DesignedToGiveFitts.html022DesignedToGiveFitts.html http://www.yorku.ca/mack/http://www.yorku.ca/mack/

GI92.htmlGI92.html

Precision Pointing Movement Precision Pointing Movement TimeTime

► Study: Sears and Shneiderman ’91 Study: Sears and Shneiderman ’91 Broke down task into gross and fine components for small Broke down task into gross and fine components for small

targetstargets Precision Point Mean Time = a + b logPrecision Point Mean Time = a + b log22(D/W+1) + c log(D/W+1) + c log22(d/W)(d/W)

► c – speed for short distance movementc – speed for short distance movement► d – minor distanced – minor distance

Notice how the overall time changes with a smaller target.Notice how the overall time changes with a smaller target.► Other factorsOther factors

Age (Pg. 369)Age (Pg. 369)► Research: How can we design devices that produce Research: How can we design devices that produce

smaller constants for the predictive equationsmaller constants for the predictive equation Two handedTwo handed ZoomingZooming

AffordanceAffordance

►Quality of an object, or an Quality of an object, or an environment, that allows an individual environment, that allows an individual to perform an action. to perform an action.

►Gibson (’77) – Gibson (’77) – perceived perceived action action possibilitiespossibilities

►Norman – The Design of Everyday Norman – The Design of Everyday ThingsThings

Affordance ExamplesAffordance Examples

Affordance ExamplesAffordance Examples

http://jared-donovan.com/teaching/blog/hci

Affordances Matter?Affordances Matter?

►When would affordances matter?When would affordances matter? LanguagesLanguages EmergenciesEmergencies

http://jared-donovan.com/teaching/blog/hci

Novel DevicesNovel Devices► Themes:Themes:

Make device more diverseMake device more diverse► UsersUsers► TaskTask

Improve match between task Improve match between task and deviceand device

Improve affordanceImprove affordance Refine inputRefine input Feedback strategiesFeedback strategies

► Foot controlsFoot controls Already used in music where Already used in music where

hands might be busyhands might be busy CarsCars Foot mouse was twice as Foot mouse was twice as

slow as hand mouseslow as hand mouse Could specify ‘modes’Could specify ‘modes’

Novel DevicesNovel Devices► Eye-trackingEye-tracking

Accuracy 1-2 degreesAccuracy 1-2 degrees selections are by constant selections are by constant

stare for 200-600 msstare for 200-600 ms How do you distinguish w/ How do you distinguish w/

a selection and a gaze?a selection and a gaze? Combine w/ manual inputCombine w/ manual input

► Multiple degree of Multiple degree of freedom devicesfreedom devices Logitech Spaceball and Logitech Spaceball and

SpaceMouseSpaceMouse Ascension BirdAscension Bird Polhemus Liberty and Polhemus Liberty and

IsoTrackIsoTrack

Novel DevicesNovel Devices► Boom ChameleonBoom Chameleon

Pros: Pros: Natural, good Natural, good spatial understandingspatial understanding

Cons: Cons: limited limited applications, hard to applications, hard to interact (very passive) interact (very passive)

► DataGloveDataGlove Pinch glovePinch glove Gesture recognitionGesture recognition American Sign Language, American Sign Language,

musical directormusical director Pros: Pros: NaturalNatural Cons:Cons: Size, hygiene, Size, hygiene,

accuracy, durabilityaccuracy, durability

Novel DevicesNovel Devices► Haptic FeedbackHaptic Feedback

Why is resistance useful?Why is resistance useful? SensAble Technology’s SensAble Technology’s

PhantomPhantom Cons: limited Cons: limited

applicationsapplications Sound and vibration are Sound and vibration are

easier and can be a good easier and can be a good approximationapproximation► Rumble packRumble pack

► Two-Handed inputTwo-Handed input Different hands have Different hands have

different precisiondifferent precision Non-dominant hand Non-dominant hand

selects fill, the other selects fill, the other selects objectsselects objects

Ubiquitous Computing and Ubiquitous Computing and Tangible User InterfacesTangible User Interfaces

► Active Badges allows Active Badges allows you to move about you to move about the house w/ your the house w/ your profileprofile

►Which sensors could Which sensors could you use?you use?

► Elderly, disabledElderly, disabled► Research: Smart Research: Smart

HouseHouse►Myron Kruger – novel Myron Kruger – novel

user participation in user participation in art (Lots of exhibit art art (Lots of exhibit art at siggraph)at siggraph) http://www.linuxjournal.com/files/linuxjournal.com/linuxjournal/articles/030/3047/3047f2.png

Novel DevicesNovel Devices

► Paper/WhiteboardsPaper/Whiteboards Video capture of annotationsVideo capture of annotations Record notes (special tracked Record notes (special tracked

pens Logitech digital pen)pens Logitech digital pen)

► Handheld DevicesHandheld Devices PDAPDA Universal remoteUniversal remote Help disabledHelp disabled

► Read LCD screensRead LCD screens► Rooms in buildingRooms in building► MapsMaps

Interesting body-context-Interesting body-context-sensitive. sensitive. ► Ex. hold PDA by ear = phone Ex. hold PDA by ear = phone

call answer.call answer.

Novel DevicesNovel Devices

►MiscellaneousMiscellaneous Shapetape – reports Shapetape – reports

3D shape. 3D shape. ►Tracks limbsTracks limbs

► Engineer for specific Engineer for specific app (like a gun app (like a gun trigger connected to trigger connected to serial port)serial port) Pros: good Pros: good

affordanceaffordance Cons: Limited Cons: Limited

general use, timegeneral use, time

Speech and Auditory Speech and Auditory InterfacesInterfaces

► There’s the dreamThere’s the dream► Then there’s realityThen there’s reality► Practical apps don’t really Practical apps don’t really

require freeform discussions require freeform discussions with a computerwith a computer Goals: Goals:

► Low cognitive loadLow cognitive load► Low error ratesLow error rates

► Smaller goals:Smaller goals: Speech Store and Forward (voice Speech Store and Forward (voice

mail)mail) Speech GenerationSpeech Generation Currently not too bad, low cost, Currently not too bad, low cost,

availableavailable

Speech and Auditory Speech and Auditory InterfacesInterfaces

► Ray Kurzweil (’87) – first commercial Ray Kurzweil (’87) – first commercial speech recognition softwarespeech recognition software

► Bandwidth is much lower than visual Bandwidth is much lower than visual displaysdisplays

► Ephemeral nature of speech (tone, Ephemeral nature of speech (tone, etc.)etc.)

► Difficulty in parsing/searching (Box Difficulty in parsing/searching (Box 9.2)9.2)

► TypesTypes Discrete-word recognitionDiscrete-word recognition Continuous speechContinuous speech Voice informationVoice information Speech generationSpeech generation Non-speech auditoryNon-speech auditory

► If you want to do research here, If you want to do research here, review research in:review research in: AudioAudio Audio psychologyAudio psychology Digital signal processingDigital signal processing

http://www.kurzweiltech.com/raybio.html

Discrete-Word RecognitionDiscrete-Word Recognition

► Individual words spoken by a specific personIndividual words spoken by a specific person► Command and controlCommand and control► 90-98% for 100-10000 word vocabularies90-98% for 100-10000 word vocabularies► TrainingTraining

Speaker speaks the vocabularySpeaker speaks the vocabulary Speaker-independentSpeaker-independent

► Still requiresStill requires Low noise operating environmentLow noise operating environment MicrophonesMicrophones Vocabulary choiceVocabulary choice Clear voice (language disabled are hampered, stressed)Clear voice (language disabled are hampered, stressed) Reduce most questions to very distinct answers (yes/no)Reduce most questions to very distinct answers (yes/no)

Discrete-Word RecognitionDiscrete-Word Recognition► Helps:Helps:

DisabledDisabled ElderlyElderly Cognitive challengedCognitive challenged User is visually distractedUser is visually distracted Mobility or space restrictionsMobility or space restrictions

► Apps:Apps: Telephone-based infoTelephone-based info

► Study: Study: much slower for cursor movement than mouse or much slower for cursor movement than mouse or keyboard (Christian ’00)keyboard (Christian ’00)

► Study: Study: choosing actions (such as drawing actions) improved choosing actions (such as drawing actions) improved performance by 21% (Pausch ’91) and word processing (Karl performance by 21% (Pausch ’91) and word processing (Karl ’93)’93) However acoustic memory requires high cognitive load (> than However acoustic memory requires high cognitive load (> than

hand/eye)hand/eye)► Toys are successful (dolls, robots). Accuracy isn’t as importantToys are successful (dolls, robots). Accuracy isn’t as important► Feedback is difficultFeedback is difficult

Continuous Speech Continuous Speech RecognitionRecognition

► DictationDictation► Error rates and error repair are still poorError rates and error repair are still poor► Higher cognitive load, could lower overall qualityHigher cognitive load, could lower overall quality► Why is it hard?Why is it hard?

Recognize boundaries (normal speech blurs them)Recognize boundaries (normal speech blurs them) Context sensitivityContext sensitivity ““How to wreck a nice beach”How to wreck a nice beach”

► Much trainingMuch training► Specialized vocabularies (like medical or legal)Specialized vocabularies (like medical or legal)► Apps:Apps:

Dictate reports, notes, lettersDictate reports, notes, letters Communication skills practice (virtual patient)Communication skills practice (virtual patient) Automatic retrieval/transcription of audio content (like radio, CC)Automatic retrieval/transcription of audio content (like radio, CC) Security/user IDSecurity/user ID

Voice Information SystemsVoice Information Systems► Use human voice as a source of infoUse human voice as a source of info► Apps:Apps:

Tourist infoTourist info Museum audio toursMuseum audio tours Voice menus (Interactive Voice Response Voice menus (Interactive Voice Response

IVR systems)IVR systems)► Use speech recognition to also cut through Use speech recognition to also cut through

menusmenus If menus are too long, users get frustratedIf menus are too long, users get frustrated Cheaper than hiring 24 hr/day repsCheaper than hiring 24 hr/day reps

► Voice mail systemsVoice mail systems Interface isn’t the bestInterface isn’t the best

► Get email in your carGet email in your car Also helps with non-tech savvy like the Also helps with non-tech savvy like the

elderlyelderly► Potentially aides withPotentially aides with

Learning (engage more senses)Learning (engage more senses) Cognitive load (hypothesize each sense has Cognitive load (hypothesize each sense has

a limited ‘bandwidth’)a limited ‘bandwidth’)► Think ER, or fighter jetsThink ER, or fighter jets

Speech GenerationSpeech Generation

► Play back speech (games)Play back speech (games)► Combine text (navigation systems)Combine text (navigation systems)► Careful evaluation!Careful evaluation!

Speech isn’t always greatSpeech isn’t always great►Door is ajar – now just a toneDoor is ajar – now just a tone►Use flashUse flash►Supermarket scannersSupermarket scanners

Often times a simple tone is betterOften times a simple tone is better Why? Cognitive loadWhy? Cognitive load

►Thus cockpits and control rooms need speechThus cockpits and control rooms need speech►Competes w/ human-human communicationCompetes w/ human-human communication

Speech GenerationSpeech Generation► Ex: Text-to-Speech (TTS)Ex: Text-to-Speech (TTS)► Latest TTS uses multiple syllabi to make generated speech Latest TTS uses multiple syllabi to make generated speech

sound bettersound better Robotic speech could be desirable to get attentionRobotic speech could be desirable to get attention All depends on appAll depends on app Thus don’t assume one way is the best, you should user testThus don’t assume one way is the best, you should user test

► Apps: Apps: TTS for blind, JAWSTTS for blind, JAWS► Web-based voice apps: VoiceXML and SALT (tagged web Web-based voice apps: VoiceXML and SALT (tagged web

pages). pages). Good for disabled, and also for mobile devicesGood for disabled, and also for mobile devices

► Use ifUse if Message is shortMessage is short Requires dynamic responsesRequires dynamic responses Events in timeEvents in time

► Good when visual displays aren’t that useful. When?Good when visual displays aren’t that useful. When? Bad lighting, vibrations (say liftoff)Bad lighting, vibrations (say liftoff)

Non-speech Auditory Non-speech Auditory InterfaceInterface

► Audio tones that provide informationAudio tones that provide information►Major Research AreaMajor Research Area

Sonification – converting information into audioSonification – converting information into audio AudiolizationAudiolization Auditory InterfacesAuditory Interfaces

► Browsers produced a click when you clicked Browsers produced a click when you clicked on a linkon a link Increases confidenceIncreases confidence Can do tasks without visual cognitive loadCan do tasks without visual cognitive load Helps figure out when things are wrongHelps figure out when things are wrong Greatly helps visually impairedGreatly helps visually impaired

Non-speech Auditory Non-speech Auditory InterfaceInterface

► Terms:Terms: Auditory icons – familiar Auditory icons – familiar

sounds (record real world sounds (record real world sound and play it in your sound and play it in your app)app)

Earcons – new learned Earcons – new learned sounds (door ajar)sounds (door ajar)

► Role in video games is hugeRole in video games is huge Emotions, Tension, set moodEmotions, Tension, set mood

► To create 3D soundTo create 3D sound Need to do more than stereoNeed to do more than stereo Take into account Head-Take into account Head-

related transfer function related transfer function (HRTF)(HRTF)► Ear and head shapeEar and head shape

► New musical instrumentsNew musical instruments ThereminTheremin

► New ways to arrange musicNew ways to arrange music

DisplaysDisplays

► Primary Source of Primary Source of feedbackfeedback

► Properties:Properties: Physical DimensionPhysical Dimension ResolutionResolution Color Depth and Color Depth and

correctnesscorrectness Brightness, contrast, Brightness, contrast,

glareglare PowerPower Refresh rateRefresh rate CostCost ReliabilityReliability # of users# of users

Display Display TechnologyTechnology

► Monochrome displays Monochrome displays (single color)(single color) Low costLow cost Greater intensity range Greater intensity range

(medical)(medical)► ColorColor

Raster Scan CRTRaster Scan CRT LCD – thin, brightLCD – thin, bright Plasma – very bright, thinPlasma – very bright, thin LED – large public displaysLED – large public displays Electronic Ink – new product Electronic Ink – new product

w/ tiny capsules of negative w/ tiny capsules of negative black particles and positive black particles and positive whitewhite

Braille – refreshable cells with Braille – refreshable cells with dots that rise updots that rise up

Large DisplaysLarge Displays►Wall displaysWall displays

InformationalInformational►Control rooms, military, flight Control rooms, military, flight

control rooms, emergency control rooms, emergency responseresponse

►ProvidesProvides System overview System overview Increases situational Increases situational

awarenessawareness Effective team reviewEffective team review

InteractiveInteractive►Require new interaction Require new interaction

methods (freehand sketch, methods (freehand sketch, PDAs)PDAs)

►Local and remote collaborationLocal and remote collaboration►Art, engineeringArt, engineering

Large DisplaysLarge Displays

►Multiple Desktop DisplaysMultiple Desktop Displays Multiple CRTs or Flat panels Multiple CRTs or Flat panels

for large desktopsfor large desktops CheapCheap FamiliarFamiliar Spatial divide up tasksSpatial divide up tasks Comparison tasks are easierComparison tasks are easier Too much info?Too much info?

► Eventually -> Every surface Eventually -> Every surface a pixela pixel

Mobile device displaysMobile device displays

► PersonalPersonal Reprogrammable picture Reprogrammable picture

framesframes► Digital family portrait Digital family portrait

(GaTech)(GaTech)

► MedicalMedical Monitor patientsMonitor patients

► Research: Research: Modality Modality Translation Services Translation Services (Trace Center – (Trace Center – University of University of Wisconsin)Wisconsin) As you move about it As you move about it

auto converts data, info, auto converts data, info, etc. for youetc. for you

Mobile device displays Mobile device displays guidlinesguidlines

► Bergman ’00, Weiss, ’02Bergman ’00, Weiss, ’02► Industry led research and design case Industry led research and design case

studies (Lindholm ’03)studies (Lindholm ’03)► Typically short in time usage (except Typically short in time usage (except

handheld games)handheld games)► Optimize for repetitive tasks (rank Optimize for repetitive tasks (rank

functions by frequency)functions by frequency)► Research: Research: new ways to organize large new ways to organize large

amounts of info on a small screenamounts of info on a small screen► Study: Study: Rapid Serial Visual Presentation Rapid Serial Visual Presentation

(RSVP) presents text at a constant (RSVP) presents text at a constant speed (33% improvement Oquist ’03)speed (33% improvement Oquist ’03)

► Searching and web browsing still very Searching and web browsing still very poor performancepoor performance

► Promising: Hierarchical representation Promising: Hierarchical representation (show full document and allow user to (show full document and allow user to select where to zoom into)select where to zoom into)

3D Printing3D Printing► Create custom objects Create custom objects

from 3D modelsfrom 3D models► Create physical models Create physical models

forfor Design reviewDesign review ConstructionConstruction