Interaction DevicesInteraction Devices

Human Computer InteractionHuman Computer Interaction

CIS 6930/4930CIS 6930/4930

Section 4188/4186Section 4188/4186

Interaction PerformanceInteraction Performance► 60s vs. Today60s vs. Today

PerformancePerformance► Hz -> GHzHz -> GHz

MemoryMemory► k -> GBk -> GB

StorageStorage► k -> TBk -> TB

InputInput► punch cards -> punch cards -> ► Keyboards, Pens, tablets, mobile Keyboards, Pens, tablets, mobile

phones, mice, digital cameras, web phones, mice, digital cameras, web camscams

OutputOutput► 10 character/sec10 character/sec► Megapixel displays, color laser, Megapixel displays, color laser,

surround sound, force feedback, surround sound, force feedback, VRVR

► Substantial bandwidth Substantial bandwidth increase!increase!

Interaction PerformanceInteraction Performance► Future?Future?

Gestural inputGestural input Two-handed inputTwo-handed input 3D I/O3D I/O Others: voice, wearable, Others: voice, wearable,

whole body, eye trackers, whole body, eye trackers, data gloves, haptics, force data gloves, haptics, force feedbackfeedback

Engineering research!Engineering research! Entire companies created Entire companies created

around one single technologyaround one single technology► Current trend: Current trend:

Multimodal (using car Multimodal (using car navigation via buttons navigation via buttons oror voice)voice)

Helps disabled (esp. those w/ Helps disabled (esp. those w/ different levels of disability)different levels of disability)

Keyboard and KeypadsKeyboard and Keypads

► QWERTY keyboards been QWERTY keyboards been around for a long timearound for a long time (1870s – Christopher (1870s – Christopher

Sholes)Sholes) Cons: Cons: Not easy to learnNot easy to learn Pros: Pros: FamiliarityFamiliarity Stats:Stats:

► Beginners: 1 keystroke per Beginners: 1 keystroke per secsec

► Average office worker: 5 Average office worker: 5 keystrokes (50 wpm)keystrokes (50 wpm)

► Experts: 15 keystrokes per Experts: 15 keystrokes per sec (150 wpm)sec (150 wpm)

► Is it possible to do Is it possible to do better? Suggestions?better? Suggestions?

Keyboard and KeypadsKeyboard and Keypads► Look at the piano for possible Look at the piano for possible

inspirationinspiration► Court reporter keyboards Court reporter keyboards

(one keypress = multiple (one keypress = multiple letters or a word) letters or a word) 300 wpm, requires extensive 300 wpm, requires extensive

training and usetraining and use► Keyboard properties that Keyboard properties that

mattermatter Size Size

► large - imposing for novices, large - imposing for novices, appears more complexappears more complex

► mobile devicesmobile devices AdjustableAdjustable

► Reduces RSI, better Reduces RSI, better performance and comfortperformance and comfort

Mobile phone keyboards, Mobile phone keyboards, blackberry devices, etc. blackberry devices, etc.

Keyboard LayoutsKeyboard Layouts► QWERTYQWERTY

Frequently used pairs far apartFrequently used pairs far apart Fewer typewriter jamsFewer typewriter jams Electronic approaches don’t jam.. why Electronic approaches don’t jam.. why

use it?use it?► DVOARK (1920s)DVOARK (1920s)

150 wpm->200 wpm150 wpm->200 wpm Reducing errorsReducing errors Takes about one week to switchTakes about one week to switch Stops most from tryingStops most from trying

► ABCDE – styleABCDE – style Easier for non-typistsEasier for non-typists Studies show no improvement vs. Studies show no improvement vs.

QWERTYQWERTY► Number padsNumber pads

What’s in the top row? What’s in the top row? Look at phones (slight faster), then look Look at phones (slight faster), then look

at calculators, keypadsat calculators, keypads► Those for disabledThose for disabled

Split keyboardsSplit keyboards KeyBowl’s orbiTouch (screenshot)KeyBowl’s orbiTouch (screenshot) Eyetrackers, miceEyetrackers, mice Dasher - 2d motion with word predictionDasher - 2d motion with word prediction

KeysKeys

► Current keyboards have Current keyboards have been extensively testedbeen extensively tested SizeSize ShapeShape Required forceRequired force SpacingSpacing

► Speed vs. error rates for Speed vs. error rates for majority of usersmajority of users

► Distinctive click gives audio Distinctive click gives audio feedbackfeedback Why membrane keyboards Why membrane keyboards

are slow (Atari 400?)are slow (Atari 400?)► Environment hazards might Environment hazards might

necessitate necessitate ► Usually speed is not a Usually speed is not a

factorfactor

Keys GuidelinesKeys Guidelines► Special keys should be Special keys should be

denoteddenoted► State keys (such as caps, etc.) State keys (such as caps, etc.)

should have easily noted should have easily noted statesstates

► Special curves or dots for Special curves or dots for home keys for touch typistshome keys for touch typists

► Inverted T Inverted T Cursor movement Cursor movement keys are important (though keys are important (though cross is easier for novices)cross is easier for novices)

► Auto-repeat featureAuto-repeat feature Improves performance, but Improves performance, but

only if repeat is customizable only if repeat is customizable (motor impaired, young, old)(motor impaired, young, old)

► Two thinking points:Two thinking points: Why are home keys fastest to Why are home keys fastest to

type?type? Why are certain keys larger? Why are certain keys larger?

(Enter, Shift, Space bar)(Enter, Shift, Space bar)

► This is called This is called Fitt’s LawFitt’s Law

Keypads for small devicesKeypads for small devices► PDAs, Cellphones, Game consolesPDAs, Cellphones, Game consoles► Fold out keyboardsFold out keyboards► Virtual keyboardVirtual keyboard► Cloth keyboards (ElekSen)Cloth keyboards (ElekSen)► Haptic feedback?Haptic feedback?► Mobile phonesMobile phones

Combine static keys with dynamic Combine static keys with dynamic soft keyssoft keys

Multi-tap a key to get to a characterMulti-tap a key to get to a character Study: Study: Predictive techniques Predictive techniques

greatly improve performancegreatly improve performance Ex. LetterWise = 20 wpm vs 15 wpm Ex. LetterWise = 20 wpm vs 15 wpm

multitapmultitap► Draw keyboard on screen and tap Draw keyboard on screen and tap

w/ penw/ pen Speed: 20 to 30 wpm (Sears ’93)Speed: 20 to 30 wpm (Sears ’93)

► Handwriting recognition (still hard)Handwriting recognition (still hard) Subset: Graffiti2 (uses Subset: Graffiti2 (uses unistrokesunistrokes))

Pointing DevicesPointing Devices► Direct manipulation needs some pointing Direct manipulation needs some pointing

devicedevice► Factors:Factors:

Size of deviceSize of device AccuracyAccuracy DimensionalityDimensionality

► Interaction Tasks:Interaction Tasks: Select – menu selection, from a listSelect – menu selection, from a list Position – 1D, 2D, 3D (ex. paint)Position – 1D, 2D, 3D (ex. paint) Orientation – Control orientation or provide Orientation – Control orientation or provide

direct 3D orientation inputdirect 3D orientation input Path – Multiple poses are recorded Path – Multiple poses are recorded

► ex. to draw a lineex. to draw a line Quantify – control widgets that affect Quantify – control widgets that affect

variablesvariables Text – move textText – move text

► Faster w/ less error than keyboardFaster w/ less error than keyboard► Two types (Box 9.1)Two types (Box 9.1)

Direct control Direct control – device is on the screen – device is on the screen surface (touchscreen, stylus)surface (touchscreen, stylus)

Indirect control Indirect control – mouse, trackball, – mouse, trackball, joystick, touchpadjoystick, touchpad

Direct-control pointingDirect-control pointing► First device – lightpenFirst device – lightpen

Point to a place on screen and press Point to a place on screen and press a buttona button

Pros: Pros: ► Easy to understand and useEasy to understand and use► Very fast for some operations (e.g. Very fast for some operations (e.g.

drawing)drawing) Cons: Cons:

► Hand gets tired fast! Hand gets tired fast! ► Hand and pen blocks view of screenHand and pen blocks view of screen► FragileFragile

► Evolved into the touchscreen Evolved into the touchscreen Pros: Pros: Very robust, no moving partsVery robust, no moving parts Cons: Cons: Depending on app, accuracy Depending on app, accuracy

could be an issue could be an issue ► 1600x1600 res with acoustic wave1600x1600 res with acoustic wave

Must be careful about software Must be careful about software design for selection (land-on design for selection (land-on strategy).strategy).► If you don’t show a cursor of where If you don’t show a cursor of where

you are selecting, users get confusedyou are selecting, users get confused User confidence is improved with a User confidence is improved with a

good lift-off strategygood lift-off strategy

Direct-control pointingDirect-control pointing

► Primarily for novice Primarily for novice users or large user users or large user basebase

► Case study: Disney Case study: Disney WorldWorld

►Need to consider Need to consider those who are: those who are: disabled, illiterate, disabled, illiterate, hard of hearing, hard of hearing, errors in usage (two errors in usage (two touch points), etc.touch points), etc.

Indirect-Control PointingIndirect-Control Pointing► Pros: Pros:

Reduces hand-fatigue Reduces hand-fatigue Reduces obscuration problemsReduces obscuration problems

► Cons: Cons: Increases cognitive load Increases cognitive load Spatial ability comes more into playSpatial ability comes more into play

► MouseMouse Pros:Pros:

► FamiliarityFamiliarity► Wide availabilityWide availability► Low costLow cost► Easy to useEasy to use► AccurateAccurate

Cons:Cons:► Time to grab mouseTime to grab mouse► Desk spaceDesk space► Encumbrance (wire), dirtEncumbrance (wire), dirt► Long motions aren’t easy or obvious (pick up and replace)Long motions aren’t easy or obvious (pick up and replace)

Consider, weight, size, style, # of buttons, force feedbackConsider, weight, size, style, # of buttons, force feedback

Indirect-Control PointingIndirect-Control Pointing► TrackballTrackball

Pros:Pros:►Small physical footprintSmall physical footprint►Good for kiosksGood for kiosks

► JoystickJoystick Easy to use, lots of buttonsEasy to use, lots of buttons Good for tracking (guide or Good for tracking (guide or

follow an on screen object)follow an on screen object) Does it map well to your Does it map well to your

app?app?► TouchpointTouchpoint

Pressure-sensitive ‘nubbin’ Pressure-sensitive ‘nubbin’ on laptopson laptops

Keep fingers on the home Keep fingers on the home positionposition

Indirect-Control PointingIndirect-Control Pointing

► TouchpadTouchpad Laptop mouse deviceLaptop mouse device Lack of moving Lack of moving

parts, and low profileparts, and low profile Accuracy, esp. those Accuracy, esp. those

w/ motor disabilitiesw/ motor disabilities►Graphics TabletGraphics Tablet

Screen shotScreen shot comfortcomfort good for cad, artistsgood for cad, artists Limited data entryLimited data entry

Comparing pointing devicesComparing pointing devices

► Direct pointingDirect pointing Study: Faster but less accurate than indirect (Haller ’84)Study: Faster but less accurate than indirect (Haller ’84)

► Lots of studies confirm mouse is best for most tasks Lots of studies confirm mouse is best for most tasks for speed and accuracyfor speed and accuracy

► Trackpoint < Trackballs & Touchpads < MouseTrackpoint < Trackballs & Touchpads < Mouse► Short distances – cursor keys are betterShort distances – cursor keys are better► Disabled prefer joysticks and trackballs Disabled prefer joysticks and trackballs

If force application is a problem, then touch sensitive is If force application is a problem, then touch sensitive is preferredpreferred

Vision impaired have problems with most pointing devices Vision impaired have problems with most pointing devices ► Use multimodal approach or customizable cursorsUse multimodal approach or customizable cursors► Read Vanderheiden ’04 for a case studyRead Vanderheiden ’04 for a case study

► Designers should smooth out trajectoriesDesigners should smooth out trajectories► Large targets reduce time and frustrationLarge targets reduce time and frustration

ExampleExample►Five fastest places to click on for a Five fastest places to click on for a

right-handed user?right-handed user?

ExampleExample►What affects time?What affects time?

Fitts’s LawFitts’s Law► Paul Fitts (1954) developed a model of human hand movementPaul Fitts (1954) developed a model of human hand movement► Used to predict time to point at an objectUsed to predict time to point at an object► What are the factors to determine the time to point to an object?What are the factors to determine the time to point to an object?

D – distance to targetD – distance to target W – size of targetW – size of target

► Just from your own experience, is this function linear?Just from your own experience, is this function linear? No, since if Target A is No, since if Target A is DD distance and Target B is distance and Target B is 2D2D distance, it distance, it

doesn’t take twice as longdoesn’t take twice as long What about target size? Not linear there eitherWhat about target size? Not linear there either

► MT = a + b logMT = a + b log22(D/W + 1)(D/W + 1) a = time to start/stop in seconds (empirically measured per a = time to start/stop in seconds (empirically measured per

device)device) b = inherent speed of the device (empirically measured per b = inherent speed of the device (empirically measured per

device)device) Ex. a = 300 ms, b = 200 ms/bit, D = 14 cm, W = 2 cmEx. a = 300 ms, b = 200 ms/bit, D = 14 cm, W = 2 cm

► Ans: 300 + 200 logAns: 300 + 200 log22(14/2 + 1) = 900 ms(14/2 + 1) = 900 ms

Fitts’s LawFitts’s Law►MT = a + b logMT = a + b log22(D/W + 1)(D/W + 1)

a = time to start/stop in seconds (empirically a = time to start/stop in seconds (empirically measured per device)measured per device)

b = inherent speed of the device (empirically b = inherent speed of the device (empirically measured per device)measured per device)

Ex. a = 300 ms, b = 200 ms/bit, D = 14 cm, W = 2 Ex. a = 300 ms, b = 200 ms/bit, D = 14 cm, W = 2 cmcm►Ans: 300 + 200 logAns: 300 + 200 log22(14/2 + 1) = 900 ms(14/2 + 1) = 900 ms

Question: If I wanted to half the pointing time (on average), Question: If I wanted to half the pointing time (on average), how much do I change the size?how much do I change the size?

► Proven to provide good timings for most age groupsProven to provide good timings for most age groups► Newer versions taken into account Newer versions taken into account

Direction (we are faster horizontally than vertically)Direction (we are faster horizontally than vertically) Device weightDevice weight Target shapeTarget shape Arm position (resting or midair)Arm position (resting or midair) 2D and 3D (Zhai ’96)2D and 3D (Zhai ’96)

Very Successfully StudiedVery Successfully Studied► Applies toApplies to

Feet, eye gaze, head mounted sightsFeet, eye gaze, head mounted sights Many types of input devicesMany types of input devices Physical environments (underwater!)Physical environments (underwater!) User populations (even retarded and drugged)User populations (even retarded and drugged) Drag & Drop and Point & ClickDrag & Drop and Point & Click

► LimitationsLimitations Software accelerated pointer motionSoftware accelerated pointer motion TrainingTraining Trajectory Tasks (Accot-Zhai Steering Law)Trajectory Tasks (Accot-Zhai Steering Law) Decision Making (Hick’s Law)Decision Making (Hick’s Law)

► Results (what does it say about)Results (what does it say about) Buttons and widget size?Buttons and widget size? Edges?Edges? Popup vs. pull-down menusPopup vs. pull-down menus Pie vs. Linear menusPie vs. Linear menus

Precision Pointing Movement Precision Pointing Movement TimeTime

► Study: Sears and Shneiderman ’91 Study: Sears and Shneiderman ’91 Broke down task into gross and fine components for small Broke down task into gross and fine components for small

targetstargets PPMT = a + b logPPMT = a + b log22(D/W+1) + c log(D/W+1) + c log22(d/W)(d/W)

► c – speed for short distance movementc – speed for short distance movement► d – minor distanced – minor distance

Notice how the overall time changes with a smaller target.Notice how the overall time changes with a smaller target.► Other factorsOther factors

Age (Pg. 369)Age (Pg. 369)► Research: How can we design devices that produce Research: How can we design devices that produce

smaller constants for the predictive equationsmaller constants for the predictive equation Two handedTwo handed ZoomingZooming

Novel DevicesNovel Devices► Themes:Themes:

Make device more diverseMake device more diverse► UsersUsers► TaskTask

Improve match between task Improve match between task and deviceand device

Improve affordanceImprove affordance Refine inputRefine input Feedback strategiesFeedback strategies

► Foot controlsFoot controls Already used in music where Already used in music where

hands might be busyhands might be busy CarsCars Foot mouse was twice as Foot mouse was twice as

slow as hand mouseslow as hand mouse Could specify ‘modes’Could specify ‘modes’

Novel DevicesNovel Devices► Eye-trackingEye-tracking

Accuracy 1-2 degreesAccuracy 1-2 degrees selections are by constant selections are by constant

stare for 200-600 msstare for 200-600 ms How do you distinguish w/ How do you distinguish w/

a selection and a gaze?a selection and a gaze? Combine w/ manual inputCombine w/ manual input

► Multiple degree of Multiple degree of freedom devicesfreedom devices Logitech Spaceball and Logitech Spaceball and

SpaceMouseSpaceMouse Ascension BirdAscension Bird Polhemus Liberty and Polhemus Liberty and

IsoTrackIsoTrack

Novel DevicesNovel Devices► Boom ChameleonBoom Chameleon

Pros: Pros: Natural, good Natural, good spatial understandingspatial understanding

Cons: Cons: limited limited applications, hard to applications, hard to interact (very passive) interact (very passive)

► DataGloveDataGlove Pinch glovePinch glove Gesture recognitionGesture recognition American Sign Language, American Sign Language,

musical directormusical director Pros: Pros: NaturalNatural Cons:Cons: Size, hygiene, Size, hygiene,

accuracy, durabilityaccuracy, durability

Novel DevicesNovel Devices► Haptic FeedbackHaptic Feedback

Why is resistance useful?Why is resistance useful? SensAble Technology’s PhantomSensAble Technology’s Phantom Cons: limited applicationsCons: limited applications Sound and vibration are easier Sound and vibration are easier

and can be a good and can be a good approximationapproximation► Rumble packRumble pack

► Two-Handed inputTwo-Handed input Different hands have different Different hands have different

precisionprecision Non-dominant hand selects fill, Non-dominant hand selects fill,

the other selects objectsthe other selects objects► Ubiquitous Computing and Ubiquitous Computing and

Tangible User InterfaceTangible User Interface Active Badges allows you to Active Badges allows you to

move about the house w/ your move about the house w/ your profileprofile

Which sensors could you use?Which sensors could you use? Elderly, disabledElderly, disabled Research: Smart HouseResearch: Smart House Myron Kruger – novel user Myron Kruger – novel user

participation in art (Lots of participation in art (Lots of exhibit art at siggraph)exhibit art at siggraph)

Novel DevicesNovel Devices

► Paper/WhiteboardsPaper/Whiteboards Video capture of annotationsVideo capture of annotations Record notes (special tracked Record notes (special tracked

pens Logitech digital pen)pens Logitech digital pen)

► Handheld DevicesHandheld Devices PDAPDA Universal remoteUniversal remote Help disabledHelp disabled

► Read LCD screensRead LCD screens► Rooms in buildingRooms in building► MapsMaps

Interesting body-context-Interesting body-context-sensitive. sensitive. ► Ex. hold PDA by ear = phone Ex. hold PDA by ear = phone

call answer.call answer.

Novel DevicesNovel Devices

►MiscellaneousMiscellaneous Shapetape – reports Shapetape – reports

3D shape. 3D shape. ►Tracks limbsTracks limbs

► Engineer for specific Engineer for specific app (like a gun app (like a gun trigger connected to trigger connected to serial port)serial port) Pros: good Pros: good

affordanceaffordance Cons: Limited Cons: Limited

general use, timegeneral use, time

Speech and Auditory Speech and Auditory InterfacesInterfaces

► There’s the dreamThere’s the dream► Then there’s realityThen there’s reality► Practical apps don’t really require freeform Practical apps don’t really require freeform

discussions with a computerdiscussions with a computer Goals: Goals:

►Low cognitive loadLow cognitive load►Low error ratesLow error rates

► Smaller goals:Smaller goals: Speech Store and Forward (voice mail)Speech Store and Forward (voice mail) Speech GenerationSpeech Generation Currently not too bad, low cost, availableCurrently not too bad, low cost, available

Speech and Auditory Speech and Auditory InterfacesInterfaces

► Bandwidth is much lower than visual displaysBandwidth is much lower than visual displays► Ephemeral nature of speech (tone, etc.)Ephemeral nature of speech (tone, etc.)► Difficulty in parsing/searching (Box 9.2)Difficulty in parsing/searching (Box 9.2)► TypesTypes

Discrete-word recognitionDiscrete-word recognition Continuous speechContinuous speech Voice informationVoice information Speech generationSpeech generation Non-speech auditoryNon-speech auditory

► If you want to do research here, lots of research in If you want to do research here, lots of research in the audio, audio psychology, and DSP field you the audio, audio psychology, and DSP field you should understandshould understand

Discrete-Word RecognitionDiscrete-Word Recognition

► Individual words spoken by a specific personIndividual words spoken by a specific person► Command and controlCommand and control► 90-98% for 100-10000 word vocabularies90-98% for 100-10000 word vocabularies► TrainingTraining

Speaker speaks the vocabularySpeaker speaks the vocabulary Speaker-independentSpeaker-independent

► Still requiresStill requires Low noise operating environmentLow noise operating environment MicrophonesMicrophones Vocabulary choiceVocabulary choice Clear voice (language disabled are hampered, stressed)Clear voice (language disabled are hampered, stressed) Reduce most questions to very distinct answers (yes/no)Reduce most questions to very distinct answers (yes/no)

Discrete-Word RecognitionDiscrete-Word Recognition► Helps:Helps:

DisabledDisabled ElderlyElderly Cognitive challengedCognitive challenged User is visually distractedUser is visually distracted Mobility or space restrictionsMobility or space restrictions

► Apps:Apps: Telephone-based infoTelephone-based info

► Study: Study: much slower for cursor movement than mouse or much slower for cursor movement than mouse or keyboard (Christian ’00)keyboard (Christian ’00)

► Study: Study: choosing actions (such as drawing actions) improved choosing actions (such as drawing actions) improved performance by 21% (Pausch ’91) and word processing (Karl performance by 21% (Pausch ’91) and word processing (Karl ’93)’93) However acoustic memory requires high cognitive load (> than However acoustic memory requires high cognitive load (> than

hand/eye)hand/eye)► Toys are successful (dolls, robots). Accuracy isn’t as importantToys are successful (dolls, robots). Accuracy isn’t as important► Feedback is difficultFeedback is difficult

Continuous Speech Continuous Speech RecognitionRecognition

► DictationDictation► Error rates and error repair are still poorError rates and error repair are still poor► Higher cognitive load, could lower overall qualityHigher cognitive load, could lower overall quality► Why is it hard?Why is it hard?

Recognize boundaries (normal speech blurs them)Recognize boundaries (normal speech blurs them) Context sensitivityContext sensitivity ““How to wreck a nice beach”How to wreck a nice beach”

► Much trainingMuch training► Specialized vocabularies (like medical or legal)Specialized vocabularies (like medical or legal)► Apps:Apps:

Dictate reports, notes, lettersDictate reports, notes, letters Communication skills practice (virtual patient)Communication skills practice (virtual patient) Automatic retrieval/transcription of audio content (like radio, CC)Automatic retrieval/transcription of audio content (like radio, CC) Security/user IDSecurity/user ID

Voice Information SystemsVoice Information Systems► Use human voice as a source of infoUse human voice as a source of info► Apps:Apps:

Tourist infoTourist info Museum audio toursMuseum audio tours Voice menus (Interactive Voice Response IVR systems)Voice menus (Interactive Voice Response IVR systems)

► Use speech recognition to also cut through menusUse speech recognition to also cut through menus If menus are too long, users get frustratedIf menus are too long, users get frustrated Cheaper than hiring 24 hr/day repsCheaper than hiring 24 hr/day reps

► Voice mail systemsVoice mail systems Interface isn’t the bestInterface isn’t the best

► Get email in your carGet email in your car Also helps with non-tech savvy like the elderlyAlso helps with non-tech savvy like the elderly

► Potentially aides withPotentially aides with Learning (engage more senses)Learning (engage more senses) Cognitive load (hypothesize each sense has a limited ‘bandwidth’)Cognitive load (hypothesize each sense has a limited ‘bandwidth’)

► Think ER, or fighter jetsThink ER, or fighter jets

Speech GenerationSpeech Generation

► Play back speech (games)Play back speech (games)► Combine text (navigation systems)Combine text (navigation systems)► Careful evaluation!Careful evaluation!

Speech isn’t always greatSpeech isn’t always great►Door is ajar – now just a toneDoor is ajar – now just a tone►Use flashUse flash►Supermarket scannersSupermarket scanners

Often times a simple tone is betterOften times a simple tone is better Why? Cognitive loadWhy? Cognitive load

►Thus cockpits and control rooms need speechThus cockpits and control rooms need speech►Competes w/ human-human communicationCompetes w/ human-human communication

Speech GenerationSpeech Generation► Ex: Text-to-Speech (TTS)Ex: Text-to-Speech (TTS)► Latest TTS uses multiple syllabi to make generated speech Latest TTS uses multiple syllabi to make generated speech

sound bettersound better Robotic speech could be desirable to get attentionRobotic speech could be desirable to get attention All depends on appAll depends on app Thus don’t assume one way is the best, you should user testThus don’t assume one way is the best, you should user test

► Apps: Apps: TTS for blind, JAWSTTS for blind, JAWS► Web-based voice apps: VoiceXML and SALT (tagged web Web-based voice apps: VoiceXML and SALT (tagged web

pages). pages). Good for disabled, and also for mobile devicesGood for disabled, and also for mobile devices

► Use ifUse if Message is shortMessage is short Requires dynamic responsesRequires dynamic responses Events in timeEvents in time

► Good when visual displays aren’t that useful. When?Good when visual displays aren’t that useful. When? Bad lighting, vibrations (say liftoff)Bad lighting, vibrations (say liftoff)

Non-speech Auditory Non-speech Auditory InterfaceInterface

► Audio tones that provide informationAudio tones that provide information►Major Research AreaMajor Research Area

Sonification – converting information into audioSonification – converting information into audio AudiolizationAudiolization Auditory InterfacesAuditory Interfaces

► Browsers produced a click when you clicked Browsers produced a click when you clicked on a linkon a link Increases confidenceIncreases confidence Can do tasks without visual cognitive loadCan do tasks without visual cognitive load Helps figure out when things are wrongHelps figure out when things are wrong Greatly helps visually impairedGreatly helps visually impaired

Non-speech Auditory Non-speech Auditory InterfaceInterface

► Terms:Terms: Auditory icons – familiar Auditory icons – familiar

sounds (record real world sounds (record real world sound and play it in your sound and play it in your app)app)

Earcons – new learned Earcons – new learned sounds (door ajar)sounds (door ajar)

► Role in video games is hugeRole in video games is huge Emotions, Tension, set moodEmotions, Tension, set mood

► To create 3D soundTo create 3D sound Need to do more than stereoNeed to do more than stereo Take into account Head-Take into account Head-

related transfer function related transfer function (HRTF)(HRTF)► Ear and head shapeEar and head shape

► New musical instrumentsNew musical instruments ThereminTheremin

► New ways to arrange musicNew ways to arrange music

DisplaysDisplays

► Primary Source of Primary Source of feedbackfeedback

► Properties:Properties: Physical DimensionPhysical Dimension ResolutionResolution Color Depth and Color Depth and

correctnesscorrectness Brightness, contrast, Brightness, contrast,

glareglare PowerPower Refresh rateRefresh rate CostCost ReliabilityReliability # of users# of users

Display Display TechnologyTechnology

► Monochrome displays Monochrome displays (single color)(single color) Low costLow cost Greater intensity range Greater intensity range

(medical)(medical)► ColorColor

Raster Scan CRTRaster Scan CRT LCD – thin, brightLCD – thin, bright Plasma – very bright, thinPlasma – very bright, thin LED – large public displaysLED – large public displays Electronic Ink – new product Electronic Ink – new product

w/ tiny capsules of negative w/ tiny capsules of negative black particles and positive black particles and positive whitewhite

Braille – refreshable cells with Braille – refreshable cells with dots that rise updots that rise up

Large DisplaysLarge Displays► Wall displaysWall displays

InformationalInformational►Control rooms, military, flight Control rooms, military, flight

control rooms, emergency control rooms, emergency responseresponse

►ProvidesProvides System overview System overview Increases situational Increases situational

awarenessawareness Effective team reviewEffective team review

►Old: Array of CRTsOld: Array of CRTs InteractiveInteractive

►Require new interaction Require new interaction methods (freehand sketch, methods (freehand sketch, PDAs)PDAs)

►Local and remote Local and remote collaborationcollaboration

►Art, engineeringArt, engineering

Large DisplaysLarge Displays

►Multiple Desktop DisplaysMultiple Desktop Displays Multiple CRTs or Flat panels Multiple CRTs or Flat panels

for large desktopsfor large desktops CheapCheap FamiliarFamiliar Spatial divide up tasksSpatial divide up tasks Comparison tasks are easierComparison tasks are easier Too much info?Too much info?

►HMDHMD► Eventually -> Every surface Eventually -> Every surface

a pixela pixel

Mobile device displaysMobile device displays

► ApplicationsApplications PersonalPersonal

► Reprogrammable picture Reprogrammable picture framesframes

Digital family portrait Digital family portrait (GaTech)(GaTech)

BusinessBusiness► PDAs, cellphonesPDAs, cellphones

MedicalMedical► Monitor patientsMonitor patients

Research: Research: Modality Modality Translation Services Translation Services (Trace Center – (Trace Center – University of Wisconsin)University of Wisconsin)► As you move about it As you move about it

auto converts data, info, auto converts data, info, etc. for youetc. for you

Mobile device displaysMobile device displays

► Actions on mobile Actions on mobile devicesdevices Monitor information and Monitor information and

alert (calendar)alert (calendar) Gather then spread out Gather then spread out

information (phone)information (phone) Participate in groups and Participate in groups and

relate to individual relate to individual (networked devices)(networked devices)

Locate services and Locate services and identify objects (GPS car identify objects (GPS car system)system)

Capture and then share Capture and then share info (phone)info (phone)

Mobile device displaysMobile device displays► Guidelines for designGuidelines for design

Bergman ’00, Weiss, ’02Bergman ’00, Weiss, ’02 Industry led research and design case Industry led research and design case

studies (Lindholm ’03)studies (Lindholm ’03) Typically short in time usage (except Typically short in time usage (except

handheld games)handheld games) Optimize for repetitive tasks (rank Optimize for repetitive tasks (rank

functions by frequency)functions by frequency) Research: Research: new ways to organize large new ways to organize large

amounts of info on a small screenamounts of info on a small screen Study: Study: Rapid Serial Visual Presentation Rapid Serial Visual Presentation

(RSVP) presents text at a constant speed (RSVP) presents text at a constant speed (33% improvement Oquist ’03)(33% improvement Oquist ’03)

Searching and web browsing still very Searching and web browsing still very poor performancepoor performance

Promising: Hierarchical representation Promising: Hierarchical representation (show full document and allow user to (show full document and allow user to select where to zoom into)select where to zoom into)

Animation, Image, and VideoAnimation, Image, and Video► Content quality has also Content quality has also

greatly increasedgreatly increased► 3D rendering is near life-like3D rendering is near life-like► Digital Photography is Digital Photography is

commoncommon► Scanned documentsScanned documents► Video compression Video compression ► Multimedia considerations for Multimedia considerations for

the disabledthe disabled► PrintersPrinters

3D Printers create custom objects 3D Printers create custom objects from 3D modelsfrom 3D models