devices 1

Tactile: keyboard, mice, stylus / tablets (chord keyboards, 3D gloves)

Visual: monitors, printers (helmets, see-through, microscreens)

Acoustic: speech synthesis & recognition, music

Smell: burn chemicals, control air flow, in movies across head rest

Taste ?

Neurological Implants, “R U wired?”, Sci-Fi

cochlear implants

Virtual Retinal Display (VRD)

Laser-based projected image

onto the retina

Visual Cortical Implant

Interaction Devices

devices 2

QWERTY designed for manual (slow) typewriters

Dvoark minimizes finger travel distance

Dvoark takes a week or so to learn:

Benefits not greater than effort to learn?

Resistance to change?

Chord Keyboards

Several keys are pressed to enter a symbol.

Speed up to 300w/m with high accuracy.

Steep learning curve, constant practice.

E.g.: court recorders, piano keyboards...

Keyboards

QWERTY Dvoark

Words / minute 150 200

Error more less

qaz

wsx

edc

r tf gv b

space

u yj hn b

ik,

o l.

p;/

.458.2

.957.9

.6818.4

.5422.9

.42

.5421.6

1.177.2 .41

12.6.601.3

% Error% Work Load

symbols

devices 3

QUERTY

Work Load and Errors

common device performance

metrics / analysis

task performance

devices 4

Special keys often require "homing" or positioning of hands off

keyboard's "home row".

Homing is a source of error (description) and often requires a change in

visual focus from task to keyboard.

Function keys require homing. Side function keys were operated by

single hand.

Control sequence (shortcuts / accelerator) commands require least

homing like function key use.

Cursor positioning keys vary both home position and key layout across

keyboards.

Homing Position

devices 5

3rd hand - homing problem

Direct Pointers: fingers, stylus / pen (lightpen, touchscreen)

+ User selects and manipulates objects on display directly with pointer

device. More natural (touchscreen).

+ Fast positioning

- Pointing on vertical display causes fatigue.

- Less accurate, pointing (hand) can block view of display. In-Out

touchscreen positioning can help accuracy.

Touchscreens useful in novice user and environmental critical areas.

automated tellers, information kiosk, factory floor computers

controlling manufacturing, civil engineers on construction sites,

survey workers in the field.

phones, tablets, laptops (vertical fatique)

Pointing Devices

devices 6

- User positions a "pointer" icon on screen with a pointer device.

The pointer device does not contact the display. After indirectly

positioning icon the user can select and manipulate displayed

objects.

- Positioning less "natural", a learned skill

+ View not blocked, less fatigue

Indirect pointers have a resolution – sensitivity to movement

("pixels"/inch, ballistic mode)

Mouse is dominant indirect pointer.

Card et.al. showed arrow key faster

than mouse for very short distances only.

Fitt's law applies to the study of pointing tasks.

Distance

0

1

2

3

4

1 2 4 8

Centimeters

Se

co

nd

s

Arrow Mouse

Indirect Pointers (mice, joystick)

devices 7

Direct pointers are faster than keyboard cursor keys in most cases.

Indirect pointers excel at fine resolution position of on screen targets and

far positioning tasks.

Cursor keys excel at low resolution position of on screen targets (form

fill in tasks) and close position tasks.

Cursor keys preferred on tasks that mix text processing (keyboard

required) and positioning

Less 3rd hand homing problem with cursor keys and keyboard

commands.

Indirect pointer are often "preferred" position and selection devices in

"easy to use" oriented software.

Are small mobile devices “easy” or “convenient” to use?

User satisfaction due to closure, control, direct manipulation interface.

Keyboard VS. Pointer Devices

devices 8

Feet based pointer devices (no 3rd hand)

Head (eye) based pointers -- infrared headbands, glasses

IR head tracker

Trackir

3D Manipulators

Data Glove: manipulates objects, has sense of contact with object. No

"mass/resistance" feedback.

Joystring: manipulates objects, no sense

of contact with object, "mass/resistance" feedback.

3D “mouse”: eg. 3DConnexion.

Dimensions = (x, y ,z, pitch, yaw, and roll).

Interfaceboard

trackingsensor

tactilefeedbackflexing

sensors

Dataglove

foot pointerUncommon pointers

devices 9

Interfaceboard

trackingsensor

tactilefeedbackflexing

sensors

Dataglove

foot pointer

Joystring

hand grabs "inverted T"

wires attached to

stepper motors provide

resistance.

devices 10

Natural I/O interface: selection, input

Metaphor to existing tools

Expressive and portable

Smart Paper & Electronic Ink

Smart paper (use awareness in the media)

text OCR convert, auto complete, spell (grammar) check, justify,

evaluate math

draw CAD conversion, snap, stretch...

Electronic writing - pen strokes (vs bitmaps) can be edited, layered

(“post-its”), attached to objects in smart paper.

Pen I/O

devices 11PenPoint -- eg. pen commands, common edit cmds

brackets, selects pieces of text

caret, insert text

check, display options for selected text, objects, icons, tools, etc

circle, opens edit pad to modify text fields and labels

cross out deletes object beneath it

scratch out deletes sfsff

flick, scrolls document in flick direction {up, down, right, left}

pigtail deletes a character

tap selects or invokes

press - hold initiates drag (move)

tap - press initiates drag (copy)

devices 12

Palm Pilot's graffiti reduced stroke input language

write letters here write numbers here

Graffiti

Help screen

Division marks

Start

stroke at

heavy dot

Lift

Stylus

here

devices 13

devices 14

3 interacting variables of color vision:

Hue color

Brightness intensity (bright - dull)

Saturation % color in field

Opponent process theory of color vision

These colors can't be seen in same patch of light.

They produce shadows and edges.

Avoid use of opponent (opposite) colors.

Color is very useful to have user selected items stand out in a display.

Spatial and temporal representations of blue colors is worse than

other colors.

Selection / Applicability

Color can also be used to indicate whether a menu option is valid in

the current state or not ("greyed options").

blue / yellow

green / red

white / black

Color Vision

devices 15

Alert / Attention.

Change of color represents change in state (green, yellow, red).

• Use few colors that are easy to discriminate

• Use warning colors sparingly.

• Consistent system wide analysis of color use.

Element Discrimination.

Color provides contrast and improves discrimination. Need high

contrast difference. Contrast a function of luminance or hue.

Category grouping & field definition. Color can help group display

elements and facilitate visual search. Visual search is affected by:

• number of items

• color separation of categories

• legibility of coded symbols

• relationship between color coding and targets

Color Usage

devices 16As screen density increases color effect increase.

Color can define visual fields on display - weather maps

Size & Visual Acuity

As number of colors increase size of text should also increase.

Color can't be assumed! Redundantly code display.

Designer's color perception != user's color perception

color & text codes (categorization)

color, size & text

color, size, text & icon ....

Color Memory:

5 - 7 color memory for codes.

Don't tax Working Memory use around 4 colors!

devices 17Strong color connotations:

red danger, error, hot, revolution

green OK, go on, well, alive, healing

yellow slow down, caution, sun light

Weak color connotations:

blue fluid / liquid, wet, calm, hidden

black empty, death, anarchy

brown earth, warm

Color Preferences

• Children prefer warm colors: reds, browns

• Adults prefer cool colors: blues, greens

• Occupations (degrees) have color associations:

green health

orange engineering

devices 18

No more than 6 colors (including black) should be used on one screen.

Backgrounds should not be brighter than foregrounds. Grid lines should

be half intensity

Do not have extreme color contrasts between foreground and

background colors -- causes afterimages due to rod fatigue.

Use white for critical / important dynamic information (color gun fails)

A monochrome flash of twice intensity is as effective as color use.

Separate significant information on display by size, distance, intensity

or highlighting.

Use screen position consistently. Time / Date info or page number in

same place.

Groups screen elements. Provide title for elements. Hard to title than

grouping is poor.

Rules of Thumb for Visual Displays

devices 19

US Government suggested color use in (critical) monitoring systems

Color State Result

Flashing red Emergency immediate action

red Alert corrective action

magenta Emergency out of limit indicator

warning

yellow Advise Caution, recheck

Blue Advisory ( use only as

background / filler )

devices 20

Pixels (picture elements) / inch - density measure. For text higher

resolution implies larger font dimensions. (Text same visual size)

Pixel shape (saturation)

Display Fields

Layout (tiled)

center in fields

uncluttered fields

Resolution

devices 21

Tiled vs Overlapping Windows.

Titled: Applications with static window sizes and little or no window

manipulation. E.g.: form entry, message windows, popups.

Overlapped: Applications with dynamically sized windows with user

window manipulation.More robust - let user control display rather than

application. E.g.: text or graphic editors.

Experts tend to prefer overlapped (control ,customization).

Novices tend to prefer tiled (full screen / switch apps) when applicable.

Preference

full screen Vs overlapped

mobile devices ?

tiles Vs full screen

Windowed Displays

devices 22Refresh rate

CRT Vs LCD / LED displays

CRT (85 Hz)

+ faster refresh rate, less motion blur, less pixel persistence

+ independent of age and temperature

? better color resolution, better black / white

- flicker, eye fatique

- burn in, constant display burns phospors

LCD / LED (60 or 75Hz, set @ 75Hz)

+ less eye fatique, slower refresh rate (60hz), pixel persistence

120Hz / 144 Hz LCD availability

+ text display

+ LED black / white

- degrades with age and temperature

- LCD black / white

Older users have greater eye fatique with CRT.

devices 23

Display glasses user sees a full-sized

display floating in front of their eyes.

Projected Displays - displays projected

transparent surfaces: glass windows, cockpits, visors

opaque surfaces: SixthSense, Cave

Enable user to view displays w/o moving eyes in critical environments.

Proposed for displays in cars. (dashboard controls, in car TV / movies

Wearable displays

devices 24

Visual display can appear 360 degrees, or at least always be present

regardless of head position.

HMD w/ trackers can update display based on head movement (inside a

virtual world).

Sound (and possibly smell) can also be presented (3D sound effects).

Complete control of user's visual and acoustic interface. No external

environment interference.

Comfort?

Head Mounted Displays

devices 25

Speech Synthesis (generation)

+ Good quality - issue of digitized sampling resolution and complexity.

+ Basic speech phonemes can be edited to make comprehensible

speech.

? Speaking devices: cars, computers, houses...

Acoustic Interfaces

devices 26

hearing & understanding limited w/o training

Trained systems. User speech commands are sampled and stored for

pattern matching.

Production Systems:

spoken cmd pattern1 : action1

spoken cmd pattern2 : action2

: :

- Sensitive to surrounding noise

- Low transference across users

- High storage overhead for representing spoken commands and

searching command pattern set. (Can be hierarchical in nature).

Speech Recognition

devices 27

+ High Quality, actual sound

? capacity function of sampling

rate / compression

CD R/W provides large capacity

Using Recorded (or Digitized) Sound

Sampling & manipulations (music -rap)

Sound Bites (commercials, film, TV) HCI Applications:

Voice Mail -- Computer Phone surveys - digitized voice

production with user input by touch tones.

Acoustic menu system...

Acoustic Desktop

Sonic Icons: pilots, blind

Desktop becomes a spherical environment.

Digital Sound Processing

devices 28

(Reality) = = the product of our 3 lb. universe (brain)

Reality consists of a person's perception of the world around them. That

perception is the product of their sensory inputs and expectations.

Illusions, "magic" are the interactions of our sensations and

expectations. We see and hear what we want, what we expect.

Current computer controlled interactive devices have the rudimentary

capability to generate a truly artificial reality for the user:

Vision and Acoustic control of environment via helmets, goggles,

headphones, ...

Tactile interaction and feedback via data glove like devices for

hands, feet, body glove ...

Existing, accepted artificial realities: movies, books, television, games,

simulators for training, sensory deprivation, dreams...

Virtual Reality

devices 29Uses of VR, AR (augmented reality)

Multimedia interfaces like interactive CD-ROM for graphics,

animations, and sound, helmets, and data gloves enable complex

adventure/fantasy environments (caves) for entertainment.

Will books and movies become interactive ?

Scientific Visualization - engineers and scientist can move through

dangerous or theoretical worlds.

Boeing Dreamliner first virtual (VR assisted) designed aircraft

Medicine - surgeons walk through accurate 3D representations of

patients gathered w/ CAT or MRI scans. (Robotic/Waldo surgery)

Waldo enhancements for workers in dangerous environments.

Programming as a visual/intuitive process - looking at data and processes

aka Neuromancer. Software walkthrough gets a new meaning!

Is software modeling 2D (UML...) or can it be 3D

devices 30

"We try to move our designs closer to the

human being and blend the machine with

the body." - Hideji Takemasa, NEC

Wearable Data Terminal an optical scanner

worn on forearm enables reading bar codes,

OCR, graphics, with writable optical disk and

CD-ROM database computer located in

terminal worn on neck.

The MIThril hardware platform combines

body-worn computation, sensing, and

networking in a clothing-integrated design.

The MIThril software platform is a

combination of user interface elements and

machine learning tools built on the Linux

operating system.

Wearable Computers

devices 31

annual wearable computing conference urls

http://iswc.gatech.edu/archives.htm

http://www.media.mit.edu/wearables/

http://wearables.gatech.edu/

"A person's computer should be worn, much

as eyeglasses or clothing are worn, and

interact with the user based on the context of

the situation. With heads-up displays,

unobtrusive input devices, personal wireless

local area networks, and a host of other

context sensing and communication tools, the

wearable computer can act as an intelligent

assistant, whether it be through a

Remembrance Agent, augmented reality, or

intellectual collectives." -- MIThril

devices 32I, Cyborg

In 1998 Porfessor K. Warwick surgically implant a silicon chip

transponder surgically implanted into his forearm.

http://www.kevinwarwick.com/index.asp

He can be monitored using a signals

emitted by the chip. He could operate

doors, lights, heaters and other computers

without lifting a finger...

In 2002 a 100 electrode array was

implanted into the median nerve fibres of

his left arm. The implant can send signals

back and forth between Warwick's

nervous system and a computer.

This bi-directional functionality was

demonstrated with the aid of Kevin’s

wife Irena and a second, less complex

implant with her nervous system...