CS260: Research Topics in HCI Fall 2006

John Canny

UCB EECS

The Course

• CS260 this semester is a focused overview of current research in HCI. The focus is set by the students taking it.

• There will be at least one class presentation by each student, based on their area of interest.

• There is also a semester-long class project, either individual or pair.

Sample Projects:Tactile Feedback in Music Applications

Kerry Kimes

+ Sample Projects:Usability analysis of UC-WISE

MISS - Multiple-Interface Scheduling System

CS260, Spring 2002

Juha Johansson

February 28

User Study for Designer’s OutpostKatie Everitt

Sample Projects:

Xiaodong JiangScott Lederer

February 28, 2002

Privacy Management in Ubiquitous Computing

CS260 Project Checkpoint

Livenotes:

Second Iteration of UI

Matthew KamOrna Tarshish

ObjDraw – A tool for use in CS61a at Berkeley

Project Checkpoint

Ryan Stejskal

Feb. 28, 2002

Create a new tool that allows students to explore one of the hardest “big ideas” – object-oriented

programming – outside of class time

Similar to an existing tool, “envdraw”

A graphical interface to object-oriented programming as implemented in 61a

Sound Visualization for the Hearing-Impaired

Wai-ling Ho Ching

• Convenient, low cost home audiometry is desirable and implementable with the power of current home computers

• Pure tones of varying frequencies and degrees of loudness are played to each ear to determine range and degree of hearing loss

Source Code Visualization

Zachary Weinberg• Decades of history, thousands of modifications • No easy way to find the changes you care about • Poor collective memories

HCI: the state of the union

• Ideas appeared in special issue of Queue Magazine on HCI (JFC guest editor).

• Outline: brief recent history of modern HCI

• Why things are different

• What it means for interactive system design

Modern HCI

• We can define modern HCI as the iterative, user-centered design of systems.

• There are two key data points in this evolution, the Xerox Alto and the Xerox Star.

• They represent two very different approaches to design, and two very different outcomes.

Xerox Alto

• Began in1970, soon after PARC formed. Design team: Alan Kay, Chuck Thacker, Butler Lampson

• Real target was a laptop (dynabook), but a personal computer was the closest you could achieve in the 1970s.

• Features:– Mouse– Overlapping windows– Ethernet

• But– Still “mostly text” UI– Lacked a killer app

Xerox Star

• Very different pedigree. Not a research prototype.• Created by Xerox’s product division (Don Masarro).• Goal was to support generic office work. • Project leader David Liddle tapped

experts from PARC to help with Star’sdesign process.

• Started with a “best practice” designdocument:

– Scenarios– Task analysis– Conceptual modeling– Rapid prototyping

Xerox Star

• The Star’s design process is completely modern: it’s a perfectly good example of best practice today.

• The result was a completely modern UI design (a WIMP interface).

• Liddle stated that the Star was a “big improvement on its successors”

• It’s a fair statement. Star’s UI wascopied in Macs and later PCs.

• Its object-oriented design wascleaner than the leading OSes decades later.

Design process

• Neither machine succeeded in the marketplace, although the Star’s design is arguably the most influential of any machine on today’s UIs.

• They represent two different design processes:• Visionary, techno-centric (and theory-centric) design,

embodied in the Alto.• User-centric, market-driven, evolutionary design,

embodied in the Star.

Where we are:

• Desktop PCs are primarily office (knowledge work) machines.

• The IT market is much broader now. Intel’s Taxonomy– Office– Mobile– Health– Home (+ emerging regions?)

• The last 3 are different environments, top-to-bottom. The hardware is different too: In mobile we have cell phones, PDAs, cameras, GPSes, portable games, iPods etc. The Health and Home market’s seem to be still in flux.

Where we are:

• Looking again at the new markets:– Mobile– Health– Home

• There are many differences in how people access information:

– Many more short episodes– More proactivity (reminders, alerts, automation etc.)– More interaction between “device” and “world”– Scarcity of a “desktop” with keyboard, screen, mouse– Increased role of context: where, when, who, history,…

Where we are:

• HCI has mostly followed an evolutionary strategy: its target (the knowledge work environment) has stayed the same for most of its history.

• We’re currently in a “disruptive period” where evolution gives way to revolution.

Technology themes:

Perceptual interfaces (vision, speech, sensing)

Context modeling

Technology themes:

• Perceptual Interfaces: Translate sensed data into relevant (system) actions for the user’s task/activity. Sensing is everywhere in new IT domains (cameraphones, occupancy sensors).

• Context: Is what is “understood” by humans when they interact with each other, and makes efficient communication possible. Very important for emerging IT domains like mobile and home.

Each supports the other.

Context:

• Traditional HCI definition (Abowd et al.):– Place and time– User preferences– User activity (task)

• Most of the papers on context focus on what data to use, not on what to do with it.

• Alternative notion: Context comprises– User activity (what is it, and user’s role?)– Situation (social convention, what would a stranger do?)

• This allows us to assign meaning to observations:– What are the anticipated consequences of the

observations?

Perceptual Interfaces:

• Perceptual Interfaces (like computer vision and speech) have fared poorly in knowledge work environments. Why?

• They are doing much better in the new domains. – E.g. the largest market for speech software now appears

to be health care.• But new environments are physically challenging: noisy,

erratic lighting etc. • But these environments are also context-rich (not true for

the office).

Today’s mobile phone

This year’s Smartphone (free with service contract)• 150-200 MHz ARM processor • 32 MB ram• 2 GB flash (not included)

Windows-98 PC that boots quickly!

Plus:• Camera• AGPS (Qualcomm/Snaptrack)• DSP cores, OpenGL GPU• EV-DO (300 kb/s), Bluetooth

200 mips

Context-Aware Face Recognition

Perceptual Interfaces - Vision

Cameraphones are capable serious computer vision now. Right now, the vision algorithms available include:

• Motion• Barcodes• OCR text (business cards etc.)

Technically feasible:• Face recognition• Building or streetscape recognition

Context-Aware Face Recognition

• Face recognition alone - 43% accurate(state of the art computer vision)

• Context analysis alone - 50% accurate(Face prediction from contextual data on the phone)

• Context+Content analysis - 60% accurate

Figure 1. (Top) Subjects with frontal pose, (Bottom) Same

Context-Aware Place Recognition

• Image analysis alone - 30% accurate

• Context analysis alone - 55% accurate

• Context+Content analysis - 67% accurate

Photo Share Guesser

1 2 3 4 5 6 7 8 9 100.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

SFA prediction

Baseline

Perceptual Interfaces - Vision

TinyMotion is a software mouse for cameraphones.

By moving the camera against any background, real-time image motion estimation provides mouse coordinates.Also great for games

Perceptual Interfaces - Speech

Speech recognition technology has improved steadily in the last ten years, particularly in noisy environments.

Speech was never a good match for office environments.

But the mobile playing field is completely different.

Mobile users often need their eyes and hands free, and the phone will always have a voice channel for telephony.

Speech on Mobile Phones

Restricted speech recognition is available on many phones.

Large-vocabulary recognition just appeared on cell phones last year (Samsung P207). Its a huge step. It enables the next generation of mobile speech-based apps:

• Message dictation• Web search• Address/business lookup• Natural command forms

(no need to learn them)…

Most of this technology was developed in the US by VoiceSignal Technologies.

Speech for Developing Regions

Speech is an even more important tool in developing regions.

Literacy is low, and iconic (GUI) interfaces can be hard to use.

Unfortunately, IT cannot help most of these people because they lack even more basic skills – fluency in a widely-spoken language like English or Mandarin.

This project focuses on teaching English in an ecologically appropriate way.

Speech-based phones are ideal for this.

Speech for Developing Regions

Speech (with headset) allows students to learn while working.

It leaves their eyes and hands free, and engages their minds during tedious, manual work.

Some game motifs:• Safari: hear sound & say the name in English• Karoake: in English• Listen and summarize: BBC, cricket etc. • Treasure hunt: leave LB clues in English• Adventure games: dialog-driven scenarios

Context-Awareness

Context-awareness is the holy grail for next generation mobile applications:

• Location (e.g., video store, kitchen) heavily shapesthe user’s likely actions. So does time,place, identity of friends, etc.

• These data are often inferred byperceptual systems.

• But when people say “context” theymean much more: con-text is literallywith-the-text; its all the other informationneeded to make sense of a text (or a user interface action).

Making more of context…

There is an enormous amount of social sciences that points to two sources of high-level context:

• Activity: based on subject’s personal history and what they are engaged in.

• Situation: a set of socially-understood normed environments and behaviors within them.

• In both cases there is a structure of– People (and their roles)– Objects– Actions or scripts

Making more of context…

There is an enormous amount of social sciences that points to two sources of high-level context:

• Activity: based on subject’s personal history and what they are engaged in.

• Situation: a set of socially-understood normed environments and behaviors within them.

• In both cases there is a structure of– People (and their roles)– Objects (and their genres)– Actions or scripts

Making more of context…

Dealing with context is a great challenge because it covers so many fields, from sociology and linguistics to machine learning.

But the potential payoffs are great as well…

E.g. JCR Licklider’s OLIVER (Online Vicarious Expediter and Responder)

Summary

We are at a fork in the road in HCI. We face a new playing field (app. Domains) with new demands.

We have an opportunity (and probably a necessity) to build systems that are much more context-aware.

Future systems can also leverage machine perception to build and exploit context.

The net result is systems that understand (and respond to) us much better.

Next time…

I propose to start on learning systems for Wednesday. I can cover the lecture.

I’d like a volunteer for Monday’s lecture.