HCI in Ubiquitous Computing

HCI in Ubiquitous Computing

양 현 승AIM (AI & Media) Lab

KAIST 전자전산학과hsyang@cs.kaist.ac.kr

http://mind.kaist.ac.kr

KAIST, AIM Lab

ContentsContents

HCI in U-C

HCI in U-C Embedding Interaction U-C HCI Researches

KAIST, AIM Lab

Ubiquitous ComputingUbiquitous Computing

Ubiquitous computing is the method of enhancing computer use by making many computers available throughout the physical environment, but making them effectively invisible to the user (Mark Weiser, Xerox PARC)

KAIST, AIM Lab

Ubiquitous ComputingUbiquitous Computing

We are surrounded by computing Computing and processing is embedded into

everyday devices There are many computers/processors per person Information access and communication is possible

virtually everywhere Dedicated computing devices – information

appliances – are all around us Devices can be connected and networked

What gets us here?

KAIST, AIM Lab

Ubiquitous ComputingUbiquitous Computing

Mark Weiser: Computers enter everyday life Help people with everyday tasks in the office and

at home (at any time, any place)

A good tool is an invisible tool. By invisible, I mean that the tool does not intrude on your consciousness; you focus on the task, not the tool. [Weiser 94]

KAIST, AIM Lab

HCI themes with U-LifeHCI themes with U-Life

Three past interaction themes: Natural Interfaces Context-Aware Interaction Automated Capture & Access to Life

Experiences

New interaction theme proposed: Everyday Computing

KAIST, AIM Lab

Natural InterfacesNatural Interfaces

Forms of Natural Interfaces Speech, gestures handwriting (pen-based/free-form)

Issues Encountered Need a way to represent information with new

interface Error-prone, even humans can’t perfectly read

handwriting

KAIST, AIM Lab

Context-Aware InteractionContext-Aware Interaction

What is appropriate context to use? Current – user and location Future – time, history, other users

How to represent this context? Incorporate hierarchal info and relations

Truly Ubiquitous? Limitation of many technologies.

KAIST, AIM Lab

Context-Aware InteractionContext-Aware Interaction

location

identity

objects

KAIST, AIM Lab

Everyday Computing: Things to be ConsideredEveryday Computing:

Things to be Considered

No clear beginning & end to all activities Interruption is expected Multiple activities operate concurrently Time is important discriminator Associative models of information

Embedding InteractionEmbedding Interaction

KAIST, AIM Lab

U-LifeU-Life

Web servers

Web browsers

Mobilebrowsers

Electronicservers

?

KAIST, AIM Lab

Change of UI ParadigmChange of UI Paradigm

Single Screen-based UI

Interact with a number of U-devices

(distributed + interconnected)

• Highly personal and mobile appliances • Systems that are integrated in everyday environment

Ubiquitous Computin

g

KAIST, AIM Lab

User interface in U-CUser interface in U-C

Requirements Distribution of UI

All U-devices are distributed. Implicit HCI

To reduce the need for explicit HCI To let explicit interfaces virtually disappear into the

environment

Awareness of the situation, the environment and the aims of the user

Being noticed only when needed

KAIST, AIM Lab

User interface in U-CUser interface in U-C

Current Interaction Explicit HCI

By command-line By direct manipulation using a GUI, gesture, or speech

input

Interaction in U-C Implicit HCI

It allows the computer to interpret the user’s behavior and the surrounding situation and use this information as input

KAIST, AIM Lab

What is different from traditional‘HCI’ and ‘HCI in UbiComp’ ?

What is different from traditional‘HCI’ and ‘HCI in UbiComp’ ?

Output modalities not just an audio visual channel all senses!

Input modalities more than pressing buttons and

moving an object in two dimensions Distribution – physical and

conceptual Magic beyond the screen … it is a vivid physical

relationship

KAIST, AIM Lab

Development Process?Research Approach?

Development Process?Research Approach?

Not anymore designing and programming a GUI Interdisciplinary teams – ethnography, design, CS

It is about creating an experience by Understanding the interaction and process Designing and constructing a set of devices and an

environment Implement the human-information interface based on the

created devices/environment Test it yourself Test it with users

… go back an refine the hardware and start again

KAIST, AIM Lab

PrototypesPrototypes

Functional prototypes are essential to learn, understand and experience how to interact with the ubiquitous computer

From the idea to knowledge Prototyping has been central to hallmark research in the

area (e.g. ParcTab, ActiveBadge) Learning occurs when along the prototyping process as

well as in use

Evaluation Functional prototypes are the means for evaluation “Confronting” real people – already with version 0.001 Deployment in a living lab environment Facilitating everyday environments with real users

KAIST, AIM Lab

Ubi-Comp Environment is itself the Interface

Ubi-Comp Environment is itself the Interface

Everyday objects augmented with sensing table chairs glasses …

Creating a digital shadow reflecting the interaction

KAIST, AIM Lab

Embedding InteractionEmbedding Interaction

Basic technologies for embedding interaction Sensing technologies

Environmental conditions Users’ location Co-location with others Physiological and emotional state of the user User goals User schedules …

Agent technologies Combining a multitude of sometimes contradictory inputs to

make sense at a higher level Adopting a system’s output to be appropriate to whatever

situation might arise

KAIST, AIM Lab

Implicit Interaction (1/2)Implicit Interaction (1/2)

Implicit Human-Computer Interaction (iHCI) iHCI is the interaction of a human with the environment

and with artifacts which is aimed to accomplish a goal. Within this process the system acquires implicit inputs from the user and may present implicit output to the user.

Implicit Input Implicit inputs are actions and behaviour of humans,

which are done to achieve a goal and are not primarily regarded as interaction with a computer, but captured, recognized and interpret by a computer system as input.

Implicit Output Output of a computer that is not directly related to an

explicit input and which is seamlessly integrated with the environment and the task of the user.

U-C HCI ResearchesU-C HCI Researches

KAIST, AIM Lab

OXYGEN ProjectOXYGEN Project

Speech and vision technologies enable us to communicate with Oxygen as if we’re interacting with another person, saving much time and effort

MIT Media Lab

KAIST, AIM Lab

AwareHomeAwareHome

Designing the Interactive Experience Digital Family Portrait

reconnects geographically distant extended family members by allowing them to remain aware of each other in a non-obtrusive, lightweight manner

What Was I Cooking? a context-aware system that captures the

transient information of recent activities and passively displays them as visual cues.

Gesture Pendant Gesture Pendant recognizes and then

translates gestures into commands for your home appliances

AwareHome with human-like perception could improve quality of life for many, especially seniors.

Georgia Tech.

KAIST, AIM Lab

Easy Living(1)Easy Living(1)

EasyLiving is developing a prototype architecture and technologies for building intelligent environments

System Architecture

Key features Computer vision for person-

tracking and visual user interaction.

Multiple sensor modalities combined.

Use of a geometric model of the world to provide context.

Automatic or semi-automatic sensor calibration and model building.

Fine-grained events and adaptation of the user interface.

Device-independent communication and data protocols.

Ability to extend the system in many ways.

RulesEngine

PersonTracker

PersonDetector

PersonDetector

SeatSensors

PCLogon

Fingerprint

Logon

RoomLights

A/V Media

Systems

TerminalServer

RoomControl UI

KB/Mouse

Redirect

DesktopManage

r

WorldModel

AgentLookup

personpersontrackingtracking

world modelworld model room controlroom control

authenticationauthentication

Microsoft

KAIST, AIM Lab

Easy Living(2) . . . . Easy Living(2) . . . .

Personal Detection Stereo Processing with

commercial software Background subtraction and

person detection Reports sent to central

personal tracker about 7Hz

Personal Tracking Process each new report from

a sensor

color depth

patches people

Past locations

Predicted location

New sensor measurement

“Person creation zone”

Microsoft

KAIST, AIM Lab

HomeLabHomeLab

Philips HomeLab appearance looking and feeling like a regular home for testing its new home technology prototypes in the most

realistic possible way WWICE PHENOM EASY ACCESS POGO: an interactive game for children

virtual story world interfaced by active tools Intelligent Personal-Care Environment

based on measurements from the activity monitor and heart rate sensor

Philips Research

KAIST AIM Lab ResearchKAIST AIM Lab Research

KAIST, AIM Lab

Role of Wearable Computer in Ubiquitous Computing Environment

Role of Wearable Computer in Ubiquitous Computing Environment

It easily acquires personal data (personalization). It guarantees safety of personal data (privacy). It enhances user’s interaction with many devices. It reduces network traffic about personal data

transmitting. It assists us to work (agent).

KAIST, AIM Lab

BackgroundBackground

• This system should understand a user’s intention or preference.• This system should communicate with various electronic media.

A system that assists us in interacting with those mediain our daily life is required.

A system that assists us in interacting with those mediain our daily life is required.

We will frequently interact with those media.(We will feel much annoyed with this interaction.)

We will frequently interact with those media.(We will feel much annoyed with this interaction.)

Various electronic media will be scattered around usin the near future (ubiquitous computing environments).

Various electronic media will be scattered around usin the near future (ubiquitous computing environments).

KAIST, AIM Lab

Research ObjectiveResearch Objective

♦ To Establish Some Concepts - IEM - IWAS

♦ To Establish Some Concepts - IEM - IWAS

♦ To Propose an IWAS prototype and IEM prototypes♦ To Propose an IWAS prototype and IEM prototypes

♦ To demonstrate interaction of IWAS and IEM♦ To demonstrate interaction of IWAS and IEM

KAIST, AIM Lab

IEMIEM

Interactive Electronic Media electronic media in ubiquitous computing environment

that are not only controlled by a user’s command but that also respond to context or the user’s emotional state

. . .

Wireless Control

IEM

KAIST, AIM Lab

IEMIEM

IEM Examples IEM encapsulated electronic appliances

such as a TV, a video player, a radio, a computer, and etc. Responsive digital media

interactive media artworks All objects with embedded computer chips or sensors

an automatic curtain that rises or falls according to a user’s intention or preference

a lamp that intelligently controls the intensity of light according to a user’s emotional state

IEM Features Wireless control => ultimately, automation (agent system) Unique ID Interaction capability

KAIST, AIM Lab

IWASIWAS

Intelligent Wearable Assistance System

KAIST, AIM Lab

IWAS H/W DesignIWAS H/W Design

KAIST, AIM Lab

IWAS H/W DesignIWAS H/W Design

Self-contained System to Wear integrating all components of wearable computer with a

suit User-friendly Interface

input: speech recognition, key-pad, mouse, etc. output: see-through HMD, small speakers.

Various Sensors FSR and postural sensing unit Infra-red tag reading unit

Wireless networking Wireless LAN, IEEE 802.11b

KAIST, AIM Lab

Functions of IWASFunctions of IWAS

Intelligent User-Assistance local identification using IR sensor device direct control of IEM using IR remote controller communication via wireless LAN or Bluetooth information service such as schedule alert, email

check interacting with media

KAIST, AIM Lab

Functions of IWASFunctions of IWAS

Home Network(Home RF, IEEE902.11, ···)

Home G/W

TV

PC Audio Player Phone

Sensing Control

Intelligent Agent

• IEM identification using IR sensor • wireless control using IR remote controller

UbiComp Environment

• providing personalized information service

Lamp

KAIST, AIM Lab

IWAS H/W PrototypeIWAS H/W Prototype

FSR sensor

See-through HMDwith speech head set

IR tag reader &IR remote controller

3-axis postural sensorIWAS suit

KAIST, AIM Lab

Interaction with IWAS and IEM

Interaction with IWAS and IEM

CASE 1: Operating a laptop computer CASE 2: Turning on TV

KAIST, AIM Lab

Interaction with IWAS and IEM

Interaction with IWAS and IEM

CASE 3: Controlling virtual system