Ubiquitous Computing Toolkits

Ubiquitous Computing Toolkits

Tara MatthewsAdvance User Interface Software Fall 2004

Goals of Ubicomp

Scalable interfaces multiple devices inter-device communication multiple users

Natural interaction getting away from the desktop augmenting surrounding environment provide context-appropriate services

Calmness more info w/o overwhelming users aware of user’s interruptiblity

Toolkits for Goals

Scalable interfaces multiple devices Ubicomp

Infrastructure inter-device communication multiple users



Toolkits for Goals

Scalable interfaces multiple devices inter-device communication multiple users CSCW (not covered)



Toolkits for Goals


Natural interaction getting away from the desktop Physical UIs /

Mobile augmenting surrounding environment (other lectures)

provide context-appropriate services Calmness

more info w/o overwhelming users aware of user’s interruptiblity

Toolkits for Goals


Natural interaction getting away from the desktop augmenting surrounding environment provide context-appropriate services Context-Aware


Toolkits for Goals



Calmness more info w/o overwhelming users Peripheral Displays aware of user’s interruptiblity

Toolkits for Goals



Calmness more info w/o overwhelming users aware of user’s interruptiblity Interruptibility

(not covered)

Toolkits for Goals




Goals accomplished? Evaluation

Toolkit Categories

Ubicomp infrastructure Context-aware Peripheral displays Evaluation

Not covering: CSCW (separate field) Interruptibility (no toolkits yet) Physical UIs (covered in Jack’s lecture) Mobile Devices (covered in an unclaimed lecture)

Ubicomp Infrastructure

iROS Stanford Interactive Room Operating System

iStuff Physical application toolkit build on iROS

Aura CMU “Distraction-Free Ubiquitous Computing” project Distributed services infrastructure

Gaia University Of Illinois at Urbana-Champaign Infrastructure for Active Spaces

iROS

Interactive Room (iRoom) Operating System Definition: A ubiquitous computing

environment where groups come together to collaborate on solving problems (not a toolkit)

Contains: Embedded devices: large display screens, table-

top display, & other output devices Mobile devices: mobile devices brought into

space can be used with embedded facilities

iROS Goals Facilitate common Interactive Workspace usages observed

in iRoom prototype workspace: Move data between devices Control devices & apps from other devices Coordinate apps, including ones not written to work together

Additional goals: Provide for heterogeneity of devices in workspace,

& new devices being added over time Allow easy integration of legacy devices Robust to transient failures Portable across installations Actions appear instantaneous to users

iROS

3 sub-systems: Event Heap: stores and forwards events Data Heap: facilitates data movement in an

interactive workspace iCrafter: provides a system for service

advertisement and invocation, along with a user interface generator for services.

Event Heap Tuple space model

Associated memory: content is addressable using a pattern Distributed systems based on blackboard model are easily

created in tuple space Decouples apps in space & time

Added semantics: Apps not designed to work together, so need

Self-describing tuples, Flexible typing, Typed tuples Apps can snoop or interpose

Tuple sequencing Data shared & can build up

Tuple expiration Benefits

Anonymous coordination Interposability Snooping

Event Heap

Figure: Example operation showing placed events, and using template events for matching

Data Heap

Provides type & location independent storage

Machine independent: Don’t need explicit location

of data

Type independent: If a set of type converters

are available, system automatically transforms the data

Figure: Shows automatic retrieval of a Word document in PostScript form

iCrafter

Universal control system: control anything from anywhere

Services advertise how they can be controlled System returns appropriate interface per

device Interfaces can range from fully custom to

dynamically generated

iCrafter

Active Interface

UIRenderer

Generator Selector

Generator Processor

Environment Database

Generator Database

Appliance Description

Service Description

ServiceUser

InterfaceManager

Appliance

132

4

Remote Invocation

Discovery

Active Interface

UIRenderer

Generator Selector

Generator Processor


Generator Database


Service Description

ServiceUser

InterfaceManager

Appliance

132

4

Remote Invocation

Discovery

1. USER REQUEST

The user requests an interface for a service. The appliance sends this request to the Interface Manager, along with appliance description information.

iCrafter

Active Interface

UIRenderer

Generator Selector

Generator Processor


Generator Database


Service Description

ServiceUser

InterfaceManager

Appliance

132

4

Remote Invocation

Discovery

Active Interface

UIRenderer

Generator Selector

Generator Processor


Generator Database


Service Description

ServiceUser

InterfaceManager

Appliance

132

4

Remote Invocation

Discovery

2. GENERATOR SELECTION

Upon receiving the request, the Generator Selector selects an appropriate generator based on the service(s) selected and the appliance description.

iCrafter

Active Interface

UIRenderer

Generator Selector

Generator Processor


Generator Database


Service Description

ServiceUser

InterfaceManager

Appliance

132

4

Remote Invocation

Discovery

Active Interface

UIRenderer

Generator Selector

Generator Processor


Generator Database


Service Description

ServiceUser

InterfaceManager

Appliance

132

4

Remote Invocation

Discovery

3. GENERATOR PROCESSING

The Generator Processor runs the selected generator with access to all context (appliance, service, environment), which outputs a UI description.

iCrafter

Active Interface

UIRenderer

Generator Selector

Generator Processor


Generator Database


Service Description

ServiceUser

InterfaceManager

Appliance

132

4

Remote Invocation

Discovery

Active Interface

UIRenderer

Generator Selector

Generator Processor


Generator Database


Service Description

ServiceUser

InterfaceManager

Appliance

132

4

Remote Invocation

Discovery

4. USER INTERFACE RENDERING

The UI description is sent to the UI Renderer on the appliance, which renders the interface and handles user interactions, including service invocations over the EH.

iStuff

Toolkit to prototype physical UIs Flexible, lightweight components Protocol independence Platform independence with cross-platform capabilities Ease of integration with existing applications Support for multiple simultaneous users

Built on iROS iStuff wireless devices (e.g., iButton, iSlider, iLight)

send/receive events to/from Event Heap via a proxy PatchPanel translates events (e.g., iButton: light on | light off) Apps get events from devices via PatchPanel + Event Heap

iStuff Architecture

Event Heap

PatchPanel Proxy

iStuff Device

Transceiver

iStu

ff c

ompo

nent

Application

Wireless connection

TCP/IP connection

iStuff Devices

iSpeaker iLight

iBuzzer

Anoto PeniMouse

iMike

iDog

X10

iStylusiSlider

RF

iButtons

InputInput

OutputOutput

Aura

CMU “Distraction-Free Ubiquitous Computing” theme: Mobility: allow users to move computational tasks

from one place to another Maximize use of available resources Minimize user distraction & drains on attention

Personal information aura: spans wearable, handheld, desktop, & infrastructure devices

Requires new system design: hardware, network, OS, middleware, & UI support

Aura Research Framework

Users

Tasks

Services

OS / Network

Physical Devices

Research Examples

UI / Agents / Speech

Task-driven computing Rapid service configuration

Energy-aware adaptation Multi-fidelity computation Nomadic data access

Intelligent networking Power-aware devices Wearable computers

Aura Example Scenario

Fred prepares presentation & is late Fred gets PDA & Aura transfers slides to it Finishes slides on PDA walking to meeting Aura finds location of meeting from calendar

& uploads slides to projection machine Aura recognizes unfamiliar meeting

attendees faces & skips budget slides

Aura Architecture

User tasks are represented as set of distributed services Database query interface to access distributed services Easily search mixed environments for needed services A key service is People Location Service

Environments self-monitor & re-negotiate task support given runtime variation of capabilities & resources Can detect when task requirements (e.g., min response

time) aren’t met, & search & deploy alternative configurations to support task

Uses software architecture models (graph of components & connectors) for runtime adaptability

Aura Architecture

Task Manager: tracks user context, environment, & task changes Context Observer: provides info on physical context & reports

relevant events Task & Environment Managers Environment Manager: handles access to distributed services Suppliers: provide services for tasks (text editing, video playing,...)

Privacy in Aura

Aura services depend on user location info Introduces privacy concerns

Location policies: define who gets location info, where, when, and at what granularity e.g., “Bob is allowed to know Alice’s location

when she is in NSH” Similar to Bell Labs’ Houdini system’s “rules”

Not based on how user’s actually disclose location

GAIA

Infrastructure for Active Spaces, UIUC Like Aura:

Distributed services architecture accessed via structured queries

Main difference: Focus on user customization of apps based on

resources in current space

Toolkit Categories


Context

Context is key to ubicomp environment that reacts correctly to user’s current situation

Definition: Info used to characterize the situation of a person, place or object relevant to the interaction between a human & a computational service

Primary context types: Identity (who), Location (where), Time (when), Activity (what)

Secondary context types: e.g., for identity: email address, phone number, birthdate, etc

Context-Aware Applications

Definition: Uses context to provide relevant info and/or services to the user, where relevancy depends on the user’s task

3 categories: Present information & services to users

e.g., nearby wireless networks, directions Automatically execute services

e.g., phone vibrates when in meeting Tag info with context for later retrieval

e.g., class lectures and notes

Toolkits for Context 3 main approaches:

1. Widget model Context Toolkit Dey, Abowd, & Salber, 2001 Based on architecture of GUIs

2. Distributed services model Context Fabric Hong & Landay, 2001 Based on client-server dialog

3. Blackboard model iRoom Winograd, 2001 Based on artificial intelligence apps (Engelmore, 1988)

Widget Model: Context Toolkit

Operating system view Abstraction to hardware (sensors);

data is secondary Similar to GUI input handling: uses widget

abstractions for handling context input Uses 3 abstractions:

widgets, interpreters & aggregators

Benefits

Separates semantics from low-level input sensor handling

Apps can focus on context, not sensors or analysis of sensor input

Allows re-use of context widgets, interpreters & aggregators

Drawbacks

Tight coupling – less robust to component failure

Single-manager (Discoverer) control – less flexible

Context Toolkit Architecture

Widgets

Encapsulate info about a single piece of context (e.g., location or activity)

Hide details of underlying sensors from apps Provide customizable & reusable building

blocks of context sensing Provide historical context info

Widgets

Include optional services, or actuation of output in environment Like Interactors: responsible for input & output e.g., light widget could sense light level & adjust

lights accordingly Get context by subscription or polling

Callback methods used to notify subscribers

Aggregators

Widgets + ability to aggregate context info Collect sensory info about an entity (person,

place, thing, etc.) from multiple sources into one widget

Hide more complexity about context-sensing mechanisms by combining multiple sensors

Enable maintainability and efficiency

Interpreters

Abstract sensor info Use lower level sensory information to

deduce higher level info e.g., identity + location + sound sensors

“is there a meeting?”

Putting an App Together

Discoverer enables apps to locate and subscribe to context components

Distributed infrastructure uses p2p comm.

Distributed Services Model

Database view (vs. OS view of CTK) Data is primary, hardware is secondary Focus on how data is modeled, distributed, protected, & used

Middleware technologies that can be accessed through a network Example: infrastructure=Internet; service=DNS

Any kind of device or application can use context gathering & processing services by adhering to predefined data formats & network protocols

Benefits

Interoperable: independent of hardware platform, OS, & programming language

Decoupling: Sensors, services, & apps can be changed w/o affecting each other

Simpler devices: sensors, processing power, data, & services w/in infrastructure shared

Drawbacks

Applications must poll for data (proactively) Servers are specialized per sensor

Context Fabric

Includes 3 pieces1. Distributed data model of people, places, things

each device has a space: private data goes to local space multiple views of context (mine, yours, theirs)

2. Context Specification Language (like SQL) apps specify what context info they need in uniform way e.g., "What are the nearby movie theaters?”

3. Context service as API (per device) interpret CSL queries and provide context info

Privacy in Context Fabric

Decentralized architecture allows it to capture, store, process, & share in privacy-sensitive manner Capture, store, & process data on user’s device

Provides mechanisms for end-user control of sharing

Plausible deniability built-in e.g., if request for context info denied, system can

reply “unknown”

Aura

Also follows distributed services model Uses SQL-like interface to access a set of

distributed contextual services Handles privacy through user-set policies

Blackboard Model Communication goes through a centralized server Components

post messages shared blackboard subscribe to get posted messages matching specified pattern

Route by matching message content to subscriber's pattern Pattern matching uses AI (often apply sophisticated inference

procedures to logical representations) Direct communication between components simulated by including

an identifier for path endpoints as a field in message Announce-listen style

Components that provide services periodically post events Component that uses the data listens to these events, & if one does

not appear within the expected time can initiate restart operations

Benefits

Ease of configuration Robustness to component failure

When component fails, only communication link broken is from it to blackboard

Dependent components can detect failure and call for restart

Simplicity provided by a uniform communications path (to the blackboard) No complex protocol for finding ports or

resources, establishing connections

Drawbacks

Communication less efficient Each communication requires 2 hops General message structure not optimized for

particular data or interaction protocol

iRoom & iStuff

Event Heap: the blackboard & central communication server

Review of Context Approaches

Widget Model – CTK Widget architecture where Discoverer acts as a manager of

context widgets

Distributed Infrastructure Model – ConFab, Aura, Gaia Client-server architecture where client apps communicate

directly w/ services using structured language

Blackboard Model – iRoom, iStuff Central server architecture where apps communicate only w/

server & events are routed using subscriber pattern matching

Toolkit Categories


Peripheral Displays

Provide awareness with min attention Separate from primary task Important to ubicomp vision of many devices to 1 user

Non-focal display not used unless providing peripheral info

Enable you to monitor many info sources while maintaining a calm environment But, only calm if designed to manage attention they attract

Why is creating PDs hard?

Need to abstract info to be glance-able Need mechanisms for dynamically

managing attention PDs attract: Deciding attention levels to attract

(notification levels) Displaying info appropriately (transitions)

Peripheral Display Toolkit (PTK) Supports these 3 key issues in PD creation

Example PTK Applications Remote Activity

Social Guitar Audio Monitor Motion Monitor Remote Awareness Display

Bus Displays Bus Mobile Bus LED

Instant Messenger Status

IM Picture FrameSocial Guitar Bus LED BusMobile

Orb showing remote activity

PTK Architecture

1. Support for managing impact on human attention using abstraction, notification levels, and transitions

2. Simplified code design and code re-use

3. Library of common PD components

Output

TransitionInput

NotificationMap

Abstractor

Architecture DiagramInput-Side Discovery

ServerOutput-Side

Peripheral Display 1

(Abstractors) (NotificationMaps)

(Output Widgets w/ optional Transition)


Input 1

Input 2

Input N

Abs. 1

Abs. NPTK

DiscoveryServer

Abs. 2

Abs. N

Abs. 1 N.M. 1

N.M. 2

N.M. N

Out 1Trans

Out 2

Out N

.

.

. …

Abs. 2

Abs. N

Abs. 1 N.M. 1

N.M. 2

N.M. N

Out 2Trans

Out 1

Out N

…

Input-Side DiscoveryServer

Output-Side


(Abstractors) (NotificationMaps)

(Output Widgets w/ optional Transition)


Input 1

Input 2

Input N

Abs. 1

Abs. NPTK

DiscoveryServer

Abs. 2

Abs. N

Abs. 1 N.M. 1

N.M. 2

N.M. N

Out 1Trans Out 1Trans

Out 2

Out N

.

.

. …

Abs. 2

Abs. N

Abs. 1 N.M. 1

N.M. 2

N.M. N

Out 2Trans Out 2Trans

Out 1

Out N

…

Library Components Input

audio, camera, Phidgets, Context Toolkit, online calendars, news, stocks, weather, Web page parser, serial port

Input audio, camera, Phidgets, Context Toolkit, online calendars,

news, stocks, weather, Web page parser, serial port

Output ticker text, Ambient Orb, Phidgets

Library Components




Abstractors motion, people counting, voices, phone ringing

Library Components




Abstractors motion, people counting, voices, phone ringing

Notification exact match, threshold, contains, degree of change

Library Components

info urgency user attention

Toolkit Categories


Evaluation

3 tools for evaluating people in natural settings from Intille et al.:

1. Context-aware experience sampling based on GPS or other sensors

e.g., ask questions when user goes to store

2. Ubiquitous sensing system that detects environmental changes

e.g., tape-on touch, proximity, & light sensors

3. Image-based experience sampling system i.e., take a picture & ask user about it later

Summary: Ubicomp Goals




Summary: Ubicomp Toolkits

Ubicomp infrastructure iROS, iStuff, Aura, Gaia

Context-aware Context Toolkit, Context Fabric, iRoom

Peripheral displays Peripheral Display Toolkit

Evaluation Intille’s toolkit

Thanks!

CSCW

Groupware Architectures Phillips, W.G., 1999. Architectures for Synchronous

Groupware, Tech. Rep.. http://phillips.rmc.ca/greg/pub/ Greenberg, S. and Roseman, M., 1999. Groupware

Toolkits for Synchronous Work. In: Beaudouin-Lafon, M. (Ed.), Trends In CSCW'99, No. 7 in Trends in Software, John Wiley & Sons, New York, NY, USA, ch. 6, pp. 135–168.

Roseman, M. and Greenberg, S., 1992. GROUPKIT: a groupware toolkit for building real-time conferencing applications. In: Proceedings of the conference on Computer-supported cooperative work, ACM Press, pp. 43–50. http://doi.acm.org/10.1145/143457.143460

Documents

Ubiquitous Computing Toolkits