Fast And Robust Interface Generation for Ubiquitous

ApplicationsThe SUPPLE Project

University of Washington, Seattle

Krzysztof Gajos, David Christianson, Raphael Hoffmann, Tal Shaked, Kiera Henning,

Jing Jing Long, and Daniel S. Weld

Fast And Robust Interface Generation for Ubiquitous

Applications

Automatic, On The Fly

The SUPPLE ProjectUniversity of Washington, Seattle

Krzysztof Gajos, David Christianson, Raphael Hoffmann, Tal Shaked, Kiera Henning,

Jing Jing Long, and Daniel S. Weld

Motivation

Motivation

Motivation

Motivation

Motivation

Motivation

• Variety of display devices & interaction contexts makes hand-designed interfaces expensive⇒ Adapt to device characteristics

Motivation

• Variety of display devices & interaction contexts makes hand-designed interfaces expensive⇒ Adapt to device characteristics

• Current interfaces: complex & “One size fits all” ⇒ Adapt to users and tasks

Motivation

• Variety of display devices & interaction contexts makes hand-designed interfaces expensive⇒ Adapt to device characteristics

• Current interfaces: complex & “One size fits all” ⇒ Adapt to users and tasks

⇒ Automatic interface generation is a scalable

solution

Approach

• Develop abstract representation for:

• Interfaces

• Display devices

• Users

• Automatically generate interfaces from the abstractions

SUPPLE Architecture

Display

Device Model

Target Device

SUPPLEInterface Model

Applicationor

Appliance

User Model

User's InfoSpace

SUPPLE Architecture

Display

Device Model

Target Device

SUPPLEInterface Model

Applicationor

Appliance

User Model

User's InfoSpace

Road MapMotivationModeling user interfaces in SUPPLE

User interface generation as optimizationAutomatically adapting user interfaces

A preliminary user studyAdaptation in SUPPLE

Customization support in SUPPLE

Conclusions

• simple types:

• media types:

• containers:

• derivative types:

• vectors:

• actions: !1 ! !2

!!, C! "

Modeling User Interfaces

vector(!)

i nt| f loat|st r i ng|bool

{!ii!1...n}

image|interactiveMap

Modeling User Interfaces

Light Level: τ: <int, [0,10]>

Power: τ: bool

Light: τ: { , }

Light Bank: τ: { , , }

Light ... Light ...

A/V: τ: { , }

Projector: τ: { , }

Classroom:τ: { , , }

Input:τ: <string, {data1,data2, video}>

Vent:τ: <int, [0,3]>

Power: τ : bool

Screen:τ: bool

Modeling User Interfaces

Light Level: τ: <int, [0,10]>

Power: τ: bool

Light: τ: { , }

Light Bank: τ: { , , }

Light ... Light ...

A/V: τ: { , }

Projector: τ: { , }

Classroom:τ: { , , }

Input:τ: <string, {data1,data2, video}>

Vent:τ: <int, [0,3]>

Power: τ : bool

Screen:τ: bool

Light Level:

!: <int, [0,10]>Power:

!: bool

Light:

!: { , }

Light Bank:

!: { , , }

Light ... Light ...

A/V:

!: { , }

Projector:

!: { , }

Classroom:

!: { , , }

Input:

!: <string, {data1,data2, video}>Vent:

!: <int, [0,3]>Power:

! : boolScreen:

!: bool

Modeling User Interfaces

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

Click!

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

Click!

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

Media Types

Subtyping

Subtyping

Subtyping

Click!

Road MapMotivationModeling user interfaces in SUPPLE

User interface generation as optimizationAutomatically adapting user interfaces

A preliminary user studyAdaptation in SUPPLE

Customization support in SUPPLE

Conclusions

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

User Interface Generation as Optimization

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

• Driven by a Cost function ($) -- estimated user effort to manipulate a rendering of the interface

User Interface Generation as Optimization

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

• Driven by a Cost function ($) -- estimated user effort to manipulate a rendering of the interface

• Cost function derived from device model

User Interface Generation as Optimization

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

• Driven by a Cost function ($) -- estimated user effort to manipulate a rendering of the interface

• Cost function derived from device model

• Algorithm: branch-and-bound search with full constraint propagation at each step

User Interface Generation as Optimization

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

• Driven by a Cost function ($) -- estimated user effort to manipulate a rendering of the interface

• Cost function derived from device model

• Algorithm: branch-and-bound search with full constraint propagation at each step

• Performance: 0.2 - 2.0 seconds on a desktop computer

User Interface Generation as Optimization

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

• Driven by a Cost function ($) -- estimated user effort to manipulate a rendering of the interface

• Cost function derived from device model

• Algorithm: branch-and-bound search with full constraint propagation at each step

• Performance: 0.2 - 2.0 seconds on a desktop computer

User Interface Generation as Optimization

User Interface Generation as Optimization

• Flexible with respect to screen size

• Versatile: Same algorithm for different devices

• Allows new concerns to be included in the rendering process, e.g.:

• Cross-device consistency

• Adaptation to usage patterns

Robustly Adapting to Different Screen Sizes

Robustly Adapting to Different Screen Sizes

Robustly Adapting to Different Screen Sizes

Single Algorithm -- Many Devices

Single Algorithm -- Many Devices

Single Algorithm -- Many Devices

Single Algorithm -- Many Devices

Single Algorithm -- Many Devices

Adapting To Usage Patterns

SUPPLE with an empty

trace

Adapting To Usage Patterns

SUPPLE with an empty

trace

SUPPLE with a “lights-heavy”

trace

Road MapMotivationModeling user interfaces in SUPPLE

User interface generation as optimizationAutomatically adapting user interfaces

A preliminary user studyAdaptation in SUPPLE

Customization support in SUPPLE

Conclusions

Adapting To Usage Patterns By Complete Makeover

SUPPLE with an empty

trace

SUPPLE with a “lights-heavy”

trace

Adapting To Usage Patterns By Complete Makeover

SUPPLE with an empty

trace

SUPPLE with a “lights-heavy”

trace

Is this the only way to adapt?

Content Eliding

Content Eliding

Content Eliding

Content Eliding

Visual Popout

Visual Popout

Split Interfaces

Split Interfaces

Split Interfaces

Adaptation StrategiesEfficiency

Confusion

Adaptation StrategiesEfficiency

Confusion

Efficiency

Confusion

Complete

Makeover

Adaptation StrategiesEfficiency

Confusion

Complete

Makeover

Content eliding

Adaptation StrategiesEfficiency

Confusion

Visual Popout

Complete

Makeover

Content eliding

Adaptation StrategiesEfficiency

Confusion

Split

Visual Popout

Complete

Makeover

Content eliding

Adaptation StrategiesEfficiency

Confusion

Split

Visual Popout

Complete

Makeover

Content eliding

Preliminary User Study

Split Interface

Preliminary User Study

Split Interface

Preliminary User Study

Split Interface

Preliminary User Study

Split Interface

Preliminary User Study

Split Interface

Visual Popout

Interface

Preliminary User Study Results

Preliminary User Study Results

• Split Interface strongly preferred over non-adaptive

Preliminary User Study Results

• Split Interface strongly preferred over non-adaptive

• Both adaptive interfaces slightly faster than non-adaptive

Preliminary User Study Results

• Split Interface strongly preferred over non-adaptive

• Both adaptive interfaces slightly faster than non-adaptive

• But: Visual Popout interface often found distracting

Sneak Peak: Followup Study

with Mary Czerwinski and Desney Tan

Sneak Peak: Followup Study

• In a different setting, we tested three adaptation strategies including Split Interface and Visual Popout Interface

with Mary Czerwinski and Desney Tan

Sneak Peak: Followup Study

• In a different setting, we tested three adaptation strategies including Split Interface and Visual Popout Interface

• Users strongly preferred and were significantly faster using Split Interface

with Mary Czerwinski and Desney Tan

Sneak Peak: Followup Study

• In a different setting, we tested three adaptation strategies including Split Interface and Visual Popout Interface

• Users strongly preferred and were significantly faster using Split Interface

• Users strongly disliked Visual Popout Interface

with Mary Czerwinski and Desney Tan

Split Interfaces in SUPPLE

• Adapting by promoting hard-to-reach but frequently used functionality

Split Interfaces in SUPPLE

Click!

Split Interfaces in SUPPLE

Split Interfaces in SUPPLE

Click!

Split Interfaces in SUPPLE

Click!

Split Interfaces in SUPPLE

Click!

Click!

Split Interfaces in SUPPLE

Split Interfaces in SUPPLE

Click!

Split Interfaces in SUPPLE

Split Interfaces in SUPPLE

Split Interfaces in SUPPLE

Four extra clicks required just to print in

landscape mode!

Split Interfaces in SUPPLE

Split Interfaces in SUPPLE

Split Interfaces in SUPPLE

System- Vs. User-Initiated Adaptation: Customization

• Customizing any part of the interface with drag and drop

• Out of order undo

• Generalization

Customization

Customization

Customization

Customization

Customization

Customization

Road MapMotivationModeling user interfaces in SUPPLE

User interface generation as optimizationAutomatically adapting user interfaces

A preliminary user studyAdaptation in SUPPLE

Customization support in SUPPLE

Conclusions

Conclusions

• A powerful, flexible and practical tool for automatically generating user interfaces for multiple devices

• Adaptation and customization offer the possibility of creating custom-made UIs for each user

• Support for distributed operation and caching makes it practical even on small devices

Can I Have It?

Can I Have It?

• We are awaiting permission to release SUPPLE as an open source toolkit

• Visit SUPPLE web site to sign up for the user’s mailing list

Yes!

Contributors

Daniel Weld Raphael Hoffmann

Kiera Henning Jing Jing Long meAnthony Wu

Dave Christianson Tal Shaked

More Information

• SUPPLE:http://www.cs.washington.edu/ai/supple/ : supple interfaces

• Krzysztof Gajos: kgajos@cs.washington.edu

• Daniel Weld: weld@cs.washington.edu

• SUPPLE:http://www.cs.washington.edu/ai/supple/ : supple interfaces

• Krzysztof Gajos: kgajos@cs.washington.edu

• Daniel Weld: weld@cs.washington.edu

More Information