Upload
merryl-lang
View
220
Download
4
Tags:
Embed Size (px)
Citation preview
MIDAS Multi-device Integrated Dynamic
Activity SpacesUnmil P. Karadkar
Ph.D. CandidateDept. of Computer Science
Texas A&M University
• Theory– Which aspects of information content do humans find valuable?
• Techniques– How to adapt content while optimizing for human perception? – How to enable heterogeneous device co-use?
• Frameworks– What are the attributes of multi-device systems? – How options are available to system designers for each attribute?
• System– Design a flexible architecture to embody multi-device system behaviors– Proof-of-concept implementation and analysis
• Practice– Harness device multiplicity to accomplish significant tasks
Research Questions
• Motivation• Related Work• 12C Framework• Approach• MIDAS Architecture• Content Transformation• Demonstration• Analysis• Contributions• Future Work
Outline
• Motivation• Related Work• 12C Framework• Approach• MIDAS Architecture• Content Transformation• Demonstration• Analysis• Contributions• Future Work
Outline
14” 17” 40”- 60”21-24”3.5”
Custom
13”- 17”
75”
10”
4.5” 6” 7”
– Display characteristics– Network bandwidth
– Processing power– Interaction modalities
Societal Changes
• User demographics• Nature of activity• Access location• Affordability– Free phones with contract– $150 e-readers
• Multiplicity – iPhone, iPad, MacBookPro
Heterogeneous contexts of information access
MIDAS
Multi-deviceIntegratedDynamicActivitySpaces
MIDAS
1. Deliver information elements to multiple devices simultaneously– Content– Interaction
2. Rely on widely available infrastructure– Wired and wireless computer networks– Mobile phones, e-readers, tablets, desktop computers– Sensors embedded in the environment
3. Support content transformation– Within form (image scaling, text summarization)– Across forms (text to speech, audio extraction from
video)
Design Goals
4. Strive to retain content integrity from a human perspective– Select transformations that a user perceives to be
closest to the original form
5. Direct content elements to devices that can present them optimally
6. Reconfigure presentation dynamically in response to changes in device availability– New devices become available– Current devices become unavailable
Design Goals
• Motivation• Related Work• 12C Framework• Approach• MIDAS Architecture• Content Transformation• Demonstration• Analysis• Contributions• Future Work
Outline
Background
Individual devices
device-based content adaptation
Ubiquitous Computing
Infrastructure-enabled device co-use
• Service separation– Yahoo!, CNN, Weather Channel, UT Mobile
• Image distillation– Digestor (Fox 1996)
• Text and Web form summarization– Pythia (Bickmore 1997)– Power Browser (Buyukkokten 2002)– Proteus (Anderson 2001)
• Web page content extraction– Baseline SVPs + device-based PVPs (Chua 2007)
• Location-aware services– Computer for the 21st century (Weiser1991)– Java Ring (Olsen 2001)– IntelliBadgeTM (Cox 2003)
• Mobile devices in interactive spaces– Personal Server (Want 2002)– InfoStick (Kohtake 1999)– Elope (Pering 2005)
Opportunistic device co-use
Background—Device co-use• Pebbles (Myers 1997)
– Desktop and PDA co-use– Extend Windows applications to PDAs– PDAs as input and control devices
• WebSplitter (Han 2000)
– Split Web content across a user’s devices– Support for multiple views
• ANMoLE (Haneef 2004)
– Dynamic content reformulation (splitting video)– Integration of phone and computer networks
Background—Device co-use• Ubiquitous Display System (Aizawa 2002)
– Large public displays and mobile phones– Control from mobile phone
• Multibrowsing (Johanson 2001)
– Desktops, notebooks, PDAs, and large displays– Specially encoded links for content routing
• Interface Distribution Daemon (Luyten 2005)
– Web interface as well as stand-alone applications– Multiple content allocation modes– Usability analysis - Completeness and Continuity
• Motivation• Related Work• 12C Framework• Approach• MIDAS Architecture• Content Transformation• Demonstration• Analysis• Contributions• Future Work
Outline
• Concurrency– Simultaneous multi-device content presentation
• Control– Interaction support
• Comity– Permissible diversity in device characteristics
• Completeness– Presentation of relevant information elements
• Coverage– Distribution of content to optimize device usage
• Conversion– Content transformation to suit target device characteristics
12C Framework
• Composition– Relationship of components to the interface
• Coherence– Consistency in presentation of content elements
• Coordination– Reallocation of elements when devices change
• Continuity– Support for users to interpret and evaluate system state
• Constancy– Reuse of content instances that have been shown before
• Confidence– Security and trustworthiness in communication
12C Framework
• Motivation• Related Work• 12C Framework• Approach• MIDAS Architecture• Content Transformation• Demonstration• Analysis• Contributions• Future Work
Outline
Development Platform
Web or something else?
• WWW– Dominant infrastructure– World-wide audience
• Stateless server– Client-initiated communication– Content embedded within structure
• <p>, <h1>
– Changing devices means restarting the activity
• Standardized browsers– Little support for custom behavior
WorldWide Web
context-aware Trellis (caT)• Petri net-based hypertext system (Furuta, Stotts - 1989)
– Formal, graph-based model– Browsing semantics (user privileges, time of day, location)
• Decouple specification from information content– Content not subsumed within structure– <img>, <a>– Separation allows devices to select suitable content
• Decouple content from presentation– Browsers present content autonomously
• Stateful server– Support for multiple clients– Easy propagation of user actions and content to multiple
devices
• Motivation• Related Work• 12C Framework• Approach• MIDAS Architecture• Content Transformation• Demonstration• Analysis• Contributions• Future Work
Outline
Device 3
Browser 4Browser 3
Browser 2
Device 2
Browser 4Browser 3
Browser 2Device 1
Browser 4Browser 3
Browser 2
MIDAS
Browser
ArchitectureSpecification
Information ServicecaT
Users
Device 3
Browser 4Browser 3
Browser 2
ApplicationCoordinator
Device 2
Browser 4Browser 3
Browser 2
ApplicationCoordinator
Device 1
Browser 4Browser 3
Browser 2
MIDAS
Browser
Architecture
Device Manager
BrowserCoordinator
ResourceRealizer
Specification
Information Service
Users
Device 3
Browser 4Browser 3
Browser 2
ApplicationCoordinator
Device 2
Browser 4Browser 3
Browser 2
ApplicationCoordinator
Device 1
Browser 4Browser 3
Browser 2
MIDAS
Architecture
instance propertiescontent handle
user actions
content handles
Specification
Information Service
Users
instance handle, actions
status,user actions
instance handle
content
Browser
user actions content
ResourceRealizer
ResourceRepository
BrowserCoordinator
Device Manager
content handles instance
properties
(instances, devices)
Presentation onlyInput only Interactive
Device Manager
Instance properties
Device Selection Scheme
Device properties, interaction modeInstance Scorer
Content-Device Map
Content Handles, properties
Information Service
ResourceRealizer
BrowserCoordinator
Fully ReplicatedInteraction Replicated
Instance OptimalMIDAS Optimal
• Motivation• Related Work• 12C Framework• Approach• MIDAS Architecture• Content Transformation• Demonstration• Analysis• Contributions• Future Work
Outline
Content Transformation• Images– Scaling (automatic)– Color reduction (automatic)
• Text– Text extraction from formatted documents (automatic)
• HTML, Word, PDF– Summarization (semi-automatic)– Formatted data such as tables (semi-automatic)– Visual or audio rendering (automatic)
• Video– Audio extraction, down-sampling (automatic)– Video summarization (semi-automatic)
Resource Manager• Authoring support
• Device dependence for content instances
• Automatic transformation of images (ImageMagick)
• Author validation
Perception-based Image Transformation
• Images– Changes to attributes of photographs– Fastest growing information content on the Web
• Goal of user study– Understand human perception of nearness• Scaling of images• Color reduction (and gray scaling)
– Obtain actionable metrics to design rules for providing content in its most suitable form
Study Design• Two stages– Stage 1: Image classification– Stage 2: Similarity of image pairs
• Subject demographics– Five subjects per stage– Graduate students and staff– Different disciplines
Stage 1• Subjects classified 100 photographs– Taken by four photographers– Everyday and travel pictures– Prints of digital photos to make piles
• No guidelines regarding classification– Subjects created between 12 and 40 categories
• I coalesced categories based on image content– Four final categories
Image Types
People
Structures
Nature
Text
Stage 2• Subjects viewed image pairs on identical displays– No overlap or switching windows
• Each pair differed in size or color– Variables manipulated independently
• 20 image pairs for each variable
• Answered three questions– Similarity of images (9-point scale)– Suitability of automatic substitution (yes/no)– Acceptance of informed substitution (9-point scale)
Scaling
Color Reduction
Data Analysis
160x120
320x240
640x480
800x600
1024x768
2 (1-bit)
4 (2-bit)
16 (4-bit)
256 (8-bit)
16 Million (24-bit)
Size Colors1
2
3
4
Effect of Information Content Loss
Distance
Score Content Loss
0 6.89 67%
1 6.03 54 %
2 4.57 79 %
3 4.93 90 %
4 4.14 96 %
Distance Score Content Loss
1 7.12 56 %
2 7.11 80 %
3 6.87 93 %
4 6.14 98 %
Scores on a 9 pt. Scale
320 X 240 X 24 vs. 320 X 240 X 1
1024 X 768 X 24 vs. 160 X 120 X 24
(no loss) (98 %)
(no loss) (96 %)
Scaling Color depth change
Similarity by image type
• Images of nature and structures scale well• Human faces• Legibility of textual elements• Algorithms for face and text detection in images exist
Distance nature structures people text
1 6.57 7.67 7.17 6.78
2 7.33 6.78 7.09 7.25
3 7.75 7.00 7.00 6.29
4 7.00 7.50 5.71 5.00
Scaling
Distance Automatic substitution
Acceptability Informed Substitution
Acceptability
1 44% 7.66 56% 6.05
2 49% 7.42 51% 6.00
3 50% 7.13 50% 6.50
4 21% 7.67 79% 4.18
Scaling– Image Substitution
• No clear trends in terms of distance• Subjects allowed automatic substitution
– Corresponded with higher replacement suitability
Distance Automatic substitution
Acceptability Informed Substitution
Acceptability
0 44% 7.50 56% 5.10
1 36% 7.43 64% 4.44
2 10% 8.00 90% 3.89
3 29% 8.75 71% 4.00
4 29% 8.00 71% 3.80
Color Reduction – Image Substitution
• Acceptance of automatic substitution is lower than that for scaled images
• The suitability ratings are correspondingly higher
Perception-informed Rules • Principles
– Prefer scaling over color reduction– Avoid scaling of images that contain people and text– Gray scaling is the best option when used by itself
• Normalized score: Original image = 0 (best score)
+ Wcolor
(9 - Scoredist, color )
9x
Image Rating = Wsize
(9 - Scoredist, size)9
x Wtypex0.4
0.6
Wtogray = 1.0 (grayscale)Wtogray = 0.0 (otherwise)
Wtype
+ Wtogray
(9 - Scoredist, color )
9x
Wtype (people, text) = 0.53Wtype (nature, structures) = 0.47
• Motivation• Related Work• 12C Framework• Approach• MIDAS Architecture• Content Transformation• Demonstration• Analysis• Contributions• Future Work
Outline
• Motivation• Related Work• 12C Framework• Approach• MIDAS Architecture• Content Transformation• Demonstration• Analysis• Contributions• Future Work
Outline
1. Deliver information elements to multiple devices simultaneously– Content– Interaction
2. Rely on widely available infrastructure– Wired and wireless computer networks– Mobile phones, e-readers, tablets, desktop computers– Sensors embedded in the environment
3. Support content transformation– Within form (image scaling, text summarization)– Across forms (text to speech, audio extraction from
video)
Analysis - Design Goals
✔
✔
✔
4. Strive to retain content integrity from a human perspective– Select transformations that a user perceives to be
closest to the original form
5. Direct content elements to devices that can present them optimally
6. Reconfigure presentation dynamically in response to changes in device availability– New devices become available– Current devices become unavailable
Analysis - Design Goals
✔
✔
✔
Analysis - 12C FrameworkConcurrency ComityControl
ConversionCoverageCompleteness
Coordination
CoherenceComposition
ConfidenceConstancy Continuity
Present information elements on multiple devices
• Inclusion of devices in MIDAS is optional• WebSplitter, ANMoLE
• A user can choose the interaction mode for each device
• Inclusion of public devices is not supported• Ubiquitous Display System
• Content is directed to devices automatically• Multibrowsing, Ubiquitous Display System
12C FrameworkConcurrency ComityControl
ConversionCoverageCompleteness
Coordination
CoherenceComposition
ConfidenceConstancy Continuity
Interaction support
• Flexible – dependent on device choices• Most architectures support centralized or distributed
• It is possible to have no input devices at all
• May change during a session
12C FrameworkConcurrency ComityControl
ConversionCoverageCompleteness
Coordination
CoherenceComposition
ConfidenceConstancy Continuity
Permissible diversity in device characteristics
• MIDAS places few restrictions on devices• Pebbles, Ubiquitous display system
• Device must run the Browser Coordinator
• No requirement of proximity to a user• Remote devices may be included - printers
12C FrameworkConcurrency ComityControl
ConversionCoverageCompleteness
Coordination
CoherenceComposition
ConfidenceConstancy Continuity
Presentation of relevant information elements
• Ensures completeness by presenting all resource ids
• Resource Manager aids content authors• Support includes textual description as fallback
• IDD assertion is based on fixed content instances• ANMoLE and MIDAS convert content
12C FrameworkConcurrency ComityControl
ConversionCoverageCompleteness
Coordination
CoherenceComposition
ConfidenceConstancy Continuity
Distribution of content to optimize device usage
• Flexible – dependent on Device Manager mode• Fully replicated - redundancy• Interaction replicated • Instance optimal - coverage for individual instances• MIDAS optimal – system-wide coverage
• Coverage may not apply to some systems• Ubiquitous Display System, Multibrowsing
12C FrameworkConcurrency ComityControl
ConversionCoverageCompleteness
Coordination
CoherenceComposition
ConfidenceConstancy Continuity
Content transformation to suit target device characteristics
• Resource Manager• Authoring-time syntactic conversion with user validation
• Device• Presentation-time implicit conversion
12C FrameworkConcurrency ComityControl
ConversionCoverageCompleteness
Coordination
CoherenceComposition
ConfidenceConstancy Continuity
Relationship of components to the interface
• Generative approach• Instances and actions together form the interface
• Many Web-based systems take a degenerative approach• Content filtering
12C FrameworkConcurrency ComityControl
ConversionCoverageCompleteness
Coordination
CoherenceComposition
ConfidenceConstancy Continuity
Consistency in the presentation of content elements
• Content presented on all devices reflects the current state
• Device Manager mode dependent• Replication modes may present inconsistent information if the content
is not vetted by the author
12C FrameworkConcurrency ComityControl
ConversionCoverageCompleteness
Coordination
CoherenceComposition
ConfidenceConstancy Continuity
Reallocation of elements when devices change
• In response to • Device changes• User actions• Environmental properties changes (inherited from caT)
• Potential for optimization• Not all events threaten the integrity of a presentation
12C FrameworkConcurrency ComityControl
ConversionCoverageCompleteness
Coordination
CoherenceComposition
ConfidenceConstancy Continuity
Reuse of instances that have been shown before• Not supported (yet)
• MIDAS always picks the most optimal instance
• Systems that use fixed instances support constancy better• Web-based presentations
12C FrameworkConcurrency ComityControl
ConversionCoverageCompleteness
Coordination
CoherenceComposition
ConfidenceConstancy Continuity
Support for users to interpret and evaluate system state
• Supported with user-initiated state changes• Device changes• User actions
• Presentation changes due to environmental properties• Causes may be subtle – off-peak hours in another time zone
12C FrameworkConcurrency ComityControl
ConversionCoverageCompleteness
Coordination
CoherenceComposition
ConfidenceConstancy Continuity
Security and trustworthiness in communication
• Reliance on the underlying communication infrastructure• Connection-oriented sockets
• Security or encryption is not supported
• Motivation• Related Work• 12C Framework• Approach• MIDAS Architecture• Content Transformation• Demonstration• Analysis• Contributions• Future Work
Outline
Contributions• Theory– 12C Framework for characterizing multi-device systems– Human perception of nearness in image adaptation
• Nature of content loss is important• Not all images scale equally gracefully• Gray scaled images are perceived to be the closest
• Techniques– Perception-informed metrics for image adaptation
• System– MIDAS architecture and prototype
• Flexible infrastructure • Minimal requirements for including diverse devices• Expression of various server- and client-side behaviors• Coordination in response to users, environment, and devices
• Motivation• Related Work• 12C Framework• Approach• MIDAS Architecture• Content Transformation• Demonstration• Analysis• Contributions• Future Work
Outline
Future Work• Infrastructure
– WebSockets API (HTML5), CSS3 (device-specific templates)– Clients for Android, iOS, and Windows– Inclusion of large-screen displays — GoogleTV(?)
• Interaction and Usability– Realistic setting, contemporary devices– Attention, cognition, privacy– Controlled studies, data from long-term use– Exploration of collaborative use
• Content transformation– Other image adaptations– Suitability of lessons learned from presentation for disabled
populations– Cross-form adaptation– Normalization of scores across different content forms
The most profound technologies are those that disappear.
- Weiser 1991.