Upload
may-james
View
222
Download
5
Tags:
Embed Size (px)
Citation preview
Plan
Context Motivations Thesis statement Requirements for multimodal web UIs Proposal
Reference framework for multimodal web UIs Design space for multimodal web UIs Transformational method for producing multimodal UIs Tool support
Case studies Conclusion & Future work
Context
Graphical user interfaces – classical human-computer interface paradigm Users with limited literacy and typing skills Handheld devices
Need for hands and eyes free Difficult to support mouse and keyboard interaction
Web access anyplace, anywhere, anytime
Need for newer, better interface paradigm Multimodal user interface is the answer… Adaptation to the context of use (user, platform, environment) More natural and efficient human-computer interaction
Motivations
Modality interaction flexibility Faster interaction Lower incidence of errors and esier error recovery True device mobility Usability improvement Robust systems Enhancement of device effectiveness Improved experience Lack of multimodal applications
Thesis statement
Development of multimodal web user interfaces Three types of user interfaces:
Graphical Vocal Multimodal
Conceptual and methodological aspects for a model-to-model transformational approach
Development life cycle based on a design space composed of design options
Requirements
Support for multimodal input and output CARE properties support for input and output
modalities Approach based on design space Openness to new modalities Separation of modalities
Requirements (cont’d)
Ontological independence of modalities Transformation-based development Machine processable Human readable Ontological homogeneity Reuse of specification Methodological explicitness Methodological extendibility Support for tool interoperability
Reference framework
Camelon reference framework UsiXML – selected language:
Structured according to four levels of abstraction
Progressive transformation from Task & Concept level to Final UI Support for the extension of the transformational methodology (graph transformation) Underlying unique formalism (graph-based syntax) Support for reusability of specification and transformations thanks to
the XML structure Support for modality independence Allows introduction of new interaction modality Cross-toolkit development
Vocal Concrete Interaction Objects
Shortcomings of UsiXML previous version:
No specialized containers for user-system dialog No specialized container for multiple selection No differentiations between the system’s output types No object dedicated to system interruptions
Vocal Concrete Interaction Objects (cont’d)
VocalContainers: VocalGroup VocalForm VocalMenu VocalConfirmation
VocalIndividualComponents: VocalOutput:
VocalFeedback VocalPrompt VocalMenuItem
VocalInput VocalNavigation Break Exit
EventTypes: error, help, noInput, noMatch
Multimodal Concrete Interaction Objects
MultimodalCIO: combination of graphical CIO and vocalCIO
outputText + inputText vocalPrompt + vocalInput outputText + inputText + vocalInput + vocalFeedback + guidance
groupBox + radioButtons
vocalMenu + vocalItems + vocalInput
groupBox + radioButtons + vocalMenu + vocalItems +
vocalInput + guidance
Graphical Vocal Multimodal
Concrete UI Relationship
GraphicalRelationships GraphicalTransition GraphicalAdjacency GraphicalContainment
VocalRelationships VocalTransition VocalAdjacency VocalContainment
Synchronization: between vocalCIOs and graphicalCIOs
Multimodal Concrete Interaction Objects
Syntax example: multimodal text field
with equivalent input and redundant output
Specification of transformation
Graph-based patterns Syntactically typed patterns Textual concrete syntax of
patterns: UsiXML syntax
How transformation rules work…
LHS: Find an occurrence of LHS into G NAC: Check that NAC does not match
into G. If there is a match then skip to another occurrence of LHS
RHS: Replace LHS by RHS
Transformation rule
Initial Model
Resultant Model
The four steps of the transformational approach
Step 1: Construct the Task&Domain Models
Step 2: From Task & Domain to AUI
Step 3: From AUI to CUI Step 4: From CUI to FUI
Design space for web UIs
Support for the progressive development life cycle towards a final system
Composed of a set of design options Each design option has a finite set of design values Advantages:
Clarifies the development process in terms of options Facilitates the incorporation in the development life cycle of
design options as an abstraction covered by a software Supports the tractability of more complex design problems
Design options for multimodal web UIs
Prompt: vocal (A), graphical (A), multimodal (E, C, R) Grouping/distinction of items:
Input: vocal (A), graphical (A), multimodal (E) Output: vocal (A), graphical (A), multimodal (E, C, R)
Immediate feedback: vocal (A), graphical (A), multimodal (E, C, R)
Guidance: Input: textual (A), Iconic (A), Acoustic (A), Speech (A),
Multimodal (E, C, R) Immediate feedback: textual (A), Iconic (A), Acoustic (A),
Speech (A), Multimodal (E, C, R)
Step 2: From Task & Domain to AUI
Separated sub-task presentation
Combined sub-task presentation
T11 T12 T13
T1
AC11
AC12
AC13
AC1
T11 T12 T13
T1
AC11
AC12
AC13
NAC LHS RHS
NAC LHS RHS
Step 3: From AUI to CUI
Multimodal text field Prompt: multimodal (R) Input: multimodal (E) Immediate feedback: multimodal (R) Guidance:
Input: iconic (A) Output: iconic (A)
Prompt: multimodal
Please say your name
(vocal +graphical)1
Input: multimodal
Juan Gonzalez
2
(vocal +graphical)
Juan Gonzalez
Guidance for input: iconic
Guidance for feedback: iconic
Feedback:multimodal
Your answer was: Juan Gonzalez
3
(graphical + vocal)
Transformation rule
NAC: LHS:
RHS:
Multimodal prompt
Guidance for input
Multimodal input
Guidance for output
Vocal feedback
Tool support
IdealXML TransformiXML GrafiXML CFB Generator Communication Flow
Builder XHTML+Voice
Generator tool
Conclusions
Model-drive approach for the development of multimodal web user interfaces
Requirements fulfillment: Support for multimodal input/output CARE properties support for input/output modalities Approach based on design options
Methodology covers a subset of the UIs which can be developed with X+V
Conclusions Advantages of combining design options into a design space:
Descriptive Generative Comparative
Conclusions
Work positioned at the intersection of SE and HCI Focus on the feasibility of code generation (2 case
studies) Usability point of view:
Inexistent published studies on multimodal UI design in SE and HCI
Few ongoing works caused by the reduced number of multimodal applications
Internal validation of the Ph.D. (study based on qualitative principles derived from the cognitive psychology)