1EMODE workshop – 27-28 September 2007 Automatic Usability Assessment of Multimodal User Interfaces Based on Ergonomic Rules Adrian Stanciulescu Jean Vanderdonckt

1 EMODE workshop – 27-28 September 2007

Automatic Usability Assessment of Multimodal User Interfaces

Based on Ergonomic Rules

Adrian StanciulescuJean Vanderdonckt

Benoit Macq

Université catholique de Louvain (UCL)http://www.isys.ucl.ac.be/bchi

http://www.similar.cchttp://www.openinterface.org

http://www.isys.ucl.ac.be/bchi

http://www.similar.cci/




Outline

• Introduction• Context

– OpenInterface platform– UsiXML language– UsiXML for OpenInterface

• Navigation in large scale images– Existing solutions– Proposed solution

• Structure of the navigational tasks– Reaching the structure– Working hypopothesis

• Expanded UsiXML model for vocal interaction• Conclusions & future work


Introduction

• Information vizualization: succesful paradigm in HCI

• Zoomable user interfaces– Organize information in space and scale– Interaction techniques: translation and scale– E.g.: storytelling, web browsing, browsing of maps– Few usability studies, inconclusive results

• Interaction devices– Traditional interaction: mouse and keyboard– New emergent interaction: vocal and tactile– The problem: optimize the interaction– Combining vocal and tactile vs. Vocal-only: faster, less error

prone, less linguistic complexity, less time to complete the task


Introduction

• Our objectives:– Automatic usability assessment of MM navigational tasks over

large scale images– Interactions types: translation and zooming– Interaction modalities: graphical (keyboard, mouse), vocal(speech

input), tactile (finger, stylus pen)– Specification of composing elements: sequential or order-

independent interaction


OpenInterface platform

• Provide an open source platform for the design and rapid development of multimodal prototyped applications – Components: pieces of software that provide services (CIDL)– Pipelines: inter-connections between components (PDCL)

• Advantages:– Allow seamless integration of heterogeneous software (C/C++,

Java, Matlab)– Allows rapid prototyping of multimodal applications (easy

software connection)– Reusability of delivered software

• To address:– No OI module for usability guidance at design time– A knowledge base of multimodal ergonomic rules


UsiXML language

• Structured according to four levels of abstraction (Cameleon framework)

• Advantages:– Support for multimodal I/O– Separation of modalities– Support for CARE properties– Extendibility to new modalities

• Shortcomings:– Non-hierarchical structure of the grammars– Undefined order of specification for grammar elements– No means to specify the cardinality of grammar elements


UsiXML for OpenInterface

Usability adviser tool

Multimodal OpenInterface Applications

UsiXML Specification

Usability Evaluation

Ergonomic rules

Abstracti

on

Abstra

ction


Usability adviser tool

• Shortcomings of the existing tools:– No tool to assess the usability of both web and non-web UIs– Formalized language to specify the ergonomic rules: limitations

with respect to the quality of the evaluation

• Our objectives:– Real time guidance during the development life cycle– Evolutional knowledge base– Multiplatform ergonomic rule validation (UsiXML specifications)– Ergonomic rules expressed in a formal language close to natural

language– No need to learn the formal language (rule specification assistant)– Portability of the knowledge base (single file storing the rules)


Navigation in large scale images

• Existing solutions– Combine multiple smaller displays

into a single one– Multiple LCDs butted together

into a single one– Cost-intensive, space-intensive, or both

• Our proposal:– Navigational techniques are required– 3 X 3 overlaying grid– Meaningful symetrical pairs

(Top/Bottom, Left/Right)


Taxonomy of the navigational tasks

• Translation: top / bottom / left / right / main diagonal up (Bottom Left) / main diagonal down (Top Right)/ second diagonal up (Bottom Right)/ second diagonal down (Top Left)– Translate without specifying a value: the user decides when to stop the

translation – Translate by specifying the number of grid units

• Zooming: zoom in / zoom out – Zooming without specifying the zoom factor: it is inferred from the

selected zoomed area and the size of the display window– Zooming with a zoom factor:

• Fixed zoom factor: zoom by steps• Specified zoom factor: the value is specified by the user

– Zooming in a single selected grid unit (e.g., zoom in Top Left / zoom in Centre, zoom in Bottom Right)

– Zooming in multiple selected grid units: • Adjacent horizontal grid unit selection: zoom in Horizontal Top/ zoom

in Horizontal Centre / zoom in Horizontal Bottom• Adjacent vertical grid unit selection: zoom in Vertical Left / zoom in

Vertical Centre / zoom in Vertical Right• Diagonal grid unit selection: (e.g., zoom in Left Centre to Bottom

Right / zoom in Centre to Top Right.


Reaching the structure

• General structure:Instruction:= {Action, Object, Parameter}

• EBNF specific format:Instruction:= {{Action}- [Object] [{Parameter}]}-

• Instanciation of the instruction:


Working hypothesis

A. Hypothesis 1: Action applied over 1 optional Object which has 2 Parameters

B. Hypothesis 2: Action-Parameter order• Ordered instruction: Instruction: = Action >> Parameters• Order-independent instruction: Instruction: = Action |=|

Parameters


Working hypothesis

C. Hypothesis 3: Equivalent navigational instructions:

{ Sub-instruction} :• groups one or more elements

of an instruction• specified by employing a

monomodal/multimodal interaction


Expanded UsiXML model

• VUI support added:– Grammar– Part– Item

• Advantages:– Flexibility

– Reusability


Conclusions & future work

• Conclusions:– Large number of choices when deciding in the favour of a

particular set of navigational tasks and the employed modality– Usability studies required: speed, error rates and subjective

satisfaction– Usability guideline are limited and no yet validated– Automatic usability assesment based on a knowledge base of

ergonomic rules

• Future work:– Address the tactile interaction– Expand UsiXML to support tactile interaction– Adding new ergonomic rules related to MM interaction


Thank you very much for your attention

http://www.isys.ucl.ac.be/bchi BCHI Lab

http://www.similar.ccSimilar netork of Execellence

http://www.usixml.orgUser Interface eXtensible Markup Language

http://www.openinterface.org OpenInterface



http://www.similar.cc/

http://www.usixml.org/

http://www.openinterface.org/

Documents

1EMODE workshop – 27-28 September 2007 Automatic Usability Assessment of Multimodal User Interfaces Based on Ergonomic Rules Adrian Stanciulescu Jean Vanderdonckt