PhD Thesis Defense - Enhancing Software Quality and Quality of Experience through User Interfaces

Enhancing Software Quality and Quality of Experiencethrough User Interfaces

Pedro Luis Mateo Navarro

Advisors:

Dr. Diego Sevilla Ruiz,

Dr. Gregorio Martınez Perez

Murcia, July 2014

Pedro Luis Mateo Navarro () PhD Thesis Defense Murcia, July 2014 1

Agenda

Enhancing Software Quality and Quality of Experiencethrough User Interfaces

I. Motivation, Goals, and Context

II. Solutions Proposed in this PhD Thesis

III. Conclusions and Further Work

IV. Contributions Summary


IMotivation, Goals, and Context

Table of contents


1 Motivation

2 Goals of this PhD Thesis

3 Context of this PhD Thesis


Quality of Software and Interaction

Quality of Software is a broad concept...

encompasses many disciplines: from technical to psychological

can be achieved in many ways:

checking functional requirements of softwareevaluating users perceived qualityetc.

This thesis focuses on User Interfaces......a field in which there is too much work to do


User interfaces

The link between the human and the machine...

users interact with the interface → they can see/touch/feel it

system provide responses through the interface

mean of communication

User interfaces are essential;guaranteeing their quality is critical


Software Testing

Processes to guarantee software and interfaces quality...

a system or application works as expected?

a system or application works within its context?

guarantee that the requirements are met

Testing is essential before a software is released


Human-Computer Interaction (HCI) Assessment

HCI assessment ⇔ discipline within Software Testing

Study of a conversational process involving:

user → has a task to accomplish; provides input

system → processes data to elaborate a response

context → surrounds them; affects behavior and response

Interaction has changed in the past years


What is Good-Quality Software?

Quality software ⇔ requirements are met

The software fulfills:

functional requirements → the behavior is the expected

non-functional requirements → structural quality

Quality can also be measured in terms ofUsability and Interaction Experience


Software Usability & Quality of Experience (QoE)

Usability: How easy it is to use a software +how easy it is to learn how to use that software

assure software effectiveness (is it useful?) and efficiency

assure user satisfaction

QoE: Subjective measure of users experiences with a softwarebased on user perception and user feelings

encompasses user behaviors + attitudes + emotions

dynamic → changing context/circumstances


Achieving Good-Quality Software?

Different ways to achieve software quality

We try to narrow the problem...

focusing on user interfaces

analysis of user-system interaction in real-time

How can we assess interaction?

Since interaction = user inputs + system outputs then...

...we posed 2 different approaches


Achieving Software Quality through User Interfaces

Approach 1: Decompose interaction into its components

main components ⇒ input + output

components quality is enhanced separately

analysis/testing processes → more focused, more effective

Approach 2: Assess interaction process as a whole

whole dialog → keeps totality of the process

conversational nature → cause-effect relationships

analysis of external factors ⇒ context


Table of contents


1 Motivation




Goal of this PhD Thesis

Improve methods for software testing and interaction analysis

design better tools → +effective, +robust

design easy-to-integrate tools → +adaptable, +lightweight

ease the process of achieving software quality

Too abstract/ambitious ⇒we defined 4 main Goals (2 Blocks)


Research Goals in Block 1

Block 1: assure quality of interaction components separately

Goal 1.1: Ease the validation of system response

automation of GUI testing processes → simulate expert actions

ease the integration into diverse scenarios

Goal 1.2: Allow verification of user inputs at runtime

protect GUI data using rules/constraints → prevent malfunction

dynamic and interactive verification process


Research Goals in Block 2

Block 2: assure quality of user-system interaction

Goal 2.1: Describe multimodal interaction for its assessment

generic and dynamic description of interaction

allow instrumentation, analysis, and comparison

Goal 2.2: Allow analysis of user experiences in mobile scenarios

combine interaction/context data with users subjective data

generic description of mobile context and users perceived quality


Table of contents


1 Motivation




Context of this research

Catedra SAES-UMU:

research initiative created by SAES and the University of Murcia

improve development processes

promote the use and development of open-source tools


SAES

SAES (Cartagena, Spain) is the only Spanish company specialized insubmarine electronics and acoustics

Besides funding our work, SAES provides us with:

real problems to be solved

realistic quality requirements to be achieved

real scenarios in which integrate and use the developed methods


Research Internship at TLabs

Telekom Innovation Laboratories = Deutsche Telekom + TU Berlin(Berlin, Germany)

Central research institute of Deutsche Telekom:

more user-centered approaches

real problems in interaction with current technology


IISolutions Proposed in this PhD

Thesis

Table of contents


4 Block IValidating System OutputVerifying Users Input

5 Block IIEvaluating Multimodal InteractionEvaluating Mobile QoE


Table of contents





Goal 1.1 & Problems Validating System Response


↪→ Goal 1.1: Ease the validation of system response

Problems with GUI output validation:

expensive/cumbersome processes → automation

high dependency on experts → encapsulation + simulation

changes during GUI design/development → robust testing

difficult integration of testing tools

diverse testing environments → adaptable architecturecode-intrusiveness → non-invasive tools


Solution: Open HMI Tester (OHT)

OHT: a framework to develop GUI-testing tools

↪→ Capture/replay approach

GUI Testing ⇒ uses test cases to check requirements

1 Capture → create test cases with interaction

experts interact with parts to be testedGUI introspection → test cases created automatically

2 Replay → execute test cases to validate app response

functional & regression testingexpert simulation: GUI events → +tolerance to changes


HMI TESTER

PRELOAD MODULE

TESTED APPLICATION

dep. on data-model

dep. on OS

dep. on GUIcomm comm

preloadingcontrol

eventconsumer

eventexecutor

logicrecording

controlplaybackcontrol

logic data-modeladapter

user interface

preloadingaction

EVENT DATA

EVENT DATA

GUI EVENTS

Open architecture divided into 2 elements:

HMI Tester → recording/playback control

Preload Module → deployed into the target application↪→ event capture and execution

Adaptable modules + LIB Preload ⇔ Easy integration

How does it work?

OHT example video:


OHT Conclusions

basis to develop GUI testing tools based on GUI introspection

adaptable architecture + adaptable recording/execution processes

direct validation of system response → no model, no test oracles

we still need visual validation of results

Open-source implementation...

cross-platform (C++) + agnostic to GUI platform + extensible

...used in SAES → Qt and ILOG versions


Table of contents





Goal 1.2 & Problems Verifying User Input


↪→ Goal 1.2: Allow verification of user inputs at runtime

In Runtime Verification (RV) a program is monitored atruntime to determine if it satisfies correctness properties

Problems of using RV to verify GUI input:

GUI properties are simpler → +effective method

complex languages and formal logic → +options

formal specification of acceptable behavior → lightweight method

aspect-oriented bindings are static → dynamic languages

feedback to users? → interactive verification


Solution: S-DAVER (Script-based Verification)

S-DAVER: lightweight framework to verify user inputs in GUI

↪→ verification layer between GUI and business logic

Based on fundamental pillars of RV...

GUI data verified when produced → runtime

efficient verification → while user interaction

...but:

RV verifies app behavior 6= S-DAVER verifies data

rules written in scripting languages → no formalisms

included in separate filescan be changed at runtime → dynamic approach

lightweight + easy-to-integrate → smooth verification process

rules loaded and verified transparently


verificationlayer

business logic

1

2

rules logsrule loadinginteractionextraction rule checking feedback

3

gui

dep. on rules

dep. on GUI

Independent/autonomous verification layer:

1 works while user interaction → no perceived overhead

2 only changed data is verified → efficiency

3 if rules fulfilled then go to business logic else do interactive feedback

Adaptable architecture ⇒ GUI platform + rules language

How does it work?

S-DAVER example video:

verification layer

rules logs

1 4

rule search rule check2 3

Event: FocusLostWidget: le_bufferSize

le_bufferSize change was invalid. Rule: "le_bufferSize <= 20"


S-DAVER Conclusions

lightweight, easy-to-integrate verification solution

trust relationship GUI ⇔ business logic

interactive verification experience

no guarantee of the internal correctness of rules


cross-platform + agnostic to GUI platform and verification language+ extensible

...tested in SAES → Qt + Lua version


Table of contents





Table of contents





Goal 2.1 & Problems Assessing Multimodal Interaction


↪→ Goal 2.1: Describe multimodal interaction for its assessment

Problems with multimodal interaction assessment:

many representations → abstract + generic

different modalities analyzed separately → equivalences

static parameters or average metrics

dynamic nature of dialog process? → stepwise descriptionconversational nature of dialog? → interaction turns


Solution: PALADIN (Multimodal Interaction Model)

PALADIN: a generic, runtime multimodal interaction model

↪→ dynamic description of multimodal interaction

Dialog = sequence of alternative user/system turns

interaction described stepwise

relationship data⇔time → runtime model

allows dynamic analysis of interaction + cause-effect relationships

system turn

user turnends

systemfeedback

starts

infopresentation

starts

system turnends

user turnends

user startsexploring

user startstransferring

action

user turn

delay feedback presentation delay exploring transferring


Generic description of multimodal interaction:

model → relationships between data

model → same metrics + same structure → comparison

unified criteria to describe multimodal interaction

agnostic to system and modalities used

Model with 2 levels of abstraction:

generic metrics → main aspects of communication

specific metrics → peculiarities of modalities

Multimodal Interaction ⇒ described as a single flow of actions

PALADIN Integration and Usage

1 PALADIN model instance = 1 interaction episode

Instances intended to be created automatically:

1 interaction extraction → automatic user-system instrumentation

2 PALADIN instantiation → helping framework

3 interaction analysis, comparison, transformation, and decision

PALADINinstances

1. Interaction tracking 2. Model instantiation 3. Interaction analysis

INSTANTIATIONFRAMEWORK

HCIEXTRACTOR

INTERACTIONANALYZER

DECIDER

PALADINimplementation

app


PALADIN Conclusions

generic description of multimodal interaction + stepwise

multimodal interaction = single, seamless flow of actions

unified criteria to analyze and compare different systems


model design + model implementation (Ecore + Java)

instrumentation tool + instantiation framework (Java + Android)

...tested with real users in TLabs

What about mobile interaction?What about quality of experiences?


Table of contents





Main Problems when Assessing Mobile User Experiences


↪→ Goal 2.2: Allow analysis of user experiences in mobile scenarios

Problems analyzing mobile user experiences:

many interaction and context data → select useful ones

low standardization in context definition → generic context

incorporate user ratings into interaction analysis

no robust methodologies combining...

quantitative data evaluating performancequalitative data assessing usability/QoE


Solution: CARIM (Context & Ratings Interaction Model)

CARIM = PALADIN + context data + users subjective ratings

combines mobile interaction data

runtime instrumentation of mobile interaction

ModalityTurnDialog

System TurnUser Turn

Metacommunication

Dynamic Context

Physical environment Social context Location/time Device Communication

Trial

User data

Ratings Mood

Attitude

PALADIN (base)

CARIM (new)


Incorporating Context Data

Generic definition of thesurrounding context:

parameter-based

6 dimensions

user is the subject

extensible

CARIM

deviceuser

location/time

communication social context

human-computerinteraction

dynamicparameters

staticparameters

user ratings

environment

PALADIN


Incorporating User Ratings

User ratings = users quality perception + 2 influencing factors

Quality perception:

AttrakDiff questionnaire → pragmatic, hedonic, attractiveness

mini version (10 pairs of opposite adjectives)

Factors → high influence on quality perception

user mood →attitude towards technology use → 5-item questionnaire


How does it work?

CARIM experiment scenario + video:


CARIM Conclusions

unified criteria to assess QoE from the perspectives of...

user-system interactionits surrounding contextquality perception of the users

basis for context- and QoE-based decision

we are still asking too much data to users


model design + model implementation (Ecore + Java)

instrumentation tool + instantiation framework (Java + Android)

...tested in our labs with real users


IIIConclusions and Further Work

Table of contents


6 Conclusions

7 Future Lines of Work


Conclusions - Goals Achieved (i)

2 Approaches to assess interaction:

Block 1: Decompose interaction into its components

Block 2: Assess interaction as a whole

We outlined 4 main goals


Conclusions - Goals Achieved (ii)


↪→ Goal 1.1: Ease the validation of system response

expensive, hard-to-integrate testing processes

OHT ⇒ adaptable architecture for developing GUI testing tools

implement GUI testing on a robust basis

validate app response using a battery of test cases

test software with precision and tolerance to changes


Conclusions - Goals Achieved (iii)


↪→ Goal 1.2: Allow verification of user inputs at runtime

no effective GUI data verification + no feedback

S-DAVER ⇒ lightweight, runtime, interactive verification framework

independent layer to verify GUI data

dynamic rules

unified interactive feedback


Conclusions - Goals Achieved (iv)


↪→ Goal 2.1: Assess and compare multimodal interaction

static, no abstract description of modalities

PALADIN ⇒ generic, runtime, stepwise interaction model

1 model ⇔ 1 description ⇔ many scenarios

seamless description of modalities

runtime interaction model


Conclusions - Goals Achieved (v)


↪→ Goal 2.2: Allow analysis of mobile user experiences

no methods combining quantitative + qualitative

CARIM ⇒ runtime model = interaction + context + user ratings

measure and analyze QoE from different perspectives

combined + structured data into model

generic definition of context


Table of contents


6 Conclusions

7 Future Lines of Work


Future Work (i)

Support automatic validation of GUI output

OHT ⇒ still need visual validation of results

graphical computing → simulate visual validation

Ensure correctness and consistency of specifications

S-DAVER ⇒ no method to guarantee rules correctness

explore alternatives → CLP?

Outline Block 1 concepts into a multimodal context

from only-GUI to multimodal scenario

validation of multimodal input/output


Future Work (ii)

Enhance/extend design of interaction models

PALADIN & CARIM ⇒ limitations in design

fix potential errors/new emerging needs

instrumentation of new modalities → gestures

incorporate new dimensions of interaction

task: interaction issues ⇔ problem being solveduser emotions: interaction issues ⇔ user state


Future Work (iii)

Exploit the full potential of the interaction models

PALADIN & CARIM ⇒ structured data

model transformation/code generation

decision systems based on live data → adaptation

continuous authentication of users


Future Work (iv)

Study interaction in multi-person, concurrent interfaces

multiple users interacting with the system concurrently

e.g., smart home, tabletop devices

reconsider turn-based approach and interaction models


IVContributions Summary

Table of contents


8 Scientific Publications

9 Software Contributions


JCR Journals

1 Pedro Luis Mateo Navarro, Gregorio Martınez Perez, Diego SevillaRuiz. OpenHMI-Tester: An Open and Cross-PlatformArchitecture for GUI Testing and Certification, InternationalJournal of Computer Systems Science and Engineering (IJCSSE),Special Issue on Open Source Certification, 2010.

2 Pedro Luis Mateo Navarro, Stefan Hillmann, Sebastian Moller, DiegoSevilla Ruiz, Gregorio Martınez Perez. Runtime Model BasedFramework for Automatic Evaluation of Multimodal Interfaces,Journal on Multimodal User Interfaces (JMUI), 2014. In press.

3 Pedro Luis Mateo Navarro, Gregorio Martınez Perez, Diego SevillaRuiz. A Context-aware Interaction Model for the Analysis ofUsers QoE in Mobile Environments, International Journal ofHuman-Computer Interaction (IJHC), Taylor & Francis, 2014. Inpress.


JCR Journals under Review

4 Pedro Luis Mateo Navarro, Diego Sevilla Ruiz, Gregorio MartınezPerez. A Lightweight Framework for Dynamic GUI DataVerification Based on Scripts, Journal on Software Testing,Verification and Reliability, 2014. Minor revision.

Solution JCR journal

OHT 4

S-DAVER minor revisionPALADIN 4

CARIM 4


Other Journals

1 Pedro Luis Mateo Navarro, Gregorio Martınez Perez, Diego SevillaRuiz. Aplicacion de Open HMI Tester como frameworkopen-source para herramientas de pruebas de software, RevistaEspanola de Innovacion, Calidad e Ingenierıa del Software (REICIS),2009.

2 Pedro Luis Mateo Navarro, Diego Sevilla Ruiz, Gregorio MartınezPerez. A Proposal for Automatic Testing of GUIs based onAnnotated Use Cases, Advances in Software Engineering journal,Special Issue on Software Test Automation, vol. 2010, 2010.

3 Pedro Luis Mateo Navarro, Diego Sevilla Ruiz, Gregorio MartınezPerez. CARIM: Un Modelo de Interaccion para el Analisis de laQoE en Entornos Moviles, FAZ: Revista de Diseno de Interaccion,2014. In press.


Conferences and Workshops (i)

1 Pedro Luis Mateo Navarro, Diego Sevilla Ruiz, Gregorio MartınezPerez. A Context-aware Model for QoE Analysis in MobileEnvironments, XIV International Congress on Human-ComputerInteraction, Madrid (Spain), September 2013.

2 Pedro Luis Mateo Navarro, Diego Sevilla Ruiz, Gregorio MartınezPerez. A Context-aware Model for the Analysis of UserInteraction and QoE in Mobile Environments, CENTRIC 2012,Lisbon (Portugal), November 2012.

3 Pedro Luis Mateo Navarro, Stefan Schmidt. Model-basedMeasurement of Human-Computer Interaction in MobileMultimodal Environments, aMMI (Assessing MultimodalInteraction), NordiCHI 2012, Copenhagen (Denmark), October 2012.

4 Pedro Luis Mateo Navarro, Diego Sevilla Ruiz, Gregorio MartınezPerez. Towards Software Quality and User Satisfaction throughUser Interfaces, Fourth IEEE International Conference on SoftwareTesting, Berlin (Germany), March 2011.


Conferences and Workshops (ii)

5 Pedro Luis Mateo Navarro, Gregorio Martınez Perez, Diego SevillaRuiz. Verificacion de Datos en la GUI como un AspectoSeparado de las Aplicaciones, PRIS 2010, Jornadas de Ingenieriadel Software y Bases de Datos (JISBD 2010), Valencia (Spain),September 2010.

6 Pedro Luis Mateo Navarro, Gregorio Martınez Perez, Diego SevillaRuiz. Open HMI Tester: un Framework Open-source paraHerramientas de Pruebas de Software, PRIS 2009, Jornadas deIngenieria del Software y Bases de Datos (JISBD 2009), SanSebastian (Spain), September 2009.

7 Pedro Luis Mateo Navarro, Diego Sevilla Ruiz, Gregorio MartınezPerez. Automated GUI Testing Validation guided by AnnotatedUse Cases, MOTES 09 - Model-based Testing Workshop, Lubeck(Germany), September 2009.


Table of contents


8 Scientific Publications

9 Software Contributions


Main Software Contributions

OHTTesting framework (C++, Qt4)http://www.catedrasaes.org/wiki/OHT

S-DAVERVerification framework (C++, Qt4) + Rules (Lua/Chaiscript)http://www.catedrasaes.org/wiki/GuiVerification

PALADINInteraction model (EMF, Java) + HCI Extractor 8 (Java, Android)http://www.catedrasaes.org/wiki/MIM

CARIM:Interaction model (EMF, Java) + HCI Extractor 11 (Java, Android)http://www.catedrasaes.org/wiki/Carim


http://www.catedrasaes.org/wiki/OHT

http://www.catedrasaes.org/wiki/GuiVerification

http://www.catedrasaes.org/wiki/MIM

http://www.catedrasaes.org/wiki/Carim

Main Software Contributions — Validation

Tools validated into different scenarios:

OHT SAES developmentsS-DAVER and internal processes

PALADIN experiments in TLabs

CARIM experiments in Catedra-SAES lab

All tools are open-source, downloadable,and ready-to-use


Acknowledgements

www.catedrasaes.org

Supported by:

cátedrasaeslabs


Technology

PhD Thesis Defense - Enhancing Software Quality and Quality of Experience through User Interfaces