Chapter 12 cont.: Introducing Evaluation

Three evaluation case studies

• Improving a design: the HutchWorld patient support system.

• Multiple methods help ensure good usability: the Olympic messaging system (OMS).

• Evaluating a new kind of interaction: an ambient system.

The HutchWorld patient support system

• This virtual world supports communication among cancer patients.

• Privacy, logistics, patients’ feelings, etc. had to be taken into account.

• Designers and patients speak different languages.

My Appearance

Evaluation

• Informal evaluation• Observation of users

– No critical mass– Users prefer asynchronous communication– Games were popular

• Usability evaluation in labs • Evaluation of the revised version• When it is time to stop testing?

Multiple methods to evaluate the 1984 OMS

• Early tests of printed scenarios & user guides.

• Early simulations of telephone keypad. • An Olympian joined team to provide

feedback.• Interviews & demos with Olympians

outside US.• Overseas interface tests with friends

and family.

Multiple methods to evaluate the 1984 OMS cont.

• Free coffee and donut tests (65 users).• Usability tests with 100 participants.• A ‘try to destroy it’ test (24 students). • Pre-Olympic field-test at an

international event.• Reliability of the system with heavy

traffic (2800 and 1000 users).• ->cultural differences

Evaluating an ambient system

• The Hello Wall is a new kind of system that is designed to explore how people react to its presence.

• What are the challenges of evaluating systems like this?

Key points

• Evaluation & design are closely integrated in user-centered design.

• Some of the same techniques are used in evaluation as for establishing requirements but they are used differently (e.g. observation interviews & questionnaires).

• Three main evaluation approaches are:usability testing, field studies, and analytical evaluation.

Key points cont.• The main methods are:observing, asking users, asking

experts, user testing, inspection, and modeling users’ task performance.

• Different evaluation approaches and methods are often combined in one study.

• Triangulation involves using a combination of techniques to gain different perspectives, or analyzing data using different techniques.

• Dealing with constraints is an important skill for evaluators to develop.

Chapter 13: An evaluation framework

The aims are:

• To discuss the conceptual, practical and ethical issues involved in evaluation.

• To introduce and explain the DECIDE framework.

DECIDE: a framework to guide evaluation

• Determine the goals.• Explore the questions.• Choose the evaluation approach and

methods.• Identify the practical issues.• Decide how to deal with the ethical

issues.• Evaluate, analyze, interpret and present

the data.

Determine the goals

• What are the high-level goals of the evaluation?

• Who wants it and why?• The goals influence the approach

used for the study.• Some examples of goals:

− Check to ensure that the final interface is consistent.

− Investigate how technology affects working practices.

− Improve the usability of an existing product .

Explore the questions

• All evaluations need goals & questions to guide them.

• E.g., the goal of finding out why many customers prefer to purchase paper airline tickets rather than e-tickets can be broken down into sub-questions:– What are customers’ attitudes to these new tickets? – Are they concerned about security?– Is the interface for obtaining them poor?

• What questions might you ask about the design of a cell phone?

Choose the evaluation approach & methods

• The evaluation approach influences the methods used, and in turn, how data is collected,analyzed and presented.

• E.g. field studies typically:– Involve observation and interviews.– Do not involve controlled tests in a

laboratory.– Produce qualitative data.

Identify practical issues

For example, how to:

•Select users•Stay on budget•Stay on schedule•Find evaluators•Select equipment

Decide about ethical issues

• Develop an informed consent form

• Participants have a right to:- Know the goals of the study;- Know what will happen to the findings;- Privacy of personal information;- Leave when they wish; - Be treated politely.

Ethical issues – Norway

• Norsk samfunnsvitenskapelig datatjeneste NSDhttp://www.nsd.uib.no/personvern/

• Innsamling av opplysninger om personer (personopplysninger) må meldes til Personvernombudet for forskning

• Personopplysninger er opplysninger som direkte eller indirekte kan identifisere en person.

• Direkte personidentifiserbare opplysninger: navn, personnummer eller andre personlige kjennetegn.

• Indirekte personidentifiserbare opplysninger: bakgrunnsopplysninger som kan gjøre det mulig å spore opplysningene tilbake til en enkeltperson, for eksempel bostedskommune eller institusjonstilknytning kombinert med opplysninger om alder, kjønn, yrke, nasjonalitet, diagnose, etc.

An example – Norway

Formålet med denne undersøkelsen er å utforske brukernes opplevelse av Internettjenester. Undersøkelsen er en del av et forskningsprosjekt hvor blant annet TV2/SUMO og SINTEF deltar. Det er i alt 19 spørsmål som tar ca. 10 minutter å besvare. Du velger selv om du vil svare på alle spørsmålene.

Dersom du vil være med i loddtrekningen av to gavekort, hver på 1000 kroner, må du oppgi e-postadressen din.

Vi ber om at det kun er personer over 15 år som besvarer denne brukerundersøkelsen.

SINTEF og TV2/SUMO garanterer at samlet data vil bli behandlet strengt konfidensielt, og senest innen prosjektslutt 15. mai 2010 vil opplysningene om deg anonymiseres. Ingen enkeltpersoner vil kunne identifiseres i fremtidige publikasjoner.

Our Experience

21

COOL

Technology

SE artefacts

0 %10 %20 %30 %40 %50 %60 %70 %80 %90 %

Exp

erim

enta

lM

ater

ial

Task

-pe

rform

ing

Act

ions

Pla

nnin

g,S

trate

gy a

ndR

efle

ctio

n

Com

preh

ensi

onan

d P

robl

ems

Inte

ract

ion

obse

rver

Categories

Perc

enta

ge CTARTAFCT

• Developers don’t care about monitoring• People give away sensitive information (bank account, passwords)

Karahasanovic, A. (IS, 2003).

Evaluate, interpret & presentdata

• The approach and methods used influence how data is evaluated, interpreted and presented.

• The following need to be considered:- Reliability: can the study be replicated?

• Field studies versus experiments

– Validity: is it measuring what you expected?• Average performance – recording user errors

Considerations

• Biases: is the process creating biases?– Expert evaluations

• Scope: can the findings be generalized?– Novices versus expert system users

• Ecological validity: is the environment influencing the findings? - i.e. Hawthorn effect. – Expectations of the users (placebo effect)

Key points

• There are many issues to consider before conducting an evaluation study.

• These include the goals of the study, the approaches and methods to use, practical issues, ethical issues, and how the data will be collected, analyzed and presented.

• The DECIDE framework provides a useful checklist for planning an evaluation study.

…and some lessons learned

• Encourage collaboration. Investigating motivation, user engagement, user involvement, co-experience and sociability at the level of communities and families is essential for applications aiming to support sharing and co-creation of UGC. Both tasks and evaluation methods should reflect this priority. Extending well known methods such as interviews, focus groups, and group-based expert walkthroughs with hands-on sessionsand usage of collaborative tasks has been very useful for capturing these factors.

• Start to evaluate UX as early as possible. Early feedback is very valuable to the developers. In particular, feedback on motivation, emotions, and anticipated engagement is valuable. However, one should adapt both the methods and the measurement to the evaluation phase. As the project progresses, one can move towards finer granularity evaluation. For example, one can measure the emotions related to a general idea of a tool for collaborative writing early in a project and emotions related to a particular function of the tool later in the project.

• Evaluation should be playful and provide added value for the participants. One cannot overemphasize the importance of providing a safe, comfortable and playful evaluation environment, and giving ‘something extra’ to the study participants. The opportunity to learn and try something completely new and to affect the development of new applications is not only very stimulating and rewarding for the communities of users and experts participating in the evaluation, but also positively affects usefulness of the evaluation methods. When working with communities it is very important to build a trustful relationship for ensuring a successful long term relationship.

• Prepare for diversity. In depth knowledge of your communities—the different groups of users and non-users—is essential for successful data collection. Different versions of questionnaires and focus group guidelines should be prepared for different user groups (e.g., professional cabaret artists, amateur artists, and theatres) and evaluators/moderators should be able to speak ‘different languages’ (e.g., to talk to children, teenagers, and elderly people) at the same time.

• Be best friends with the developer. Good knowledge of the application under development is very important for the success of the evaluation. Evaluators/moderators should be able to explain ideas behind paper prototypes and screenshots. Communicating the results of the evaluation clearly and in formats understandable to the developers is extremely important for uptake of the evaluation

results.Karahasanovic, A., and Obrist, M. (2010). Investigating the Usefulness of Methods for Evaluating User

Experience of Social Media Applications, Proceedings of the International Workshop on the Interplay between User Experience and Software Development (I-UxSED 2010), NordiCHI workshop, Reykjavik, Iceland, October 2010.