Evaluating usability through claims analysis

Suzette KeithAnn Blandford, Bob Fields,

Richard Butterworth, Yin Leng Theng

2

Aims

• Testing claims analysis as an analytical usability evaluation technique for digital libraries

• Validating claims with user trials• Testing usability of claims analysis for

designers• … towards developing a suite of usable

usability tools for digital libraries

Claims analysis

“claims analysis is an analytical evaluation method involving the identification of scenario features that have significant positive and negative usability consequences” (Rosson, Carroll 2002)

Case studies

1. Exploring claims analysis with the

developers

2. Applying claims analysis to an add-on

feature

3. User trial to investigate claims

Claims analysis: overview of method

1 Create scenarios - users, goals, story, action

2 Develop description of the artifact3 Generate claims4 Analyse claims5 Reason about design rationale

Data sources:• Interviews with users• Cognitive processing model• Research on information seeking• Research on digital library usage…Natasha is a recent graduate investigating

a new project. She decides to try the digital library and finds some results…

1 Create scenarios

2 Description of the artifact

• Design rationale• Usability specification• Functional specification• Mock-ups• Scripts• Partial implementation• Prototyping…she sees the search-box and types in a

query…

3 Generate claims

The Action cycle:How does the artifact:• evoke goals in the user?• support planning?• indicate action to take?• guide interpretation?• help evaluation and error recovery?

Goal claim

...Natasha is seeking information on a topic to support on-going project activity..

Claim about users goals:The digital library holds appropriate

information accessed by keyword searching and browsing

4 Analyse goal claim

• User can search or browse for information

• but • May not know appropriate terms• Content may not be appropriate

5 Reasoning about design rationale

• What factors might cause the user to fail• How can the effect of negative claims be

moderated• What trade-offs might effect other parts

of the system design

Case study 1:Exploring claims

Findings from developers:• Developers had designed the ‘best possible’

system • Informal use of novice scenarios• Focus on problems of awareness and

exploratory use• Difficult to identify negative consequences

of design decisions

Case study 2:Applying claims

Add-on feature in development:

• Hypothetical novice user scenarios

• Prototype interface

• Developers wanted support for design decisions

• Analysis strategy: plan, execute and evaluate

15

From ‘add a URL to this information space’

Plan? Action? Evaluation?

Claim for evaluation

….Natasha enters details into a form, the web-site or document url is notified to all profiled users..

Implemented feature: automatic completion sequence:

+ rapid execution minimises user effortbut- reduced opportunity to correct errors, maximum

opportunity for public embarrassment

Reasoning about evaluation claim

• Trade-off between speed and accuracy• Importance of feedback and consequence

of error• Exploration of alternative scenarios• Examination of technical issues• Proposed changes to interface

Case study 3:Novice user trials

The ‘ Google generation’• Positive and negative effects of prior

experience on goal, planning, action, evaluation

• Very specific information seeking task• Difficulty dealing with very large results sets• Progress measured by relevant full-text

documents found

The challenge

…what is exciting about digital libraries is the power they offer ordinary users…

The promise that information can be stored,accessed, manipulated and moved in ways that were hereto impossible…(Dillon)

20

Claims analysis: reprise

Can claims analysis contribute to the understanding of information seeking and the design of digital libraries ?

- task-artifact cycleCan claims analysis be specified to support the development of innovative and creative solutions to usability issues?

- concrete yet flexible

21

Thank you

s.keith@mdx.ac.ukwww.cs.mdx.ac.uk/ridl