Putting Evidence-based Practice to Work: Usability Testing at Northwestern University Library

Frank CervoneAUL for Information Technology

Monday October 29, 2007Internet Librarian 2007

Overview

The problem of web site design Using more formal methods Formal methods in practice Outcomes

The problems of web site design

Web development/HCI is an intricate mix of technology and design– While there are exceptions, the majority of

librarians haven’t been trained in HCI – Gaps in understanding the significant differences

between the on-line and in-person experience Once a site is done, most people are not

eager to have others suggest adjustments or changes, particularly to those things they “know” are right

Liberally adapted from Problems with in-house website development by “The IT Guy”http://www.newfangled.com/contentmgr/showdetails.php/id/7441

Web site design and organizational responsiveness

The more an organization depends upon its publics for achieving its mission, the more it should employ dialogic features into its Web site design

The challenge is to move the tasks associated with Web site design and maintenance away from being a "B-list" job to being an imperative for the survival of highly stakeholder-dependent organizations

Kent, M. L., Taylor, M., and White, W. J. (2003). The relationship between Web site design and

organizational responsiveness to stakeholders. Public Relations Review, 29(1), pp. 63-77.

Which leads to …

Evidence-based information practice– “…an approach to information practice that

promotes the collection, interpretation, and integration of valid, important, and applicable user-reported, librarian-observed, and research-derived evidence”

Andrew BoothSenior Lecturer in Evidence-based Healthcare InformationSchool of Health and Related ResearchUniversity of Sheffield

What is evidence-based librarianship?

Data provides the primary evidence for making decisions– Not anecdotal stories– Not “common sense”– Evaluation occurs early in the process

How is this different from what often happens now? – Decisions are made based on

Beliefs of what is needed– Frequently, these beliefs are inherently biased

Assumptions, anecdotal evidence, and preferences

– Evaluation, if it occurs, happens afterward

Evidence-based professional practice

Derived from the evidence-based model of medicine

Fundamental precepts– Study the phenomenon– Contrast results to other studies of the same or

related phenomena– Combine results to better understand the

phenomenon at hand

Evidence-based practice process

Find the evidence

Evaluate the evidence

Apply results of the evaluation

Evaluate change

Define the problemRedefine the problem

Defining the problem

Very similar to concepts used in “user-centered design”

Five points– Yes, there’s an acronym - SPICE– SETTING

Where is this being used; what is the context?– POPULATION

Who are the (potential) users?– INTERVENTION

What is being done to/for them?– COMPARISON

What are the alternatives?– EVALUATION

What does success mean?

Levels of evidence

1. Systematic review of a wide-body of rigorous studies2. Systematic review of a wide-body of less rigorous

studies3. Randomized controlled trials4. Controlled-comparison studies5. Cohort studies6. Descriptive surveys7. Case studies8. Decision analysis9. Qualitative research (focus groups, ethnography,

historic, Delphi techniques, interviews)10. Surveys, audits

Adapted from Eldredge, J. (2000). Evidence-based librarianship: An overview. Bulletin of the Medical Library Association, 88(4). pp: 289-302. Online at http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=35250

Which brings us to the Northwestern story…

First usability test in 2001 Focused in on “Electronic Resources”

– A home-grown resource finder Investigation and questions based on

anecdotal evidence with ER interface Learning experience

– Provided model for organization Whetted people’s appetite for usability

testing

The second test – Searching in NUcat

Began summer 2002 How are people using Voyager?

– Are there discernable patterns? – Most importantly are there patterns of errors?– Can we do something to fix that?

What we did– Looked at searching patterns in our Voyager

system– Used a more rigorous research approach

Based on grounded theory Simple data mining techniques

And in the process…

0

200000

400000

600000

800000

1000000

1200000

1400000T

itle

Au

tho

r

Jo

urn

al T

itle

Su

bje

ct

Gu

ide

d K

eyw

ord

Ke

yw

ord

Gu

ide

d S

ea

rch

Bu

ilde

r

Su

bje

ct

He

ad

ing

Ad

va

nce

d K

eyw

ord

Ca

ll N

um

be

r

Co

mm

an

d

Ke

yw

ord

Ph

rase

Su

bje

ct

Bro

wse

ISB

N

Searches by type

…dispelled some myths

050000

100000150000200000250000300000350000400000450000500000

Title

Au

tho

r

Jo

urn

al T

itle

Su

bje

ct

Ke

yw

ord

Gu

ide

d K

eyw

ord

Su

bje

ct

He

ad

ing

Bu

ilde

r

Gu

ide

d S

ea

rch

Ad

va

nce

d K

eyw

ord

Co

mm

an

d

Ca

ll N

um

be

r

Ke

yw

ord

Ph

rase

ISB

N

ISS

N

No hits within searches by type

Which lead to our tackling a bigger problem

Existing site did not work especially well for the Library's various user communities– Both students and faculty expressed frustration

with the web site Ability to navigate the site Overall look and feel

– Problem areas determined by gathering information in focus groups, online surveys, interaction with librarians and staff

Hours and locations Collections Electronic resources

Moving to a more evidence-based process

Restructuring of the web development process Creation of a active web group

– Library Web Advisory Group Representation from all divisions of the Library

– Reference/Instruction– Collection management – Bibliographers– Branch libraries – Technical Services– Library Public Relations– Digital Library Systems Development– Library Information Technology Support Services

Web Communications Group at University Relations

Induction process

Required reading list– Extensive list of supplemental readings

Required training– In usability– Conducting a usability test– Other “soft skills”

Evidence-based procedures

Required participation in design and testing– Usability test moderator– Usability test observer– Transcriber– Missionary/representative

Current results

Overall, site usability has improved– And we can provide it with statistical measures!– People find resources more quickly and easily

But there is still an unfortunate amount of confusion Debates are less rancorous about how to

proceed– We can always go back to the data– We can always go back to the studies

Easier to develop a strategy for incremental improvements over time– No longer locked into a tight academic schedule

Remaining issues

Jargon– I don’t know what

– Metasearch– E-journals– Virtual reference– Electronic resources

means! Why should I go here? I can find everything

in Google (or Google Scholar)– To some degree, this is true…

The big issue

Designing for a world where they don’t even come to the library web site– When the day comes that Google Scholar or

Microsoft Academic Live do provide a “universal index”

– Pluggable modules for the University portal

Thanks

Frank CervoneAssistant University Librarian for Information TechnologyNorthwestern University Library1970 Campus DriveEvanston, IL 60208

email: f-cervone@northwestern.eduAIM/YAHOO: hfcervoneSkype: frank.cervone