Assess for Success: Proving Library Value

Steve Hiller

Director, Planning and Assessment University of Washington Libraries

Assessment Basics

• What do we need to know about our communities and customers to make them successful?

• How do we measure the effectiveness of our services, programs and resources?

• What do our stakeholders need to know in order to provide the resources needed for a successful library?

Assessment Is More than Numbers

Library assessment is a structured process:

• To learn about our communities

• To respond to the needs of our users

• To improve our programs and services

• To contribute to the success of our communities

Assessment Process

• Focuses on understanding customer needs and offering services that meet those needs

• Collects, analyzes and uses quantitative and qualitative data for more effective management and decision-making

• Emphasizes ongoing communication with customers and stakeholders,

• Seeks opportunities for collaboration and comparisons within the organization, institution and beyond

What Do We Need to Know About Our Customers?

• Who are our customers (and potential customers)?

• What areas/fields/courses are they working in?

• How do they work? What’s important to them?

• How do they find information needed for their work?

• How do they use our services? What would they change?

• How do they differ from each other in use/needs?

• Outcomes

How do we add value to their work?

How do we contribute to their success?

The Value of Community Assessment

• Identify actual and potential customers• Understand needs and use preferences• Use funding and staff effectively• Understand and address “competition”• Encourage community involvement and “ownership” • Target marketing, market penetration and outreach• Measure, demonstrate, present the value of the library

to the community and stakeholders

Thinking Strategically About Library Futures

• What is our central work and how can we do more, differently, and at less cost?

• What important services do we provide that others can’t?

• What advantages do we possess?

• How is customer behavior changing?

• How do we add value to our customers work?

• What are the essential factors responsible for library success now and in the future?

The Challenge for Libraries

• Traditional statistics and data are no longer sufficient– Emphasize inputs – how big and how many– Do not tell the library’s story– May not align with organizational goals and plans– Do not measure service quality

• Need data from the user’s perspective• Need to provide organizational accountability to

funders and other stakeholders• Need to define where we want to be in 3/5/more years• Need to answer: What value do we provide our

primary community and how do we show it?

Defining Success in a Digital Environment

• Crafting new indicators and measures of success

• Moving from measuring inputs to outputs and outcomes

• Understanding impact of library/partner roles and services on the community

• Agreeing on qualitative measures of success: user perceptions, user success, creating value, advancing community goals.

• Reallocating resources and managing capabilities to achieve success.

• Demonstrating value of library contributions

Some New Metrics for Academic Libraries

• Uniqueness of collections

• Value of consortia

• Administrative and budgetary efficiencies

• Student outcomes/student learning/graduate success

• Contributions to faculty productivity

• Social frameworks/intellectual networks

• Generating new knowledge

• Creating the collective good with reusable assets such as an institutional repository or e-science

Good Assessment Starts Before You Begin . . . Some Questions to Ask

• Define the question– What do you need to know and why

• How will you use the information/results

• Where/how will you get the information– What are appropriate methods

– Is there existing information

– New data (where or who will you get it from)

– Is it cost and resource effective

• How will you analyze the information

• Who will act upon the findings

How Do We Get Customer Information?

• Statistics/data mining (local, institutional)• Surveys• Focus groups• Observation• Usability• Interviews• Embedding• Logged activities• Comments, suggestions, “over the counter”, being there

Another View of Metrics

Getting The Right Data Isn’t Enough

“…but to suppose that the facts, once established in all their fullness, will ‘speak for themselves’ is an illusion.”

Carl BeckerAnnual Address of the President of the American Historical Association, 1931

Understanding what the data means and how to follow-up are crucial

Make Data Meaningful

• Summarize

• Compare

• Analyze

• Present

• Go below the surface to examine results by:– Demographic group

– Users and non-users

– Stakeholders vs non-stakeholders

• Compare current data with past; look for trends

• How can we use the data for action?

Data Presentation: 3 Key Questions

• What’s the message?– Fewer “messages” means greater impact

• Who’s the audience?– Multiple audiences may need different presentations

• How do we present?– Quantitative data - Be graphic , provide meaning/understanding

– Qualitative data - Be selective, provide corroborating data

• Keep it simple AND focused!

And Remember the Content!

Moving from Analysis to Action

• From all of the data, determine what can and should be addressed

• Prioritize the important/critical action items– Make evidence-based decisions

– Align with mission, vision and goals of parent organization

– Address what is important to customers

• Establish action guidelines– Who, how, when

What Makes it Hard to Be Evidence-Based?(From Pfeffer and Sutton, 2006)

• There’s too much evidence

• There’s not enough good evidence

• The evidence doesn’t quite apply

• People are trying to mislead you

• You are trying to mislead you

• The side effects outweigh the cure

• Stories are more persuasive anyways

When the Evidence Isn’t Used

Collection Assessment: The 4 Questions

What’s important from the:

Library perspective?

Customer perspective?

Stakeholder perspective?

Vendor/publisher perspective(s)?

Some Good Reasons to Assess Collections

• Meeting user needs • Costs/Budget/Space• Changes in the information discovery/use processes• Accountability to stakeholders, funding agencies• Breadth and depth; comprehensiveness• Environmental changes (social, information, technical)• Changes in publisher/vendor marketing and packaging• Return on Investment (ROI); contingent valuation

What Numerical/Cost Data Do We Need?

• Costs – Collections (annual/over time)

• Titles

• Packages, consortia

– Operational• Staff

• Facilities

• Vendors

• Usage – Print (on-site, check-out, ILL)

– Electronic/online• Type of use

– Packages, consortial, doc delivery

– Change over time

• Cost per use– What type of use

• Downloads

• Searches

• Loans

• Page views

– Packages, individual titles

• Global information– Collection comparisons

– Publishing output

– Bibliometric tools, citation analyses, collection

– Community demographics

So We Need Systems/Products That:

Enable us to track and validate usage Enable us to track item level and package level costsProvide comparative informationProvide “canned”/custom reports easily• Provide data in flexible formats we can manipulate/analyse• Provide information that meets established standards• Present snapshot and trend data• Are cost-effective

E-Metrics Tools: COUNTER, SUSHI and ERMI

• Counting Online Usage of NeTworked Electronic Resources– Standardized way to count usage

– Defined reports for journals, databases, and eBooks

• Standardized Usage Statistics Harvesting Initiative – Model for automation of statistics harvesting

– SUSHI feed: usage data that magically appears!

• Electronic Resources Management Initiative– Can accept SUSHI feeds

– Easy to generate COUNTER-style reports within ERM module

– Brings in data from order records to calculate cost per use

Genesis Statistics Detail

2007 payment / 2007 Use = 2007 CPU

$1950.71 / 649 = $3.00

Title2007 CPU

2007 Use

2008 CPU

2008 Use

Genesis $3.00 649 $3.67 93

Don’t Forget the Qualitative

• Discovery pathways and user expertise

• Usability

• User needs

• User outcomes– What did the information enable the user to accomplish?

Integrated Collection Assessment

• Collections-centered– Size, growth, formats, authors, subjects, comparisons

• Cost-centered– Expenditures, price, inflation, cost per use, return on

investment

• Use-centered– Loans, e-usage, purchase requests, ILL

• Customer centered (secondary)– Community demographics, programs, research, bibliometric

studies, consortial

• Customer centered (primary)– Behavior, impact, outcomes, success, discovery

What Will We Do With the Information?

• Analyze and understand

• Present and share

• Make good decisions and choices

• Ensure we add value to the work of our community

Customers and stakeholders make the ultimate determination of value!

Focus on the UW Libraries Contribution to the Research Enterprise

• How does the library support sponsored research?

• What is the value of the library contribution?

• Some measures & indicators of library contributions– Collections budget and usage data

– Survey information

– Bibliometrics

– Frequency of use

– Time saved/contingent valuation

– Information seeking behavior

– Qualitative information

– Services used

– Library research projects/collaborations

The Importance of the Research Enterprise University of Washington Operating Revenues

$2.4 Billion in 2007-08

Tuition 18%

Other 11%

Gifts8%

State funding 15%

Investment Income 3%

Research Grants 45%

Research Grants $1.1 BillionHealth and Human Services

$510 millionNational Science Foundation

$80 millionOther federal agencies

$200 millionIndustry/Foundations

$110 millionOther non-federal

$160 million

University of Washington Libraries Assessment Methods Used

• Large scale user surveys every 3 years (“triennial survey”) beginning in 1992. In 2007:– 1500 faculty responses; 600 grad; 500 undergrad

• In-library use surveys every 3 years beginning 1993– 4000 surveys returned in 2008

• Focus groups/Interviews (annually since 1998)• Observation (guided and non-obtrusive)• Usability• Usage statistics/data miningInformation about assessment program available at:

http://www.lib.washington.edu/assessment/

Traditional Library Core Business (Usually in-person)

• Physical Collections – Print (primarily) – Microforms – Other

• Facilities – House collections– Customer service & work space– Staff work space

• Services– Reference– Instruction– Access

UW Libraries Usage Data Items Used In-Library800,000 in 2002-03300,000 in 2007-08(6.0 million article downloads in 2007-08)

Gate Counts4.6 million in 2002-034.3 million in 2007-08(8.7 million Web site user sessions in 2007-08)

In-Person Reference Questions140,000 in 2002-03 90,000 in 2007-08(18% of all ref queries are virtual in 2007-08)

The Changing Business Model: Trends in Library Use at the University of Washington

34

Qualitative Information Provides Context BioScience Interview/Focus Groups (2006)

• Content is primary link to the library– Identify library with ejournals; want more titles & backfiles

• Provide library-related services and resources in our space not yours– Discovery begins primarily outside of library space with

Google and Pub Med; Web of Science also important

– Library services/tools seen as overly complex and fragmented

• Print is dead, really dead– If not online want digital delivery

– Go to physical library only as last resort

• Personal connection is important

UW Faculty Mode of Use by Academic Area 1998/2007 (w eekly or more often)

0% 1% 1%Visit Only 6%

Visit Only10%Visit Only10%

Remote & Visit 32%

Remote & Visit 42%

Remote & Visit 51%

Remote & Visit 17%

Remote & Visit 39%

Remote & Visit

7%

Remote Only45% Remote

Only87%

Remote Only26% Remote

Only 72%

Remote Only 23%

Remote Only 47%

Non- Weekly 17%

Non- Weekly5%

Non- Weekly 25%

Non- Weekly 9% Non- Weekly

15%

Non- Weekly,9%

Health Sci1998

Health Sci2007

Science-Engin1998

Science-Eng2007

Hum-Soc Sci1998

Hum-Soc Sci2007

36

Off-Campus Remote Use 1998-2007(Percentage using library services/collections at least 2x week)

Faculty

Undergrad

Grad

0%

10%

20%

30%

40%

50%

60%

70%

1998 2001 2004 2007

0%

10%

20%

30%

40%

50%

60%

70%

76% of faculty (80% of those using federal research funds) connect online at least 2x week

Primary Reasons for Faculty Use of Libraries Web Sites by Academic Area (2007 Triennial Survey, at least 2x per week)

15%

25%

35%

45%

55%

65%

75%

Health Sci Science-Engin Hum-Soc Sci

Library Catalog Bib Database Online journal articlesThe ability to access full-text or pdf research articles online through the library subscriptions is my primary use of the library and is central to my research. Neurobiology Grad Student

Importance of Books, Journals, Databases byAcademic Area Scale of 1 “not important” to 5 “very important)

Books

Journals<1985

Bib Databases

Journals>1985

3.2

3.4

3.6

3.8

4

4.2

4.4

4.6

4.8

Health Sciences Science-Engineering Hum-Soc Science

Books Journals<1985 Bib Databases Journals>1985

UW Libraries Serial Purchasing and Usage

• 85% of collections budget spent on serials/backfiles– Ranked 4th in 2006-07 ARL serial expenditures ($11.4 million)

• 6 million journal article downloads (Counter compliant)

– 30% increase over 3 years

– 75% of downloads from scholarly journals in science-engineering-health sciences

• Bibliographic database use has declined– Most pronounced in subject specific databases

– Web of Science unchanged at 160,000 login sessions annually (13% of all log-in sessions)

UW Libraries Contribution to Sponsored Research

Among faculty receiving federal research funding:

• 97% rate Libraries as very important to their work

• 96% rate journals as most important info resource for their work

• 93% rate the Libraries as making a major contribution to keeping current in their field

• 93% rate the Libraries as making a major contribution to their research productivity

• 80% connect to the Libraries at least twice per week from office, lab or off-campus

Differences in Library Contribution by Funding

Source: (Scale of 1 “Low to 5 “High”)

4 4.2 4.4 4.6 4.8 5

Journal importance

Overall Importanceto work

Keeping current inyour field

Productive research

Fed Funding

No fed funding

Libraries Contribution to Your Being a More Productive Researcher by Academic Area and

Funding Source

4

4.25

4.5

4.75

5

All Humanities-Soc Sci Science-Engineering Health Sciences

No federal research funds

Federal research funds

Closing the Loop: Success with Assessment

• Assess what is important• Keep expectations reasonable and achievable • Use multiple assessment methods; corroborate• Mine/repurpose existing data• Focus on users; how they work, find & use information • Use the data to improve and add customer value• Keep staff, customers and stakeholders involved and

informed

Eye to the Future

Measuring performance is an exercise in measuring the past. It is the use of that data to plan an improved future that is all important. Peter Brophy

• Data trends can inform the future

• Strategic planning can frame the future

• Organizational performance models can align ongoing operations with future aspirations

• Understanding how customers work and how that work is changing is key to our future