EDUCAUSE 2002 Service Quality Assessment in a Digital Library Environment Service Quality Assessment in a Digital Library Environment Atlanta, Georgia

EDUCAUSE 2002

Service Quality Assessment in a Service Quality Assessment in a Digital Library Environment Digital Library Environment

Atlanta, GeorgiaOctober 3, 2002

Duane Webster

Overview of DiscussionsOverview of Discussions

1. New Models for Understanding and Describing Library Success

2. ARL’s New Measures Initiative

3. LibQUAL+ Project Development

4. Experience with LibQUAL+

5. Conclusions and Next Steps

The Association of Research Libraries

www.arl.org

ASSOCIATION OF RESEARCH LIBRARIES

Mission: Shaping and influencing forces affecting the

future of research libraries in the process of

scholarly communication.

Members: 123 major research libraries in North America.

Ratios: 4% of the higher education institutions

providing 40% of the information resources.

Users: 3 million students and faculty served.

Expenditures: $2.35 billion annually, $727 million for

acquisitions of which 9% is invested in access

to electronic resources.

ARL New Measures InitiativeARL New Measures Initiative

Collaboration among member leaders with strong interest in this area

Specific projects developed with different models for exploration

Intent to make resulting tools and methodologies available to full membership and wider community

LibQUAL+LibQUAL+™™ Description Description

LibQUAL+TM is a research and development project undertaken to define and measure library service quality across institutions and to create useful quality-assessment tools for local planning.

Project ResourcesProject Resources

LibQUAL+TM is an ARL/Texas A&M University joint effort. The project is supported in part by a 3-year grant from the U.S. Department of Education’s Fund for the Improvement of Post-Secondary Education (FIPSE) and a 3-year grant from the National Science Foundation (NSF)

The ImperativeThe Imperativefor our Researchfor our Research

In an age of accountability, there is a pressing need for an effective and practical process to evaluate and compare research libraries. In the aggregate, among the 122 Association of Research Libraries (ARL) alone, over $2.8 billion dollars were expended in 1999/2000 to satisfy the library and information needs of the research constituencies in North America (Kyrillidou & Young, 2001, p. 5).

The Problem of Assessment in The Problem of Assessment in Research LibrariesResearch Libraries

The lack of metrics

ARL Membership Criteria Index variables emphasize inputs, primarily expenditures

No demonstrable relationship between expenditures and service quality

To rise in the ARL Index it is only necessary to spend more

AssessmentAssessment

“The difficulty lies in trying to find a single model or set of simple indicators that can be used by different institutions, and that will compare something across large groups that is by definition only locally applicable—i.e., how well a library meets the needs of its institution. Librarians have either made do with oversimplified national data or have undertaken customized local evaluations of effectiveness, but there has not been devised an effective way to link the two.” Sarah Pritchard, Library Trends, 1996

LibQUAL+LibQUAL+™™ Project Goals Project Goals

Establishment of a library service quality assessment program at ARL

Development of web-based tools for assessing library service quality

Development of mechanisms and protocols for evaluating libraries

Identification of best practices in providing library service

LibQUAL+LibQUAL+TM TM ParticipantsParticipants

Year 1

Year 2

Year 3

Spring 2001Spring 2000 Spring 2002

12 Participants

43 Participants

164 Participants

For More Information about Participants:

Visit the LibQUAL+ web site.

Relationships: perceptions, Relationships: perceptions, service quality and satisfactionservice quality and satisfaction

….only customers judge quality;all other judgments are essentiallyirrelevant”

Zeithaml, Parasuraman, Berry. (1999). Delivering quality service. NY: The Free Press.

Source: Parasuraman, ARL Symposium on Measuring Service QualityWashington, DC, October 2000

70+ Interviews conducted 70+ Interviews conducted

York University University of

Arizona University of

Connecticut University of

Houston University of Kansas

University of Minnesota

University of Pennsylvania

University of Washington

Smithsonian Northwestern

Medical

Empathy

Ubiquity and Ease of Access

Comprehensive Collections

Reliability

Responsiveness

Symbol

Utilitarian space

Assurance

Formats

Timely access to resources

Physical location

Self-reliance Library as Place

LibraryServiceQuality

Figure 1: Dimensions of Library Service Quality

?

Refuge

Affect of Service

Atlas TI Dimensions of AnalysisAtlas TI Dimensions of Analysis

“By habit, I usually try to be self-sufficient. And I’ve found that I am actually fairly proficient. I usually find what I’m looking for eventually. So I personally tend to ask a librarian only as a last resort.”

Graduate student

Self-relianceSelf-reliance

“…first of all, I would turn to the best search engines that are out there. That’s not a person so much as an entity. In this sense, librarians are search engines [ just ] with a different interface.”

Faculty member

Self-relianceSelf-reliance

Affect of ServiceAffect of Service

“I want to be treated with respect. I want you to be courteous, to look like you know what you are doing and enjoy what you are doing. … Don’t get into personal conversations when I am at the desk.”

Faculty member

Comprehensive CollectionsComprehensive Collections

“I think one of the things I love about academic life in the United States is that as a culture…, we tend to appreciate the extraordinary importance of libraries in the life of the mind.”

Faculty member

Comprehensive CollectionsComprehensive Collections

“I sense that if I were in an institution that didn’t have the rich collections as this library and the very effective staff members that this library has that I would imperceptibly slip in my discipline….”

Faculty member

Ubiquity of AccessUbiquity of Access

“Over time my own library use has become increasingly electronic. So that the amount of time I actually spend in the library is getting smaller and the amount of time I spend at my desk on the web … is increasing.”

Faculty member

“You put a search on a book and it’s just gone; it’s not reacquired. … There’s more of a problem of lost books, of books that are gone and nobody knows why and nobody’s doing anything about it.”

Faculty member

ReliabilityReliability

ReliabilityReliability

“I put something on reserve. And it didn’t show up, and somebody complained. I went back and said ‘I’ve asked for this to be put on reserve’ and they had lost the form. So I had to do it again.”

Faculty member

“I guess you’d call them satisfiers. As long as they are not negatives, they won’t be much of a factor. If they are negatives, they are a big factor.”

Faculty member

Library as PlaceLibrary as Place


“The poorer your situation, the more you need the public spaces to work in. When I was an undergraduate, I spent most of my time in the library, just using it as a study space.”

Faculty member


“One of the cherished rituals is going up the steps and through the gorgeous doors of the library and heading up to the fifth floor to my study. … I have my books and I have six million volumes downstairs that are readily available to me in an open stack library.”

Faculty member

Dimensions of LibraryDimensions of LibraryService QualityService Quality

Affect of Service

Empathy

Information Access

Personal Control

Responsiveness

Symbol

Utilitarian space

Assurance

Scope

Timeliness

Convenience

Library as Place

LibraryServiceQuality

Refuge

Reliability

Ease of Navigation

Convenience

Modern Equipment

Affect of ServiceAffect of Service

Emerged as the dominant factor early in our work

Absorbed several of the original SERVQUAL questions measuring Responsiveness, Assurance and Empathy

In the current analysis also includes Reliability

All in all: the Human Dimension of Service Quality


Transcends the SERVQUAL dimension of Tangibles to include the idea of the library as the campus center of intellectual activity

As long as physical facilities are adequate, library as place may not be an issue

Personal ControlPersonal Control

How users want to interact with the modern library

Personal control of the information universe in general and web navigation in particular

Access to InformationAccess to Information

Ubiquity of access: information delivered in the format, location and time of choice

Comprehensive collections

42 — ARL Libraries 35 — Health Sciences Libraries 36 — State Colleges & Universities

(excluding ARL) 34 — Private Colleges & Universities

(excluding ARL) 15 — Community Colleges 2 — Special & Public Libraries

(Smithsonian & NYPL)

LibQUAL+LibQUAL+ 2002 Iteration 2002 Iteration

The Challenge of AnalysisThe Challenge of Analysis

There are few, if any useful conclusions to be drawn from aggregate data of all institutions, because their missions and subsequent user expectations for service are too diverse.

There are commonalities in service delivery profiles that merit further investigation.

In the long run, information that may be derived from demographic responses of individuals may yield the richest data.

Two Interpretation Two Interpretation FrameworksFrameworks

Score Norms

Zone of Tolerance

Zone of ToleranceZone of Tolerance

The area between minimally acceptable and desired service quality ratings

Perception ratings ideally fall within the Zone of Tolerance

Aggregate DimensionAggregate DimensionSummary (n=70,445)Summary (n=70,445)

Note: LibQUAL+ Spring 2002 Aggregate Survey Results. (2002). vol. 1, p. 24

Note: LibQUAL+ Spring 2002 Aggregate Survey Results. (2002). vol. 2, p. 40

Faculty Item SummaryFaculty Item Summary

Score NormsScore Norms

Norm Conversion Tables facilitate the interpretation of observed scores using norms created for a large and representative sample.

LibQUAL+TM norms have been created at both the individual and institutional level

ARLTop 40

6.84(.46)

ARLOther6.74(.27)

PrivateColleges

6.90(.49)

State Colleges& Universities

6.38(.30)

CommunityColleges

7.26(.55)

Overall Mean Scores and Overall Mean Scores and Service Adequacy Gap ScoresService Adequacy Gap ScoresBy Cohort GroupBy Cohort Group

2002 LibQUAL+2002 LibQUAL+ Iteration Iteration(n=162)(n=162)

AAHSL7.07(.56)

LibQUAL+LibQUAL+™ ™ Fundamental Contributions Fundamental Contributions to the Measurement of Effective Delivery to the Measurement of Effective Delivery

of Library Servicesof Library Services

Shift the focus of assessment from mechanical expenditure-driven metrics to user-centered measures of quality

Re-ground gap theory for the library sector, especially academic libraries

Grounded questions yield data of sufficient granularity to be of value at the local level

Determine the degree to which information derived from local data can be generalized, providing much needed “best practices” information

Demonstrate the efficacy of large-scale administration of user-centered assessment transparently across the web

Makes little demand of local resources and expertise

Recognize the limitations of Recognize the limitations of listening to customerslistening to customers

Customers have a limited frame of reference and tend to offer incremental, rather than bold, suggestions

– A better slide rule

– The microwave oven, Post-it Notes, Velcro

Innovation is the responsibility of staff

Anthony W. Ulwick, Harvard Business Review, January 2002

Shift the focus to outcomesShift the focus to outcomes

Plan outcome-based customer interviews Capture desired outcomes Organize the outcomes Rate the outcomes for importance and satisfaction

– Opportunity algorithm:

(Importance+(Importance-Satisfaction)=Opportunity) Use the outcomes to jump-start innovation


When desired outcomes When desired outcomes become the focus of customer become the focus of customer research, innovation becomes research, innovation becomes

a manageable, predictable a manageable, predictable discipline.discipline.


SummarySummary Survey can handle large numbers Survey can be turned around quickly Limited local expertise required Interpretations should be across chosen

cohorts Lots of opportunities for using demographics

to discern user behaviors Q-technique and other tests will provide

opportunities to observe how institutions may cluster

LibQUAL+LibQUAL+ Related Related DocumentsDocuments

LibQUAL+LibQUAL+ Web Site http://www.arl.org/libqual/

LibQUAL+LibQUAL+ Bibliographyhttp://www.coe.tamu.edu/~bthompson/servqbib

Survey Participants Procedures Manual http://www.arl.org/libqual/procedure/lqmanual2.pdf

NDSL LibQUAL+ ActivitiesNDSL LibQUAL+ Activities

• 120-200 qualitative interviews to contribute to identifying dimensions of digital library service quality

• Test and refine dimensions of digital library service quality and self-sufficiency through development of total market survey

• Implement survey across variety of organizational and digital library implementations

NDSL LibQUAL+ GoalsNDSL LibQUAL+ Goals

• Define dimensions of digital library service quality from the users’ perspectives

• Develop tool for measuring user perceptions and expectations of digital library service quality across NSDL digital library contexts

• Identify digital library best practices that permit generalizations across operations and development platforms

NSF GrantNSF Grant

• Assess service quality in digital libraries

• 3 year period

• Adopt LibQUAL+ instrument for use in the Science, Math, Engineering and Technology Education Digital Library community (NSDL)

LibQUAL+LibQUAL+TM TM Project HistoryProject HistorySep

t. 19

99

Jan20

00

July

200

0

Sept

. 200

0

Oct

. 200

0

Jan.

200

1

June

200

1

9/99 - ARL launches “New Measures Initiative” which includes the study of service effectiveness known as SERVQUAL spearheaded by Texas A&M University.

1/00 - Initial 12 institutions begin the SERVQUAL study led by the Texas A&M team.

7/00 - LibQUAL+ as a distinct library-based assessment tool is presented.

9/00 - ARL and Texas A&M awarded a FIPSE grant to fund further development of the LibQUAL+ project.

10/00 - The ARL symposium, “New Culture of Assessment in Academic Libraries Measuring Service Quality” attracts a group of 170 people.

1/01 - Representatives from 43 research and university libraries participating in the Spring 2001 Implementation meet in Washington, DC during ALA midwinter.

6/01 - National Science Foundation awards grant to ARL and Texas A&M to adapt LibQUAL+ for NSDL

LibQUAL+LibQUAL+TM TM TeamTeam

ARL

• Duane Webster

• Martha Kyrillidou

• Kaylyn Hipps

• Julia Blixrud

• Jonathan Sousa

• Consuella Waller

TAMU

• Fred Heath

• Colleen Cook

• Bruce Thompson

• Yvonna Lincoln

• Trey Thompson

• Julie Guidry

Overview of DiscussionsOverview of Discussions

1. New Models for Understanding and Describing Library Success

2. ARL’s New Measures Initiative

3. LibQUAL+ Project Development

4. Experience with LibQUAL+

5. Conclusions and Next Steps

The Problem of Assessment in The Problem of Assessment in Research LibrariesResearch Libraries

• ARL Membership Criteria Index variables emphasize inputs, primarily expenditures

• To rise in the ARL Index it is only necessary to spend more

• No demonstrable relationship between expenditures and service quality

• The lack of metrics describing performance