Beyond LibQUAL+ ® : Effective, Sustainable, and Practical Assessment Other tools: ClimateQUAL ®, MINES for Libraries ®, annual statistics Other approaches:

• Beyond LibQUAL+®: Effective, Sustainable, and Practical Assessment

• Other tools: ClimateQUAL®, MINES for Libraries®, annual statistics

• Other approaches: Return on Investment, Value & Impact

Martha Kyrillidou, Glasgow, May 24 2010

ARL Mission

• Non-profit organization of the libraries of research institutions in North America

• Forum for exchange of ideas

• Agent for collective action

What makes a research Library?

• Breadth and quality of collections and services• Sustained institutional commitment to the library• Distinctive resources in a variety of media• Services to the scholarly community• Preservation of research resources• Contributions of staff to the profession• Effective and innovative use of technology• Engagement of the library in academic planning --from ARL ‘Principles of Membership’

•What makes a quality research library?

“Quality much like beauty is in the eye of the beholder”

What’s in a “Library”

• A word is not crystal, transparent and unchanged; it is the skin of a living thought, and may vary greatly in color and content according to the circumstances and time in which it is used. – --Justice Oliver Wendell Holmes

Thinking Strategically About Library Futures

• What is the central work of the library and how can we do more, differently, and at less cost?

• What important set of services does the library provide that others can’t? What new roles are needed?

• What advantages does the research library possess?

• What will be the most needed by our community of users in the next decade? How is user behavior changing?

• What should our libraries aspire to be ten years from now? What are the implications of technology driven change?

• What are the essential factors responsible for the success of the library?

www.arl.org

A gateway to assessment tools: ARL StatsQUAL™:

ARL Statistics -- E-Metrics

LibQUAL+®

ClimateQUAL™

DigiQUAL™

MINES for Libraries™

•Library Assessment Conferences•Service Quality Evaluation Academy•Library Assessment blog•Effective, Sustainable, Practical Assessment•Profiles … Scenarios …. Balanced Scorecard …. LibValue

Assessment at ARL

www.arl.org

http://www.climatequal.org

ARL Tools for Library Assessment

As we look to the future…

New Tool New

Tool

New Tool

Common Database

StatsQUAL®

Common User Interface

Unified Data Structure

Enhanced Data Mining/Warehousing Opportunities

Faster Development Cycle for New Tools

Common Workflow

Users


ARL Tools for Library Assessment

As a result of the work of the New Measures and Assessment Initiative (1999)…

ARL Statistics™

Since 1907-08

LibQUAL+®

Since 2000

MINES for Libraries™

Since 2003

DigiQUAL®

Since 2003

ClimateQUAL™

Since 2007

Partners

2008• Arizona State

University• Cornell University• Duke University• Emory University• Kansas State

University• New York University• Northwestern

University• University of

Houston• University of

Maryland• University of

Massachusetts, Amherst

2007 • Texas A&M

University• University of

Arizona• University of

Connecticut• University of Iowa• University of

Kansas

2009 George Mason

University Illinois State

University Johns Hopkins

University Oberlin College University of

California, Berkeley University of Hawaii

at Manoa University of

Nebraska-Lincoln University of

Wyoming


Goals & Research Question for Project

• Develop a tool that assesses the “health” of your library.– Whether the policies, procedures and practices of your

library are supporting your mission and facilitating meeting current and future customer needs.

• Develop large database of norms to help libraries interpret their results.

• Develop an active community of libraries that share experiences and interventions to improve the effectiveness of libraries participating in community

• Track changes in libraries over time to permit objective feedback regarding effectiveness of attempted practices/interventions.

• Empirical validation of the healthy organization theory.


Core Concepts

• Climate for Diversity• Climate for Fairness

– Distributive Justice– Procedural Justice– Informational Justice– Interpersonal Justice

• Climate for Innovation• Climate for Continual Learning• Climate for Teamwork• Climate for Psychological Safety• Climate for Customer Service• Immediate Supervisor Scales

– Authentic Leadership; Leader-Member Relationship Quality• Work Group Conflict• Worker Attitudes


2000 U. Maryland Survey

Results• Issues identified:

– Managerial training– Mentoring– Standardization of procedures– Decision-making practices, etc.

• Recruitment/selection of ethnic minorities• Performance review• Rewards and recognition

Transformative Changes

• Holding All-Staff Meetings

• Emphasis on learning

• Creating training for supervisors

• Created new position - Coordinator of Personnel Programs

• Focus on team building


2004 U. Maryland Survey

Results• Support of diversity and

communication practices• Employees feel they are

treated fairly• Teamwork is valued• Increase in

Organizational Withdrawal from 2000 to 2004

• Indication of consistent ethnic and divisional differences

Transformative Changes

• Conducted focus groups within two divisions

• Groups and teams identifying strategies

• Developing supervisor core competencies

• Incorporating Organizational Citizenship Expectations into work plans

• Beginning workforce planning


Community Building

• No one-size-fits-all method of interpreting results or implementing changes

• Creation of intimate community of participants

• In-person events, an online shared workspace, and conversations

• Allows for shared experiences concerning improvement strategies

Assessing the Value of Networked Electronic Services:

Measuring the Impact of Networked Electronic Services (MINES) - MINES for Libraries™

www.arl.org/stats/newmeas/mines.html

The MINES survey

• A research methodology consisting of a web-based survey form and a sampling plan.

• Measures who is using electronic resources, where users are located at the time of use, and their purpose of use.

• Adopted by the Association of Research Libraries (ARL) as a part of the “New Measures” toolkit May, 2003.

• Different from other electronic resource usage measures that quantify total usage (e.g., COUNTER, EQUINOX, E-Metrics, ICOLC guidelines, ISO and NISO standards) or measure how well a library makes electronic resources available (LibQUAL+®, DigiQUAL™).

What is MINES?

• How extensively do sponsored researchers use the new digital information environment?

• Are researchers more likely to use networked electronic resources from inside or outside the library?

• Are there differences in usage of electronic information based on the user’s location (e.g., in the library; on-campus, but not in the library; or off-campus)?

• What is a statistically valid methodology for capturing electronic services usage both in the library and remotely through web surveys?

• Are particular network configurations more conducive to studies of digital libraries patron use?

Questions Addressed

Effective, Sustainable and Practical Assessment

• Association of Research Libraries (ARL) project “Making Library Assessment Work” began in 2005 to “Assess the state of assessment efforts in individual research libraries, identify barriers and facilitators of assessment, and devise pragmatic approaches to assessment that can flourish in different local environments”

• Funded by participating libraries• Conducted by Steve Hiller and Jim Self

– 1.5 day site visit– Presentation and best practices– Interviews and meetings – Report to the library with recommendations

Library Assessment Process

• Focuses on customer needs, defining measurable “outputs” and offering services that meet those needs

• Collects, analyzes and uses data for management, program development, and decision-making

• Emphasizes ongoing communication with customers, opportunities for collaboration, qualitative measures and circular process of continuous improvement

MLAW/ESP:Data Collection Methods

Pre-Visit• Survey on assessment activities, needs etc.• Telephone follow-up• Mining library and institutional web pages

Visit (1.5 days)• Presentation on effective assessment • Group meetings

Follow-up and report• Pursue leads and additional information

Commonly Identified Assessment Needs (30 Libraries)

67%

70%

80%

87%

90%

97%

0% 25% 50% 75% 100%

Build AssessmentCulture

Learning Outcomes

Staff Data Use Skills

Data Analysis

Data Collection

Use Data Effectively

Organizational Indicators of Effective Assessment

• Library leadership/management “truly” supportive

• Customer focus is a shared library value

• Organizational culture receptive to change & improvement

• Assessment responsibility recognized and supported

• Library has strategic planning process and prioritizes

• Evidence/Data used to improve services/programs

– Web sites (usability)

– Facilities (qualitative methods)

– Serial subscriptions (emetrics)

– LibQUAL+™ results are followed-up

Building a Community of Practice

• Biennial Library Assessment Conference– 220 registrants for 2006 conference in Charlottesville, VA– 380 registrants for August 2008 in Seattle, WA

• Workshops– Biennial Service Quality Evaluation Academy– Full day and half day workshops

• Library Assessment SPEC Kit (December 2007)– SCONUL/ARL collaboration for UK and Irish libraries

• Assessment tools– LibQUAL+® (“Millions served”)– MINES for Libraries®– ClimateQUAL™, DigiQUAL™ and more

Library Assessment Conference

Library Strategy Maps as of

April 14, 2009

Customer Needs:

Internal Processes

Marketing & Communications

Library Services Operational Efficiency

LearningAndGrowth

Customer Perspective Financial Perspective

Recruit, develop, and retain productive, highly qualified

staff

Recruit, develop, and retain productive, highly qualified

staff

Strategic Skills / Workforce Infrastructure

Implement space and technology upgrades to

support cutting-edge services and initiatives

Implement space and technology upgrades to

support cutting-edge services and initiatives

CulturePromote culture of

assessment and accountability

Promote culture of assessment and accountability

Develop “community” model of leadership and organizationDevelop “community” model

of leadership and organization

The intellectual crossroads where The intellectual crossroads where research and teaching, tradition and research and teaching, tradition and innovation, faculty and students all innovation, faculty and students all

intersectintersect..Facilitate world-class research, teaching and learning through leadership in the creation, access, use, and preservation of

information and knowledge for scholars at the University of Virginia

UW Strategy Map Based on Strategic Plan Mission

Enhance undergraduate

services

Enhance undergraduate

services

Reshape library spaces

Reshape library spaces

Customer Perspective

Demonstrate library value to research

productivity

Demonstrate library value to research

productivity

Optimize organizational structure and

efficiency

Optimize organizational structure and

efficiency

Financial Perspective

Internal Perspective

Learning and Growth Perspective

Marketing & Communications

Library Services Operational Excellence

Communicate and market library services &

resources

Communicate and market library services &

resources

Develop and promote scholarly communication

plan

Develop and promote scholarly communication

plan

Integrate resources & services into user

environments

Integrate resources & services into user

environments

Strengthen library liaison

program

Strengthen library liaison

program

Build a management information structure Build a management

information structure

Create a workplace of choice

Create a workplace of choice

Enriching the Quality of Life and Advancing Intellectual Discovery by Connecting People with Knowledge

Enriching the Quality of Life and Advancing Intellectual Discovery by Connecting People with Knowledge

Develop staff expertise that supports current & future

TLR

Develop staff expertise that supports current & future

TLR

Strengthen digital and

physical delivery services

Strengthen digital and

physical delivery services

Provide technological infrastructure to support

staff and user needs

Provide technological infrastructure to support

staff and user needs

Culture Infrastructure Strategic SkillsStrategic Skills

Assess effectiveness of services & programs Assess effectiveness of services & programs

Provide access to information resources for world-class TLR

Provide access to information resources for world-class TLR

Strategy Map

Improve discovery of and

access to scholarly resources

Improve discovery of and

access to scholarly resources

Create world-class teaching

& learning environments

Create world-class teaching

& learning environments

Strive for exemplary

service that is responsive to

user needs

Strive for exemplary

service that is responsive to

user needs

Marketing and Communications

Services Operational Effectiveness

Promote the Library’s role in the discovery, dissemination,

and preservation of knowledge

Promote the Library’s role in the discovery, dissemination,

and preservation of knowledge

Integrate the Library into the

University’s teaching,

learning, and research mission

Integrate the Library into the

University’s teaching,

learning, and research mission

Develop highly-trained,

technologically-fluent

superlative staff

Develop highly-trained,

technologically-fluent

superlative staff

Nurture a healthy,

collaborative, and dynamic organization

Nurture a healthy,

collaborative, and dynamic organization

Grow an evidence-

based culture

Grow an evidence-

based culture

Align the Library’s budget with the

University’s mission

Align the Library’s budget with the

University’s mission

Increase alternate sources of revenue

e.g. fundraising, grants and revenue

generation

Increase alternate sources of revenue

e.g. fundraising, grants and revenue

generation

Mission: advance teaching, learning and research at McMaster by teaching students to be successful, ethical information seekers, facilitating access to information resources, providing welcoming spaces for intellectual discovery and promoting the innovative adoption of emerging learning technologies

Track efficiency and effectiveness of Library programs and services

Track efficiency and effectiveness of Library programs and services

Enhance the Library’s commitment to strategic

planning

Enhance the Library’s commitment to strategic

planning

Encourage innovation and

risk taking

Encourage innovation and

risk taking

Mission: To advance research, teaching, learning at Johns Hopkins University by providing relevant and significant collections, services, and instruction for faculty, students, and staff. The Sheridan Libraries through the University Libraries Council partners with other Johns Hopkins Institutions

libraries to maximize support for our constituents while minimizing costs.

Mission: To advance research, teaching, learning at Johns Hopkins University by providing relevant and significant collections, services, and instruction for faculty, students, and staff. The Sheridan Libraries through the University Libraries Council partners with other Johns Hopkins Institutions

libraries to maximize support for our constituents while minimizing costs.

Customer Needs Provide

productive, user-centered

workspaces (virtual and physical)

Provide productive, user-

centered workspaces (virtual and physical)

Ensure fast, easy access to resources and services needed

for research and teaching

Ensure fast, easy access to resources and services needed

for research and teaching

Financial Perspective

Develop strong support base

Develop strong support base

Internal Processes

Communications Programs Operational Efficiency

Promote resources

and services

Promote resources

and services

Expand and preserve access to information

including digital library development

Expand and preserve access to information

including digital library development

Create productive research, learning, study, and social

space

Create productive research, learning, study, and social

space

LearningAndGrowth

Recruit, develop, and retain

productive highly qualified staff

Recruit, develop, and retain

productive highly qualified staff

Promote an adaptive, collaborative, diverse, engaged,

innovative workplace

Promote an adaptive, collaborative, diverse, engaged,

innovative workplace

Create a culture of assessment and accountability

Create a culture of assessment and accountability

JHU Library Strategy Map

Strategy Map

Minimize costsMinimize costs

Identify and improve processes and

infrastructure with greatest impact

Identify and improve processes and

infrastructure with greatest impact

Integrate and strengthen

instructional and research support

services and programs

Integrate and strengthen

instructional and research support

services and programs

ARL Profiles

• Describes the here and now and strategies

• 86 qualitative descriptions of research libraries• Five pages each• Address issues related to collections, services and

collaborative relations• Preliminary report available: directors.arl.org• Working on identifying key areas and defining an

ARL Statistics 2.0 survey

ARL Scenarios

• Process led by Karla Hahn, Associate Executive Director for Transforming Research Libraries, with external consultant

• Interview and focus groups on out of the box thinking attempting to describe different scenarios of the future of research libraries in 20 years or so

• A visioning tool• FAQ on ARL website

IMLS Lib-Value Grant

Partner Institutions: U of Tennessee, U of Illinois at Urbana-Champaign and ARL

Primary Goals• Establish a fact-based articulation of the value

and ROI of the university library resources and services within the wider mission of university administration.

• Develop a model for ROI and tools that implement this model which can be used by other academic libraries.

Impact

• More informed librarians who understand how to conduct and present ROI studies for their libraries and who can measure all aspects of library services and collections

• Improved understanding by university library funders of the ROI and value of university libraries to the overall mission of the university and what products and services are the best investments

01/04/2010

Introductions - Management Team

• Carol Tenopir – UT ([email protected])

• Paula Kaufman – UIUC ([email protected])

• Martha Kyrillidou – ARL ([email protected])

• Donald King - UNC ([email protected])

• Bruce Kingma -Syracuse ([email protected])

• Tina Chrzastowski – UIUC ([email protected])

• Gayle Baker – UT ([email protected])

• Ken Wise – UT ([email protected])

• Rachel Fleming-May – UT ([email protected])

• Regina Mays – UT ([email protected])

• Crystal Sherline – UT ([email protected])

• Andrea Baer – UT ([email protected])

01/04/2010

Overview of Functional Areas

02/01/2010

02/01/2010

All Stakeholders considered -- emphasis on faculty and students)

Overview of Sub-Areas

02/01/2010

June Workshops

In Washington DC ….

Return on Investment – Lib-Value with Paula Kaufman, Don King and Bruce Kingma

Value and Impact with Stephen Town

Thank You

Documents

Beyond LibQUAL+ ® : Effective, Sustainable, and Practical Assessment Other tools: ClimateQUAL ®, MINES for Libraries ®, annual statistics Other approaches: