Our Most Important Challenge

Raising the Assessment Bar: A Challenge to our Community

Rick Luce, Emory University

Library Assessment ConferenceSeattle – August 4, 2008

Environmental Scan

1. University mission evolving: Education reform and the Spellings commission focus on

outcomes and accountability Globalization and competition

2. New research methods in a networked world: Rise of eScience / eResearch = new ways to work, new

needs and expectations Data science & data scientists require new organizational

environments

3. Social drivers: Technology enabled social tools to connect & collaborate

My View after 2 years at Emory

Aggressive Strategic Plan completed in first 4 months:

Strategic direction: (1) digital innovations, (2) special collections, (3) delivery of 21C access, resources, and services

$100M requested in new funds over 5 years Concept approval for ~$36M for 3 new capital projects

Implement annual Business Plan – reviewed quarterly

None of which utilize current ARL statistics

We Need a Systems Approach The human body is a system, our subsystems work together

to keep us healthy

Research libraries are systems, requiring a management system to keep the subsystems working together to be healthy

Anyone who learns to see the organization as a system can never again feel satisfied with “improvement” initiatives which simply change staffing and the org chart but do not tackle the system itself

Where Assessment Fits

Assessment – a method of planning for improvement Catalyst for organizational change (not a quick fix)

Gain staff understanding for need for improvement & commitment to shared improvement goals

Ideally underpinned by a performance measurement matrix balancing: Quality = customer defined goodness – internal &

external Time = speed, how fast is the response, agility Cost = resources spent on people, processes, or

organizational shifting or rework

Performance Measures: our Vital Signs

Statements without performance measures are wishful thinking -- without data, we don’t know

We all have volume or transaction data - provides no process insight

Move focus on product metrics to process metrics Process performance (statistics, run charts, variation)

Getting to the ‘right’ metrics What is the value equation? How do we compare & differentiate ourselves? Focus on : Customer, process, sponsorship metrics

Hedgehog View: Constancy of Purpose

What are webest at?

What are wepassionate

about?

What drives our economic

(or value)engine?

Adapted from: Jim Collins. Good to Great. 2001

ID Key Success Factors

Characteristics of successful organizations:*

1. Do something others cannot do

2. Do something well that others do poorly, or

3. Do something others have great difficulty doing well

~10-15% (max.) of research libraries content / services are unique

What % of the budget resources support that?

*Prahalad and Hamel. The Core Competencies of the Corporation. HBR, May/June 1990.

Management Principles of Successful Organizations

Strong customer focus

Effective leadership

Continuous improvement and learning

Management by fact

Fast response

Long-range view of the future

Results orientation

Cooperation, teamwork, partnering

A Value Proposition Example (LANL) Scientific Productivity & Competitiveness

Enhancing research productivity and competitiveness ROI = $4.50 : 1.00

Quality and Business Focused 10 year average: 96% satisfaction rate; 25% delighted Reduced cost per transaction by a factor of 16 Quality New Mexico ‘Roadrunner’ awards: 1997 & 2000 Federal Library of the Year 1999 “World’s Best Science Library” 2005 Blue Ribbon Review

Follow the Leaders

Malcolm Baldrige National Quality Award: recognizes organizations practicing the most effective management methods

High performance is sustainable due to good management practices

Analysis of 600 winners over 10 years: growth = > 2.5 times as fast as peers, more than 2X more profitable

Examines approach, deployment and results

Baldrige Criteria for Performance Excellence

1. Leadership

2. Strategic Planning

3. Student, Stakeholder, and Market Focus

4. Measurement, Analysis, and Knowledge Management

5. Workforce Focus

6. Process Management

7. Results

Requires a system both in approach and deployment

Baldrige Applications: Lessons Learned

Accelerated learning using Baldrige framework

“System” is tough to integrate all at once“Be patient, have discipline” Deming

Importance of supplier partnerships

Difficulty of language translation

Benchmarking data - time series data for competitors – couldn’t be obtained from libraries

Benchmarking Process for gaining and applying knowledge to

improve business process performance from a study of current practices. A means of using data to identify magnitudes and reasons for

variances in performance.

Intent: comparative process data, best practices Analyze the operation, know the competition & industry

leaders Incorporate the best of the best - become the new

benchmark

Missing from Research Library Portfolios

Customer satisfaction index: (delight & loyalty) and perceived value

Product / service quality (defined by the customer)

Process and operational performance - cycle time, productivity

Employee satisfaction – learning, morale, training, alignment of strategy direction and rewards

Measuring supplier performance - quality, process variables, price competitiveness, overall ease of doing business

Financial: cost/value matrix, return on investment, cost avoidance

Customer Satisfaction Metrics

Maturing our satisfaction assessment

Level 1 – Satisfaction surveys: ‘happiness meters’

Level 2 - What’s important analysis of customer importance & satisfaction

levels

Level 3 - How do we rate against best in industry

See it from the customers eyes

Measures that matter

Align library vital signs with the organization’s drivers

Quality of product and processes

Innovation

Research leadership

Brand identity

Growing market share

Reducing new product development time

Ability to attract and retain employees

Credibility

Avoiding Pitfalls

Measures that don’t focus on strategy

No accountability

Too many initiatives

Forgetting larger organizational drivers

Lack of discipline

Insulating researchers and managers from scholarly communication issues

No action without a plan, no plan without data

1. Customer focus: Satisfaction, loyalty, value-added

2. Product quality: E.g., accessibility, usability, accuracy, completeness

3. Operational process performance: Productivity, competitiveness, cycle time

4. High performance workforce: Organization performance assessment, formal process

changes, employee satisfaction5. Prestigious reputation -- output results

Strategic performance results, benchmark results, external assessment scores

Business Scorecard - Desired Business Results

A Tale of 2 Libraries: Budget Allocations

$M

Payoff for Successful Quality Implementation

Any road will do if the destination is unknown

The journey to truly superior performance is neither for the faint of heart nor for the impatient.

The development of genuine expertise requires struggle, sacrifice, and honest, often painful self-assessment.

HBR: Anders Ericsson (FSU), Michael Prietula (Emory), Edward Cokely (Max Planck)

If you wish to do something for the community, build a road. If you wish to do something better for the community, build a bridge – Chinese Proverb

Together let’s build the bridge to a new level of assessment practice,

supporting continuous improvement and focusing on outcomes and impact.