1

OrientationALA Midwinter Meeting

January 15, 2010

2

Workshop Goals

Create a common understanding and increased knowledge of outcome-based planning and evaluation so grantees may refine and finalize Smart investing @ your library evaluation and marketing plans.

Set the stage for effective program implementation.

3

Overview & Agenda

Why Measure Outcomes? What is Outcome-Based Evaluation? How to Measure Outcomes How to Collect Data How to Use Data Closing Thoughts and Next Steps

4

Why Measure Outcomes?

See if programs really make a difference in the lives of people.

Improve programs. Improve planning. Increase accountability. Ensure best use of funds. Compare programs consistently. Demonstrate impact. Funders demand it.

5

Why Measure Outcomes?

"Outcome measurement has the potential to be a powerful tool to help us substantiate the claims we know to be true about the impact of libraries in our institutions and in our societies.

Will it be an easy road to travel? No, but it will absolutely be worth the trip!"

-- Peggy Rudd, Texas State Librarian

6

What Is Outcome-Based Evaluation?

Outcome-based evaluation is a systematic way to assess how well a program has achieved its intended results. How has my program made a

difference? How are the skills of the program

participants improved as a result of my program?

7

What Is Outcome-Based Evaluation?

8

Get Ready

Audience is key Users Staff Program partners

Key terms Situation Inputs Activities Outputs Outcomes

9

Get Ready

10

Outputs/Outcomes

Three programs held

Web site developed

Print, electronic collection increased 10%

100 PSAs run

50 people from target audience attend each; report they would recommend the program

Increased use of investment Web site from beginning to end of the program.

Increased use of investment collection.

10% of participants report PSA motivated attendance

Outputs/Outcomes

Good: Train 10 staff.

Better: 80% of trained staff will show improved skills as a result of training.

Best: 50% of trained staff will lead trainings and receive positive evaluations from library patrons.

11

Outputs/Outcomes

Good: Program attendance will average at least 20 people.

Better: 70% of participants will demonstrate increased financial literacy.

Best: 50% of participants will open a savings account within six months of the program end.

12

13

Get Ready

Smart investing program elements Staff training Classes/programs/training/exhibits Web presence Collection development and positioning

in physical/virtual library Marketing/outreach Partnerships

14

Choose the Outcomes You Want to Measure

Smart investing overall goals: Community members will view the

library as a reliable place for unbiased investor information

Community members will make increased use of library programs and resources

Community members will be more knowledgeable about key investment issues.

15

Choose the Outcomes You Want to Measure

Smart investing outcomes at the library level: Increased requests for investment

materials Increased visits to library Web site Increased awareness of library

resources Increased staff competencies

16

Choose the Outcomes You Want to Measure

Smart investing outcomes at the library level: Users demonstrate increased skills

and/or knowledge Users take action with new skills (i.e.

start investing, reduce debt) Program partners report a positive

experience working with the library PR/Marketing activities boost program

participation

17

Specify indicators for your outcomes

Smart investing indicator examples Program attendance Web hits Survey feedback Increased circulation of related materials Trained staff will provide patron training New library card enrollments New saving or investing behavior.

18

Specify indicators for your outcomes

Measurable: That is, you can test for the change or observe it. But if you made a movie of success, the camera would focus on people, not on mechanisms or processes used to create the hoped-for results.

Changes in participants: Remember we’ve defined an outcome as a change in a target audience’s skills, attitudes, knowledge, behavior, status, or life condition brought about by experiencing a program.

Define success:  Does the outcome represent a benefit for the target audience? Do key stakeholders accept the outcome as valid for your program? Finally, is the outcome realistic and attainable?

Participating in your program:  Is it sensible to claim your program services influenced the outcome?

19

Marketing Example

GOAL: Develop an outreach plan to inform local citizens that the library is a resource center for financial planning. The marketing plan will identify community

members in each targeted audience who will benefit most from the project.

Community members will be well-informed of the spectrum of activities available throughout the project.

20

Staff Example

GOAL: Reference staff will know sources of accurate and unbiased information on investing. Participation by reference staff in

investment education resources training.

Reference staff will identify new web resources and add them to library Web page.

21

Hands-on

Take 15 minutes to review your goals and outcomes and make two to three improvements based on our conversation. How can you focus on outcomes, as well as outputs? How can you make the objectives more specific and measurable?

22

Prepare to collect data on your indicators

Pre-test and Post-test You can’t measure success without a

baseline.

What is the “current state of affairs” – what do people know, perceive and do before the program… and how does the program move the audience forward?

Include all stakeholders

23

Prepare to collect data on your indicators

Data source options Feedback forms/short surveys Point-of-use inquiry by staff Focus groups Interviews Skills tests Observation Instructor assessments Library use statistics

24

Prepare to collect data on your indicators

Other considerations: When will you collect data? How often will I collect data? Include all participants or a sample? Who will collect data? Who will record/compile data? How will confidentiality be protected? How will participants be informed about

the data collection process?

25

Try out Your Measurement System

Pilot or “beta test” your surveys or questionnaires. Clarity of questions Ease of use Are you measuring what you intended

to measure? Are you asking the most appropriate questions?

REVISE instrument as needed!

26

Analyze and Report Findings

Collect, input data at regular intervals Mid-project reporting Troubleshooting Revising outcomes as appropriate

It’s as much about the journey as the destination

27

Analyze and Report Findings

o Review feedback from participantso Get very familiar with the datao Look for and note oddities in

reportingo Peruse the data and identify

patternso Substantiate patterns – do data

sources corroborate each other?

28

Analyze and Report Findings

Organize data logically (tables/charts)

Analyze and interpret data to develop narrative for final report

Document findings Maintain files or database of

outcomes and activities Determine outcomes you want to

continue monitoring

Analyze and Report Findings

Hiring an outside evaluator What to look for? What to expect? Cost?

29

30

Improve Your System

More important to discontinue something that isn’t working than to report back that you did everything you said you would do at start of the program when it failed.

31

Hands-on

Take 15 minutes to review your evaluation plan and the data you gathered for today’s workshop. What baseline data do you need that you don’t yet have? How will you get it?

32

Use Your Findings

Tell your story! Marketing Accountability and long-term

assessment Improved services and/or programs Resource (re)allocation

33

Marketing

Outcomes can inform key messages about the library and what it does well.

Outcomes allow stakeholders to understand, in users’ own words, the powerful role of the library.

34

Marketing

Identify your audiences Funder Targeted media outlets Partners

Match audiences and outcomes Determine dissemination strategies

Annual report Press release Flyers and brochures

35

Closing

Questions?

One thing you’ve learned today

One thing you’re going to do as a result of what you’ve learned today

Evaluation form

36

Additional resources

IMLS National Leadership Grants tutorial: http://www.imls.gov/project_planning/index1.asp

NEFE Financial Education Evaluation Toolkit:

http://www.nefe.org/eval/ Florida State Library workbook:

http://dlis.dos.state.fl.us/bld/Research_Office/OutcomeEvalWkbk.doc

37

ALA Office for Research & Statistics

Denise Davis, Director312-280-4273

Larra Clark, Project Manager202-628-8410 x8213