Introduction to EvaluationJanuary 26, 2005

Slide 2Innovation Network, Inc.

Who We Are: Innovation Network

National nonprofit organization Committed to evaluation as a tool for empowerment Build evaluation capacity of nonprofits and funders so

they can better serve their communities Practice a participatory approach

Slide 3Innovation Network, Inc.

Objectives of This Session

Identify how evaluation can be useful in your work

Understand both implementation and outcome evaluation

Understand how evaluation planning feeds into data collection

Slide 4Innovation Network, Inc.

What is Evaluation?

The systematic collection of information about a program that enables stakeholders to better understand the program, improve program effectiveness, and/or make decisions about future programming.

Slide 5Innovation Network, Inc.

What’s in it for you?

Understand and improve your program Test your theory of change/program theory Tell your program’s story Be accountable Inform the field Support fundraising efforts

Slide 6Innovation Network, Inc.

Evaluation Principles

Evaluation is most effective when it: Is connected to program planning and delivery Involves the participation of stakeholders Supports an organization’s capacity to learn and reflect Respects the community served by the program Enables the collection of the most information with the

least effort

Slide 7Innovation Network, Inc.

Implementation and Outcomes

Evaluating Implementation/Process: What did you do? How well did you do it?

Evaluating Outcomes: What difference did you make through your work? Or, what changes occurred because of your work?

Slide 8Innovation Network, Inc.

Evaluating Outcomes

Outcomes: the changes you expect to see as a result of your work

Indicators: the specific, measurable characteristics or changes that represent achievement of an outcome. They answer the question: How will I know it?

Slide 9Innovation Network, Inc.

Evaluating Outcomes:Common Types of Change

New knowledge

Increased skills

Changed attitudes, opinions or values

Changed motivation or aspirations

Modified behavior

Changed decisions

Changed policies

Changed conditions

Slide 10Innovation Network, Inc.

Evaluating Outcomes: Sample Plan

Outcomes Indicators Data Collection Method

Data Collection Effort

(have, low, med, high)

Participants improve job-seeking skills

#/% of participants who meet criteria in mock interview

Observation of mock interview at end of training session using checklist

Low

#/% of participants who develop a quality resume

Job counselors review resumes based on quality checklist

Have

Participants improve money management skills

#/% of participants who balance their checkbooks

Review participants’ check registers & work sheets

Low

#/% of participants who pay credit card bills in full & on time

Review participants’ credit card statements

Low

Slide 11Innovation Network, Inc.

Evaluating Implementation

Activities and Outputs: The “what” —the work you did, and the tangible products of that work

Additional Questions: The “why”—understanding how well you did, and why

Slide 12Innovation Network, Inc.

Evaluating Implementation: What Did You Do?

Examine Activities and Outputs Did you conduct activities as planned? Did those activities produce the outputs you envisioned?

How to measure? Program documents

Slide 13Innovation Network, Inc.

Evaluating Implementation: How well did you do it? What information will help you understand your program

implementation? Think about:: Participation Quality Satisfaction Context

How to measure? Program documents, surveys, interviews, comment functions, focus groups, other methods

Slide 14Innovation Network, Inc.

Evaluating Implementation: Sample Plan

Activities Outputs &Implementation Questions

Data Collection Method

Data Collection

Effort(have, low, med, high)

TRAININGDevelop/revise curriculum for training seriesMeet with potential program clientsCoordinate logisticsProvide training series to two groups of clients

Outputs Curriculum (developed/revised)2 training series heldCompletion by 30 of 33

Program recordsProgram recordsProgram records/attendance logs

Have

Have

Questions

Are we reaching the clients we expected?

Review of participant intake data

Low

Slide 15Innovation Network, Inc.

Data Collection: 3 Steps

Choose the method Decide which people or records will be the source of

the information Determine the level of effort involved in using that

method with that population

Slide 16Innovation Network, Inc.

Data Collection Steps 1 & 2:Choose Method, Identify Source

Review documents Observe Talk to people Collect written responses Pictorial/multimedia

Slide 17Innovation Network, Inc.

Data Collection Step 3:Level of Effort

Instrument development Cost/practicality of actually collecting data Cost of analyzing and presenting data

Also consider: Communication Power Proxy Power

Slide 18Innovation Network, Inc.

Good Data Collection Characteristics: Culturally appropriate (survey, focus group, who

asks, manner of asking)

Ethical

Respectful of participants

Slide 19Innovation Network, Inc.

Continuous Learning Cycle

Program Plan

Analysis, Reflection &Improvement

Evaluation Planning

Data Collection

Slide 20Innovation Network, Inc.

Next Steps

We can do more! Data Collection and Analysis

Jan 27 @ 9 am Online, instructor-led trainings Online, self-paced modules Individual technical assistance

Slide 21Innovation Network, Inc.

Thanks for Your Participation!

Measure results.

Make informed decisions.

Create lasting change.Innovation Network, Inc.

1625 K St. NW, 11th FloorWashington, DC 20006

(202) 728-0727www.innonet.org

Ehren Reed: ereed@innonet.org

www.innonet.org/services/LSC_Train