Performance Measures AmeriCorps Project Director Training Saratoga, NY October 7 th – 9 th, 2013

Performance Measures

AmeriCorps Project Director TrainingSaratoga, NY

October 7th – 9th, 2013

Theory of Change

Activity

Performance Measure Basics Overview

Activity/Questions

AGENDA

Theory of Change

INTRODUCTIONTheory of Change Perspective

Looks at cause and effect relationships

Identifies specific interventions to achieve the desired result

Shifts thinking from “what are we doing” to focus on “what do we want to achieve”

Overview of Prohibited ActivitiesTheory of Change Elements

STATISTICS documenting the need


EVIDENCE •Guides choice of intervention

•Supports cause-effect relationship



Prohibited ActivitiesEveryday Life Example

I have strep throat (problem) I will take antibiotics (intervention) I will get better (outcome)





Prohibited ActivitiesEveryday Life Example cont.

EVIDENCE Penicillin Amoxicillin×Tetracycline

EVIDENCE Penicillin Amoxicillin×Tetracycline

There are many antibiotics out thereWhich antibiotic do I take for strep?Look at evidence to make the choice

Overview of Prohibited ActivitiesTheory of Change Elements







Community Problem/Need is the specific issue your project, with its specific intervention (service activity), is designed to address. What is the extent and severity of this need in the community?

Overview of Prohibited ActivitiesTheory of Change Elements - Community Need

Data documenting problem/need should answer these questions:

SCOPE: Who and how many are directly affected? How severe is this?

SIGNIFICANCE: What makes this a compelling need? Is it likely to become worse? What will happen if we do nothing?

CAUSE(S): Why does the need exist? How is it perpetuated?

Overview of Prohibited ActivitiesTheory of Change Elements - Statistics



The best data comes from reputable primary sources like government agencies, institutions, and universities that do their own research that is: Locally relevant Up-to-date

Note: New reports are not primary sources but could be used as back-up to primary evidence.

Data documenting problems/needs should be statistic that document the extent and severity of the problem.

Overview of Prohibited ActivitiesTheory of Change Elements - Outcome







What change are you hoping to make related to the identified need?

Overview of Prohibited ActivitiesTheory of Change Elements - Outcome

Healthy Futures Program Example

Possible outcomes to measure: Increased knowledge of what is “healthy food” More frequent choice of healthy foods to eat More frequent involvement in physical activity or exercise Improved physical condition

Identifying the Intended Outcome.

Overview of Prohibited ActivitiesTheory of Change Elements - Intervention







An intervention is the specific set of activities in which participants and volunteers will be engaged. What is the best way to achieve the intended outcome?






Design – who does what with whom? Dosage

Frequency – how many sessions a week? Intensity – length of each session. Duration – how many total weeks of sessions?

Describe the design and dosage of your intervention (service activity):


Design: national service participants implement the Shape Up curriculum with economically disadvantaged urban girls ages 14-16 to increase physical activity (30 minutes/session) and educate them on healthy eating

Frequency: twice a week afterschool Intensity: 60 minutes per session Duration: 12 weeks

Healthy Futures Program Example

Overview of Prohibited ActivitiesTheory of Change Elements - Evidence





Evidence demonstrates that the proposed intervention (design, dosage) is likely to lead to the outcome.

Service activities that are supported by research to demonstrate a high likelihood that these activities will address the need, and result in intended changes.






Evidence provides a reality check for theory of change

New programs: What is the recommended design (specific program activities) and dosage (frequency, intensity, and duration) to achieve an intended outcome? For existing programs: Is there sufficient evidence for the intervention to continue its use? Based on the evidence, are there modifications to the intervention that should be considered to have greater impact? Do you need to choose a new intervention?

EVIDENCE: Information that supports your choice. It is NOT enough to just say “we believe” our intervention is “likely” to be successful.






Your past performance measurement outcome data

Results from an impact evaluation of your program

Research that documents the outcomes of similar programs

Impact evaluations that document outcomes of similar programs

Possible sources of evidence:

Overview of Prohibited ActivitiesTesting Your Theory of Change

PLAUSIBLE: Does the logic of the model seem correct: “if we do these things, will we get the results we expect?”

FEASIBLE: Are resources sufficient to implement the chosen intervention?

MEANINGFUL: Are intended outcomes important? Is the magnitude of expected change worth the effort?

Is your Theory of Change:

Overview of Prohibited ActivitiesTheory of Change Summary

A theory of change identifies cause/effect

The three elements of a theory of change; community problem/need, intervention, intended outcome are supported by data and evidence

Data documenting community need should show scope, significance, and causes

Summary of Key Points:

Case Study

Performance Measure Overview

Overview of Prohibited ActivitiesPerformance Measures

Accountability to funders and stakeholders Tell your story, justify funding

Recognizing progress; measuring change Reliable information collected in a systematic way

Program improvement Spot and correct problems Strengthen the intervention Determine where to allocate limited resources

Why Measure Performance?


Using Performance Measurement in Program Management


Using Performance Measurement in Program Management continued

Overview of Prohibited ActivitiesPerformance Measures – Outputs and Outcomes

OUTPUTSAmount of services being

providedo People servedo Products createdo Programs developed

OUTCOMESReflects the changes or

benefits that occur

Can reflect changes in individuals, organizations, communities, or environment

Address changes in attitudes/beliefs, knowledge/skill, behavior, or conditions

Overview of Prohibited ActivitiesPerformance Measures – Outcomes

Outcome Example: EDUCATION

Overview of Prohibited ActivitiesPerformance Measures - Alignment

Aligning Theory of Change with Performance Measures


Alignment: Logical flow and connection between the different elements.

Alignment helps you:

Strengthen your theory of change

Articulate you performance measures

Measure more accurately

Cleary report change


Example: Aligned Output-Outcome with Performance Measure

Overview of Prohibited ActivitiesPerformance Measures – Impact Evaluation

Differences Between Impact Evaluations and Performance Measures

Key AreasPerformance Measurement

Impact Evaluation

CausalityAssumes causality; does not “prove” theory of change

Seeks to show causality and “prove” theory of change

ImplementationTracks outputs and outcomes on a regular outgoing basis

May occur from time to time, but not on a regular ongoing basis

Time FocusShorter term focus; what can be measured within one year

Longer term focus

DesignBalance rigor with practicality

Uses most rigorous evaluation design

Overview of Prohibited ActivitiesPerformance Measures- Summary

Performance measurement is a systematic process of measuring progress (outputs and outcomes)

Performance measurement does not seek to “prove” a theory of change, but can provide snapshots.

Impact evaluation can determine if results occurred because of the intervention.

Strong performance measures align with the theory of change (need, intervention and outcome)

Key Points

Overview of Prohibited ActivitiesPerformance Measures- Summary

CNCS Priorities and Performance Measures:

http://www.nationalservice.gov/resources/npm/home

Program-Specific Notices of Funding Opportunities and Application Instructions:

http://www.nationalservice.gov

Additional Resources

Project Director TrainingSaratoga, NY

October 7th – 9th, 2013

Questions?

Documents

Performance Measures AmeriCorps Project Director Training Saratoga, NY October 7 th – 9 th, 2013