Serve DC Training 9/28/11. Theories of Change and Logic Models Evidence Performance Measurement 101 Reviewing Performance Measures eGrants Tips Serve

Serve DC Training 9/28/11

Theories of Change and Logic Models

Evidence

Performance Measurement 101

Reviewing Performance Measures

eGrants Tips


Learning Objectives:

Know the definition of “theory of change”

Understand relationship between theory of change and program design

Understand how logic models articulate a theory of change


A theory of change:

Looks at cause and effect relationships

Identifies specific interventions to achieve the desired result

Uses evidence to articulate assumptions


PROBLEM: The identified community need

INTERVENTION: The activities of members and community volunteers supported by AmeriCorps members

OUTCOME: The change that occurs because of the intervention

EVIDENCE: Why you believe a certain set of actions (the intervention) will lead to the intended outcome


If the intervention (X) is delivered, at a certain dosage, then the expected outcome (Y) will happen.

X → Y


I have strep throat (PROBLEM).

If I take antibiotics (INTERVENTION), then I will get better (OUTCOME).

Antibiotics → I get better.X → Y


If I take penicillin, I will get better.

If I take a different antibiotic, will I get

better?

Some interventions (antibiotics) work better

than others. Some don’t work at all.


How do I know which antibiotic is best? I look at the evidence. There is research

that shows which antibiotic is likely to get the best result.

I consider constraints that may preclude the ideal intervention. (Penicillin may be too expensive.)

If I can’t have the most promising intervention, I need to understand the tradeoffs.


Two types of evidence are required:

Data that documents the community need

Data that documents why you think your intervention (using AmeriCorps members and community volunteers) will achieve the intended outcome


Data that demonstrates that the proposed intervention is likely to solve the identified problem

For example: Evidence says that x hours of tutoring leads to academic outcomes…so the intervention features X hours of AmeriCorps members tutoring


The evidence basis for an intervention may

include:Past performance measurement dataResults from a program evaluationResearch studies that document the outcomes of similar programsEvaluations that document outcomes of similar programs


Preliminary → Moderate → Strong


Variance in executing the ideal program intervention

Little evidence to support your intervention


PROBLEM: Children at risk of failing third grade reading exam

INTERVENTION: Individualized tutoring on five “building block” literacy skills

OUTCOMES: Students master skills, pass state reading exam


What is your theory of change?


Logic models are a visual way of expressing the cause and effect reasoning behind a theory of change. They move from left to right in a linear fashion.

X → Y


ProblemProblem InterventionIntervention OutcomesOutcomes

I have strep I have strep throat.throat.

I take penicillin.I take penicillin. I get better.I get better.


PROBLEM INTERVENTION OUTCOME

Children at risk of failing third grade reading exam

Evidence: Statistics on the number of students at risk of failing in program’s service area; Research on why reading proficiency by 3rd grade is important.

Individualized tutoring on five “building block” literacy skills

Evidence: Research on how children learn to read supporting theory that mastering building block skills will lead to proficiency. Research on design, frequency, duration of tutoring sessions.

Constraints?

Students master five building block skills.

Students pass state reading exam.


Logic Model Practice



Review the two types of evidence that support a theory of change

Practice evaluating evidence


Data that documents the community need

Data that documents why the intervention is likely to lead to the outcome(s)


Relevant

Compelling

Up-to-date

Reliable Source

Evidence Continuum (Preliminary, Moderate, Strong)


Where do we find evidence to document the community need?


Past performance measurement data Results from impact evaluation of your

program Research studies that document the

outcomes of similar programs Evaluations that document outcomes of

similar programs


University or research organizations

Names of known professionals/thought leaders

Similar sounding programs/descriptions

Meta-articles that review multiple studies


Recommended design and dosage (frequency, intensity, duration)

Do we need to alter the design of our intervention?

Do we need to choose a new intervention?



Know the definition of performance measurement

Understand differences between performance measurement and evaluation

Know performance measurement requirements for AmeriCorps grants


Performance measurement is the process of regularly measuring the amount of work done by your program and the outcomes of this work on your program beneficiaries.


Performance MeasurementCaptures near term changes

EvaluationCaptures lasting changesAttempts to demonstrate cause and effect between intervention and outcome


Performance Measurement

Evaluation

Systematic data collection and information about:•What took place•What outputs were generated•What near term outcomes were generated


Performance Measurement

Evaluation

• Tracks outputs and outcomes on a regular, ongoing basis

• Does not show causality

• Seeks to show causality

• Longer term focus• Uses the most rigorous

methodology that is right for the program (often quasi-experimental design)


The most important difference: Evaluation seeks to “prove” the theory of change (X→Y). Performance measurement does not.


Performance measurement can show the outcome (change ) occurred but not causality (the change occurred because of the intervention)

Performance measurement does not seek to “prove” a theory of change but can provide evidence that informs your theory

Performance measurement data can inform evaluation efforts


Performance MeasurementIndividual benchmark assessments on Dynamic Indicators of Basic Early Literacy Skills (DIBELS) three times/yearState Reading Exam --Number of students who graduate from the Minnesota Reading Corps who pass state reading exam


EvaluationMatched sample research project in Minneapolis School District—Reading Corps pre-school participants scored significantly higher in phonemic awareness, alphabetic principle, and total literacy than children in matched comparison group entering kindergarten


If performance measurement doesn’t prove that my intervention worked, then why do it?


If the evidence for an intervention is strong, PM helps show the program is on track

If the evidence basis is weak or not well-defined, PM can provide evidence that a change occurred


Improve performance

Inform decision making

Demonstrate accountability (internally and

externally)

Justify continued funding

Enhance customer service

Improve quality of services

Set targets for future performance


Measuring prevention or long-term outcomes

Time

Cost

Establishing reasonable targets

Brief service interventions

Attributing impact to the intervention


Counts of the amount of service that members or volunteers have completed.

They do not provide information on benefits to or other changes in the lives of members and/or beneficiaries.


Number of students who complete participation in an AmeriCorps education program

Number of veterans engaged in service opportunities

Number of individuals receiving support, services, education and/or referrals to alleviate long-term hunger


Outcomes specify changes that have occurred in the lives of members and/or beneficiaries. They should be:RealisticMeasurable during grant periodRelevant to theory of change


Outcomes measure changes in:

AttitudeBehaviorCondition

Most programs should aim to measure a quantifiable change in behavior or condition.


Applicants are required to create at least one aligned performance measure to capture the output and outcome of their primary service activity.

Note: Applicants may create additional performance measures provided that they capture significant program outcomes.


An aligned performance measure has two components:OutputOutcome

Alignment refers to whether:The outcome is logical and reasonable given your intervention and output(s)The output and outcome measure the same beneficiary



Learn how CNCS assesses performance measures

Practice using assessment checklist


Applicants must describe the following theory of change elements:The problem(s) identified (Need)The actions that will be carried out by AmeriCorps members and community volunteers (Evidence-Based Intervention)The ways in which AmeriCorps members are particularly well-suited to deliver the solution (Value Added by AmeriCorps)The anticipated outcomes (Outcomes)


Measures align with the need, activities and outcomes (theory of change) described in the narrative

Outputs and outcomes are correctly aligned

Measures utilize rigorous methodologies to demonstrate significant outcomes


Choose an intervention that will lead to the specific desired outcomes.

Choose outcomes that can measure the intervention.

Example: Improving academic performance


Intervention: After school enrichment program

Outcome: Improved academic performance in reading


Intervention: Tutoring program focused on helping kindergarten students master the most critical emergent literacy skills



Intervention: Homework help program focusing on multiple subjects



Need a clear link between:

The intervention (design, frequency, duration)

The specific change (outcome) that is likely to occur as a result of the intervention


Intervention: AmeriCorps members lead classes to educate smokers about the health risks associated with smoking.

Outcomes: Individuals increase their knowledge of the health risks of smoking. Individuals stop smoking.

Alignment Issue: Simply telling people that smoking is bad for them may not help them to quit.


Intervention: Members provide financial literacy trainings to economically disadvantaged adults.

Outcome: Economically disadvantaged adults will open savings accounts after receiving financial literacy training.

Alignment Issue: If beneficiaries do not have enough money to meet their basic needs, a savings account may not be realistic.


National Measures must be aligned as directed in CNCS guidance

Aligned measure includes output and outcome for primary service activity

Outcomes likely to result from outputs

Outputs and outcomes measure the same population


Output H4: Clients participating in health education programs

Outcome: Community members will decrease costly emergency room visits


Output H5: Youth engaged in activities to reduce childhood obesity

Outcome: Children experiences at least an 8% increase in aerobic fitness


Output EN4: Acres of parks improved

Outcome: Acres of park certified as satisfactorily restored by land manager partners


Output EN4: Acres of parks improved

Outcome: Public parks will be cleaner as the result of removing 140,000 pounds of trash and debris


Data collection methods are rigorous

Outcomes capture a significant change. It is helpful to consider:So what?Is this change worth measuring?


Do outcomes capture the change you want to accomplish?

Will proposed methods/instruments capture these outcomes?

Are methods rigorous but realistic?

Is there a clear plan/timeline for developing instruments and collecting data?


Proposed methods are not realistic because:Too ambitiousCan’t get dataUnable to obtain a representative sample


A grantee plans to use a standardized pre/post test but has difficulty administering the test and aggregating the data within the grant period. Would like to measure improvement in grades instead.

A grantee is unable to create a sampling frame that defines the population from which they will sample.


Objective vs. Subjective

Not tested ahead of time

Don’t measure what they are supposed to measure (Validity)

Biased


What is a valid way to measure my driving proficiency?

Survey that asks how I feel about driving?

Survey that asks if I think I’m a good driver?

Written Test?

Driving Test?


A survey scale that only measures improvement

A survey that is only returned by individuals who feel strongly


The AmeriCorps State and National Performance Measurement Assessment Checklist:Alignment with theory of changeAlignment of outputs and outcomesQuality (Rigorous, Worth Measuring)


Practice using Performance Measurement Checklist


Learning Objectives

Learn tips for entering PMs in eGrants

Understand how the eGrants language is sometimes different from other CNCS language for performance measures


Strategy = Intervention

Result = Output, Intermediate Outcome, or End Outcome

Indicator = A description of the measurable change that will occur (Number of beneficiaries who...)

Target Statement = The indicator plus the expected number (100 beneficiaries will…)


Target – The number in the target statement (100)

Instrument – The specific tool that will be used to collect data (AIMSweb Letter Sounds and Letter Names pre/post test)

Data Collection Methodology – How data will be collected (survey, pre/post test, etc.)


The strategy (intervention) will be the same for all components of the measure (output, intermediate outcome, end outcome) because all of these should result from the same intervention


Within each output or outcome, the result statement, indicator, target statement and target number will seem repetitive:Result Statement: Students will demonstrate improved academic performanceIndicator: Number of students with improved academic performanceTarget Statement: 100 students will demonstrate improved academic performanceTarget: 100


The data collection methodology is how you will collect the data. For example, administering a standardized test is a method of collecting data.

The instrument is the actual tool that will be used. For example, AIMSweb Letter Sounds and Letter Names Pre/Post test is one standardized test that might be an acceptable instrument.


Resource Center

2012 AmeriCorps NOFO and Performance Measures Instructions

Performance Measurement by Harry Hatry

Stanford Social Innovation Review


Documents

Serve DC Training 9/28/11. Theories of Change and Logic Models Evidence Performance Measurement 101 Reviewing Performance Measures eGrants Tips Serve