Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
Champions for Change Workshop
EVALUATING COLLECTIVE IMPACT | FEBRUARY 2014
2
© 2014 FSG
“My question is: Are we making an impact?”
Source: The New Yorker
3
© 2014 FSG
EVALUATING COLLECTIVE IMPACT VS. SHARED MEASUREMENT
Evaluation
Shared measurement systems (SMS)
- a common set of indicators to
determine if, where, and for whom the
initiative is making progress
SMS can be both an input to evaluation (by providing data and/or
shaping evaluation questions) and an object of evaluation
Shared
Measurement
Systems
(SMS)
Evaluation - a range of activities that
involve the planned, purposeful, and
systematic collection of information
about the activities, characteristics, and
outcomes of a CI initiative
Collective Impact Evaluation Framework
Context
Long-term social or
environmental
impact goals
Impact Changes in Systems
and Behavior
Intermediate outcomes,
related to changes in the
way people, organizations,
institutions, and systems
behave
Time and Impact
CI Design & Implementation
Five key elements; initiative’s capacity
and learning culture; implementation
of programs, activities, and
campaigns, as part of the initiative’s
strategy
Capacity
CI learning culture
Changes in individual
behavior and professional
practice
Changes in systems
(funding, norms, policy)
Changes in lives
Developmental Formative evaluation Summative
Planning for Effective CI Evaluation Begins With Choosing Specific Areas of Focus
Context 1 Key
Potential evaluation
focus Continuous learning
Elements of context
CI Process
Outcomes & Indicators
CI Design & Implementation
CI /Partner Activities
Outcomes & Indicators
2 Intermediate
Outcomes
Behavior Change
Outcomes & Indicators
Systems Change
Outcomes & Indicators
Impact
CI Impact Goals
Outcomes & Indicators
4
3
Time and Impact
SMS (in operation) Early performance indicators
6
© 2014 FSG
Within Each Selected Focus Area, CI Partners Should Identify the Key Learning Questions they Seek to Answer
Intermediate Outcomes Impact
• To what extent has the CI initiative
achieved its ultimate outcomes?
• What has contributed to or hindered the
achievement of the CI initiative’s goals?
Context
• What are the cultural, socioeconomic,
and political factors that are influencing
the design and implementation of the CI
initiative?
1
3 4
CI Design & Implementation 2
• Backbone Infrastructure: To what
extent and in what ways is the backbone
infrastructure providing the leadership,
support, and guidance partners need to
do their work as planned?
• To what extent / in what ways are the
flows of philanthropic and public funding
shifting to support the goals of the CI
initiative?
• To what extent / in what ways are social
and cultural norms evolving in ways that
support the goals of the CI initiative?
Changes in Systems
Sample Learning Questions
Developmental Formative evaluation Summative
Example 1: Outcomes and Indicators for CI Process, Backbone Infrastructure
Context 1 Key
Potential evaluation
focus Continuous learning
Elements of context
CI Process
Outcomes & Indicators
CI Design & Implementation
CI /Partner Activities
Outcomes & Indicators
2 Intermediate
Outcomes
Behavior Change
Outcomes & Indicators
Systems Change
Outcomes & Indicators
Impact
CI Impact Goals
Outcomes & Indicators
4
3
Time and Impact
SMS (in operation) Early performance indicators
8
© 2014 FSG
Example Outcomes and Indicators – Backbone Infrastructure
Sample Outcomes Sample Indicators
The backbone infrastructure
(BBI) effectively guides the CI
initiative’s vision and strategy
BBI builds and maintains hope and motivation to
achieve the initiative’s goals
BBI celebrates and disseminates achievements of CI
partners internally and externally
Partners look to the BBI and SC for initiative support,
strategic guidance and leadership
The backbone infrastructure
ensures alignment of existing
activities, and pursuit of new
opportunities, toward the
initiative’s goal
BBI provides project management support, including
monitoring progress toward goals and connecting
partners to discuss opportunities, challenges, gaps,
and overlaps.
BBI convenes partners and key external stakeholders
to ensure alignment of activities and pursue new
opportunities
Learning Question: To what extent / in what ways is the backbone providing the
leadership, support, and guidance partners need to do their work as planned?
Developmental Formative evaluation Summative
Example 2: Outcomes and Indicators for Systems Change, Funding Flows
Context 1 Key
Potential evaluation
focus Continuous learning
Elements of context
CI Process
Outcomes & Indicators
CI Design & Implementation
CI /Partner Activities
Outcomes & Indicators
2 Intermediate
Outcomes
Behavior Change
Outcomes & Indicators
Systems Change
Outcomes & Indicators
Impact
CI Impact Goals
Outcomes & Indicators
4
3
Time and Impact
SMS (in operation) Early performance indicators
10
© 2014 FSG
Example Outcomes and Indicators – Funding Flows
Sample Outcomes Sample Indicators
Philanthropic funding in the
targeted issue area/system is
increasingly aligned with the
goals of the CI initiative
Overall philanthropic funding for the targeted issue
area or system has increased
New / Existing philanthropic resources are aligned
towards evidence-based strategies in the targeted
issue area/system
Philanthropic funding is increasingly designed to
allow for program innovation and experimentation in
the targeted issue area/system
Philanthropic and public funders
leverage funding for the targeted
issue area/system through
partnerships and collaborative
funding efforts
Grantmakers participate in funding collaboratives to
leverage resources toward the targeted issue
area/system (as relevant)
Philanthropic and public funders engage in public-
private partnerships to leverage resources toward the
targeted issue area/system
Learning Question: To what extent and in what ways are the flows of philanthropic and
public funding shifting to support the goals of the CI initiative?
Shared Measurement
12
© 2014 FSG
Shared Measurement Is a Critical Piece of Pursuing a Collective Impact Approach
Identifying common metrics for tracking progress toward a common
agenda across organizations, and providing scalable platforms to share data, discuss learnings, and improve strategy and action
Definition
Benefits of Using Shared Measurement
Source: Breakthroughs in Shared Measurement and Social Impact, FSG, 2009
Learning and Course Correction
Catalyzing Action
Improved Data Quality
Tracking Progress Toward a
Shared Goal
Enabling Coordination and
Collaboration
13
© 2014 FSG
There Are Three Phases to Developing a Shared Measurement System
Design Develop Deploy
1 2 3
• Shared vision for the
system and its relation to
broader goals, theory of
change or roadmap
• View of current state of
knowledge and data
• Governance and
organization for
structured participation
• Identification of metrics,
data collection approach
(incl. confidentiality/
transparency
• Development of web-
based platform and
data collection tools
• Refinement and
testing of platform
and tools
• Staffing for data
management and
synthesis
• Learning forums and
continuous
improvement
• Ongoing infrastructure
support
• Improve system based
on ongoing evaluation
of usability and impact
Source: FSG Analysis
14
© 2014 FSG
Several Challenges Can Occur When Developing and Implementing Shared Measurement Systems
Coming to agreement on common outcomes and indicators
Concerns about relative performance / comparative measurement across
providers
Limited capacity (time and skill) for measurement and data analysis within
organizations
Alignment among funders to ask for the common measures as part of their
reporting requirements
Time and cost of developing and maintaining a system (human capital and
technology)
15
© 2014 FSG
There Are a Number of “Tips and Tricks” to Bear in Mind When Developing Shared Measures
Collecting and
Presenting Data
• Specific, time-bound goals; report progress relative to targets
• Data on whole populations, where possible
• Numbers as well as percentages make goals more tangible
Identifying
Indicators
• “Top-level” indicators limited to a manageable number (~15),
with additional contributing indicators if needed
• A set of criteria to guide the identification and prioritization of
potential indicators
Leveraging
Existing Efforts
and Expertise
• Voluntary team of data experts to advise on the design,
development, and deployment of a SMS
• Crosswalk of what partners are already measuring
• Leverage existing indicators adopted by relevant efforts at the
local, provincial, or federal level
16
© 2014 FSG
The Road Map Project Uses Indicators to Guide its Progress toward Doubling the Number of College Graduates in the Seattle Area
“The Road Map Project’s goal is to double the number of
students in South King County and South Seattle who are
on track to graduate from college or earn a career credential
by 2020. We are committed to nothing less than closing the
unacceptable achievement gaps for low income students
and children of color, and increasing achievement for all
students from cradle to college and career.”
17
© 2014 FSG
Work Groups Used a Multi-Step Process to Identify, Vet and Refine the Road Map Indicators
• Used criteria to identify and prioritize indicators
• Work group chairs calibrated indicators across continuum
• Short-list of top-level indicators for time-bound targets; also
supporting indicators
• Four work groups
• Identified indicators of student success for their part of the
continuum
• Baseline and annual reports to track progress on all
indicators
• Data experts advise on indicators over time so the list can
evolve
Source: FSG Analysis
Identifying
Indicators
Tracking
Progress and
Refining over
Time
Developing
Work Groups
18
© 2014 FSG
• Indicator must be valid measure linked to ultimate goal of postsecondary attainment and/or four sub-
goals
• Indicator must be easily understandable to local stakeholders
• Data must be produced by trusted source
• Priority given to including indicators comparable across school districts (or neighborhoods), and
having ability to be compared
• All or most indicators must be affordable to gather and report
• Data should be available consistently over time (preferably on annual basis or more frequently),
should be recent (2008 or later), and easily disaggregated by county, City / S. King County
community, and school (or neighborhood)
• Priority given to data that can be disaggregated by ethnicity, socio-economic status, ELL, and gender
• Trend data should be provided over at least 3-year period (beginning with or including 2007)
• Each indicator should be able to be influenced to significant degree by local action, and be useful in
daily work of this Working Group, other educators and organizations that are working to improve
education
Source: Based on criteria identified by Cincinnati’s Strive Partnership
Work Groups Were Given a Set of Criteria to Guide
Indicator Selection
19
© 2014 FSG
From this Process, Top Level Indicators Were Identified, Targets Were Set, and Progress Is Tracked Over Time
Healthy and
ready for
Kindergarten
Supported and
successful in
school
Graduate from
high school
college and
career-ready
Earn a college
degree or career
credential
• % of students proficient in:
- 3rd grade reading
- 4th grade math
- 5th grade science
- 6th grade reading
- 7th grade math
- 8th grade science
• % students triggering Early
Warning indicators
• % of students who
graduate high school
on-time
• % of graduating high
school students meeting
minimum requirements
to apply to a
Washington State 4-
year college
• % of students at
community and technical
colleges enrolling in pre-
college coursework
• % of students who
enroll in
postsecondary
education by age 24
• % of students
continuing past the
first year of
postsecondary
• % students who earn a
post-secondary
credential by age 24
• % of children ready to
succeed in school by
kindergarten
Readiness Attainment Achievement
Source: Road Map Project, http://www.roadmapproject.org/
20
© 2014 FSG
The Road Map Project Baseline Report Includes Data to Measure Progress over Time on Key Indicators
21
© 2014 FSG
Annual Results Reports Detail How the Project Is Progressing Relative to Targets for Key Indicators
22
© 2014 FSG
THANK YOU!
• Thank you for being part of the conversation today
• For additional guidance on this topic, see resources on the Collective
Impact Forum website, and look for FSG’s Guide to Evaluating
Collective Impact, Coming in Spring 2014
Goals of the Guide
Illustrate the general process by which CI initiatives address complex
problems 1
Explore the ways in which evaluation and learning support CI success 2
Answer common questions about planning for and implementing
evaluation activities 3
collectiveimpactforum.org