6/3/2014

1

Evaluating Progress and Outcomes Through a Community Lens

Shauna MacKinnon (PhD)Assistant ProfessorUrban and Inner City StudiesUniversity of WinnipegCo-Investigator, Manitoba Research Alliance©Shauna T. MacKinnon University of Winnipeg Personal collection 2014

Presentation for University of Manitoba Summer Institute in Program evaluation

June 4, 2014

The basics… Decisions about what to measure and the

methods to be used must be made in partnership with funding agencies, researcher-evaluator, community being evaluated and members of the community as identified by the community in which the inquiry takes place

This should apply to summative as well as formative evaluation

6/3/2014

2

Questions to consider How to evaluate government

programs/investments/activities through a community lens;

How government evaluation processes can be more responsive to community needs;

How intergovernmental alignment of evaluation processes would benefit the community

What we hear from community organizations about evaluation

Who decides what and how to measure? What about unexpected outcomes? Community priorities don’t always align with

government/funder priorities So what we measure and how it does not

always align well

Who is accountable to the community?

6/3/2014

3

What to do? Take a step back and decide

What should the purpose of evaluation be? What do we want to learn? What will we do with what we learn?

What Good Evaluation IS… Includes formative-process evaluation (not only

sumative-outcome evaluation) Designed and implemented early in the process

part of the process of program implementation Centred within cultural norms

Responsive to the history, needs, and dreams of the people participating in and being affected by the program being evaluated (American Indian Higher Education Consortium 2009)

Engages ‘community’ in the design Focuses on strengths Aligns with the goals and objectives of the program

This means the goals and objectives identified by the community based organizations in collaboration with other stakeholders – not just the goals of the funding agency

6/3/2014

4

Flexible Includes both ‘countable’ and

‘uncountable’ things Outcome evaluation tends to focus on

‘countables’ Comprehensive evaluation is more than testing

whether our assumptions about an intervention were correct Did doing ‘A’ result in ‘B’? Focusing on ‘outcomes’ alone without an

indepth analysis of ‘how’ the outcome occurred can lead to many problems, including unsuccessful replication

Example – replication of Pathways to Education model

Properly resourced and not rushed Evaluation that is useful will be implemented

early so that it can inform further program development/refinement

Evaluation should be used to learn how to improve a program that was developed from basic assumptions

Evaluation can be empowering to program participants if they are included in the designand implementation

6/3/2014

5

Valuing unintended outcomes

“I always thought that I was stupid, but now I know that I am smart”

What good evaluation isn’t… An add on and‘backward’ focused

but RATHER in tandem with program design and implementation and ‘forward’ thinking.

Weakness focused Developed in a vacuum

(narrowly defined pre-determined objectives)

Focused only on ‘countables’

6/3/2014

6

How governments can better respond

Work in collaboration with communities to come to an agreement on: What we are evaluating Why we are evaluating How we are evaluating Who will evaluate How we intend to use the evaluation

Holistic Programs call for Holistic evaluation

We tend to want to evaluate ‘one thing’ or ‘one program’ but is this effective?

It is difficult to resist looking for specific information that aligns with our priorities but what do we miss when we do this? And what about the priorities of program participants?

What do they want?

6/3/2014

7

Examples: Assessing ‘school readiness’ may be useful,

but how will this resolve the complex issues that face many children after they begin school? (eg. Poverty, racism)

Funding of one program in an organization with multiple programs meeting multiple/interconnected community needs

CEDA Pathways to Education Pathways Canada focus is on evaluation ‘countables’ credit accumulation graduate rates attendance

We also tend to look at the countables through a single lens, comparing to ‘the norm’ Students should be attending full day

everyday Students should graduate within four years

6/3/2014

8

Answering the Questions How to evaluate government

programs/investments/activities through a community lens;

How government evaluation processes can be more responsive to community needs;

How intergovernmental alignment of evaluation processes would benefit the community?

Government evaluation processes can be more responsive to community needs if they are designed in collaboration (honest/real) with community All ‘parties’ have to be able to be ‘upfront’

about their goals, objectives, needs and interests so that evaluation can be effectively designed and implemented

6/3/2014

9

Intergovernmental alignment of evaluation processes would benefit the community – as well as governments It would ensure that we are gathering all of

the information we need to develop holistic programs that meet multiple goals

If we want to “evaluate government programs/investments/activities through a community lens” then we need to engage the community in determining whether the program/investment/activities are guided by the priorities that they see in the community

6/3/2014

10

Acknowledging the power imbalance We like to talk about ‘partnerships’ but

community organizations are well aware that these are not ‘equal’ partnerships

The agency funding the evaluation will have influence on the direction evaluation takes

Funding agencies need to be open to ‘redefining’ evaluation questions and approaches to reflect the interests of community organizations

Reassessing where evaluation will lead us Part of evaluation through a community lens is

looking at our goals, objectives, expectations differently. Should we be changing our expectations to better

align with realities rather than trying to force a different reality? (High School graduation)

Evaluation may show intervention has improved ‘school readiness’, but if parents remain poor, disengaged, disempowered, what will be the long term impact?

6/3/2014

11

Open to other priorities We become uncomfortable when it

comes to evaluating other priorities and/or recognizing the importance of other factors-other goals

For example – Indigenous goals and objectives

The Four QuestionsThe Indigenous community has established that “success” (however defined) requires individuals to have a strong sense of:Who am I?Where did I come from?What is my purpose?Where am I going?

If the Indigenous community has identified this as critical to all that follows, should we not be trying to evaluate progress here? And if so, how?

6/3/2014

12

If we agree that we want transformative change, then we need to measure progressive in a way that is transformative Forward looking Strength based Community driven Empowering