Community Partnership Evaluation for Injury Prevention

Susan J. Snelling, PhD

Health Promotion Field Support Specialist, Evaluation

Instructions• Audio: You can listen and speak

through your computer or phone – this is an option for you to select in the “Audio Mode”

2

Instructions• Question: When it comes to

the Q & A session you can let me know you have a question by clicking the “Raise Your Hand” button

3

Instructions• Question: If you have a

question you would like to type, you can do so in the textbox provided

4

WEBINAR OUTLINE• Partnerships• What is a partnership?• Why do we work in partnerships? What are the challenges?• How do partnerships work and what makes a successful

partnership?

• Evaluating Community Partnerships• What is evaluation?• Options for evaluating partnerships: Strategies, evaluation

options, measures, methods, and tools

• Resources

5

WHAT IS A PARTNERSHIP?

6

Different Types of Inter-Organizational Work• Networking: The exchange of information for mutual benefit.

• Coordination: Exchanging information and altering activities for a common purpose. Match and coordinate needs and activities. Limit duplication of services.

• Cooperation: As above plus sharing resources.

• Collaboration: As above plus, collaboration includes enhancing the capacity of the other partners for mutual benefit and a common purpose. Longer-term and more deliberate efforts of organizations and groups to undertake new, joint activities. Building interdependent systems to address issues and opportunities.

Adapted from Labonte (2003) and Integrated Health Promotion (2003), Victoria, Australia.7

What is a Partnership?Community Partnership involves building collaborative relationships between different groups in the community to achieve a common goal of promoting health. Within the partnership, decision-making is shared across partners. The mutual action of the partnership has the potential to achieve more than any one partner working independently.

- (Sears 2009 for Heart Health Resource Centres)

8

WHY DO WE WORK IN PARTNERSHIPS IN INJURY PREVENTION?

9

Why do we Work in Partnerships?• Resource limitations; share workload• Reduce duplication• Increase influence and ‘reach’• Include a variety of perspectives• Learn from others• Develop shared messaging; reduce public confusion• Ontario Public Health Standards are built on a principle of

partnership and collaboration• Social change goes beyond public health: see OPHS logic

models!

10

WHAT CHALLENGES DO YOU EXPERIENCE IN INJURY PREVENTION PARTNERSHIPS?

11

What Challenges do we Experience in Partnerships?• Processes: How we work together• Partnership building and maintenance takes time• Difficulty coming to consensus may impede current and future efforts• Power imbalances; contribution inequities• Changes in membership

• Outcomes: What we achieve together• “Opportunity costs” of engaging in partnerships • Focus on information sharing or endless planning rather than achieving

results • Focus on long-term goals can lead to loss of focus and commitment in

the short term

Adapted from Partnerships: The Benefits. (2007) Institute of Public Health in Ireland.12

HOW DO PARTNERSHIPS WORK AND WHAT MAKES A SUCCESSFUL PARTNERSHIP?

13

Three Levels at Which Partnerships Operate• Level 1: Partnership infrastructure, function, and processes

• Level 2: Partnership programs and interventions

• Level 3: Health and systems change outcomes

14

Five Conditions of Collective Success• Common agenda

• Shared measurement systems

• Mutually reinforcing activities

• Continuous communications

• Backbone support organizations

- John Kania and Mark Kramer, “Collective Impact” (2011)

15

WHAT IS EVALUATION?

16

What is Evaluation?

The application of systematic methods to address questions about program operations and results.

- Wholey, 2010, Handbook of Practical Program Evaluation

17

10 Steps in Evaluation ProcessStep 1: Clarify your program Step 2: Engage stakeholdersStep 3: Assess resources for evaluationStep 4: Select the type of evaluationStep 5: Determine methods of measurement and proceduresStep 6: Develop workplan, budget and timelineStep 7: Collect dataStep 8: Process the data and analyze the resultsStep 9: Interpret and disseminate results Step 10: Take action

18

Process Evaluation• How do we do our work?

• Measures activity and implementation

• Sometimes known as tracking or monitoring

19

Outcome Evaluation• Are we achieving the objectives we set out?• Estimates success of program (or partnership)

at meeting goals and objectives• Measures effects of activities at producing

change• It can be difficult to measure long-term

outcomes• Ii can be difficult (or impossible) to isolate the

contribution of your work to a societal result

20

OPTIONS FOR EVALUATING PARTNERSHIPS

21

Strategies for Implementing Effective Evaluation in Partnerships

• Build in evaluation concepts from the start of the partnership• Establish principles: Here are a few suggestions:• No shame, no blame – do evaluation to learn and improve• Don’t evaluate something that can’t change• Commit to taking action based on the results• ‘We’re all in this together’ (the problems and the solutions are not

the leader’s responsibility)

• Establish clear evaluation questions• Have a plan for using the results of an evaluation

22

Options for Evaluating Partnerships• Start with Evaluability Assessment

see Butterfoss: Are You Ready to Evaluate Your Coalition?

http://coalitionswork.com/documents/are_you_ready_to_evaluate_your_coalition.pdf

• Since there are three levels at which partnerships operate, there are also three levels on which evaluation can focus:• Level 1: Partnership infrastructure, function, and processes• Level 2: Partnership programs and interventions • Level 3: Health and systems change outcomes

23

Measures for Evaluating Levels of PartnershipsLevel 1: Structures and Functions• Number and type of partners• Skills of partners• Diversity • Commitment, sense of ownership• Partnership processes such as

meetings, workload• Leadership style and

effectiveness• Member satisfaction• Perceived effectiveness among

members and community

Level 2: Program Implementation• Implementation of plan• Programs and services offered• Community actions taken• Reach; Media coverage• Knowledge and behaviour change• Improved or new policies or

practices (Level 2 or 3)Level 3: Outcomes and Impacts• Health status • Community changes• Community capacity• Partnership sustainability

24

Methods and Tools for Evaluating PartnershipsMethods• Active data collection: Surveys, interviews, focus groups, self-report• Unobtrusive data collection: Observation, monitoring, statistics• Descriptions of partnerships can be done through network mapping

Tools• Coalition Member Survey; Coalition Effectiveness Inventory (Butterfoss)

• Vic Health Partnerships Analysis Tool

• Network Health Scorecard

• Mental Health Mapping Tool

• Partnership Effectiveness Tool

25

RESOURCES

26

Health Promotion Capacity Building Services

• Free to those working on Ontario-focused projects.

• Scope varies, depending on need:• training sessions; • brief, one-time advice; • consultations;• review your work or product;• links to other sources of information and resources.

• Consultation request form

http://www.thcu.ca/consultation/request_form.cfm27

QUESTIONS?

28

Thanks to Ontario Injury Prevention Resource Centre for supporting us on this webinar!

29

Public Health Ontario wishes to acknowledge and thank THCU staff and many partners who contributed to an earlier version of this document. THCU (originally known as The Health Communication Unit, started in 1993 at the University of Toronto) moved to Public Health Ontario’s Health Promotion, Chronic Disease and Injury Prevention Department in 2011.