IUHPE Global Programme: The Public Health Agency of Canada’s Effectiveness Project

Marcia Hills, RN, PhD

Simon Carroll (PhD Student)

Canadian Consortium for Health Promotion Research

Sylvie Desjardins, Director, Strategic Policy, PHAC.

Rio de Janeiro, Brazil May 12, 2005

Overview

1) Typical strategies for assessing effectiveness…systematic reviews and an alternative approach

2) Policy-maker’s need for evidence of effectiveness and frameworks/tools

3) The Public Health Agency of Canada’s Effectiveness Project

Why should health promoters care about ‘evidence’ and ‘effectiveness’

Increasing demand on policy/decision makers to provide transparent and explicit justification of decisions

Ignoring the resource allocation issue is neither possible nor desirable

Valuable lessons are discounted Valuable evidence is ignored What counts as evidence “Gambit of compliance” Leads to poor evaluations, poor data gathering, and a

waste of valuable resources (i.e. 10% of funds!)

Evaluation…

Programmatic Focuses on goals, objectives and outcomes of a

particular program Highly contextualized Evaluation – programs, people & practice Concerns practice with people in communities

– What is the relationship between the evaluator and those delivering the program?

– What should be accepted as evidence upon which to base practice?

Effectiveness…

Looks for an evidence across programs Driven by desire for evidence-based policy…

practice…decision-making Accountability for public funds Decides what constitutes ‘evidence’ and

influences how evaluations are conducted

Typical Strategies for Assessing Effectiveness…Systematic Reviews

Numerical Meta- analysis – Randomized Control Trials– Quantitative– Statistical notion of causality (probability)

Narrative Review – Multiple Methodologies (interpretative)– Qualitative– Ignores or rejects causal explanation

Different philosophies, assumptionsParadigm “wars”

Difficulties with both approaches…

Based on “best buy” Accumulation of evidence in similar

programs Beliefs about causality that are based on

statistics or reactions to it Either context–free or over

contextualized lessons

Realist synthesis approach (Pawson, 2000)

Based on philosophy of science that argues that it is possible to have a logic of comparison that is not statistical

Causal powers of an intervention (program) lie in its underlying mechanism/s - its basic theory about how program resources will affect participant’s action in certain contexts

NOT …does the program work ? …BUT what are the conditions (context) under which the resources the program offers have an impact on participant’s actions

Mechanisms, Contexts, Outcomes

Want to discover those contexts that have produced solid and successful outcomes FROM…those contexts that induce failure

All cases are important …successful policy depends as much on avoiding previous errors as by “imitating” successes

Causality…

The language of causality is crucial because it refers to the elements of a programme that make them work…or not work.

Improving the priority setting of funders and improving the quality of practice in health promotion demands attention to what causes positive and negative outcomes

We need to move beyond the either/or of relying on probabilistic reasoning or retreating into the defensive position of denying the capability of causal assertions

We need to conceptualize our phenomena into discernable structures, objects and relations, rather than a obsessive focus on producing data for statistical modeling.

Public Health Agency of Canada: Effectiveness Project

Goal..

To develop a framework for assessing the effectiveness of federally-funded community initiatives to improve health

Individual, Community& Systems

Change

Framework for Assessing the Effectiveness of Community Initiatives That Promote Health

Improvements inPopulation

Health Outcomes While Reducing

Health Inequity

Impact on Determinants of

Health

IndividualChange

CommunityChange

SystemsChange

CommunityOrganization& Action

Collaborative Planning

TransformationalChange

Social

Economic

Political

Environment

Gender

Cultural

Contexts

Component 1 Mechanisms: Collaborative Planning

1) Meaningful participation of all relevant stakeholders

2) Critical dialogue

3) Shared power

4) Project action planning and evaluation

Component 2: Community Organization and Action

1) Ongoing education and training opportunities

2) Evolving leadership

3) Sustained mobilization of resources

4) Critical reflection and systematic monitoring

Component 4: Transformational Change

1) Develop and attract champions

2) Generate publicity of project successes

3) Influence Public Policy and Decision-making Bodies

4) Work with relevant social movements and provincial and/or national advocacy groups

A Realist Synthesis Approach to Assessing Effectiveness…

Allows us to learn from past successes and failures

Focuses our attention on the issues so important to community practitioners

Uncovers what is often undervalued and discounted

Builds credibility for community initiatives by using a theory of causation based on a realist philosophy of science

A Snapshot of the NARO Effectiveness WebsiteURL <http://dynosite.romulin.com/web/uofvic>

Until one is committed, there is hesitancy, the chance to draw back, always ineffectiveness, concerning all acts of initiative (and creation). There is one elementary truth, the ignorance of which kills countless ideas and splendid plans: that the moment that one definitely commits oneself, then Providence moves too. All sorts of things occur in one’s favour that would never otherwise have occurred. A whole stream of events issues from the decision, raising in one’s favour all manner of unforeseen incidents and meetings and material assistance which no man could have dreamed would come his way. Whatever you can do or dream you can, begin it. Boldness has genius, power and magic in it. Begin it now!

Johann Wolfgang Von Goethe