8/4/2019 Mass Engaging Report
1/119
3masslbp.com
Engaging with impact:Targets and indicatorsfor successfulcommunityengagement by
Ontarios Local HealthIntegration Networks
A citizens report
from Kingston,Richmond Hilland Thunder Bay
r.1
8/4/2019 Mass Engaging Report
2/119
Engaging with impact:Targets and indicatorsfor successful
communityengagement byOntarios Local HealthIntegration Networks
A citizens reportfrom Kingston,
Richmond Hilland Thunder Bay
We know that community engagement
matters especially to our public
health system. As Ontarios Local Health
Integration Networks strengthen their
focus on community engagement,
what are the common benchmarks
and commitments that citizens thinkmatter most?
Engaging with Impact addresses the
challenge of evaluating engagement
and proposes a series of indicators that
can be used to assess performance and
develop a culture of engagement across
Ontarios public health system.
This report features:
A special essay on the challenge of
evaluating deliberative engagement by
Professor John Gastil;
Two essays comparing the commitment
of Canadian and UK health systems to
greater community engagement by the
Wellesley Institute and the British think
tank, Involve;
Interviews with the directors responsible
for community engagement in the North
West, Central and South East LHINs;
An account of three Citizens Workshops
that provide the basis for the
recommendations in this report;
An engagement scorecard for OntariosLHINs which proposes principles,
recommendations and indicators.
MASS LBP is reinventing
public consultation
masslbp.com
This project was commissioned by the Ontario Ministry of
Health and Long Term Care, Health System Strategy Division
and the Central, North West and South East LHINs.
r.1 Local Health IntegrationNetwork
8/4/2019 Mass Engaging Report
3/119
8/4/2019 Mass Engaging Report
4/119
8/4/2019 Mass Engaging Report
5/119
MASS LBP is a new kind o company
that works with visionary governments
and corporations to deepen and improve
public consultation and engagement.
We design impartial and ully transpar-ent public learning processes that build
awareness, consensus and insight.
MASS LBP provides an unparalleled
range o consultation and engagement
services or government, corporate and
not-or-prot clients. From conception
to execution to evaluation, MASS LBP
delivers highly innovative engagement
strategies that increase public under-standing, legitimacy and support.
Our services include:
comprehensive process design and
delivery rom 20 to 200+ participants
strategic advice, analysis and recom-
mendations concerning eective
public engagement and stakeholder
consultationcorporate and public needs
assessment
program evaluation and analysis
custom research and dissemination
acilitation and learning
public communications and curriculum
development
event coordination and logistics
We regularly make presentations to
audiences about our work concerning
the uture o responsible government,
public systems design and civic engage-
ment. We also oer seminars and relatedprogramming to clients on many o these
themes.
Inspired by Canadas rst Citizens
Assemblies, MASS LBP was ounded
in 2007 by Peter MacLeod and George
Gosbee to extend this model and
reinvent public consultation.
MASS LBP is based in Toronto, withassociates in Vancouver, Ottawa and
London, U.K.
8/4/2019 Mass Engaging Report
6/119
First published in 2009
MASS LBP.
Some rights reserved.
392A King Street East
Toronto, OntarioCanada M5A 1K9
1 800 369 7136
masslbp.com
ISBN 978-0-9811005-0-1
Series design by Concrete Design
Set in Helvetica Neue
and Berthold Baskerville Book
Cover paper: Cascades Cover 80
Text paper: Cascades Natural 60(FSC Mixed Sources)
Open access. Some rights reserved.
As the publisher o this work, MASS LBP
wants to encourage the circulation o our
work as widely as possible while retain-ing the copyright. We thereore have an
open access policy, which enables any-
one to access our content online without
charge. Anyone can download, save,
perorm or distribute this work in any or-
mat, including translation, without written
permission provided that the ollowing
conditions are met:
MASS LBP and the author(s) arecredited.
This summary, the MASS LBP com-
pany prole and the address
www.masslbp.com are displayed.
The text is not altered and is used
in ull.
The work is not resold.
A copy o the work or link to its use
online is sent to MASS LBP.
MASS LBP grateully acknowledges the
work o Creative Commons in inspiring
our approach to copyright. To nd out
more, go to:
www.creativecommons.org
8/4/2019 Mass Engaging Report
7/119
4Chapter Title
8/4/2019 Mass Engaging Report
8/119
5masslbp.com 5masslbp.com
Contents
Section 1: Evaluating Engagement
Introduction
Section 2: Understanding Engagement
A Comprehensive Approach to Evaluating Deliberative Public Engagement
The Canadian Experience: Observations and Lessons from the Canadian
Health Sector
The English Experience: Evaluating Patient and Public Engagement in Health
Learning from the LHINs
Working with the Citizens Workshops on Health and Engagement
Section 3: A Scorecard or Evaluating Engagement
A Scorecard for Evaluating Engagement
Appendices
Denitions of Community Engagement
Members of the Citizens Workshops on Engagement and Health
Indicators and Ideas Exercise
Endnotes
7
15
28
43
59
71
93
100
102
103
110
8/4/2019 Mass Engaging Report
9/119
Section 1:EvaluatingEngagement
8/4/2019 Mass Engaging Report
10/119
7masslbp.com
Engaging with impact: Targets and indicators or successul community
engagement by Ontarios LHINs ocuses on the value o community
engagement. Specically, it deals with the challenge o evaluating
engagement and proposes a series o recommendations and indica-
tors that can be used to assess perormance and develop a culture o
engagement that will help to rewrite the relationship between health
administrators and their public.
Local Health Integration Networks were created in 2006 with
an explicit mandate to engage stakeholders and their communities.
More than this, the idea o engagement was central to their rationale.
Proponents o the LHIN system argued that regional planning authori-ties would be better positioned than ministry ocials to assess and
interpret local needs. LHINs could do this because they would be in
closer contact with the communities they served and because o the
strength and number o local relationships they could orge and sus-
tain.
Many o Ontarios LHINs have spent their rst three years dem-
onstrating the easibility and merit o this rationale. Using their ownexpertise and intuition and sometimes relying on simple trial and error,
they are working to better engage stakeholders and members o the
public and to connect their eorts to other planning and integration
processes.
For these organizations, the debate concerning the value o
engagement has largely been settled. In its place is the growing
recognition that a commitment to integrating engagement into the
abric o their organization requires upending many o the traditionalassumptions that have dened health systems planning and public
administration.
The capacity to engage with Ontario communities is one o the
Introduction
8/4/2019 Mass Engaging Report
11/119
8Evaluating Engagement
LHINs dening and most distinctive competencies. As this com-
petency evolves, it promises to change how LHINs respond to the
interests and needs o their communities and to gradually transorm
how health systems planning is perormed. In this way, Engaging with
Impact is addressed to those who believe that community engagement
can be a major driver o health systems reorm.
Despite its length, this report is not exhaustive. Instead, we hope
it is stimulating and useul a rst installment amidst a broad range o
research and initiatives that the Ministry o Health and Long-Term Care
and the LHINs have begun to seed. Our purpose is to oer recommen-
dations and to be ameliorative rather than denitive a purpose that
is consistent with the sentiments o our citizen-participants, who in the
course o their work clearly understood that something as complexand amorphous as creating better systems o engagement rarely sub-
mits to single measures or immediate solutions.
Engaging with Impact begins with an essay by Proessor John
Gastil rom the University o Washington, one o North Americas
leading theorists concerned with the value o community engage-
ment. Gastil tackles the particular challenge o evaluating deliberative
engagement, where citizens and experts work together to examine and
solve problems. The essay oers ruitul reading or anyone wrestling
with the heightened challenges associated with designing, managing
and evaluating intensive engagement processes.
Subsequent papers rom the British think-tank Involve and
Canadas Wellesley Institute provide a comparative perspective on
evaluating engagement within centralized and decentralized
health systems.
Involve looks at the eorts being made by the English NationalHealth Service (NHS) to promote and evaluate engagement. In
England, a system-wide standard requires health service providers to
engage with patients and the public at large. Their paper examines the
ecacy o this standard and describes three initiatives that exempliy
the NHSs attempts to measure the outcomes o their investments in
public engagement.
The Wellesley Institute surveys a range o health agencies in
Canada and provides our examples o localized innovation. Without
a national standard or champion, interest in incorporating community
engagement has only recently begun to mature in Canada. Their paper
describes the challenges that need to be overcome or community
8/4/2019 Mass Engaging Report
12/119
9masslbp.com
engagement to become more deeply embedded within Canadian
health agencies.
A ourth paper looks at the experiences o our three sponsoring
LHINs, each located in a dierent part o the province and conronting
very dierent demographic pressures. Interviews with the directors o
planning, integration and community engagement at the North West,
Central and South East LHINs describe their on-the-ground eorts to
build a local practice o engagement that inorms the operations o
their organizations and the decisions o their boards.
The second section o this report describes three Citizens
Workshops that were hosted in Kingston, Richmond Hill and Thunder
Bay between November 27 and December 6, 2008. These workshops
involved representatives rom each o the sponsoring LHINs, as wellas independent experts who oered their insight on the strengths and
weaknesses o the health system. Billed as an opportunity to learn and
contribute ideas or improving engagement, more than 3,000 invita-
tions were mailed to randomly selected households in each region.
Ultimately, 80 citizens came orward to participate during the daylong
events. Their work culminated in a series o presentations that are the
basis or the principles that underlay the evaluation scorecard ound at
the end o this report.
Creating a culture o engagement
The overarching theme o this report is how to create a culture o
engagement. It is a culture that LHINs, unique among the wider health
sector, are singularly able to develop and it is a culture that citizens
want and increasingly expect.
During the workshops, the inseparable nature o engagement and
integration also became clear. At its core, integration requires a will-
ingness to try new things, in new combinations. While it is easy to get
lost in the technical minutiae o integration agreements, it is harder to
remember that integration is rst and oremost an act o imagination
and guiding that imagination should be a common sense o purpose
a desire to improve the quality and eciency o health services
available to Ontarians. LHINs need to engage the imagination o the
public and their health service providers i they are to achieve theirobjectives or health reorm.
In order to create a culture o engagement that helps the LHINs
move towards these goals, we urge the ollowing:
8/4/2019 Mass Engaging Report
13/119
10Evaluating Engagement
1. LHINs as health service providers: Community Engagement generates
real health outcomes.
Community engagement is not peripheral but central to the work o
Ontarios LHINs. It is the service they provide to the health system ingeneral and to citizens in particular. LHINs must continue to invest
in their ability to provide and enhance this service. LHINs should be
at the centre o an ongoing and lively conversation about the values,
views and priorities o their stakeholders and the public at large.
Building such relationships will help to rebuild citizens trust in the
health system, nd new opportunities or integration and increase the
sense o shared ownership and responsibility or the perormance o
the health system as well as or the populations general health and
well-being. In this sense, the LHINs are health service providers, and
the service they provide is community engagement.
2. Citizens are ready and waiting.
Citizens are willing, capable and ready to make important contributions
to the work o Ontarios LHINs. However, the opportunities or citizens
to make a contribution either directly or indirectly remains limited and
episodic. The proessional expertise o health service providers and
the input and interest o the public are integral assets that cannotaord to be let on the table. LHINs need to work to make engagement
a routine and more visible part o their repertoire. Moreover, because
good communication is a precursor to eective engagement, LHINs
should work to align their communications and engagement strategies.
3. To harness public input, emphasize learning.
Most citizens are unamiliar with the inner workings o the health care
system and, consequently, with the work o Ontarios LHINs. But citi-zens are not only willing and ready to make a contribution, they are
also eager to learn. They want to become better inormed and they
want to better understand a system they rely on and value. LHINs can
add value to public input by creating opportunities or the public to
become better inormed. With this in mind, LHINs need to ask or more
than public opinion they must help citizens understand the nature
and constraints o their health care system or any other issue they are
being asked to address.
4. Make it real.
Facing many competing pressures and demands, citizens have a
good sense or the value o their time. They will engage most deeply
8/4/2019 Mass Engaging Report
14/119
11masslbp.com
and meaningully when something is real and at stake. Their commit-
ment will always be proportionate to their sense o infuence and the
likely impact o their contribution. In this way, LHINs will only get out
o their eorts at engagement what they are prepared to put in. As
our scorecard explains, this means clearly dening the purpose o an
engagement and the role the public is expected to play. It means being
accountable and responsive to the publics contributions.
5. Focus on creating t-to-purpose engagement.
LHINs need to expand their repertoire and work to create a better t
between the processes they use to engage the public and the out-
comes they expect. In this report, we propose three classications
that describe the characteristics and the objectives o a wide range o
engagement processes.
6. Community engagement is mission critical.
Successul engagement is a key to meeting the LHINs objectives or
health systems reorm and unlocking the trust, imagination and com-
mitment o health service providers and the public. Poorly designed,
incomplete or insincere eorts to engage will only uel cynicism and
estrangement. Learning how to engage with impact is essential or
system-wide transormation.
Conclusions
The act that the LHINs have a clear mandate to invest in community
engagement demonstrates that the health system is eager to respond
to the concerns, needs and desires o citizens. Translating this man-
date into an eective culture o engagement should be a major ocus
o the LHINs over the next three years.
To help achieve this culture o engagement, the ministry should:
requirededicatedprogrambudgetsforthepurposeofengagingcommunities on substantive and ongoing issues
recognizeandrewardinnovationinengagement evaluatetheprogressofeachLHINtoimproveitseffortstowards
this goal
The LHINs should:
createengagementplansthatsupportandarecongruentwiththeir strategic objectives
diversifyanddeepentheirrangeofengagementofferings aligntheircommunicationsandengagementstrategies
8/4/2019 Mass Engaging Report
15/119
12Evaluating Engagement
encourageHealthServiceProviders(HSP)todeveloptheirownengagement plans and integrate these plans with their core oper-ations
Just as preventative health is about ordinary citizens taking control o
their lives, engagement is about the capacity o citizens to contribute
to the systems that serve them. I there has been a change o philoso-
phy rom reactionary to proactive health care provision, an analogous
philosophical shit is required to revolutionize the way health care
systems work and respond to the needs o citizens. In this light, com-
munity engagement is not just a task to be completed. It is an ongoing
process through which health outcomes are improved, trust is built,
public legitimacy is enhanced and systems transormation can
be pursued.
8/4/2019 Mass Engaging Report
16/119
13masslbp.com
8/4/2019 Mass Engaging Report
17/119
Section 2:UnderstandingEngagement
8/4/2019 Mass Engaging Report
18/119
15masslbp.com
I you turned back the clock just 20 years, it would be dicult tond a person in public oce, academia or civil society talking about
the virtues o citizen deliberation. At that time, a ew innovativepublic deliberation programs, such as the Citizens Jury in the UnitedStates and the Planning Cell in Germany, existed, but they did soin an unortunate kind o isolation, sometimes overlooked even bythose who would develop deliberative programs o their own in thecoming years.1
Today, the landscape could not be more dierent. Growinginterest in citizen engagement has spurred a prolieration o new,
more sophisticated deliberative practices designed to ellicit substan-tive public involvement in policy-making and public aairs. Now theissue or planners and administrators isnt scarcity but choice.
With dierent agencies and organizations deploying diverseapproaches to deliberative citizen engagement, it has become moreimportant than ever to take seriously the evaluation o these var-ied processes. It is not pessimistic to say that we currently have nosystematic comparisons o alternative deliberative methods, thoughmany civic reormers, researchers and agency ocials have ideasabout when to use one process instead o another. To improveour knowledge o deliberation and upgrade the practice o citizeninvolvement, we must begin to evaluate the design, process and out-comes o our civic engagement activities.2
In this chapter, I aim to provide the tools necessary or doing so.I begin by clariying the meaning o deliberative public engagement
and discussing broad evaluation categories. I review each evaluationcriterion and suggest measurement tools and then conclude with asummary recommendation or conducting evaluations.
Proessor John Gastil
Department o Communication, University o Washington
A Comprehensive Approach to EvaluatingDeliberative Public Engagement
8/4/2019 Mass Engaging Report
19/119
16Evaluating Deliberative Public Engagement
Dening Deliberative Public Engagement
It is imperative that reerences to deliberative public engagementconvey a suciently specic meaning so we can distinguish it romgeneric public involvement processes, such as ormal hearings orinormal consultations. For the purpose o this chapter, I dene this
term as an ocial or quasi-ocial process whereby policy-makers,policy/scientic experts and lay citizens work together on a publicproblem or concern, with the citizens careully examining a problemand seeking a well-reasoned solution through a period o inormed,inclusive and respectul consideration o diverse points o view.3
Breaking this down, the players in a deliberative public engage-ment need to include (1) appointed or elected ocials with some
degree o authority, (2) persons with content-relevant expertise and(3) lay citizens, whether randomly selected or otherwise recruited ina ashion that seeks diverse members o the general public. The citi-zens are at the heart o the process, but public ocials typically serveas the catalyst or initiating the deliberation and acilitating the imple-mentation o its ndings. The experts play a role behind the scenes(e.g., preparing brieng materials) or as personal resources thatcitizens can call on in the course o their deliberations (e.g., as key
witnesses). Together, the interplay o these participants constitutes apublic engagement process.
For such a process to be deliberative, it must meet a higher stan-dard or the quality o the dialogue, debate, discussion and other talkin which citizens participate. Table 1 shows a denition o a delibera-tive public meeting that I have ound helpul. First, a deliberativemeeting involves a rigorous analytic process, with a solid inorma-
tion base, explicit prioritization o key values, an identication oalternative solutions (sometimes pre-congured but oten still subjectto amendment) and careul weighing o the pros and cons. (Researchon group decision-making has ound that o these analytic elements,careul consideration o cons is oten the key to a high-quality pro-cess, and the emphasis on hard choices and trade-os in manydeliberation processes refects this.)4
Exclusive ocus on problem-solution analysis, per se, would
make our conception o deliberation overly rationalistic and over-look the social aspect o deliberation. One might say that the socialcomponent o deliberation is what makes it democratic deliberation,by requiring equal opportunity, mutual comprehension and consid-
8/4/2019 Mass Engaging Report
20/119
17masslbp.com
eration, and respect. The social requirements also make clear theimplicit emphasis on inclusion and diversity in deliberation.5
Analytic Process
Social Process
Combine expertise and professional research with
personal experiences to better understand the
problems nature and its impact on peoples lives.
Prioritize the key
values at stake.
Create a solid
information base.
Integrate the publics articulation of its core values
with technical and legal expressions and social,
economic and environmental costs and benefits.
Identify a broadrange of solutions.
Identify both conventional and innovative solutions,including governmental and non-governmental
means of addressing the problem.
Weigh the pros, cons
and trade-offs
among solutions.
Systematically apply the publics priorities to the
alternative solutions, emphasizing the most
significant trade-offs among alternatives.
Make the bestdecision possible.
Identify the solution that best addresses the problem,potentially drawing on multiple approaches when
they are mutually reinforcing.
Adequately
distribute speaking
opportunities.
Mix unstructured, informal discussion in smaller
groups with more structured discussion in larger
groups. Create special opportunities for the reticent.
Ensure mutual
comprehension.
Ensure that public participants can articulate general
technical points and ensure that experts and officials
are hearing the publics voice.
Consider other
ideas and
experiences.
Listen with equal care to both officials and the
general public. Encourage the public to speak in their
authentic, unfiltered voices.
Respect other
participants.
Presume that the general public is qualified to be
present by virtue of their citizenship. Presume
officials will act in the publics best interest.
8/4/2019 Mass Engaging Report
21/119
18Evaluating Deliberative Public Engagement
General Evaluation Criteria
The question o whether a process even aspires to approximatedeliberative citizen engagement precedes any serious attempt atevaluation. Ater all, one can justiy the considerable eort evalua-tion requires only i the process being examined claims to be (or has
some reasonable expectation o being) related to the particular idealso public engagement and deliberation. Presuming that the delibera-tive engagement program, project or event aspires to these ideals,then the ollowing our evaluative criteria are appropriate or assess-ing its overall quality on these terms.6
When implemented, a deliberative public engagement processshould be evaluated on its own terms. That is, the best way to judgeits eectiveness is to assess the extent to which it achieves the goalsthat such a process strives to achieve. Because deliberative engage-ment programs share some common ideals, however, they do sharea concern with (1) design integrity and (2) sound deliberation andjudgments. Ater all, such programs ail immediately i their designor the ensuing deliberation does not meet basic requirements, asdescribed below.
In addition, these engagement processes can be assessed in terms
o the outcomes their public events engender. Here, more varia-tion occurs among dierent programs, and the third criterion thusrequires (3) infuential conclusions and/or actions. For some pro-cesses, it will be enough or deliberation to yield recommendationsthat carry infuence, whereas other programs will emphasize takingdirect action, whereby citizens not only talk but work together toexert their infuence.7
Finally, the greatest variation in purposes comes rom the
wide range o (4) additional benets or public lie that delibera-tive engagement processes hope to realize. Herein, I will considermethods or evaluating a range o these, rom benecial eects onindividual citizen participants to broader impacts on the communityor even the larger political culture. I call this nal criterion second-ary benets because it reaches beyond the immediate purpose andimpact o citizen deliberation, but nearly every deliberative enterprisecarries ambitions that extend outward in this way.8
The sections that ollow consider these our criteria in greaterdetail and suggest the eective means whereby one might assess theaccomplishment o each.
8/4/2019 Mass Engaging Report
22/119
19masslbp.com
Criterion 1: Design Integrity
A high-quality deliberative engagement process gains its power partlyrom the integrity o its development, design and implementation.This criterion can be broken down into three more specic sub-components:
1. Unbiased raming. The process by which issues are ramed ordeliberation should be transparent, subject to open criticism byall interested parties. The resulting issue rame should be a airrepresentation o conficting views and arguments. Even when theorganizers imagine that they have an undened, open issue rame(e.g., political reorm, without speciying any options), its stillthe case that they selected that issue and generated language todescribe it.
2. Process quality. The deliberative procedures themselves shouldbe developed in consultation with (or at least subjected to commentrom) interested parties, particularly those with dierent points oview on the issue at hand, and the resulting process should beconsistent with the best practices or deliberation (e.g., rigorous ana-
lytic process or studying the problem and generating and evaluatingsolutions, along with respectul and egalitarian relationsamong participants).
3. Representative. The selection o citizen participants should givebroad opportunity to all potentially interested parties (excluding onlythose with public oces or unusually high personal/nancial stakesin an issue). The resulting body o citizen participants (hereater
called a citizen panel) should prove representative o the generalpopulation and, in particular, include representatives rom any per-manent minorities (i.e., groups or whom public policy consistentlygoes against their interests) and even smaller-numbered culturallyrelevant identity groups (i.e., sub-publics or communities who seekvisible representation in any public deliberative body).
One can assess these design eatures through direct inspectiono relevant event and design records, along with interviews withorganizers and interested third parties. Specically, I recommend theollowing evaluation methods:
1. Evaluating the issue rame. Whenever possible, the issue rames
8/4/2019 Mass Engaging Report
23/119
20Evaluating Deliberative Public Engagement
airness should be evaluated beore the deliberative body convenesand reaches its conclusion. This way, evaluations will not refectreactions to the outcomes. A neutral third party (e.g., unaliateduniversity researcher or program evaluation specialist) can evaluateindependently, through inspection o project documents and proce-
dures, whether the raming process was neutral and transparent, butideally this process is evaluated by interested parties rom all relevantperspectives. The latter approach oers a more varied perspective onthe procedures airness to the particular concerns o dierentinterest/advocacy groups.
2. Evaluating process quality. This ollows the same basic protocol asissue rame evaluation, with two exceptions. It is useul to get prelim-inary process assessments beore deliberation begins, but wheneverpossible it is helpul to complement these with assessments duringand ater deliberation. The actual implementation o the delibera-tive procedures may shape the nal evaluations thereo. To ensurecommensurate evaluations, it is also important to discuss with eachevaluator including interested parties the conception o delib-eration underlying the process design. (This parallels the present
chapters eort to careully dene deliberation.)
3. Assessing representativeness. The nal body o citizens whoattend the event (versus those who register or pledge to attend)should be surveyed to determine their relevant demographic andideographic (attitudinal) characteristics. These characteristics canthen be compared against relevant census and survey data or thetargeted geographic/political region. This can be more expensive
when the target area does not have a readily available census orsurvey prole, as in the case o a watershed, transit area, biozone orother non-standard region.
Criterion 2: Sound Deliberation and Judgment
Beyond their process eatures, deliberative civic engagement pro-grams should show signs o high-quality judgment. Thus, theyshould produce the ollowing outcomes:
1. Maniest disagreement. Public deliberation should include periodso debate among the citizens (hereater called panelists, as in theinstance o a citizen panel) on both questions o act and more un-
8/4/2019 Mass Engaging Report
24/119
21masslbp.com
damental moral issues. The absence o such a clash would suggestexcessive consensus-seeking among citizens who surely have genuinedierences in experiences and values.
2. Supermajorities. Deliberative groups should be able to work
through their dierences and oten reach broad agreement whenassessing initiatives. Narrow majority views should sometimes growinto large majorities, and minority viewpoints should sometimesprevail.
3. Inormed and coherent judgments. Citizens judgments shoulddevelop in light o the inormation presented, the views put orwardand the careul, honest discussions among participants. As a result,participants should demonstrate more inormed and coherent viewson initiative-related issues ater participating in panel discussions.Participants should be able to give reasons or their views and shouldbe able to explain the arguments underlying alternative pointso view.
One can assess these outcomes through direct observation o thedeliberative process, complemented by systematic surveys and inter-
views with participants, event moderators and otherinterested observers.
4. Assessing disagreement level. Systematic coding o an audio (orpreerably video, or ease o transcription) record o the deliberationcan establish whether disagreement took place. This can be comple-mented with interviews o participants to determine whether theysubjectively experienced such disagreements and whether there were
any potential disagreements they chose not to bring orward (i.e.,internally censored).9
5. Assessing supermajorities. This is assessed directly rom the eventrecords when ormal votes are taken by the citizen deliberators. Inall cases, it helps to survey the participants aterward, to nd out thedegree to which they (privately) supported any nal recommenda-tions.
6. Evaluating judgments. The citizens nal judgment should beevaluated by a neutral third party, as well as interested parties, toobtain their varied assessments o its soundness. In these cases (and
8/4/2019 Mass Engaging Report
25/119
22Evaluating Deliberative Public Engagement
those where no nal judgment is reached), it is also helpul to com-bine an analysis o the deliberation with a survey o participants,so that one can assess the degree to which the inormation andperspectives provided in the event shaped citizens individual viewson the issue. In particular, post-deliberation citizens should be more
knowledgeable, have better correspondence between their views andrelevant acts, and understand the cons o whatever recommendationthey ultimately made.
Criterion 3: Infuential Conclusions/Actions
Once implemented, successul deliberative processes should showclear evidence o their infuence on the policy-making process oron the actions o the wider public. Depending on whether theyemphasize policy recommendations and/or direct action, eectivedeliberative citizen engagement should produce the ollowing results:
1. Infuential recommendations. Deliberative engagement processesshould prove to be an eective mechanism or making a policyproposal succeed or ail in light o the citizens recommendations.Specically, when a clear majority o panelists avour a particular
policy initiative, its chances o prevailing should increase, and thereverse should be true when citizens oppose a policy.
2. Eective, coordinated action. Deliberative bodies that attemptto generate change through direct action should be able to coordi-nate their post-deliberative eorts to thereby change the relevantvoluntary actions taken by the larger public, which may indirectlyspark policy changes (depending on whether the citizens action plan
involves public policy change).One can assess these outcomes through institutional, policy and
sociological analysis, which involves a history o the relevant policiesand public actions through examination o records and interviewswith ocials, activists and lobbyists.
3. Assessing infuence. This is a tricky undertaking because it is otendicult to establish baseline probabilities o policy outcomes.10 Themost eective approach is probably employing a third-party evalu-ator who combines all relevant documentation with interviews,preerably both beore and well ater a deliberative event. Long-termassessment, in particular, could determine whether the infuence o
8/4/2019 Mass Engaging Report
26/119
23masslbp.com
the deliberative engagement builds (or erodes) over time.
4. Assessing action eectiveness. The same basic methods apply toaction as to policy, with the emphasis shiting rom policy analysis tosociological investigation. The latter should entail large-scale longitu-
dinal surveys to assess public behaviour.
Criterion 4: Secondary Benets
I deliberative processes are implemented and the evidence showsthat they are reaching sound and infuential judgments and/ortransorming public action, that would be enough to warrant theirwidespread adoption. Nonetheless, it is important to examine otherpotential outcomes because many deliberative civic engagement pro-grams stress the impact they have on the participants themselves, thewider public or macro-level political processes. To give a sense o therange o these secondary benets in relation to governance, herein Idescribe and suggest evaluation approaches or three: transormingpublic attitudes and habits, changing the attitudes and habits o pub-lic ocials and altering strategic political choices.11
1. Transorming public attitudes and habits. In the long term, delib-erative panels could transorm not only their participants, but alsothe larger public. Those participating in, engaged with or captivatedby the panels should report stable (or rising) levels o public trustand signs o reduced civic neglect. Voter turnout in elections mightincrease, and citizens should develop political belies (e.g., a sense opolitical sel-condence) conducive to varied orms o public partici-pation (e.g., attending public meetings, using public aairs media).
2. Changing public ocials attitudes/behaviour. Citizen deliberationcould also change how public ocials think and behave in relation tothe larger public. Government ocials could develop more avour-able views o the judgments that citizens make during deliberativeevents. Ocials should also demonstrate an awareness o the impor-tance o citizen deliberation and come to respect panel judgments. Asa sign o improved leadership, elected representatives (and agencyocials) could also begin to step away rom conventional publicopinion on initiatives in anticipation o deliberative panel judgmentsto the contrary.
8/4/2019 Mass Engaging Report
27/119
24Evaluating Deliberative Public Engagement
3. Altering strategic political choices. In addition, the public delibera-tion could change the strategic choices made by political campaignproessionals during initiative campaigns. Panels will have succeededin transorming the electoral environment i initiative and policycampaigns begin to ocus more o their energy on addressing the
issues raised by deliberative panels (e.g., holding debates ocused onpanel issues) and incorporating deliberative panel results into cam-paign advertising. A more ar-reaching eect o the panels could bethe emergence o routine pilot-testing potential initiatives with low-cost varieties o deliberative polling, trying to understand how thepublic will view the initiative ater deliberating.12
The methods o evaluation used to assess these secondary out-comes would be as varied as the potential impacts themselves.
Measuring shits in public attitudes and habits. One can assessimpacts on participants and the larger public through surveyresearch and inspection o election records (in those countries wherevoting is not mandatory). Examples abound or what to include insuch surveys and how to assess it, but the best examples includelongitudinal assessment (to establish change over time), comparisongroups (to dierentiate deliberations impact rom those eects o
other social/political orces) and a wide variety o measures (e.g.,breaking down ecacy into multiple sub-components, such as sel-ecacy versus collective ecacy, i.e., a sense o eectiveness whenacting in a group).13
4. Measuring changes in public ocials attitudes/behaviour. Toassess changes in public ocials, survey methods likely will ail,owing to poor response rates conventionally obtained among elites.
Instead, one should assess these outcomes through interviews withpublic ocials and in-depth, longitudinal legislative and policy analy-sis that compares processes beore and ater the deliberative civicengagement, in light o other changes in the political/legislativeenvironment.
5. Detecting shits in strategic political choices. One can assessthese outcomes through interviews with public ocials, lobbyists,campaign ocials and political activists. This can prove especiallychallenging, as it requires accessing internal strategic decisions (ordocumentation thereo) within organizations whose interests maynot be well served by such investigation. I one can obtain such data,
8/4/2019 Mass Engaging Report
28/119
25masslbp.com
however, it is possible to detect signs o the deliberative processexerting its infuence. For instance, policy initiatives that ail to passmuster in trial runs (i.e., in the mock deliberative polls describedabove) are subsequently withdrawn; this can indicate that antici-pation o the eventual deliberative citizen engagement process is
causing more careul vetting o the proposals such a group might putbeore policy-makers and the general public.
Conclusion: Integrating Evaluative Methods
Table 2 summarizes the preceding discussion and breaks downevaluative methods into two columns. The rst describes a basicevaluation those methods most readily deployed on a modestbudget and within a narrower time rame. The second column aug-ments these basic methods with additional assessment tools, whichmay require more labour, money and time. Whether the evaluationrequires more than a basic method depends on the resources andgoals, but it is important to recognize the limitations o the basicevaluation approaches in terms o their reliability and validity.
In conclusion, it is important to consider how one integratesthese various evaluation metrics. That is, how does one move rom
separate assessments o each criterion (or sub-component) to anoverall evaluation o the deliberative citizen engagement process as awhole? This depends, again, on ones conception o the project, butthe ollowing approach will apply to many such programs.
Each o the three elements o design integrity count as pass-ailelements, and a subpar evaluation on any one o these yields a nega-tive summary evaluation o the entire process. That is, i any aspecto the design ailed to meet basic standards or integrity, the other
outcomes o the process are all suspect.The three elements o sound deliberation and judgment should
be viewed as parts o a coherent whole, such that one arrives at asingle assessment o deliberation/judgment in light o each element.The third o these might be most important (i.e., the coherence andsoundness o the groups judgments), but this should be weighed byhow rich the disagreement was and how eectively the group couldmove toward a supermajority. Outstanding perormance on twoo these criteria might obviate lower perormance on another, butoutright ailure on either the rst (disagreement) or third (quality ojudgment) should yield an overall assessment o program ailure.
8/4/2019 Mass Engaging Report
29/119
26Evaluating Deliberative Public Engagement
Criterion Basic Evaluation Additional Evaluative Method
Sound Deliberation and Judgment
Unbiased
framing
Third-party document
inspection prior todeliberation
Inspection by interested parties
Third-party inspection ofprocedure instructionsand direct observation ofprocess
Inspection by both third-partyand interested parties before,during and after deliberation
Compare citizenparticipant demographics
with census data
Conduct detailed survey ofcitizens and target population to
check for differences in bothcensus and attitudinal variables
Direct inspection ofdeliberation for signs ofdisagreement
Survey participants to judgetheir subjective experience ofdisagreement and check forself-censorship of potentialdisagreements
Check final vote tallies Survey participants to learn theirdegree of private support fortheir public recommendations
Third-party assessmentof the citizens finaljudgment in light ofavailable information
Inspection by interested partiesand survey of participantsrelevant knowledge/perspective
Third-party documentinspection prior todeliberation
Inspection by interested parties
Process quality
Representative
Manifestdisagreement
Supermajorities
Informed andcoherentjudgments
Influentialconclusions/actions
Third-party assessmentof policy impact
Take longer-term assessmentsto capture gradual/eventualimpact (or detect erosion ofinfluence)
Influentialrecommenda-tions
Third-party assessmentof impact on publicbehaviour
Inclusion of large-scale,longitudinal population surveys
Effective,coordinatedaction
Design Integrity
8/4/2019 Mass Engaging Report
30/119
27masslbp.com
The infuential conclusions/actions criteria are dierent in that someprograms will emphasize only one or even neither o these cri-teria. All deliberative citizen engagement programs, however, shouldorient toward one or the other to at least a degree, lest deliberationbecome seen as merely discussion, disconnected rom action. Eventhen, poor perormance on a programs relevant infuence criteriondoes not impugn the entire exercise; rather, it suggests the need orimproving the component o the program that leverages infuence.
Finally, assessment o secondary benets stands apart romthese other criteria in that program success may not require evidenceo these impacts. I a program is well designed, deliberative and
infuential, these become bonus eects, not strictly necessary orjustiying the citizen engagement program per se. In the long run,however, these secondary benets could be o tremendous value ora public and its political culture. A more engaged public, legitimateinstitutions and responsible, deliberative politics could dramaticallyincrease the capacity or shared governance and public action and,ultimately, yield much better public policy. Such potential impactsshould be assessed, or evidence o these changes could increase theestimated value o deliberative citizen engagement, thereby warrant-ing the time and resource expense it requires.
Criterion
Secondary Benefits
Basic Evaluation Additional Evaluative Method
Transforming
public attitudesand habits
Post-deliberation survey
of participants
Longitudinal survey (and
analysis of voting records) forboth deliberation participantsand wider public
Interviews with publicofficials
Legislative and institutionalpolicy analysis
Third-party assessmentof changing politicalclimate
Intensive interviews andstrategic document analysiswithin policy-relevantinterest/advocacy groups
Changingpublic officialsattitudes/behaviour
Altering strategicpolitical choices
8/4/2019 Mass Engaging Report
31/119
28The Canadian Experience
This chapter evaluates the eectiveness o community engagementand public participation eorts in Regional Health Authorities
(RHAs) in Canada. An extensive review o the literature examiningpublic engagement theories and evaluative methodologies and inter-views with health care and other social services practitioners helpedestablish some common themes and directions in public engagement.Though not meant to be comprehensive, case studies are includedthat represent a more detailed investigation o particular publicengagement processes.
Because health care delivery is a provincial responsibility, RHAs
across Canada operate independently o one another. Even RHAswithin the same province have signicant dierences in approach andmethodologies. This is largely due to the act that the legislation andpolicies that are used by provinces to mandate community engage-ment or RHAs are oten very general in their requirements andreporting rameworks.
The literature and interviews suggest that in regions where a
commitment to public participation in a health care delivery systemis being implemented in a meaningul way, there is a growing con-sensus about what constitutes an eective engagement, in terms oboth processes and outcomes. Outcomes are increasingly measuredby population health and patient-centred metrics such as increasedinvolvement in health programs and client satisaction with healthcare service. Common themes or eective processes in publicengagement are oten dened using terms such as respect, diversity,
meaningul participation, accountability and equity.A growing body o work in the eld o community engage-
ment evaluation categorizes three types o evaluation summative,ormative and developmental. Summative evaluations measure
Brian Eng, Wellesley Institute
The Canadian Experience:Observations and Lessons rom theCanadian Health Sector
8/4/2019 Mass Engaging Report
32/119
29masslbp.com
the outcomes that are the end result o engagement exercises, suchas uptake o services, client satisaction, better health outcomes.Formative evaluations measure progress in achieving process-oriented goals, such as participant satisaction with the process,appropriate inormation, meaningul dialogue, adequate repre-
sentation o community diversity. Developmental evaluation isan emerging eld that attempts to measure change and is used inengagement eorts that are working on complex problems in whichoutcomes tend to be unpredictable and goals, purposes, contexts andso on may change as the engagement process develops. An exampleis attempting to measure the relative impact o particular processes inmoving toward change is the way systems unction.
Summative evaluation has been used extensively in the healthcare system even prior to the advent o community engagementprocesses. The metrics and indicators are well understood, and thereis a general consensus about how to apply them.
Formative evaluation is less well developed in public engagementactivities in the health sector. There are some promising initiativesin this area and a growing body o practice in the health care andother social services sector around evaluation o processes such as
stakeholder analysis, comprehensible inormation dissemination andaccountability. It is important to note that ormative evaluations takeplace regularly in engagement processes, albeit on an ad hoc andoten personal basis. Practitioners oten point to the need or morerigorous methodologies that can transcend personal and institutionalbias (both positive and negative) and where the tools and outcomesare comparable across engagement activities.
Developmental evaluation is very much an emerging eld andis not being addressed in any signicant way in the health sector.Engagement in the health sector is still driven mostly by the needso health authorities to deliver health care programs. While thedenitions o health care programs has expanded beyond access tomedical care and now includes programs to encourage healthy liv-ing, the engagement processes are still driven by predetermined goalsand anticipated outcomes that lend themselves to summative and
ormative analysis. However, as health authorities begin to grapplewith their role in looking at the broader social determinants o healthand the systemic changes required to make progress in these areas,developmental evaluation will become increasingly important in
8/4/2019 Mass Engaging Report
33/119
30The Canadian Experience
engagement processes.1The case studies in this chapter will help to illuminate some o
these concepts and provide some indicators o common themes andchallenges.
Drivers or change
Since the mid-1990s, governments in Canada have been devolvingthe responsibility or allocating resources in social services deliveryto community levels. The theory behind this action is that local com-munities are better at determining their social service needs thancentralized bureaucracies. The relative merit o this approach is stillopen to debate, but the process is well advanced in many areas.
With this devolution o responsibility has come a greateremphasis on nding ways to involve the community members,sector organizations and other partners in discussions and decision-making about resources allocation. Public participation, communityconsultation and community engagement have become importantcornerstones in the delivery o social services.
Traditionally, community engagement generally meant inorm-ing the public about available services and encouraging them to
use those services. This is now being augmented by processes suchas roundtables, advisory committees, town halls and open orums,where community members and stakeholders receive inormation,discuss options, and sometimes have decision-making power aboutthe nature o services.
This is certainly the case in the health sector. Provincial govern-ments across Canada are moving away rom the model o directly
unding service delivery organizations such as community healthcentres (CHCs) and hospitals. They are establishing Regional HealthAuthorities and local networks responsible or developing compre-hensive service delivery programs. In many cases, the provincialgovernment requires these regional bodies to have a communityengagement strategy to inorm and guide the development o theirplans.
The traditional method o public participation in health care
delivery was through involvement on the boards o directors osel-governing institutions such as community health centres andhospitals. These bodies were unded directly by the government andmay or may not have had other community engagement strategies to
8/4/2019 Mass Engaging Report
34/119
31masslbp.com
develop their programs. A report commissioned by the Associationo Ontario Health Centres in 2006 indicated that this model waschanging signicantly.
Across Canada, there has been a general shit in how publicparticipation in health care is being carried out. Community gov-
ernance has been shiting away rom direct democracy o locallyelected community boards toward engagement through variousother mechanisms such as inormation sharing and consultation, andby the establishment o community advisory committees, councils orgroups. From the perspective o governments, devolved authority toregional structures and the encouragement o citizen participation inplanning and priority setting through these various means is seen as
moving health care closer to communities. But locally governed com-munity health organizations and individual community memberssee these trends as a movement toward more remote and centralizedgovernance. Although community engagement is being promoted asa means to involve citizens in health care planning, empowerment olocal citizens (including the most vulnerable populations) achievedthrough local community governance may be declining.2
Given the caution expressed in this report, it is clear how impor-
tant it is to have an evaluation ramework o community engagementto determine whether the goals o community involvement andempowerment are being met.
Cape Breton
The Cape Breton District Health Authority (CBDHA) is one o ninehealth authorities in Nova Scotia. It is primarily a rural catchment
with several medium-sized towns and an urban centre o 25,000people in Sidney.The Nova Scotia Health Authorities Act requires that each
District Health Authority (DHA) establish Community HealthBoards (CHBs) to serve as the eyes and ears o the community.This is a primary vehicle or public consultation and participationin Cape Breton. There are six CHBs in the Cape Breton DHA thatcover a range o rural and urban catchments. Board members are
recruited through local advertising and word o mouth.CHBs provide advice to their health authorities about the needs
o their community. Depending on the CHBs internal inrastructurecapabilities, this advice can result rom internal board discussions or
8/4/2019 Mass Engaging Report
35/119
32The Canadian Experience
public consultations such as community orums.As well, CHBs distribute unding to community agencies imple-
menting DHA programs. They are required to report back to theDHA about the use o those unds and the impact on health out-comes in their community.
The Cape Breton District Health Authority (CBDHA) pro-vides a summative evaluation across the district o this inormationand makes it available to the public on its website. This evaluationincludes a description o key indicators o health and liestyle out-comes. A progress report is updated each year.3
One problematic process is the recruitment and retention oCHB members. An ad hoc evaluation using an inormal survey
method and conversations with existing and past board membersidentied key issues such as volunteer burnout, transportation andunderstanding the role o CHB members.
CHBs communicate with one another within each region andacross the province through a council o chairs, enabling them toevaluate their experiences with other engagement processes that havesimilar mandates. In act, a new sta position Community HealthBoard coordinator was recently established at the Cape Breton
District Health Authority ater members heard about the eective-ness o similar stang components in other health authorities in theprovince.
The CBDHA interviewee indicated that ormative evaluation oCHB work is on the agenda. At the moment, there are limited toolsin use, but additional tools are being contemplated. The intervieweeexpects that the major challenges will be nancial i the implementa-
tion is resource-intensive. It is assumed that there will be signicantsupport rom senior sta and policy-makers at the health authority.According to the interviewee, They are not araid o change.
Winnipeg Regional Health Authority
The Winnipeg Regional Health Authority (WRHA) has a thoroughand well-researched Community Development Framework.4 Theramework promotes the regions organizational development and
acilitates networking, inter-sectoral collaboration, public participa-tion initiatives and local area development. The ramework denescommunity, establishes a participation model and outlines methodso public participation.
8/4/2019 Mass Engaging Report
36/119
33masslbp.com
The WRHA oers the ollowing rationale or having a compre-hensive community development model.
Community development empowers people to have morecontrol over the decisions that infuence their own health and thehealth o their community through increasing personal control over
their own health behaviour change and by addressing the underly-ing health determinants such as poverty, housing, or environmentalthreats. The concept o empowerment is ocused on achieving equityin health and increased public participation in health program deci-sion-making.
The public participation process involves six advisory councilsand dozens o place-based and program-specic working groups.
Rigorous evaluation determines whether the advisory councils arerepresentative o the communitys diversity. Potential advisory coun-cil members are interviewed and asked to provide inormation aboutthemselves, including sel-identication with minority or marginal-ized groups. Advisory council members are chosen with a view tothe overall makeup o the councils being representative o the com-munity. There is also an evaluation ramework or the work o thecouncils based on sel-reported perceptions o the work and processes
o the group, as well as indicators o community interest in partici-pating in the work o the councils. These reports are rolled up andanalyzed by sta to the health authority and reported to the WHRAboard.
The ramework also outlines extensive evaluation tools or thevarious working groups. These tools are used by acilitators to moni-tor the progress o the group as it denes goals, begins to understand
issues, assesses participation and so on. This process has been inplace or a short time only and, at present, is used primarily bythe acilitators to track and rene processes. However, the WRHAinterviewee elt that standardizing the evaluation will help to create amore general picture o the eectiveness o the community develop-ment model once the resources are in place to do a ull-scale rollup othe inormation. The interviewee also indicated that one o the pur-poses o the standardization was to nd evaluative tools that t with
models that are more prevalent and understood in the health carecommunity. In other words, to nd models and tools that evaluatequalitative actors in a culture that is more used to and adept at usingsummative, quantitative tools.
8/4/2019 Mass Engaging Report
37/119
34The Canadian Experience
An example o an innovative WRHA development is in theirwork with other government departments and social service agen-cies. The models and evaluative tools along with human resourcesare now being made available as a part o an inter-sectoral collabora-tion, and it is hoped that comparing outcomes across sectors will help
to rene the evaluative tools and the methods and processes o publicparticipation.
Saskatoon Regional Health
In many ways, Saskatchewan has one o the longest histories opublic participation in health care. When one sta member at theSaskatoon Health Region was asked, Why do you engage the pub-
lic? they answered, Because we are the province o medicare. TheSaskatoon Regional Health interviewee reinorced this point by say-ing that the region is itsel as a steward o public unding and elt anobligation to engage the public or eective resource allocation.
There is a legislative requirement in Saskatchewan to have pub-lic input into the health care system. However, the province doesnot monitor this except as a complaint-based system. The SaskatoonHealth Region has a clear Community Development Framework.5
Like many others, it has identied reasons or community engage-ment that include encouraging community participation in health,ocusing on the creation o healthier communities and expanding theunderstanding o actors that sustain health o communities. It alsoidenties principles and methods.
According to the interviewee, there is very little rigorous evalu-ation o the program as a whole. However, there is evaluation o
particular initiatives, primarily driven by external unders. Theexample cited was o an Aboriginal partnership in which they willbe hiring an evaluator to assess the outcomes o the program and, ashow well important values such as respect, equity, integrity and soon are being incorporated and to help drive the engagement process.
The Saskatoon Health Region realized that its advisory councilsystem seemed to be foundering, so it surveyed advisory councilmembers about what was working and what was not and asked
or suggestions about what could be improved. Implementing theserecommendations, however, has been hampered by a recent changeo government that has resulted in a transition period while thegovernment examines new bureaucratic processes. The interviewee
8/4/2019 Mass Engaging Report
38/119
35masslbp.com
recalled that this kind o delay has happened in the past and oeredthe opinion that ormative evaluations o this kind are most use-ul when they can be applied over time since they are intended tomeasure progress rather than outcomes. The evaluations becomeineective when there is constant restructuring and repositioning
o ormats and methodologies that are the result o bureaucratic orpolitical imperatives rather than o the evaluations themselves.
The Saskatoon Health Region comprises both the urban cen-tre o Saskatoon and surrounding rural areas. The intervieweementioned one rural community engagement ocused on usingdiscretionary public unding to preserve a local acute care acility.The questions were asked: How do you evaluate that engagement
outcome against the act that it seems relatively clear that the sameamount o resources put into programs or healthier living, coupledwith programs to reimburse transportation costs or medical carein the nearby urban municipality, will result in better health out-comes or the community as a whole? Are people not receiving theright inormation? Are they not assimilating it? In other words thequestion that is being asked is how to evaluate whether or not theengagement is meaningul.
The Saskatoon Health Region is keen on developing a bet-ter understanding o community engagement and how it can beeectively evaluated. To this end, it is involved with a Regional Inter-sectoral Committee that has commissioned an evaluation o publicparticipation in social service delivery in the province.
Vancouver Coastal Health
Vancouver Coastal Health (VCH) includes 25% o the populationo British Columbia in an area that covers the city o Vancouver, itssuburbs and as ar along the coast as Powell River. VCH has veCommunity Health Advisory Committees: three are geographicallybased, one works with the Aboriginal community and, in 2006,a Palliative Care Community Reerence Committee was established.As well, VCH delivers numerous project-based engagement exercisesintended to provide advice to the health authority on program
development.
In 2006, VCH hired a consultant to help sta develop evalua-tion methodologies and tools. The resulting ramework identied the
8/4/2019 Mass Engaging Report
39/119
36The Canadian Experience
purpose o the evaluation as to assess the practice (or process) ocarrying out community engagement processes and the impact (out-come) that they have on VCH decision making.6 The rameworkalso established a set o questions or consideration.
A set o surveys to be completed by participants and project
leaders at the conclusion o selected community engagement consul-tations was created as an evaluation tool. Follow-up surveys are alsosent out. Several dierent templates have been developed or use indierent contexts.
The process evaluation is intended to evaluate a broad range oprocess-oriented questions, including demographics and participantmotivation and satisaction. Sample questions include Did partici-
pants eel like their opinions matter to the organization? and Didthey have enough inormation to be able to contribute ully?The outcome evaluations have their own particular nuance in
that they are not evaluating program outcomes in the traditionalsense. Instead, they are being used to determine the value o commu-nity engagement (CE) processes to particular projects. For example,questions are asked to determine whether project leaders eel thatthe CE process was useul to their project outcomes and what moti-
vates project leaders to integrate a CE process into their projectwork plan.
The intention o the VCH sta has been to compile thesereports as part o their yearly reporting process. However, as withmany other engagement processes, it is a challenge to nd the statime and other resources to properly systemize, collate and analyzethese reports. As the VCH interviewee put it, there is a constant
need to do, do, do. Time spent in evaluative work by departmentsta can be perceived as time taken away rom delivering the com-munity engagement program.
Nevertheless, VCH sta do spend time in ad hoc evaluationand process analysis as part o their ongoing work. This is a naturalpart o the day-to-day discussions and collaboration among sta.The interviewee did oer the observation that a more systematicapproach to collating the learnings rom these evaluations might
prove valuable in bridging the culture cap between the communitydevelopment unctions and the health care delivery unctions. Butshe oered the caution that any evaluative process needs to be used,analyzed and understood in terms o the contexts in which engage-
8/4/2019 Mass Engaging Report
40/119
37masslbp.com
ments take place. Any attempt to overly systemize evaluations in anattempt to make them applicable across wide varieties o engagementprocesses will likely result in ailure due to receiving insucientinormation or the potential or misinterpreting inormation withoutknowing the context.
St. James Town Initiative
One o the many things to be learned about evaluation rom otherelds o community engagement practice in the health care sector isthe burgeoning eld o Community-Based Research (CBR). Duringthe development o CBR projects, ormative, process-based evalua-tions are oten used to dene and oten rene the purpose, scope and
methodologies o the project.The St. James Town Initiative o the Wellesley Institute, orexample, was envisioned by its initiators as a research project thatwould look at neighbourhood actors in newcomer health outcomes.It is well known that new immigrants to Canada tend to be healthierthan the general population but that their health outcomes declineover time. There is a general understanding about the drivers o thisphenomenon in terms o lower incomes and more dicult access to
culturally appropriate health care. The Wellesley Institute wantedto examine neighbourhood actors in a distinct geographic area thathas a high immigrant population. North St. James Town in Torontowas a likely candidate. The methodology initially envisaged was aqualitative study that would ollow a select group o individuals andamilies or a period o time using periodic surveys, ocus groups andso on. There was also an intention to do a quantitative study.
Sta at the Wellesley Institute began to work with community-based organizations in St. James Town to recruit people or thequalitative study and to seek input on the indicators. Almost imme-diately, the community members engaged in an ad hoc ormativeevaluation to determine whether they wanted to participate. TheWellesley Institute was told that i the sole purpose o the engage-ment was to learn more, then St. James Town community memberswerent interested, concluding: Our community has been studied to
death. What we need is action.
The Wellesley Institute repurposed the engagement to includean action component, evaluating the initial methodologies to deter-
8/4/2019 Mass Engaging Report
41/119
38The Canadian Experience
mine whether they could result in action. They concluded that theyhad to introduce some new, more participatory research componentsthat would allow community members to dene early in the pro-cess some areas o action that might be pursued as the engagementproceeded. As a result, the rst methodologies employed in the quali-
tative research were Photo Voice, Community Mapping and ConceptMapping. This allowed participants to make an early identication oneighbourhood actors that might be actionable.
Common Themes
Some common themes have emerged rom the research.First, the health care system is relatively adept at using sum-
mative, quantitative evaluations to assess the impact o communityengagement and public participation processes. It can look at pro-gram evaluations, uptake models and so on to determine whetherhealth outcomes are improving as a result o engagement. Theindicators and metrics are reasonably well understood and agreedupon. The evaluations have expanded beyond just access to medicalservices and now include other elements o population health.
Second, it is clear that ad hoc ormative evaluations take place
during many engagement processes. These are natural and otenunintended evaluations that occur because acilitators or partici-pants want to know that their eorts are eective and valuable.Interviewees and other practitioners have indicated that more rig-orous ormative evaluations would be valuable in assessing andimproving engagement processes.
Third, as the health care sector moves into more complex
engagement processes that examine the larger systemic issues thatimpact health outcomes, more complex evaluative tools will need tobe developed and used to assess the outcomes and processes o theengagements.
Challenges
There are a number o challenges in evaluating community engage-
ment and public participation in the health sector, particularly romthe view o ormative or developmental evaluation.
One challenge is that common denitions or ormulationso various key terms and components do not exist. What is pub-
8/4/2019 Mass Engaging Report
42/119
39masslbp.com
lic participation in one place is analogous to civic engagement inanother and community development somewhere else. There areno common denitions o key community, stakeholder, user and soon. Goals and principles o community engagement vary rom oneprocess to another. This occurs not only in the heath care sector but
across the entire eld o community engagement and development.Advances have been made in this area,7 but urther opportunitiesexist to rene our denitions by collaborating across processes. Onebenet would be to move the ormative evaluations rom the ad hocand sel-reported methods to more rigorous methods in which theoutcomes and learnings can be more easily shared across jurisdic-tions and sectors.
Another challenge is that the more rigorous evaluations tend tobe resource-intensive. The long-term benets o diverting resources,particularly in the health care sector, rom immediate problem-solving into evaluation have to be clearly articulated. As well, thelevel o unding required can sometimes be unpredictable at the starto an engagement process. This is problematic or unders, whorequire predictable costs.
On the cultural ront, large institutions oten resist new tools and
techniques. Without a champion o innovation within the institution,there is a tendency to use what appears to have worked in the past.There is also a resistance to evaluating or outcomes that appear tobe outside the scope o the engagement process. While strengtheningtrust in democratic processes is regularly identied by practitionersas a probable and desirable outcome o a successul engagement,there is very little work being done by institutions to identiy this and
to try to monitor outcomes. Community capacity building is alsooten cited as a goal o community development and engagementprocesses. But it is hard to nd any indication that this is being moni-tored or evaluated in any signicant way.
There is also some resistance on the part o the participants inthe process. Many o the new evaluative tools are highly resource-intensive in terms o both the time required rom participants in theengagement and rom acilitators and evaluators. It also requires a
substantial buy-in rom the participants. This can be particularly di-cult i some individuals eel they have time constraints. There is alsoa tendency to want to roll up the sleeves and work on the problem,and people may not appreciate the value o regular breathing spaces
8/4/2019 Mass Engaging Report
43/119
40The Canadian Experience
to evaluate where they are and where they might be going.
Conclusions
The health care system in Canada has undergone undamentalrestructuring in the last 20 years. There has been a greater emphasis
placed on public participation in developing and allocating resources,setting priorities and creating programs. The degree to which this callhas been taken up has varied considerably across the country.
It is important to note that most health care authorities andinstitutions are in the business o allocating resources, developingpolicies and creating programs that address particular health prob-lems. However, it must be recognized that many public engagement
processes (particularly those that genuinely involve grassroots com-munity members) will tend to move beyond these limitations toaddress the larger issues o public policy, community values, equityand accountability, thereore:
Use a mix o evaluative methods
There are dozens o possible community engagement tools available,ranging rom public awareness programs to online surveys, ocus
groups, public meetings, citizens assemblies, advisory councils andcommunity health boards. People will participate in these orumsby a variety o means. They may volunteer to be part o a process.They may be randomly selected. They may be elected rom theircommunities or appointed by institutions.
Because o this variety o methodology, a variety o evaluationtools must be available to determine whether the engagements are
eective. Traditional summative methods may be most useul orgroups working on particular programs. More ormative evaluationsmethods may be necessary to determine the eectiveness o thoseengagements working to dene values and priorities. Developmentalevaluations will be most useul in complex collaborative eorts seek-ing systemic change.
Plan or evaluation
Evaluation needs to be addressed at the beginning o the process.Particularly with ormative and developmental evaluation; theengagement participants should be involved in the planning processrom the start. Eective community engagement is an iterative,
8/4/2019 Mass Engaging Report
44/119
41masslbp.com
evolving process, and regular evaluation o the process itsel will leadto outcomes that have greater impact.
Plan or dierent orms o evaluation or dierent types oengagement and even or dierent stages in particular engagements.Focus groups that are empowered to develop new programs may
need specic data sets and benet rom summative orms o evalu-ation. Citizens assembly engagements may need to evaluate thequality o inormation being provided to them as they go throughtheir process. Advisory councils may need to evaluate whether theyare representative o the community they speak or or work with.
Be fexible
Use the outcomes o evaluation to rethink priorities, directions andmethods being used. The impression that original goals arent beingreached shouldnt be seen as ailure. The engagement process maybe raising new questions, problems and solutions that need to beexplored urther. Be ready to use dierent evaluative processes asneeds and directions change.
Learn, share and collaborate
Good practices, resources and even talent related to evaluative meth-ods should be shared, both within the health care sector and withthe broader social service and community development sector. Thisis particularly true in the use o ormative and developmental evalu-ation. These types o evaluation tend to ocus on processes that areexperiential and oten values-laden. The more we can begin to iden-tiy common denitions and methodologies, the more we will be able
to compare outcomes across dierent engagement processes. Whileit is important to be able to nuance and adapt denitions and tools todierent contexts, it is also important to develop shared understand-ings o broad-stroke concepts. This will lead to evaluative methodsand tools that will produce more meaningul learnings and improvethe eectiveness o community engagement and public participation.
8/4/2019 Mass Engaging Report
45/119
42
8/4/2019 Mass Engaging Report
46/119
43masslbp.com
You will be involved. The local NHS [National Health Service] willinvolve patients, carers, the public and other key partners. Those
aected by proposed changes will have the chance to have their sayand oer their contribution. NHS organizations will work openlyand collaboratively.
NHS Next Stage Review: Leading Local Change, May 2008
This chapter examines existing indicators used in England to mea-sure meaningul engagement and public condence in the health caresector.1 It also examines the challenges that exist in measuring patient
and public involvement in health and lessons that can be learnedrom the British experience. Recommendations and conclusions aredrawn rom a comprehensive literature review as well as interviewswith ve leading engagement specialists in the health eld, carriedout in late 2008. These experts are spearheading many o the initia-tives to improve the quality o health engagement evaluation andassessment.
Traditionally, the National Health Service (NHS) has aordedvery limited involvement to patients and the public. NHS cultureassumed that patients are passive recipients o health care servicesand that their needs were best anticipated and managed through top-down structures that let little room or meaningul consultation ordevolved autonomy. Today, that culture is changing. The desire tosustain public condence and a newound recognition o the exper-tise and experience o patients is driving calls or innovation and
reorm.
Drivers or change
Three reasons explain the shit toward greater patient and public
The English Experience: Evaluating Patientand Public Engagement in Health
Edward Anderson and Emily Fellen, Involve
8/4/2019 Mass Engaging Report
47/119
44The English Experience
engagement. First, across the wider public sector, the appetite amongcivil servants or involving citizens in service design and decision-making exercises has grown. This appetite stems rom the practicaldesire to improve the quality and responsiveness o public services,and serves a secondary interest by connecting the experience o pub-
lic services to a broader democratic agenda. The drive or public andpatient engagement in health care should be viewed within this widercontext.
Second, a new ocus on prevention and behavioural change hasemerged within the health sector in order to meet the needs o anaging population, the rising cost o medical interventions and thegrowth o complex chronic disease. Practitioners and health admin-
istrators have come to view enhanced engagement as an importanttool or encouraging behavioural change and healthy living.Third, a series o widely publicized incidents required NHS
administrators to restore public trust in the health service and its gov-ernance. An early example was the 2000 Kennedy Inquiry into thehigh mortality rate at a childrens heart surgery in Bristol. Among itsrecommendations, the inquirys report included no ewer than10 recommendations aimed directly at the issue o public involve-
ment and empowerment.
Key developments and challenges
Legislative and regulatory changes have also had a powerul eect.Since 2001, all health bodies in the United Kingdom have beenrequired to consult and involve patients in service planning andoperation. As o October 2008, these requirements were extended to
include relevant communities in assessing commissioning decisions.The new duties, embedded in section 242 o the National HealthService Act 2008, extend patient and public engagement rom theservice delivery arena to strategic decision-making. Increasingly,patients and the public are being viewed as ull operational andstrategic partners in the provision o British health care.
Another key change has been the introduction o new structuresor engagement, such as Local Involvement Networks (LINks) set up
in early 2008. Designed to provide a link between citizens and ser-vices, these networks are a vehicle or ongoing engagement. LINkshave been established alongside local councils. Unlike previous NHSengagement structures that relied on articial or unrecognized health
8/4/2019 Mass Engaging Report
48/119
45masslbp.com
boundaries, LINks serve existing and well-established communitiesand integrate easily with local governance structures.
But despite new on-the-ground inrastructure such as LINks anda strong legislative mandate, the NHS aces diculties in implement-ing section 242 o the 2008 NHS Act. Little headway has been made
in gauging the uptake or success o these recent requirements.Some existing indicators show troubling developments. For
example, public trust in NHS is in decline; the British SocialAttitudes Survey 2006 highlighted that just 12% o respondents had agreat deal o trust that the NHS would spend money wisely or thebenet o citizens.2 The challenge o accurately and meaningullymeasuring the impact and quality o public engagement is not unique
to the health arena, although it does ace its own particular obstacles.First, any indicators o successul engagement need to take intoaccount the variety o organizations working within the health eld;good engagement practices at a commissioning organization arelikely to look quite dierent rom the engagement practices amongservice providers. There is also a considerable dierence betweenengaging patients in their own care and engaging members o thepublic in policy-making or planning.
Second, the NHS has traditionally been driven by quantitativetargets and indicators based on clinical outcomes. These hard tar-gets are not well suited or easily adapted to the qualitative and highlycontextual work o patient and public engagement. Not surprisingly,many researchers and organizations have struggled to develop robustindicators that can measure meaningul engagement outcomes.
Moreover, most eorts at engagement do not typically yield
immediately identiable and causal clinical improvements. Theseshortcomings can uel a clash among clinicians, administrators andproponents o engagement who argue that scarce resources shouldbe spent on these activities.
A third challenge is the relative novelty o engagement or theNHS and the constant change o NHS policies, structures and priori-ties, which has hampered attempts to evaluate and refect on healthengagement structures. This bureaucratic churn has muddied the
waters and made it diculty to properly evaluate the ecacy omany programs and initiatives.
Together, these actors explain why a denitive rameworkor assessing engagement has yet to be developed in the United
8/4/2019 Mass Engaging Report
49/119
46The English Experience
Kingdom, though eorts by the Healthcare Commission, theDepartment o Health and the National Centre or Health andClinical Excellence are each underway.
The evaluation o health engagement in England
Drawing on interviews with individuals at the oreront o theengagement in the health arena and in particular with individualsinvolved in setting up rameworks or evaluating engagement thissection examines the extent and quality o evaluations o Patient andPublic Engagement (PPE) activity in England. It then looks at spe-cic case studies and outlines several o the methods or evaluatingpublic engagement in health currently in use in England. It concludes
by elaborating on the themes emerging rom the literature reviewand interviews, and discusses what needs to happen in order