Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
123
Corresponding author: J.K. Popp, Director, Southern Alberta Child and Youth Health Network, Room 3517, c/o Alberta Children’s Hospital, 1820 Richmond Rd. SW, Calgary, AB T2T 5C7; <[email protected]>
HOW DO YOU EVALUATE A NETWORK? A CANADIAN CHILD AND YOUTH HEALTH NETWORK EXPERIENCE
Janice K. PoppLaura N. L’HeureuxSouthern Alberta Child and Youth Health NetworkCalgary, Alberta
Carly M. DolinskiCalgary Health RegionCalgary, Alberta
Carol E. AdairUniversity of CalgaryCalgary, Alberta
Suzanne C. ToughCalgary Health RegionCalgary, Alberta
Ann L. CasebeerUniversity of CalgaryCalgary, Alberta
Kathleen L. Douglas-EnglandCalgary Health RegionCalgary, Alberta
Catherine C. MorrisonSouthern Alberta Child and Youth Health NetworkCalgary, Alberta
Over the past decade, approximately 20 child and youth health networks have been initiated in Canada. The value of any net-work depends on its effectiveness in achieving stated goals; how-ever, description and measurement of network effectiveness is challenging, given the complex multi-sectoral and/or multidisci-plinary relationships involved and the relative dearth of evalua-tion methods specifi c to networks. Since its inception in 2001, the Southern Alberta Child and Youth Health Network (SACYHN) has reviewed the network evaluation literature and developed
The Canadian Journal of Program Evaluation Vol. 20 No. 3 Pages 123–150ISSN 0834-1516 Copyright © 2005 Canadian Evaluation Society
Abstract:
THE CANADIAN JOURNAL OF PROGRAM EVALUATION124
and implemented an evaluation framework. This article describes key fi ndings from the literature on networks and their evaluation, and the experience and learning from network evaluation activi-ties conducted by SACYHN. These may inform evaluation efforts in other networks or similar inter-organizational initiatives.
Au cours de la dernière décennie, le Canada a connu l’établisse-ment d’environ 20 réseaux pour la santé des enfants et des jeunes. La valeur de tout réseau dépend de son effi cacité à atteindre les buts précisés; toutefois, la description et la mesure de l’effi cacité du réseau posent tout un défi étant donné les relations plurisec-torielles complexes et/ou pluridisciplinaires dont il est question et la pénurie relative de méthodes d’évaluation précises pour les réseaux. Depuis sa création en 2001, le Southern Alberta Child and Youth Health Network (SACYHN) a étudié la littérature sur l’évaluation des réseaux et a préparé un cadre d’évaluation et l’a mis en oeuvre. Cet article décrit les conclusions principales tirées des études sur les réseaux et leur évaluation ainsi que l’expé-rience et les leçons tirées des activités d’évaluation des réseaux réalisées par le SACYHN. Ces constatations pourront informer les efforts d’évaluation dans d’autres réseaux ou des initiatives inter-organisationnelles semblables.
Canadian communities face challenges in delivering health and related services to children and youth as a consequence of limited availability of specialty paediatric expertise, competition for scarce resources within largely adult-focused service systems, a need to provide access to specialized services over large geographic areas, multiple organizational mandates of different child-serving organizations and sectors, and the necessity of considering a child’s health and well-being within the family and environmental context. To address these challenges, organizations are using networks to develop shared visions, collaborative relationships, and joint initia-tives (Weiss, Anderson, & Lasker, 2002). A network is “a grouping of individuals, organizations, and agencies on a non-hierarchical basis around common issues or concerns, which are pursued proactively and systematically based on commitment and trust” (World Health Organization, 1998) and which include attachment to a common purpose (Chisholm, 1998).
In Canada, at least 20 known child and youth health networks have been established to attend to some or all of the following functions: provision/coordination/planning of services, policy development, pri-ority setting, research, communication and information sharing. These networks may focus exclusively on health care or include other
Résumé:
125LA REVUE CANADIENNE D’ÉVALUATION DE PROGRAMME
child-serving sectors such as education and social services, and vary in scope from local to regional to provincial (Southern Alberta Child and Youth Health Network, 2004). By promising to address complex child health issues through a voluntary multisectoral and multidisci-plinary approach with a horizontal rather than hierarchical structure, networks face daunting challenges in relation to evaluating their functioning and in demonstrating that they add value beyond basic organizational connection or communication. One such network cur-rently experiencing these challenges is described below.
Established in September 2001, the Southern Alberta Child and Youth Health Network (SACYHN) is a dynamic, voluntary, cross-sector collaboration that geographically spans four health regions across southern Alberta and is concerned with the health and well-being of children, youth, and families (see Figure 1). Its focus is on creating inter-sectoral and inter-regional linkages that will increase capac-ity to respond seamlessly to the needs of children and youth (Popp, Douglas-England, Casebeer, & Tough, 2005). SACYHN membership includes families (both parents and youth), government ministries, provincial organizations, regional authorities, First Nations, and not-for-profi t agencies, all from a variety of child-serving sectors such as health, education, justice, and children’s services. Representa-tives from these groups participate in various committees, working groups, and regional groups. While families participate in many ways, parents have specially designated representation on the SACYHN Steering Committee and youth participate primarily through the Child and Youth Advisory Council. A small number of SACYHN staff and regional network coordinators manage the key areas of SACYHN activity, such as outreach services and telehealth, a family and com-munity resource centre, primary care capacity-building projects, communications, and evaluation. Funding for SACYHN activities and initiatives comes from direct contributions of partners to core network functions, in-kind support, joint cost-sharing on specifi c initiatives, and proposals to external agencies.
Given the complex structure and context described above, how can SACYHN demonstrate that it is an effective mechanism for address-ing equally complex issues in delivering health and related services to children and youth? The purpose of this article is to describe key fi ndings from the literature on networks and their evaluation, and the experience and learning from network evaluation activities conducted by SACYHN. These may inform evaluation efforts in other networks or similar inter-organizational initiatives.
THE CANADIAN JOURNAL OF PROGRAM EVALUATION126
Stee
rin
gC
om
mit
tee
Reg
ion
alIn
ter-
sect
ora
lC
om
mit
tees
Wo
rkin
gG
rou
ps
SAC
YHN
Dir
ecto
rate
Net
wor
k C
oo
rdin
ato
rs
Ou
r Vis
ion
The
hea
lth
an
d w
ell-
bei
ng
of c
hild
ren
,yo
uth
an
d fa
mili
es is
su
pp
ort
ed b
y h
igh
qu
alit
y, c
oo
rdin
ated
pro
gra
ms,
serv
ices
and
info
rmat
ion
reso
urc
es t
hat
are
acce
ssib
le a
nd
as
clo
se to
ho
me
asp
oss
ible
.
Prov
ides
str
ateg
ic d
irec
tion
and
advo
cacy
for S
AC
YHN
Inte
r-se
ctor
al g
roup
s th
at id
enti
fy a
ndw
ork
on p
rior
itie
s w
ithi
n th
eir
geog
raph
ic re
gion
s in
kee
ping
wit
hSA
CYH
N v
isio
n an
d m
issi
on
Task
ori
ente
d gr
oups
that
und
erta
kesp
ecifi
c SA
CYH
N p
roje
cts/
stra
tegi
es(c
ross
-reg
iona
l and
/or c
ross
-sec
tora
l)
Figu
re 1
SACY
HN S
chem
atic
127LA REVUE CANADIENNE D’ÉVALUATION DE PROGRAMME
FINDINGS FROM THE LITERATURE ON NETWORKS AND THEIR EVALUATION
A literature review on networks undertaken by SACYHN (Hill, 2002) uncovered general characteristics of networks, specifi c characteristics that challenge network evaluation, and proposed evaluation frame-works. Examples of implementing evaluation frameworks were rare. An important conclusion of the literature review was that “network studies tend to focus on network composition and function rather than elements crucial to meeting network goals or developing the vision” (Hill, p. 43).
General Characteristics of Networks
Networks are identifi ed by their voluntary connections and relation-ships among systems, organizations, and individuals. Individuals or organizations enter into network relationships at their own desired level and intensity, allowing them to maintain their autonomy. A network can have members who are strongly tied together as well as others who are more loosely connected but still bound through a commitment to a core vision (Chisholm, 1996). All members contribute to the overall strength and effectiveness of the network. In general, the greater the number and variety of connections, the more likely the network will have the capacity to creatively tackle complex is-sues, explore innovations, and sustain itself over time (Provan & Milward, 2001).
Provan and Milward (2001) describe effective networks as having the following characteristics:
• multiple levels of involvement (within organizations, front-line staff to senior management; local to regional to provin-cial);
• focused integration (targeted and appropriate linkages among individuals, organizations, communities, and systems);
• tie strength (interdependence among key members across multiple types of involvement);
• network governance (in a form necessary to sustain a net-work);
• involvement (interaction based on trust and commitment to network goals);
• legitimacy (both within the network and in the larger com-munity or system);
THE CANADIAN JOURNAL OF PROGRAM EVALUATION128
• resources (suffi cient to build and sustain the network);• goals (specifi c, attainable, and appealing to members); and • stability (over time with no major or frequent system upheav-
als).
Chisholm (1996) identifi es four functions of effective networks: (a) creating and maintaining a vision that binds partner organizations together, (b) serving as a forum for dealing with complex issues, (c) identifying the importance of attitudes and perceptions for broad development, and (d) providing ways of communicating. Critical to these network functions are the underlying elements of mutual trust, willingness and desire to facilitate change, interest in gaining legitimacy among network agencies and within the community, and political power.
Networks are also described as learning organizations that are self-aware and self-regulating. To generate this self-knowledge, communi-cation and continuous evaluation are critical components of network functioning (Mays, Halverson, & Kaluzny, 1998).
Characteristics of Networks That Make Them Challenging to Evaluate
Studies evaluating the effectiveness of networks are few. This may be attributed to the nebulous nature of networks, perceptions of connectedness, role confusion, divergence in defi ning criteria for success, and the diffi culty in identifying measurable outcomes, as detailed below.
The nebulous nature of networks poses challenges for evaluation (Rose, 2004). As described above and in more detail in Hill (2002), networks are a complex arrangement of multisectoral/multi-agency and/or multidisciplinary relationships with shared processes and objectives. Provan and Milward (2001) articulate this as the “joint production problem,” where multiple agencies are responsible for one or more components of a single service creating blurred or indistinct organizational boundaries. This unifi ed delivery of services may sat-isfy clients, but presents challenges to the attribution of outcomes.
According to Provan, Veazie, Staten, and Teufel-Shone (2005), vari-ation in connectedness of network relationships is desirable. A mix of strong and weak ties improves effi ciency and allows for additional sources of information and ideas to be forthcoming. The value of strong ties is that members are signifi cantly and consistently invested
129LA REVUE CANADIENNE D’ÉVALUATION DE PROGRAMME
in multiple network initiatives, whereas the value of weak ties is that members bring a potentially broader or unique perspective. Though this mix is attractive, some network participants, particularly those who are weakly tied, may not self-identify as network members, even when recognized by the network as such. This incongruence in perceived member status is a challenge for network evaluation, as data collection and measurement can be impacted if those with weak ties do not see their relevance to the evaluation and decline to participate (Dolinski, 2005; Provan et al., 2005). The result is a lack of understanding of the perceptions and experiences of more weakly tied members, which in turn impedes achieving a complete under-standing of how the overall network is or is not effective.
Other impediments to the evaluation of networks are related to con-fusion about roles, and diverging views of success and health. First, differences in opinion amongst network leaders and members as to how the overall work done by networks should be defi ned, articu-lated, and implemented (Child and Youth Health Networks, 2003) can hamper assessment of effectiveness due to lack of agreement on what should be evaluated and how success should be defi ned (Provan et al., 2005). Second, networks and other collaborative partnerships often seek outcomes in multiple domains, and improvement in any given domain may be defi ned differently by individual participants, thus challenging the development of evaluation criteria (Provan et al., 2005; Taylor-Powell, Rossing, & Geran, 1998). Finally, traditional health outcome indicators are often medically defi ned and therefore clinical in nature. Many child and youth health networks subscribe to a broader defi nition of health (SACYHN, 2004). Consequently, appro-priate measurable outcome indicators that encompass this vision need to be identifi ed, potentially increasing the complexity of evaluation.
All of these factors that contribute to network complexity suggest the need for a correspondingly complex or systemic approach to network evaluation.
Proposed Evaluation Frameworks: Developing a Network Evaluation Matrix
The challenges to evaluating inter-organizational partnerships and collaborations are not new. However, our literature review uncovered only one framework (that of Provan & Milward, 2001) that attempted to evaluate a complex arrangement of partnerships and collaborations — that is, networks — with a comparably sophisticated multidimen-
THE CANADIAN JOURNAL OF PROGRAM EVALUATION130
sional approach. This multidimensional framework fi gures heavily into a broader systems approach to evaluation that identifi es levels of analysis and associated evaluation outcomes as culled from the literature. The resulting network evaluation matrix can be used to assess the effectiveness of network processes and activities from multiple stakeholder perspectives.
Networks operate on four levels (vision, structure, process, and service delivery) and thus network evaluation must pay attention to func-tioning at each level:
1. The overarching operational concern is vision (Mays et al., 1998), as it is the vision that binds the participants together (Chisholm, 1998). Therefore, networks need to develop a clear and well-articulated vision early in their formation.
2. The second level involves structure, or how the network is arranged (Bolland & Wilson, 1994; Gray & Wood, 1991; Mays et al., 1998). Structural functions include assembling, ad-ministering, evolving, regulating, maintaining, and funding the network, as well as planning activities. Issues associated with the structure include the need for an identifi ed form, network composition, consideration of stakeholders and end-users, resources, technical assistance, and e-health.
3. A third level is that of process, or how the network fulfi lls its vision (Gray & Wood, 1991; Mays et al., 1998). Proc-ess functions include confi guring, interacting, coordinating, learning, sensing, developing, deliberating, confl icting, com-municating, planning, collaborating, surveying, and evaluat-ing. Issues associated with process include trust, the need to develop and sustain relationships, time, managing confl ict, communication strategies, and the need for quick wins.
4. The fourth level of network operation is service delivery, which includes exchanging information, raising public aware-ness, case management, co-locating, establishing a program resource centre, and standardizing referral and intake proc-esses (Bolland & Wilson, 1994). Issues associated with service delivery include the need to plan regarding systems integra-tion, the need for joint funding, and the immense challenges of integrated service delivery.
In addition, networks have impacts at different levels and therefore must also be evaluated on each of these (community, network, and organization/participant) to capture adequately the perspectives of their multiple and diverse stakeholders (Provan & Milward, 2001):
131LA REVUE CANADIENNE D’ÉVALUATION DE PROGRAMME
1. The community level involves the impact of network activi-ties and processes on stakeholder groups such as client ad-vocacy groups, funders, regulators, and the general public. Outcomes associated with this level of analysis may include cost to community, building social capital, public perceptions that the problem is being solved, changes in incidence of the problem, and aggregate indicators of client well-being (Provan & Milward, 2001).
2. The network level is about whether stakeholder groups rep-resented by primary funders and regulators, network admin-istrative organizations (NAO), and member organizations believe that network processes and initiatives are successful. Outcomes may include network membership growth, range of services provided, absence of service duplication, relation-ship strength (or multiplexity), creation and maintenance of NAO, integration and coordination of services, cost of network maintenance, and member commitment to network goals (Provan & Milward, 2001).
3. Provan and Milward’s (2001) third level to consider, the organization/participant level, can be separated into the distinct levels of individual and organizational, an approach favoured by SACYHN. Both levels include the perspective of stakeholders such as the member agency board and manage-ment, agency staff, or individual clients. Outcomes associated with the individual level include service access, client level health outcomes, and staff outcomes, whereas organizational outcomes include agency survival, enhanced legitimacy, re-source acquisition, service costs, member satisfaction, refer-rals, and collaborative attitudes.
Therefore, a comprehensive assessment of network effectiveness should consider the level on which the network is operating and the varying levels on which the impact is felt, both of which provide the basis for our own evaluation matrix (see Table 1). Networks that seek, adhere to, and measure a well-articulated vision and forge trustful and committed relationships among network participants are more likely to operate at numerous levels and impact on multiple and diverse stakeholders (Hill, 2002).
While the literature on network extinction or failure is scant (Hill, 2002), what is there suggests that issues related to network disband-ment should be included in their monitoring and measurement. Ab-sence of a clear vision (Fawcett, Francisco, Paine-Andrews, & Schultz,
THE CANADIAN JOURNAL OF PROGRAM EVALUATION132Ta
ble
1Ne
twor
k Ev
alua
tion
Mat
rix w
ith S
ampl
e Ac
tiviti
es a
nd In
dica
tors
for S
ACYH
N
Le
vel o
f im
pact
and
sam
ple i
ndica
tors
Leve
l of n
etwor
k ope
ratio
n Co
mm
unity
Ne
twor
k Or
gani
zatio
nal
Indi
vidua
l
Visio
n
Stru
cture
Proc
ess
Serv
iceDe
liver
y
Sour
ce: A
dapt
ed fr
om H
ill (2
002)
.* I
ndica
tors
relat
ed to
netw
ork d
isban
dmen
t/exti
nctio
n.
• Sh
ared
und
ersta
ndin
g of
pu
rpos
e and
goa
ls
• Ar
rang
emen
t or c
ompo
sitio
n of
the n
etwor
k•
Activ
ities
inclu
de as
sem
-bl
ing,
fund
ing,
adm
inist
er-
ing,
regu
latin
g, et
c.
• W
hat t
he n
etwor
k doe
s to
fulfi
ll its
visio
n•
Activ
ities
inclu
de co
ordi
nat-
ing,
lear
ning
, dev
elopi
ng,
com
mun
icatin
g, et
c.
• Sh
ared
deli
verin
g of
serv
ices
and/
or fa
cilita
tion
of se
rvice
de
liver
y •
Activ
ities
inclu
de ca
se
man
agem
ent,
exch
angi
ng
info
rmati
on, r
eferra
l and
in
take,
etc.
• Po
licies
alig
n vis
ion
and
miss
ion
• Po
licy d
evelo
pmen
t an
d ad
optio
n
• Po
licy i
mpl
emen
tatio
n•
Com
mun
icatio
n m
ateria
ls
• Ex
posu
re•
Abse
nce/
redu
ction
of
serv
ice d
uplic
ation
• Af
fi liat
ion
Agre
emen
t sig
ned
• Pe
rcep
tions
that
netw
ork l
acks
us
efuln
ess*
• Co
hesio
n•
Cent
ralit
y•
Relat
ions
hip
stren
gth
(or m
ulti-
plex
ity)
• De
velo
pmen
t and
gro
wth
• Co
llabo
rativ
e atti
tude
s•
Cost
over
runs
*•
Stak
ehol
der t
urno
ver*
• St
ructu
re•
Delib
erati
on•
Lack
of t
rust*
• La
ck o
f com
mitm
ent*
• In
suffi
cient
tim
e to
build
relat
ion-
ship
s*
• Sh
arin
g in
form
ation
(e.g
., co
nfer
-en
ces,
news
letter
s, m
eetin
gs)
• In
abili
ty to
chan
ge se
rvice
*•
Inab
ility
to re
alloc
ate m
ainstr
eam
re
sour
ces*
• Se
rvice
s alig
n vis
ion
and
miss
ion
• Su
cces
sful j
oint
pro
gram
s, pr
ojec
ts•
Agre
emen
ts, p
roto
cols,
pr
ogra
m p
olici
es•
Shar
ed cl
ient b
ase
• Sh
ared
data
base
• Pa
rticip
ation
in n
etwor
k pr
ojec
ts
• Ap
prop
riate
refer
rals
• In
crea
sed
awar
enes
s•
Incr
ease
d kn
owled
ge
• Pa
rent
inpu
t (at
Stee
ring
Com
mitt
ee le
vel)
• Pa
rent
and
yout
h in
put (
at W
orkin
g Gr
oup
level)
• Co
llabo
rativ
e atti
tude
s
• En
hanc
ed kn
owled
ge o
f ch
ild h
ealth
• Sh
arin
g in
form
ation
with
pa
rent
s/ fa
mili
es•
Impr
oved
conn
ectio
ns
133LA REVUE CANADIENNE D’ÉVALUATION DE PROGRAMME
2000); changes in leadership or stakeholder representation (Fawcett et al., 2000; Fonner, 1998); perceptions that networks lack usefulness (Chisholm, 1998); early confl ict, lack of time to build relationships, and inability to effect changes in services (Goss, 2001); and discomfort with an ambiguous structure (Gilchrist, 1995) are some of the issues associated with network failure. Investigating these dimensions in network evaluation should help networks avoid major pitfalls in their ongoing development.
Overall, few studies reported in the literature evaluate networks using multiple levels of analysis (Hill, 2002), and the lack of data linking networks to outcomes (Lehman, Postrado, Roth, McNary, & Goldman, 1994; Provan & Milward, 2001) limits understanding of how networks may be mechanisms for addressing complex health issues (O’Toole, 1997).
EVALUATION EXPERIENCE OF SACYHN
SACYHN has undertaken or been involved in a number of evalua-tion activities that demonstrate how it has evolved its approach to assessing network effectiveness.
In its fi rst year, in addition to undertaking a network literature re-view, SACYHN participated in an external research project on change in health care delivery which involved more than 30 interviews with SACYHN stakeholders between September 2001 and December 2002 (Rose, 2004). This formative evaluation provided SACYHN with feed-back on its position in the community, how the vision and mission were perceived, and what issues would be important for sustainability. The results challenged SACYHN to consider how best to achieve its initiatives. That is, should networks prioritize building network infra-structure in an attempt to accomplish goals, or invest in the creation of network processes (e.g., building community capacity) so that the work can be done differently (and more effectively)? Rose (2004) also highlighted the need for network strategies on how to facilitate the diffi cult conversations that will arise about membership agreement and changes, outcome discrepancies, process discrepancies, and un-intended consequences of action, both positive and negative.
In 2003, to encourage a national dialogue about network models, evaluation strategies, and outcomes, SACYHN prepared and dis-tributed a National Network Survey through the Canadian Associa-tion of Paediatric Health Centres general membership mailing list
THE CANADIAN JOURNAL OF PROGRAM EVALUATION134
(SACYHN, 2004). Thirteen questionnaires were returned, represent-ing 12 unique child and youth health networks. Respondents most frequently reported that evaluation activities were conducted in relation to specifi c projects (53.8%) or focused on member satisfaction (46.2%). The survey also revealed that only three networks (23.1%) believed they had evaluation instruments that would be useful for other networks.
Development of the SACYHN Implementation Evaluation Framework
Coinciding with the formative evaluation and subsequent to the SACYHN literature review, SACYHN developed an Implementa-tion Evaluation Framework (Figure 2) to guide evaluation efforts. The framework identifi ed activity/process (i.e., output) and process and health outcome (i.e., success) indicators from the evaluation matrix derived from the literature. The indicators were articulated in reference to level of impact (individual, organizational, network, or community), thereby promoting multiple levels of analysis within evaluation of the network.
The Implementation Evaluation Framework was titled as such to ac-knowledge that evaluation activities of the network might vary de-pending on the developmental phase. For example, evaluation early in a network’s development might focus on engagement or the degree to which the vision is shared, a critical step in network development; whereas in a mature network, which the literature suggests can take several years to achieve (Dockery, 1996; Goss, 2001), the concrete bene-fi ts of the network should be clearer and one might expect evaluation to identify actual outcomes or the degree to which the network has met its goals (Hill, 2002). Thus, evaluation activities appropriate to a network in its early development may not fi t for a mature network. The point that evaluation must account for evolution has also been made in literature on collaborative partnerships (Taylor-Powell et al., 1998).
Of note, the Implementation Evaluation Framework also considered the relational and technical quality of the SACYHN vision (an opera-tional level of analysis), and acknowledged the linkages and collabora-tions required to achieve that vision. SACYHN’s vision is: “The health and well being of children, youth and families is supported by high quality, coordinated programs, services and information resources that are accessible and as close to home as possible.” The framework defi ned the following four critical elements to capture the relational quality of SACYHN’s vision:
135LA REVUE CANADIENNE D’ÉVALUATION DE PROGRAMME
Figu
re 2
SACY
HN Im
plem
enta
tion
Eval
uatio
n Fr
amew
ork
(pre
sent
ed in
par
t: Ne
twor
k go
als
1 &
2 o
nly)
Mis
sio
n:
The
Net
wo
rk w
ill u
se it
s co
llect
ive
stre
ng
th &
exp
erti
se t
o b
uild
& s
ust
ain
lin
kag
es t
hat
op
tim
ize
the
hea
lth
& w
ell b
ein
g o
f all
child
ren
, yo
uth
& fa
mili
es.
SAC
YH
N V
isio
nTh
e h
ealt
h a
nd
wel
l-b
ein
go
f ch
ildre
n, y
ou
th &
fam
ilies
is s
up
po
rted
by:
Co
mp
on
ents
/Dim
ensi
on
sN
etw
ork
Go
als
Year
s 1
to
3
Hig
h qu
alit
y p
rog
ram
s,se
rvic
es &
info
rmat
ion
re
sou
rces
Coo
rdin
ated
pro
gra
ms,
serv
ices
& in
form
atio
n
reso
urc
es
Acc
essi
ble
pro
gra
ms,
serv
ices
& in
form
atio
nre
sou
rces
Rela
tio
nal
Qu
alit
y:
- Ch
ild-f
ocu
sed
& fa
mily
-cen
tred
(fam
ilies
as
par
tner
s, sh
arin
g in
form
atio
n,
stre
ng
ths
bas
ed, r
esp
ectf
ul,
ho
no
ur d
iver
sity
, rec
og
niz
e d
evel
op
men
tal
nee
ds,
flexi
ble
ser
vice
s, cu
ltu
rally
sen
siti
ve, r
esp
on
sive
)
- Rec
og
nit
ion
of u
niq
ue
hea
lth
nee
ds
of A
bo
rig
inal
fam
ilies
- S
up
po
rt &
val
ue
all s
ervi
ce p
rovi
der
s
- Fo
cus
on
bro
ader
det
erm
inan
ts o
f hea
lth
Tech
nic
al Q
ual
ity:
- S
kille
d s
ervi
ce p
rovi
der
s
- Co
nsi
sten
t st
and
ard
s o
f car
e
- Evi
den
ce b
ased
car
e
- Co
nti
nu
ou
s im
pro
vem
ent/
on
go
ing
eva
luat
ion
Lin
kag
es/C
olla
bo
rati
on
(pla
nn
ing,
pri
ori
ty s
etti
ng,
ser
vice
del
iver
y):
- B
etw
een
pro
gra
ms/
serv
ices
- B
etw
een
ag
enci
es
- Bet
wee
n s
ecto
rs
- Bet
wee
n re
gio
ns
- B
etw
een
pro
vid
ers
- B
etw
een
fam
ilies
- B
etw
een
fam
ilies
& p
rovi
der
s
- In
-per
son
&/o
r ele
ctro
nic
- M
ult
iple
mo
des
of a
cces
s (e
.g.,
lan
gu
ages
, tel
ehea
lth
)
- Mu
ltip
le p
oin
ts o
f acc
ess
- M
ult
iple
refe
rral
pat
hs
(e.g
., se
lf, p
rovi
der
)
- Clo
ser t
o h
om
e
- Aff
ord
able
- A
deq
uat
e co
mm
un
ity
cap
acit
y
1. D
evel
op
& a
do
pt
a sh
ared
vis
ion
& m
issi
on
2. Id
enti
fy &
cre
ate
op
po
rtu
nit
ies
for p
osi
tive
c
han
ge
in s
ervi
ce d
eliv
ery
3. In
volv
e fa
mili
es &
ser
vice
pro
vid
ers
in
s
ign
ifica
nt
way
s in
ser
vice
pla
nn
ing
&
del
iver
y
4. B
uild
& s
ust
ain
inte
r-re
gio
nal
, cro
ss-
s
ecto
ral &
clin
ical
co
nn
ecti
on
s
5. D
evel
op
or i
mp
rove
ele
ctro
nic
lin
kag
es
THE CANADIAN JOURNAL OF PROGRAM EVALUATION136Fi
gure
2SA
CYHN
Impl
emen
tatio
n Ev
alua
tion
Fram
ewor
k (p
rese
nted
in p
art:
Netw
ork
goal
s 1
& 2
onl
y) —
con
tinue
d
Net
wo
rk
Go
al #
1M
eth
od
So
urc
e o
f In
form
atio
n
Act
ivit
y/P
roce
ss(O
utp
ut)
Ind
icat
ors
Pro
cess
& H
ealt
h O
utc
om
e
In
dic
ato
rs (S
ucc
ess)
Dev
elo
p &
ado
pt
ash
ared
vis
ion
& p
urp
ose
- Ag
reed
up
on
su
stai
nab
ility
pla
n
(lon
g te
rm fi
nan
cial
su
pp
ort
&
ov
ersi
gh
t)- S
har
ed re
sou
rcin
g /
will
ing
nes
s to
re
sou
rce
wo
rk o
f th
e N
etw
ork
- Su
stai
ned
mem
ber
ship
(lo
w t
urn
over
rate
)- I
ncr
easi
ng
# o
f mem
ber
s ov
er t
ime
- In
crea
sin
g #
of s
ecto
rs in
volv
ed- I
ncr
ease
d in
ter-
reg
ion
al c
olla
bo
rati
on
- In
crea
sed
cro
ss-s
ecto
ral c
olla
bo
rati
on
- Par
tici
pan
t sa
tisf
acti
on
(85%
tar
get
)
(Net
wor
k &
org
aniz
atio
nal l
evel
out
com
es)
- In
crea
sed
co
mm
un
ity
& a
gen
cy
u
nd
erst
and
ing
of t
he
Net
wor
k’s
visi
on
& m
issi
on
- Co
nsi
sten
cy o
f mes
sag
es a
bo
ut
the
N
etw
ork
- In
crea
sed
cro
ss-s
ecto
ral r
esp
on
sib
ility
for c
hild
& y
ou
th h
ealt
h- I
ncr
ease
d p
arti
cip
atio
n in
Net
wor
k
wo
rkin
g g
rou
ps,
com
mit
tees
& p
roje
cts
- In
crea
sed
lin
kag
es w
ith
pro
vin
cial
& n
atio
nal
init
iati
ves
- In
crea
sed
lin
kag
es w
ith
oth
er c
hild
&
yo
uth
hea
lth
net
wo
rks
(Net
wor
k &
com
mun
ity
leve
l out
com
es)
- # o
f Sig
nat
ori
es to
Aff
iliat
ion
Ag
reem
ent
- Net
wor
k ac
tio
n p
lan
dev
elo
ped
- # o
f reg
ion
al N
etw
ork
com
mit
tees
form
ed- R
egio
nal
act
ion
pla
ns
dev
elo
ped
- # o
f Net
wor
k C
oo
rdin
ato
rs a
pp
oin
ted
(by
reg
ion
& s
ecto
r)- N
etw
ork
invo
lvem
ent
in re
sear
ch o
n
o
rgan
izat
ion
al c
han
ge
in t
he
hea
lth
care
sec
tor (
Dr.
Tere
sa R
ose
)
- Co
mm
un
icat
ion
pla
n d
evel
op
men
t &
im
ple
men
tati
on
- # o
f pre
sen
tati
on
s o
r mee
tin
gs
wit
h
ke
y st
akeh
old
ers
& b
road
co
mm
un
ity
- Bro
chu
re /
new
slet
ter d
evel
op
men
t- P
ost
er p
rese
nta
tio
n m
ater
ials
- Net
wor
k w
ebsi
te- #
of a
rtic
les
or n
ewsl
ette
r ite
ms
p
ub
lish
ed a
bo
ut
the
Net
wor
k- C
om
ple
tio
n o
f lit
erat
ure
revi
ew o
n
n
etw
ork
s- P
ub
licat
ion
of a
rtic
les
fro
m li
tera
ture
revi
ew- P
rovi
nci
al &
nat
ion
al li
nka
ges
esta
blis
hed
- Ste
erin
g C
om
mit
tee
- Reg
ion
al N
etw
ork
C
om
mit
tees
- Net
wor
k C
oo
rdin
ato
rs- W
ork
ing
Gro
up
s
- Co
mm
itte
e
par
tici
pat
ion
- Co
mm
un
icat
ion
- Co
mm
un
icat
ion
W
ork
ing
Gro
up
- Eva
luat
ion
Wo
rkin
g
Gro
up
- Ste
erin
g C
om
mit
tee
137LA REVUE CANADIENNE D’ÉVALUATION DE PROGRAMME
Figu
re 2
SACY
HN Im
plem
enta
tion
Eval
uatio
n Fr
amew
ork
(pre
sent
ed in
par
t: Ne
twor
k go
als
1 &
2 o
nly)
— c
ontin
ued
Net
wo
rk
Go
al #
2M
eth
od
So
urc
e o
f In
form
atio
n
Act
ivit
y/P
roce
ss(O
utp
ut)
Ind
icat
ors
Pro
cess
& H
ealt
h O
utc
om
e
In
dic
ato
rs (S
ucc
ess)
Iden
tify
& c
reat
eo
pp
ort
un
itie
s fo
rp
osi
tive
ch
ang
ein
ser
vice
del
iver
y
- Im
pro
ved
acc
essi
bili
ty o
f car
e (c
lose
r
to h
om
e; le
ss t
rave
l, co
st &
fam
ily
st
ress
)- 1
00%
of r
egio
ns
& s
ervi
ces
usi
ng
Ou
trea
ch S
ervi
ces
Fram
ewo
rk- I
ncr
ease
d c
linic
al c
on
nec
tio
ns
acro
ss
re
gio
ns
- In
crea
sed
co
nti
nu
ity
of c
are
- In
crea
sin
g #
of f
orm
al N
etw
ork
m
emb
ers
over
tim
e- I
ncr
ease
d in
ter-
reg
ion
al c
olla
bo
rati
on
- In
crea
sed
cro
ss-s
ecto
ral c
olla
bo
rati
on
- Im
pro
ved
un
der
stan
din
g o
f bu
sin
ess
p
roce
sses
& a
ctiv
itie
s ac
ross
sec
tors
- In
crea
sed
alig
nm
ent
of p
olic
ies
ac
ross
reg
ion
s &
sec
tors
- Do
cum
ente
d p
ract
ice
chan
ges
(Net
wor
k &
org
aniz
atio
nal l
evel
out
com
es)
- Su
cces
sfu
l res
ou
rcin
g o
f pro
ject
s- I
mp
lem
enta
tio
n o
f pro
ject
s- N
ew p
artn
ersh
ips
dev
elo
ped
- En
chan
ced
pro
fess
ion
al c
apac
ity
(Net
wor
k &
org
aniz
atio
nal l
evel
out
com
es)
- Dev
elo
pm
ent
& a
do
pti
on
of O
utr
each
Se
rvic
es F
ram
ewo
rk- #
of c
linic
al a
reas
usi
ng
Ou
trea
ch
Se
rvic
es F
ram
ewo
rk- #
of r
egio
ns
usi
ng
Ou
trea
ch S
ervi
ces
Fr
amew
ork
- Tel
ehea
lth
lau
nch
- # o
f org
aniz
atio
ns
sig
nin
g t
he
A
ffili
atio
n A
gre
emen
t- P
lan
nin
g &
pri
ori
ty s
etti
ng
acr
oss
sect
ors
wit
hin
reg
ion
s
- Id
enti
ficat
ion
of r
egio
nal
& s
ecto
r
pri
ori
ties
for a
ctio
n- I
den
tific
atio
n o
f nee
ds,
gap
s &
edu
cati
on
al p
rio
riti
es- N
etw
ork
pro
ject
dev
elo
pm
ent
(s
ervi
ce/p
rog
ram
, lo
cal/
reg
ion
al,
b
road
Net
wor
k le
vel,
cro
ss-s
ecto
ral)
- Ou
trea
ch S
ervi
ces
Fr
amew
ork
- Tel
ehea
lth
init
iati
ve- A
ffili
atio
n A
gre
emen
t- R
egio
nal
Net
wor
k
Co
mm
itte
es
- Po
licy
d
evel
op
men
t
- Co
mm
itte
e
par
tici
pat
ion
- Ste
erin
g C
om
mit
tee
- Reg
ion
al N
etw
ork
C
om
mit
tees
- Sta
keh
old
er m
eeti
ng
s
(CEO
s, B
oar
ds,
co
mm
un
ity,
etc
).
*Net
wor
k go
als
#3,
4 &
5 a
ll co
ntri
bute
to th
is g
oal (
i.e.,
invo
lve
fam
ilies
,im
prov
e lin
kage
s)
THE CANADIAN JOURNAL OF PROGRAM EVALUATION138
1. Be child-focused and family-centred (i.e., family as partners, share information, strengths-based, respectful, honour di-versity, recognize developmental needs, fl exible services, culturally sensitive, and responsive);
2. Recognize the unique health needs of Aboriginal families;3. Support and value all service providers; and4. Focus on broader determinants of health.
Additionally, the Implementation Evaluation Framework was de-signed in relation to the fi ve goals of SACYHN:
1. Develop a shared vision and purpose;2. Identify and create opportunities for positive change in serv-
ice delivery;3. Involve families and service providers in signifi cant ways in
service planning and delivery;4. Build and sustain inter-regional, cross-sector, and clinical
connections; and5. Develop or improve electronic linkages.
The framework identifi ed both activity/process and health outcome indicators for each goal, and articulated how each outcome could be achieved through SACYHN’s committees and working groups. It also outlined valid and reliable data sources for each goal-specifi c outcome in order to facilitate meaningful interpretation of results.
Applying the Implementation Evaluation Framework
In the fall of 2004, with the framework established, SACYHN undertook an evaluation of the network in regard to its fi rst two goals, namely:
1. Develop and adopt a shared vision and purpose, and 2. Identify and create opportunities for positive change in serv-
ice delivery.
Six steps were involved in the evaluation:
1. Document review: A standard matrix was developed to record information related to each indicator for each of the two goals. This matrix was systematically applied to all docu-ments, commencing with the minutes of the SACYHN Steer-ing Committee meetings from inception to December 2004.
139LA REVUE CANADIENNE D’ÉVALUATION DE PROGRAMME
The document review also targeted SACYHN’s Background and Communications and Working Group materials.
2. Member survey: A questionnaire was developed, pretested, and revised. Content included current SACYHN involve-ment, individual and organizational impact, and benefi ts to service delivery; satisfaction with involvement; and thoughts about SACYHN’s future. The questionnaire was circulated to all individuals the SACYHN staff considered members, regardless of role or tie strength.
3. Semi-structured interviews with key stakeholders: Eight key informants were interviewed to provide context to the docu-ment review and survey fi ndings, and to validate preliminary conclusions. Key informants included parent participants; representatives from multiple child-serving sectors, the net-work governance structure, and network working groups; and network staff.
4. Child and Youth Advisory Council focus group: The ques-tions for SACYHN’s youth participants were parallel to the member survey but more tailored and administered in a focus group format. Members of the Council ranged in age from 12 to 19 years and length of participation in the Council ranged from one month to over one year.
5. SACYHN Director input: Interviews with the Director of SACYHN were designed to again provide additional context and validation as well as to gain a unique perspective on the network and its development.
6. Investigator analysis: Results from all sources were cross-referenced, themes and congruencies related to the two SACYHN goals were identifi ed, and a fi nal report was pre-pared which included recommendations for future considera-tion or action.
Substantive Findings about SACYHN That Provide Lessons about Network Characteristics and Evaluation
Overall, the evaluation fi ndings indicated positive outcomes related to the orientation of SACYHN’s structure, processes, and initiatives with respect to its fi rst two goals. The evaluation report included recommendations for future consideration that could have a positive impact on SACYHN realizing its vision of providing high quality, coor-dinated care as close to home as possible, of building and maintaining professional linkages, and of increasing capacity across regions. Full results from the SACYHN evaluation have been reported elsewhere
THE CANADIAN JOURNAL OF PROGRAM EVALUATION140
(Dolinski, 2005); only results that illustrate issues related to network characteristics and evaluation are reported here. Comparisons are also made with the formative evaluation completed by Rose (2004).
The ability of evaluation to inform network practices is illustrated by changes in understanding of commitment by SACYHN members. Initially, the formative evaluation by Rose (2004) identifi ed diffi culties amongst members in achieving consensus on the level of formality appropriate for SACYHN membership. However, the subsequent evaluation found that consensus had been achieved, with an associ-ated increase in formal membership in 2004. This positive fi nding supports the utility of evaluation in directing the efforts of networks, ultimately contributing to the probability of sustainability given that high turnover in stakeholder representation can lead to network extinction (Fawcett et al., 2000; Fonner, 1998).
The nebulous nature of networks was previously discussed as a chal-lenge in assessing network effectiveness. In the formative evaluation, Rose (2004) questioned whether the goal of SACYHN was to create “an organization or centre of excellence, or a system of excellence brought about by individuals and groups interacting in new ways?” (p. 20). The recent evaluation found that SACYHN must encourage its membership to appreciate that there are multiple ways to instill or operationalize the SACYHN vision in their community, and there-fore generate awareness about the network and build community capacity.
The recent evaluation also found that some individual members re-main uncomfortable with the “nebulosity” of SACYHN. However, use of the word “nebulous” does not imply that SACYHN has no structure or decision-making process. Rather, the nebulosity of SACYHN is represented by the confl ict or incongruence between the network’s desire to support more fl uid processes and individual expectations and experience of participation. From a network vision perspective, the drive to eliminate physical and psychological rigidity, in order to create an environment where ideas and information can circu-late freely, may be easily adopted by members. However, when this requires personal or organizational change or fl exibility (that is, challenges to traditional ways of working), members may experience considerable discomfort and the tendency can be to revert to creating structure to manage the anxiety. Many members remain confl icted between wanting/needing more structure, and wanting to be open to participating in what was described by an interviewee as “having
141LA REVUE CANADIENNE D’ÉVALUATION DE PROGRAMME
the conversation.” SACYHN’s literature review on networks identi-fi ed the “tyranny of structurelessness” (Gilchrist, 1995) as an issue that may lead to network disbandment. Therefore, SACYHN needs to continue to provide enough structure to offset member anxiety while still challenging traditional organizational frameworks in its pursuit to effect change.
A further question posed by Rose (2004) in the formative evaluation was whether the structure of SACYHN could sustain more work and, if so, what kinds of work would meet expectations for effectiveness/success in the long term? This question is directly related to the citation of role confusion as a challenge to assessing network effectiveness. However, too much activity can also be a pitfall for networks if mem-bers fall into a trap of engaging in activities that “give the illusion of accomplishment but are not appropriate for the network” (Chisholm, 1998, p. 194). As SACYHN currently identifi es priority areas of interest and action using stringent criteria, the potential for what Chisholm describes is signifi cantly reduced. Still, the recent evalua-tion detected some ambivalence around what work was being done, and what work should be done. Thirty-eight percent of questionnaire respondents felt that over the next fi ve years SACYHN should review how it decides upon its areas of activities.
Furthermore, some SACYHN respondents/informants wanted to focus on achieving more tangible outcomes, and in realizing these outcomes felt a need for them to be directly attributable to the network itself. In SACYHN, as well as in other collaborative partnerships, there is a need to link network successes back to participating organizations in order to demonstrate the value of the network to its participants (Taylor-Powell et al., 1998). Given the nature of networks described above, this direct attribution of outcomes will be extremely diffi cult to realize. Numerous members questioned whether it is even appro-priate for concrete outcomes to be linked back to SACYHN, as this is incongruent with the rationale and ethos of networks.
Connectedness was also cited as a challenge to the assessment of network effectiveness. Rose (2004) stated that “Networking (building relationships) requires a signifi cant amount of time and other invest-ment at the same time that individuals and organizations are feeling a squeeze on their time” (p. 20). Despite this sentiment, the majority of respondents acknowledged and appreciated the fact that trust bonds, and therefore connectedness, were developing well amongst SACYHN participants. It was critically important that the effective-
THE CANADIAN JOURNAL OF PROGRAM EVALUATION142
ness criteria in the Implementation Evaluation Framework enabled this fi nding, given that a lack of trust and commitment between and by network members has been attributed to network extinction (Child & Faulkner, 1998).
Lessons About Tools and Methods for Network Evaluation
The discussion below pinpoints a number of additional challenges and lessons related to the use of evaluation tools and methods in as-sessing network effectiveness, many of which can again be directly linked to the network literature.
Combining qualitative and quantitative data collection allowed for improved understanding of network impact by providing an in-depth and contextual description of how SACYHN was positioned both with its member base, and in the community. Goal 1 was measured against effectiveness criteria that encompassed “participant satisfaction” to “increased linkages with other child and youth health networks.” Goal 2 was measured against a broad range of effectiveness criteria that were not simply activity-focused. That is, as SACYHN (and many child and youth health networks in Canada) subscribe to a broad defi nition of service delivery, a purely quantitative investiga-tion of this goal would not have captured the entirety of initiatives. This fi nding is in keeping with recommendations in the literature for strengthening evaluation of collaborative partnerships through use of multiple designs (Taylor-Powell et al., 1998).
Generalized network member surveys should probably not be the dominant form of data collection. In our case, the use of a general e-mail member survey was not the most effective method. A response rate of 28% to the survey left questions about the extent to which the fi ndings were representative of all members. Anecdotal comments obtained from non-responders suggest that the questionnaire was too detailed in relation to the level of connection for many, and that some participants would have preferred to complete the questionnaire during scheduled committee meetings. Response rates in similar circumstances have ranged from 17% (Child Health Network for the Greater Toronto Area [CHN], 2003) to 48% (Child and Youth Health Network for Eastern Ontario [CHEO], 2000).
Despite its limitations, the SACYHN member survey did reveal es-sential information about member connectedness. Some members who completed the survey viewed themselves as strongly connected, while
143LA REVUE CANADIENNE D’ÉVALUATION DE PROGRAMME
others did not see themselves as formally connected. These fi ndings were of value in determining the extent of fragmentation amongst members, and provided insight into the distribution of strong versus weak ties throughout the network. Such information is critical for network development, as weak ties, once identifi ed, are generally easy to strengthen with targeted attention (Provan et al., 2005).
Differences in opinion as to the role of a network also arose as a challenge in SACYHN’s evaluation. Specifi cally, responses on the member survey as to what criteria should be used to assess network effectiveness may have differed depending on the respondent’s view of the primary role of SACYHN; yet the survey did not enable re-spondents to clearly articulate just what they consider the work of SACYHN to be. For example, the majority (in line with SACYHN’s vision) indicated an effectiveness criterion was “accessibility of child and youth health services” (87.5%), whereas for some it was their organization realizing “cost effectiveness” from participation (39%). It can be assumed that these differences would infl uence how par-ticipants responded to other questions related to how well SACYHN functions (Dolinski, 2005).
If surveys are used, some preliminary work on individual perceptions of membership ought to be done, as described in CHEO (2000), par-ticularly given the diversity of members and their varying connected-ness and ideas about network roles. Furthermore, network member surveys should be customized to each type of member role and perhaps to level of commitment or activity. As well, consideration should be given to using natural opportunities (i.e., scheduled meetings) to al-low completion of the survey. Another option for networks might be to consider using a social network analysis tool early on, since that method is designed to communicate to stakeholders (visually) where the relationships are, where they are not, where connections can be leveraged, and so on (Hawe, Webster, & Shiell, 2004; Provan, Veazie, Teufel-Shone, & Huddleston, 2004). This might afford later compari-sons with the increasing multiplexity, or connection of participants in more than one way, of the network over time as well as capture changing perspectives of participants.
The number of indicators should be limited, or better defi ned between goals, so that results can be accurately attributed when reporting against them. One of the SACYHN evaluation recommendations in-cluded review and revision of the indicators contained in the Imple-mentation Evaluation Framework (Dolinski, 2005). In an attempt to
THE CANADIAN JOURNAL OF PROGRAM EVALUATION144
be comprehensive, the framework identifi ed multiple indicators at varying levels, some of which were duplicated, ultimately making it diffi cult to operationalize the evaluation. Furthermore, to reduce both investigator and participant burden, it may be prudent to apply a time frame for measurement to each indicator. This would enable an im-proved focus on new initiatives as they develop, and provide a clearer perspective on network development and growth at any given time.
Triangulation of multiple sources of qualitative data assists with understanding network effectiveness. Completion of a SACYHN docu-ment review provided a perspective on how SACYHN was positioned. Qualitative data gleaned from the participant survey, semi-structured interviews, and focus groups added context to the data from the docu-ment matrix. The qualitative data clarifi ed what “high quality” means to SACYHN participants when one considers programs, services, and information resources. The recent evaluation of SACYHN’s fi rst two goals provided valuable information regarding actual commitment to vision and purpose as well as the impact on service delivery.
CONCLUSION AND FUTURE DIRECTIONS
Considerations for the Next Phase of SACYHN Evaluation
There were several challenges to applying what may be described as program evaluation methods and tools to evaluating our network. However, much was learned and SACYHN is now well positioned to plan and develop innovative evaluation tools and methodologies for the next stage of evaluation in order to capture the effectiveness of the complex inter-organizational relationships involved. Developing these innovative methods may be made easier by focusing on what now needs to be measured as an indicator of network effectiveness. That is, does SACYHN, now four years old, need to move away from evolutionary dynamics to focus on the impact on health outcomes? Or is it still too early to be measuring such outcomes, given that much of what SACYHN does is related to and designed to impact (but one step removed from) the service delivery that will encourage an improvement in, or degradation of, health outcomes? That is, what can be properly attributed to SACYHN initiatives and coordination? Discussion of these issues will assist SACYHN in developing goals that address the health outcome component of its mission.
With respect to methods, SACYHN now needs to consider whether it will narrow the scope of evaluation to focus less on the entire member
145LA REVUE CANADIENNE D’ÉVALUATION DE PROGRAMME
base, and more on recording and analyzing the development and out-comes related to particular initiatives, or to subgroups of participants. The focus will determine whether other approaches such as case studies, participatory action research, and/or social network analysis might provide valuable information about network effectiveness.
Considerations for Future Research on Evaluation of Networks
Evaluation is an issue of concern to child and youth health networks in Canada and, given the proliferation of networks, there is an ur-gent need to better understand network effectiveness, determine the indicators of a healthy or successful network, and develop net-work-specifi c evaluation methods (SACYHN, 2004). In the absence of thorough evaluation of network practices the progress of networks will be very diffi cult to measure, affecting the credibility of the net-work movement. SACYHN’s literature review (Hill, 2002) on network concepts and evaluation identifi ed areas for further research related to evaluating network effectiveness, including a need for:
• evaluations regarding the relationship between inter-organi-zational network structures and network effectiveness;
• evaluations that consider several levels of network opera-tional and impact analysis;
• evidence of impact at the community level;• evaluation of a network’s internal organizations;• consideration of both external and internal impacts of net-
works in evaluation; and • more description of networks as distinct from less complex
partnerships or community collaborative initiatives.
There is current deliberation over the potential to develop a multi-site network evaluation strategy across Canada that would provide child and youth health networks with common indicators of effec-tiveness. Individual networks are supportive of this direction, with 66.7% of respondents in the National Network Survey (SACYHN, 2004) indicating that it is very important to study network effective-ness. Furthermore, as evaluation mechanisms are rarely funded and developing outcome indicators for networks is a diffi cult process (Provan & Milward, 2001), there is value in working together so that all networks benefi t.
THE CANADIAN JOURNAL OF PROGRAM EVALUATION146
SUMMARY
This article presents key fi ndings from the literature relevant to network concepts and evaluation, and highlights learning from the evaluation activities of the Southern Alberta Child and Youth Health Network. Key lessons for evaluators examining network effectiveness include: consider a matrix approach that includes network operational and impact levels of analysis in evaluation; be thoughtful about the issue of membership connectedness when assessing effectiveness; carefully consider how well program evaluation methods and tools fi t in the more complex network context; and consider what outcomes can realistically and appropriately be tied directly to a network.
REFERENCES
Bolland, J.M., & Wilson, J.V. (1994). Three faces of integrative coordina-tion: A model of inter-organizational relations in health and human services. Health Services Research, 29, 341–366.
Child and Youth Health Network for Eastern Ontario (CHEO). (2000). Final report on evaluation from the Evaluation Task Group to the Network Steering Committee. Ottawa: Author.
Child and Youth Health Networks. (2003, June). Think tank on a national child and youth health network study: A dialogue on a scope and strategy for a national multi-site evaluation framework. Proceedings from a satellite event of the 2003 Canadian Association of Paediatric Health Centres (CAPHC) Conference, Calgary, AB.
Child Health Network for the Greater Toronto Area (CHN). (2003). Taking a fi rst step: Evaluating system performance for mothers, newborns, children and youth. Toronto: Author.
Child, J., & Faulkner, D. (1998). Strategies of cooperation: Managing al-liances, networks and joint ventures. New York: Oxford University Press.
Chisholm, R.F. (1996). On the meaning of networks. Group and Organiza-tion Management, 21, 216–236.
Chisholm, R.F. (1998). Developing network organizations: Learning from practice and theory. Reading, MA: Addison Wesley Longman.
147LA REVUE CANADIENNE D’ÉVALUATION DE PROGRAMME
Dockery, G. (1996). Rhetoric or reality? Participatory research in the NHS. In K. deKoning & M. Martin (Eds.), Participatory research in health: Issues and experiences (pp. 164–176). London: Zed Books.
Dolinski, C. (2005). Southern Alberta Child and Youth Health Network: Evaluation report. Calgary, AB: Southern Alberta Child and Youth Health Network.
Fawcett, S.B., Francisco, V.T., Paine-Andrews, A., & Schultz, J.A. (2000). A model memorandum of collaboration: A proposal. Public Health Reports, 115(2–3), 174–179.
Fonner, E., Jr. (1998). Relationship management: Assessing coalitions, networks, and other workgroups. Administrative Radiology Journal, 17(6), 18–21.
Gilchrist, A. (1995). Community development and networking. London: Com-munity Development Foundation.
Goss, S. (2001). Making local governance work: Networks, relationships and management of change. New York: Palgrave.
Gray, B., & Wood, D.J. (1991). Collaborative alliances: Moving from practice to theory. Journal of Applied Behavioral Science, 27, 3–22.
Hawe, P., Webster, C., & Shiell, A. (2004). A glossary of terms for navigat-ing the fi eld of social network analysis. Journal of Epidemiology and Community Health, 58, 971–975.
Hill, C. (2002). Network literature review: Conceptualizing and evaluating networks. Calgary, AB: Southern Alberta Child and Youth Health Network.
Lehman, A.F., Postrado, L.T., Roth, D., McNary, S.W., & Goldman, H.H. (1994). Continuity of care and client outcomes in the Robert Wood Johnson Foundation program on chronic mental illness. Milbank Quarterly, 72, 105–122.
Mays, G.P., Halverson, P.K., & Kaluzny, A.D. (1998). Collaboration to improve community health: Trends and alternative models. Joint Commission on Quality Improvement, 24(10), 116–123.
THE CANADIAN JOURNAL OF PROGRAM EVALUATION148
O’Toole, L.J. (1997). Treating networks seriously: Practical and research-based agendas in public administration. Public Administration Re-view, 57, 45–52.
Popp, J., Douglas-England, K., Casebeer, A., & Tough, S. (2005). Creating frameworks for providing services closer to home in the context of a network. Healthcare Management FORUM, 18(2), 27–33.
Provan, K.G., & Milward, B. (2001). Do networks really work? A framework for evaluating public-sector organizational networks. Public Admin-istration Review, 61, 414–424.
Provan, K.G., Veazie, M.A., Staten, L.K., & Teufel-Shone, N.I. (2005.) The use of network analysis for strengthening community partnerships in health and human services. Public Administration Review, 65, 603–613.
Provan, K.G., Veazie, M.A., Teufel-Shone, N.I., & Huddleston, C. (2004). Network analysis as a tool for assessing and building community capacity for provision of chronic disease services. Health Promotion Practice, 5, 174–181.
Rose, T. (2004). Research project report: Evolution, development, and main-tenance of the Southern Alberta Child and Youth Health Network. Calgary, AB: Southern Alberta Child and Youth Health Network.
Southern Alberta Child and Youth Health Network (SACYHN). (2004). Child and youth health networks: A national survey. Calgary, AB: Author.
Taylor-Powell, E., Rossing, B., & Geran, J. (1998). Evaluating collaboratives: Reaching the potential (Document G3658-8). Madison, WI: University of Wisconsin-Extension, Cooperative Extension.
Weiss, E.S., Anderson, R.M., & Lasker, R.D. (2002). Making the most of col-laboration: Exploring the relationship between partnership synergy and partnership functioning. Health Education and Behavior, 29, 683–698.
World Health Organization (WHO). (1998). Health promotion glossary. Di-vision of Health Promotion, Education and Communications (HPR), Health Education and Health Promotion Unit (HEP). (Document WHO/HPR/HEP/98.1). Geneva: Author.
149LA REVUE CANADIENNE D’ÉVALUATION DE PROGRAMME
Janice K. Popp, M.S.W., R.S.W., is a clinical social worker and Director of the Southern Alberta Child and Youth Health Network (SACYHN), an inter-regional, cross-sector health network for children and youth. Since SACYHN’s inception, Janice has had responsibility for the ongoing development and evaluation of the network and has a keen interest in connecting network theory to practice. She also has administrative responsibility for family-centred care, the Fam-ily and Community Resource Centre, and Palliative Care and Grief Support Services within the Child and Women’s Health portfolio in the Calgary Health Region. Previously, Janice worked in the area of children’s mental health as a therapist, program manager and in provincial policy and planning, all roles entailing extensive cross-sector collaboration.
Laura N. L’Heureux, M.A., is the Evaluation Coordinator for the Southern Alberta Child and Youth Health Network; the Principal and Owner of Panoptik Research and Consulting in Calgary, Alberta; and has served as the Vice-President of the Alberta Chapter of the Cana-dian Evaluation Society since 2003. Special research interests include the application of systems theory to evaluation, network evaluation, sustainable development, practice and policy outcomes related to health research, organizational effectiveness, harm reduction, and access to health care. She obtained her interdisciplinary graduate training in program evaluation and applied research from the Applied Social Psychology Department at the University of Saskatchewan.
Carly M. Dolinski, M.P.H., received her master’s degree from the Department of Population Health, University of Western Australia, where her research explored the health of child care workers from a policy perspective. She has worked as a clinical nurse, project co-or-dinator, evaluator, and research assistant, with particular interests in remote health care, Aboriginal health, and maternal and child health. Most recently, she completed a research assistant post with the Calgary Health Region’s Decision Support Research Team based at the Alberta Children’s Hospital. Now back in Australia, Carly will soon be commencing a new position as a program offi cer with the Strategic Planning Branch of the Offi ce of Mental Health Division at the Health Department of Western Australia.
Carol E. Adair, M.Sc., Ph.D., is Associate Professor in the Depart-ments of Psychiatry and Community Health Sciences and co-lead of the Population Mental Health Research Program in the Hotchkiss Brain Institute at the University of Calgary. Her research interests
THE CANADIAN JOURNAL OF PROGRAM EVALUATION150
around health services include evaluation/outcomes research and paediatric and psychiatric epidemiology.
Suzanne C. Tough, Ph.D., is Associate Professor in the Departments of Paediatrics and Community Health Sciences as well as Associate Director for the Institute of Maternal and Child Health at the Uni-versity of Calgary. She is also the Scientifi c Director of the Alberta Centre for Child, Family and Community Research and an Alberta Heritage Foundation for Medical Research (AHFMR) Population Health Investigator. Her research program focuses on the health and well-being of mothers and infants beginning prior to conception through infancy to optimize birth and childhood outcomes.
Ann L. Casebeer, Ph.D., is Associate Professor in the Department of Community Health Sciences at the University of Calgary. Her research efforts focus on understanding change supporting health improvement, with special interests in organizational innovation in the context of health systems, advancing policy incentives, and enhancing public participation in decision making for health and health care. As Faculty Director for SEARCH Canada (Swift Effi cient Application of Research in Community Health; a public service or-ganization dedicated to improving knowledge development and use in practice), Ann is particularly interested in supporting learning for innovation and in sharing and using learning across challenging boundaries and borders.
Kathleen L. Douglas-England, M.Sc., is an adjunct professor in the Department of Paediatrics, University of Calgary, and a Senior Research Consultant in the Quality, Safety and Health Information Portfolio, Calgary Health Region. Her program of research and train-ing includes survey instrument development, program evaluation for health services, and expertise in patient and family feedback about the health care experience.
Catherine C. Morrison, M.S.W., is the Calgary Region Network Manager for the Southern Alberta Child and Youth Health Net-work. She has over 20 years of experience within the health and children’s services sectors. Her graduate work focused on networks, their functioning and evaluation, and she has an ongoing interest in strengthening connectivity between organizations that serve children and their families.