Upload
ann-larson
View
631
Download
0
Embed Size (px)
Citation preview
Evaluation amidst complexityEight questions evaluators should ask
Originally presented at Australasian Evaluation Society Annual Meeting, Melbourne, September 2015.Revised November 2015.
Ann Larson, PhDwww.socialdimensions.com.au
Complexity is often invoked as a reason for a project’s failures. My key message is that evaluators can use a complexity lens to
understand and facilitate success – if we know what to look for.
Overview of presentation
• Framing the problem• Explaining characteristics of complex
adaptive systems• Eight questions to ask in an evaluation• Closing remarks
Framing the problem
Characteristics of projects introducing an evidence-informed intervention which are affected by complexity
Signs of resistance or lack of support among some stakeholders or intended beneficiaries
Slow progress in starting implementation
Small or lack of adoption of new practices despite different strategies to change behaviour
Not clear how to sustain gains when the project finishes
Context for evaluatorsInvolved only at the end of the project, not at the beginning or middle.
The task is to make a narrative to explain success or failure that is convincing to implementers and donors.
We also must make recommendations that are grounded in evidence.
There is rarely a logic model or an M&E system that is particularly helpful for the evaluation.
Explaining complex adaptive systems
Properties and behaviours of complex adaptive systems
• Path dependency• Diversity• Multiple interdependent relationships • Self-organising emergent behaviours • External shocks• Non-linear outcomes
Path dependency
Diversity
External shocks & outside influences
Interdependent relationships
Self-organising and emergent behaviour
Non-linear outcomes
Harnessing complexity
An eclectic list of strategies for projects to harness rather than control complexity
• Flexible, long term funding• New behaviour grounded in relevant history and
saliency• Build coalitions around a vision for change• Understand motivations for behaviour change:
introduce accountability and incentives• Start small, be flexible and experiment• Balance local initiative with quality standards• Monitor, review and act in a timely manner
Evaluation strategies
Evaluation designs for coping with CAS
Suggestions from evaluatorsDevelopmental evaluations
Push-back from clientsEvaluation only budgeted at end, take too long, too expensive, too much like internal evaluation
Intensive case studies to elucidate challenges and successes
Lacks generalisability, doesn’t address original project objectives and logic models
8 complexity-sensitive evaluation questions
Summary of complexity-sensitive evaluation questions
Is there evidence that project designers and implementers ….
1. Were sensitive to history and current priorities?2. Accommodate diversity in its design and implementation by employing
different approaches depending on capacity and circumstance?3. Understood dynamics of relevant behaviour?4. Effectively influenced those dynamics?5. Monitored, reviewed and took action based on regular information?6. Recognised and embraced emergent behaviours that supported the
intervention?7. Responded to external change?8. Were focused on what happens after the project?
1) Has the project aligned itself with history and current priorities? Has that had an effect on its acceptability?
Shape of the Busselton Sheds, planks from the old
jetty
2) Did the project take into account the differences in regions, local context, workforce or beneficiaries? How could it have been more effective if it had?
States -2Districts -5Facilities-5
States -19Districts -35Facilities-53
States -19Districts- 40Facilities - 81
States -19Districts – 206Facilities - 371
States -19Districts – 236Facilities - 458
Different strategies employed in different states and facilities
In this sanitation project, households build their own latrines using standardized construction methods but individualised shelters.
3) Did project designers and implementers understand relationships and dynamics in the workplace, families or agencies?
This can be done by being part of the culture, involving the group whose behaviour you want to change, or running small scale pilots.
Projects within complex adaptive systems need to identify and work ‘with’ or work ‘around’ all of the important components, such as supply chains.
4) What did they put into place to change those dynamics to change behaviour?
Training alone is rarely effective in changing behaviour because there are so many other influences reinforcing the old behaviour.
Among strategies to change behaviour affected by local norms and reinforced by feedback loops are coaching, measures to increase accountability, rewarding performance and removing obstacles.
Encourage or enable communities to demand the new service
Provide incentives to local leaders
5) Did the project routinely collect, review and respond to information about activities and behaviour change?
6) Did the project recognise emergent behaviours and, if so, how did they respond?
Local co-option of men’s health strategies
7) How did the project respond and adapt in face of external shocks and changes? Why were they able to do this?
Projects need good relationships and the capacity to be flexible to respond to changes in policies, funding or security issues through advocacy or problem solving at the appropriate level.
8) How is the project preparing for what will happen when it ends?
What aspects will government or local NGOs retain? Will the principal goal continue to inspire action? How is the implementing agency altering their approach?
Summary of complexity-sensitive evaluation questions
Is there evidence that project designers and implementers ….
1. Were sensitive to history and current priorities?2. Accommodate diversity in its design and implementation by employing
different approaches depending on capacity and circumstance?3. Understood dynamics of relevant behaviour?4. Effectively influenced those dynamics?5. Monitored, reviewed and took action based on regular information?6. Recognised and embraced emergent behaviours that supported the
intervention?7. Responded to external change?8. Were focused on what happens after the project?
Closing remarks
These questions can be used in an interview guide with stakeholders or as a framework for analysis.
We need more evidence on how to influence behaviour in complex adaptive systems. Evaluators
can contribute to the evidence base.
A personal reading listPaina, L. and D. H. Peters (2012). "Understanding pathways for scaling up health services through the lens of complex adaptive systems." Health Policy and Planning 27(5): 365-373.Preskill H et al. Evaluating Complexity: Propositions for improving practice. FSG working paper, 2014.Everything by Lant Prichett but especially Pritchett, L. and F. de Weijer (2010). Fragile States: Stuck in a Capability Trap? World Development Report 2011 Background Paper. Washington DC, World Bank.Everything by Trish Greenhalgh but especially Greenhalgh, T., J. Russell, R. E. Ashcroft and W. Parsons (2011). "Why National eHealth Programs Need Dead Philosophers: Wittgensteinian Reflections on Policymakers’ Reluctance to Learn from History." Milbank Quarterly 89(4): 533-563.Axelrod, R., & Cohen, M. D. (1999). Harnessing Complexity: Organizational Implications of a Scientific Frontier Chandy, L., A. Hosono, H. Kharas and J. Linn, Eds. (2013). Getting to Scale: How to Bring Development Solutions to Millions of Poor People. Washington DC, Brookings Institutions Press.Sutton, R. I. and H. Rao (2014). Scaling Up Excellence: Getting to More without Settling for Less. New York, Random House Business Books.