Upload
magdalen-oconnor
View
217
Download
1
Tags:
Embed Size (px)
Citation preview
Outcome Mapping
Team MeetingFebruary
2010
2
Key evaluation challenges
• measuring development results
• establishing cause & effect in an open system
• timing
• encouraging iterative learning
• clarifying values
3
problem with « impact »
Impact Implies:
• Cause & effect• Positive, intended results• Focus on ultimate effects• Credits a single contributor• Story ends when program
obtains success
Development Implies:
• Open system• Unexpected positive &
negative results occur• Upstream effects are
important• Multiple actors create
results & need credit• Change process never ends
4
Recognizing that change is…
• Continuous
• Complex
• Non-linear
• Multidirectional
• Not controllable
5
OM - a flexible, multiple-use tool
• Planning
• Monitoring
• Evaluation
Thanks to Paul Crawford
Boundary Partners
Ultimate beneficiaries
RESOURCES
OUTPUTS
OUTCOMES
IMPACT
GeSCI
Strategic partners
Strategic partners
7
focus on direct partners
• Key concept is « boundary partners »
• The individuals, groups, and organizations you work with directly and anticipate opportunities for influence
Focus of Outcome Mapping
Behavioural Changes
Local partners / beneficiaries ownership
increases
Program influence decreases
Inputs Activities Outputs Outcomes Impacts
9
Why behaviour changes?
• To stress that development is done by and for people
• To illustrate that although a program can influence the achievement of outcomes, it cannot control them because ultimate responsibility rests with the people affected
10
Contribution not attribution
You can influence but not control change in your partners
11
• PLANNING: articulate goals & define activities
• MONITORING:assess program performance & partners’ outcomes
• EVALUTION:design & conduct a use-oriented evaluation
Primary uses
The Questions that Outcome Mapping will The Questions that Outcome Mapping will help to answer :help to answer :
• In designing and articulating the program’s logic
- What are our development goals?- How can our programme contribute to those development goals?- Who are our beneficiaries?- How can we help our beneficiaries contribute to the broader
development goals?
• Recording internal and external monitoring data
- How far have our beneficiaries progressed towards achieving outcomes?- What are we doing to support the achievement of those outcomes?- How well have we performed?
• Indicating cases of positive performance and areas of improvement
- What worked well? Why?- How can we maximise our contributions?
Kiely, Team Meeting, December 2007
The Questions that Outcome Mapping will The Questions that Outcome Mapping will help to answer help to answer (continued)(continued)::
• Evaluating intended or unexpected results
- Who changed / what changed? How?- If they did not change as expected, do we need to do something different or
reorient our expectations?
• Gathering data on the contribution that a program made to bringing about changes in its partners
- What activities / strategies were used?- How did the activities influence the individuals, groups, or institutions to
change?
• Establishing evaluation priorities and an evaluation plan
- What strategies, relationships, or issues need to be studied in depth?- How, and from where, can we gather relevant data?
Components of Outcome MappingComponents of Outcome Mapping
• A description of the macro-level changes to which program intends to contribute
vision, mission, boundary partners and outcome challenges, progress markers
• Strategy maps
• Organisational Practices Journal
• A monitoring framework outcome journal, strategy journal, performance journal
• Evaluation Plan
15
boundary partners
Those individuals, groups, and organizations with whom the program
• interacts directly to effect change• anticipates opportunities for influence
• engages in mutual learning
16
Boundary partners have boundary partners
program program’s bp bp’s bp
17
describe:
• the ideal behavioural changes (relationships, activities & actions) of a boundary partner
• how they will contribute ideally to the vision.
Outcome challenges
18
Example: Outcome Challenge Statement (IDRC)
Vibrant and dynamic regional and national knowledge building and sharing communities which will produce and share high quality evidence based research and experiences on ICT4E and KS. These communities will work with GeSCI in a pro-active and committed way to identify critical issues, develop appropriate solutions and support one another on a peer basis. They will make optimal use of the GeSCI resources, tools and expertise. They will commit themselves to participate in leadership capacity building for ICT4E and KS and acquire and apply critical identified skills in their respective fields for system transformation and innovation.
19
Progress markers
✓ A graduated set of statements describing a progression of changed behaviours in the boundary partner
✓ Describe changes in actions, activities and relationships leading to the ideal outcome; shows story of change
✓ Articulate the complexity of the change process
✓ Can be monitored & observed
✓ Permit on-going assessment of partner’s progress (including unintended results)
ExampleExampleBoundary partner Outcome challenge
Government officials and policymakers
Ministries of Education(s) Endorsement of the ICT initiatives, approval of the implementation details and establishing guidelines for future implementation.
Progress markers
Expect to see1. Negotiating with SchoolNet and other service providers on details of implementation2. Setting a priority list for implementation3. Taking control of the initiative
Like to see1. Efficient coordination of partnerships for delivery of ICT services to schools.2. Establishing guidelines for integrating ICT effectively into learning and teaching 3. Setting operational objectives for implementation based on the ICT policy for education4. Direct budgeting for expenditure by the Ministries of Education on the programme to be
specified in the Policy Implementation Plan
Love to see1. Creating the critical mass of ICT skills at school level to be able to realise the country’s
vision for the future.2. Setting specific guidelines on cost-effectiveness for ICT solutions for schools or putting
implementation out to tender to see who offers the best option3. Using the ICT initiative to facilitate a change in emphasis to student-centred learning and
teaching.
21
Taken together, progress markers
facilitate mid-course corrections and improvements
stimulate the program to consider how it can contribute to the most profound transformation possible
articulate the complexity of change
22
X
23
Strategy Maps
• Outlines the program`s approach in working with the boundary partner
• Breaks strategies down into different types so you can be as strategic and influential as possible
24
6 types of strategies
Aimed at the
Boundary Parnter
Aimed at the Boundary Partner`s
Environment
Strategy SupportivePersuasiveCausal
I-1
• Direct Output
I-2
• Arouse New Skills/ Thinking
I-3
•Supporter who guides change over
time
E-2
•Modify the information system
E-3
•Create / Strengthen a Peer
Network
E-1
•Alter physical or regulatory environment
25
Practice: Facilitation Questions
I-3I-2I-1
E-1 E-3E-2
What will bedone to producean “immediate”
output?
What networks/relationships willbe established
or utilized?
How willsustainedsupport,
guidance, ormentoring be
provided?
How will you usethe media orpublications?
What will bedone to build
capacity?
What will bedone to changethe physical or
policyenvironment?
M&E Planning
27
Use(s) and User(s) – Who will take action(s) or make decisions with the help of the M&E process or findings? What actions and/or decisions will be taken?
Questions Timing Methods & Data
Sources
Responsibility & Roles
Dissemination & Communication
Costs
What do you want to know about our programmes?
When are the findings needed in order to be useful to the user(s)?
When will the data be collected?
How will data be collected? (I.e., instruments) From what sources?
Who is going to lead the process?
Who needs to play what role to accomplish the work? (eg., who will manage collection? analysis? writing/reporting?)
How will the findings be communicated to the intended user(s)?
What should the final products look like in order to be useful?
Who else will be interested in the findings and should be informed of them?
What human and financial resources are needed and who contributes?
What resources are available or not?
28
different tools you can use to collect data
• Outcome/ strategy mapping journals
• most significant change methodology
• project cycle analysis
• activity system analysis
• any social science tool or method
• ….
30
outcome journal
To understand the change process in boundary partners. Collects information about:
• Story of change and reasons for change
• Unexpected changes
• The actors and factors that contributed to that change
• How we know the change occurred
• Learnings
31
outcome journal
Work Dating From/To:
Contributors to Monitoring Update:
Low =
Medium =
High =
Outcome Challenge:
LMH
Expect to see: Who?
3
1
2
32
set values for low, medium & high
• Can capture either quantity or quality of change(s):
– Quantity (number of boundary partners)– Quality (depth of change)
34
strategy journal
To understand how the programme is contributing to changes in boundary partners. Collects information on:
• Resources that have been invested• Strategies and Activities• Products• Effectiveness• Changes that need follow up• Learnings
35
strategy journal
Working Dating From/To:
Contributors to Monitoring Update:
Strategy to be Monitored:
Effectiveness?
(How did it help the boundary
partner?)
Outputs
36
how to use the journals
• alone or combined with other tools and methods
• focus groups, interviews, e-surveys, conference calls, revision of documents
• can incorporate quantitative and qualitative methods
• face-to-face workshops
• other…
37
new technologies for data collection
• discussion forums • handhelds• participatory video• survey monkey• blogs• chat• skype• what else…?
38
Expanding accountabilities
OM helps you:
• manage multiple accountabilities: up, down, horizontal by:– being able to document & communicate outcomes
(traditional understanding of accountability)– engage in social learning with your boundary partners
making you more accountable to yourself, your organization, and them
39
Taking up M&E findings
• What should we keep doing?
• What do we need to change in order to improve?
• What strategies/practices do we need to add?
• What strategies/practices do we need to drop? (i.e., they produced no results, they require too much efforts or resources to produce results?)
• Has any issue come up that we need to evaluate in greater depth? What? When? Why? How?
40
Reviewing programme design
1. Read the vision statement
Does this still reflect the program's dream?
2. Read the mission statement
Is this the greatest contribution our program can make? Have we been doing this? Why? Why not? Should we add anything or take anything away?
3. Review boundary partners
Is this who we are working with directly? Do we need to work with anyone else?
4. Review outcomes Do these accurately describe the ideals way that our boundary partners could act to contribute to the achieving of the vision?
5. Review progress markers
Was the change process we set out accurate and useful? What now needs to be added or taken out?
6. Review strategies What did we plan to do? Have we implemented these activities? Why? Why not?
7. Review organizational practices
Are we doing everything we can to maintain our capacity to support our partners?
41
Discussion questions
• What elements of OM would we like to integrate that we think would be helpful? Why?
• What do we need to do to make this happen? • What spaces, methodologies & tools do we already use
that OM could complement? (how can we integrate OM into what we are already doing?)
• What challenges will we face? What opportunities are there to take advantage of?
• How will we communicate to others? • What will be our next steps?
OM 2009Review of Pilot OM Implementation within Culture for Organizational Learning
STRENGTHSIdentify areas where OM was used to provide a benefit to organization, teams and partners in 2009
WEAKNESSESIdentify areas of organizational practice which needed improvement in order to achieve desired outcome of OM integration in 2009
OPPORTUNITIESIdentify areas of OM that could support organizational learning and development in 2010 - that are not currently being utilized
THREATSIdentify possible obstacles or problems to the integration of OM horizontally across our programmes in 2010