Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
1 |
SUSTAINABLE LIVELIHOODS AN EVIDENCE-BASED APPROACH TO IMPROVING PRACTICE AND OUTCOMES
Evaluation Café Presentation
September 23, 2015
©Resources for Results and
2 |
Today’s Evaluation Café Session…
Agenda (35 minutes)
Part 1 – Why a Theory of Change?
Part 2 – Sustainable Livelihoods
Part 3 – Our Action Research Project
(Mini Case)
Part 4 – Results and Findings
Part 5 - Discussion
Purpose
To explore the merits of a theory of
change in promoting effective
program practice and improving
outcomes
Resources for Results &
3 |
Why a theory of change?
How many of you facilitate the development of Theories of Change in your evaluation work?
Resources for Results &
4 |
It can be:
A ‘road map’ for your project
A drawing that communicates the logical connections between your actions and desired outcomes
A statement of the chain of inter-connected outcomes and key milestones of progress
What is a Theory of change?
Resources for Results &
5 |
Evaluation and the Program Planning Cycle
Program design
Implementation planning
Design of evaluation and
monitoring
Ongoing results documentation
Program implementation
Reflection and
learning
Results assessment
Redesign
Resources for Results &
6 |
Envisioning and planning for outcomes...
Supports a developmental
approach to evaluation
Supports the planning of
complex, multi-faceted initiatives
Engages partners and supports a
collaborative, learning-based
approach
Enhances a results-orientation in
your planning
Promotes learning about
effective practices
Improves your results
Advances the policy agenda
Resources for Results &
7 |
The Sustainable Livelihoods Framework
A quick overview of our grounding Theory of Change
Resources for Results &
8 |
What is Sustainable Livelihoods?
The Sustainable Livelihoods approach offers a framework and language for understanding the assets of people who are socially excluded and living in poverty.
Resources for Results &
9 |
Think “livelihoods”
A “livelihood”
includes…
Livelihood thinking
goes beyond the idea
of “having a job”
Being a citizen who is fully included in society
Being able to work and earn income, and being a part of the economy
Empowerment and independence – the ability to make decisions and have a say
Long-term economic and social security
Resources for Results &
10 |
What does “sustainable” mean?
Sustainability is defined broadly as…
Resilience The ability to cope with and recover from life
shocks and stresses Thee ability to be more resourceful and
“entrepreneurial” in navigating the job market Future mode The ability to build assets starting with a small
resource base Social inclusion Fairness and the opportunity to be a part of
society and the economy
Resources for Results &
11 | Resources for Results &
12 |
Client Engagement Work
Resources for Results &
13 |
14 |
Sustainable Livelihoods - Features of the Approach
Client-centred – focuses on clients and
“where they are at”
Holistic – deals with all areas of clients’ lives,
not just one
Positive – looks at what assets clients have,
and proposes strategies to build on their
strengths
Results-oriented – starts with what clients
want and promotes their priorities
Problem solving – identifies and seeks to
deal with systems and policies that hold
clients back
15 |
Our Action Learning Project
A theory driven approach to program innovation
It is critical to build a foundation for participation before clients can move on to more substantial employment and income security gains. We now have a clear understanding of what must be in place for clients to participate in, and benefit from program interventions. LEF has now developed client-centred practices that intentionally build constructive, asset-building behaviours and resilience in LEF clients .
16 |
Introduction - LEF Sustainable Livelihoods Project
Strategy 1: Co-creating effective client
engagement practices
Embedding client-focused Sustainable Livelihoods
and motivational interviewing practice into LEF
client engagement processes through a
participatory, action learning approach
Strategy 2: Outcomes evaluation
Building an evidence-based strategy for learning
and continuous improvement of emerging CEW
practices
Project Purpose : • To enhance client engagement and retention in LEF’s programs and services • To advance LEF’s efforts to restructure and de-silo its organizational structure and program delivery. • To deepen the livelihood outcomes for clients.
Collaborative Learning to integrate Sustainable
Livelihood
Evidence-based approach
Resources for Results &
17 |
Early Stage Pathways to Livelihoods A preliminary theory of change developed through intensive workshops with CEWs
Welcome
Personal welcome and orientation to
organization's services
Personal, non-judgmental
registration and intake
Immediate internal and external referral
"Triage" and problem
identification
Problem solving to stabilize people (e.g.
access to income boosting
entitlements)
Put in place supports to participation (e.g.
child minding, transit)
Decision re: "fit" for problem solving and
coaching
Pathway 1: Develop the conditions for
participation
Pathway 2: Develop the capacity for
participation
Asset mapping and assessment
Promoting action planning practice
In house referral to further training and
support
Ongoing support for decisionmaking and
asset building
Hold clients accountable for
action and progress
Pathway 3: Coaching for
behaviour change/asset
building
"Anchor" relationship
continues parallel to program
participation
Promote self-direction and self-advocacy (e.g. goal
setting and decisionmaking)
Manage and sequence asset development
(including money)
Support transition to engagement at
organization
Resources for Results &
18 |
Client Engagement Worker: Creating a culture and process that engages LEF clients
Design changes to promote client
engagement ...
Streamlined client flow
Client-focused approach
A shift to holistic coaching
Program design that promotes
client accessibility (hours, location)
Evidence-based learning approach
The Concept
The Client Engagement Worker role has
been established at LEF to offer a single
entry point and personalized, ongoing
support for new clients who come to LEF.
Clients are engaged through a one-to-
one interaction that offers holistic,
client-centred coaching and supports,
breaking down silos which can create
barriers to service, and addressing the
immediate needs and priorities of clients.
Resources for Results &
19 |
Welcome
•Day 1:
•Orientation
•Problem Solving
•Triage/needs assessment
•Action planning
•Client Service Plan
Problem Solving
•Early Meetings:
•Building trust and a working partnership
•Continued problem solving as required
• Form filling as required
•Ongoing action planning
Asset Mapping
•Within 2-3 weeks:
•Early program look at assets
• Facilitated self-assessment
•Mapping strengths
• Setting priorities and planning for building assets
Ongoing Coaching
•Ongoing as required:
• Support for self-direction & independence
•Checking in and holding clients accountable
• goal setting and decision-making
• self-advocacy
Ongoing Mapping
•At 3 and 6 months:
• Facilitated client self-assessment
•Assessment of progress and outcomes
Building the conditions and capacity for success in settlement and employment
LEF Client Engagement Work What happens at each stage of client engagement?
Resources for Results &
20 |
The Next Generation of SL Tools
Based on the original CWF Sustainable Livelihood
tools, Mary Ferguson and Janet Murray worked
with LEF managers and staff to design, refine and
pilot the following tools :
Welcome Form – Client intake and consent form
What can we do for you? – An orientation to
LEF’s program offerings
My Plan 1 – A form to support client-led planning
and action
My Plan 2 – An asset mapping tool that supports
client self-assessment and longer-term planning
and asset building
LEF has partnered with Resources for Results, Eko Nomos and the Ontario Institute for Studies in Education (OISE) to develop and pilot a new level of Sustainable Livelihood tools and practices.
Resources for Results &
21 |
Motivational Interviewing integrated well into coaching and asset mapping practice
The Spirit of MI
MI Is a new client-centred practice that is
taking hold in the social services sector.
Grounded in empowering psychological
counselling, MI offers constructive and
empowering language and practices that
fit well with SL’s holistic, asset-based
approach.
COLLABORATION – Relationship as
collaboration or partnership
EVOCATION – Elicit vs. Impart
AUTONOMY – Ability to choose
COMPASSION - Empathy
GUIDING – vs. following or leading
(From: What is Motivational Interviewing? (PPT), Goldstein and Anthony,
OISE, 2015) Resources for Results &
22 |
LEF Client Flow – Sequencing Client Engagement priorities
Resources for Results &
23 |
Livelihood
Stage
Building the Conditions for
Participation
Building the Capacity for
Participation
Engaging in the
economy
Steps
1 Welcome 2 Problem
Solving
3 Asset
Mapping
4 Ongoing
Coaching
5 Follow-up –Asset
Mapping
Outcomes
Access to services
Action planning becomes a
client practice
Improved program enrollment
Increased client trust and
engagement
Ongoing employment and
settlement problems solved
Improved program retention
Realistic self-assessment
Strategy for building assets
Planning skills
Improved connections
Enhances skills to get and keep a
job
Enhanced decision making
Stronger self-organizing and
action
Ability to self-advocate
Increased independence
Improved stability
Enhanced hope and
wellbeing
Ability to think
about and plan for
the future
Employment
Onward education
Improved health
Coping Strategies Asset Building Strategies Future mode
Expected Outcomes of Client Engagement
Resources for Results &
24 | Resources for Results &
25 | Resources for Results &
26 |
Results and findings of our developmental evaluation
A Client outcomes survey validates our theory of change
Resources for Results &
27 |
Resources for Results &
28 |
Table 2 - Survey Likert scales – Aggregate Asset Area Scores and Sub-sets of Indicators/scores Mean N
Getting Connected 1.15 133
I am less alone than when I first came to the Learning Enrichment Foundation 1.00 130
I have better access to the supports and services I need 1.18 131
I have a strong connection to at least one staff person here 1.26 131
Solving Problems 1.04 133
I am more likely to seek help and advice for problems I face 1.08 132
I can find ways of solving the problems I face 1.03 131
I try out new ideas and ways of doing things 1.03 129
I am more likely to let people know what I think and what I need 1.06 132
Basic Needs .79 128
My housing is better .42 102
I have better access to information and resources 1.02 126
I have better access to computers and the internet .80 122
Building skills to find and keep a job .97 132
I have a better understanding of my strengths and abilities .94 130
I have a stronger set of skills to help me find and keep a job .91 131
I have better connections to help me find work 1.03 131
I have stronger English language skills 1.01 106
Action Planning .88 130
I am clear about what I want, and where I am going in my life 1.09 129
I often use action plans and lists to keep me on track .69 127
I am more able to get things done on my own .86 127
Money .32 113
I have a better understanding of my finances and how to deal with my money .63 106
My income has increased since I first came here .12 101
I am more financially stable now .15 103
Standing up for myself .80 128
I know how to deal with government to get what I need .38 125
I can stand up for myself 1.03 124
I have a better idea of what my rights are 1.08 118
Improving Health .81 130
I am less stressed than when I first came here .82 125
I am more healthy mentally since I came here - I am more positive and less 'down' .83 128
Building Identity 1.06 127
I have a clearer idea of who I am and what I can do 1.02 125
I can keep going no matter what happens 1.02 124
I am more hopeful about the future 1.18 125
I am more confident 1.11 127
I have more control over my life 1.00 121
Growing wellbeing .90 126
My life is more stable .71 123
I am better able to think about my future 1.02 125
Resources for Results &
29 |
Getting connected
1
1.18
1.26
I am less alone than when I first came to the Learning Enrichment Foundation
I have better access to the supports and services I need
I have a strong connection to at least one staff person here
Getting Connected (composite score: 1.15, N=133)
Note: These Likert scores are based on a scale from -2 to +2 where -2 means "Strongly disagree"; 0 means "Neutral" and +2 means "Strongly agree“.
Resources for Results &
30 |
Skills and Employability
0.94
0.91
1.03
1.01
I have a better understanding of my strengths and abilities
I have a stronger set of skills to help me find and keep a job
I have better connections to help me find work
I have stronger English language skills
Building skills to find and keep a job (composite score: 0.97, N=132)
1.09
0.69
0.86
I am clear about what I want, and where I am going in my life
I often use action plans and lists to keep me on track
I am more able to get things done on my own
Action Planning (composite score: 0.88, N=130)
1.08
1.03
1.03
1.06
I am more likely to seek help and advice for problems I face
I can find ways of solving the problems I face
I try out new ideas and ways of doing things
I am more likely to let people know what I think and what I need
Solving Problems (composite score: 1.04, N=133)
Note: These Likert scores are based on a scale from -2 to +2 where -2 means "Strongly disagree"; 0 means "Neutral" and +2 means "Strongly agree“.
Resources for Results &
31 |
Resources for Results &
32 |
Scale,
n = completed all items in the scale
Number of
items in
scale
Internal consistency
(Cronbach's Alpha)
Composite score -2
to +2 (all
participants)
Basic Needs, n=100 3 0.864 0.79±0.81
Growing wellbeing, n=121 3 0.830 0.90±0.68
Getting Connected, n=127 3 0.746 1.15±0.74
Solving Problems, n=128 4 0.861 1.04±0.67
Action Planning, n=124 3 0.809 0.88±0.69
Building skills to find and keep a job, n=102 4 0.842 0.97±0.78
Money, n=94 3 0.898 0.32±1.01
Building Identity, n=118 5 0.905 1.06±0.66
Standing up for myself, n=112 3 0.749 0.80±0.79
Improving Health, n=123 2 0.816 0.81±0.89
Table 1: Chronbach’s Alpha - Analysis of scales Note: composite score reported as mean ± SD. All scales have high degree of internal consistency (value of Cronbach's Alpha about 0.7 or above).
Resources for Results &
33 | Resources for Results &
34 |
CEW Asset Mapping Pilot Asset Growth – pre/post assets (N=23)
ASSET AREA Pre-Training Post-Training Pre-Post Difference
Connections 6.42 7.63 1.21 (N=24)
Identity 7.23 8.26 0.74 (N=23)
Skills and Employability 6.79 7.46 0.67 (N=24)
Money 5.71 5.92 0.21 (N=24)
Foundation Building 6.58 7.46 0.33 (N=23)
Health 7.96 7.89 -0.41 (N=23)
Pre-Post Difference - change between Asset Maps
Resources for Results &
35 |
CEW Asset Mapping Pilot Livelihood Progress Case Study (N=8)
Sustainable Livelihood
Stage 5
Consolidation
Stage 4
Transition
Stage 3
Foundation Building
Stage 2
Engaging
Stage 1
Stabilizing
Resources for Results &
36 |
Promising Practices - CEW coaching and asset mapping support intentional asset building and livelihoods progress When combined with MI, Sustainable Livelihoods offers an
effective, client-centred practice that promotes outcomes.
We have confirmed that the following practices build a
foundation for client empowerment and independence:
1. Investment in long-term, face to face, “anchor” relationships of trust that focus on clients’ priorities – not funders’
2. Holistic, asset-based self-assessment using asset mapping resulting in livelihood development strategies.
3. Emphasis on the development of planning and decision-making skills through the ongoing re-enforcement of action planning, priority setting and problem solving skills.
4. Non-judgmental, motivational coaching that evokes clients’ own solutions to problems and desire for change.
5. Following up to hold clients accountable for their progress.
Resources for Results &
37 |
Finding 1 – Our theory of change has guided the development of evidence-based effective practices in client engagement
Finding 3 – There is strong potential for a Collective Impact approach and shared measures
Finding 2 – We now have a model of change. Our developmental evaluation has provided evidence validating our theory. Confirming that one-to-one client relationships build a foundation for participation and asset building
Resources for Results &
38 |
Thank You
Evaluation/ Report Credits: Janet M. Murray, Resources for Results Tel: 647-233-7711 Mary Ferguson, Eko Nomos Tel: 519-599-2410 Robyn Hoogendam, LEF Statistician: Anton Svendrovski Graphics: Nicholas Smith Graphic Production: Sara Gaudon
For SL resources go to: www.ekonomos.com
This report is the result of the intensive collaborative effort and persistence of a large team of people. We all worked hard on this project ! The authors wish to thank everyone who contributed their vision, resources and dedication to this project: Peter Frampton SL Design Team: Elisa Martinez-Reyes , LEF Robyn Hoogendam, LEF Dr. Immaculate Antony, OISE Dr. Abby Goldstein, OISE Janet M. Murray, Resources for Results Mary Ferguson, Eko Nomos Emmett Ferguson, Eko Nomos LEF Conductors: Roopa Dittakavi Cheryl Partridge Ray Stephens Fernando Rivas LEF CEW Staff