Embed Size (px)
Citation preview
A clash of mindsets - finding a middle way
between different impact methodologies
Joy Mackeith, Triangle, [email protected] Reeder, London School of Economics,
Social Impact Analysts Association annual conference, December 2013
Aims of session
• Lessons from practical experience
• Outline of findings from academic research project
• Initiate debate on key issues arising from different mindsets
Objectivity and subjectivity(or direct and indirect measurement)
Direct measures (often end outcomes)
Things that can be directly observed,
eg housing, offending
Associated with direct cost savings
Indirect measures (often intermediate
outcomes)Things that cannot
be directly observed but have an impact
on directly observable
behaviour, eg confidence, improved
relationshipsAssociated with sustainability of
outcomes
Objectivity and subjectivity: Middle ground
Use both!Bring objectivity into indirect outcomes through tools for standardising professional
judgement or subjective report ie psychometric tools
Indirect measuresDirect measures
The Outcomes Star
Research on validity, reliability and practical application
Dickens et al (2012) “Recovery Star: validating user recovery” in the Psychiatrist
Dr Helen Killaspy (2012) “Psychometric properties of Mental Health Recovery Star in British Journal of Psychiatry Harris and Andrews (2013) “Implementing the Outcomes Star well in a multi-disciplinary environment” RMIT University, Australia (Homelessness Star, Drug and Alcohol Star and Empowerment Star)
York Consulting (2013) “Family Star Evaluation” published by Family Action
Standardised systems versus case by case approaches
Standardised systems
Provides a shared framework for
use across many projects or people
Can be aggregated
Case by case approach
Responds to the uniqueness of each project or
personCaptures all impacts and
reflects values and language of
those being measured
Standardised systems versus case by case approaches : Middle ground
Frameworks that allow case by case approaches to be analysed within a
standardised system (eg BSC)Tools that are modified for different
client groups (eg Outcomes Star)
Case by case approachStandardised systems
Example: Big Society Capital Outcomes Matrix
Outcome Area Individual Community, Sector and Society
Employment Education and TrainingHousing and Local Facilities
Income and Financial Inclusion
Physical Health
Mental Health
Family Friends and RelationshipsCitizenship and community
Arts, Heritage, Sports and Faith
Conservation of the Natural Environment
Example: Outcome Star versions
Recovery Star (mental health)
Life Star(Learning
Disabilities)
Work Star
Older Person’s Star
Youth Star
Empowerment Star
(domestic violence)
Teen StarWell-being Star
(long-term health conditions)
Spectrum Star(Autistic
spectrum)
Family Star
Community Star
Shooting Star(schools)
Homelessness Star
Alcohol Star
Music Therapy Star
Drug and Alcohol Star
Experimental versus exploratory approaches Experimenta
l Simplif
y complexity
by isolating one variable and identif
ying its
impact (ie
RCT)Can
produce very persua
sive eviden
ce
Exploratory
Explore relation
ships betwee
n variables and recogni
se complexity of systems and
relationships (eg St
Mungo’s work)Cheape
r and can
produce richer insights
into cause and
effect
Experimental versus exploratory approaches : Middle ground
Look when each approach is appropriate. Perhaps
experimental approaches appropriate when sufficient
exploration carried out and in very stable environments
Exploratory
Experimental
Comprehensive versus pragmatic
Comprehensive versus pragmatic: Middle ground
Comprehensive provides good intellectual framework
but measurement in practice must be pragmatic.
Walk first, run later!
PragmaticComprehensive
