Mindset - A Clash of Mindsets Part 1

A clash of mindsets - finding a middle way

between different impact methodologies

Joy Mackeith, Triangle, Joy@triangleconsulting.co.ukNeil Reeder, London School of Economics,

n.b.reeder@lse.a.uk

Social Impact Analysts Association annual conference, December 2013

Aims of session

• Lessons from practical experience

• Outline of findings from academic research project

• Initiate debate on key issues arising from different mindsets

Objectivity and subjectivity(or direct and indirect measurement)

Direct measures (often end outcomes)

Things that can be directly observed,

eg housing, offending

Associated with direct cost savings

Indirect measures (often intermediate

outcomes)Things that cannot

be directly observed but have an impact

on directly observable

behaviour, eg confidence, improved

relationshipsAssociated with sustainability of

outcomes

Objectivity and subjectivity: Middle ground

Use both!Bring objectivity into indirect outcomes through tools for standardising professional

judgement or subjective report ie psychometric tools

Indirect measuresDirect measures

The Outcomes Star

Research on validity, reliability and practical application

Dickens et al (2012) “Recovery Star: validating user recovery” in the Psychiatrist

Dr Helen Killaspy (2012) “Psychometric properties of Mental Health Recovery Star in British Journal of Psychiatry Harris and Andrews (2013) “Implementing the Outcomes Star well in a multi-disciplinary environment” RMIT University, Australia (Homelessness Star, Drug and Alcohol Star and Empowerment Star)

York Consulting (2013) “Family Star Evaluation” published by Family Action

Standardised systems versus case by case approaches

Standardised systems

Provides a shared framework for

use across many projects or people

Can be aggregated

Case by case approach

Responds to the uniqueness of each project or

personCaptures all impacts and

reflects values and language of

those being measured

Standardised systems versus case by case approaches : Middle ground

Frameworks that allow case by case approaches to be analysed within a

standardised system (eg BSC)Tools that are modified for different

client groups (eg Outcomes Star)

Case by case approachStandardised systems

Example: Big Society Capital Outcomes Matrix

Outcome Area Individual Community, Sector and Society

Employment Education and TrainingHousing and Local Facilities

Income and Financial Inclusion

Physical Health

Mental Health

Family Friends and RelationshipsCitizenship and community

Arts, Heritage, Sports and Faith

Conservation of the Natural Environment

Example: Outcome Star versions

Recovery Star (mental health)

Life Star(Learning

Disabilities)

Work Star

Older Person’s Star

Youth Star

Empowerment Star

(domestic violence)

Teen StarWell-being Star

(long-term health conditions)

Spectrum Star(Autistic

spectrum)

Family Star

Community Star

Shooting Star(schools)

Homelessness Star

Alcohol Star

Music Therapy Star

Drug and Alcohol Star

Experimental versus exploratory approaches Experimenta

l Simplif

y complexity

by isolating one variable and identif

ying its

impact (ie

RCT)Can

produce very persua

sive eviden

ce

Exploratory

Explore relation

ships betwee

n variables and recogni

se complexity of systems and

relationships (eg St

Mungo’s work)Cheape

r and can

produce richer insights

into cause and

effect

Experimental versus exploratory approaches : Middle ground

Look when each approach is appropriate. Perhaps

experimental approaches appropriate when sufficient

exploration carried out and in very stable environments

Exploratory

Experimental

Comprehensive versus pragmatic

Comprehensive versus pragmatic: Middle ground

Comprehensive provides good intellectual framework

but measurement in practice must be pragmatic.

Walk first, run later!

PragmaticComprehensive