Improving Futures Evaluation and Learning

Evidence Briefing PresentationLaurie Day and James Ronicle, Ecorys UK

21st January 2014

Part 1:

Evaluation Overview & Methodology

Laurie Day, Evaluation Director

Key Indicators Improving Futures

• Launched by the Big Lottery Fund in March 2011

• Total £26 million grant funding distributed between 26 pilot projects across the UK

• Aiming to test different approaches to improve outcomes for children in families with multiple and complex needs

• Requirements: – led by VCOs partnership with statutory services– include mechanisms to engage 'hardest to reach' families; and. – oldest child 5-10 years at entry stage

Measuring Impact Aims of the Evaluation

Evaluation aims 1.To measure the effectiveness, impact and outcomes of the 26 projects; and the programme as a whole2.To support the projects with measuring progress 3.To capture and share learning across the programme, and to disseminate to policymakers & practitioners across the UK

Scope •Consortium led by Ecorys, with Ipsos MORI, Family Lives and University of Nottingham •Five year evaluation (Oct 2011 - Dec 2016)•Evaluation and learning

Health and wellbeing

Emotional and behavioural

development

Educational outcomes

Safeguarding and social care

Crime and ASB

Parenting skills and confidence

Health and wellbeing

Educational outcomes

Safeguarding and social care

Crime and ASB

Family functioning and relationships

Financial wellbeing and

security

Secure and safe environment

Social networks and belonging

Strategic influence over UK policyLeverage over external resourcesSynergy with other programmes

Knowledge transferReplication of effective practice

Net improvements to children’s life chancesNet risk reduction for families with complex needs

Net attributable savings

Programme Effectiveness

Programme-levelEffectiveness of programme design,

development and implementation

Project-levelEffectiveness of governance;

partnerships; strategy and planning; user involvement and delivery

Quality of provision Sustainability

Children Adults Families

Evaluation FrameworkProgramme Impacts

Programme Outcomes

Strategic Benefits

Measuring Impact Methodology

Programme-Level Evaluation

•Literature review

•Monitoring data collection (IFMIS): secure online system (practitioner-led)

•Longitudinal survey of families: baseline; +12 mths, and +24 mths

•Stakeholder telephone survey: key local stakeholders (2013 & 15)

•Case study research: mix of longitudinal (n=6) & snapshot (n=14)

•Family Advisory Panel

Project-level evaluation

•Co-production of local evaluation plans: support with identifying project outcome measures & data sources

•Bi-annual monitoring against local grant milestones, and qualitative feedback on lessons learned

•Support for self-evaluation: advice on methods, tools and training

•Enhanced evaluation activities: CBA case studies; impact case studies; participatory research case studies

Measuring Impact Key considerations •Need to gather comparable data: measuring changes to families’ status over time •26 local projects, with different local data systems

Format Secure online databaseData entry by project workers, based upon data recorded within service plan (or equivalent) Three mandatory data entry points: entry, exit, + 6mths post-exit (with optional interim measure)

IFMIS: Overview

Key Indicators Key Indicators

• Child, adult and family indicators, based on a review of the research literature, plus socio-demographic data

• Problems and strengths; hard and softer measures

• Three indicator types : dynamic (subject to change), event-based (time-specific), and status (interpretive only)

• Record type and amount (hours) of support received

Measuring Impact Creating a family record – example

Measuring Impact Measuring Impact Adding and updating indicators – example Impact

Measuring Impact Measuring Impact and Outcomes

1. Desk research to develop key indicators (risk factors & strengths)

2. Embed within monitoring framework (IFMIS)

3. Measure short-term change over time (monitoring data)

4. Measure longer-term outcomes, up to +2yrs (survey)

5. Test predictive ‘strength’ of indicators (compare survey & MI)

6. Scale-up to estimate programme-wide effects

Measuring Impact

• Need to establish ‘additionality’ of the programme

• Counterfactual analysis: explore alternative scenario (no intervention). Two approaches:

1. Self-reporting of impacts: all projects (exit questionnaire and via Ipsos MORI survey)

2. Quasi-experimental design: 4-5 impact case study projects: compare outcomes between participants and matched comparison group (“difference-in-differences”)

Counterfactual

Evaluation Activities Completed to Date

1. IFMIS system live and fully embedded

2. Bespoke evaluation plans in place for all 26 projects

3. Baseline survey of families underway

4. First local stakeholder survey underway

5. Six ‘trailblazer’ case study visits completed (England, Wales, and Scotland)

6. Participatory action research training delivered

Measuring Impact Learning and dissemination strand

Policy Makers, Practitioners, Wider StakeholdersImproving

Futures Projects

Good Practice Publications

Webinars

Policy Workshops

Face to Face Learning Events

Project Exchanges

Shared Learning Environment

Electronic Newsletters

Press Releases

Policy Briefings

Policy Makers, Practitioners, Wider

Stakeholders

Measuring Impact Learning activities completed to date

Internal1.National launch event for the programme 2.Secure online portal for projects3.Project networking events and webinars 4.Newsletter

External 1.Launch of evaluation web-site: http://www.improvingfutures.org/ 2.Social media activities 3.Policy round table session

Measuring Impact

Part 2:

Key Findings from Year 1 of the Evaluation

James Ronicle, Evaluation Project Manager

Introduction

- Progress in Establishing Projects

- Supporting Families

- Who are the Improving Futures Families?

- Early Outcomes for Families

- Conclusions and Next Steps

- Implications and Lessons Learnt for Future BIG Programmes

• Focus to date has been on building local partnerships and delivery teams.

• Projects still in early stages of delivery & models are evolving

• Number of families supported by projects are lower than anticipated. However, referral numbers picked-up considerably during 2013.

• Projects face more challenging financial climate than when launched.

• Projects faced barriers to partnership working due to heightened sense of competition & widespread restructuring

• Despite this, projects building strong local partnerships: - Engaging public services - Collaborations between VCSs - Engaging ‘micro’ VCSs

• Consequently, projects sharing best practice & early signs becoming embedded within local structures for children & families.

Progress in Establishing Projects

Supporting Families: Examples of Improving Futures ProjectsThe Neighbourhood Alliance, Sunderland•Led by Foundation of Light, Sunderland Football Club•Developed ‘Neighbourhood Menu’ of 45 ‘micro- VCSs’ to support families

BIG Manchester•Created co-located team to tackle ‘toxic trio’ of substance misuse, mental health & domestic violence•Provides three areas of support:

• Holistic family support through Key Worker• Peer and community engagement• Spot purchasing of specialist support

Family Pathways, Lewisham•Focuses on children demonstrating destructive or anti-social behaviour•Runs ‘Roots of Empathy’; classroom-based programme where children spend time with babies to strengthen emotional competence• Delivers training for

practitioners on promoting attachment between parents & children

Tyne Gateway, N & S Tyneside•Recruits Family Entrepreneurs as ‘barefoot professionals’:

• Outreach• Support in assessing family needs• Support in delivering services• Direct support to families

Supporting Families:Examples of Projects

Supporting Families:Identification and Engagement

• Primary schools have provided a hub for many projects (59% of families engaged through schools)

• Challenges around reaching out to ‘harder to engage’ families who do not traditionally participate

• Previous negative experiences of contact with services having an impact on willingness to engage

• Some challenges around 5-10 age criteria

• Greater number of families with complex needs than anticipated, in some projects

Supporting Families:Packages of Support

• Replicating many aspects of good practice :• ‘Key Worker’ model• Holistic support packages• Evidence-based parenting programmes:

• Incredible Years, Triple P, Roots of Empathy

• Innovative finance mechanisms:

• Family budgets• Spot purchasing

• Peer support

• However, not all projects providing ‘holistic’ support

• Key success factors:

• ‘Out of hours’ provision• Frequent Key Worker contact• Hands-on approach with referrals to other services

Measuring Impact Who are the Improving Futures Families?Demographics

• Free School Meals significantly above national averages (77%)

• Minority ethnic background – above UK averages (at 31%). Variation from previous family programmes (FIPs: 88% white, and FPs: 77% White-British)

• High proportion of lone parent families – approaching two thirds, at entry stage (62%).

• Predominately female adult profile (75%)

Measuring Impact Who Are the Improving Futures Families?Family Risk Factors

Figure 1: Family risk factors affecting 10% or more of the cohort

Base: 891 families

• ‘Upper-middle’ in continuum of need. More prevalent than general population; less prevalent than other intensive intervention programmes.

Measuring Impact Who Are the Improving Futures Families?Adult indicators

Risks (-)

•Parenting difficulties the most prevalent issue: parenting anxiety or frustration (56%), problems with discipline / boundaries (49%)

•Well over one third with suspected or reported stress or anxiety (41% - twice the population average)

•Far lower prevalence of adult ASB or crime indicators than some other programmes (2.1% ASB, compared 79% for FIPs).

Strengths (+) •Over half of adults within the cohort (51%) demonstrating awareness of how to keep children safe from harm, and around one third setting appropriate boundaries (34%)

•Fair levels of face-to-face contact with school staff and positive relationships, ranging from under half for all adults (45%) to just over half for lone parents (52%)

Measuring Impact Who Are the Improving Futures Families?Child indicators

Risks (-)

•Child behavioural problems by far the most prevalent risk, when low-level (30%) and persistent disruptive behaviour (16%) are combined

•One in five children with suspected or reported stress or anxiety (22%); mirroring similar problems in adults within the cohort, and one in ten under-achieving at school

•Lower propensity for project workers to report risk factors for children; could indicate recording difficulties

Strengths (+) •Most children adopt some aspects of a healthy lifestyle: well over half (61.5%) attend regular medical appointments; a fair proportion participate in play (37%), and exercise or sport activities (27%)

•Over one third of children have supportive peer friendships at school (43%), whilst just over one quarter have regular contact with friends outside school (27%).

Measuring Impact Who Are the Improving Futures Families?Towards some typologies…

• Factor analysis of IFMIS data (in SPSS) – clusters of variables that are strongly correlated with each other but relatively uncorrelated with variables in other clusters

• Iterative approach to find the most robust and realistic groupings

• Eight typologies identified on this basis...

Measuring Impact Who Are the Improving Futures Families?Towards some typologies (2)

• Typology 1 - Families with strong structures and behaviours

• Typology 2 – Low skilled families with financial and housing difficulties

• Typology 3 – Families with significant discipline and behavioural issues

• Typology 4 – Transient families with domestic abuse, mental health and substance misuse

• Typology 5 – Families engaged in their community

• Typology 6 – Families known to the police for minor disorder

• Typology 7 – Socially excluded families with health problems

• Typology 8 – Families known to the police for more serious incidents

Early Outcomes for Families

1. Projects able to achieve ‘quick wins’ for families:

1.Resolving financial problems

2. Improving housing circumstances

2. Parents reporting improved well-being:

1.Confidence & self-esteem

2.Reduced Anxiety

3.Parenting capacity

3. Improved behaviour and attendance at school

Evaluation Conclusions•Improving Futures is becoming integrated into wider local support infrastructure for children and families

•The programme is strengthening links between statutory and VCS organisations

•Projects showcased diverse range of practices for working with families at different levels of need. Some are replicating good practice, others developing good practice

•Too early to offer robust conclusions about impact of programme, but positive early signs that projects making different to families’ lives.

Areas for Development•Reaching families not in contact with mainstream services

•Taking stock of ‘whole family’ approaches

Conclusions and Next Steps

Evaluation design

•Need to strike a balance between a) freedom for grant-holders to innovate, and b) setting parameters for measuring impact

Programme design

•Value of statutory (DCS) endorsement of VCS bids•Development phase critical for building multi-agency partnerships (…but make next stages conditional on early success?)

Key areas of added value / potential legacy…

•Understanding dynamics of change for families over time•What works with the 5-10 age group, and VCS-school links •New forms of service commissioning (esp. neighbourhood level)•Models of engagement and support for BME families

Implications / Lessons Learned for Future Programmes