OPM Report Template version 5...December 2017 Childonomics Malta and Romania: lessons learned from applying the Childonomics conceptual framework and methodology in practice Joanna

December 2017

Childonomics

Malta and Romania: lessons learned from applying

the Childonomics conceptual framework and

methodology in practice

Joanna Rogers, Jean Anne Kennedy, Adrian Gheorghe and with

contributions from Volodymyr Kuzminskyi and Claudia Costea.

Childonomics: Malta and Romania: lessons learned from applying the Childonomics conceptual framework and methodology in practice

This assignment was carried out by Oxford Policy Management in association with IFCO and Partnership for Every Child CEE/CIS Consultancy group. For further information contact [email protected]

The contact point for the client is Jana Hainsworth [email protected]

Oxford Policy Management Limited Level 3, Clarendon House +44 (0) 1865 207 300

52 Cornmarket Street Fax +44 (0) 1865 207 301

Oxford, OX1 3HJ Email [email protected]

Registered in England: 3122495 United Kingdom Website www.opml.co.uk

© Oxford Policy Management, December 2017 i

Acknowledgements

First and foremost, we would like to thank all of our colleagues in Malta and Romania, who

patiently engaged with us as we worked through the piloting of the Childonomics methodology, and

who participated in validation meetings and workshops that have contributed to this report.

In Malta, our special thanks to the social workers, clients and managers of the Appogg Intake

and Family Support Service, the five community centres, the Foster Care Service, the Looked After

Children Service and the Sibling Group residential service. Thanks also to the information

management and financial management teams at the Foundation for Social Welfare Services and

to the National Foster Care Association Malta, the Kummissjoni Ejjew Ghandi, the Ministry of

Finance and the Ministry of Family, Children’s Rights and Social Solidarity. Our thanks especially

to the care leavers, children in residential care, foster carers and children in foster care who took

part in consultations.

In Romania, special thanks to the social workers, case managers and service managers of the

Alexander and Samuel Day Care Centres and the creche in Bucharest, who met with us and

shared information. Thank you also to parents, young people in care and care leavers who gave

their time and shared their views in interviews and group discussions. Thanks also to the National

Agency for the Protection of Children’s Rights and Adoption, which supported our field work, Bacau

County Council and Bucharest Sector 6 authorities, UNICEF Romania and Romanian non-

governmental organisations that took part in our consultations.

Many thanks also to our team members, Andy Bilson, Richard Carter, Maria Herczog and Chris

Rayment for comments and contributions to the thinking that helped to produce this report, and to

Clara Picanyol from Oxford Policy Management for peer reviewing the final version.

Thank you also to members of the project expert group, especially Andrew Goudie and Jennifer

Davidson for their active and meaningful contributions to the development and testing of the

methodology. Thanks also to Jana Hainsworth and Agata D’Addato at Eurochild for their support

and important contributions to the development and testing of the Childonomics methodology.

mailto:[email protected]

mailto:[email protected]

http://www.opml.co.uk/


© Oxford Policy Management ii

Executive summary

Childonomics is a research project that has developed the first iteration of an instrument that can

help decision-makers, researchers, child care specialists, social workers, child rights advocates

and other stakeholders to reflect on the long-term social and economic return of investing in

children and families within a given national or subnational context. The Childonomics

methodology proposes an approach to economic modelling that can be used in a number of ways

to inform decision-making, advocacy, research and practice development. The approach is based

on cost-consequence analysis (CCA) and incorporates elements of social return on investment

methods especially the inclusion of service user perspectives in the analysis of return on

investment. CCA presents costs and outcomes side by side in a disaggregated manner; it is a form

of cost-effectiveness analysis which presents the range of benefits identified alongside costs

incurred without aggregating them in a single metric (e.g. a cost-effectiveness ratio), leaving the

users of the methodology to incorporate their own considerations when judging the merits of an

intervention or programme.

Eurochild commissioned the testing of the instrument that was developed by the Childonomics

project and chose Malta and Romania as the countries for piloting as it had a strong partnership

with the Ministry of Family, Children’s Rights and Social Solidarity (MFSS) in Malta and the

National Agency for the Protection of Children’s Rights and Adoption (NAPCRA) in Romania at the

time when the project was being planned. Both government agencies had an interest in using the

instrument to analyse new policies and programmes and were committed to providing the

necessary data for the pilots.

The main purpose of conducting the pilots in Malta and Romania in 2017 was to test the draft

Childonomics methodology and to inform changes and adjustments to the final version. This report

presents the results from the country pilots and, as such, sets out the challenges and lessons

learned from piloting the Childonomics instrument at an early stage of its development. This report

should be read in conjunction with the Childonomics Conceptual Framework (Bilson et al, 2017)

and the Childonomics Methodology paper (Gheorghe et al, 2017).

The data gathered through the pilots were sufficient to field test the methodology, but not to

provide conclusive or even moderately strong evidence of a connection between investments and

outcomes or outputs for any of the services included in the pilots in both Romania and Malta. The

results of applying the methodology in these pilots do not provide sufficiently robust evidence for

advocacy campaigns or to inform decision making on policy or funding in the two pilot countries.

This report is structured according to the steps and outputs of the Childonomics methodology

(Figure 1).


© Oxford Policy Management iii

Figure 1: Outline of the Childonomics methodology

Step 0: Establish an understanding of the context in which the methodology is being used

Policies and programmes in Malta and Romania were mapped and analysed (presented in Section

1.1 and Annex A of this report) in order to inform decision-making about the scope of the pilots and

to gather data to inform the selection of outcomes and measurement of indicators (presented in

Section 3 of this report). The data instruments used and guidance on ethical considerations are

presented in Annex B.

Primary and secondary data were gathered from:

key informant interviews with national government specialists and decision makers, service

managers, social workers, social pedagogues and case managers;

case studies: family support cases and interviews with care leavers and young people in care;

service user consultations with children in residential care, children in foster care, foster carers

and parents;

service statistics analysis;

review of child protection and service delivery database parameters;

service manuals/procedures/standards; and

policy analysis and service provision mapping.

In both pilot field tests no existing research could be identified which could provide a sufficiently

robust basis for identifying outcomes (and indicators) for children, families, communities and

society and then to identify the inputs from specific services that could be directly attributable to

achieving those outcomes. The general lack of rigorous evaluations meant that the application of

the methodology was largely limited to the data that was gathered during the field work for the

pilots.


© Oxford Policy Management iv

The approach used to analyse qualitative data and measure indicators for selected outcomes in

both Malta and Romania is presented in Section 3 of this report.

Step 1: Determine scope

For the purpose of testing the data collection tools and methodology, the research team selected

which services the pilots would focus on, in order to test different levels of granularity:

In Malta, it was decided to undertake a whole-system comparison of specialised family support

services (Intake and Family Support Service (IFSS) and Community Services (CS)) that cover the

whole country, as well as two types of alternative care services (foster care and residential care).

This meant that associated services were also included within the scope of the pilot, such as

psychologists, home-based family therapy, and the Looked After Children’s Service (LACS).

In Romania, it was decided to compare similar, specialised community-based family support

services in urban and rural settings to see if the Childonomics methodology is sensitive to micro-

level comparisons and if it is possible to compare investments and outcomes for services working

with different target groups (e.g. young Roma men who are dropping out of school, children with

disabilities, babies and infants from low-income households). It was only possible to include two

day centres for children with disabilities. Financial data were not made available for the other two

services that were initially included within the scope of the pilot.

The time horizon selected for costing, in each case, was one year of serving a client. Some

process indicators that measure immediate service outputs were considered, such as caseload

(number of cases per social worker) or attendance at school, but the whole-childhood perspective

and longer-term outcomes for children as young adults was given consideration in terms of the

time horizon for measuring impact.

Step 2: Specify outcomes

Analysis of the data gathered permitted the identification of outcomes valued by stakeholders at

different levels, and indicators for measuring them. These were then presented in an outcomes

matrix, as illustrated in the Romania pilot outcomes matrix in Table 1.

The existence of common outcomes and indicators for different types of investments (services) is

critical if the services are to be included in the CCA. The process of establishing the scope and

specifying – and then measuring – outcomes and outputs is an iterative one, whereby the users of

the Childonomics methodology may need to adjust the scope and outcomes specification until the

set of services that is included in the CCA can be confirmed as working towards common

outcomes that can be measured with common indicators.

Table 1: Outcomes matrix for comparing day centres for children with disabilities based on analysis of data gathered during the Childonomics pilot in Romania

Domain Indicator Data sources

Ou

tpu

ts

Outp

uts

for

serv

ice u

sers

Child development

Age- and ability-appropriate developmental milestones (including early childhood development (ECD)) met, progress in communication, social skills and behaviour

Development, speech/communication assessments, case files, interviews with service providers and parents


© Oxford Policy Management v

Increased capacity of parents to provide care

Parent knowledge and competency and engagement in partnership with service staff

Interviews with carers and parents, interviews with children and young people, case files

Ou

tco

mes

Child

Children prepared for school/education

Literacy, progress in education during and after service use

Official statistics, service data and case files, interviews with young people (and former service users), education records

Fam

ily Parents

employed and relationships stable

Household income and structure Service data and case files

Com

mun

ity

Reduced separation of children from parents

Number of children with disabilities from communities targeted by day centres entering long-term alternative care

Official statistics, service data and case files

Socie

ty Improved

education and inclusion of children with disabilities

At risk of poverty and exclusion rate (AROPE), disaggregated for children with disabilities from different services at different stages of adulthood


Step 3: Measure outcomes

The ability to measure outcomes and outputs accurately and meaningfully is highly dependent on

the availability of valid data. In the case of the pilots, a limited number of interviews and group

consultations were conducted, and the available quantitative data were reviewed and analysed, but

ultimately generalisable conclusions could not be made. This is not to minimise the importance or

validity of the views and perspectives of service users and staff who were interviewed, but a

general absence of rigorous evaluation or granular monitoring data meant that analysis was

constrained and generalisability limited.

Step 4: Calculate costs

In regard to this step, the report sets out the assumptions that were made in order to allocate

costs. However, given that these assumptions were based on interviews and a review of

secondary data they cannot be considered to be underpinned by strong evidence. In this regard,

the costing work carried out during the course of the pilots can be considered as ‘good enough’ for

the purpose of testing the methodology and comparing investments using similar assumptions, but

should not be considered as reporting the real or actual expenditure on each type of service or set

of services included in the pilots.

Step 5: Present net costs and outcomes

The Childonomics methodology does not return a single metric but rather a more complex and

nuanced picture of the interplay of investments and outcomes at different levels presented in a

results matrix linking outcomes to investments and incorporating consideration of the strength of

evidence for these links.

The financial data and measurement of outcomes presented in the two pilot results matrices in

Tables 17 and 18 (Section 4 of this report) should be considered as illustrative only and not as


© Oxford Policy Management vi

reporting real or actual data, given that the strength of evidence was very weak and was based on

low validity and non-generalisable data. The data and measurement of outcomes are presented to

demonstrate how the Childonomics methodology culminates in a single matrix linking outcomes to

investments, while also reflecting on the strength of evidence for these links.

With this cautionary note in mind, it is nevertheless possible to see from the results matrices how

the Childonomics methodology presents information in such a way that it could help users,

especially if moderate or strong evidence can be found, to form judgements on the services being

analysed, and, with only indicative or no evidence, it highlights the gaps in knowledge that could

help to inform the development of monitoring systems or further research.

Step 6: Validation and summative narrative

The results matrix alone does not capture the nuances of the context in which the services that are

the subject of the analysis are being delivered: it has to be accompanied by a summative narrative

that presents context, interprets the results and facilitates a more informed understanding of the

data presented in the matrix. The methodology proposes that workshops are held with

stakeholders in order to validate the outcomes and results matrices, and to explore the implications

of the results from the CCA.

Researchers presented the purpose of the pilot process and the preliminary analysis to

stakeholders in Malta and Romania, both in groups and in individual interviews, and facilitated a

discussion of: the validity of the outcomes, outputs and indicators selected for the results matrices;

interpretation of the findings; and the challenges and opportunities that emerge from the findings.

Service users, social workers, specialists, managers and decision makers largely agreed that,

subject to some refinement and clarification, the outcomes and outputs identified were relevant to

the services included in the pilots, both in Romania and Malta, and can be applied across the

different comparator services. They also recognised them as reflecting their experience and

contributions made during the pilot studies.

While bearing in mind the low validity of the evidence presented in the matrices, the consultations

with stakeholders were important in regard to interpreting contextual factors affecting the links

between investment and results.

Stakeholders generally found the summative nature of the results matrices helpful in thinking about

the whole system and the wider results that can emerge from each part of the system.

The summative narrative summarises the contextual issues, nuances of interpretation and

feedback from stakeholders in a final analysis that should always accompany the results matrix as

the final output from applying the Childonomics methodology. The summative narrative includes

information from each step of the analysis and takes into account a range of stakeholder

perspectives. A combined Malta and Romania summative narrative given in Section 4 of the report

illustrates how a summative narrative can be structured. The results matrix should always be

accompanied by a summative narrative and cannot be considered valid without one.

Conclusions and lessons learned

Lesson 1: the purpose of using the Childonomics instrument needs to be clear from the outset and

consistently communicated to stakeholders.

The instrument can be used in a number of ways: by government policy makers to support

decision-making, by child-rights advocates seeking to strengthen the evidence-base of their


© Oxford Policy Management vii

advocacy; and by service-providers to compare and contrast different service options. Depending

on the purpose behind using the instrument, different levels of focus or granularity can be applied.

Users of the instrument need to be clear about what it can and cannot do and its strengths and

limitations in different circumstances. It is also important to recognise that the instrument is applied

within a specific socio-cultural context, where choices of policies and programmes are not value-

neutral. The instrument works best when used as a tool for open enquiry which can challenge

underlying assumptions about what system best supports families and children.

Lesson 2: the Childonomics methodology is highly dependent on the quality and validity of the data

that are available and cannot replace absent data.

Data gathering for the purpose of applying the Childonomics methodology can highlight gaps in the

availability of data as much as it can provide an opportunity for gathering new information. The

data gathered for the pilots in Malta and Romania do not replace rigorous evaluation, are mainly

anecdotal, and are at best indicative. Future users of the instruments should be rigorous in

ensuring all available research is reviewed and, where appropriate, they should commission

research to fill gaps in data.

Lesson 3: strong research skills are needed to undertake reliable and valid qualitative data

analysis and to ensure that the views and perspectives of all stakeholders, including service users,

managers and staff, are taken into account and valued.

Lesson 4: it is important to concentrate attention on the value of outcomes rather than on the

value of inputs, and to ensure that the complexity and context of services and their inputs and

outputs are given due consideration in the presentation of the results matrix.

Many stakeholders may be learning for the first time about the financial ‘worth’ of the service they

are involved in. In order to facilitate stakeholder contributions to defining outcomes of value and

providing feedback on the implications of CCA, it is important to present financial data as neutrally

as possible and to take care to ensure discussion focuses duly on outcomes, indicator

measurements and outputs.

Lesson 5: the Childonomics methodology convenes stakeholders across institutional and

organisational boundaries and has the potential to be a powerful tool for planning and monitoring

system change.

Users of the Childonomics instrument must be sensitive to the power dynamics in the systems in

which stakeholders participating in the CCA exist, and they must ensure that all stakeholder views

are given equal value.

Conclusions

The piloting of the Childonomics methodology has demonstrated its potential to support analysis at

all levels of the links between investments in children and families, and the resulting outputs and

longer-term outcomes for children and families themselves, but also for communities and society at

large. The piloting process has highlighted the need for good quality monitoring data and greater

transparency generally in child and family services, as well as the need for valid evidence on long-

term outcomes for children and families who have used certain types of services. Without these

types of data, there could be risks in using the results of the Childonomics CCA to take

programmatic decisions.

The Childonomics methodology nevertheless helps people to marshal the best available evidence

on the costs and outcomes of different services. Those using the methodology then need to decide


© Oxford Policy Management viii

if the evidence is strong enough to make judgements about future directions, and whether other

evidence from new research – or perhaps pilots – or further information on costs is required. Once

they are satisfied that they have a sound enough basis for future planning they can use the

methodology to model costs based on different assumptions or projections based on evidence

about the impact of, for example, increasing some aspect of community-based services or the

need for out-of-home care. This will thus require them to make projections based on assumptions

or evidence, where available, about the impact of different forms of investment.

There is no perfect model for assessing return on investment and the Childonomics methodology is

no exception, but it does help to bring out assumptions and to examine the evidence for these

assumptions and what they mean for children, families and the system as a whole, both in terms of

maintaining the status quo and exploring alternatives.

Remaining questions and challenges to be taken forward include: how to make best use of results that are underpinned by weak evidence; the validity of comparing services and the challenges the full evaluative methodology presents in terms of appearing to introduce an element of ‘competition’; and how to ensure the rights-based perspective set out in the Childonomics conceptual framework (Bilson et al., 2017) guides the selection of outcomes.


© Oxford Policy Management ix

Table of contents

Acknowledgements i

Executive summary ii

List of abbreviations xii

1 Introduction 1 1.1 Mapping and analysis of policy, programmes and services 3

1.1.1 Summary of policy priorities and outcomes sought by the governments of Malta and Romania at the time of the pilot 4

1.1.2 Examples of indicators for monitoring policy implementation in Malta and Romania 5

1.1.3 Programmes and services for children and families in Malta and Romania mapped onto the Childonomics conceptual framework 6

2 Description of data gathered to inform application of methodology 8 2.1 Primary data 9

2.1.1 Key informant interviews 9 2.1.2 Case studies 9 2.1.3 Consultations with service users 11

2.2 Secondary data 15

3 Analysis of qualitative data 18

4 Applied methodology - detailed costing and CCA in Malta and Romania pilots 30 4.1 Step 1: Determine scope 30 4.2 Step 2: Specify outcomes 31 4.4 Step 4: Calculate costs 38 4.5 Step 5: Present net costs and outcomes 43 4.6 Step 6: Stakeholder validation and the summative narrative 48

5 Lessons learned and conclusions 52 5.1 Lessons learned 52 5.2 Conclusions 54

References 56

Annex A Mapping and context - Malta and Romania 57 A.1 Romania 57 A.2 Malta 65

Annex B Data collection instruments and ethics guide 69 Ethical considerations and data collection instruments 69 Bibliography 74

Annex C Sample Case Study 107


© Oxford Policy Management x

List of tables, figures and boxes Table 1: Outcomes matrix for comparing day centres for children with disabilities based on

analysis of data gathered during the Childonomics pilot in Romania iv Table 2: Summary of characteristics of 10 family support cases, including inputs, outputs,

outcomes and indicators presented by social workers as part of the Malta pilot 10 Table 3: Breakdown of child participants in focus group discussions 12 Table 5: Summary of care experience pathways, reported by eight young people in Malta pilot 13 Table 6: Summary of care experience pathways, reported by seven young people in Romania

pilot 14 Table 6: Data entry and classification to define inputs or resources and outputs or outcomes

resulting from investments – example from Malta pilot 19 Table 7: Data entry and classification to define inputs or resources and outputs or outcomes

resulting from investments – example from Romania pilot 22 Table 8: Areas of consensus across a range of stakeholders and all services under review –

example from Malta pilot 25 Table 9: Areas of consensus across a range of stakeholders and all services under review –

example from Romania pilot 26 Table 10: Classifying valued outcomes and indicators, inputs according to impact levels – example

from Malta pilot 28 Table 11: Classifying valued outcomes and indicators, inputs according to impact levels – example

from Romania pilot 28 Table 12: Assumptions on staff numbers and caseload to inform scope – Malta pilot example 30 Table 13: Outcomes matrix for comparing family support and alternative care services based on

analysis of data gathered for Childonomics pilot in Malta 31 Table 14: Outcomes matrix for comparing day centres for children with disabilities based on

analysis of data gathered for Childonomics pilot in Romania 32 Table 15: Measuring selected indicators – examples based on some of the indicators in the

outcomes matrix and data available from the Malta pilot 33 Table 16: Calculating family support costs, Malta pilot (2016 data) 39 Table 17: Calculating family support costs, Malta pilot (2016 data) 39 Table 18: Calculating foster care costs, Malta pilot (estimated 2016 data) 40 Table 19: Budget and capacity: Centre 1 – Romania pilot 41 Table 20: Estimated costs for Centre 1 (EUR, 2015–2016) – Romania pilot 42 Table 21: Budget and capacity: Centre 2 – Romania pilot 42 Table 22: Costs for Centre 2 (EUR, 2015-2016) – Romania pilot 43 Table 23: Definitions of strength of evidence categories 43 Table 24: Malta pilot results matrix for illustration purposes presenting estimated measurements of

outcomes and financial data 45 Table 25: Romania pilot results matrix for illustration purposes presenting estimated

measurements of outcomes and financial data 47 Table 26: A breakdown of the general objectives of the Strategy by type of policy 59 Table 27: Cash (shaded dark blue) and services (shaded light blue) available to support children

and families in Romania 63 Table 28: Cash (shaded dark blue) and services (shaded light blue) available to children and

families in Malta mapped on to the Childonomics conceptual framework 67 Table 29: Focus group discussion with 8–12-year-olds 93 Table 30: Focus group discussion with 12–18-year-olds 94

Figure 1: Outline of the Childonomics methodology iii Figure 2: Outline of the Childonomics methodology 3 Figure 3: Example from Malta of services and programmes mapped on to part of the Childonomics

conceptual framework 6 Figure 4: Example from Romania of services and programmes mapped on to part of the

Childonomics conceptual framework 7 Figure 5: Ecomap summarising change in situation from baseline to current 108


© Oxford Policy Management xi

Box 1: Examples of inputs, outcomes and indicators described in service specifications and normative frameworks (Malta pilot) 16

Box 2: Measuring outcomes and outputs for two day centres in Romania pilot based on interviews and group discussions with staff and parents whose children use the day centre services 37


© Oxford Policy Management xii

List of abbreviations

AROPE At risk of poverty and exclusion rate

CAFR Children’s Allowance Flat Rate

CC Community centre (Malta)

CCA Cost-consequence analysis

CPD Child Protection Department (Romania)

CS Community Services (Malta)

ECD Early childhood development

CPS Child Protection Service (Malta)

EU European Union

EUR Euro

FSWS Foundation for Social Welfare Services (Malta)

IFCO International Foster Care Organisation

IFSS Intake and Family Support Service (Malta)

LACS Looked After Children’s Service (Malta)

MFSS Ministry of Family, Children’s Rights and Social Solidarity (Malta)

NAPCRA National Authority for the Protection of Children’s Rights and Adoption (Romania)

NGOs Non-government organisations

PwC PriceWaterhouseCoopers

UNICEF United Nations Children’s Fund

YPEET Young people in education, employment or training


© Oxford Policy Management 1

1 Introduction

Childonomics is a research project that has developed the first iteration of an instrument that can

help decision-makers, researchers, child care specialists, social workers, child rights advocates

and other stakeholders to reflect on the long-term social and economic return of investing in

children and families within a given national or subnational context. The instrument provides an

approach to economic modelling that can be used in a number of ways to inform decision-making.

The methodology enables consideration of the different types of costs of a range of services and

approaches to supporting children and families in vulnerable situations, or children and families

who are at risk of being in vulnerable situations, and explores their links to the expected outcomes

of using these services.

The project and methodology adopt a rights-based foundation and an outcomes focus as a basis

for understanding what investing in children and families requires in order to ensure the well-being

of children. It has a particular emphasis on supporting children, families and communities in order

to prevent and reduce any form of developmental delay and harm – and especially the

unnecessary separation of children from their parents. It is anticipated that governments, service

providers, non-governmental organisations (NGOs) and researchers will be able to use the

methodology developed through the Childonomics project in a variety of ways to support decision

making, advocacy and research. One such aim is that it will provide a framework for considering

the social and economic cost of a change in policy and investment in services in the short and long

term, and that it can thus be used as part of the planning process for policy and practice change. It

could also be used as part of a strategy for the reform of systems that do not provide sufficient

support for parents and children, and which have led to poor outcomes for children. Whilst it is

initially being developed in a European context, the longer-term aim for the Childonomics

methodology is for it to be relevant and capable of being applied internationally.

The Childonomics methodology is based on CCA, which compares the costs of implementing a

programme or policy with its consequences, such as child development outcomes, and which also

incorporates elements of social return on investment methods, especially ensuring a service user-

informed perspective on the value of the programme. This method has been selected as it best

meets the goals of Childonomics to examine the interplay of inputs and outcomes at multiple

levels, and to enable consideration of contextual factors that may be influencing outcomes. CCA

shows outcomes in their natural units rather than as a single metric and ‘it is left to decision-

makers to determine whether, overall, the [intervention] is worth carrying out’ (National Institute for

Health and Care Excellence, 2017). The methodology is explained in full in the Childonomics

methodology paper (Gheorghe et al., 2017).

This report presents the results from piloting the Childonomics methodology and conceptual

framework in Malta and Romania in 2017 and, as such, describes and analyses the challenges

and lessons learned from piloting the Childonomics methodology at an early stage of development.

This report should be read in conjunction with the Childonomics conceptual framework (Bilson et

al., 2017) and the Childonomics methodology paper (Gheorghe et al., 2017).

Why Malta and Romania? Background to the Malta and Romania Childonomics pilots

Eurochild, a network of European child rights NGOs, commissioned Oxford Policy Management

and its partners Partnership for Every Child consultancy group and International Foster Care

Organisation to design and test a method for appraising the long-term return on investment in

children and families. Eurochild’s goals in commissioning this study were twofold:



1. to generate a valid methodology for generating robust data to calculate the long-term social return on investment in children and families; and

2. to create and test a methodology that can be used by NGOs and other stakeholders to generate evidence to support advocacy for investment in children and families, and by governments to inform decision making about strategies for improving child care and family support systems.

Eurochild recognised that in order to test the methodology developed by the project there would be

a need to find partners who are strongly committed to providing data and information about child

care and family support services and their finances. Malta and Romania were chosen as the

countries for piloting the Childonomics methodology as Eurochild had a strong partnership with

MFSS in Malta and NAPCRA in Romania at the time when the project was being planned. The

MFSS had expressed an interest in being able to use the Childonomics methodology in assessing

the results from new child protection legislation (Law on Child Protection, 2016) and family support

programmes, such as the LEAP1 programme, that were being introduced at the time when the

project was being planned. NAPCRA was interested in examining the case for greater investment

in family support services, such as multi-disciplinary teams based in county child protection

services. MFSS and NAPCRA were therefore committed to providing the necessary data for the

field-testing of the Childonomics methodology during the pilots.

Purpose of the Malta and Romania pilots

The main purpose of conducting the pilots in Malta and Romania was to test the draft

Childonomics methodology, which is based on CCA and incorporates elements of social return on

investment methods (see the Childonomics methodology paper, Gheorghe et al., 2017 for details),

and thus to inform changes and adjustments to the final version of the methodology that is the

main output of the Childonomics project.

Limitations of the Malta and Romania pilots

The pilots are constrained by the limitations of the first iterations of the data collection instruments

and the methodology, which have subsequently been adjusted, and by the availability of data. As

the methodology is tested further beyond the lifetime of this project it is likely to go through further

changes and adjustments. This report focuses on illustrating how the methodology was applied in

practice, and on analysing lessons learned from applying the first iteration of the methodology in

Malta and Romania.

The results from applying the methodology in Malta and Romania do not give robust evidence for

advocacy campaigns or decision making on policy or funding. In all cases, the evidence used for

testing this Childonomics CCA methodology was mainly anecdotal and only indicative at best, as it

was based on a limited number of case studies and interviews with a non-representative sample of

managers, social workers, day centre specialists, residential staff and service users. Additional

systems monitoring data – especially for measuring long-term outcomes – were either not available

or could not be incorporated into the analysis due to lack of time and other constraints.

The data gathered were sufficient to pilot the methodology, but not to provide conclusive or even

moderately strong evidence for the connection between investments and outcomes or outputs for

any of the services included in the pilots in both Romania and Malta.

1 LEAP is the name (not an acronym) of a programme providing food support, employment services and other supports to low-income households in Malta: https://fsws.gov.mt/en/leap/Pages/default.aspx

https://fsws.gov.mt/en/leap/Pages/default.aspx



Structure of the report

This report on the pilots follows the structure and logic of the Childonomics methodology (see

Figure 2). It sets out the way in which the methodology was applied in Malta and Romania, as well

as the lessons learned from this pilot exercise and the way they have contributed to final

adjustments in the methodology. The report includes recommendations for further testing and

development of the next iteration of the methodology.

Figure 2: Outline of the Childonomics methodology

Source: Gheorghe et al, 2017

The first section of this report summarises the context in Malta and Romania, using the

Childonomics conceptual framework to conduct a policy analysis and to map services and

programmes in the two countries (presented in more detail in Annex A). The second section

establishes the scope, constraints and boundaries of the pilot in each country. The third section

sets out the data gathered to inform the application of the methodology and the fourth section

illustrates the process for selecting the outcomes that are the subject of the analysis (the outcomes

matrix). The fifth section illustrates how the financial and economic analysis can be applied and

summarised in the results matrix and discusses issues of data validity and attribution and

incorporating stakeholder feedback from the final validation step of the methodology. It also sets

out lessons learned from the pilots and offers concluding thoughts on the Childonomics

methodology and potential further developments.

1.1 Mapping and analysis of policy, programmes and services

Step 0 in the Childonomics methodology requires that the research team conducting the analysis

establishes an understanding of the context in which the methodology is being used. This step

helps to identify policy priorities of, and outcomes being sought by, the government for children,

families and communities, and at the societal level, as well as the indicators that are being used in

existing monitoring systems to measure these outcomes. It also maps the current types of

investments being made by the government in supporting children and families.



Questions that guided the mapping exercise during the pilots included the following:

1. What outcomes for children are important for the key stakeholders – government, NGOs and the academic community?

2. What programmes and services have been/are successful in achieving these outcomes?

3. Which services are considered as the priorities for development and funding, and why? What factors underpin the process of introducing or withdrawing programmes and services? How does intersectoral or inter-ministerial cooperation take place in decisions on money allocation or reallocation according to needs? What are the levels of governance involved, what are the competencies and responsibilities of decision makers, and what degree of accountability is there?

4. What are the core inputs and activities of priority programmes and services? (For example, social workers visiting families, counselling, psychological accompaniment, training/education, income maximisation, fulfilling social administration tasks.) Are there critical aspects to how services are delivered that impact on their effectiveness (e.g. working culture and core values, application of basic standards)?

What are the available data sources to inform the answers to questions 2–4?

Methods for mapping included:

policy review and analysis that incorporated key pieces of child protection and social protection

legislation, position papers, policies and strategies;

a review of published academic and ‘grey’ literature, including programme and service

evaluations, NGO reports and academic papers; and

stakeholder consultations and interviews with key informants in government, NGOs and

academic institutions.

Examples of policy priorities, outcomes and indicators identified in Malta and Romania are

summarised here. Examples of how the national programmes and services of the two countries

can be mapped on to the Childonomics conceptual framework, and more details on the national

policy priorities, are provided in Annex A.

1.1.1 Summary of policy priorities and outcomes sought by the governments of Malta and Romania at the time of the pilot

The Romanian government aims to reduce the poverty and social exclusion rate overall. In order

to do this, it is targeting social transfers and social services to particularly vulnerable geographic

areas/communities and groups, including children and families, people with disabilities, children

without parental care and Roma communities, in order to improve access to education,

employment, housing, health etc, which it is believed can help to achieve specific objectives

relating to breaking the cycle of intergenerational poverty. This approach also aims to reduce the

numbers of children needing alternative care, in support of the implementation of the National

Strategy for the Promotion and Protection of Children’s Rights 2014–2020 (which aims to reduce

by 25% the number and rate of children entering the special protection system by 2020, but which

is otherwise still very focused on de-institutionalisation of the old child care system that is still

heavily reliant on residential care).

There appears to be an extensive national monitoring system in development that is focused on

many process indicators. There appears to be a gap in disaggregation that could support

Childonomics return on investment analysis: users of special protection services and community-

based integrated services (both representing areas of significant investment by the government)

should have improved outcomes, but it is not clear whether the monitoring system will permit this



type of analysis. The outcomes for the child protection system that are recommended in the United

Nations Children’s Fund (UNICEF), World Bank and NAPCRA study may be useful for the

monitoring outcomes in a Childonomics framework once data become available, especially if they

will be disaggregated for all types of service users and not only children in the alternative care

system.

Maltese government policies on social investment generally can be characterised as latent and

redistributive, aiming to, in the case of children and families, address current situations and

‘improve chances of less stressful situations later on in life’ (Bouget, 2015). The overall goals of the

National Children’s Policy launched for public consultation2 in 2016 and specific outcomes refer to

financial or economic security and stability, both during childhood and once children reach

adulthood. A key outcome relevant to the focus of the Childonomics framework is ‘Children living in

a safe and economically stable family environment’ (National Children’s Policy, 2016, p.6). The

concept of an ‘adequate living environment’ is also important as it suggests both economic

adequacy but also adequacy in meeting children’s other needs – developmental and psychological.

These needs should be defined and measured accordingly when assessing the achievement of

outcomes.

1.1.2 Examples of indicators for monitoring policy implementation in Malta and Romania

The governments of Malta and Romania, along with the other member states of the European

Union and Eurostat, the EU statistics agency, systematically gather data on the following

standardised Europe 2020 strategy indicators:

At risk of poverty or social exclusion rate (AROPE) 3

The European Commission adopted a broad ‘at risk of poverty or social exclusion rate’ indicator,

AROPE, to serve the purposes of the Europe 2020 strategy. This indicator is an aggregate of three

sub-indicators: (1) monetary poverty, (2) material deprivation, and (3) low work intensity. This

indicator can also be disaggregated for particular sub-groups of the population: for example, in

Romania where poverty rates are significantly higher among the Roma than non-Roma

population.4

Young people in education, employment or training (YPEET)

The EU 2020 target is to reduce the percentage of the population aged 18–24 who leave education

and training early to a level of under 10%.

Monitoring the proportion of families with children at risk of poverty or social exclusion that are the

families of care leavers or other former service users (e.g. family support or child protection cases)

and / or monitoring the AROPE of former service users across different types of services, including

family support, community support and alternative care, could help to build up understanding of the

outcomes from differing investments in the long term.

Within the 18–24-year-old age group, monitoring the number and percentage of young people in

employment, education or training for each type of former service user could help to monitor the

achievement of maximum potential in education and employment, and therefore the return on

investment for each type of service and intervention in relation to this outcome.

2 The Child Policy was launched for public consultation in 2016 and redrafted based on feedback in 2017. 3 http://ec.europa.eu/eurostat/statistics-explained/index.php/Europe_2020_indicators_-_poverty_and_social_exclusion 4 www.worldbank.org/en/news/speech/2016/05/31/romanias-development-and-anti-poverty-policies

http://ec.europa.eu/eurostat/statistics-explained/index.php/Europe_2020_indicators_-_poverty_and_social_exclusion

http://www.worldbank.org/en/news/speech/2016/05/31/romanias-development-and-anti-poverty-policies



Intimate partner violence rates

The European Union Agency for Fundamental Rights reports on this indicator for EU member

states and it could provide some proxy monitoring evidence for an ‘adequate environment’ for

children, alongside child protection data.

Number and percentage of children in out-of-home care or in child protection services – stock and

flow data

Both Malta and Romania regularly monitor indicators relating to the child protection system and

children in out-of-home care, both of which can be proxy indicators for the effectiveness of the child

protection and family support system.

1.1.3 Programmes and services for children and families in Malta and Romania mapped onto the Childonomics conceptual framework

The Childonomics conceptual framework was used to map policies, programmes and services that

make up the total system of support for children and families (see Figure 3 and Figure 4). During

testing in both Malta and Romania, this step helped to engage stakeholders in a discussion to think

broadly about the different types of mechanisms being used nationally or at the local level, across

a range of sectors, which are impacting on families and their ability to care for children. It was

possible to map the existing provisions for cash assistance and services for children and families

on to the Childonomics conceptual framework and this helped participants to consider how they

may impact on policy outcomes being sought.

Figure 3: Example from Malta of services and programmes mapped on to part of the Childonomics conceptual framework



Figure 4: Example from Romania of services and programmes mapped on to part of the Childonomics conceptual framework

Source: Childonomics project

The Childonomics methodology does not require that the services and other types of assistance

are sorted into a rigid typology, but rather that the conceptual framework is used to establish the

broad existing context, both in terms of the current system of support and services and the

outcomes for children, families, communities and society that are being achieved (or being sought).

Figures 3 and 4 do not map systems and services exhaustively (and for reasons of space do not

present universal services at all), but rather serve to illustrate how the system of support can be

laid out across organisational and ministerial divides. In Malta, for example, the services specified

in the columns are delivered by a range of different government (such as the Appogg Agency) and

non-government agencies, but this mapping helps stakeholders to consider the system as a whole

rather than in the silos of institutions, agencies or organisations.



2 Description of data gathered to inform application of methodology

The goal of data gathering is to establish the context (Step 0), as set out in the previous section; to

define the scope of the CCA (Step 1); to identify and define the outcomes and indicators for the

outcomes matrix (Step 2); to measure the outcomes and outputs (Step 3); and to identify the inputs

and costs in order to define and establish the amounts of investment (Step 4) in each service.

The primary data collection methods used in the pilots were:

key informant interviews with national government specialists and decision makers, service

managers, social workers, social pedagogues and case managers;

case studies: family support cases, and interviews with care leavers and young people in care;

and

service user consultations with children in residential care, children in foster care, foster carers

and parents.

Secondary data collection included:

service statistics analysis;

review of child protection and service delivery database parameters;

service manuals/procedures/standards; and

policy analysis and service provision mapping.

The data gathering exercise focused on defining:

the goals and objectives of each service generally, with reference to examples from individual

cases;

the inputs provided by the service to the client and their family;

the results from these inputs, i.e. the outputs and outcomes;

the indicators used to monitor the achievement of the goals, objectives and outcomes.

The interviews with managers, staff and service users, and the review of service documentation

(standards, manuals, guidance for staff and manager) aimed to elicit triangulated information about

service standards that could help to define service inputs, outputs and outcomes. The key

parameters that were examined included: the caseload of frontline staff (e.g. number of cases per

case manager or per family support, foster care or residential social worker); the length of service

provision; thy intensity of casework in each type of service and at different stages of service

provision; and performance indicators for staff and services.

Data gathering instruments and guides were approved by the Oxford Policy Management Ethical

Review Board and are attached in Annex B, along with guidance for conducting consultations with

children, young people and care leavers.

It should be noted that this piloting of the Childonomics methodology was conducted by a group of

researchers that included an economist, social services specialist and child and youth participation

specialists, with support being provided by national consultants, and government and non-

government stakeholders.



Important note on data gathered and strength of evidence

In all cases, strength of evidence used for testing this Childonomics CCA methodology was mainly anecdotal and only indicative at best, as it was based on a limited number of case studies and interviews with a non-representative sample of managers, social workers, day centre specialists residential staff and service users. Additional systems monitoring data, especially for measuring long-term outcomes, were either not available or could not be incorporated into the analysis due to lack of time and other constraints.

The data gathered were sufficient to pilot the methodology but not to provide conclusive or even moderately strong evidence for the connection between investments and outcomes or outputs for any of the services included in the studies in both Romania and Malta.

2.1 Primary data

2.1.1 Key informant interviews

Semi-structured interviews were conducted in:

Malta with a number of MFSS and Foundation for Social Welfare Services (FSWS) managers, five

Appogg service managers, and 14 social workers from family support, foster care and residential

care services run by Appogg and other agencies; and

Romania with a number of NAPCRA specialists and managers, two county child protection

managers, and social workers, social pedagogues and case managers from multi-disciplinary

teams, day centres and other community-based services for children and families.

2.1.2 Case studies

Social workers and case managers in both country pilots were asked to describe:

1. the baseline situation of the child and family at the point of referral to the service;

2. the full range of inputs provided to the child and family as part of the service, including referrals to other services and;

3. the outputs or interim outcomes from the service interventions for the child and family.

This exercise helped to deepen understanding of the services and service users, and the links

between different types of services.

Malta case studies

As part of the Malta pilot 10 case studies were presented to interviewers by 10 social workers from

family support services (IFSS and five Community Centres (CC)) that are run by Appogg, one of

the FSWS service delivery agencies. All of the cases described were still open and the outcomes

were the current situation of the family at the time of interview. Some social workers used

genograms and ecomaps to present their cases and others provided only a narrative.

Five of the cases presented involved parents who themselves had experience of care and whose

children were now being referred by the child protection services for family support interventions

because of concerns about neglect, violence or abuse. One case described how the family support

service was working with the parents of two young children who have been in foster care for two

years, to support planned reintegration. Two other cases involved children who are living with

grandparents while social workers from the family support services work with parents to address

relationship issues in the family or work to formalise the care being provided by the grandparents.

In both of these cases, siblings have been separated from each other, with at least one sibling



remaining with the parents. The other two cases were presented as concerning relationship issues

between parents, or challenges in education.

Parent mental health issues and referrals to mental health services featured in many of the cases.

The family support services social workers described how they provide accompaniment services

and support families to access other services in the community, including Homestart (a volunteer

staffed family mentoring service), home-based family therapy services, and psychologist services,

as well as addressing health and education issues. The baseline, inputs, outputs and outcomes

described by social workers for these cases are summarised in Table 2 and a full case description

of one case, with ecomaps illustrating the ‘before’ and ‘after’ situation of the family, is provided in

Annex C as an illustration of how case studies helped to inform the definition of family support

inputs, outputs, outcomes and indicators as part of applying the Childonomics methodology.

Table 2: Summary of characteristics of 10 family support cases, including inputs, outputs, outcomes and indicators presented by social workers as part of the Malta pilot

Characteristics of child and family support cases 10 family cases

Age of parents at baseline 17–42 years

Age of children at baseline 0–14 years

Number of children in each family 1–4 children

Pre-history

Adults in family with child protection concerns and care experience in their childhoods

5 mothers with care experience

Other children in the family previously taken into care 2 families

Characteristics of cases at baseline

Child protection service referral (concerns about neglect, abuse, violence towards child/children in family)

4 cases

Other referrals 2 self-referrals, 3 referrals from other services (e.g. education or health)

Reintegration case 1 case – 2 siblings

Mental health issues of mother 2 women with post-natal depression; 4 women with other mental health issues mentioned

Family conflict / relationship problems 3 cases

Domestic violence 3 cases

Informal kinship care 2 cases – 2 grandmothers caring for 7 children aged 2–8 years

Homelessness or housing issues 3 cases

Education/school issues 3 cases (7 children)

Interventions

Intensive (daily or weekly) social worker accompaniment, followed by lower intensity support

Duration: 4 months – 5 years

Mental health services 6 cases

Psychologist 1

Home-based family therapy (weekly) 2

Social assistance (child allowance, unemployment allowances and/or housing support)

10 cases

Homestart volunteer (weekly) 5

Education support/referrals 3

Employment support/referrals 3



Speech therapy for child/children 1

Current situation

Improved education attendance and results 2 cases (6 children)

Improved parent interaction with child 7 cases

Improved child development or behaviour 6 cases

Improved basic care (food, clothing, hygiene) 4 cases

Siblings separated 2 cases

Improved relationships between adults in the family and among extended family

6 cases

Ongoing support still needed (including mental health services)

10 cases

Romania case studies

During the Romania pilot some case presentations were made by case managers from a child

protection department. These were largely unstructured narratives, although questions were put in

a structured way. It was difficult to elicit a clear idea of outputs and outcomes from these

respondents. Other cases presented in a day centre elicited more structured narratives from some

specialists, who provided some examples of outcomes: for example, a physiotherapist who said

that her exercises with a boy with disabilities helped him to develop new motor skills. In another

example a specialist said: ‘the child could not eat by himself. Due to regular exercises and the

provision of a special technical aid, the child can now eat by himself.’

During interviews with specialists structured around the discussion of case studies other examples

of outcomes that were mentioned included the following:

‘develop skills for independent living for a child with severe disabilities’;

‘reintegrate a child into the biological family’;

‘develop motor and learning skills to be able to attend school’;

‘teach parents how to look after and develop [teach] children with disabilities’; and

‘prevent separation of a child and provide adequate education’.

Documenting case studies was a useful method for gaining an understanding of the nature of each

service included in the pilots and the way it links to other services in the system, and for identifying

inputs, outputs, outcomes and indicators. It is worth noting, however, that more structured case

presentations (including, for example, genograms and ecomaps, as illustrated in Annex C) elicited

more useful data than the less structured narratives that resulted from semi-structured interview

questions.

2.1.3 Consultations with service users

Children in out-of-home care were consulted in groups using play and other age-appropriate

methods and activities (see Annex B for the data collection instruments, including guides on

consulting with children and young people).

Children and young people in out-of-home care

The aim of consulting children in out-of-home care was to explore the subjective experience of the

formal system, as well as exploring what children perceived to be the purpose or value of the

intervention and its outcomes.



Children in out-of-home care were recruited by local gatekeepers (social services) to participate in

two focus groups in Malta, one for children in residential care and one for children in foster care,

and two in Romania, one for girls in residential care and one for male care leavers living in a

transition centre.

Table 3: Breakdown of child participants in focus group discussions

Type of current care placement

Participants Age range

Malta

Residential care 6: 3 girls and 3 boys Between 8 and 17 years of age

Foster care 4: 3 girls and 1 boy Between 8 and 10 years of age

Romania

Residential care 4: 7 girls and 2 boys. Between 16 and 23 years of age

Reflecting on the process of consultation with these groups of children and young people, some

learning points are as follows:

1. Age is not always the best aggregating factor for undertaking a group activity: in the Malta and Romania consultations there was a wide range of abilities as regards concentrating on topics and tasks.

2. A one-off focus group activity for 90 minutes might not be the best way to consult children about this specific question. For one group, the children did not know each other well, and while they happily participated, the facilitator felt that this lack of cohesion (and the time constraint) meant that they did not have time to reflect or muse on the questions. The other focus group included a sibling group and some children who were known to each other before the activity. However, because they did not have consistent contact, some children chose not to engage with the activity but rather to connect with each other outside of the suggested activities. Perhaps this type of consultation can work best with children who know each other as part of an established group, or else considerable time or multiple facilitated meetings will need to be planned to establish a productive shared forum for communication, in order to ensure meaningful participation.

3. It was necessary to adapt the planned activities and respond to the children’s interests and responses during the consultations. The skills and capacity of the facilitator are key to achieve this.

4. The minimum requirements for consulting with children and youth set out in the guidance given ahead of the fieldwork were not met, which had an impact on the methodology and how some children and youth participated.

During two of the sessions adults (carers and social workers) remained in the room with the

facilitators and the children and young people. The sessions where this happened resulted in the

least amount of information or stories being offered by the children and youth. Also, the children

and young people subverted the expectation that they would talk in front of carers and social

workers by doing one of the following: taking their group task and completing it individually in their

rooms and not wanting to discuss afterwards in a group setting; or suggesting an individual

interview rather than a group activity.

Consultations should be held at a venue away from the usual living space of the children. One-on-

one interviews held in a child or youth’s bedroom should be avoided for reasons of safety, for both

the facilitators and the children/youth, and due to considerations of research ethics. The minimum

requirements and guidance included in Annex B are compulsory.



Please see Annex B for a description of the roles and responsibilities of the lead facilitator and co-

facilitator, as well as guidance for consideration when designing child consultations for the purpose

of Childonomics analysis.

Nevertheless, the data gathered, in spite of their limitations and the constraints experienced in

conducting interviews and group consultations, have contributed to the defining of outcomes,

inputs and indicators set out in the next step of the methodology.

Care leavers

Semi-structured interviews were conducted for the pilots with eight care leavers in Malta and seven

young people in Romania who had mixed experience of care, with episodes in foster care and / or

residential care, as well as, in the cases of five respondents, periods of reintegration with parents

and subsequent further placements in residential or foster care. The interviews were conducted

with an interview guide asking about their ‘pathways’. Pathways is an approach that looks at the

various moments when decision making has an impact on an individual’s life and interactions with

social services (see Annex B for the interview schedule and the matrix used to lay out the data).

Two of the respondents in Malta, both now 29 years of age, had experienced initial placements in

residential care as young children and then entered foster care between seven and 10 years of

age for long-term placements through to reaching adulthood. Both were supported well into young

adulthood by their foster carers. One respondent had moved between residential care placements

10 times.

In Romania several of the young people interviewed were still transitioning out of care. In both

pilots the care leaver interviews were conducted with only a small sample of young people and

adults and cannot be considered as representative of any particular group or experience. They

were useful, however, for testing the challenge of attributing particular outcomes or outputs to

specific inputs.

Table 4 and Table 5 summarise information about the care pathways and current outcomes for 15

respondents, illustrating how generally positive outcomes, with many of the respondents currently

in employment or education and some caring for their own children, have emerged from complex

pathways through care. The information speaks to the resilience of the individuals concerned as

much as it does to the quality of the services they have encountered or their experiences in care.

Table 4: Summary of care experience pathways, reported by eight young people in Malta pilot

Current age at the time of interviews Between 19 and 29 years of age

Gender 6 female, 2 male

Experience in alternative care placements

Entry into the alternative care system

Entered care as a baby5 6 young people

Entered care before the age of 10 years

2 young people

Number of placements6

Total number of placement changes before they aged out of care

2 placements 1 young person

3 placements 4 young people


10 or more placements 1 young person

5 Some young people did not know the age they were when they came into care but used indicative phrases like: ‘I was a baby’ or ‘I wasn’t walking yet’. 6 Average placement changes per person were 5.6, with 45 placement changes in total across all interviewees.



Type of care experienced

Was in foster care 3 young people

Was in residential care 8 young people

More information about their experiences with their birth family while in care

Was separated from siblings in care 5 young people

At some stage returned home but came back into care

4 young people

Reports positive experience of contact with birth parents/family

2 young people

Reports negative experience of contact with birth parents/family

6 young people

Educational attainment

Self-reported level of formal education achieved

Completed secondary-level exams 4 young people

Did not complete secondary-level exams

3 young people

Started but did not complete tertiary level studies

2 young people

Currently in education 2 young people

Has completed tertiary education course

1 young person

Current status

Self-reported information about their lives currently

Employed 5 young people

Unemployed 2 young people

Recovering addict 2 young people

Is a parent 4 young people

Table 5: Summary of care experience pathways, reported by seven young people in Romania pilot

Current age at time of interview 16–23 years of age

Gender 5 female and 2 male

Experience in alternative care placements

Entry into the alternative care system

Entered care as a baby 1 young person

Entered care before the age of 10 years

6 young people

Number of placements7

Total number of placement changes to date

2 placements 1 young person




Type of care experienced

Was in adoption 1 young person

Was in foster care 1 young person

Was in kinship care 1 young person

Was in residential care 7 young people

More information about experiences with birth family while in care

Was separated from siblings in care 4 young people

Has contact with siblings 2 young people

Wants contact with parents, sibling or other family members

3 young people

Educational progress or attainment

7 Average placement changes per person were 3.7, with 26 placement changes for all interviewees



Self-reported level of formal education achieved

Has problems at school and has repeated a year or entered special education

3 young people

Is in high school 3 young people

Is in tertiary education (college) 2 young people

Wants to continue with education and get a qualification

5 young people

Has additional psychologist or counselling support (either at school or in the care setting)

4 young people

The interviews with care leavers can be seen as having produced a set of process indicators about

experiences in care that are valued by the young people. For example, the interviewers did not ask

questions initially about contact with, or separation from, siblings while in care, but this issue came

up during the course of the first interviews and questions were subsequently added to the interview

in order to ensure later interviews captured information on this issue.

Foster carers in Malta

Focus group discussions were conducted with six foster carers in Malta who have experience of

providing both long- and short-term care to children, where the child was aged from one month up

to five years of age at the time when the child was placed with them. The length of placements

ranged from one week to 10 years at the time of the consultation and nearly all were under care

orders rather than voluntary placements.

Parents of children with disabilities and parents of very young children in Romania

Focus group discussions were conducted in Romania with eight parents of children with disabilities

who have children using the services at one of the day centres where data were gathered for the

Childonomics analysis, and with six mothers at a creche for babies and infants who are referred by

the child protection department as they are assessed as having additional needs.

2.2 Secondary data

Document review: reports, evaluations and databases

In Romania the documents that were reviewed included an NGO report on the financial impact of a

reform of services (Hope and Homes for Children, 2013) and a financial impact assessment of

scaling up multi-disciplinary teams(PwC and UNICEF, 2016) . The team also reviewed the

Information Management System, AURORA (www.auroraproject.ro), which has the potential to

provide very detailed baseline, inputs and outcome/outputs data for users of multi-disciplinary

community-based services. There were not enough longitudinal data, however, to use information

from this database at the time of the pilot.

In Malta, annual reports on servicesfrom the Appogg agency (Appogg agency, 2015a and 2016a)

and the parameters of the child protection database managed by FSWS were reviewed. Studies on

children aged over 16 years in care (Appogg agency, 2016b), low-income households (Caritas

Malta, 2016) and children in specialised foster care were reviewed (Appogg agency, 2015b) as

well as service manuals and procedures for a range of services, and a new case management

system and outcomes monitoring framework being prepared by FSWS.

http://www.auroraproject.ro/



Box 1: Examples of inputs, outcomes and indicators described in service specifications and normative frameworks (Malta pilot)

The IFSS mission statement and key objectives can be interpreted as seeking the following outcomes: improved child / parent relationships; strengths-based assessments of home and family situation help service users to define and achieve their goals. Measurement would rely heavily on assessing improvement in the child/parent relationships and on the client’s own assessment of whether their goals have been achieved.

Looked After Children’s Service (LACS) Manual, 2014: principles, activities and mission statement can be understood as seeking the following outcomes: the needs of children in care are met and their best interests determined and pursued; children leave care able to live independently, or to return to their families who have been helped to better meet their needs.

There are a number of measurable, immediate outputs defined in the manual that are being purchased with the investment in the service, such as: the child’s voice being heard and their views being represented when decisions are being taken; siblings kept together; contact with family maintained; and involvement of parents facilitated.

Residential care (summarised from procedural manuals provided by Appogg agency and Kummissjoni Ejjew Ghandi)

The outcomes being sought include: optimised growth and development; achievement of personal goals and increased motivation for new learning; children supported to regulate emotions and maintain positive relationships with adults; healthy development and life success; and improved resiliency, self-sufficiency and self-concept.

The outputs or interim outcomes being sought include: competency in problem-solving, critical thinking skills and emotional regulation skills; constructive family contact; and activities that are appropriate to each child’s development level across all functional domains.

The inputs include: key workers and other staff implementing individualised care plans that create an engaging and supportive milieu; purposeful and goal-oriented interactions and activities; trauma-sensitive responses to challenging behaviour; healthy attachments and trusting, personally meaningful relationships with children in their care; and partnership with families.

Indicators fall out of the service standards as outcomes, outputs and inputs and include for example: developmental measures; measures of resiliency, skills and competencies; process indicators, such as the individual care plans and other procedures that underpin the application of the principles and help to achieve the desired outcomes and outputs.

In both pilots there were no existing studies that provided the data that were required to first

identify outcomes (and indicators) for children, families, communities and society, and then to

identify the inputs from specific services that could be directly attributable to achieving those

outcomes. The general lack of rigorous evaluations meant that the application of the methodology

was limited by the data that were gathered during the fieldwork for the pilots, and from the reports

and documents reviewed.

In both pilots there are extensive databases available that have the potential to provide more

granular data linking investments with longer-term outcomes for children. For example, the Appogg

database contains data on service users that have been accumulated over more than 25 years and

that have the potential to provide information about intergenerational use of out-of-home care

services, child protection and family support services, or to compare outcomes from community

family support services and the national IFSS. The database is not configured to provide this

information, however, and further detailed research would be required in order to mine the

database to answer questions about long-term outcomes.



A note on multi-disciplinary teams in Romania:

Interviews with key informants and the document review conducted during the Romania pilot indicated that multi-disciplinary teams represent a potentially important new investment by the government of Romania in supporting children and their families in the community. UNICEF Romania commissioned a financial impact analysis of scaling up the service. The report suggested that there are existing services with identifiable costs that can be scaled up. It was not, however, possible to interview staff from these services during both field visits to Romania for the pilot.

It appears that the multi-disciplinary teams are not a clearly identifiable standalone service, but rather a collection of services that is understood in different ways by different local authorities. In one local authority,8 for example, the multi-disciplinary team is a group of professionals who are mandated by a regulation to work together on particular cases, but in practice do not actually collaborate in the way they have been instructed. These professionals are working in their own professional setting (school, health system, social care system) and do not necessarily have additional resources to be able to dedicate time to working in the multi-disciplinary team.

With more time, the pilot in Romania might have been able to identify a functioning team and include it in the scope of the analysis, but within the timeframe and resources available to the project, the multi-disciplinary teams could not be included.

In future, however, the AURORA database, which is gathering data on households with children at community level, could provide a rich source of granular data for linking interventions by multi-disciplinary teams to specific outcomes being sought in relation to poverty reduction, child protection and social inclusion.

A note on LEAP in Malta

LEAP is a new government programme piloted in 2014 and rolled out in 2016–2017 that aims to address intergenerational poverty and a culture of unemployment and benefits dependency. The programme aims to work at the community level through regional programmes to take into account cultural and social specifics in each region. Success measures for LEAP managers and workers include:

• improved literacy;

• lower rates of early pregnancy;

• reduced unemployment; and

• reduced mental health problems and addictions.

The programme aims to assess the strengths and weaknesses of each household living below a certain income level and to support household members to access the employment, child care or other services they need to be able to graduate into employment. If the programme is successful, there should be fewer clients requiring more specialised family support services. The LEAP programme conducts assessments of all households at the point of entry and these are automatically stored in a database.

In future, it should be possible to compare the outcomes for households of families with children compared to baseline, in terms of household income, literacy and other key indicators. If these outcomes indicators can be harmonised across all services then it should be possible to track the progress of different groups of services users and to identify the contributions that different types of services are making to achieving certain types of outcomes. At the time of the pilot in Malta, not enough data were available to be able to include LEAP in the CCA.

8 Interview with a social worker who is part of a multi-disciplinary team, 2017.



3 Analysis of qualitative data

The data gathered from interviews with service managers and staff and with service users were

analysed to inform the selection of outcomes or outputs and identification of inputs for the

Childonomics CCA in both pilots.

In qualitative research, a three-step approach to aggregating raw data is often undertaken: reduce,

display and verify (Dey, 1993; Miles and Huberman, 1994; Miles, Huberman and Saldana, 2014).

Data reduction refers to the process of reducing raw data to data that are relevant to the question.

Data display is a continuous process of reorganising the data to show connections that are

relevant to answering the question being asked. The third and final stage of data analysis is to start

developing conclusions addressing the research aim: initial conclusions can then be validated,

references made to the data collected and gaps and areas for further or future data collection

identified.

Qualitative data from interviews, case reviews and focus group meetings with service users,

service workers and managers elicited a range of perspectives about the main research questions

put during the data gathering exercise (described in Section 2 of this report):

What are the outcomes that the service is aiming to achieve?

What are the inputs that are intended to achieve these outcomes?

What are the actual results from these inputs (i.e. outputs, interim outcomes and longer-term

outcomes)?

How do you know that the outcomes have been achieved (i.e. what indicators are used to

measure the outcomes)?

Annex B sets out the data collection instruments and the interview and discussion guides that were

used to test the methodology in Malta and Romania for the Childonomics project. The narratives

that resulted from the interviews were analysed using usual qualitative data analysis methods; this

involved sifting, sorting and classifying the statements of respondents as inputs, outputs,

outcomes or intended outcomes and indicators and arranging them according to the type of

respondent (service user, staff or manager) as summarised in Table 6 and Table 7. This was the

‘reduce’ step.



Table 6: Data entry and classification to define inputs or resources and outputs or outcomes resulting from investments – example from Malta pilot

Respondent Inputs Outputs or interim outcomes

Achieved outcomes Targeted (planned) outcomes

Indicators

Family support

Service manager

General support, social worker accompaniment, home visits, material aid, psychologist, home-based family therapy, Homestart, referrals to other services (employment, LEAP, psychiatrist, health services), legal support, support in claiming benefits

Increase in formal and informal support

Increased stability in family relationships

Increased household income and improved housing

Improved parent care provision (i.e. level of meeting needs of the child)

Managing risk to child at community level and empowering parents to do something better for their children

Improved stability in relationships between adults in the family and between adults and child

Prevent further problems (‘what long-term impact will be if the child has no need of services as an adult')

Reducing level of contact with social worker is indicator of increased stability of family situation

Client initiating contact and seeking support is an indicator of trust/quality of the service

Confidence that client is getting along independently is a case closure criterion

Reduction in referrals to child protection services

Service user9

Community support worker: ‘when I have a problem, I call her … she does home visits or I can come here.’ ‘She is very understanding which is what I need.’

‘It is mostly talking. She talks with my daughter also.’

Access to other services psychology, family therapy.

‘we have less arguments and if we argue it is less intense. I feel supported.’

‘I’m finished. I’m tired.’ Is considering placing her daughter in care

‘Now I feel stronger to talk it out. It is about calming the situation before it gets worse.’

‘I can look forward to a better relationship with her. I feel more empowered as a parent. I look forward to much improvement.’

‘I don’t know what she wants to do.’ ‘She used to draw.’ ‘She wants to have a job.’

‘It has affected my work. I have no social life.’

‘My daughter’s health has also been affected.’

9 No service users were interviewed at the time of the main fieldwork, but one service user parent was interviewed during the validation process and the data on this have been included here.



Service staff

Support with relationship issues, counselling, help parents to acquire skills

Establishing trust with client

Housing grant, day centre for mother with mental health issues

Improved relationship between child and parent, between other family members, reduced violence in family

Improved child development

Improved mental health of mother / father Mother / father gain skills and in employment

Address trauma from own childhood (of parent)

Reintegration of child from foster care to parents

Child: ECD milestones, pre-school and school attendance, grades/achievement, improved behaviour

Parent: Better attendance at appointments (health, employment etc)

Community: better outreach to families by health, education service

Residential care services

Service manager

Individualised perspective

Systemic perspective

Relationships of trust

Providing a safe place

Supporting sibling and other family relationships

Good quality relationships

Education achievements (incremental) – better attendance, better organised, doing homework, doing better

Emotional containment (e.g. less acting out/processing trauma)

Independent living

Return to parents

Placement into foster care

Adoption

Maintain/build relationships with parents and keep siblings together

Child well-being

Enabling them to succeed in life

Child in residential care has an assigned social worker but the family may not have a social worker supporting reintegration

Stability of relationships

Staff behaviour in relation to children; quality of community with children

Professionalism and knowledge of staff – ability to negotiate and understand why certain behaviour happens

Service user

Good staff should teach independence'

'Need more birth family support'

He feels free, he visits his family every weekend and feels safe there

‘There are homeless care leavers and they all need help.'

'Go back to mother when I finish studying.'

'Carers that listen, that are kind and helpful, that play football with you.'

'More freedom at weekend.'

'Children should be allowed to pick their own haircut.'

‘Help them plan and stay with them for the plan. Don't throw them away.'



Service staff

Personal emotional investment ('we live their pain')

Investing in attachment

Time investment by staff

Emotional stability Daily successes (not always compared to baseline)

Knowing where to go when they have a crisis as young adults

After care (e.g. supporting maternity appointments)

Prepare for independent living

Get an education (attend school)

Safe and healthy environment for children so they have a secure base where they feel protected, love

Prepare for reintegration

Trust in the social worker

Stable staff

Maintaining boundaries/supervision

Involving children in care planning and decision making

When staff get demotivated and burned out, the quality of care goes down

Foster care services

Service manager

Recruit, train, assess (including home study that requires frequent visits and takes five to six months to complete)

Continuous training and support for foster carers

Children in the care of people who have been assessed and trained to meet their needs according to a care plan

Adoption, ageing out or full reintegration

Individual care plans, but not always possible 'it is crisis intervention rather than planning'

Social worker is assigned to monitor child in foster care but the family may not have a social worker assigned to support reintegration

Service user

'Social workers should talk with us, talk about our feelings we have inside.'

'The school helps me. Everybody helps me. I want to go see my mother on Saturdays.'

'I feel safe.'

'One of my biggest regrets is that I'm not close to my biological siblings. The youngest one was placed into adoption and we've never met'

Still supported by foster family through college

'Now I'm self-employed, I have a qualification and I'm happy with my life.'

'I want to see my mother, I miss her the most.'

'I want to be adopted [by my foster carers].'

'My social worker she always cancels that day [the same day the appointment is made for] with only a few minutes notice. I don’t see her that much.'

Service staff (foster carers)

Every child has his or her own situation

We need more information about the child and his or her health

Foster care allowance but no clothing allowance or children's allowance

Foster care provides security

Improved health and development Increased skills

Children make significant progress, feeling loved and building positive and safe relationships

Return to mother

Placement in residential care

The plan is for reintegration but decisions seem to be driven by parents’, not children's, needs

Children are being left for too long in dysfunctional, abusive families

The biggest challenges are dealing with the behaviour

We expect to be valued and to have consistency



Play therapy is subsidised, we paid for behaviour therapy and it has not yet been reimbursed

I had to reduce my work hours in order to be able to take children to appointments (e.g. paediatrician, speech therapist)

and reliability from the social workers

Table 7: Data entry and classification to define inputs or resources and outputs or outcomes resulting from investments – example from Romania

pilot

Respondent Inputs Outputs or interim outcomes

Achieved outcomes Targeted (planned) outcomes

Indicators

Day Centres 1 and 2

Service manager (day centres and creche)

Attitude of parents: ‘parents are educated and do everything they can for their child’ (Day Centre 2 – ‘parents just bring their children and leave them here, they don’t engage very much with us or with their children’)

‘We think that the children are progressing once they are placed here (usually for three years)’

‘We can see families are doing better when they have more income’

Service users (parents from Day Centre 1 only)

Psychological support at the centre and at home

Parent group support

Individual therapy (including education, behaviour, speech, physiotherapy)

Counselling for parents

Home visits and centre-based services

We meet other parents

We can learn how to support our child in his own home environment

The parents were at risk of divorce/separation ‘because of the stress of having a child with disability’ but now ‘we are better as a couple and as parents’

‘My child now knows how to better control his behaviour and to say short sentences, he is more relaxed and happy’

‘My child is more confident and has acquired independent living skills, dressing and eating by himself’

Indicators of quality of service:

Ease of access ‘it is very easy for us to come here’

Professionalism of staff

Friendly environment ‘this centre is like a second home for us’



Mothers are receiving allowances as carers and fathers are working (in most cases)

‘I have learned how to be a better parent, to communicate with and understand my child, I am a more confident parent.’

‘We now have hope and confidence for the future.’

‘We are ready to integrate our child into mainstream school.’

Attitude to parents: ‘respect and care for our needs as a family.’

Parent as partners: ‘we are part of decisions about our child.’10

Free services (parents do not have to pay for therapy)

Service users (parents from creche)

Free service ‘the biggest help is that this is for free, because we pay rent’.

The centre provides clothes, shoes and other necessities

Trips and outings ‘that we could never afford’.

Parents have to prove eligibility every three months: ‘the big issue is that we need to redo the papers every three months to prove we still need this, yes but this service is important so it is worth it.’

‘One of my girls couldn’t speak very well before – she made progress here.’

‘The children are learning a lot and get used to playing with other children.’

Parents can work: ‘’I couldn’t go to work until I found this place.’

Working individually with each child (‘this is not happening’).

Seeing progress in child’s learning (e.g. learning to write)

‘We notice the development is very good.’

Service quality indicators: flexible times for arriving and pick up, open until 7 pm

Staff qualifications (only one educator with a degree)

Access (transparent process): ‘child protection only inform the ones they want to, not everyone. The truth is that I would wait one year to access this.’

Service staff

Informational support to families with children with disabilities

Prevent child/family separation

Supporting children to reach their maximum potential

10 This point was further emphasised and given strong priority by parents during the validation workshop.



Adequate education for children with disabilities

Changing attitudes and mentality about children with disabilities

Integrate children with disabilities into mainstream schools

Social integration of children with disabilities



Case review data have been treated as representing the perspective of ‘service staff’. Only

statements made by service users have been entered as data from service users.

The process of sifting through the statements of respondents resulted in: (i) identifying groups of

outcomes that are referred to consistently by all respondents; and (ii) groups of outcomes that are

referred to exclusively by one or other group. The next step is weighing the extent to which there is

consensus on any given set of outcomes and assessing the value that is given to each group of

outcomes by respondents.

This is not an exact science as qualitative data are always subject to interpretation and

consideration has to be given to the dominance of certain narratives in particular groups (e.g.

staff/managers, as compared to service users) and subjective bias (on the part of the researcher

as well as the respondent). Coding systems can help to sift and group statements in order to

identify areas of consensus and diversity. In this case, data were entered into Excel worksheets

and reviewed by service, by respondent, and by theme or type of question (on outcomes/

indicators etc).

The statements can be grouped into areas of consensus and into outcomes, inputs and indicators

that are valued by a range of stakeholders across a range of services, as summarised in Table 8

and Table 9 – part of the ‘display’ step.

Table 8: Areas of consensus across a range of stakeholders and all services under review – example from Malta pilot

Themes/groups of issues

where there is consensus

across stakeholder groups

Types of potential indicators

mentioned by a range of

stakeholders

Inputs to be costed mentioned

by a range of stakeholders

across all services

Quality of care – 'meeting

needs and ensuring well-

being' of children

Individual approach to each child

and family (group of siblings)

Staff time and service standards

(residential social workers, social

workers / caseload, timeliness and

quality of support to children and

carers)

Well supported carers (residential

staff, parents, foster parents,

grandparents)

Specialists (psychologist,

therapists, paediatrician,

psychiatrist)

Quality of child

relationships with carers,

parents, siblings, friends

Safety, development, levels and

quality of contact

Foster carer time (allowances,

respite care)

Parent training; mentoring; home-

based family therapy

Mental health diagnosis and

treatment of parent/child

Running costs (office, premises,

cleaning, maintenance, cooking,

food, administration and

management)

Additional support services:

supervised visiting services, Child

Protection Service (CPS), LACS,

Homestart, psychologist, home-

based family therapy



Key outcomes to emerge

with consensus among

stakeholders and reflected

in policy / procedures

Potential indicators Comments

Reintegration with birth

family

Number of children returning

home and length of stay/number

of repeat referrals to CPS and

LACS

This is an output that is specific to

the alternative care system

Children prepared so that

independent living will 'be

a success'

YPEET, housing, quality of

relationships as adults, literacy,

integration into the community

All of these outcomes can apply to

children receiving a range of

services, including children living

with their families and children

living in alternative care

Improved health and

development of child

During and after service inputs

(i.e. as children and adults): age-

and ability-appropriate

developmental milestones

(including ECD), progress in

health/communication/speech

indicators, mental health/

emotional well-being

Reduced intergenerational

use of services

Number of users of CPS,

IFSS/CC, residential care, foster

care services using CPS,

IFSS/CC social services and

alternative care services for their

own children

Longitudinal data are available for

many types of service in Malta


poverty and exclusion

AROPE disaggregated for

different service users at different

stages of adulthood

The LEAP programme database

will have the potential to generate

longitudinal data that can be used

to monitor these indicators

Table 9: Areas of consensus across a range of stakeholders and all services under review –

example from Romania pilot

Themes/groups of issues

where there is consensus

across stakeholder groups

Types of potential indicators

mentioned by a range of

stakeholders

Inputs to be costed mentioned

by a range of stakeholders

across all services

Child development –

improvements in

development,

socialisation, behaviour

Individual approach to each child

and family

Involvement of parents in

partnership for decision making

and therapy

Staff time and service standards

(mobile teams/home visits, parent

group, quality of support to children

and parents)

Parents’ improved understanding

of child’s condition and skills in

communication and care

provision

Specialists (psychologist, speech,

physio, behaviour and education

therapists)



Parent / family ability to

provide care

Parents’ confidence in the future

Reduced stress in family

Parents’ ability to work

Increased household income

Flexible opening (including

evenings), qualifications of staff

Carer allowances and disability

subsidies

Running costs (offices, therapy

rooms, transport, communications,

premises, cleaning, maintenance,

cooking, food, administration and

management)

Key outcomes to emerge

with consensus among

stakeholders and reflected

in policy / procedures

Potential indicators Comments

Integration into education

and society of children and

young adults with

disabilities

Number of children entering

education (mainstream or

specialised); education attainment

and YPEET

Disaggregated by disability/ability

age and gender through into young

adulthood

Families staying together

and providing better care

Number of child service users

entering residential care or other

alternative care

Improved development of

child


(i.e. as children and adults): age-

and ability-appropriate



communication/ speech, social

skills and behaviour

Reduced poverty and

exclusion of children and

young adults with

disabilities, and their

families

AROPE for children with

disabilities and young adults with

disabilities

There is room for extensive debate on how to define an indicator, an input and an outcome. The

final phase of the methodology, the validation workshop, facilitates stakeholder participation in

analysis and decision making about classifying the areas of consensus, giving them value and their

inclusion in the final outcomes matrix. Tables 6–11 illustrate how this process of ‘displaying’ the

data, and the sifting, classifying and analysing can work, based on examples from the Romania

and Malta pilots, with inputs potentially emerging as indicators and outcomes becoming outputs, or

vice versa, as the analysis progresses.

Ultimately, analysis of the data is finalised by sorting statements in each column to identify groups

of outcome statements related to the children using the service, other family members, the

community where the service is operating, and wider society or national policy objectives, as

summarised in Section 1 of this report. These four levels correspond to the Childonomics levels for

determining the outcomes that can accrue from investments in child and family services, and can

be transferred into the outcomes matrix, as set out in the Childonomics methodology.



Table 10: Classifying valued outcomes and indicators, inputs according to impact levels – example from Malta pilot

Level

Inputs

Outputs or

interim

outcomes

Targeted (planned)

outcomes Indicators

So

cie

ty

Policy, legislation,

standards, budget for

services

Reduced

intergenerational

poverty and

exclusion

AROPE, disaggregated for

different service users at

different stages of adulthood

Co

mm

un

ity

Services: staff, quality

standards, network

Reduced

intergenerational use

of services

Number of users of CPS,

IFSS/CC, residential care,

foster care services using CPS,

IFSS/CC social services and

alternative care services for

their own children

Fam

ily

Staff and specialists,

material support,

referrals to other

services

Quality of care –

'meeting needs

and ensuring

well-being' of

children

Carers provide care

that meets individual

needs of children

Repeat referrals to CPS,

reduced risk to children,

reduced incidence of violence in

the family, parents attending

appointments/ acquiring skills

Ch

ild

an

d

Fam

ily

Individual approach to

each child and family

(group of siblings)

Keeping siblings

together,

maintaining

contact with

family members

Quality of child

relationships with

carers, parents,

siblings, friends

Safety, development, levels and

quality of contact

Ch

ild

Well supported carers

(parents, foster carers,

residential care

workers) – staff,

specialists, social

assistance/ adequate

funding

Mental health

diagnosis and

treatments of

parents/child

Improved health and



(i.e. as children and adults):

age- and ability-appropriate



health/ communication/ speech

indicators, mental health/

emotional well-being

Children prepared so

that independent

living will 'be a

success'




Table 11: Classifying valued outcomes and indicators, inputs according to impact levels – example from Romania pilot

Level

Inputs

Outputs or

interim

outcomes

Targeted (planned)

outcomes Indicators

So

cie

ty

Policy, legislation,

standards, budget for

services and benefits

Integration of

children and young

adults with

disabilities


different service users at

different stages of adulthood



Co

mm

un

ity Services: qualified staff,

quality standards,

transport, carer

allowances, disability

benefits

Children with

disabilities being

cared for by their

families

Numbers of children with

different disabilities entering

residential care

Fam

ily

Staff and specialists;

material support;

referrals to other

services

Parents are

informed and

supported to

reduce family

stress and

provide better

care

Parents provide care

that meets individual

needs of children

Parent self-assessment on

relationships, child progress,

confidence and future outlook

before, during and after service

provision

Fam

ily

Parents are

able to work

Increased income for

families

Household income before,

during and after service

provision

Ch

ild

Individual habilitation

and confident parents –

staff, specialists, social

assistance/ adequate

funding

Improved health and



(i.e. as children and adults):

age- and ability-appropriate


(including ECD), progress

communication/ speech

indicators, behaviour, social

skills

Children

prepared for

education

Children prepared for

independent living

YPEET, integration into the

community as children and

young adults

Tables 6–11 illustrate how some outcomes (e.g. for children) can be achieved only with inputs (well

supported carers or confident parents) that are in themselves also indicators of outcomes or

outputs at other levels (family – quality of care meeting needs and ensuring well-being of children).

At the pre-validation stage, these tables were only drafts as the process of analysis, sifting,

classification and allocation was subsequently informed by stakeholders and reflected in these final

versions. Researchers can guide and facilitate the process, but determining whether certain

outcomes are valued and shared by stakeholders across a range of services can only be done by

stakeholders themselves.



4 Applied methodology - detailed costing and CCA in Malta and Romania pilots

4.1 Step 1: Determine scope

In Malta, based on the context analysis, interviews conducted and resources available to the

project, a decision was taken by the research team to attempt to undertake a whole-system

comparison for specialised family support services (IFSS and CS) that cover the whole country and

two types of alternative care services (foster care and residential care). This involved costing

associated services, such as psychologists, the home-based family therapy service, CPS and

LACS, as well as core services. A review of service delivery data from Appogg agency (Appogg

agency, 2015a and 2016a) summarises the assumptions that can made in terms of caseloads for

associated services (Table 12).

Table 12: Assumptions on staff numbers and caseload to inform scope – Malta pilot example

Service users reported in Appogg Yearly Statistical Reports (except

where noted otherwise) 2014 2015 2016

Staff numbers reported by service managers or in normative documents

Caseload = clients

per social worker

Social

workers Manage

ment

IFSS 1,743 1,766 15,5 1 114

(CC) (number of cases) 854 863 14 5 62

Foster care service (number of foster carers)

239 216 193 10 1 19

Foster care allowance recipients (according to MFSS)

215 204

LACS 434 451 24 1 19

Siblings Community Home 8 8 8 10 1 0,80

Supervised Access Visits /Courts 397

Homestart (estimate based on Annual Report 2013)

100 100 100

CPS 1,592 1,284 25 1 51

Source: Appogg agency, 2015 and 2016 annual statistical reports, MFSS, service manuals and interviews with staff and managers of services

An assumption has been made that all children in foster care or residential care are clients of

LACS and of the CPS, and that half of IFSS and CC clients are families with children and a third of

those are also CPS clients. All Homestart clients are also IFSS or CC clients. The proportion of

children in care using supervised access visits is not known and has not been factored into the

analysis. The numbers of children and families using psychological support services, mental health

services and home-based family therapy were also not available and have not been factored into

the analysis for piloting purposes.

In contrast, in Romania, based on the data made available to the pilot, the research team decided

to compare similar, specialised, community-based family support services in urban and rural

settings to see if the Childonomics methodology is sensitive to micro-level comparisons, and if it is

possible to compare investments and outcomes for services working with different target groups –

e.g. young Roma men who are dropping out of school, children with disabilities, babies and infants



from low-income households. In the end it was only possible to include two day centres for children

with disabilities as financial information was received for only these two services, out of the four

services that were initially included within the scope of the pilot.

Time horizon

The time horizon selected for both pilots for costing was one year of serving a client. Costs and

service data (stock and flow of service users) were requested for multiple years in order to assess

whether there are significant changes year to year, but only costs for a single year were included in

the final calculations of Step 4 of the methodology.

Some process indicators, such as caseload (number of cases per social worker) or attendance at

school, that measure immediate service outputs were considered, but the whole-childhood

perspective and longer-term outcomes for children as young adults was given consideration in

terms of measuring impact.

4.2 Step 2: Specify outcomes

Table 13 and Table 14 summarise the outcomes, outputs and domains specified for the services

being compared in Malta and Romania. All elements included in the matrix are drawn from the

analysis of primary and secondary data gathered for Childonomics and analysed in Tables 6-11

and then validated by stakeholders in the final step of the methodology.

Table 13: Outcomes matrix for comparing family support and alternative care services based on analysis of data gathered for Childonomics pilot in Malta


Ou

tpu

ts

Outp

uts

for

serv

ice u

sers

Quality of relationships with parents, siblings and other family members

Safety of child; child development; levels and quality of contact with parents, siblings, grandparents

Case worker reports; IFSS, CS, foster care, residential care and LACS service data; CPS referrals; development assessments; interviews with carers and parents; interviews with children and young people

Quality of care – 'meeting needs and ensuring well-being' of children

Well supported carers (residential staff, parents, foster parents, grandparents) Mental health diagnosis and treatment of child/parents

Case worker reports; LACS, IFSS, CC, foster care and residential service data; service data on caseload, client contact and networking with other services; CPS referrals; interviews with carers and parents

Ou

tco

mes

Child

Improved health and development of child

During and after service inputs (i.e. as children and adults): age- and ability-appropriate developmental milestones (including ECD) met; progress in health/ communication/ speech indicators, mental health/ emotional well-being

Development, mental health, speech/communication and emotional well-being assessments; health records; case files; interviews with service providers

Children prepared so that independent living will 'be a success'

Disaggregated by type of service user and other characteristics: YPEET, housing, quality of relationships as adults, literacy, integration into the community

Official statistics, service data and case files, interviews with young people (former service users), education records



Fam

ily

Families providing care that meets needs of children and ensures well-being

Repeat referrals to CPS/LACS, reduced risk to children, reduced incidence of violence in the family, parents attending appointments/ acquiring skills


Com

mun

ity Reduced

intergenerational use of services

Number of users of CPS, IFSS/CC, residential care, foster care services using CPS, IFSS/CC social services and alternative care services for their own children


Socie

ty Reduced

intergenerational poverty and exclusion

AROPE, disaggregated for different service users at different stages of adulthood


Table 14: Outcomes matrix for comparing day centres for children with disabilities based on analysis of data gathered for Childonomics pilot in Romania


Ou

tpu

ts

Outp

uts

for

serv

ice u

sers

Child development

Age- and ability-appropriate


(including ECD) met, progress

in communication, social skills

and behaviour

Development,

speech/communication

assessments; case files;

interviews with service

providers and parents

Increased capacity

of parents to

provide care

Parent knowledge and

competency and engagement in

partnership with service staff

Interviews with carers and

parents; interviews with

children and young people;

case files

Ou

tco

mes

Child

Children prepared

for

school/education

Literacy, progress in education

during and after service use

Official statistics, service data

and case files, interviews with

young people (and former

service users), education

records

Fam

ily

Parents employed

and relationships

stable

Household income and structure Service data and case files

Com

mun

ity Reduced

separation of

children from

parents

Number of children with

disabilities from communities

targeted by day centres entering

long-term alternative care


and case files

Socie

ty

Improved

education and

inclusion of

children with

disabilities


children with disabilities from

different services at different

stages of adulthood


and case files



For the systems-level pilot in Malta the process of specifying outcomes was complex because in

some instances family support service managers and staff framed service outputs and outcomes in

terms of ‘preventing entry into out-of-home care’. If this is the goal of family support services then it

is not possible to compare them with alternative care services, the goals of which are focused on

child well-being, education and preparation for independent living. Longer-term outcomes being

sought by family support services were also identified by service staff and in-service manuals and

so it became possible to compare the three types of services in the Malta system for the purpose

of piloting. What makes it possible to include different types of investments (services) in the CCA is

the existence of common outcomes and indicators, and not the characteristics of service users.

Childonomics allows, for example, a comparison of the cost and consequences of earlier

intervention, before the situation in the family becomes such that the child has to enter alternative

care, as well as allowing consideration of how changes to gatekeeping, assessment and decision

making could help to ensure better outcomes and more efficient and targeted use of all types of

services.

The process of establishing the scope of the CCA and specifying and then measuring outcomes

and outputs is always likely to be an iterative one: the users of the Childonomics methodology may

need to adjust the scope and outcomes specification until the set of services that is included in the

CCA can be confirmed as working towards common outcomes that can be measured with common

indicators.

4.3 Step 3: Measure outcomes

The data available to the Childonomics project team in both Romania and Malta for measuring

indicators for the outputs and outcomes for each set of services, and the strength of evidence for

linking inputs with outputs in each case, is summarised in Table 15 for the Malta pilot and in Box 2

for the Romania pilot. In all cases, as mentioned in Section 2 of this report, the evidence for this

Childonomics CCA can be classified mainly as anecdotal and only indicative at best as it is based

mainly on a limited number of case studies and interviews with a non-representative sample of

managers, social workers, residential staff and service users.

Table 15: Measuring selected indicators – examples based on some of the indicators in the outcomes matrix and data available from the Malta pilot

For each type of service:

Indicator Means of

measurement Source of data for this

CCA

Comments on evidence for this CCA and potential for

stronger evidence

Well supported carers (residential staff, parents, foster parents, grandparents)

Carer/social worker caseload ratios and level of contact

Case studies; interviews with managers, social workers, residential staff, service users and foster carers

Service delivery metrics can provide strong evidence if monitored systematically, without relying only on data entered by case workers and if triangulated by direct feedback from clients

Family support (IFSS and five CC)

89 per social worker per year on average 30 per social worker per year for intensive support cases Daily/weekly contact

Appogg annual review (service data); case studies and interviews with social workers and managers

Case management system combined with feedback from service users will provide strong evidence. There are slight differences in caseload ratios between IFSS and CC

Residential care (two examples of services)



0.72 children per residential social worker (eight-hour daily shifts) Three residential care workers per LACS social worker Weekly supervision

Case studies and interviews with social workers and managers

The level of supervision and management support for residential care workers would be a better equivalent for comparing with the level of social worker support for foster carers and parents living in poverty. Anecdotal evidence that residential social workers are paying for private therapy

Foster care

19 carers per foster care social worker (on average per year)

Appogg annual review (service data for Foster Care Service, LACS, foster carer payments); foster carer and foster children focus group discussions

Anecdotal evidence that frequency of contact between foster carers and social workers is not adequate.

Indicator Means of


CCA


stronger evidence

Mental health diagnosis and treatment of child/parents

Parents and children accessing services they need for mental health issues

Yes, no, sometimes: based on case studies; interviews with managers, social workers, residential staff, service users and foster carers

Number of referrals to mental health services in themselves will not be a good measurement of this indicator, but could complement qualitative data

Family support

Parents: sometimes Children: yes

Several anecdotal cases of ‘undiagnosed’ mental health issues among parents and post-natal depression triggering care placements of young children

Disaggregating mental health records of family support service users and monitoring cross-referrals to mental health services could help to generate stronger evidence

Residential care

Young children: probably not Older children: sometimes Carers: sometimes

Residential care, no matter how well organised, is unlikely to provide individualised 24/7 care required by very young children. Older children receive support of staff and external specialists, but are extremely traumatised and the residential setting may be meeting their mental health needs only partially. Residential care workers require considerable support to prevent burn-out and enable full engagement with children

Case files, staff interviews and supervision records, child health records and interviews with young people could all help to monitor whether children and staff are receiving the mental health support they require

Foster care

Children: sometimes Foster carers: sometimes

Children may arrive in foster care extremely traumatised and parents and children may not be accessing all the support required

Anecdotal evidence that support for trauma required by children not fully accessible and that foster carers are ‘left on their own with the problems’



Indicator Means of


CCA


stronger evidence

Quality of relationships with parents, siblings and other family members

Siblings kept together Yes/no/sometimes – based on interviews, case studies

Case management system and existing child protection database should provide strong evidence on this indicator

Family support

Yes, with some exceptions

Anecdotal evidence that children may have siblings living with grandparents or in formal care while they continue to live with parents

Residential care

Yes No

Some residential units are specifically designed to keep large sibling groups together Other residential units separate siblings by age and gender

Foster care

Sometimes

Anecdotal evidence that small sibling groups (two siblings) are placed together, but that large sibling groups are difficult to place together in foster care as it is currently organised

Indicator Means of


CCA


stronger evidence

Improved health and well-being

Age- and ability-appropriate developmental milestones and well-being targets are met

Yes/no/maybe, based on interviews, case studies

Development, mental health, speech/communication and emotional well-being assessments, health records, case files, interviews with service providers and service users could provide strong evidence

Family support

Maybe

Mainly yes, but anecdotal evidence from residential and foster care services of extreme neglect and developmental delay at the point of entry into care

Regular assessments are required to be able to monitor these types of indicator, but they can provide moderate evidence if done systematically by trained and qualified personnel who are not assessing their own work, or through carer assessments that are then analysed by qualified personnel

Residential care

Probably not for younger children. Maybe for older children

Foster care

Yes Bearing in mind that children entering foster care may have experienced extreme abuse, neglect or violence, anecdotal evidence suggests strong recovery and development

Indicator Means of


CCA


stronger evidence

YPEET, housing, quality of relationships as adults, literacy,

Good, bad, mixed: based on interviews with care leavers, case studies and interviews with managers and service staff

If disaggregated for different types of former service users, official statistics, service data and case files, interviews with young people (former service users) and education records could provide strong evidence




Family support

Yes (at national average levels)

Very little evidence, and need to disaggregate former care leavers from purely ‘family support’ cases

Longitudinal data are required if evidence is to have high validity and to be strong

Residential care

Mixed

Services report higher than national average YPEET, literacy and integration, but care leavers present a mixed set of results and case studies from family support services indicate a number of former residential care leavers among CPS referred cases struggling with relationships, housing issues etc.

Foster care

Yes

Only very anecdotal evidence that care leavers with long, stable stays in foster care and ongoing contact with foster families into adulthood are doing better than care leavers with more chaotic care pathways

Indicator Means of


CCA


stronger evidence

Repeat referrals to CPS/LACS, reduced risk to children and violence in family, parents attending appointments/ acquiring skills

• High repeat referrals = 1; Low = 3

• High reduced risk = 3; Low = 1

• High attendance/skills = 3; Low = 1

High, medium, low levels scored 1–3: based on

interviews and case studies. Total possible score = 9 (for

three indicators)

Official statistics, service data and case file reviews could help to generate stronger

evidence

Family support

Repeat referrals = 2 Reduced risk = 2 High attendance/ skills = 2 TOTAL 6

In some cases, anecdotal evidence suggests good results (warranting a score of 3) and in others much weaker performance. This scoring represents an average across all interviews and case studies

Several cases of care leavers reported as returning to their families of origin and soon becoming homeless as the neglect, abuse or violence that caused their entry into care has not changed

Residential care

Repeat referrals = 1 Reduced risk = 2 High attendance/ skills = 1 TOTAL 4

Residential facilities largely do not work with families to reduce violence or build skills. Children leaving residential care may often return for further stays. Children in residential care may be at lower risk than if they remain in their own families but only if the care provision is of a high standard with strong safeguards in place: ‘the nuns pinched me so I asked to be moved’.

Foster care

Repeat referrals = 2 Reduced risk = 2 High attendance/skills = 1 TOTAL 5

Foster care, as it is practised currently, tends to be long term but children may not be in a situation of permanence, with parents still retaining parental rights. Children may move between foster care and residential care or reintegration. Foster care services do not work with parents to reduce violence or build skills. Children may be safer in foster care if adequate safeguards are in place and high standards are maintained

Indicator Means of


CCA


stronger evidence

Number of users of CPS, IFSS/CC, residential care, foster

No data reviewed for this CCA but a good/satisfactory/poor rating has been proposed based on anecdotal evidence from interviews and document review (FOR PURELY ILLUSTRATIVE PURPOSES)

Official statistics, service data and case files can provide strong evidence for key indicators, both historically and for ongoing monitoring. Data will need to be



care services using CPS, IFSS/CC social services and alternative care services for their own children

disaggregated for different service users

Family support

Satisfactory

Some cases cited of families using services from generation to generation, but also examples of real change and graduation into employment and stability, no longer requiring support

Residential care

Poor

Many cases of care leavers from residential care being referred for their own children to child protection services, mental health services, homelessness and domestic violence shelters

Foster care

Satisfactory Some cases of care leavers from long-term foster care who have stable housing and are in employment and have not required further support and services

Indicator Means of


CCA


stronger evidence

AROPE

No data reviewed for this CCA

Strong evidence can be gathered from official

statistics and service data if they are disaggregated for different service users at

different stages of adulthood. Case file analysis can also provide strong evidence if

case files are systematically and consistently maintained (e.g. on household income

and exclusion)

Family support: Satisfactory (national average)?

Residential care: Not known

Foster care: Not known

Box 2: Measuring outcomes and outputs for two day centres in Romania pilot based on

interviews and group discussions with staff and parents whose children use the day centre services

Parents in one centre note the following outcomes/outputs: family stability, reduced stress in parent relationships, positive adjustment of siblings, employment of parents, improved child development and behaviour, and socialisation.

All parents interviewed are long-term service users (four to five years or more) and they greatly value being an active and an equal partner in the process of service delivery and engaged in detailed discussion with staff of the centre on all aspects of the child’s progress and the service inputs.

The minimum service does not meet all needs (there are other services being paid for by families (e.g. therapy, sport)).

Based on staff interviews in two centres: children develop skills and abilities that they did not have before (speech and communication, motor and learning skills, self-care) and parents develop skills and knowledge.

Staff in one centre also highlight the impact that accrues at the community level in terms of changes in attitudes and inclusion for children with disabilities, but no specific measurements are made of these changes.

Staff in one centre say that around 50% of children enter mainstream school from the centre, but this is not measured or monitored systematically after children leave the centre.

The ability to measure outcomes and outputs accurately and meaningfully is highly dependent on

the availability of valid data. As the methodology makes clear, different kinds of data can be used,

including qualitative data, but consideration has to be given to the extent to which the data can be

said to be valid in relation to linking inputs or interventions to specific outcomes or outputs. The

methodology (Gheorghe et al., 2017) gives the following guidance on the steps for measurement:



Identifying the relevant data: understood as obtaining information on the target outcome

indicators. Depending on the indicator, data can be either quantitative (e.g. poverty rates) or

qualitative (e.g. perceived experience of using the service). There may be cases where data

are already available: for example, in databases, published studies or reviews of the literature.

In other cases, data may have to be collected as part of the analysis.

Assessing the extent to which data constitute evidence for the said outcomes. This

involves making a judgement on validity and generalisability. What these concepts mean

depends on the nature of the relevant data.

In the case of the pilots, a limited number of interviews and group consultations were conducted,

and the available quantitative data were reviewed and analysed, but in the end generalisable

conclusions could not be made. This is not to minimise the importance or validity of the views and

perspectives of the service users and staff who were interviewed, but a general absence of

rigorous evaluation or granular monitoring data meant that analysis was constrained and

generalisability was limited for the two pilots.

4.4 Step 4: Calculate costs

We present here the calculation steps and assumptions that we made in deriving the total, average

and variable cost for each intervention being compared in the two pilots. Official financial data were

provided in Malta by FSWS and the MFSS, and in Romania by the service managers. The

assumptions that have been made in order to allocate costs, however, are based on interviews and

a review of secondary data and, as such, cannot be considered to be underpinned by strong

evidence. In this regard, the costings carried out during the pilots can be considered as ‘good

enough’ for the purpose of testing the methodology and comparing investments using similar

assumptions, but should not be considered as reporting the real or actual expenditure on each type

of service or set of services included in the analysis conducted for the pilots.

How to determine which costs to include for family support – Malta example

The following assumptions, based on interviews and data review, were made in order to determine

which costs to include for family support:

Homestart is a service that is entirely dedicated to supporting families with children, therefore

its entire annual budget was added in the calculation.

IFSS and CC have both adults and children as service users. We assumed, based on

interviews and service data reviews, that half of their service users are families with children,

therefore 50% of each of their respective annual budgets was added in the calculation. A

necessary assumption is that the distribution of service intensity is no different between

children and adults, otherwise the 50% budget allocation does not hold.

CPS also serves a range of users, including those receiving family support. On the basis of

interviews and case studies we estimate that 30 out of 89 cases per social worker in the family

support services are child protection referrals or are being monitored and followed by child

protection. We have added a per client amount for use of CPS services for one-third of 50% of

family support service users who are families with children and child protection cases.

The total number of beneficiaries of family support services was assumed to be 50% of users

of the combined IFSS and CC, as these are the only services assumed to be contributing



unique, mutually exclusive users. Beneficiaries of Homestart and CPS would also be captured

by IFSS and CC, therefore adding them would constitute double-counting.

Table 16: Calculating family support costs, Malta pilot (2016 data)

Service

Estimated annual

expenditure

(EUR)

Proportion of

budget assumed

towards families

with children or

CPS clients

Annual expenditure

assumed towards

family support (EUR)

IFSS 487,569 50% 243,785

CC 972,743 50% 486,372

CPS for 438 clients (one-third of 1,315 clients)

302,202 100% 302,202

Homestart 29,283 100% 29,283

Estimated total expenditure (on families with children)

- 1,061,641

Of which: estimated total expenditure on 438 CPS clients

66% 707,761

How to determine which costs to include for residential care – Malta example

The cost for residential care was calculated based on data provided for only two residential homes

and cannot be assumed to be representative for all residential homes. The costs for the residential

care were calculated using the costs for the homes themselves and a proportion of CPS costs and

LACS costs. The details of the calculation are summarised in Table 17 and the following

assumptions were made:

Both residential homes are dedicated exclusively to this service, therefore their annual budgets

were added in full. Beneficiaries of these two services are also unique, mutually exclusive

users, which were added without risk of double-counting.

All beneficiaries of both residential homes would be previously seen and followed by CPS and

LACS. We have allocated a cost for these inputs based on the per client cost for each of these

services.

Table 17: Calculating family support costs, Malta pilot (2016 data)

Service Estimated annual expenditure EUR

% budget assumed towards two residential

care homes

Residential Home 1 242,897 100%

Residential Home 2 610,800 100%

CPS for 26 clients 17,939 100%

LACS for 26 clients 41,002 100%

Estimated total expenditure (two residential care homes)

912,638 -

Estimated costs Costs per beneficiary per year EUR

Per child per year for 26 children using two residential care services

35,102



If this cost per care beneficiary in these two homes is taken as an average that is representative of

the costs per care beneficiary in the system as a whole, then the total cost of the system, assuming

there are 208 children in total in the care of residential homes, can be estimated at around 208 x

EUR 35,102 = EUR 7,301,216 per year.

N.B.: These assumptions and estimates were made for purely illustrative purposes in order to test the application of the methodology and cannot be taken as reflecting the real cost of the residential care system in Malta, or the actual cost per child of residential care in Malta.

How to determine which costs to include for foster care – Malta example

The cost for foster care was assumed to include the following services: foster care service, foster

care allowances, CPS and LACS. The details of the calculation are summarised in Table 18 and

the following assumptions were made:

There were 243 children in foster care in 2016.

The foster care service and foster care allowances are two services dedicated exclusively to

foster care, therefore their annual expenditure were added in full. While the fostering budget

was provided by FSWS, expenditure on allowances for 2016 was estimated by calculating the

average annual amount based on actual amounts spent by MFSS on allowances in 2014 and

2015, and assuming the same was spent in 2016 for 243 children.

Similar to residential care (above), it was assumed that all children in foster care would be

previously seen by CPS and supported by LACS. We have allocated a cost for these inputs

based on the per client cost for each of these services.

Table 18: Calculating foster care costs, Malta pilot (estimated 2016 data)

Service Estimated annual expenditure EUR

% budget assumed towards foster care

Foster care service 390,804 100%

Foster care allowances 1,553,000 100%

CPS for 243 clients 167,670 100%

LACS for 243 clients 383,211 100%

Total estimated expenditure (foster care system)

2,494,685 -

Estimated costs per beneficiary EUR

per child for 243 children using foster care services 10,266



N.B.: For all three sets of services included in the Malta pilot, the costs of the physical space in which the child is being cared for have not been included – the home of the family, whether rented or owned, the home of the foster carers or the building in which the residential care is provided. Children’s allowances, unemployment benefits or other cash allowances that families using family support services might be receiving have not been included. Salaries from paid work and other income of families or foster carers have not been included. This is because the scope of the analysis was to determine the costs to the government budget of the service provision alone. It would be possible, if data were available, to add in some, or all, of these other types of investment in order to explore the full investment that is being made, not only by the state services budget, in achieving particular outcomes for children and families (and communities and society). As the Childonomics methodology emphasises, however, the question of how to attribute different types of investment to specific outcomes becomes extremely tricky, requiring granular data providing highly valid and generalisable evidence. The field-testing during the pilots, however, indicates that the methodology could be applied in this way as long as data are available.

How to determine which costs to include for day centres – Romania example

We present here the calculation steps and assumptions that we made in deriving the total, average

and variable cost for the two day centre services for children with disabilities being compared in the

Romania pilot. Financial data were provided by each organisation.

The budget, staff and number of services users for Centre 1 for the years 2015–2016 are

presented in Table 19 and costs are presented in Table 20. The following assumptions were made:

Staff data and the number of service users were available only for 2016. Based on interviews

and annual reports we concluded that no significant changes occurred from 2015 to 2016.

We did not have data on the exact salary of each staff member or staff role. We used a system

of weights to approximate salary differences: 1 for care staff, 1.5 for nurses and clerks, 2 for

inspectors and specialised staff, and 2.5 for the manager. These weights were used to derive

the cost of staff directly involved in child services within the centre – this was assumed to

include specialised staff and care staff.

Table 19: Budget and capacity: Centre 1 – Romania pilot

Item 2016 2015

Assumed direct,

child-facing

resources (%)

Main budget items (Romanian lei)

Salaries and contributions 537,723 333,692 56%

Stationery 1,363 2,12 100%

Cleaning materials 1,004 37,426 60%

Lighting 11,630 4,594 0%

Water, sewage, cleaning 3,738 3,746 0%

Internet, phone, post 6,672 40,904 0%

Other goods and services from third

parties 34,430 215,209 70%

Repairs in the centre 287,195 9,983 50%

Food 4,569 3,945 100%

TOTAL expenditure 888,324 651,631

Staff

Manager 1 1

Therapist 4 4

Speech therapist 1 1

Kinethotherapist 2 2



Medical nurse 1 1

Driver 1 1

Carer 2 2

Cleaner 2 2

Clerk 3 3

Inspector 2 2

TOTAL staff 19 19

Average # service users 40 40

Table 20: Estimated costs for Centre 1 (EUR, 2015–2016) – Romania pilot

Type of cost 2016 2015

Average annual exchange rate, 1 leu = EUR 0.222 0.224

Total annual cost (EUR)

Total annual cost of centre 197,208 145,965

Average total cost per child (EUR) 4,930 3,649

Average direct cost per child (EUR) 2,625 2,070

The expenditure budget, staff and service users for Centre 2 in the years 2015–2016 are

presented in Table 21. These data are then combined in Table 22 to produce the total cost, total

average cost per child and the average direct cost per child.

Table 21: Budget and capacity: Centre 2 – Romania pilot

Item 2016 2015 Assumed direct,

child-facing resources (%)

Main budget items (Romanian lei)

Salaries and contributions 2,744,294 2,072,280 87%

Goods and services 234,381 276,909 10%

Current repairs 5,736 17,023 0%

Food 326,656 350,189 100%

Drugs and medical consumables 19,724 15,011 100%

Uniforms - 36,000 100%

Other material objects - 27,000 50%

Non-financial actives 16,262 14,004 50%

TOTAL expenditure 3,536,780 2,886,299

Staff

Managers 2 2

Specialised staff 72 72

Care staff 18 18

Administrative staff 7 7

TOTAL staff 99 99

Average # service users

Residential care homes 84 84.5

Rehabilitation centre (non-residential) 127 127



Table 22: Costs for Centre 2 (EUR, 2015-2016) – Romania pilot

Type of cost 2016 2015

Average annual exchange rate, 1 leu = EUR 0.222 0.224

Total annual cost (EUR)

Total cost of centre 785,165 646,531

Total cost of residential care homes 505,355 441,746

Total cost of rehabilitation centre 279,810 204,785

Average total cost per child (EUR)

Child in residential care homes 6,280 5,208

Child in rehabilitation centre 2,300 1,607

Average direct cost per child (EUR)

Child in residential care homes 5,806 4,531

Child in rehabilitation centre 2,126 1,398

The calculation relies on the following assumptions:

The centre’s financial officer could provide disaggregated budget allocations for the centre’s

two main types of services, residential (foster homes) and non-residential (rehabilitation

centre). These apportionments were assumed to be correct.

Staff data were available only for 2016. Based on interviews and annual reports we concluded

that no changes occurred in the staffing structure from 2015 to 2016.

We did not have data on the exact salary of each staff member of staff role. We used a system

of weights to approximate salary differences: 1 for care staff, 1.5 for nurses, 2 for administrative

and specialised staff, and 3 for managers. These weights were used to derive the cost of staff

directly involved in child services within the centre – this was assumed to include specialised

staff and care staff (with the exception of cooks).

4.5 Step 5: Present net costs and outcomes

The Childonomics methodology does not return a single metric, but, rather, a more complex and

nuanced picture of the interplay of investments and outcomes at different levels, presented in a

results matrix linking outcomes to investments and incorporating consideration of the strength of

evidence for these links.

The methodology proposes a simple taxonomy of strength of evidence based on the concepts of

validity and generalisability and scored according to a grading framework (Petticrew and Roberts,

2003, cited in Gheorghe et al., 2017, pp. 28–29), based on a research design, as summarised in

Table 23.

Table 23: Definitions of strength of evidence categories

Validity Generalisability

High – evidence is generated as a result of a research design that can be mapped to grade +++ in the Petticrew and Roberts framework.

High – net impacts are informed by context-specific data (e.g. the same country or region) and there are no reasons to suspect meaningful variations.

Moderate – evidence is generated as a result of a research design that can be mapped to grade ++ in the Petticrew and Roberts framework; or evidence is generated by combining results from several research products that can be

Moderate – net impacts are informed by data from a different context and there may be variations, but their magnitude is unlikely to influence the net impact (e.g. service cost data from a neighbouring county or country).



mapped to grade ++ in the Petticrew and Roberts framework (e.g. decision modelling).

Low – evidence is generated as a result of a research design that can be mapped to grade + in the Petticrew and Roberts framework; or evidence is generated by combining results from several research products that can be mapped to grade + in the Petticrew and Roberts framework.

Low – net impacts are informed by evidence generated in a different context, variations are likely to exist and their impact on the net impact may be substantial.

High validity and high generalisability, for example, equates to strong evidence; a combination of

high and low validity and generalisability equates to moderate evidence; and combinations of low

or moderate validity and generalisability equate to only indicative evidence.

The financial data and measurement of outcomes presented in the two pilot results matrices in

Table 24 and Table 25 should be considered as illustrative only and not as reporting real or actual

data, given the low strength of evidence scores. They are presented here only to demonstrate how

the Childonomics methodology culminates in a single matrix linking outcomes to investments, while

also reflecting on the strength of evidence for these links.

In the case of the pilots, the strength of evidence was in many instances weaker than the weakest

category of evidence proposed in the methodology (indicative) as it was based mainly on low

validity and low generalisability data. With this cautionary note in mind, it is nevertheless possible

to see from the results matrices how the Childonomics methodology presents information in such a

way that it could help to inform decision making and support advocacy, especially if moderate or

strong evidence can be found. In the presence of only indicative or no evidence, it highlights the

gaps in knowledge that could help to inform the development of monitoring systems or further

research.

The column ‘difference from comparator’ is intended to quickly summarise the differences between

the indicators reported for the comparator and the services being compared for outputs, each level

of outcomes and each dimension of investment (overall expenditure, cost per beneficiary and

marginal cost). Where there is strong evidence, many of these cells would be filled with numbers or

strongly supported statements of difference. In the case of the pilots, a simple +/- scheme is used

for purely illustrative purposes, given the indicative – or even weaker – nature of the evidence

available.

Key for reading the matrices in Tables 24 and 25

Key for ‘difference from comparator’ column:

* = cannot be compared, no data

+ = higher value than comparator

/ = the same value as comparator

- = lower value than comparator

/- = the same or lower value

Key for ‘strength of evidence’ column:

Strong = valid and generalisable

Moderate = moderately valid and generalisable

Indicative = low validity and generalisability of evidence reviewed for the pilot studies

Anecdotal = almost no validity or generalisability of evidence reviewed for the pilot studies

No evidence and data not available = evidence may exist but was not reviewed for the pilot studies

Purely illustrative = information based on guesswork to illustrate how the matrix can be used if evidence is available



Table 24: Malta pilot results matrix for illustration purposes presenting estimated measurements of outcomes and financial data

Dimension Domain Indicator

Comparator – Family support

Alternative 1 – Residential care Alternative 2 – Foster care

Impact Impact Difference

from comparator

Strength of evidence

Impact Difference

from comparator


Outputs

Quality of relationships with parents, siblings and other family

members Quality of care -–'meeting needs

and ensuring well-being' of children

Siblings kept together

Well supported carers: social

worker caseload per carer and

level of contact

Yes, with some exceptions

89 families per social worker (average), 30 parents per

social worker (reported) for

intensive support with daily/weekly

contact

Sometimes

18 residential social workers per supervisor

(average) – weekly

supervision

/-

+

Indicative

Sometimes

19 carers per social worker

(monthly or less frequent contact)

/- /-

Indicative

Outcomes

Improved health and development

of child

Age- and ability-appropriate

developmental milestones met

Maybe

No for young children; maybe

for older children

/- Anecdotal Yes? +? Anecdotal

Children prepared so that

independent living will 'be a success'




National average?

Mixed - Anecdotal National average? / Anecdotal

Families providing care that meets

needs of children and ensures well-

being

Repeat referrals to CPS/LACS, reduced risk to

children, reduced incidence of

violence in the family, parents

attending appointments/ acquiring skills

6

4 - Anecdotal 5 - Anecdotal

Reduced intergenerational use of services

Number of users of CPS, IFSS/CS, residential care,

foster care services using CPS, IFSS/CS and alternative

care services for their own children

Low? Medium? - Purely

illustrative Low? /

Purely illustrative




poverty and exclusion

AROPE, disaggregated for different service users at different

stages of adulthood

Not known Not known * No evidence Not known * No evidence

Costs

Total cost (EUR)

Total cost of programme per

year for all children (for CPS

cases only)

1,061,641 (707,761)

7,301,216 X 7

(X10) Purely

illustrative 2,494,685

X2.4 (X3.5)

Indicative

Total cost per beneficiary (EUR)

Total cost per child (for CPS cases only)

Total cost per family

807 (1,616)

Not known

35,102

Not known

X44 (X22)

Purely illustrative

10,266

Not known

X13 (x6)

Indicative

Marginal cost per beneficiary (EUR)

Marginal cost per child

Marginal cost per family

Not known Not known * Data not available

Not known * Data not available


© Oxford Policy Management, December 2017 47

Table 25: Romania pilot results matrix for illustration purposes presenting estimated measurements of outcomes and financial data

Dimension

Domain Indicator

Comparator – day

Centre 1 Alternative 1 – day Centre 2

Impact Impact Difference

from comparator


Outputs

Child development Increased capacity of parents to provide care

Age- and ability-

appropriate

developmental

milestones

(including ECD)

met, progress in

communication,

social skills and

behaviour

Parent knowledge

and competency

and engagement in

partnership with

service staff

Yes

Yes

? ?

/ /

Anecdotal

Outcomes

Children prepared for school/education

Literacy, progress

in education during

and after service

use

Yes No data * Anecdotal

Parents employed and relationships stable

Household income and structure

Yes No data * Anecdotal

Reduced separation of children from parents

Number of children with disabilities from communities targeted by day centres entering long-term alternative care

Satisfactory Poor - Purely

illustrative

Increased education and inclusion of children with disabilities

AROPE, disaggregated for children with disabilities assisted by different services at different stages of adulthood

Not known Not

known * No evidence

Costs

Total cost (EUR)

Total cost of programme per year

197,208 279,810 - Indicative

Total cost per beneficiary (EUR)

Total cost per child Total cost per family

4,930

Not known

2,300

Not known

- Indicative

Marginal cost per beneficiary (EUR)

Marginal cost per child Marginal cost per family

Not known Not

known * Indicative



4.6 Step 6: Stakeholder validation and the summative narrative

The results matrix alone does not capture the nuances of the context in which the services that are

the subject of the analysis are being delivered: it has to be accompanied by a summative narrative

that presents context, interprets the results and facilitates a more informed understanding of the

data presented in the matrix. The methodology proposes that workshops be held with stakeholders

in order to validate the outcomes and results matrix, and to explore the implications of the results

from the CCA. This permits the stakeholders to engage not only in defining the outcomes of values

and the indicators of importance, but also in shaping the way in which the results matrix is

interpreted and presented. The summative narrative should be informed by the validation

workshop and should present, in one page, any considerations of context, any factors affecting (for

example) access, availability, quality and effectiveness of services, and stakeholder interpretations

of the results and implications for decision making or next steps.

Validation workshops

Researchers presented the purpose of the pilots and the preliminary analysis to stakeholders in

Malta and Romania, both in group discussions and in individual interviews, and facilitated a

discussion of the following points:

Do the outcomes, outputs and indicators selected for the outcomes and results matrix reflect

the outcomes and outputs that are recognised as valuable by stakeholders, and are the

indicators the correct ones for their measurement?

What do the findings mean to whom? What can be done with them?

The challenges and opportunities that emerge from the findings.

Validation consultations were held with managers, social workers and service users from each of

the comparator services or service systems and, in most cases, with the same participants as in

the first stages of data collection.

Given that the main purpose of the pilots was to test the methodology, and given that the data

presented in the results matrices were incomplete and with low validity, the validation discussions

were focused more on trying to understand the usefulness of the methodology and its results for

stakeholders, its applicability and its relevance in the different contexts where it was being piloted.

Some discussion focused on interpreting the results, but this was hampered by concerns about the

validity or accuracy of data in some cases.

Key points to emerge from the stakeholder consultations on the results matrices, and the

implications in terms of validating the selected outcomes and indicators, include the following:

1. Service users, social workers, specialists, managers and decision makers largely agree that the

outcomes and outputs identified were relevant to the services included in the pilots, both in

Romania and Malta, and can be applied across the different comparator services. They also

recognise them as reflecting their experience and contributions made during the field-testing of the

methodology in both country pilots. The indicators need to be refined and more clearly defined if

they are to become meaningful and measurable across all sets of services: ‘What do we mean by

quality of relationships or level and quality of contact?’ one participant asked in the Malta pilot, for

example. A manager emphasised, in another example, that the caseload for social workers is ‘not

a useful indicator by itself and may be an over simplification’. The honing of outcomes and

indicators together with stakeholders represents best practice in planning services, defining service

standards and creating and implementing monitoring frameworks. Validation consultations with

parents of children using day centre services, in another example from the Romania pilot,



highlighted the value that parents place on strong partnerships with staff and being respected as

an equal partner in their child’s care and habilitation.

2. While bearing in mind the low validity of the evidence presented in the matrices, the importance

of context and making clear some of the nuances that may be affecting the links between

investment and results is critical. Examples from the validation workshop consultations included

the following:

Social workers may have to make decisions based on ‘what is available, not what is needed’

so, for example, children may be in residential care not because they need that form of care,

but because there are no other alternatives at the time when decisions are being taken.

Challenges in implementing new legislation in the child protection and alternative care system

will require more resources, and recruiting trained social workers is an ongoing challenge so

‘both of these issues need to be factored in’.

Contributions to the effectiveness of social services for children and families are also made by

other allied services, such as the education and mental health systems. These contributions

(investments) can also be taken into consideration to maximise impact and value in achieving

certain outcomes.

Some children in residential care may be very traumatised and require intensive, therapeutic

services delivered by highly qualified staff, and these inputs carry high costs.

Day care services in a rural area may seem to be achieving poorer results at a lower cost than

a day centre in an urban area, but the education level of parents in the rural area may be lower

and their ability to engage as partners in their child’s care may also be lower as they have to

travel greater distances to bring their child to the service. More investment in outreach services

in rural areas could help to bring outcomes and outputs achieved up to the level of those of the

urban-based centre.

3. Feedback on interpreting the results presented in the results matrix and understanding what

can be done with the findings included the following:

The estimated cost of residential care provision may not be high enough as too much

residential care is under-funded, which can affect the quality of care provided and therefore the

outputs and outcomes that can be expected from this kind of care.

The apparent lower cost of one day centre compared to the other could be attributed to the

centre sharing some overhead and administrative costs with other types of services, or it could

reflect greater opportunities in urban areas to attract additional private funding to support

service delivery.

A foster care system stakeholder reacted to the cost of foster care being 13 times more than

the cost of family support services with the question: ‘does this mean that the results should be

13 times better from foster care?’

Acknowledgement that there is a need for more independent, quality research in order to

address the weakness and validity of evidence available for measuring outputs and outcomes.

Staff and managers also acknowledge the need to be better at systematic long-term monitoring

of outputs and outcomes: for example, following up on the children who leave the day centres

or different types of care or family support services, in terms of education progress and social

inclusion.

The Childonomics model was felt by some participants to be useful as it can help to change the

conversation from a discussion about inputs and costs to a discussion about investments in

future outcomes:



‘[the Ministry] pays out per bed nights, not per outcome or per identified need. This

makes it hard to argue for more money. Also, when they see overtime for social

workers, it is viewed as a “cost” not as an “investment”…’ (Senior manager,

validation workshop participant, Malta)

Stakeholders generally found the summative nature of the results matrix helpful in thinking

about the whole system and the wider results that can emerge from each part of the system.

One social worker said: ‘it has given me a clearer understanding of what we are contributing

to’. Stakeholders felt it can be used to measure system improvements at all levels of provision

(i.e. child, family, community and society). This was largely interpreted as positive and

reinforcing best practice.

Summative narrative

This narrative summarises the contextual issues, nuances of interpretation and feedback from

stakeholders into a final analysis. Such a final analysis should always accompany the results

matrix as the final output of applying the Childonomics methodology. The summative narrative

should include key information from each step of the analysis, from establishing the context and

scope through to presenting and validating the results and taking into account a range of

stakeholder perspectives.

A combined Malta and Romania pilot summative narrative is presented here in order to illustrate

how such a narrative can be structured. The results matrices in Table 24 and Table 25 should

always be accompanied by this summative narrative.

Malta and Romania pilots: combined summative narrative

The purpose of the pilots was to test whether the Childonomics CCA methodology can be applied

across whole systems of services from different parts of the Childonomics conceptual framework,

as well as to more micro-level analysis of services of a similar type. The pilot in Malta has shown

that common outcomes and indicators can be identified across a range of services in order to

facilitate a systems-level CCA and the Romania pilot has demonstrated that useful analysis can

also result from applying the method to the detailed analysis of very similar services.

Both pilots have relied on low validity and non-generalisable evidence and even though some of

the financial information and service provision data have higher levels of validity, evidence has

generally not been more than indicative, at best, during the pilots. The results matrices and this

narrative cannot be understood as providing an evaluation of the effectiveness of the system of

family support and alternative care in Malta, or of the quality and effectiveness of two day centres

in Romania.

In Malta, the results from the pilot suggest that a full case review and analysis of service data from

the CPS, IFSS and community centres, LACS and the foster care service has the potential to

establish more rigorously the links and pathways between family support and alternative care

services and their longer-term outcomes.

The pilot in Romania can be interpreted as suggesting that there is a need to review the single

standard for funding government day centres for children with disabilities in order to take more

account of the different contexts in which these services are being delivered. This could lead, for

example, to support for more outreach work with parents in rural areas and thus to the

achievement of more equitable access to quality services, supporting social inclusion for children

with disabilities and their families.



The indicators included in both results matrices are illustrative and further engagement with

stakeholders to specify and define the indicators of importance at each level would be needed

before finalising the outputs/outcomes matrices and therefore the results matrices. The inclusion,

for example, of separation from siblings in the Malta pilot results matrix as an indicator of quality of

relationships is justified by the strong priority given to this indicator by some of the care leavers

interviewed. Similarly, parent partnership with staff is included as an indicator in the Romania pilot

because of the strong priority given by parents to this indicator. The measurements for the

indicators in the results matrices are also illustrative as they are based mainly on evidence that can

be classified, at best, as indicative and as largely anecdotal.

The costs of services and the cost per child for each set of services included in the results matrices

are based on real cost data from 2016, but in some cases the figures are very rough estimates

based on only partial data. The costs are therefore also assessed as being only indicative or purely

illustrative in terms of the strength of evidence for both pilots.

The stakeholders consulted during the validation workshops and meetings confirmed that the

outcomes and outputs selected are largely valid, but are complex and may need further honing and

fine-tuning, as would the indicators, if they are to be used in any way as part of a planning and

monitoring framework.



5 Lessons learned and conclusions

5.1 Lessons learned

Lesson 1: the purpose of using the Childonomics methodology needs to be clear from the

outset and consistently communicated to stakeholders.

Childonomics is trying to do something very challenging and for which there are no existing

standard models. These challenges are described in detail in the conceptual framework and

methodology papers. Without reinventing the wheel, the Childonomics methodology draws on

various established methods for appraising return on investment and creates something slightly

different that can help to put different types of investment and outcomes side by side for

comparison. The methodology is very dependent on the quality and validity of the data that are

available. At the project outset there were high expectations among some stakeholders that

Childonomics could provide answers to a whole range of questions about the impact on society,

communities, families and individual children of different types of investments. As the project

unfolded, these expectations have had to be modified in the face of constraints on data and

concerns of stakeholders about how data will be used and the ultimate goal of the study. The

methodology works best when used as a tool for open inquiry which can challenge underlying

assumptions about what system best supports families and children.

The Childonomics methodology does not exist in a vacuum, but, rather, in a value-laden socio-

cultural system of policies and programmes aimed at supporting children and families in any given

national context. The methodology can be used at a number of different levels of focus or

granularity and choices about this will depend on decisions about the reason for using it. For the

purpose of testing the data collection tools and methodology, the research team selected the

services on which the pilots focused in order to test different levels of granularity (broad-sweep

national systems in Malta and more granular comparisons of similar services in Romania). The

decisions were also informed through consultation with stakeholders on their priorities and the

availability of data in both countries. Future users of the Childonomics methodology will have their

own priorities for selecting particular services or programmes for analysis and will have their own

understanding of the context in which they are making these choices, but the methodology can

help to ensure that these decisions are informed by a range of stakeholder perspectives.

The use of the tool is context-dependent – it relies on the availability of the right sort of evidence

and information about costs – but all policymaking faces the conundrum that we cannot fully

predict the future impact of actions. Users of the Childonomics methodology can use it to look at

the range of current investments and to make the best possible judgements about how and

whether they can reinvest and change the balance between services. It should be emphasised that

the methodology is user-driven and informed by Step 0. The conceptual framework is a tool that

supports a 360-degree landscape analysis that helps to make apparent the priorities and

objectives of the CCA and helps to answer the question ‘what are we trying to achieve?’ Other

questions to emerge could include, for example, ‘where to prioritise funding?’, ‘how to improve

continuity of care?’, ‘how to make sure no groups are left behind?’, ‘how to align budgeting with

service performance from the child’s perspective?’ and so on. The results from the CCA can then

support planning regarding potential reverberations throughout the system, and with a constant

eye on outcomes for the child and family, but also the community and society as a whole.

Lesson 2: the Childonomics methodology is highly dependent on the quality and validity of

the data that are available and it cannot replace absent data.



Data gathering for the purpose of applying the Childonomics methodology can highlight gaps in the

availability of data as much as it can provide an opportunity for gathering new information. The

data gathered for the pilots in Malta and Romania ended up being mainly anecdotal, based on a

limited number of interviews and consultations. Future users of the methodology should be

rigorous in ensuring all available research is reviewed, and, where appropriate, should commission

research to fill gaps in data.

However, the potential for gathering stronger evidence is noted in the pilots – especially in relation

to the existing FSWS database and the new case management system and monitoring system that

is not yet fully in place in Malta, and the AURORA database in Romania. The Childonomics

methodology could make an important contribution to the process of refining and defining

outcomes and indicators for these systems to ensure cross-sectoral and cross-organisational focus

on common outcomes at all levels (child, family, community and society), and measurement of

meaningful indicators of progress in working towards these outcomes.

Lesson 3: strong research skills are needed to undertake reliable and valid qualitative data

analysis, and to ensure that the views and perspectives of all stakeholders, including

service users, managers and staff, are taken into account and valued

The step of classifying and defining inputs, outcomes and indicators is of critical importance. If the

purpose of using the methodology is, for example, to inform future decisions about resource

allocation and to create a monitoring framework highlighting key, valued outcomes and ways of

measuring important indicators in order to track the consequences or returns from investment of

resources, then it is of critical importance to involve stakeholders in this process of precisely

defining the exact outcomes and indicators to be tracked. In this case, the methodology was

applied for the purpose of testing: if the governments of Malta and Romania want to take forward

the application of the methodology fully, then a facilitated stakeholder consultation workshop, to

refine the wording and definitions of the outcomes and indicators, would be advisable.

Lesson 4: it is important to concentrate attention on the value of outcomes rather than on

the value of inputs, and to ensure that the complexity and context of services and their

inputs and outputs are given due consideration in the presentation of the results matrix

Many stakeholders may be learning for the first time about the financial ‘worth’ of the service they

are involved in. In order to facilitate stakeholder contributions to defining outcomes of value and

providing feedback on the implications of the CCA it is important to present financial data as

neutrally as possible, and to take care to ensure discussion focuses duly on outcomes, indicator

measurements and outputs. Care is also needed in presenting the measurements of outcomes as

the results matrix does not permit much nuanced presentation of the factors that contribute to

differences in results from each type of comparator. The results matrix should always be presented

together with the summative narrative, which should be created following a stakeholder analysis

and validation process.

Lesson 5: the Childonomics methodology convenes stakeholders across institutional and

organisational boundaries and has the potential to be a powerful tool for planning and

monitoring system change.

Users of the Childonomics methodology must be sensitive to the power dynamics in the systems in

which stakeholders participating in the CCA exist, and must ensure that all stakeholder views are

given equal value. The lead organisation commissioning the use of the methodology (originally the

MFSS in Malta and NACPRA in Romania in the case of the pilot studies) needs to be prepared to

support the process through to the end if the results are to be used in a meaningful way.



5.2 Conclusions

The piloting of the Childonomics methodology has demonstrated its potential to support analysis at

all levels of the links between investments in children and families and the resulting outputs and

longer-term outcomes for children and families themselves, but also for communities and society at

large. The pilots have highlighted the need for good quality monitoring data and greater

transparency generally in child and family services, as well as the need for valid evidence on long-

term outcomes for children and families who have used certain types of services. Without these

types of data, there could be risks in using the results of the Childonomics CCA to take

programmatic or funding decisions.

Nevertheless, the Childonomics methodology can help users to marshal the best available

evidence on the costs and outcomes of different services. Users would then need to decide if the

evidence is strong enough to make judgements about future directions, or whether further

evidence from new research or pilots or additional information on costs is required. Once satisfied

that they have sound enough evidence, users can use the methodology for planning by modelling

costs based on different assumptions or projections. This will require projections to be made based

on assumptions (or evidence, where available) about the impact of different forms of investment.

Decisions about the scaling up and rolling out of services such as the multi-disciplinary teams in

Romania and LEAP in Malta appear to have been taken without much evidence of the returns that

can be expected from this investment. There is acknowledgement in both countries of the

importance of monitoring and data, but the new information frameworks and databases that have

been created to support the roll-out of programmes and services in both countries were at too early

a stage of development to be included in the pilot studies.

There is no perfect model for assessing return on investment and the Childonomics methodology is

not perfect, but it does help to bring out assumptions, and to examine the evidence for these

assumptions and what they mean for children, families and the system as a whole – both in terms

of maintaining the status quo and exploring alternatives. Childonomics as it is has some rough

edges that need ironing out in the next stages of development. These include questions and

challenges in the following areas:

1. The value of conducting studies and applying the methodology when evidence is weak.

The pilot studies, even with weak evidence, have supported stakeholders to: a) convene across

sectoral or organisational silos and recognise common goals for children, families and

communities; b) identify gaps in data and monitoring systems that need to be addressed; and c)

recognise that differences in the size of investment exist and therefore there is a need to review

the reasons for these differences and to be sure that decision making about directing children and

families towards particular interventions is rigorous and in the best interests of children.

This suggests that, even if strong conclusions to inform decision making cannot be formed

because of weak evidence, the exercise of using the Childonomics methodology itself has value in

focusing attention on issues of importance in relation to achieving system improvements.

The application of the methodology brings costs to the surface and makes them visible, as well as

helping to make explicit and to test the assumptions that have been used in planning, monitoring or

defining goals. The methodology draws out inputs and the interplay of different parts of the system,

ensuring that complex relations do not remain hidden.



2. The validity of comparing services and the challenges the full evaluative methodology presents in terms of appearing to introduce an element of ‘competition’ through such comparisons.

In the health sector, or indeed in any sector where resources are finite, the relative value of

different interventions is of prime importance in making decisions about resource allocation. This

should be no exception in the social care or social services sector. The interventions in the lives of

children and families may seem ‘messier’ and less standardised than health or education

interventions, but they should nevertheless be subject to scrutiny and rigour in ensuring that

resources are being used to the greatest effect. However, to be sure that interventions are

achieving outcomes there is a need for research – and preferably long-term and rigorous research

that meets the +++ grading under Petticrew and Roberts.

3. It is very challenging to define outcomes. It is especially challenging to define long-term outcomes that are being sought for children as a result of service provision as the outcomes essentially add up to their whole adult lives and successful outcomes have to therefore be defined in terms of how society defines ‘success’ for an adult.

Most outcomes research that is currently being conducted ends when children reach the age of 18

years and become adults. There is some research that continues into young adulthood, up to 23 or

25 years of age, but it is not clear at what point the impact can be considered to be measurable

from services delivered in childhood, such as day care for children with disabilities with the aim of

supporting inclusion in mainstream education, family support services to ensure adequate levels of

care in the family and to mitigate the effects of poverty, or alternative care services, such as foster

care and residential care.

The rights-based perspective set out in the Childonomics conceptual framework (Bilson et al.,

2017) should more clearly guide the outcomes selected for inclusion in the outcomes and results

matrices, and this aspect of the methodology may require strengthening and further testing.

All of these challenges require further exploration and elaboration in future iterations of the

Childonomics methodology and in further conversations on developing methods for measuring

return on investment in children and families.



References

Appogg agency. (2015a). Annual Statistical Report, FSWS.

Appogg agency. (2015b). Children in Specialised Foster Care (mimeo), FSWS.

Appogg agency. (2016a). Annual Statistical Report, FSWS.

Appogg agency. (2016b). Youth 16+ in Care, Out Of Home Care, Residential Homes (mimeo).

Bilson, A., Herczog, M., Kennedy, J.A., Kuzminskyi, V. and Rogersl, J. (2017). Childonomics: A

Conceptual Framework, Eurochild/OPM.

Bouget, D., Frazer, H., Marlier, E., Sabato, S. and Vanhercke, B. (2015). Social Investment in

Europe: a study of national policies, ESPN, European Commission.

Caritas Malta. (2016). A minimum essential budget for decent living – 2016, FSWS.

Gheorghe, A., Rogers, J., Kennedy, J., Kuzminskyy, V. (2017) Childonomics: Methodology for

Appraising the Return on Investment of Social Services for Children and Families, Eurochild/OPM.

Government of Malta. (2016). Child Protection and Alternative Care Act.

Government of Romania. (2015). National Strategy on Social Inclusion and Poverty Reduction

2015–2020, Ministry of Labor, Family, Social Protection and Elderly. Available:

http://antisaracie.mmuncii.ro/en/biblioteca.html Last accessed 20th February 2017.

Hope and Homes for Children. (2013). The Financial Impact Of Reform Of The Public Child

Protection System In Romania. Available:

https://drive.google.com/file/d/0B3UFn1Xvp24mV015dTRVdXNRWHc/view Last accessed 14th

December 2017.

Ministry for the Family and Social Solidarity. (2016). National Children Policy. Government of

Malta.

National Institute for Health and Care Excellence. (n.d.). Glossary. Available:

https://www.nice.org.uk/Glossary?letter=C Last accessed 14th December 2017.

PwC and UNICEF. (2016). Financial Impact Analysis For Scaling Up A Model Of Community

Based Services At National Level, Alpha MDN, ISBN 978-973-139-342-1.

World Bank, UNICEF, NAPCRA. (2016). Children in Public Care. World Bank, UNICEF,

Government of Romania.

World Bank/Government of Romania. (2016). Increase Income Support For The Poor And

Introduce Pro-Work Incentives For Programme Beneficiaries.

http://antisaracie.mmuncii.ro/en/biblioteca.html

https://drive.google.com/file/d/0B3UFn1Xvp24mV015dTRVdXNRWHc/view

https://www.nice.org.uk/Glossary?letter=C



Annex A Mapping and context - Malta and Romania

A.1 Romania

Demographic snapshot11:

Total population: 19, 870, 647.

Child population under 18 years of age: 3,734,667.

Children in care: 57, 279 (1.53% of child population).

Of these, 20,291 are in residential care (35.4%) and 36,988 are in family-based care (64.6%).

The 2016 report ‘Children in Public Care: Romania’ by NAPCRA, UNICEF and World Bank neatly

summarises the policy context. A history of a de-institutionalisation-focused policy has heavily

influenced the development of the child protection system, prioritising processes such as reducing

the numbers of children in institutional care and ensuring that children in need of alternative care

are placed in family-based care. In the last 10 years or so, policies and legislation have moved

towards recognising the systemic issues driving the high numbers of children entering alternative

care and measures adopted have included better-targeted social assistance, more integrated

social services and strengthened social inclusion and inclusive education.

National policies are currently aligned with EU priorities under the Europe 2020 Strategy and the

2013 European Commission Recommendation on Investing in Children – Breaking the Cycle of

Disadvantage. Key policy documents are the National Strategy for the Promotion and Protection of

Children’s Rights 2014–2020 and National Strategy on Social Inclusion and Poverty Reduction

2015–2020.

The National Strategy on Social Inclusion and Poverty Reduction (hereafter the Strategy)

emphasises the need to break the intergenerational cycle of poverty (Government of Romania,

2015, p.11) and to address deep-seated poverty among those who are very social excluded with

‘people-based and area-based policies’ (ibid., p.12). The aim is to target measures towards

particularly poor geographic areas, specific communities or people with specific vulnerabilities, and

to create more effective inter-disciplinary responses by adopting a life-cycle approach. Children are

recognised in the strategy both as being particularly vulnerable to poverty12 (World Bank, UNICEF,

NAPCRA, 2016) and as key agents13 in breaking the cycle of intergenerational poverty through

achieving better education levels and consequently better employment opportunities and earning

potential than their parents and grandparents.

The social inclusion strategy focuses on equality of opportunity for children: ‘For children (those

between the ages of 0 and 17 years old), the goal of the Strategy is to ensure that all children have

the opportunity to develop their full potential regardless of their social background by ensuring the

conditions necessary to develop the skills, knowledge acquisition, and experience needed to

11 National Institute of Statistics – total population and child population 1 January 2015; NAPCRA child protection data for 31 December 2015. 12 The executive summary, in English, states: ‘…one-third of Romania’s 3.8 million children live in poverty… the poverty rate for children has consistently been much higher than the overall national rate and in rural areas this rises to around 53% of children living in poverty compared to 17% of children in urban areas – 74% of poor children live in rural areas … income poverty coupled with the lack of access to basic services and an inefficient labor market increases the vulnerability of these children to chronic poverty when they become adults…’ 13 Ibid. p. 11 ‘The intergenerational cycle of exclusion is perpetuated when low educational achievement and poor health severely limit the labor market opportunities that are accessible to the next generation of children. Breaking the intergenerational cycle of poverty and exclusion will require targeted interventions designed to address the multiple drivers of inequality.’



achieve their full potential as successful students, confident individuals, responsible citizens, and

effective contributors to society’s development’ (Government of Romania, 2015, p. 11).

Nine groups of interventions are envisaged in the Strategy: education, health, housing, social

services, employment, social transfers, participation, capacity building and area-based (see Table

26, which is extracted from the World Bank/Government of Romania ‘Monitoring and Evaluation

(M&E) Plan for the National Strategy and Action Plan on Social Inclusion and Poverty Reduction,

Romania’, 2016).



Table 26: A breakdown of the general objectives of the Strategy by type of policy



For the purpose of Childonomics, the social services interventions are of key importance, with

some of the measures foreseen in education and area-based policies also relevant as they have

the potential to significantly impact on children and families and the outcomes the investments are

intended to achieve are clearly defined:

Examples of key social services interventions14:

3.5 – Develop integrated intervention community teams to provide social services (in education,

employment, healthcare, social protection and other public services) and social intermediation and

facilitation programmes at local level, especially in the poor and marginalised areas, rural and

urban, Roma and non-Roma, by:

i. developing clear methodologies, protocols, and work procedures for community-based

workers; and

ii. developing, in the larger marginalised areas, multi-functional community centres to provide

integrated services to (primarily though not exclusively) families in extreme poverty.

(Proposed Flagship Initiative).

3.6 – Strengthen social services for child protection by:

i. developing and strengthening the capacity of community-based prevention and support

services;

ii. reconsidering the ways and means (including cash benefits) of providing family support in

order to prevent child-family separations;

iii. revising the existing child protection services to enhance the quality of care provided while

reducing the duration of stays to the minimum necessary (proposed Flagship Initiative);

iv. de-institutionalisation and transition to community-based care; and

v. providing funds for the social infrastructure needed in order to deliver social services to

children (day care centres, group homes, etc.).

3.6 – Develop social services for vulnerable groups by:

i. increasing the financing of social services and improving the procedures for contracting out

social services to non-government and private providers; and

ii. strengthening the role played by the local government child protection authorities in strategic

planning, methodological coordination and supporting municipal authorities at community

level, as well as in monitoring and evaluating service providers within the county.

Key area-based interventions:

8.4 – Reduce concentrated and persistent poverty in marginalised communities by implementing

‘packages’ of integrated social services (integrated intervention community teams and multi-

functional community centres): coordinated access to education, health, employment and housing.

Key education interventions:

4.2 – Design and implement a national programme focused on children at risk of dropping out, or

already dropped out, of primary and secondary education by:

14 All interventions are from Table 1 of the Strategy, p. 15 (accessed at http://antisaracie.mmuncii.ro/en/biblioteca.html) but numbers have been assigned to correspond with the summarised interventions in the M&E plan in Table 26 above.

http://antisaracie.mmuncii.ro/en/biblioteca.html



i. designing a coherent referral to education system, with entry points in all the systems dealing

with children from vulnerable families; and

iv. designing a programme to provide assistance, parental education, support and monitoring for

the households of out-of-school children of compulsory school age.

The M&E plan sets out ‘key headline indicators’ for each objective for which data are being

regularly gathered and monitored. These include the following:

Indicators for social services:

14. – Total number of children in the special protection system (irrespective of measure, service or

provider) by: age group; sex and ethnicity; urban/rural status; population size category; and at

county, regional or national level. Also, quarterly and annual means of this number.

15. – Number of children with disabilities in the special protection system (irrespective of measure,

service or provider) by: age group; type of disability; sex and ethnicity; urban/rural status;

population size category; and at county, regional or national level. Also, quarterly and annual

means of this number.

Indicators for education:

17. – Out-of-school rate for children of primary school age (7–10 years), by: (equalised disposable)

income quintile; sex and ethnicity; population size category; urban/rural status; and at regional or

national levels. Also, annual mean of this rate.

18. Out-of-school rate for children of secondary school age (11–14 years), by: (equalised

disposable) income quintile; sex and ethnicity; population size category; urban/rural status; and at

regional or national levels. Also, annual mean of this rate.

19. Share of children with disabilities (0–17 years) enrolled in (mainstream or special) education,

by: educational stage; age group and sex; at sirsup (commune/town) level; population size

category; urban/rural status; and at county, regional or national levels. Also, annual mean of this

share.

20. Share of adolescents aged 15–18 not in education or training (early school leaving rate), by:

(equalised disposable) income quintile; sex and ethnicity; population size category; urban/rural

status; and at regional/national levels. Also, annual mean of this share.

21. Early school leaving rate for youth aged 18–24 (Eurostat definition), by (equalised disposable):

income quintile; sex and ethnicity; population size category; urban/rural status; and at regional or

national levels. Also, annual mean of this rate.

Many of these indicators are, however, process indicators, which do not give insight into longer-

term outcomes for children and families. There are indicators under the employment and social

transfer objectives that could offer some insights if they could be further disaggregated for whether

young people or adults had been enrolled in the special protection system. The AROPE, which is

being used to monitor effectiveness of social transfers, can, if applied to particular groups of

children and families (those in particularly poor and marginalised communities, those who have

been receiving certain types of family support or alternative care services) tell us if the goal to

break the intergenerational cycle of poverty is being achieved.

The National Strategy for the Promotion and Protection of Children’s Rights 2014–2020 has three

objectives(summarised in UNICEF, World Bank, NAPCRA, 2016):



i. increasing the efficiency and effectiveness of the current family-type child care system;

ii. banning the institutionalisation of young children; and

iii. closing down old-style institutions and developing community-based services.

This third objective is expected to lead by 2020 to two key outcomes:

i. all old-style institutions will have been closed down; and

ii. at least 25% of children at risk of being separated from their families will not have ended up

in the special protection system.

These objectives are largely absorbed within the Strategy on Social Inclusion and Poverty

Reduction, which was introduced later. The report Children in Public Care (UNICEF, World Bank

and NAPCRA, 2016) makes recommendations that are mainly focused on children already in the

system, including in relation to the theme of improving outcomes for children leaving the system

(ibid., theme 5 p. 18) – these outcomes are focused on housing and ability to live independently;

education, skills training and employment; rehabilitation and habilitation for young people with

disabilities to maximise functioning; access to services, including health and legal services, cultural

and outdoor activities; and ability to lead a balanced life (i.e. family counselling and sex education).

Existing services and provisions

Social security

The government makes provision for a range of cash benefits, including social allowances, child

raising allowance and emergency aid. There is also a health insurance fund allowance. Some of

these provisions directly support children and families. The allowances are set out in dark blue

cells in Table 27 and can be described as reflecting:

cash support for child raising through a universal state allowance for children, means-tested

child raising allowance and family support allowance;

a concern to provide cash and transport support for people with disabilities, both adults and

children, and persons who care for a child with disabilities – the amounts reflect different levels

of disability; and

cash support targeted to very poor households to guarantee a minimum living income

(Guaranteed Minimum Income) and support with heating – at quite low coverage, with only

24.7% of the poorest quintile benefiting from the Guaranteed Minimum Income, 30% from

family support allowance and 39.2% from heating benefit during the winter.15

The government is planning measures to improve access to these packages of support for the very

poor.

Services

A range of relevant services are being delivered, including community-based integrated social,

health and education services, ‘special protection services’ for children in need of alternative care,

certain kinds of disability services, and school transportation services. Some services are new or

are only now being established (e.g. multi-functional community centres for poor families) and

others are more established (e.g. foster care). The government appears to have decided to invest

15 World Bank, estimations using data from the 2012 HBS, first quarter (cited in p. 8 of World Bank/Government of Romania (2016) ‘Increase income support for the poor and introduce pro-work incentives for program beneficiaries’, accessed from http://antisaracie.mmuncii.ro/downloads/IP_2_VMI_June_2016.pdf )

http://antisaracie.mmuncii.ro/downloads/IP_2_VMI_June_2016.pdf



significantly in the roll-out of community-based integrated teams of social, health and education

specialists, and this appears to be a newly emerging ‘core’ service aimed at supporting families to

care for children and to prevent the need for more specialised services or alternative care services

(PwC and UNICEF, 2016).

Table 27: Cash (shaded dark blue) and services (shaded light blue) available to support children and families in Romania

Universal

provisions and

services to

support children

and families

Targeted

provisions and

services to

support children

and families

Specialised

provisions and

services

Highly specialised

provisions and

services

Alternative

care

provisions and

services

State allowance

for children

(3,961,195

beneficiaries*)

Guaranteed

Minimum Income

(245,545

beneficiaries*)

targeted to poorest

5%**

Social allowances

+ personal

monthly

complementary

budget for

children and

adults with

disabilities

according to

degree of

disability

Monthly food

allowance HIV/AIDS

Foster home

allowance and

child raising

incentive? (no

details about

the differences

between these

two provisions)

Health Insurance

Fund allowance?

(unsure if means-

tested or

universal)

Family support

allowance targeted

to families with

children in the

poorest three

deciles**

Attendant

allowance for

adult caring for

child or adult with

disabilities

Aid for refugees

Child Protection

Department

(CPD) services:

reintegration

services;

adoption

services

Heating benefit

(targeted to

households in the

poorest 60% of

income

distribution)**

Free public

transport for

persons with

disability and

children with

disability and their

attendant (also

exemption from

vehicle tax

‘vignette’)

Night care social

centres (including for

homeless children

and families?)

Transit centres

for young

persons leaving

the protection

system

School

transportation?

Child-rearing

benefit (138,350

beneficiaries*)

means-tested and

based on ‘family

context evaluation’

Children with

disabilities: day

care and

‘prevention of

separation’

services

CPD services for

children who are

abused, neglected,

mistreated, or

abandoned:

psychological and

legal advice; financial

aid; protection

measures (see next

column)

CPD services:

foster care;

formal kinship

care

Free education for

10 grades Emergency aid

‘Second chance’

school drop-out

prevention

services

CPD services for

children with

disability: school and

employment support;

neuropsychomotor

recovery services

CPD residential

services:

emergency

reception

centres;



centres for

children with

special needs

Basic financing for

pre-school

‘High school

money’, ‘Each child

in kindergarten’

and ‘School

stationery’

programmes to

improve access to

education for

children from low-

income households

‘School after

school’ for

children outside of

formal education

CPD services for

parents: measures to

prevent abandonment

and

institutionalisation;

strengthening of

parents’ abilities and

knowledge of child

care and education;

crisis-overcoming

services

Orphanages

Small family

type residential

care homes

School mediation

services (to

prevent school

drop-out – also

works with parents)

Social assistance

for pregnant

women/mothers

in hospital

maternity

departments

CPD services

‘alternative protection

measures’: day care

centres for children,

out-of-school children

and homeless

children

Social housing

services for low-

income or

homeless people

(including families

with children)

Social

development

services –

counselling for

aggressors

CPD services: mother

and baby unit

Community centres

for Romani people

(including families

with children?)

Social

development

services – child

behaviour

rehabilitation

Social welfare

canteen (including

for families with

children?)

New emerging

services***:

Multi-functional

community centres

for families with

children in extreme

poverty

Multi-disciplinary

community teams

(social work,

health, education)

*Average number of beneficiaries per month in 2015 according to Ministry of Labour, Family and Social Protection data provided to the project

**A planned project will merge these programmes into a single means-tested programme, the Minimum Social Insertion Income, as set out in the Strategy on Social Inclusion and Poverty Reduction.(World Bank/Government of Romania, 2016). This will include school attendance and employment conditionalities.

*** Those mentioned in the government strategies as being planned for introduction by 2020 and that have already been piloted and costed (see PwC and UNICEF, 2016).



A.2 Malta

Maltese government policies on social investment can generally be characterised as latent and

redistributive, aiming to, in the case of children and families, address current situations and

‘improve chances of less stressful situations later on in life’. (Bouget, 2015)

The high-level goal of the National Children’s Policy, launched for public consultation16 in 2016, is:

‘a lively, active and happy childhood [and] … also … a prosperous future’

The Vision is:

‘children are respected, protected and aware of their rights’… ‘children’s opinions are

heard, encouraged and supported’… so they can ‘fulfil their potential … and grow into

responsible and proactive adult citizens’. (National Children’s Policy, 2016 p.4)

There is an emphasis on the empowerment of children as a pre-requisite for empowered and

engaged adults. The four areas of goals and outcomes set out in the National Children’s Policy

2016 are as follows:

Long-term goals Short-term goals

Have active and healthy current and future generations

Children live an active and healthy lifestyle

Protect children and ensure an adequate living environment

Children are living in a safe and economically stable family environment

Children grow up to be independent and responsible adults

Children achieve their maximum potential in education, and later in future employment

Children become active participants within their community and state

Children are respected and enabled to be active participants in their social surroundings

While recognising the interdependency of each of the outcomes, the key outcome relevant to the

focus of the Childonomics framework is ‘Children living in a safe and economically stable family

environment (ibid. p.6). The overall goals and these specific outcomes refer to financial or

economic security and stability, both during childhood and once children reach adulthood. The key

indicators for measuring these outcomes should therefore include some measure of ‘prosperity’

and ‘economic stability’. The concept of an ‘adequate living environment’ is also important as it

suggests both economic adequacy but also adequacy in meeting children’s’ other needs –

developmental and psychological – and these needs should be defined and measured accordingly

when assessing achievement of outcomes.

The government’s new policy is strongly oriented towards the United Nations Convention on the

Rights of the Child, as is new child protection and child care legislation that will be enacted in 2018.

(Government of Malta, 2016). A range of legislative acts, as well as the new policy, all have ‘child

wellbeing as an important objective’ (National Children’s Policy, 2016, p.10) and clarity is needed

on how child well-being is to be defined and measured.

16 The Child Policy was launched for public consultation in 2016. It was redrafted based on the feedback, and the final detailed policy was published in its final version in November 2017 just as this report was being finalised. This review was based on the draft version launched for public consultation.



One of the drivers underpinning the new policy is the need to ‘promote the mainstreaming of

children’s issues across different policy areas’ (ibid. p.11) to strengthen intersectoral and

interagency working in securing the rights and well-being of children.

Examples of proposed actions from the Family and Relationships domain (ibid. p18-19) in the

National Children’s Policy that can provide measures for outcomes for children that are related to

the care provided in the family or alternative care setting include the following:

8. – Support the family and out-of-home care providers to offer an environment that enhances

the full and harmonious development of the child.

11. – Implement and coordinate family plans and support units within the community which

offer support and also serve as a referral to more specialised services as needed.

14. – Strengthen and increase the provision of support to families whose children require

special attention due to certain conditions (e.g. mental health, addiction, disability, etc.) and/or

situations (e.g. pregnancy, etc.).

26. – Strengthen outreach and community-based initiatives to provide greater mentor-style

support to vulnerable families with children.

Existing services and provisions

Social security

The government makes provision for a range of cash benefits to support children and families, as

set out in dark blue cells in Table 2817. These benefits can be described as reflecting:

a concern to bring low-income families with children up to a minimum guaranteed income;

a low level of cash support to a high proportion of the child population (according to MFSS

data, almost 60% of the child population received Children’s Allowance in 2013–2015, at an

average amount of €987 per child per year);

a concern to ensure that benefits and social assistance do not become a disincentive to

employment;

inconsistent (for example, people with sight impairments have almost five times greater

amounts of cash assistance than those with other types of disability) and generally relatively

low levels of cash support for children and adults with disabilities compared to that available to

people with other special needs (€20 per week for each child with disabilities, regardless of

level of additional needs, compared to €24.81 per week for people with special dietary needs or

€70 per week for each child in foster care); and

a low subsidy for a child in foster care (€70) compared to unemployment assistance or a blind

person’s pension.

Services

In Malta services for families and children are delivered through two modalities: 1) government-

funded service delivery partnership agreements with NGOs and faith-based organisations; and 2)

government foundations and agencies.

The specialised government FSWS has three service delivery organisations:

17 Based on a review of information provided to the project by MFSS in January and April 2017.



1. Appogg – child and family social services and residential shelters for adults;

2. Sedqa – psychosocial services focused on addictions and violence, psychological support and family therapy; and

3. Saport – residential and community-based services for people with disabilities.

A range of services, mainly under Appogg, provide support to children and families, including

community-based social services for all community members, children and families, and more

targeted and specialised services, such as child protection services. There are some Sedqa-

managed services that work with adults on violence, substance abuse and addictions, which also

have an impact on family care and children in some cases. Key elements (but not all) of the

network of services are briefly summarised, according to the Childonomics typology, in the

unshaded cells in Table 28 (cash benefits in shaded cells).

Table 28: Cash (shaded dark blue) and services (shaded light blue) available to children and families in Malta mapped on to the Childonomics conceptual framework

Universal Targeted Specialised Highly specialised Alternative care

Children’s

Allowance Flat

Rate (CAFR) –

€450 annually per

child for all families

with children

except those

receiving targeted

children’s

allowance (CA)

Children’s Allowance

(CA) – €450 - €1,156

annually per child for

families with children

and income below

€24,620

Disabled

Children’s

Allowance

(DCA) – €20

weekly

supplement to

the CA or CAFR

for each child

with disabilities

Sickness

assistance (SKA) –

24.81 weekly for

persons with a

condition requiring

a special diet

Foster care

allowance (FCA)

- €70 weekly per

child for

approved foster

care placements

Social assistance

(SA) - €100.48

weekly

unemployment

benefit for those

seeking work (with

or without children

in the household),

those unemployed

as they are caring

for adults with

disabilities or those

who are not

capable of working.

In work benefit (IWB)

– €52 to €1,200

annually per child for

families with

children/young people

in the household

under 24 years old

and adults in

employment earning

below a certain level

Intake & Family

Support Service

– voluntary

family support

service, but also

a service to

which CPS

might refer

cases with

lower levels of

assessed risk -

1825 cases

worked with in

2016 of which

611 were

ongoing cases

or returning

cases.

Blind pension

(BLD) – €94.35

weekly for children

aged 14 years or

older and adults

with a certified sight

impairment

Milk grant – €20.62

per week for

households with

Social or

Tuberculosis

Assistance with

babies requiring

weaning or

complementary

feeding

Looked after

children service

– placement and

social worker

support for 451

children in

residential and

foster care in

2016 including

49 new children

referred

Unemployment

assistance (UA) -

€100.48 weekly for

persons registered

unemployed and

meeting means

and capital

resources test

Unemployment

assistance tapering

(UAT) – reducing

Maternity benefit (MB)

and Maternity leave

benefit (MLB) –

€86.77 weekly for up

to 14 weeks plus

€160 weekly for up to

4 weeks for pregnant

women not eligible for

maternity leave

benefit (MLB) from

Programm Ulied

Darna and

Home Start –

volunteers /

mentors

supporting

families in their

homes

Child Protection

Services (CPS) –

child protection

investigations

followed by either

1) case closure, or

2) social contract

or, 3) if problems

persist, care order

to out of home

care. 790 referrals

Fostering service

– training and

licensing of

potential foster

carers; support

to foster carers;

promotion of

fostering – 243

children were in

foster care at the

end of 2016



amounts of UA, SA

or SUP

the eighth month of

pregnancy

received in 2016

and 523 cases

carried forward

from 2015

Free health care

including post-natal

services:

CommCare by

MFSS (home visits

by midwife/nurse)

and Well Baby

Clinic by MFH (3

post-natal check-

ups at 6 weeks, 8

months and 18

months - child

attends clinic)

Supplementary

allowance (SPA) -

€4.57 - €8.13 weekly

for persons with

annual income below

a certain limit

Single unmarried

parent (SUP) - €75.36

weekly for single

unmarried parent with

income below a

certain level

Embark for Life

youth

employment

service and

other youth

services

Ghabex Shelter –

residential service

for women and

their children

experiencing

violence at home.

30 women with 51

children and 43

women with no

children received

this service in 2016

Residential care

services –

delivered by

Appogg and

through social

partnership

agreements by

Kummissjion

Ejjew Ghandi

and NGOs

funded by the

Ministry of

Family and

Social Solidarity

– 208 children

(?) at the end of

2016

Early childhood

development

services and free

primary and

secondary

education and

school transport

Community Services

(Cottonera,

Birkirkara/Msida,

Qawra, Valletta) -

numbers of cases

vary from 92 to 294

cases in 2016

depending on the

community. Also

LEAP programme -

providing food

support and referral to

community child care

and employment

services (24,000

households by 2018)

Community

Services for

people with

disabilities

(adults) – 915

service users in

2013 (day

centre, personal

assistance and

intervention,

social work

service)

Psychological and

Family Therapy

Service – for

people with

substance abuse

problems and their

families. 61 family

cases and 419

individual cases

worked with in

2015

Residential

services for adults

with disabilities

requiring high

levels of support –

83 residents in

2013

Adoption service

– 384 pre-

adoption cases

worked with in

2016 and 178

post-adoption

families (238

adopted

children)



Annex B Data collection instruments and ethics guide

Ethical considerations and data collection instruments

Introduction

This is an overview document prepared for the consideration of OPM Ethical Review Board ethical

review as Childonomics was being developed. The purpose was to have the Childonomics

approach and tools ethically reviewed and approved before publication and implementation. Users

of the Childonomics tool, should however, make appropriate arrangements for ethical review for

each CCA undertaken.

Included in the primary data collection methods are consultations with service providers (staff) and

service users (adults, youth and children). This includes staff at all levels of organisation, parents

and other adult groups, as well as children and youth voices – particularly children in alternative

care and children with disabilities.

Project description

Childonomics is a research project aimed at developing a tool to determine the long-term social

and economic return of investing in children and families within a given national or subnational

context. The tool will focus on the costs and outcomes associated with systems which rely on

institutional care for children with disabilities and children separated from their families in the

child protection system. These will be compared to systems which offer prevention, early

intervention and family-strengthening services, high-quality family-based care for children

separated from their parents, and the possibility of reintegration of children into their families and

communities.

The methodology focuses on the impact dimension in the above conceptualisation, and

specifically on valuing the investments and outcomes associated with respective services. In line

with the Childonomics scope of work, outcomes go beyond the perspective of the child to

include parents and the extended family, communities and the society at large. A key aspect

envisaged by the tool is to use the available datasets and evidence in an economic modelling

exercise that estimates long-term costs and outcomes associated with respective services.

Who will be included in primary research gathering?

Parallel to any desk research and data gathering, it is important that we also include the practical

reality of implementing services and support to families and children. In order to be able to have a

complete understanding of the policies, the procedures and the experiences, as well as being able

to link these to outcomes, impact and consequences, it is necessary to include many stakeholders,

representing different experiences. This is laid out in the table below.

Possible participants Type of method(s) used to gather data

Service providers (workers)

Managers, social workers, social care workers, child care staff, residential care staff

Key informant interviews

Focus groups

Case file reviews

Service providers (data access)

Economists, financial managers, accountants, managers


Focus groups

Case file reviews



Service users (adults) Parents, foster carers, kinship carers, guardians, adult care leavers18

Semi-structured Interviews

Focus groups

Service users (children)

All children generally, but also, specifically, children in foster care, children in residential care, children in institutional care

Semi-structured Interviews

Focus groups

Consulting children and youth

When considering which stakeholder groups to include, particularly ones with specific needs or

vulnerabilities, the ethical intrusion into their lives or work must justify the insights that you may

garner. Having already established the project as informed by the UN Convention on the Rights of

the Child, it is clear that children and youth participation should be part of the stakeholder

consultation process19. Parallel to this, while taking a systems-level and holistic approach to

understanding social investment in children and families, it is also necessary to understand the

daily reality of how this is achieved and the impact it can have on all stakeholders. To that end, it

was clear that children and young people should be included.

As children are identified as a vulnerable group in society, the ethical approach to consulting any

child necessitates particular considerations. These are addressed in the following sections.

Research design

The purpose of the instruments is to consult with various

stakeholders about how they view the long-term social and

economic return of investing in children and families. Many things

may inform the choice to carry out focus groups or interviews,

including: the competence of the organisation and of the

researchers; the gatekeeping or access considerations regarding

how many children or youth you can consult with; available

resources; and the ethical considerations.

Ethical considerations and procedures

All decisions made in relation to this research are and will be based on the belief that the research

process should be beneficial to all – the researcher, the research participants, the research

supervisors, the research sponsors and society in general; that all people deserve respect and

honesty; and that all people have the right to own their own data and narratives, and to control how

much of those data and that narrative appears in the final publications and in any related articles.

The British Educational Research Association (2011: 5) recommends that ‘this ethic of respect

should apply to both the researchers themselves and any individuals participating in the research

either directly or indirectly’.

18 Care leavers are teenagers or adults who have been in alternative care when they were under 18 and have since transitioned into adulthood, out of children’s services, and are eligible for adult services – i.e. that at some stage they lived in foster care, or institutional care, or residential care and have reached independence. 19 Particularly, but not only: Article 3 – that the best interest of the child must be a primary consideration in all actions concerning children, and Article 12 – that the child’s view must be considered and taken into account in all matters affecting them. The articles of the Convention are often grouped into survival rights, developmental rights, protection rights, and participation rights.

Ethical? ‘Just because you want to do research, doesn’t mean you can… Just because you can do research, doesn’t mean you

should.’



Informed consent and assent

The principle of informed consent is a fundamental ethical safeguard when undertaking research

with human participants. Informed consent for the pilots was achieved through providing adequate

and accurate information to all potential participants. Information on this study, including the goals

of the research and an assessment of the risks and benefits of the research, was provided to each

of the participants in two forms: an information sheet for each stakeholder group (in appendix)

was developed and provided before the day of the consultations; and, secondly, the facilitators

also verbally explained the project, the purpose of the consultation, what informed consent means

and that all of their contributions would be recorded as anonymous and treated as such.

Each adult participant was also asked to sign a consent form

(Annex B3) to show that they understand the purpose of the

research and that anything they say will be anonymised, and that

they know where the information they provide will be used. For

children who are under 18, the written consent of their

parent/guardian (Annex B4) or, for children in formal alternative

care, the consent of the local authority as their legal guardian,

was required.

At the start of the consultations, the assent of the children and

their willingness to participate was also sought. In order for children to give valid consent, it is

important that the research process engage them through developmentally appropriate activities

and age-appropriate language.

Emotional harm

This ethical consideration is particularly relevant when it comes to research with families and

children due to the vulnerable status of children as a social group. In-depth interviews and focus

groups can sometimes evoke difficult memories for research participants. In this study, the risk of

emotional harms is particularly relevant to participants, both children and adults: the interviews

may touch upon their personal lives, their experiences and their expectations for the future. Any or

all of these issues may give rise to unpleasant memories.

If any participant becomes distressed in interviews, questioning will stop until the participant

indicates that he or she wishes to resume the interview. In the context of the focus groups, there

will always be at least two facilitators so that if a participant is upset, they can have individual

attention from one of the facilitators and decide if they want to continue or not. Participants in either

an interview or focus group will be assured that the decision is totally up them. Participants will be

reminded that their participation is voluntary and that they are free to withdraw from the research at

any point and without consequence. All participants have been identified through a local service

provider and their support worker from that service will provide support and follow up after the

research events are ended.

As the main focus of the research is on the care of children and families, there is a risk that social

workers, managers, parents, and carers may feel that they are under a certain amount of personal

and professional scrutiny that could conceivably lead to feelings of inadequacy and therefore

emotional harm. Social workers and managers will be assured that the purpose of the research is

not to scrutinise their personal work practices but instead to gain general insight into the nature

and delivery of family and children services from their perspectives. Parents and carers will be

assured that the purpose of the research is not to scrutinise their parenting care practices but

instead to gain general insight into the nature and delivery of family and children services from their

perspectives. This information is also included in the information sheet.

Recommended reading: ‘Guidance for developing ethical research projects involving children” (Department of Children and Youth Affairs, 2012).



Confidentiality and anonymity

The secure storage of data and personal information, and the protection of research participants’

identity, take on special significance in a study of this nature. Various types of data may be

generated in the context of this research: field notes, participant surveys, interview recordings

(audio or written recording), art work by children (which may include identifying information), and

more.

Interviews may or may not be audio-recorded. If the interviews are audio-recorded, permission to

record the interviews will be requested from all participants, and if a participant does not wish for

the interview to be recorded, with permission, notes will be taken instead. If audio-recorded,

interviews will be transcribed immediately and following this, digital recordings will be erased.

Transcription will occur via a reputable person/organisation or by members of the research team.

All information (e.g. recordings, notes, art work) will be anonymised. All computer files will be

stored on password-protected computers. Hard copies of all information will be held in a locked

filing cabinet in a suitable setting. Staff and consultants have been made aware of their

responsibilities as data holders with regards to all data protection and data management

legislation. Any disseminated data from the project will be anonymised and any identifying factor

removed or changed to a pseudonym; this includes data relating to all individuals and location

sites. Researchers cannot guarantee that all participants in a group interview will maintain

confidentiality, but participants will be asked by group facilitators to respect group confidentiality

and explicitly acknowledge their responsibility to keep information about other participants

confidential in their participant consent form.

Choosing suitable methods

The choice of research method is an important one. There is no such thing as a wrong research

method, but, rather, one may be more or less suitable to the research question. This research

focuses on people’s experiences of interacting with systems and with other people within those

systems. The decision regarding research method is also informed by practical factors such as

how much time and resources are needed to implement each research method, or how qualified,

skilled or specialised the researcher may need to be, as well as the feasibility regarding how much

time each method would need in order to be conducted properly. Bearing these factors in mind, it

was decided to prioritise key informant interviews, case reviews, semi-structured interviews and

focus groups as the most suitable research methods.

The Childonomics toolkit includes various documents to support those wanting to implement the

research. It is designed to support focus group and interviews with stakeholders generally and

includes recommendations about the recruitment process, and the various considerations and

decisions to be made about location, facilitator and process, as well as practical advice about how

to prepare and conduct interviews and focus groups with children.

Choosing a suitable method

For the purpose of Childonomics, it was decided that three particular methods would be developed

and included in the toolkit. This was based on three broad categorisations of relevant stakeholders

and the methods that best suited both the participant and the type of information that is being

elicited.

The overall reliability, validity and ethical acceptability of research studies, especially when

involving children, can be enhanced by choosing appropriate methodologies and methods which

are in line with children’s views and the individual way in which they experience the world (Thomas

and O’Kane, 1998).



Stakeholder groups Recommended methods

Service providers (directors, management, staff)

Key informant interviews,

Semi-structured interviews

Service users (adults)

Semi-structured interviews,

Focus groups

Case reviews

Service users (children and youth)

Semi-structured interviews,

Focus groups

Case reviews

Participant recruitment

A combination of purposive and convenient sampling was used to ensure

access to the research participants and processes necessary to achieve

the objectives of the research. The sampling method was particularly

significant at two stages of the research: (i) the choice of services (and

therefore service users) to be considered; and (ii) the selection of a

sample of real cases, including the service users themselves, to include

in the primary data collection.

These factors are informed by the resources and capacity of the

implementing entity and should be guided with consideration given to the

ethical consequences of doing research with vulnerable people.

Supporting documents to facilitate reflections on ethical issues regarding how to conduct research

with children specifically, and vulnerable groups more broadly, will also be provided as part of the

Childonomics toolkit. The toolkit includes some of the initial decision making considerations

regarding how to consult with children and youth, as well as some recommended reading and

resources.

Chosen methods


Key informant interviews, also known as expert interviews, can be used to consult with and include

specific types of staff or representatives from various levels of service provision (e.g. directors,

middle management, daily contact workers). This type of intensive probing interview focuses on

understanding and recording the nuance of each person’s particular role while also understanding

how that role fits within the wider system in providing services to children and families. As key

informants, their contact information may also be gathered in order to follow up after the interview if

needed.

The researcher may have some pre-prepared questions (Annexes B5 and B9), but it is very likely

that they will diverge from these as a more in-depth understanding is developed. The information

that the informant provides may not otherwise be available to the researchers as it is anchored in

understanding and performance of their particular role. It is particularly relevant when a person has

been in that role for a relatively long time and may comment on changes or trends. While the

purpose of Childonomics is not to evaluate a person or a service, we do want to record and

understand the reality of the services, as well as include this particular stakeholder group in

providing insight into the nature of the system and any problems – and to give recommendations

for solutions based on their expertise. It may also be more or less suitable to conduct these

interviews one-to-one or as a group interview. The benefit of one-to-one is increased anonymity as

well as a particularly in-depth subjective account. The benefit of a group interview is that

Recommended reading: ‘The role of adults in creating an enabling environment’ (Save the Children, 2004: 57–62)



participants may support each other in offering facts and figures, or specific information. It may

also serve to test if there is some broad agreement if opinions are sought or whether there is

diversity or even conflict in how some key informants experience the system. Key informants were

asked to provide case studies from their practice and although this could be conducted as a paper

file review, in practice, given considerations of data protection, the case studies were presented by

key informants as part of the key informant interviews (see Annex 5).

Focus groups

Focus groups are when a small number of participants are

engaged in a group discussion, focusing on a particular topic or

issue (Wilkinson, 2011). The success of focus groups depends

on the preparation and factors that create a safe environment for

participants. Annex B1 discusses in more detail considerations

when choosing this method for consultation with children and

young people.

Annex B6 details sample group activities for each of the groups

of children (e.g. children in foster care and children in residential

care), as well as consent forms, templates for participant

information leaflets, and some practical considerations about

hosting focus groups.

Semi-structured interviews with children or young adults

Semi-structured interviews are described as ‘purposeful conversation’ (Mason, 2002). Similar to

focus groups, they focus on a particular topic or issue. A semi-structured interview is where there

are a pre-prepared set of questions that the interviewer would like to discuss but the chronology of

asking the questions is not set and is anchored in the interaction with the person, based on the

natural interaction of conversation. The nature of the conversation also means that qualifying or

follow-up questions can reveal insights that the researcher could not have foreseen.

The initial considerations for deciding to hold one-to-one semi-structured interviews with

children or young adults (Annex B1) have fewer criteria than a focus group consultation. Annex

B7 provides the interview question set that may be used to structure an interview with children, as

well as advice on some ethical and practical considerations before conducting research with

children.

Bibliography

British Educational Research Association (2011) ‘Ethical Guidelines for Educational Research’. Available at:

www.bera.ac.uk/wp-content/uploads/2014/02/BERA-Ethical-Guidelines-2011.pdf [Accessed 11 July 2017].

Department of Children and Youth Affairs (2012) ‘Guidance for developing ethical research projects involving children’. Dublin, Ireland: Department of Children and Youth Affairs. Available at:

www.dcya.gov.ie/documents/Publications/Ethics_Guidance.pdf [Accessed 19 June 2017].

Save the Children (2003) ‘So you want to consult with children? A toolkit of good practice’. London, UK: International Save the Children Alliance. Available at: www.savethechildren.org.uk/sites/default/files/docs/So_you_want_to_consult_with_children_-_Toolkit_2004_1.pdf [Accessed: 19 June 2017.]

Advice for facilitation and co-facilitation approaches: ‘So you want to consult with children? A toolkit of good practice’ (Save the Children, 2004: 29–34)



Thomas, N. and O’Kane, C. (1998) ‘The Ethics of Participatory Research with Children’, Children & Society, Vol. 12: 336–348.

B.1 Childonomics: Considerations for facilitating child and youth consultations or interviews

This is a short reference summary of some of the initial considerations that you might contemplate

when designing the child and youth participation element of Childonomics. It is intended to be used

as a reference guide and is built upon some of the best practices that are listed as ‘recommended

reading’ at the end of this document.

The purpose of the instruments is to consult children and youth about how they view the long-term

social and economic return of investing in children and families. Many factors may inform the

choice to hold focus groups or interviews, including: the skills of the researchers; the gatekeeping

or access considerations regarding how many children or youth you may consult with; available

resources; and length of time considerations; considerations about confidentiality and anonymity,

especially in relation to sensitive issues that could come up.

We recommend as part of this process that all research is done with the knowledge and support of

a local organisation, and ideally the local social services. This is to support recruitment of

participants (particularly for children and care leavers). It also has an ethical purpose so that if any

participant finds participating in the research distressing, or if they disclose abuse or violence

during the research, they have a local support person (a social worker or local organisation staff)

who can follow up with them once the researcher has left.

Focus groups:

As already mentions, focus groups refers to a small number of participants engaging in a group

discussion, focusing on a particular topic or issue (Wilkinson, 2011). The success of focus groups

is informed by the preparation and factors that create a safe environment for children and youth to

participate in. One of the benefits of choosing focus groups as the method of consultation is that

you can achieve a level of agreement or diversity in some of the experiences or themes of the

research based on the shared responses of the participants. It is often less resource-intensive to

organise focus groups, compared to individual semi-structured interviews. One of the negatives to

consider is that it is not always possible or suitable to ask follow-up questions to acquire a greater

level of detail (which would be possible with semi-structured interviews).

This is an initial list of things you might consider when deciding whether or not to host focus

groups. It is set out as a spectrum between minimum requirements and ideal requirements that you

might consider when recruiting children or youth, as well as how you might create an environment

that is conducive to a successful consultation.



Focus groups: Child/youth participants

Minimum requirements Ideal requirements Considerations

Age

Separate ages into at least under-12s and over-12s, or by level of ability to engage or type of activity designed for the person

Separate groups by age: 6–10 years, 10–11 years, 12–15 years, 16–18 years and 18 and older

Separate groups by service that they or their family receive

Under-12s and over-12s would need a different methodology. Separating by age and/or service received and/or ability might make facilitation easier but it depends on how many children you are able to recruit.

You may need to consider sibling groups too.

Engaging vulnerable or marginalised groups of children might also involve adapting the activities to their level of engagement, interest or ability, or to avoid trauma

Gender Mixed Balanced

Balanced means proportionate to service recipients gender breakdown. Mixed means that if you cannot get proportionate representation, you still aim to have some diversity of gender. Gender may be a factor in their experience of the service and, therefore, relevant for analysis

Number of participants in each group

At least four

Depending on how long you spend with each group. If under-12s: five children, and a 90-minute session for any one session.

Over-12s you can facilitate 5–10 children/youth for two hours and 15 minutes for any one session (and schedule a break in the middle)

It is recommended that you have at least four youth or children so it is possible to engage in some games/activities. It may also mean you have more diversity of service experience. Time limits, level of understanding based on age and development, as well as language translation may be factors in making these decisions. It is also recommended to consider planning more than one session with each group so that their participation is meaningful and they have more time to consider their participation and the topic

Services Received a service

Separate youth consultations for recipients of ‘universal services’, ‘targeted services’ ‘intense targeted services’ and ‘in-care services’

Depending on location and type of service being offered, you may need to have youth groups with mixed experiences of different services. This does have an impact on the consultation methodology and so is an important decision

Differing abilities

Once you know, in advance, that any attendee has any learning difficulties or disabilities, you can facilitate their participation. They might need a support worker or a higher ratio of adults to children/youth during the consultation

This may impact the choice of venue or you might need more adults to ensure that children/youth can participate to the best of their ability (or even a dedicated worker). It may also slow down the



consultation, depending on their ability, so is something to be planned for

Parent/ guardian on-site

Ideally parents/guardians would stay on-site in case the child wanted to leave for any reason. This is especially true of younger children. It depends on how many trusted adults would be available if the parents were to leave for the length of the consultation

It is better if they stay nearby, or if a childminder/staff member/suitable adult who is known to the children/youth is available, if needs be. It may also be possible to schedule a parallel parent/guardian consultation parallel to the children/youth group depending on resources and capacity

Recognition of participation Suggestions: a printed colour certificate of participation, or a ‘thank you’ card, or a small gift (Sweets? Colouring pencils? Books? A journal?, or a voucher for books?)

You are gaining a lot from their volunteering of their time and life stories. You should recognise that in some way. It is a small gesture but can mean a lot to them to show that you value their participation

Practical/Logistical requests

Minimum requirements Ideal requirements Considerations

Room

At least wide enough for a circle of chairs without tables.

Accessible for anyone with limited mobility

Space to walk around all the chairs

Access to wall or flipchart to display papers

Tables for group work or painting

A colourful and inviting room with good sunlight

Different people learn in different ways. Consider kinaesthetic, visual, auditory ways of communicating

Location Accessible to participants and their parent(s)/guardian(s), with a room for parent(s)/guardian(s) to gather or wait

In a building or location already known to them, or within a reasonable commute

Recruitment of children

You need to know in advance some basics in order to plan an intervention:

number of children per group

ages

gender breakdown

what services they have been linked to (may impact the content of the consultation)

any learning difficulties or disabilities, or particular needs that you can plan for or facilitate



Facilitator (s)

Has previous experience of facilitating consultations with children and meets the national child protection standards for working with children and youth (in some countries this includes a police check)

Has training in child protection, has previous experience of facilitating consultations with children, and is able to respond appropriately if a participant becomes distressed or discloses abuse or violence during the consultation

Facilitators need to be prepared, with relevant training, to respond to distress or to disclosure if this should occur during the event

Length of consultation

For under-12s: You may need to be flexible, but with their attention span and age, 60–90 minutes per session. (We recommend two groups of five children for 60 mins rather than one group of 10 children for 90 mins).

For over-12s: You may need to be flexible on this but ideally 2 hours, 15 mins per session (so you can have a break in the middle)

This is to facilitate a level of detailed conversation and activity to gather the data needed. This is also dependant on local logistics, time of day, availability of the children and guardians etc. You may need to be flexible about how many children/youth you can recruit, and how long you can facilitate a group for

Break time If keeping them more than 90 mins, you may need to incorporate refreshments

Refreshments Water freely available at all times. Tea/coffee or juice with some biscuits/ chocolate if you take a break

Preferably small bottles or cups per person

Time of day

Depending on children/youth and parent(s)/guardian’s availability – Are they in school? Do they have after-school activities? Are the parent(s)/guardian(s) working? How far/near is the venue? If they have to commute, can the parent/guardian stay nearby until you are finished?

You may need to work around this (e.g. evenings, weekends, or early mornings)

Translation (this may or may not be relevant, depending on the make-up of the research team)

A youth worker or social care professional who speaks really good English who is happy to be led by the facilitator, who can meet in advance to explain the participation approach and what to expect, and who is helpful and will participate in the games/activities

A professional translator who has done youth-friendly translation before, is helpful and can participate on an equal footing with the youth. Someone who can meet in advance to explain the participation approach. One youth translator for every five to six youths (so you can do some work in smaller groups if you want)

The emphasis is on the translator’s ability to relate to, and engage, young people more than their qualifications to translate from English. You should plan to meet beforehand so everyone is aware of their roles and responsibilities during the session and so the lead facilitator can explain any of the content to the translator(s) in detail, so they can translate to the best of their ability

Child protection The translator and any support worker should meet the national child protection

The translator and any support worker should meet

It may not always be possible for the researcher to have a risk assessment



standards when working with children or youth (in some countries, this includes a police check)

the national child protection standards of working with children or youth (in some countries, this includes a police check)

completed. By including at least one locally-based, child protection-approved worker/volunteer who can be in the room at all times, the minimum requirement may be met. This person may also be the translator or they might be a support person during the activities.

A list of child protection resources for children and youth locally should be prepared and shared with participants.

Researchers and other facilitators/translators in the room should be prepared, preferably through appropriate training, to respond to distress or disclosure during the activity

Number of adults in the room

Any adults in the room should have specified (pre-assigned) roles. No other adults should be present during the consultation. The minimum would be the lead facilitator and at least one co-facilitator (who might also be the translator and has clearance to work with children/youth). If there is a child/youth with special or particular needs (learning difficulties, disability etc), you may need a dedicated extra person to support that child’s/youth’s participation.

It is recommended that you consider what relationship the children and youth might already have with any of the adults in the room and how this impacts on their ability and choice to participate (e.g. having a child’s social worker as one of the facilitators may or may not be beneficial and is something to consider when deciding about designated adults)

Sometimes if the translator or trusted adult is someone that they also see outside of this consultation, it can impact how they choose to participate. This can be either positive or negative. You should consult the local organiser for more information and make an informed decision in the best interest of the children/youth. Having extra adults present during the consultation changes the dynamics of the consultation and also has an impact on the researcher’s ability to promise confidentiality



Semi-structured interviews

As already noted, a semi-structured interview can be described as a ‘purposeful conversation’

(Mason, 2002). Similar to focus groups, they have a focus on a particular topic or issue. A semi-

structured interview is where there are pre-prepared questions that the interviewer would like to

discuss but the chronology of asking the questions is not set and is anchored in the interaction with

the person, based on the natural interaction of conversation. As a data collection method,

interviews may be more resource-intensive compared to focus groups. They may also be a more

intense experience for the participant. That said, they do allow for follow-up and probing questions

to achieve an in-depth knowledge of the service user’s experience of receiving services.

The initial considerations for deciding to hold one-to-one semi-structured interviews with children or

young adults involve fewer criteria than a focus group consultation. The environment, safety and

ethical considerations are the same.

Interviews: Child/youth participants

Minimum requirements Considerations

Age

That the child or young person is able to understand the types of questions that they will be asked and are able to communicate their opinions

The type of data being recorded are better suited to older children (teenagers) and young adults as they ask about their overall experiences and pathways through care

Gender Balanced Gender is a factor to consider in research design planning. Gendered experience may be relevant for analysis

Researcher/

interviewer

Has previous experience of facilitating consultations with children and meets the national child protection standards for working with children and youth (in some countries this includes a police check)

Is able to respond appropriately, preferably following relevant training, if a participant becomes distressed or discloses abuse or violence during the interview

Services Received a service

Differing abilities

Once you know in advance of any attendee with any learning difficulties or disabilities, you should aim to facilitate them. They might need a support worker with them

This may impact the choice of venue or a dedicated support person or more adults may be needed to ensure that children/youth can participate to the best of their ability. It may also slow down the consultation, depending on their ability, so is something to be planned for

Parent/ guardian on-site

Not essential but a factor to be considered

Depending on the age and capacity of the child or young person, it may be more or less relevant. Some older children might not need their parent or guardian’s support

Recognition of participation

Suggestions: A printed colour certificate of participation, or a ‘thank you card’, or a small gift (Sweets? Colouring pencils? Books? A journal?), or a voucher for books?

You are gaining a lot from their volunteering of their time and life stories. You should recognise that in some way. It is a small gesture but can mean a lot to them that you value their participation

Advice: Before beginning the sessions

The participation of children and youth in Childonomics is research-based and guidance provided

here assumes that their participation will be designed as a standalone event in order conduct a



CCA. This means it will probably be a one-time consultation (or linked sessions over time) but the

guidance can also be used to consult with existing groups of children (for example, a youth in care

network, an after-school children’s group, children who all live in a particular residential home

together, an existing grouping of children (e.g. all children in a creche group, etc.)). You may

consider having more than one session with each group/person if that is more suitable regarding

how they might engage with the questions.

1. Become familiar with the data matrix that will be filled in after each interview or focus group to make sure that each of the topics is discussed with the participant and to the level of detail required for analysis.

2. Become familiar with the interview/focus group questions. Participants may sometimes need you to reframe the question so that they better understand what you are asking them about. Without leading the conversation, it is possible to ask them more directed questions that might trigger a response from them. You can also follow up with more questions before moving to the next topic.

3. Consider using some written visuals in the interview/focus group if a participant becomes confused, or in order to better explain or illustrate complicated family and social relationships. An example of this would be a genogram, which is a graphic representation of family trees and relationships among individuals.

4. A participant may find remembering or thinking about their experiences or their future distressing or emotional. Be aware of their level of comfort and be led by it. Take a break if needed. Remind them that this is voluntary and they can stop the interview/participation at any stage and there will be no negative consequences. Be ready to refer a distressed participant to the support person (social worker, carer, parent, local organisation representative) through whom they were referred to take part in the research. Ensure that you know what you will do if a participant becomes distressed during the interview or group consultation. Have ready a list of local resources for supporting the child or adult after their participation.

5. Always be mindful of children’s rights, of child protection, and of best practice with regard to how children are treated generally, and specifically within the context of how you consult with them.

6. Be sure you know what your legal responsibilities are for reporting disclosures of abuse or violence. If a child discloses abuse, inform the child that you are obliged to pass this information on to a formal service (e.g. the child’s social worker or the local child protection authority or the police) in accordance with local legislation and child protection procedures. If the abuse is historical and mentioned by an adult, be sure you know what the local legal requirements are for reporting such disclosures/as a researcher you may not have such an obligation. Inform the interviewee that they could take up this issue with the child protection authorities and offer them a relevant local contact point.

Recommended reading and useful resources:

Save the Children (2003) ‘So you want to consult with children? A toolkit of good practice’. London, UK: International Save the Children Alliance’.

www.savethechildren.org.uk/sites/default/files/docs/So_you_want_to_consult_with_children_-_Toolkit_2004_1.pdf

Save the Children (2005) ‘Practice Standards in Children’s Participation’. London, UK: International Save the Children Alliance https://resourcecentre.savethechildren.net/sites/default/files/documents/3017.pdf

International Save the Children Alliance (2010) ‘Putting Children at the Centre; a practical guide to children’s participation’. London, UK: International Save the Children Alliance, www.savethechildren.org.uk/sites/default/files/docs/Putting_Children_at_the_Centre_final_%282%29_1.pdf

https://resourcecentre.savethechildren.net/sites/default/files/documents/3017.pdf

http://www.savethechildren.org.uk/sites/default/files/docs/Putting_Children_at_the_Centre_final_%282%29_1.pdf

http://www.savethechildren.org.uk/sites/default/files/docs/Putting_Children_at_the_Centre_final_%282%29_1.pdf



Eurochild (2011) ‘SPEAK UP! Giving a Voice to European Children in Vulnerable Situations: Methodological Framework’. Brussels, Belgium: Eurochild, www.eurochild.org/fileadmin/public/06_Projects/Methodology_Speak_Up_Final.pdf

Department of Children and Youth Affairs (2012) ‘Guidance for developing ethical research projects involving children’. Dublin, Ireland: Department of Children and Youth Affairs, Irish Government, www.dcya.gov.ie/documents/Publications/Ethics_Guidance.pdf

Council of Europe, www.coe.int/en/web/children/child-participation-assessment-tool

www.eycb.coe.int/compasito/contents.html

http://www.eycb.coe.int/compass/

An app that has been developed with the goal of giving children the opportunity to ‘speak their mind’: http://mindofmyown.org.uk/

References:

Wilkinson, S (2011) ‘Analysing Focus Group Data’, in Silverman, D. (ed.) Qualitative Research 3rd Edition, London: Sage Publications, pp. 168–184.

Mason, J. (2002) Qualitative Researching 2nd Edition, London: Sage Publications.

UN General Assembly (2010) Guidelines for the Alternative Care of Children: Resolution adopted by the General Assembly, 24 February 2010, A/RES/64/142. Available at: www.unicef.org/protection/alternative_care_Guidelines-English.pdf [Accessed 19 July 2017]

B.2 Participant information sheet template

INVITATION FOR [YOU][YOUR CHILD] TO PARTICIPATE IN [AN INTERVIEW][A FOCUS GROUP]

Childonomics: Investing in Children and Families

Introduction

This is an invitation to you [to participate] [to give permission for your child to participate] in [a focus group] [an interview] in [Country] [City/Town]. It is scheduled for [day of the week, date (day, month, year)]. We are developing a methodology to determine the long-term social and economic return of investing in children and families. We are working in partnership with [e.g. Romanian social services] to pilot and test the methodology in your [Country] [City/Town].

Before you decide whether [you] [your child] will be part of this study, it’s important for you to understand why we are doing the research consultation and what is involved. Please read this information carefully. If you have questions about the research, feel free to ask.

Purpose

We are consulting with parents and children to understand more about how services support children and families. We want to know more about what impact different types of support have on children and parents. We are inviting [you][your child] to participate because we think it is important to include the voices of [children][parents][foster carers] and understand more about how services support [them and their families][you and your family].

This study is being run separately to the service provider and will not impact the access to services that you and your child have. This study is also anonymous, so [you][your child][and your child] will not be named or identifiable in any reports or publications resulting from the study. Other children and parents in [Country] [City/Town] are also taking part in consultation groups and interviews, and

http://www.eurochild.org/fileadmin/public/06_Projects/Methodology_Speak_Up_Final.pdf

https://www.dcya.gov.ie/documents/Publications/Ethics_Guidance.pdf

http://www.coe.int/en/web/children/child-participation-assessment-tool

http://www.eycb.coe.int/compasito/contents.html

http://www.eycb.coe.int/compass/

http://mindofmyown.org.uk/

https://www.unicef.org/protection/alternative_care_Guidelines-English.pdf



are being asked about the services that they and their family have accessed. We are also interviewing and consulting with service providers and other professionals, such as social workers, directors, and financial advisers.

Procedures

If you decide to [permit your child to participate][to participate], we will ask [them][you] to participate in [an interview][a focus group].

[Delete as appropriate]

Interview with children

They will be asked to speak with us in an interview. We will ask questions about how the services work, what impact they have had on them and their family, as well as what they think is important for their future. This will last about 20–30 minutes.

Focus group with children

Your child will be asked to join a group of children of similar ages. We will use creativity, games and group activities to talk about their life and their thoughts about the future, including what services children and families have or should have. This part will take about 90 minutes.

Parent/carer/young adult interview

You will be asked to speak to a researcher in a one-to-one interview. We will ask questions about how the services work, what impact they have had on you and your family, as well as your thoughts on the future and any other services you think are important. This will last about 60 minutes.

Parent/carer focus group

You will be asked to join a discussion with a group of parents. We will ask questions about how the services work, what impact they have had on you and your family, as well as your thoughts on the future and any other services you think are important. This will last about 60 minutes.

Benefits

There is no benefit to [you personally] [your child personally] for taking part in this study. However, we hope that the results of the research will impact how systems and services that support children, parents and families are designed and delivered in the future.

Confidentiality

[Delete as appropriate]

For participants in individual interviews:

We will keep [your] [your child’s] participation in an individual interview confidential. If we publish or present the results of this study, we will not use individual names or other personally identifiable information. We will not identify which comments are from which person. Only the interviewer and his or her immediate supervisor will have access to names or identifiable information.



To help protect confidentiality, we will store all documents in secure, password-protected computers. The facilitators will not share the comments [you][your child] make to anyone outside of the Childonomics core team.

For participants in group interviews or focus group discussions:

We will keep [your] [your child’s] participation in a group discussion as confidential as possible. If we publish or present the results of this study, we will not use individual names or other personally identifiable information. We will not identify which comments are from which person. Only the focus group facilitator and his or her immediate supervisor will have access to names or identifiable information.

To help protect confidentiality, we will store all documents in secure, password-protected computers. The facilitators will not share the comments [you][your child] make to anyone outside of the Childonomics core team. [We will ask you and all other participants in group interviews (focus groups) to observe confidentiality, but we cannot guarantee that other participants in the group discussion may not share information or identifiable information.]

Compensation/payment

There is no payment for taking part in this study.

Rights

Participation in research is completely voluntary. You have the right to decline [to] [to let your child] participate or to withdraw [them] at any point in this study without penalty or loss of services or benefits to which you [or they] are otherwise entitled.

Questions

You can ask questions about this study at any time, now or later. You can talk to [contact person who gave them this information sheet] if you want to know more about the research. Before the [interview][focus group] starts, you can also speak with the interviewers [add their names] and they will answer any remaining questions.

Contact:

[Insert contact person’s name and a direct contact number]



B.3 Adult participant consent form template

CONSENT FORM FOR [FOCUS GROUP/INTERVIEW] Title of research project: Childonomics: Investing in Children and Families Name of facilitators: [Add name(s) of the facilitators]

Please tick

box

I confirm that I have read the information sheet and I understand it.

I have had the opportunity to consider the information and ask questions, and I have had these questions answered satisfactorily.

I understand that my participation is voluntary and that I am free to withdraw at any time, without giving any reason.

I understand that personal details will be confidential and anonymised and that no personal information will be included in the report or other publications.

[For focus group participants] I understand that I have a responsibility to keep information shared by all participants in a group discussion confidential.

I understand that this research is separate to the service provider and I/we will not be treated differently for participating, or for deciding not to participate.

I agree to take part in this study.

Name of participant Date Signature _____________________ _______________ ___________________ Name of person recording Date Signature the consent _____________________ _______________ ___________________



B.4 Consent form for parents/guardians of youth or child participants

CONSENT FORM FOR PARENTS/GUARDIANS OF YOUTH OR CHILD PARTICIPANTS

Title of project: Childonomics – Investing in Children and Families Name of researcher: [Add name(s) of the facilitators]

Please tick

box

I confirm that I have read the information sheet and I understand it.

I have had the opportunity to consider the information and ask questions, and I have had these questions answered satisfactorily.

I understand that my child’s participation is voluntary and that I, as their parent/guardian, or they, as the participant, are free to withdraw at any time, without giving any reason.

I understand that personal details will be confidential and anonymised and that no personal information will be included in the report or other publications.

I understand that my child and I have a responsibility to keep information shared by all participants in a group discussion confidential.

I understand that this research is separate to the service provider and I/we will not be treated differently for participating, or for deciding not to participate.

I agree to the child named below taking part in this study.

Name of participant: _____________________ Name of parent/guardian Date Signature _____________________ _______________ ___________________ Name of person recording Date Signature the consent _____________________ _______________ ___________________



B.5 Questionnaire and case study guide

Questionnaire 1 – Service providers

GENERAL INFORMATION

1. Full official name of the organisation and date of registration / incorporation

2. Contacts:

Contact person

Postal address

Telephone

E-mail

Website

3. Type of organisation Please indicate the most appropriate version

☐ government ☐ municipality ☐ NGO non-profit ☐ private profit-making Other

__________

4. Usual sources of funding of the organisation

☐ government budget _____% ☐ private donations ______%

☐ local budget _____% ☐ payments for services by users ___%

☐ others (please indicate) ____%

___________________________________________________________________________

5. Overall budget of the organisation

in 2015 _________ in 2016 __________ in 2017 ______

INFORMATION ON SERVICES

6. Overall number of service users (clients)

2015 ____________________ 2016 _____________________ 2017 (planned)_____________

7. Staff capacity and workload (number of users per 1 staff member)

Maximum: in 2015 ____________ 2016 _________________

Average: in 2015 ____________ 2016 _________________

8. Number of users by type of services (as identified by the respondent)

2015 2016 2017

Number of users – one-off services

Number of users – short-term services

Number of users – long-term services

9. Type of services provided



For example: provision of information, day care for children, consulting services, legal assistance, intervention in crisis situations, psychological assistance, social assistance, training and group meetings, social integration and adaptation, in-kind goods, habilitation / rehabilitation, financial support, services at home (household), transportation services, full-time residential care, temporary residential care, foster carer training, foster care placement support, reintegration services Others _________________________________________________________________________

10. Place where the services are provided

☐ in the house of service user

☐ in the house of another physical person (foster carer or guardian)

☐ in the premises of our organisation

☐ in the premises of other organisation (state or private)

☐ in residential institution where the child lives

☐ in other places (please indicate) _______________________________________________

11. How do you find your clients?

☐ self-referrals

☐ referral from authorities

☐ through information provided to them by friends, relatives

☐ referrals from family doctors, social protection specialists

☐ via advertisement – flyers, leaflets, ads in local

☐ other (please describe) _______________________________________________________

INFORMATION ON SERVICE USERS 12. Service users groups

Please indicate all groups of population you provide services to – for example:

☐ Children in residential institutions ☐ Families with many children

☐ Children under guardianship ☐ Guardians’ families

☐ Families with children under three years old ☐ Teenage mothers

☐ Families with children over three years old ☐ Mothers with newborn

☐ Care leavers ☐ Children with disability

☐ Families with children with disabilities ☐ Children in conflict with the law

☐ Families where parents have disabilities ☐ Children with addiction problems

☐ Families in which one parent imprisoned ☐ Children – victims of violence

☐ Adults with disabilities ☐ Alone elderly adults

☐ Children in foster care ☐ Foster carers

Others (please define) _______________________________________________________

INFORMATION ON PERSONNEL

13. Number of staff members prescribed in your service specification 2015 _______________ 2016 _________________ 2017 __________________ 14. Actual number of staff members, including



Senior managers ______________ Administrative staff ______________ Social work specialists ______________ Psychologists ______________ Disability specialists (specify which kinds) ______________ Other specialists providing services (specify which kinds) ______________ Support (technical) staff ______________

15. Education of staff members Please provide information in the table below. Please add additional page if necessary

№ Position M / F Education (highest degree and field or discipline)

Number of staff

INFORMATION ON GOALS OUTCOMES

16. What are the goals of your service? 17. Describe the inputs and outputs. 18. What are the qualitative outcomes for children and families? What are the quantitative

outcomes? How are these outcomes achieved? What are the important factors for achieving successful outcomes (intensity of service provision? characteristics of child or family?)

19. What performance indicators do you use for staff?

20. What do you classify as a success? How do you measure success of the services?



21. To whom and how often do you report on your service and with which indicators?

22. Is your funding secure? Why? What helps to inform funding decisions? How do you justify the need for funding in each budget cycle? (to explore definitions of need and performance in meeting these needs / inputs/outputs)

23. Service budget – Actual expenditure for previous two years (2015/2016) and 2017

planned expenditure Total budget Staff budget Of which budget of staff working directly with children (carers, social workers) Of which budget of staff providing specialised direct services (psychologist, other… specify) Administrative staff Manager Staff training costs Administration Rent/building maintenance Electricity/communications/heating Transport Consumables (stationery, fuel, other…) Other costs

- Does level of salary of the staff correspond with other sectors (e.g. education or health)

o It is significantly lower o It is significantly higher o It is the same level

23. Technical capacity of the organisations, including assets

Placement in the city, accessibility, premises – are they enough for services? places for meetings with clients, one-to-one consultations, transport means and other necessary equipment

How much space is used for delivering services to clients? How much space is for administration and management/kitchen/etc?

24. Please provide copies of you service specifications, standards or procedure manuals.

Case file review

Five cases from each service under consideration will be reviewed and selected purposively by

reference to the following criteria (if possible):



Case is closed and the service user is now a young adult aged 18–25 years

The service user received the full set of inputs that the service is able to provide

There are some input/output data available

All information from the case will be anonymised and details changed as necessary to preserve

confidentiality.

The following questions will be posed, with regard to the case files:

1. What was the baseline situation in the family /for the child for the key indicators at the point

when the child first entered the service?

Poverty/household income

Parent care provision (level of meeting need)

Child’s school attendance level compared to a child of the same age in the target area (or for pre-

school age the child’s developmental indicators compared to child of the same age)

Risk of separation

Risk of poverty and exclusion

2. What were the inputs provided by the service?

Intensity of contact with service staff for child/parent (daily, weekly, monthly, length of each

contact, duration of input in total)

Other direct service inputs – material support, specialist consultations, leisure activities, parent

training

Inputs from other services – child care, cash benefits, transport, tutoring

3. What were the outputs and outcomes in the short term (e.g. during the initial period of service

provision), in the medium term (at the mid-point of service provision) and at the point of exit from

the service?

Poverty/household income

Parent care provision (level of meeting need)

Child’s school attendance level compared to a child of the same age in the target area (or for pre-

school age the child’s developmental indicators compared to child of the same age)

Risk of separation

Risk of poverty and exclusion



4. Are any longer-term outcome data available? Is any ongoing monitoring being conducted?

Cases can be recorded as a narrative, following an interview with the case worker presenting the

case or a review of the case file. Ecomaps and genograms can help to present the narrative.

B.6 Childonomics children’s focus groups

The following table sets out examples of what children or youth focus groups might look like, what

types of activities might be used to answer the research questions, as well as how to integrate the

process of briefing the child on the purpose of the research (which is necessary in order to obtain

children’s assent to participate). It should be read and understood in the context of all other

Childonomics documents and not in isolation20.

The general three questions that guide these focus groups are:

What help and support have you received from the service or programme that we are

examining? (i.e. how do they define or perceive the inputs of the service?)

What have been the outcomes (or outputs) that resulted from the support of the

service/programme?

Generally, what inputs/outcomes are important from your perspective in terms of supporting

families and children?

20 Please read the recommended reading in Annex B1 explaining the ethical considerations you would consider when contemplating carrying out focus groups or interviews with children and youth. It includes more information about how to prepare and facilitate child and youth groups and provides advice about preparation for all eventualities.



Table 29: Focus group discussion with 8–12-year-olds

Time Activity Details Extra guidance Resources

10 mins

Welcome

Welcome. My name is … and this is… [co-facilitator]. We are …, together we will…, over the next hour and a half we will… If you need the bathroom, you can… We will have a break at….

Introduce the ‘car park’.

Make sure you explain the activity, confidentiality, anonymity, where the information will be used etc. This will all affirm the process of making sure the child’s assent (willingness to participate) is informed. Remind participants that they also have a responsibility to keep information from the group activity confidential.

The ‘car park’ is a facilitator’s tool. It is a large piece of paper on the wall for all the quotes, feelings, comments and messages that the young people want to share or record (and also for the facilitators to make notes).

Name badges, flip chart paper, post-its, markers.

10 mins

Name games (getting to know you)

In a circle, throw a ball to another person and say your name as you throw it. After a few minutes, throw the ball and say the name of the person you’re throwing it to.

A small soft ball.

20 mins

Art

Take a big piece of paper and draw a line down the middle of it. On the left side, draw or write what your life looked like before the service/programme. On the other side of the page, draw or write about what your life looks like now.

What is better or worse? How do you feel? What does it look like? Who is helping you? What are the things that have helped? You can write words or draw pictures.

Prompts: Who did you live with before? (Draw them or write their names and the relationship – e.g. mother, father, sister, brother). Who do you live with now? (Draw them or write their names and the relationship – e.g. foster mother, foster father, others in residential care). Write down the names of the people you love or who you trust – e.g. social worker, carer, biological aunt etc.

Post-its, flip chart papers, pens and colours.

20 mins

Art

Lie down on a piece of paper and have a partner draw your outline! In your head, draw what you think about the service. Over your heart, draw what you feel about the service and over your hands, draw what the service does. Then we want you to decorate the picture of yourself (or on the other side of the paper) with what you can do or be in the future because this programme helped you and/or your family.

Flit-chart paper, pens and colours, art accessories (glitter, coloured paper, etc).



20 mins

Group

Let’s come back to the group. On a big piece of paper, we’re going to design the perfect family home or residential centre and the community that helps them. What services and programmes will support everyone in this house to be well and live a happy life?

Allow the children to add details themselves if they like. The facilitator(s) might add some of the verbal feedback from some of the children. Ask prompt questions to see what consensus there is with some of the contributions.

Prepared diagram with house in the middle and a ‘yellow brick road’ all around it.

10 mins

Debrief

Thank you for sharing. We hope you enjoyed the group. Are you happy with everything you shared? Any last thoughts, comments or messages for the adults on the ‘car park’ wall? Pass the ball and share any last thoughts about our group. We also have a small gift and a ‘thank you’ card. [Hand these out when the parents start to collect them.]

This might be a chance to remind the group what will happen next. What will you be doing with their contributions? Remind them that it’s anonymous, that while you might use their words or their drawings, it won’t be linked back to them.

Ball, car park poster, ‘thank you’ cards or participation certificate, small gifts.

Table 30: Focus group discussion with 12–18-year-olds

Time Activity Details Resources

10 mins

Welcome

Welcome. My name is … and this is… [co-facilitator]. We are …, together we will…, over the next hour and a half, we will… If you need the bathroom, you can… We will have a break at….

Introduce the ‘car park’

Make sure you explain the activity, confidentiality, anonymity, where the information will be used etc. This will all affirm the process of making sure the child’s assent (willingness to participate) is informed.

The ‘car park’ is a facilitator’s tool. It’s a large piece of paper on the wall for all the quotes, feelings, comments and messages that the young people want to share or record (and also for the facilitators to make notes).

Name badges, flip chart paper, post-its, markers.

10 mins

Name games

In a circle, throw a ball to another person and say your name as you throw it. After a few minutes, throw the ball and say the name of the person you’re throwing it to.

A small soft ball.

30 mins

Art

We are going to draw life charts. The purpose of the life chart is to look at the length of your life and what are the things and who are the people that have had an impact on you? When you draw the X axis with your age, mark a line when the service came into your life and include the impact (if any) that it has had since.

Encourage the children to be creative. If suitable, during the activity, walk around and ask them some questions about what they’ve drawn. What does that peak represent? Would you consider adding...? Can I help you in any way?

A4 papers, pens and colours, maybe glitter or coloured paper etc.



After 30 mins, we will come back together as a group and share our charts.

10 mins

Art

Let’s go back to your life chart. It goes up to your current age. Add another length of paper to the end of your current drawing and make a projection about your life. Draw two lines for two scenarios. One line would be if the service didn’t exist. The other line is what you expect because the service helped.

If anyone in the group would like to share something about their future plans (the second part of the activity), please do.

Take your lead from the energy of the group. They may be more or less respondent to this activity. It can be emotional if they’re taking it seriously. Try and support them constructively. Don’t make verbalising the activity compulsory as some may not be ready to share that level of detail. Ask the group if it’s still ok that they keep a copy of their life chart? If they don’t want to give you the original, you can take a photo of it and they can keep it.

A4 papers, pens and colours, maybe glitter or coloured paper etc.

20 mins

Group

Carousel. On prepared flip charts laid on the ground or on tables around the room in a circle, there are topics. Add any thoughts, comments, questions on the page. Once everyone has added to each page, everyone then grabs a pen or stickers and marks (by adding a sticker or a star) each point they find valuable or they agree with).

If possible, include an ‘anything you want to share’ table for anything extra they want to share.

Prepared flip charts with statements about services, inputs, outputs and outcomes. Also include one on ‘supporting families means…’ and ‘children’s rights means…’

10 mins

Debrief

Thank you for sharing. We hope you enjoyed the group. Are you happy with everything you shared? Any last thoughts, comments or messages for the adults on the ‘car park’ wall? Pass the ball and share any last thoughts about our group. We also have a small gift and a ‘thank you’ card. [Hand these out when the parents start to collect them.]

This might be a chance to remind the group what will happen next. What will you be doing with their contributions? Remind them that it’s anonymous, that while you might use their words or their drawings, it won’t be linked back to them.

Ball, car park poster, ‘thank you’ cards or participation certificate, small gifts.

Please note the guidance provided in Annex B1 on preparing for distress or disclosure during interviews or group consultations.



B.7 Childonomics: Pathways

Semi-structured interviews for teenagers, young adults and care leavers21

This question set forms the framework for semi-structured interviews conducted with teenagers

who are in care and care leavers as part of Childonomics. It may also be adapted to be used for

any teenagers or young adults who have, or have had, experiences of service interventions. The

purpose is to ask teenagers and young adults about how they the view long-term social and

economic return of investing in children and families. To do this, it is anchored in their experience

and reflections of the child protection and/or social services system.

Some participants may need the question to be reworded to ensure that they understand what

information is being requested. There are some options added here (a., b., c., etc) to support their

understanding, if needed. If there are indicators that are generally used to measure outputs or

outcomes, you may also consider adding a question about these22.

1. Some initial information about the young person and their family

a. Can you tell me a little about your birth family?

b. How many brothers and sisters (and half-brothers and half-sisters) do you have?

What ages are they? Where do they live now? (i.e. are they also in care? Are they

living with their birth parents?)

c. Did you ever live with your parents (both or one?)

d. Were you ever cared for by other people before you entered care formally (e.g. aunts,

uncles, older siblings, family friends)?

e. What age are you?

2. What help and support have you (and your family) received from the service or programme

that we are examining ?(i.e. how do you define or perceive the inputs of the service?)

a. I want to ask you about the different decisions that were made about you as a child.

What support did your family receive before you came into care?

b. When did you come into care? Do you know why you came into care?

c. How many placements did you have and how would you describe them? If you moved

between care placements, do you know the reason? Who was included in the

decision to come into care or to change placements while in care?

3. What have been the outcomes (or outputs) that resulted from the support of the

service/programme?

a. Do/Did you have a key worker (social worker)? How often do/did you meet them?

What do/did they do for you? How many social workers have you had? Do they

change often?

b. Did you ever have therapy or extra support? If so, what was it for and how long did it

last? Why did it end or what did it address? What about other support?

21 Care leaver is a term that usually refers to any person of majority (has either turned 18 and/or has transitioned out of care). 22 An example of this included in this template is the educational outcomes for children in care and care leavers as an indicator of educational success and current or future economic independence. The questions that might be added in order to contextualise wider indicators could be gathered during Step 0 and Step 1 of the Childonomics methodology.



c. Do/Did you have a plan for leaving care? Did you have a care plan? Who was included

in the planning? Did it all work out as planned?

d. What did/will you do once you became/become 18? Where did/will you live or how

did/will you become independent?

4. Can we talk a little about your formal education?

a. Have you finished secondary school (or an equivalent)?

b. Do you want to go to college? If you do, what do you want to study? If you don’t, is

there a specific reason why?

c. If you started college, what did you study? Did you finish it? Did you get any extra

support to stay in college?

d. What are your plans for the future?

5. Generally, what inputs/outcomes are important from your perspective in terms of supporting

families and children?

a. Was you experience of the system a positive one? Can you give me an example?

b. How would you measure what good a service was? Can you give me an example?

What would you want to know about a care setting to decide if it’s a good one?

c. How should success be measured?

d. If there was something you could change about the care system (either positive or

negative), what would it be and why?

Note-taking

It is recommended to take notes during the meeting if possible23. This may include some visual

aids to understand complex relationships or family trees. The priority is to take down direct

quotes and as much detail as possible.

For each placement change or intervention (‘move’) that the person has made, please gather

as much detail as possible, noting that they may not have the formal information24:

What age were they25?

What was the change (e.g. from birth family into a foster family)?

Who was involved in this decision? (Were they consulted? Where their birth parents

consulted? Was there a social worker or judge involved?)

What was the reason for the change/move (if known)?

What was the output (immediate or short-term change that occurred)?

What was the outcome(s) (the more longer-term impact of the intervention(s))?

Be familiar with the data matrix you will be filling in after the interview so that you know what

level of detail is needed for each topic.

23 You should ask the interviewee if they are comfortable with you taking some written notes. It may also be possible to audio-record the interview. This decision must be balanced between the intrusion of recording the person’s voice and the ethical treatment of how their data (including audio records and written records) are treated. Please refer to Annex 2: Childonomics Ethical Considerations of the Country Report. 24 Make a note if they do not have the information, as this is a finding in and of itself. 25 They may not remember or know at what age it happened. A prompt question might be: Were you a baby? Do you remember if you had started school? Do you remember what grade/class you were in around then?



B.8 Childonomics: foster carer focus group questions and topics

1. Introduce the project (same as child introduction – anonymity, informed consent etc)

a. The participant may have received the leaflet but it is important to verbally remind the

group what the research is, what partners are involved and how their information may

be used.

b. Before starting the consultation, make sure to explain the consequences of anonymity

and confidentiality in regard to other participants. A useful way to do this would be to

create a group agreement that includes that the content of the discussion will not be

discussed outside of the group.

c. Informed consent means that while they agree to participate, they can still decide what

questions to answer and to what degree of detail.

d. Ask participants to sign the participant consent form.

e. It may be useful to have a flipchart or a projector so that the participants can see how

you are recording some of the content.

f. It may be useful to summarise, at the end of the consultation, some of the salient points

that were discussed. You may also want to, if suitable, collect the contact information of

all or some participants if you think you might need to clarify some points when you

start writing up the report. This is also a way of feeding back to them and possibly

holding a follow-up/validation session with the participants.

2. Focus group discussion

What help and support have you received from the service or programme – i.e. how do you define

or perceive the inputs of the service?

What have been the outcomes (or outputs) that resulted from the support of the

service/programme? For children/for yourselves? How does the service affect your own children?

Generally, what inputs/outcomes of foster care are important from your perspective in terms of

supporting families and children and community?

What are signs of good quality in foster care? (for children, for you as carers)

What was the procedure for getting approved? Were you trained before or after approval as a

foster carer? Both? Were there any costs involved in the training and procedure?

What types of foster care do you provide? (e.g. long-term / short-term / emergency / therapeutic /

specialised)

Do you have a supervising social worker or other professional contact person? Where from (which

organisation)? How often do you see him or her? Do you have to travel to the appointments or do

they take place in your home?

Do you receive any other support in caring for your foster child/ren?



Foster care allowance? How much/how often?

Services? Free medical care? Help with education costs? Transport? Preparation for independent

living? Addressing trauma? Counselling? 24-hour call service? Disability?

Costs of having a child in your home? Is there an instrument that is used to calculate the additional

costs?

Anything else you want to share?



Questionnaire for participants

1. Age 2. Gender 3. Do you have your own children? How many? Age/gender? Do they live at home now? If yes, what do they do? (e.g. study/work) _______________________________________ 4. How long have you been fostering? 5. What is most rewarding about fostering? 6. What is most challenging? 7. Please indicate in the table how many children you have fostered in the past, the age when they came to you and left you, their gender and other characteristics



Table of foster care children placed in your care

M/F

Age of child when entered placement

Reason for placement

Type of placement

Special needs? (Please describe if so)

Permanent care order or voluntary care order?

Where referred from (from birth family, hospital, residential care, other foster family, family, relatives?....)

Age of child when ended placement (current age)

Why did the placement end?

If the child has left your care, where did he/she go? If he or she is still in your care, what is the plan for his/her future?



B.9 Semi-structured interviews and analysis template

Pathways semi-structured interview

Purpose: To highlight decisions made by service users and service providers that shape pathways

through services

Potential interviewees: Care leavers, older teenagers who have been in care, parents of children

receiving services, parents as service users, young adults who are/were service users as children.

Ways of using the template:

Ways of using it: Factors to consider:

Print out a sheet with the four headings and fill it in during the interview.

Writing while talking to someone might take more time, but you can double-check with them as you’re filling it in.

Have the questions prepared, audio-record the interview. When listening back afterwards, can fill in the sheet.

Audio-recording and transcribing takes more time and you need to make sure you gather all the information to fill in the paper later. Depending on the service user, they might find the audio recorder more or less intrusive than writing quotes or answers down during the interview.

Using it as a memory tool immediately after the interview.

Sometimes it’s not possible to write or audio-record someone’s interview. As soon as possible after the interview, and while the facts are still fresh in your head, fill in the template with all the information.

Before the interview starts and after it ends, remember:

At the start of the interview, always introduce yourself and explain what the project is about. The

interviewee may or may not have read the information leaflet. Best practice is to repeat it verbally,

as well as to provide a written version.

Explain what the project is trying to do, where this information will be used and what types of

questions you will ask them. Be clear about whether they will be quoted directly and how their

quotes may be used. Explain clearly what the purpose of the research is: for example, an internal

investigation, advocacy or evaluation. If the research is connected to an internal investigation or

evaluation of a specific service, be clear about who may read it (e.g. service managers, staff,

external reviewers). If the research is connected to advocacy and the information provided by the

interviewee may be shared more widely than in research reports, specify where and in what way

the information may be used (e.g. on television, radio, social media, advertising campaigns).

Explain what anonymity means and how confidentiality works.

Anonymity is not only that your name will not be used, it also means that identifying information will be changed. Confidentiality means that no one outside of the interviewer and their line manager will know the name or identifying information of the person being interviewed.

Explain that informed consent means that they can stop the interview at any time and they can

decide at any time that they no longer want to be involved in the research.



Ask them if they have any questions at the moment. Tell them they can ask you any questions as

you go along and that they can take breaks if they need it. Now you can ask them if they have

understood everything. If they’re happy to proceed, ask them to sign the consent form. Make sure

two copies are signed and give one back to the person for their own records.

At the end of the interview, thank them for talking with you. Ask them if they have any questions

for you at this point. Are they happy with everything that was discussed or would they like to add

any more comments? (If you brought a gift for them, present it now). Make sure they have a

contact number for you in case they need to talk with you in the coming days. If you have a leaflet

with contact information about support they may use if they feel distressed, you can present it now.

You can also remind them that they have a social worker (or other social care worker or local

contact) whom they can also talk to.

Type of information recorded Possible analysis

The various age(s) or dates that represent an initiation of services, a change of services, or an ending of services

Pathways between or through various levels of service intensity.

Any capacity change (increase or decrease) of service provision – e.g. staff turnover, budget changes, recipient criteria change.

Description of the intervention How intense an intervention was it?

Who was involved in each decision (and who was not involved)?

Who was involved or excluded in the choice to initiate, change or end a service?

Was the service user included in the decision?

Was the child included in the decision?

The reason(s) and outcomes for the initiation/change/ending, if known

What was the reason for the starting, change or ending of the service?

Did it end because the need was met?

Was the initiation/change/ending planned? Was it as a reaction to an event or trauma?

Was the reason valid enough for a change/end?

An example:

Amanda, care leaver, aged 20

Interviewed in May 2017 in Waterford, Ireland

Age(s) or dates

Intervention Who was involved in each decision?

The reason(s) and outcomes for the initiation/change/ending, if known, as well as the outputs, consequences or outcomes

As a baby Taken into care and placed in an institution

The social workers, she doesn’t know if her parents decided or not

Her father had left, her mother was not coping and there was not enough food in the house

Age two and a half

Changed institutions Social workers, not her mom

The baby institution was full so she was moved to an institution for three-year-olds and older

Aged 12 Moved into foster care Social workers She wanted to live with her best friend’s mom, the best



friend’s mom agreed. Amanda was the one who initiated the change

Age 16 Moved home with her mother

At the time nobody, but later the social worker visited

She had a big fight with her foster mother. She didn’t know where to go. She didn’t know her mom very well, but she had nowhere else to go and her mom said she could come back

Age 16 Domestic violence women’s refuge

Social workers and her mom

Her mom didn’t know how to cope with a teenager and asked her to leave. It was the weekend, so the social workers took her to the domestic violence women’s refuge

Age 16 Moved into an institution Social workers After the weekend, the social workers found a place for her in a girl’s institution

Age 17 Moved to a semi-independent apartment

Nuns

Because she had finished school and was working part-time, the nuns suggested she move into an apartment that they have for girls who are preparing to become independent. The apartment is on the same property as the institution but there are fewer staff and the girls can cook and clean

Age 19 (now)

Still living in the apartment

She’s much happier now. She is happy working part-time and is thinking about studying hairdressing. She has a boyfriend that she met through work, but he’s not allowed visit the apartment. She would like to live in her own apartment before she’s 21 and is saving for that



Template

Name:

Age:

Service:

Date and time of interview:

Age(s) or dates

Intervention Who was involved in each decision?

The reason(s) for the initiation/change/ending, if known, as well as the outputs, consequences or outcomes



Analysis

There are two planned ways that this information can be analysed: one is inductive and one is

deductive.

Deductive is a more structured approach where you start with the research question and ‘test’ the

interview to see what data come out of it. The structured template is conducive to this, especially if

there are many services delivered over time, and to map pathways through services. It can be

used to offer broader, macro understandings of changes (service starting, service change or

service ending), as well as trends in reasons why service provision changes and who initiates it. A

disadvantage is that it limits the analysis to pre-selected factors. That said, you can update the

template after a piloting it, to one that is more clearly able to answer your research question (Step

0 and Step 1 of Childonomics).

If you approach the interview with an inductive analysis, then you would start with the interview and

as you go through it, take note of certain themes that are linked to the research question. Once you

have a first draft of the themes, you would then go to the next interview and use those themes to

review the next interview. You might add more themes as more patterns emerge. If you do, you

would go back to the first interview(s) and review them for the new themes also. This approach is

also conducive to analysing data with a comparison variable across services in mind. It might offer

more in-depth context since it is not limited to a pre-imagined group of variables the way a

deductive approach often is. An example of this approach that we found useful when piloting

Childonomics was that that separation of siblings in care emerged as a theme – we did not know

we would find this as such a large variable in Malta but it emerged inductively.



Annex C Sample Case Study

Inputs Outputs Outcomes

Previous background

17-year-old young person had been in contact with CPS and IFSS since early childhood

Kinship care, CPS social worker assessments, IFSS social worker contact

Homeless shelter at 17 years old

Baseline at moment of referral

Crisis in relationship with grandmother and had just become homeless

Homeless shelter for three months; IFSS social worker weekly contact and support in seeking employment

Using drugs and asked to leave homeless shelter

Situation at periodic review

Stays with friends from homeless shelter

IFSS social worker weekly contact; referral to psychologist and employment service

Does not attend employment service or psychologist appointments

Starts relationship and moves in with partner and his family, becomes pregnant and has baby

IFSS social worker weekly contact with relationship and family planning advice; informal support from family of partner; IFSS provided some supplies for the baby

Starts to attend appointments; refuses family planning services

Conflict with partner’s family and has to move out

IFSS social worker weekly contact; homeless shelter with baby

Back at homeless shelter with baby

Reunited with partner who looks for a place to live

IFSS social worker weekly contact; informal support from partner’s family

Moves in with partner; reconciled with own mother who helps with the baby

Motivated to attend appointments for the baby and to work on her issues

Living in apartment and looking after baby, partner working, some relationship problems

IFSS social worker fortnightly contact; Well Baby Clinic, home visits from midwives, child development clinic

Started to work part-time

Baby assessed at six months, development is good, reaching milestones, well-stimulated, no concerns

Ongoing relationship problems, episodes of violence

Home-based therapy (multi-disciplinary team) weekly; Home Start volunteer; employment service referral

Wants to train in skills for self-employment

Case still open: current plan is to wait until their situation is more stable



Figure 5: Ecomap summarising change in situation from baseline to current

Documents

OPM Report Template version 5...December 2017 Childonomics Malta and Romania: lessons learned from applying the Childonomics conceptual framework and methodology in practice Joanna