The NDIS Evaluation Two Years On

Kostas Mavromaras

(Chief Investigator of the NDIS trial Evaluation)

NGO Forum

Council of Social Service of NSW (NCOSS) Sydney 26 August 2015

The evaluation is funded by the Australian Government Department of Social Services (DSS) and is being carried out by an independent research team, led by the National Institute of Labour Studies (NILS) at Flinders University.

Background and acknowledgements

Flinders University was commissioned in 2013 to carry out an independent evaluation of the impact of the NDIS in the trial sites (excluding WA)

The evaluation started in May 2013 and will be completed in June 2017

The work is funded exclusively by the Australian Government Department of Social Services

The Team

• Led by the National Institute of Labour Studies (NILS)

and supported by a wide range of expertise:

• Five national State Experts in disability (including Sydney

and LaTrobe Universities)

• Three International Experts in disability, disability insurance,

and evaluation (Michigan, Maryland and Nottingham

Universities)

• Two Survey Companies: the Social Research Centre and

Iview/Ipsos

• Flinders University (Adelaide): Disability and Community

Inclusion Unit (DCIU); and Flinders Northern Territory

A Roadmap of the talk

• A description of the NDIS Evaluation and its

objectives

• The components of the NDIS Evaluation

• Putting the evidence together

• Lessons and highlights

Two main purposes of the talk

• We do not wish to influence the policy (this

would contaminate our results/conclusions)

• But Wave 2 starts soon and we do want to

make the Evaluation known to get the widest

possible representation of all voices

• We also wish to be continually informed

about the relevant thinking

• So, your ideas and comments please! (But we

will not be responding with opinions or results.)

Some of the Evaluation questions

• Is the NDIS bringing change?

• Where is it doing it?

• What is working well and what is not?

• Lessons for the broader scheme?

• Several different aspects…

• Several different viewpoints…

• 6 trial sites: VIC, NSW, TAS, SA, ACT, NT

Some of the Evaluation questions (cont.)

• Not measuring costs and funding

• Not about cost-shifting or similar issues

• Not about the NDIS financial viability or feasibility

• We want to know what all concerned (including people with disability, their families and carers, the providers of supports and their workers) think about the introduction of the NDIS

• Is it making their lives better? Is it closing the gap?

Ask what is happening (Because of the NDIS):

• Is the wellbeing and participation of people with disability and their families and carers changing?

• Is the demand for provisions changing?

• Will there be the providers to deliver?

• Will providers have the workforces to deliver?

• Is the NDIS serving the least vocal people with disability?

• Will early interventions become more prevalent?

The methodology

• We collect evidence from NDIS participants in all trial sites (the “trial” group)

• We collect evidence from NDIS potential participants not in the trial sites (the “comparison” group)

• We collect evidence twice (2014/15 & 2015/16) from the same participants (approx. one year apart) and measure how things have changed

• For similar people in the trial and comparison groups, the difference in the 2014/15-to-2015/16 change experienced by the two groups can be attributed to the NDIS

• This is called the Difference-in-Difference method

The methodology

• The D-in-D method measures the change experienced by two groups between a starting point in time and a later point in time

• It measures this change for two as similar as possible groups, but only the one group is exposed to the NDIS and the other is not

The methodology (employment as an example)

• There are many reasons why employment may change for people with disability, including what is introduced by the NDIS

• Suppose we find higher wellbeing and lower employment participation between wave 1 and wave 2 data collections

• The start of the D-in-D method measures the change experienced by two groups between a starting point in time and a later point in time

• It measures this change for two as similar as possible groups, one group having joined the NDIS and the other has not

The methodology (cont.)

• Multidisciplinary

• The role of matching

• The delights of phasing-in

• The advantages of the D-in-D method

• The role of multivariate analysis

• The delights of messy data

How do we handle this complexity?

• By collecting lots of complementary evidence big in scale and wide in scope and using a rigorous evaluation method

• Broad representative evidence from new large national surveys

• Deep and granular evidence from new in-depth qualitative studies

• Support from several national administrative data sets

• Wide expertise in disability studies, sociology, social work, evaluation, survey design, economics and statistics

• The combined evidence is analysed and evaluated by a team of national and international experts to produce a number of reports for the DSS

The main components of the NDIS Evaluation (2013-2017)

• Stakeholder Engagement Strategy

• Ethics, accessibility, privacy, age-appropriate design

• Survey of people with disability, their families and carers

(2014/15 and 2015/16)

• Survey of disability support providers, their workforce

and self employed (2014/15 and 2015/16 for all trial

sites, 2014/15 only for QLD)

• Comprehensive set of qualitative studies (2014 to 2016)

What do we do with this evidence?

• Merging of qualitative and quantitative evidence

• Statistical and econometric analysis of the evidence

• Evaluation of the impact of the NDIS on people with disability, their Families and Carers

• Assessment of the impact of the NDIS on the relevant providers and their workforces

• Synthesis and reporting

Engaging stakeholders: This is a

continuous process aimed to keep on

informing all other components

throughout the Evaluation

Stakeholder Engagement Strategy (SES)

To encourage the active participation of:

People with disability, their families & carers

Disability sector, advocacy & peak

organisations

To ensure primarily that stakeholders’ views

inform the research, but also that stakeholders

are informed about the research

State Experts are working in the trial sites/states

Information dissemination through various

methods including www.ndisevaluation.net.au

Asking the question:

What do people with disability, their

families and carers think about the NDIS?

The NDIS Survey of People with Disability, their Families and Carers

• Two waves to be conducted in trial & comparison sites

• Pilot Survey completed in May 2014

• Extensive research and testing

• Registration is continuing as we speak (August 2015)

• Wave 1 is presently in the field (ACT trial and evaluation

start later)

• Wave 2 planned for early 2016 fieldwork (approx. 10-12

months after Wave 1)

NDIS Survey of People with Disability

• Much new thinking and design has gone into:

• How we get as many people with disability as

possible to respond

• Providing choice on how to complete the survey

(online, CATI, hard copy, face to face)

• Making the questionnaires as accessible as possible

• Making sure that the opinions will be those of the

people with disability (see next slide in Part B)

NDIS Survey of People with Disability (cont.)

• Part A: Elicits mainly factual information (Proxy or person with disability)

• Part B: Elicits mainly opinions, perceptions, aspirations and assessments (only person with disability) Pictorial Easy English option where requested

Part A Part B

Disability & Supports Social Participation &

Well-Being

Supports before & since

the NDIS

Education & Employment

Aspirations

Socio-demographics Living & Housing

Needs/Aspirations

Keeping in Touch Opinions on Disability Supports

NDIS Survey of Families and Carers of

People with Disability

• Information collected includes:

• Caring – the type/access/frequency of supports

• Experiences pre- and post-NDIS

• Social participation & wellbeing

• Socio-demographics, including family circumstances

• Caring for children less than 8 years old

These are major data collections: where necessary,

interviews last over 40 minutes and questionnaires

are between 20 and 35 pages long…

NDIS Survey: Highlights/Innovation

• Independent pre-survey registration process provides choice of different completion modes to suit diversity and enhance inclusion

• We offer:

• Hard copy

• CATI

• Online

• Face to face

• All modes are well received (except face to face, which was rarely chosen as the preferred mode)

NDIS Survey: Highlights/Innovation

• Lower reliance on proxies through the use of Pictorial Easy English questionnaires for “Part B” (participation, wellbeing, aspirations, needs)

• Extensive use of experts in accessibility and inclusion

The objective is to allow the voice of the person with disability to be heard as clearly as possible (and as distinct as possible from their proxy’s voice)

Pictorial Part B has been well received

NDIS Surveys of People with Disability, and

their Families and Carers

• The survey of people with disability is well advanced

• The survey of families and carers of the people with

disability is not as advanced, but well under way

Initial response rates may me idiosyncratic: lower than

expected for “units”, but much higher for “items”. Can it

be that the survey does not look as worthwhile from the

outside, but, once started, the experience is good?

Asking the question:

Will we have the providers and the

workforces for the full NDIS roll out?

Survey of Disability Support Providers

and their Workforce

• Workplaces, workers, self-employed providers in the

NDIS trial states (2 waves). We collect information on:

• Staffing, skills & training, recruitment

• Revenue & sources of revenue

• Supports offered (incl. technological innovation)

• Client profile

• Quality assurance and performance monitoring

• Experience of NDIS & future plans

What about the depth of information that

cannot be captured by large scale

quantitative surveys?

Why Qualitative Evidence?

“BIG” quantitative surveys do not handle aspects of

diversity and accessibility with sufficient granularity

Qualitative studies offer more depth and flexibility:

they offer more depth and granularity

they can ask questions that surveys cannot

can facilitate otherwise difficult interpretations

Qualitative is asked in two-three waves (2014/15/16)

Qualitative evidence is built to be combined with our

survey evidence (mixed methods analysis)

What Qualitative Evidence?

People with disability, their families and carers

One-to-One interviews

Focus groups with eligible non-participants and non-

eligible non-participants

Face-to-Face, telephone, online, email

• Sector representatives

• Disability workforce agencies, government

agencies, NDIA, unions, peak bodies

• Specialist disability service providers

(2nd wave beginning now!)

Several important and unique aspects of

the NDIS Evaluation are covered with

additional studies.

The “special” components of the Evaluation

• Northern Territory – a separate evaluation (two waves in 2015 and 2016)

• Older people with disability: aged care/NDIS (one wave 2015/16 only)

• Mainstream providers (one wave 2015/16)

• Administrative data analysis (continuous 2013-2016)

The Northern Territory

• Small sample, remote location study, requires an

individual and flexible evaluation design

• A two-wave, large scale, qualitative study in the

Barkly region (trial) and Alice Springs (comparison)

• NDIS participants and their families and carers

• Providers of supports and peak bodies

• NDIA staff

• Culturally respectful and sensitive approach,

supported by Flinders University NT.

• Started preparations in June 2014, fieldwork in 2015

and will finish in 2016

The Older People with Disability Study

• We will compare supports and satisfaction of older

NDIS participants with older people with similar

disability but not eligible for the NDIS

• People with disability between the ages of 63 and 70

• Comparing experiences of NDIS participants aged

between 59-64 with non-participants aged between

64-70 not in trial sites and some in residential aged

care facilities

• Small samples require an in-depth qualitative study

• One Wave of interviews in 2015

The Mainstream Providers Study

• Services already provided to the general public,

which also need to be available to People with

Disability, separate to the services provided by

the NDIS.

• Content development in progress

• Expected commencement in 2015

How is the whole Evaluation project

travelling?

• Started in June 2013

• Design second half of 2013

• Piloting first half of 2014 with further design

• Wave 1 fieldwork started very end of 2014 and

continues

• Wave 2 design and start second half of 2015

• Wave 2 second half of 2015 and 2016

• Analysis and final reporting second half of 2016 and

first half of 2017