EXPERIMENTATION IN SUPPORT OF INNOVATION · What is experimentation? Different meanings to different individuals Social Experiments or Randomized Control Trials (RCTs) Trying something

EXPERIMENTATION IN SUPPORT OF INNOVATION

Using experiments to accelerate innovation and help governments design more efficient programs and policies

Partners in Innovation Symposium, BC

May 31st, 2017

2

Our Mission:

• to help policy-makers and practitioners identify and implement policies and programs that improve the well-being of all Canadians, with a special concern for the effects on the disadvantaged;

• to raise the standards of evidence used in assessing government policies and programs.

3

• In a government context, innovation means applying new insights or approaches that can be demonstrated to improve outcomes for the public compared to conventional ways of doing things.

But how do we know that an innovation constitutes an improvement?

• Demonstrating the effectiveness of an innovation requires using rigorous evaluation and structured experimental methods to generate evidence of impact.

Innovation in a government context

What is experimentation?Different meanings to different individuals

Social Experiments or Randomized Control Trials (RCTs)

Trying something new and put in place the systems to learn

Trying something new

• The most rigorous and effective way to measure the difference in outcomes resulting from an intervention/program

• Rigorous formal research and evaluation design

• No rigorous learning or evaluation strategy

• A “pilot”

Large amounts of public funds are invested in programs to serve and protect Canadians but still limited evidence on their effectiveness

Experimental approach is a smarter, cheaper and a more effective approach to develop government programs and policies

Using experimentation to learn What Works better

Typical approach

• Introduce new government programs without prior small-scale testing

Experimental approach

• Set up pilot projects to experiment with new interventions, evaluate them using rigorous methods, and scale up those that work

“You should work with your colleagues to ensure that they are devoting a fixed percentage of program funds to experimenting with new approaches to existing problems and measuring the impact of their programs. I expect you to instil a strengthened culture of measurement, evaluation and innovation and program and policy design and delivery.”

Mandate letter to the President of the Treasury Board of Canada

Research and Innovation (R&I) is administered by the Province of British Columbia to provide funding to community organizations and partners to explore, test and/or find new and innovative ways of delivering programming to help individuals keep, find or return to work. This can be done through research or the development and implementation of innovative pilot projects. The results gathered from R&I projects will identify better ways of helping unemployed British Columbians prepare for, return to, or keep employment and to become productive participants in the labour force.

Research and Innovation Applicant Guide Employment and Labour Market Services

Canadian government’s commitment towards innovation and experimentation

7

Hierarchy of Evidence

Tier Type Design FeatureEvidenceQuality

Top Tier Systematic Reviews and Meta-Analysis

Synthesize quality evidence on a specific issue

Strongest

Upper Tier Individual Studies withrandomization/credible source of exogenous variation.

Randomized experiments or natural experiments that use high-qualityexogenous variation to generate a comparison group

Very strong (if done well)

Middle Tier Limited or no source of exogenous variation, butwith credible comparison group/counterfactual

Some control in the assignment of treatment, or correlational studies including studies relying on selection of observables and case studies with a comparison group

Very strong to moderate depending on specific designfactors

Lower Tier Studies without measured comparison groups/counterfactuals

Studies without a comparison group, participant satisfaction surveys, expert opinions, exploratory case studies

Evidence should be considered suggestive and care taken to interpretthe findings accordingly.

Experimental framework

CompareImplementRandomizeDesign

Participants

Treatment group

Receive intervention

Outcome

Control group

Don’t receive intervention

Outcome

Participants can be individuals, but also

firms, public organizations,

villages, regions, etc.



Participants

Treatment group


Outcome

Control group


Outcome




Different alternatives to run the random assignment lottery (e.g., individual vs.

group level randomization)



Participants

Treatment group


Outcome

Control group


Outcome






Both groups end up being similar in any

observable and unobservable

characteristics, except for the participation

in the program



Participants

Treatment group


Outcome

Control group


Outcome









in the program

Collect data using surveys and/or

administrative data sources and estimate

impact of the intervention



Participants

Treatment group


Outcome

Control group


Outcome









in the program

Collect data using surveys and/or

administrative data sources and estimate

impact of the intervention

• The measure impact of the intervention can be attributed solely to the intervention

13

Results of Social Experiments are easy to grasp

The Self-Sufficiency Project

• Testing an innovative “making work pay” strategy that used temporary earnings supplements to help long-term welfare recipients achieve self-sufficiency through employment.

• A 10-year study involving more than 9,000 single parents in New Brunswick and British Columbia.

• Results have found employment impacts that are among the largest ever seen in a welfare-to-work program; little net cost to government, and an effective approach to raising the incomes of poor families and reducing poverty.

On the importance of having a proper counterfactual

SSP impact on full-time employmentby Months From Random Assignment

0

5

10

15

20

25

30

35

40

-12 -7 -2 4 9 14 19 24 29 34 39 44 49Pe

rce

nta

ge

Em

plo

ye

d F

ull

T

ime

Months From Random Assignment

Program Group

15

SSP impact on full-time employmentby Months From Random Assignment

0

5

10

15

20

25

30

35

40

-12 -7 -2 4 9 14 19 24 29 34 39 44 49Pe

rce

nta

ge

Em

plo

ye

d F

ull

T

ime

Months From Random Assignment

Program Group

Control Group

16

17

Why conduct experiments?

Putting new ideas to test

• Getting a sense of take-up rates

• Identifying barriers

• Measuring intended and unintended outcomes

• Measuring the differences government programs really make

• Providing measures of cost effectiveness

• Providing cost-benefit or ROI analysis

Supporting innovation

• Instill novelty and innovation into rigid systems

Promoting collaboration

• Provinces have been collaborating with the federal government to run experiments in their areas of jurisdiction

Some misconceptions about social experiments

Unethical

• Interventions should always provide a benefit and not harm recipients. Control group is never worse off compared to the status quo

• Using resources in programs that don’t work deprive other more effective programs from funding

• Several ways to get around ‘’unfair’’ perceptions: e.g. provide delayed treatment to the control group

Expensive

• It is often the program itself, not the evaluation component, that is expensive. Program expenses do not constitute new expenditures.

• RCTs do not always required large sample

• Analysis can be quite cheap, especially if administrative data can be made available

Findings not applicable to other settings

(Internal vs. external validity)

• Context matters, as is any other type of evaluation, but some lessons can be generalized. Researchers and policy makers still need to apply judgement.

Don’t tell you why there is an effect

• Social experimentation is not only about measuring impact. Monitoring the process is also important to understand how the outcomes are being generated.

Focus on a single research method

• Can be combined with qualitative methods – mixed methods provide for the most informative approach

The criticism A potential response

19

Conditions needed for experimentation in government

Political commitment

Internal champions

Know-how

Financial tools and authorities

Dedicated fund

20

Projects can be assessed with respect to

• Quality of the idea:

• Policy relevance

• Innovative

• Ethical

• Scalability

• Quality of the method:

• Impact measurement

• Cost effectiveness

• Cost-benefit analysis

• Implementation process:

• Response from participant

• Practitioners' experiences

What to look for today

Documents

EXPERIMENTATION IN SUPPORT OF INNOVATION · What is experimentation? Different meanings to different individuals Social Experiments or Randomized Control Trials (RCTs) Trying something