Regression Discontinuity Design

Regression Discontinuity Design

Thanks to Sandi Cleveland and Marc Shure (class of 2011) for some of these slides

RD DesignsA pretest-posttest, program-comparison

group strategy Review:

Advantages of Pre-tests?Detect differences between groupsDetect potential vulnerability to internal validity

threatsHelps with statistical analysis

Advantages of Comparison groups?Helps control sources of errorHelps support the counterfactual inference

Underuse of RD? Why?It’s new.Key criteria must be met for use.Perhaps it’s just misunderstood.

Overview of RD

OA C X O2

OA C O2

“pre” is ANY continuous variable that correlates with the outcome of interest

Assignment based on cutoff scoreRegression line should have vertical displacement at

the cutoff score if there is an effect

No Treatment Effect

Positive Effect

Examples1. Campell & Stanley’s Ivy League Education

Example2. Trochim’s Hospital Administration ExampleHospital Quality of Care

More about assignmentAssignment variables:

Must be continuous (or ordinal) Can be a pretest on a dependent

variable Can be by order of entry into study Cannot be caused by treatment May or may not be related to the

outcome

If implementing an RD design in your area of research, what variables would you choose for assignment?

Choosing the Cutoff ScoreNow referring to the assignment variable(s) you

identified, how would you arrive at a cutoff score?

Substantive grounds: professional judgmentNeed or MeritClinical diagnosis

Practical grounds:Available data setsAvailable resources

Choosing the Cutoff ScoreMean of the distribution of assignment scoresPolitically defined thresh-holdsComposite scores of assignment variables

Important: Assignment must be controlled! (It is as important as proper random assignment.)

Additional ConsiderationsFunctional form relating the assignment and

outcome variablesA defined population in which it is possible for all

units in the study to receive Tx regardless of the choice of a certain cutoff point

Intent-to-Treat? : Tx diffusion and cross-over participants

Variations1. Compare 2 treatment groups2. Compare 3 conditions3. Different dose treatment groups4. Multiple cutoff points5. …and many more creative ways to think of

Theory of RD – How does this work?

RDs as Treatment Effects in REs RE pretest means of Tx and Control groups nearly

identical at would be the cutoff score in an RD design through random assignment

Cutoff-based assignments creates groups with different pretest means and non-overlapping pretest distributions

RD compares regression lines, not meansBoth RDs and REs control for selection bias

Unknown variables do not determine assignment Pretests have no error IF used as the selection variable Regression lines are not affected by posttest errors

Adherence to the CutoffOverrides of the cutoffCrossoversAttrition“Fuzzy” regression discontinuity

Threats to ValidityStatistical Conclusion Validity

Nonlinearity Interactions

Internal Validity – must occur at the cutoff pointHistoryMaturationMortalitySelection-instrumentation

Interaction

Group Exercise: RD Design

Analytical AssumptionsNo exceptions to the cutoffAdhere to true function of the pre-post

relationshipUniform delivery of pretest and program

Combining RD with Randomized Experiments

7 combo examples3 advantages:

Increased power Allows estimation of both

groups at the overlap interval

Adds clarity to the cutoff point

RD – Quasi-experiment?shortfalls are not yet clearRequires more “demanding statistical analysis”Less statistical powersee table 7.2 in SCC (pg. 243)

Analysis ProblemsThe Curvilinear Problem

Steps to AnalysisTransform the pretestExamine the relationship visuallySpecify high order terms and interactionsEstimate the initial modelRefine

Comparing RD Designs with Experimental Designs

IN theory, both designs should produce similar results when all exemplary conditions of each method type exist

Question remains do they produce similar results and standard errors in practice (real world settings)?

Under exemplary conditions, experiments are 2.75 times more efficient than RDDs

If otherwise, the degree of this efficiency will varyCentral Question: How to compare these two design

options in field settings?Cook, Shadish & Wong 2008

Statistical Power for GRT and RDD

the RDD has approximately 36 per cent the efficiency of the GRT.

This implies that the RDD will require approximately 2.75 times more groups than a GRT with the same power.

The same result was found by Schochet [16] for hierarchical models in education

and by Cappelleri and Trochim [31] for trials targeting individuals rather than groups.

Pennell et al., 2010

Within Study Comparisons:

Proposed methodology from LaLondeCausal estimates derived from an experiment

compared with estimates from a non-experimentSame Tx GroupDifferent Control Group

Modifications needed to use for RDDs

7 Criteria to Improve Interpretation of Within-Study Comparisons

1. Must demonstrate variation in types of methods being contrasted

2. Both assignment mechanisms cannot be correlated with other factors related to outcome variables

3. The RE must “deserve” its status of the causal “Gold Standard”

4. The non-experiment design must also be good

AND

7 Criteria to Improve Interpretation of Within-Study Comparisons

AND

5. Both study types should estimate the same causal quantity

6. Explicit criteria must be raised on how the two design estimates relate to each other

7. Blind that data analyst!

Further Discussion?Nagging Questions?

…orInspirations?