Upload
magnar
View
206
Download
1
Embed Size (px)
DESCRIPTION
Regression Discontinuity Design. Thanks to Sandi Cleveland and Marc Shure (class of 2011) for some of these slides. RD Designs. A pretest- posttest , program-comparison group strategy Review: Advantages of Pre-tests ? Detect differences between groups - PowerPoint PPT Presentation
Citation preview
Regression Discontinuity Design
Thanks to Sandi Cleveland and Marc Shure (class of 2011) for some of these slides
RD DesignsA pretest-posttest, program-comparison
group strategy Review:
Advantages of Pre-tests?Detect differences between groupsDetect potential vulnerability to internal validity
threatsHelps with statistical analysis
Advantages of Comparison groups?Helps control sources of errorHelps support the counterfactual inference
Underuse of RD? Why?It’s new.Key criteria must be met for use.Perhaps it’s just misunderstood.
Overview of RD
OA C X O2
OA C O2
“pre” is ANY continuous variable that correlates with the outcome of interest
Assignment based on cutoff scoreRegression line should have vertical displacement at
the cutoff score if there is an effect
No Treatment Effect
Positive Effect
Examples1. Campell & Stanley’s Ivy League Education
Example2. Trochim’s Hospital Administration ExampleHospital Quality of Care
More about assignmentAssignment variables:
Must be continuous (or ordinal) Can be a pretest on a dependent
variable Can be by order of entry into study Cannot be caused by treatment May or may not be related to the
outcome
If implementing an RD design in your area of research, what variables would you choose for assignment?
Choosing the Cutoff ScoreNow referring to the assignment variable(s) you
identified, how would you arrive at a cutoff score?
Substantive grounds: professional judgmentNeed or MeritClinical diagnosis
Practical grounds:Available data setsAvailable resources
Choosing the Cutoff ScoreMean of the distribution of assignment scoresPolitically defined thresh-holdsComposite scores of assignment variables
Important: Assignment must be controlled! (It is as important as proper random assignment.)
Additional ConsiderationsFunctional form relating the assignment and
outcome variablesA defined population in which it is possible for all
units in the study to receive Tx regardless of the choice of a certain cutoff point
Intent-to-Treat? : Tx diffusion and cross-over participants
Variations1. Compare 2 treatment groups2. Compare 3 conditions3. Different dose treatment groups4. Multiple cutoff points5. …and many more creative ways to think of
Theory of RD – How does this work?
RDs as Treatment Effects in REs RE pretest means of Tx and Control groups nearly
identical at would be the cutoff score in an RD design through random assignment
Cutoff-based assignments creates groups with different pretest means and non-overlapping pretest distributions
RD compares regression lines, not meansBoth RDs and REs control for selection bias
Unknown variables do not determine assignment Pretests have no error IF used as the selection variable Regression lines are not affected by posttest errors
Adherence to the CutoffOverrides of the cutoffCrossoversAttrition“Fuzzy” regression discontinuity
Threats to ValidityStatistical Conclusion Validity
Nonlinearity Interactions
Internal Validity – must occur at the cutoff pointHistoryMaturationMortalitySelection-instrumentation
Interaction
Group Exercise: RD Design
Analytical AssumptionsNo exceptions to the cutoffAdhere to true function of the pre-post
relationshipUniform delivery of pretest and program
Combining RD with Randomized Experiments
7 combo examples3 advantages:
Increased power Allows estimation of both
groups at the overlap interval
Adds clarity to the cutoff point
RD – Quasi-experiment?shortfalls are not yet clearRequires more “demanding statistical analysis”Less statistical powersee table 7.2 in SCC (pg. 243)
Analysis ProblemsThe Curvilinear Problem
Steps to AnalysisTransform the pretestExamine the relationship visuallySpecify high order terms and interactionsEstimate the initial modelRefine
Comparing RD Designs with Experimental Designs
IN theory, both designs should produce similar results when all exemplary conditions of each method type exist
Question remains do they produce similar results and standard errors in practice (real world settings)?
Under exemplary conditions, experiments are 2.75 times more efficient than RDDs
If otherwise, the degree of this efficiency will varyCentral Question: How to compare these two design
options in field settings?Cook, Shadish & Wong 2008
Statistical Power for GRT and RDD
the RDD has approximately 36 per cent the efficiency of the GRT.
This implies that the RDD will require approximately 2.75 times more groups than a GRT with the same power.
The same result was found by Schochet [16] for hierarchical models in education
and by Cappelleri and Trochim [31] for trials targeting individuals rather than groups.
Pennell et al., 2010
Within Study Comparisons:
Proposed methodology from LaLondeCausal estimates derived from an experiment
compared with estimates from a non-experimentSame Tx GroupDifferent Control Group
Modifications needed to use for RDDs
7 Criteria to Improve Interpretation of Within-Study Comparisons
1. Must demonstrate variation in types of methods being contrasted
2. Both assignment mechanisms cannot be correlated with other factors related to outcome variables
3. The RE must “deserve” its status of the causal “Gold Standard”
4. The non-experiment design must also be good
AND
7 Criteria to Improve Interpretation of Within-Study Comparisons
AND
5. Both study types should estimate the same causal quantity
6. Explicit criteria must be raised on how the two design estimates relate to each other
7. Blind that data analyst!
Further Discussion?Nagging Questions?
…orInspirations?