Embed Size (px)
Citation preview
S-005
Intervention research:
True experiments and quasi-experiments
Intervention research• Goal is to assess the effects of some intervention or new idea or approach• Examples:
– Children’s vocabulary growth under a new curriculum– Effects of new technologies on student achievement– New approaches to professional development for teachers– New drug treatments or other interventions– Other examples?
• Key distinction between– True experiments
• We compare the new program (or treatment) to a traditional treatment (or alternative treatment or control group)
• Random assignment is the key ingredient for a study to qualify as a true experiment– Quasi-experiments
• Comparison of new program with an alternative (“new” vs. “traditional”, for example)• But random assignment not used• Comparisons may be biased by other factors• Need for great care and additional information for these studies to be convincing• Quasi-experimental designs are common in education, but they may have some important
limitations. (Some describe them as “mere observations” and not a strong basis for drawing conclusions. Others are not quite so harsh.)
True Experiments(for evaluating an intervention)
New treatment Control(or traditional or alternative)
New Comparison
True Experiments(for evaluating an intervention)
New treatment Control(or traditional or alternative)
New Comparison
Key issue: How do people get assigned to (or chosen for) the two options?
Start with a group of eligible people
Assign them randomly
True Experiments(for evaluating an intervention)
New treatment Control(or traditional or alternative)
New Comparison
Key issue: How do people get assigned to (or chosen for) the two options?
Start with a group of eligible people
Assign them randomly
Random assignment is the key ingredient required for a true experiment.
True experiments(for assessing an intervention)
New Comparison
The process of random assignment
Makes it very likely that the groups will be similar. The only difference will be which treatment they receive.The groups are very likely to be similar in age, gender ratio, prior experience, etc.
Some cautions: Compliance? Implementation?Sometimes the special conditions under which we carry out a true experiment make it hard to generalize. Others?
Quasi-experiments• Groups are compared,
but no random assignment
New Comparison
Do we really have a good comparison?
Can we really be sure about the new program?
Often we need lots of additional information to help us draw stronger conclusions?
?
Quasi-experiments
New program Traditional program
Post test average 85 80
What might we conclude?
Quasi-experimentsTwo scenarios
New program Traditional program
Post test average 85 80
Pre-test 65 60
New program Traditional program
Post test average 85 80
Pre-test 60 60
Quasi-experimentsAn even better scenario
New program Traditional program
Post test average 85 80
Pre-test 65 60
% girls 50% 52%
Parent education (HS graduates)
70% 74%
ELL 20% 18%
Adjustment strategies in quasi-experimentsNew program Traditional program
Post test average 85 80
Pre-test 65 60
% girls 50% 52%
Parent education (HS graduates)
70% 74%
ELL 20% 18%
Regression adjustmentsMultiple regression to adjust for other variablesPropensity score matchingSee where the two groups differ and where they overlapThen find good matches for those in the new program (Private schools in Mexico example)
What to do when there is non-random assignment?
