Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
Experimental Methods in the Social Sciences
Introduction to Experiments
August 5, 2013
Experiments Increasingly Important in theSocial Sciences
· Field experiments in political science as early as
1920s
· Growing use of experiments in development
economics, transportation, education, criminology,
· Global experiments in economic development,
alternative energy sources, education
2
Increase in Articles Using Experiments in
American Political Science Review
3
Why More Experiments Now?
· Better technology
· Maturation of social sciences
· Increased understanding of the complexity and
interdependence of research in social sciences
· Social sciences catching up to physical sciences
and medicine
4
Key Elements of Experiments
· Standardization
· Randomization
· Treatment versus Control Groups
· Between-Subjects versus Within-Subjects Design
· Internal versus External Validity
· Experimental Bias
5
Standardization
· The Dependent Variable (DV) – the thing to be
explained – is measured the same way in the same
context across subjects
· Independent Variables (IVs) – the explanatory
variables – also controlled and standardized
· Compared to field studies, experiments eliminate
or control unmeasured influences on the DV and
IVs
6
Treatment versus Control
· Treatment is the Independent Variable that the
experimenter manipulates.
· The Treatment Group receives the treatment.
· The Control Group that does not receive the
treatment
· An experiment may contain multiple treatments
and a control group.
7
Blinded Experiment
· In a blind experiment, subjects do not know
whether they received the treatment or control.
· A placebo is a false treatment, meant to make the
control group believe they received the treatment.
· In a double-blind experiment, the experiment
administrators do not know who received the
treatment or control
8
Randomization
· The key feature of an experiment is randomization
· In laboratory and survey experiments, subjects
assigned randomly to treatment and control
groups
· If subjects volunteer for either treatment or
control, we cannot be sure that different
outcomes are due to selection bias or to the
treatment
9
Randomization
· A simple randomizing device such as a coin flip
can assign subjects to treatment or control.
· An n-factor design includes n simultaneous
treatments.
· Each subject randomized across the n treatments.
10
Factorial Design
· Factorial design indicates the number of different
treatments and conditions per treatment
· Each subject receives a combination of treatments (or
controls)
· For example, a researcher may want to test whether a
combination of:
vitamins (vitamin or no vitamin)
exercise (exercise or no exercise
drugs (Drug A, Drug B, no drug)
affect a subject’s health
11
· A 2x2x3 factorial design indicates three separate
treatments.
· The first and second have two conditions (such as a
treatment and control)
· The third has three conditions (such as a control and two
treatments)
· The 2x2x3 design breaks subjects into 12 different groups
for analysis.
12
Between-Subjects versus Within Subjects
· Between Subjects breaks subjects into treatment
and control groups to measure differences
between the two groups after treatment.
· Within Subjects design measures subjects before
treatment and after treatment. Each subject is its
own control.
· Within Subjects also a Pre-post design.
13
Randomized Controlled Trial (RCT)
· The most powerful experimental technique in
scientific research
· RCT’s are usually necessary in a clinical trial for
medical treatments
· Randomized Trial similar to RCT but does not
contain a control group
14
RCT’s Classified by Study Design
1. Parallel Group: Each subject randomly assigned
to a group. All subjects in a group receive the
same treatment (or control)
2. Crossover: Each subject receives or does not
receive the treatment in a random sequence
15
3. Cluster: Pre-existing groups of subjects are
selected to receive the treatment (or control).
Groups may include schools, classes, villages, city
blocks, provinces, countries.
4. Factorial: Each subject randomly assigned to a
group that receives combinations of treatments
(or controls).
16
Standard Reporting Flowchart forParallel RCT
17
18
Example: Dartmouth SPORT Study(Weinstein, et al.)
· Spine Patients Outcome Research Trial (SPORT)
· RCT to assess whether surgery or conservative
care (rest, physical therapy, drugs) provided
better outcomes for herniated disc, spondyliosis,
and other ailments of the spine
· Published in the Journal of the American Medical
Association, 2007, 2008
19
2,720 Patients were screened
501 enrolled in the randomized cohort
245 were assigned to surgery 256 were assigned tonon-surgical treatment
729 Patients were ineligible
426 Were not surgical candidates 19 Had fracture, infection, or deformity 129 Had inadequate non-operative treatment 20 Had cancer 135 Had other reasons
1,991 Patients were eligible
747 Patients declined to participate
743 enrolled in the observational cohort
521 chose surgery222 chose non-surgical
treatment
203 Were available at 6 wk 40 Missed the follow-up visit 2 Withdrew 0 Died
75 (31%) Had undergone surgery
198 Were available at 3 mo 45 Missed the follow-up visit 2 Withdrew 0 Died
116 (47%) Had undergone surgery
200 Were available at 6 mo 37 Missed the follow-up visit 8 Withdrew 0 Died
133 (54%) Had undergone surgery
202 Were available at 1 yr 29 Missed the follow-up visit 14 Withdrew 0 Died
139 (57%) Had undergone surgery
187 Were available at 2 yr 35 Missed the follow-up visit 23 Withdrew 0 Died
141 (58%) Had undergone surgery
219 Were available at 6 wk 37 Missed the follow-up visit 0 Withdrew 0 Died
46 (18%) Had undergone surgery
211 Were available at 3 mo 44 Missed the follow-up visit 1 Withdrew 0 Died
73 (29%) Had undergone surgery
210 Were available at 6 mo 41 Missed the follow-up visit 5 Withdrew 0 Died
96 (38%) Had undergone surgery
213 Were available at 1 yr 27 Missed the follow-up visit 15 Withdrew 1 Died
106 (41%) Had undergone surgery
191 Were available at 2 yr 36 Missed the follow-up visit 27 Withdrew 2 Died
110 (43%) Had undergone surgery
464 Were available at 6 wk 57 Missed the follow-up visit 0 Withdrew 0 Died
471 (90%) Had undergone surgery
434 Were available at 3 mo 84 Missed the follow-up visit 2 Withdrew 1 Died
489 (94%) Had undergone surgery
443 Were available at 6 mo 70 Missed the follow-up visit 7 Withdrew 1 Died
492 (94%) Had undergone surgery
448 Were available at 1 yr 56 Missed the follow-up visit 16 Withdrew 1 Died
493 (95%) Had undergone surgery
429 Were available at 2 yr 48 Missed the follow-up visit 43 Withdrew 1 Died
493 (95%) Had undergone surgery
197 Were available at 6 wk 24 Missed the follow-up visit 1 Withdrew 0 Died
4 (2%) Had undergone surgery
187 Were available at 3 mo 34 Missed the follow-up visit 1 Withdrew 0 Died
19 (9%) Had undergone surgery
187 Were available at 6 mo 33 Missed the follow-up visit 2 Withdrew 0 Died
35 (16%) Had undergone surgery
189 Were available at 1 yr 28 Missed the follow-up visit 5 Withdrew 0 Died
44 (20%) Had undergone surgery
192 Were available at 2 yr 14 Missed the follow-up visit 15 Withdrew 1 Died
48 (22%) Had undergone surgery
180 Were available at 3 yr 35 Missed the follow-up visit 29 Withdrew 1 Died
142 (58%) Had undergone surgery
170 Were available at 3 yr 47 Missed the follow-up visit 37 Withdrew 2 Died
111 (43%) Had undergone surgery
382 Were available at 3 yr 76 Missed the follow-up visit 60 Withdrew 3 Died
493 (95%) Had undergone surgery
175 Were available at 3 yr 24 Missed the follow-up visit 22 Withdrew 1 Died
52 (23%) Had undergone surgery
149 Were available at 4 yr 47 Missed the follow-up visit 33 Withdrew 1 Died 15 Had visits pending
144 (59%) Had undergone surgery
150 Were available at 4 yr 46 Missed the follow-up visit 43 Withdrew 2 Died 15 Had visits pending
115 (45%) Had undergone surgery
342 Were available at 4 yr100 Missed the follow-up visit 76 Withdrew 3 Died
493 (95%) Had undergone surgery
166 Were available at 4 yr 28 Missed the follow-up visit 27 Withdrew 1 Died
53 (24%) Had undergone surgery
Figure 1. Exclusion, enrollment, randomization, and follow-up of trial participants. The values for surgery, withdrawal, and death are cumulative over4 years. For example, a total of 1 patient in the group assigned to surgery died during the 4-year follow-up period. [Data set 04/10/2008].
2791Surgical vs. Nonoperative Treatment for Lumbar Disc Herniation • Weinstein et al
SPORT Study Has Many of the SameProblems Social Scientists Encounter
· Many subjects did not comply. Some assigned to
surgery opted for conservative care. Some
assigned to conservative care opted for surgery
· Many subjects could not re-contacted
· Potential placebo effect from surgery
· Some outcomes measured using surveys: People
were asked how they felt
21
Internal versus External Validity
· Internally validity asks, “Did the experimental
treatment make a difference in this specific
experiment.”
· External validity asks, “To what populations,
settings, treatment variables, and measurement
variables can this effect be generalized?”
22
Internal Validity Threatened by...
· History
· Intra-experiment events (intrasession history)
· Selection bias in subjects
· Maturation of subjects
· Performance effects (conditioning)
· Regression toward the mean
· Mortality or attrition of subjects
23
External Validity Threatened by...
· Uncontrolled interactions, omitted variables,
spurious correlations
· Testing may make subjects more sensitive than
rest of population to variables under investigation
· Unrepresentative samples (college students in
particular)
24
External Validity Threatened by...
· Hawthorne effects: subjects change their behavior
when observed
· Professionalized subjects (Panel conditioning:
experiment may change subjects)
· Mortality or attrition of subjects
25
Experimental Bias
· Expectancy Effects: the experimenter influences
subjects
· Demand Effects: Subjects anticipate the purpose
of the experiment
· Experimenter Bias. in recruiting subjects,
selecting experiment time and place, coding and
analyzing data.
26
Ethical Issues
· No physical, financial, emotional harm
· Subjects must give informed consent
· Subjects should not reveal information that is
illegal or threatening to them
· Deceiving subjects acceptable as long as they are
debriefed
27
Common Types of Experiments
· Laboratory experiments
· Field experiments
· Survey experiments
· Natural experiments
· Quasi-experiments: usually non-random
28
Other Thoughts
· “You can solve only one hard problem at a time.”
· Think about causation and mechanisms
· Start with a theory. Experiments test theories.
· Proof is theoretical. Confirmation is empirical.
· Read across academic disciplines
· To paraphrase Marvin Minsky: for any problem (research
question), the best solution (experiment) is the one that
uses the least time, energy, resources
29