CHAPTER 13 EXPERIMENTS & OBSERVATIONAL STUDIES. OBSERVATIONAL STUDIES researchers don’t assign choices, they observe them a study in which no manipulation
OBSERVATIONAL STUDIES researchers dont assign choices, they
observe them a study in which no manipulation of factors has been
employed helpful for discovering trends and possible relationships
although an observational study may identify important variables
related to an outcome. there is no guarantee that we have found the
right or most important related variable
Slide 3
OBSERVATIONAL STUDIES Retrospective Study Prospective Study
subjects are selected then previous conditions or behaviors are
determined not based on random samples, they focus on estimating
differences between groups or associations between variables have a
restricted view of the world because they are usually restricted to
a small part of the population because they are based on historical
data there could be errors identifying subjects in advance and
collecting data as the events unfold
Slide 4
Observational studies that try to discover variables related to
rare outcomes are often retrospective Example: specific disease
identify people with the disease look at their history and heritage
something that could be related to their condition
Slide 5
WHO GETS GOOD GRADES? OR WHY? In 1981 a study was conducted at
a high school in California. Researchers compared the scholastic
performance of music students with that of non-music students. The
music students had a higher GPA of 3.59 compared to 2.91 that
non-music students had 16% of music students earned all As where
only 5% of non-music students earned all As Does this study tell us
that music will improve students GPAs? students that study music
might still differ from the others in some important way that we
failed to observe.
Slide 6
CAN WE PROVE CAUSE AND EFFECT?!? Experiment: studies the
relationships between two or more variables manipulates factor
levels to create treatments, randomly assigns subjects to these
treatment levels, and then compares the responses of the subject
groups across treatment levels. Random assignment: an experiment
must assign experimental units to treatment groups at random for
the experiment to be valid.
Slide 7
EXPERIMENT VOCABULARY Factor: a variable whose levels are
controlled by the experimenter. Experiments attempt to discover the
effects that differences in factor levels may have on the responses
of the experimental units Response variable: a variable whose
values are compared across different treatments. In a randomized
experiment, large response differences can be attributed to the
effect of differences in treatment levels. Subject/experimental
unit: individuals on whom an experiment is performed. (participants
for humans too) Level of the factor: the specific values that the
experimenter chooses for a factor Treatment: the process,
intervention, or other controlled circumstance applied to randomly
assigned experimental units. Treatments are the different levels of
a single factor or are made up of combinations of levels of two or
more factors
Slide 8
EXPERIMENT EXAMPLE Design an experiment to see whether the
amount of sleep and exercise you get affects your performance
Subjects: the people in the sleep study Factors: sleep and exercise
Factor levels: sleep: 4, 6, or 8 hours exercise: 0 min or 30 mins
on a treadmill Treatment: 6 total Once all this is set up you need
to RANDOMLY assign each subject to a treatment 4 hours 0 min 30
mins 6 hours 0 min 30 min 8 hours 0 min 30 min
Slide 9
MUSIC & GPA EXAMPLE only an experiment can justify a claim
like music lessons cause higher grades Music and GPA experiment
take a group of 3 rd graders (subjects) study music and never take
music classes (factor and levels) collect data about their GPA
(response variable)
Slide 10
4 Principles Of Experimental Design 1) Control make conditions
as similar as possible for all treatments groups. reduces the
variability of the responses, making it easier to detect
differences among the treatment groups Risky!! we are testing
laundry detergents. we control the water temperature at 180 0. this
would reduce variation in our results due to water temperature. BUT
now we cant say anything about the detergent in cold water we
control a factor by assigning subjects to different levels because
we want to see how the response will change at different levels we
control other sources of variation to prevent them changing and
affecting the response variable
Slide 11
2) Randomize allows us to equalize the effects of unknown or
uncontrollable sources of variation it does not eliminate the
effects of these sources, it spreads them out over all the
treatments so we can see past them protects us from things we didnt
even know about control what you can, and randomize the rest
Slide 12
3) Replicate we should repeat the experiment, applying the
treatments to a number of subjects only this can we estimate the
variability of responses an experiment is only complete once you
have assessed the variation the outcome of an experiment on a
single subject is an anecdote, not data when the subjects are not a
representative sample of the population of interest; repeat the
experiment with people from different ages and different time of
year Replication of an entire experiment with controlled sources of
variation at different levels is an essential step in science
Slide 13
(Not Required) 4) Block to reduce the effects of identifiable
attributes of the subjects that cannot be controlled. example: 10
people 2 math teachers and 8 math students we want to break them up
into two groups for a math competition. if we pick people randomly
the math teachers could be on the same team. (unfair) so we put 1
math teacher on each team then we randomly assign 4 students to
each team. we are blocking the occupation variable allowing us to
remove the variability due to the differences among the
blocks.
Slide 14
Diagrams an experiment is carried out over time with specific
actions occurring in a specified order a diagram of the procedure
can help think about experiments random allocation group #1
treatment # 1 group # 2 treatment #2 compare
Slide 15
Designing An Experiment: Step-by-step An ad for OptiGro plan
fertilizer claims that with this product you will grow juicer,
tastier tomatoes. You would like to test this claim, and wonder
whether you might be able to get by with half the specified dose.
Basically we need to buy some tomato plants and use OptiGro on some
of them. We will set up a completely randomized experiment in one
factor
Slide 16
THINK Plan: state what you want to know I want to now whether
tomato plants grown with OptiGro yield juicer, tastier tomatoes
than plants raised in otherwise similar circumstances but without
the fertilizer. Response variable Ill evaluate the juiciness and
taste of the tomatoes by asking a panel of judges to rate them on a
scale from 1 to 7 in juiciness and in taste. Treatments: specify
the factor levels and the treatments The factor is fertilizer,
specifically OptiGro. Ill grow tomatoes at three different factor
levels; some with no fertilizer, some with half the specified
amount, and some with the full dose of OptiGro. These are the three
treatments.
Slide 17
Experimental Units: Ill obtain 24 tomato plants of the same
variety from a local garden store. Experimental Design: observe the
principles of design Control: any source of variability you know of
and can control Randomly assign: experimental units to treatments,
to equalize the effects of unknown or uncontrollable sources of
variation Replicate: results by placing more than one plant in each
treatment group Ill locate the farm plots near each other so that
the plants get similar amounts of sun and rain and experience
similar temperatures. I will weed the plots equally and otherwise
treat the plants alike. Ill randomly divide the plants into three
groups. I will use random numbers from a table to determine the
assignment. There will be 8 plants in each treatment group. Draw a
picture 24 tomato plants from a garden store Group 1 8 plants
Treatment 1 control (no fertilizer) Group 2 8 plants Treatment 2
dose Group 3 8 plants Treatment 3 full fertilizer compare juiciness
and tastiness
Slide 18
Specify any other experiment details. You must give enough
details so that another experimenter could exactly replicate your
experiment. Its generally better to include details that might seem
irrelevant than to leave out matter that could turn out to make a
difference. Specify how to measure the response. I will grow the
plants until the tomatoes are mature, as judged by reaching a
standard color. I will then harvest the tomatoes when ripe and
store then for evaluation. I will set up a numerical scale of
juiciness and one for tastiness for the taste testers. Several
people will taste slices of tomato and rate them.
Slide 19
Show Once you collect the data, youll need to display them and
compare the results for the three treatment groups. I will display
the results with side-by-side box-plots to compare the three
treatment groups. I will compare the means of the groups.
Slide 20
Tell To answer the initial question, we ask whether the
difference we observe in the means of the three groups are
meaningful.
Slide 21
Control Treatment another level of the factor in order to
compare the treatment results to a situation in which nothing
happens example: the group of tomatoes that received no fertilized
Different from control (one of the 4 principles of experiments).
controlling extraneous sources of variations by keeping them
constant example: buying the plants from the same nursery, weeding
all the plots the same way
Slide 22
Blinding To avoid bias, we disguise the levels of the factors
There are two main classes of individuals who can affect the
outcome of the experiment: those who could influence the results
the subjects, treatment administrators, or technicians those who
evaluate the results (judges, treating physicians, etc) Single
Blind: when one of the groups is blinded Double Blind: when both of
the groups are blinded Tomato Experiment: we dont want our tasters
to be bias so we will not tell them plants received OptiGro
Slide 23
Placebo a control treatment that mimics the real treatment the
best way to blind subjects placebo effect: it is not unusual for
20% or more of subjects given a placebo to report reduction or
pain, improved movement, or greater alertness highlights both the
importance of effective blinding and the importance of comparing
treatments with a control
Slide 24
The best experiments are usually randomized comparative
double-blind placebo-controlled
Slide 25
Blocking Tomato plants: we want 18 plants we get 12 from store
A and 6 from store B The store the plants came from could effect
our results So we break the 18 plants into two BLOCKS one for each
store Then we randomly assign the plants in each block to one of
the three treatments This is called randomized block design
blocking is the same idea for experiments as stratifying for
samples we use blocks to reduce variability so we can see the
effects of the factors, we are nut usually interested in studying
the effects of the blocks themselves
Slide 26
Diagram 12 tomato plants from store A and 6 plants from store B
Block A: 12 tomato plants group 1 4 plants treatment 1 control
compare juiciness and tastiness group 2 4 plants treatment 2 dose
group 3 4 plants treatment 3 full dose Block B: 6 tomato plants
group 4 2 plants treatment 1 control compare juiciness and
tastiness group 5 2 plants treatment 2 dose group 6 2 plants
treatment 3 full dose
Slide 27
Matching Matching: when subjects are paired because they are
similar in ways NOT being studied in retrospective and prospective
studies Music and grades: we might match each student who studies
an instrument with someone of the same sex and similar family
income who doesnt play an instrument
Slide 28
Example Problems Describe a strategy to randomly split the 24
tomato plants into the three groups for the 3 treatments in the
OptiGro experiment.
Slide 29
A running shoe manufacturer wants to test the speed of its new
sprinting shoe on the 100-meter dash times. The company sponsors 5
athletes who are running the 100-meter dash in the 2004 Summer
Olympic games. To test the show, they have all 5 runners run the
100- meter dash with a competitor's shoe and then again with their
new shoe. They use the difference in times as the response
variable. Suggest some improvements to the design. Why might the
shoe manufacturer not be able to generalize the results they find
to all runners?
Slide 30
Athletes who had suffered hamstring injuries were randomly
assigned to one of two exercise programs. Those who engages in
static stretching returned to sports activity in a mean of 37.4
days (SD = 27.6 days). Those assigned to a program of agility and
trunk stabilization exercises returned to sports in a mean of 22.2
days (SD = 8.3 days). Explain why it was important to assign the
athletes to the two different treatments randomly. There was no
control group of athletes who did not participate in a special
exercise program. Explain the advantage of including such a group
in this experiment. How might blinding have been used in this
experiment? Do you think the difference in times is statistically
significant?
Slide 31
Adding More Factors OptiGro Experiment We want to add a factor
about watering Levels: only natural watering (rain) and watering by
hand daily Treatments: 6 Completely randomized two-factor
experiment No fertilizer fertilizerfull fertilizer No watering 123
Daily watering 456
Slide 32
12 tomato plants from a garden store Group 1 2 plants Treatment
1 control/ no water Group 2 2 plants Treatment 2 /no water Compare
juiciness and tastiness Group 3 2 plants Treatment 3 full/no water
Group 4 2 plants Treatment 4 control/water Group 5 2 plants
Treatment 5 / water Group 6 2 plants Treatment 6 full/water
Slide 33
CONFOUNDING When the levels of one factor are associated with
the levels of another factor, the factors are said to be
confounding Example: A credit card bank wanted to test the
sensitivity of the market by two factors: the annual fee charged
for a card and the annual percentage rate charged. The bank
selected 100,000 people at random from a mailing list. It sent out
50,000 offers with a low rate and no fee and 50,000 offers with a
higher rate and a $50 annual fee. They found out people signed up
(more than twice the rate) for the card with low rate and no annual
fee. Problem: The question that the bank was trying to answer was
how much of the change is due to the rate and how much was due to
the fee? How could they have avoided this?