Introduction to Statistics : Design of Experiments

Introduction to StatisticsDesign of Experimentsg p

Instructor : Siana Halim

-S. Halim -

Introduction to Statistics : Design of Experiments

TOPICS

• Controlled Experiments

• Observational Studies

References

Statistics, David Freedman, Robert Pisani, Roger Purves, , , g

3rd. Edition, Norton International Student Edition, USA, 1998

-S. Halim -

Introduction to Statistics : Design of Experiments

-S. Halim -

Introduction to Statistics : Design of Experiments

1. CONTROLLED EXPERIMENT

PROBLEM : A new drug is introducedHow should an experiment be designed to test its effectiveness ?How should an experiment be designed to test its effectiveness ?

METHOD: The basic method is comparison:The drug is given to subject in:

A treatment group VsA treatment group VsA control group (they aren’t treated)

Compare the responses of the two groups

Design :Subject should be assigned to treatment or control at randomThe experiment should be run double blind:(Neither the subjects nor the doctors who measure the responses(Neither the subjects nor the doctors who measure the responses should know who was in the treatment group and who was in the control group)

-S. Halim -

Introduction to Statistics : Design of Experiments

Example 1 : Jonas Salk’s Vaccines – POLIO

Background :(1950) In Laboratory trials, it had proved

safe and had caused the production of tib di i t li !antibodies against polio !

(1954) Try the vaccine in the real world ! outside the laboratory

Fact:Polio was an epidemic disease, whose

incidence varied from year to year

-S. Halim -

Introduction to Statistics : Design of Experiments

Design I : Random Design

The only way to find out whether the vaccine worked was to deliberately leavesome children unvaccinated and use them as controlssome children unvaccinated and use them as controlsBUT ! – (the ethics' problem)This raises a troublesome questions of medical ethics, because withholding treatmentseems cruel. However, even after extensive laboratory testing, it is often unclearWhether the benefits of a new drug outweigh the risk !Only a well-controlled experiment can settle this question !

Fact ! : 2 million children from Grade 1 Grade 2 Grade 3Fact ! : 2 million children from Grade 1, Grade 2, Grade 3

0.5 million were vaccinated

1 million were deliberately left unvaccinated

0.5 million refused vaccinated.

-S. Halim -

Introduction to Statistics : Design of Experiments

This illustrates the method of comparison

Only the subjects in the treatment group were vaccinated and the controldid not get the vaccine.did not get the vaccine.

The responses of the two groups could then be compared to see if the treatment made any difference

Note:

The treatment and control groups were of different sizes, but that did not matternot matter

• The investigators compared the rate at which children got the polio in the two groups – cases per hundred thousand

• Looking at the rates instead of the absolute numbers, adjusts for the differences in the sizes of the groups

-S. Halim -

Introduction to Statistics : Design of Experiments

D i II B P tDesign II : By Parents

Fact ! Children could be vaccinated only with their parent’s permission

(This is one possible design which also seems to solve the ethical problem)

Now:

• The children whose parents consent would go into the treatment group and get• The children whose parents consent would go into the treatment group and get the vaccine (in reality : higher income parents)

•The other children would be the control (in reality : lower income parents)

VS

-S. Halim -

Introduction to Statistics : Design of Experiments

BIAS

This design is biased against the vaccine, because children of higher i t l bl t liincome parents are more vulnerable to polio

WHY ?

Paradoxical ?

Usually most disease fall more heavily on poor !

-S. Halim -

Introduction to Statistics : Design of Experiments

BUT !

Polio is a disease of hygiene

Children who live in less hygiene surrounding tend to contract mild cases of polio early in childhood while still protected by anti bodies fromcases of polio early in childhood while still protected by anti-bodies from their mother

After being infected, they generated their own antibodies which protect them against most severe infection laterthem against most severe infection later

Children who live in more hygiene surrounding do not develop such antibodies

Comparing volunteers to non-volunteers biases the experiment !!!

STATISTICAL LESSON

The treatment and control group should be as similar as possible. Then

Comparing volunteers to non volunteers biases the experiment !!!

any difference in response between the two groups is due to the treatment rather then something else.

-S. Halim -

Introduction to Statistics : Design of Experiments

D i III B G d (NFIP)Design III : By Grade (NFIP)

Vaccinate all Grade 2 children whose parents would consent,

Leaving the children in Grades 1 and 3 as controls

BUT !U

Polio is a contagious disease, spreading through contact.

The incidence could have been higher in Grade 2 then in Grades 1 and 3

Biased the study against the vaccine or

The incidence could have been lower in Grade 2, biasing the study in favor of the vaccine.

-S. Halim -

Introduction to Statistics : Design of Experiments

D i IVDesign IV :

Control group had to be chosen from the same population as the treatment group children whose parents consented to vaccinatedgroup – children whose parents consented to vaccinated

Otherwise the effect of family background would be confounded with the effect of the vaccine

Next Problem

Assigning the children to treatment or control !

Human judgment seems necessary, to make the control group like the treatment group on the relevant variables – family income as well as the children’s general health, personality and social habits.

-S. Halim -

Introduction to Statistics : Design of Experiments

Experience Shows : BIAS !It is better to rely on impersonal chance, e.g., tossing coin, with a 50-50 chance y p , g , g ,of assignment to the treatment group or control group !

The Law of chance guarantee that with enough subjects the treatment group and the control group will resemble each other very closely w.r.t all the important variability whether or not these have been identified

RANDOMIZED CONTROLLED !!

-S. Halim -

Introduction to Statistics : Design of Experiments

D bl Bli diDouble BlindingUse PLACEBO : Children in the control group more given an injection of salt dissolve wateran injection of salt dissolve water.

During the experiment the subjects did not know whether they were in treatment or in control, so their response was to the vaccine not the idea of treatment.the vaccine not the idea of treatment.

The doctors were not told which group the child belonged to.

Thi d bl bli diThis was double blinding:

-The subjects did not know whether they get the treatment of placebo

-Neither did those who evaluated the responses.

Placebo: a pill or medicine, or procedure prescribed more for the psychological benefit to the patient of being given a prescription than for any physiological effect

Neither did those who evaluated the responses.

-S. Halim -

Introduction to Statistics : Design of Experiments

How did it all turn out ?

The Randomized Controlled The NFIP StudyDouble-blinded experiment

Size Rate

Treatment 200 000 28

y

Size Rate

Grade 2 (Vaccine) 225.000 25Treatment 200.000 28

Control 200.000 71

No consent 300.000 46

Grade 1 and 3 725.000 54

(control)

Grade 2 (no consent) 125 000 44Grade 2 (no consent) 125.000 44

Reduced Bias to a minimum Biased !

Introduction to Statistics : Design of Experiments

Confounding !

• The NFIP treatment group included only children whose parents consented to vaccination

• But the control group also included children whose parents would not have consentedhave consented

The control group was not comparable to the treatment group !

• The investigators do not have enough information to figure the chances for the outcomes !

It seems that the vaccine has no effect ! BUT this was becauseIt seems that the vaccine has no effect ! BUT this was because wrong in the design of experiment !

Introduction to Statistics : Design of Experiments

Summary1. Statisticians use the method of comparison. They want to know the effect of a

treatment on a response. To find out, the compare the responses of a treatment group with a control group.

2. If the control group is comparable to the treatment group, apart from the treatment, then a difference in the responses of the two groups is likely to be due to the effect of the treatment

3. However, if the treatment group is different from the control group w.r.t. other factors, the effects of these other factors are likely to be confounded with the effect of the treatment

4. To make sure that the treatment group is like the control group, investigators put subjects into treatment or control at random

5. Whenever possible, the control group is given a placebo

6. Use double-blind experiment.

Introduction to Statistics : Design of Experiments

2. OBSERVATIONAL STUDIES

2 1 Introduction2.1 Introduction• Controlled experiments are different from observational study

In controlled experiments the investigators decide who will be in theIn controlled experiments, the investigators decide who will be in the treatment group and who will be in the control group

In an observational study it is the subjects who assign themselves to the different groups. The investigators just watch what happensthe different groups. The investigators just watch what happens

• Jargon :

A control is a subject who did not get treatment

A controlled experiment is a study where the investigators decide who will be in the treatment group and who will not.

Introduction to Statistics : Design of Experiments

Example :

• A study of the effects of smoking are necessarily observational; No body is going to smoke for ten years just to please a statistician

• However, the treatment – control idea is still used. The investigators compare smokers (the

“treatment or “exposed” group with non-smokers –the control group to determine the effect of smoking)

Observational studies are a powerful tool, as the smoking examples shows. But they can also be quite misleading. To see if confounding is a problem, it may help to find out how the controls were selected.

The main issue was the control group really similar to the treatment group – apart from the exposure of interest ?

Introduction to Statistics : Design of Experiments

St d C PELLAGRAStudy Case : PELLAGRAFirst observed in Europe in 18th century by Gaspar Casal;

It was an important cause of ill-health, disability, and premature death among the veryIt was an important cause of ill health, disability, and premature death among the very poor inhabitants of Asturias

Seemed to hit some villages much more than others, were

S it diti i di d h h ld i itiSanitary conditions in diseased households were primitive

Flies were everywhere

One blood-sucking fly (Simulium) had the same geographical range as Pellagra, g y ( ) g g p g g ,at least in Europe; and the fly was most active in the spring, just when most Pellagra cases developed

Many epidemiologists concluded the disease was infectious, and like-malaria, yellow y p g yfever or typhus – was transmitted from one person to another by insects.

Was this conclusion justified ?

Introduction to Statistics : Design of Experiments

An Observational Study & Experiments by Joseph Goldberger (1914)

Pellagra is caused by a bad diet and is not infectious

Can be prevented or cured by foods rich in what Goldberger called the P.P (Pellagra Prevented) factor – or Niacin in market.

Ni i t ll i t ilkNiacin occurs naturally in meat, milk, eggs, some vegetables and certain corn

In the Pellagra areas :

The poor ate corn and not much else. But corn however, contains relatively little niacin ! That is why they were harder hit by the disease. The flies where a marker of poverty, not a cause of pellagra

Association is not the same as caution !

-S. Halim -

Introduction to Statistics : Design of Experiments

Confounding

Cause Effect / Outcome

(Independent Variable) (dependent Variable)

Other factors

(confounding Variable)

Confounding means a difference between the treatment and control groups – other than the treatment – which

(confounding Variable)

and control groups other than the treatment which effects the responses being studied. A confounder is a third variable, associated with exposure and with disease

-S. Halim -

Introduction to Statistics : Design of Experiments

Summary and Overview1. In an observational study, the investigators do not assign the subjects to treatment or

control Some of the subjects have the condition whose effects are being studied;control. Some of the subjects have the condition whose effects are being studied; this is the treatment group. The other subjects are the controls.

2. Observational studies can establish association : one thing is linked to another. Association may point to causation: if exposure causes disease then people who areAssociation may point to causation: if exposure causes disease, then people who are exposed should be sicker than similar people who are not exposed. But association does not prove causation

3 In an observational study the effects of treatment may be confounded with the3. In an observational study, the effects of treatment may be confounded with the effects of factors that got the subjects into treatment or control in the first place. Observational studies can be quiet misleading about cause-and-effect relationships, because of confounding

4. When looking at a study, ask the following questions

-S. Halim -

Introduction to Statistics : Design of Experiments

Studies

Controls No ControlsControls No Controls

Contemporaneous Historicalp

Controlled E i

Observational S diExperiment Studies

d d N d dRandomized Not Randomized

-S. Halim -

Introduction to Statistics : Design of Experiments

5. With Observational studies, and with nonrandomized controlledexperiments, try to find out how the subjects came to be in treatment or in control. Are the groups comparable ? Different ? What factors are confounded with treatment ? What adjustments were made to take care of confounding ? Were they sensible ?

6. In an observational study, a confounding factor can sometimes be controlled for, by comparing smaller groups which are relatively homogeneous w.r.t. the factor.g

7. Study design is a central issue in applied statistics.

-S. Halim -