Author(s): Kerby Shedden, Ph.D., 2010

License: Unless otherwise noted, this material is made available under the terms of the Creative Commons Attribution Share Alike 3.0 License: http://creativecommons.org/licenses/by-sa/3.0/

We have reviewed this material in accordance with U.S. Copyright Law and have tried to maximize your ability to use, share, and adapt it. The citation key on the following slide provides information about how you may share and adapt this material.

Copyright holders of content included in this material should contact open.michigan@umich.edu with any questions, corrections, or clarification regarding the use of content.

For more information about how to cite these materials visit http://open.umich.edu/privacy-and-terms-use.

Any medical information in this material is intended to inform and educate and is not a tool for self-diagnosis or a replacement for medical evaluation, advice, diagnosis or treatment by a healthcare professional. Please speak to your physician if you have questions about your medical condition.

Viewer discretion is advised: Some medical content is graphic and may not be suitable for all viewers.

1 / 44

Observational Studies and Experiments

Kerby Shedden

Department of Statistics, University of Michigan

April 8, 2011

2 / 44

Comparative studies

The goal of a comparative study is to estimate the difference insome characteristic between groups.

Here are some examples of comparative studies:

Comparisons within populations A specific medical treatment iscompared to giving no treatment, or a new treatmentis compared to the current standard treatment.

Comparisons over time The number of US highway fatalities in2000-2005 is compared to the number of US highwayfatalities in 1995-1999.

3 / 44

Comparative studies

Examples of comparative studies (continued):

Comparisons across populations The number of highway fatalitiesper kilometer driven in the US is compared to thenumber of highway fatalities per kilometer driven inCanada.

Comparisons across levels of a continuous factor Rates of homeforeclosure are compared based on the amount ofdownpayment paid.

4 / 44

Non-comparative studies

A non-comparative study typically aims to estimate some quantity,e.g. the proportion of US children without health insurance – butthe estimate is not immediately compared to anything.

5 / 44

Terminology

The characteristic being compared (i.e. successful treatmentresponse, traffic fatalities, home foreclosure) is called the outcomevariable (sometimes the response variable or dependent variable).

Sometimes the treatment group or factor is called the independentvariable.

Any additional variables measured at or before the time oftreatment are called covariates. Any variables measured aftertreatment are outcomes.

6 / 44

Goal of a comparative study

Comparative studies are useful for identifying associations, e.g.:

I People with persistent anxiety who took medication A had agreater reduction in anxiety symptoms than people who tookmedication B.

I Countries with higher traffic density on their roads havehigher traffic fatality rates per kilometer driven than countrieswith lower traffic density.

I The rate of home foreclosure among people who paid a higherdownpayment on their home is lower than among people whopaid a lower downpayment.

Only certain types of comparative studies can be used to identifyfactors that cause changes in the outcome.

7 / 44

Observational studies

A comparative study is observational unless treatment groupassignments are imposed by researchers following a study design orprotocol.

Here are some examples of observational studies:

I People who took either generic or brand name Tylenol arecompared in terms of their pain relief.

I Counties in eastern Iowa that that received either high or lowrainfall in a given year are compared in terms of their cornyields.

I Third grade students who followed either a phonics or awhole-language based reading curriculum are compared interms of their standardized reading test scores.

8 / 44

Controlled Experiments

A controlled experiment is a type of comparative study that aimsto isolate the effect of a single factor, by minimizing or eliminatingthe possible effects of all other factors.

One type of controlled experiment is a randomized controlledexperiment, in which units are randomly assigned to treatmentgroups.

For example, in order to determine whether a new chemotherapydrug A has a greater effect than the current standard drug B, wecould randomly assign patients to be treated by one or the other ofthe two drugs and compare the response rates.

We will discuss other forms of control later.

9 / 44

Causality

When analyzing a comparative study, many people want to drawcausal conclusions. A causal conclusion makes a statement aboutwhat would be expected to happen under an intervention.

For example, based on observed associations, the followinginterventions might be proposed:

I If anti-anxiety medication A appears to be more effective thananti-anxiety medication B, a national medical organizationmay call for doctors to preferentially prescribe medication A.

I If higher traffic density is associated with higher trafficfatalities, a national government might propose to build moreroads.

I If smaller downpayments on home loans is associated withgreater default rates, government regulators may encouragelenders to require greater downpayments.

10 / 44

Causality

For an intervention to perform as expected, it is necessary that theassociation used to justify the intervention be causal.

If anti-anxiety medication A causes a reduction in anxietysymptoms, or if traffic density causes traffic fatalities, or if smallhome downpayments cause loan defaults, these interventions willbe successful at reducing the occurrence of something bad(anxiety, traffic fatalities, foreclosures).

However for many observed associations, the variation in outcomesis wholly or partly caused by different factor than the treatmentcalled a confounder.

11 / 44

ConfoundersA confounder (or lurking variable) is a variable that is associatedwith both treatment group status and the outcome variable.

Suppose we observe an association between treatment group statusand outcomes. If confounders are present, we cannot determinewhether the treatment causes changes in the outcome, since it isalso possible that any of the confounders causes the changes.

Examples:

I Suppose anti-anxiety medication A produces a side effect thatpeople with the most severe anxiety are least able to tolerate,leading some of them to switch to medication B.

I Suppose traffic density is negatively associated withenforcement of traffic laws.

I Suppose companies that allow low loan downpayments tendto operate in states with strong laws protecting assets fromcreditors.

12 / 44

Confounders

In each of the three examples, it may be the confounder ratherthan the treatment effect that causes changes in the outcome:

I Encouraging physicians to preferentially prescribe medicationA may not lead to an overall reduction in the number ofpeople with persistent anxiety (it may even make it higher ifsome people choose to be untreated after experiencing theside effects of medication A).

I Building more roads may not reduce traffic fatalities (it mayeven make things worse if it makes enforcing traffic laws moredifficult or expensive).

I Requiring higher home downpayments may not reduce homeloan defaults.

13 / 44

Randomization and balance

Randomization guarantees that potential confounders arestatistically independent of treatment group assignment.

Balance means that a potential confounder has identical sampledistributions within all treatment groups. For example, supposepatients with alcoholism are given zero, low, and high doses of adrug to reduce craving for alcohol. The treatment groups would bebalanced for smoking status (a potential confounder) if thefraction of smokers is the same in each treatment group.

The study on the left is balanced, the study on the right is not:

DoseHigh Low Zero

Smokers 30 50 20Non-smokers 30 50 20

DoseHigh Low Zero

Smokers 30 50 10Non-smokers 50 50 20

14 / 44

Randomization and balance

Randomization ensures that potential confounders are balanced onaverage across treatment groups. However especially for smallsamples, confounding factors can fail to be balanced by chance.

If we assigned treatments at random, we might get something likethis:

DoseHigh Low Zero

Smokers 33 54 19Non-smokers 27 46 21

The benefit of randomization is that we achieve approximatebalance with respect to unknown confounders.

15 / 44

Bias due to confounders

Suppose we aim to compare the mean level of an outcome variableY between treated (T = 1) and untreated (T = 0) subjects.Suppose smoking status is a confounder that also influences theoutcome (S = 1 is a smoker and S = 0 is a non-smoker).

Suppose an additive model holds, where

E (Y |T ,S) = αT + βS

and β0 = 0 for identification (otherwise we could add a constant tothe α’s and subtract the same constant from the β’s and get thesame model).

For non-smokers, the treatment effect is

E (Y |T = 1,S = 0)− E (Y |T = 0,S = 0) = α1 − α0

16 / 44

Bias due to confounders

Smokers have the same treatment effect:

E (Y |T = 1, S = 1)− E (Y |T = 0,S = 1) = α1 + β1 − (α0 + β1)

= α1 − α0.

If we don’t control for smoking, the mean response among treatedsubjects is

E (Y |T = 1) = ESE (Y |T = 1)

= P(S = 1|T = 1)E (Y |T = 1,S = 1) +

P(S = 0|T = 1)E (Y |T = 1,S = 0)

= p1(α1 + β1) + (1− p1)α1

= α1 + p1β1,

where p1 is the proportion of smokers in the treatment group.

17 / 44

Bias due to confounders

Similarly, the mean response in the untreated subjects is

E (Y |T = 0) = α0 + p0β1,

where p0 is the proportion of smokers in the control group.

The mean difference between treatment responses in a study thatdoes not control for smoking status is therefore

E (Y |T = 1)− E (Y |T = 0) = α1 − α0 + β1(p1 − p0).

If smoking rates are not the same in the two treatment groups(p0 6= p1), the estimate of treatment response is biased.

18 / 44

Counterfactuals and control groups

The ideal comparison for a medical treatment is a counterfactual,meaning the outcomes for a set of treated people are compared tothe outcomes the same people would have had, had they not beentreated.

In this ideal experiment, there can be no confounders.

In practice a unit cannot be both treated and untreated. Thus toestimate the treatment effect we must compare treated units tountreated units that were most like the treated unit at baseline(and similarly for untreated units).

19 / 44

Known/unknown and measured/unmeasured confounders

Known confounders are potential confounding factors that can beidentified, for example, a medical treatment may have differentrates of success in women and men, or in people with differentdiets.

An unknown confounder is something that we have not recognizedas being a potential confounding variable.

Measured confounders are potential confounding factors that wecan measure either perfectly (e.g. gender) or partially (e.g. diet).

All unknown confounders are unmeasured, but a known confoundermay be impossible to measure in practice, e.g. if we are looking atcancer incidence in older people it is very unlikely that we wouldhave any useful information about their mothers’ diets duringpregnancy.

20 / 44

Balanced randomization

In small or moderate-sized samples, confounders may beunbalanced by chance even if randomization is performed.

We can force known confounders to be balanced by using balancedrandomization – we separate our sample into strata based on thevalue of the confounder, then randomly assign the same proportionof units to the treatment group within each stratum.

This is closely connected to the idea of stratified sampling – we areobtaining two proportionate stratified samples, one for thetreatment group, and one for the control group.

21 / 44

Balanced randomization

Example

Suppose we have Ns smokers and Nns non-smokers in a study, andwe want to perform a balanced randomization in which proportionp of the smokers and proportion p of the non-smokers will betreated. Our cell counts should be as follows (rounded to integers):

Cancer No cancer

Smokers pNs (1− p)Ns

Non-smokers pNns (1− p)Nns

The ratio of smokers to non-smokers in the cancer group ispNs/(pNns) = Ns/Nns and the ratio of smokers to non-smokers inthe “no cancer” group is (1− p)Ns/(1− p)Nns = Ns/Nns – theratios are the same, so we have balance.

22 / 44

Balanced randomizationExample

Another simple example of balanced randomization would be ifgender was thought to be a confounder.

If we have 50 men and 50 women in our study, and we assigntreatments at random (say by flipping a coin), we will on averagehave 25 treated men and 25 treated women, but these numberswill fluctuate by chance – e.g. in a given study we may have 28treated women and 21 treated men.

Under balanced randomization, we would randomly select exactly25 men and exactly 25 women to be treated. This ensures exactbalance for gender.

Balanced randomization is useful for exactly balancing known,measured confounders. For unknown and/or unmeasuredconfounders, we must rely on randomization to provide balance onaverage (but cannot expect to achieve exact balance).

23 / 44

Strategies for analyzing observational studies

Suppose we have data from an observational study in whichknown, measured confounders were not controlled.

Is there anything we can do to provide some sense of what weshould expect from a better controlled study?

24 / 44

Common support

Suppose we have an outcome variable Y , a treatment variable T ,and a potential confounder X .

The support of X for treated subjects is the range of X values thatoccur for treated subjects (T = 1). Similarly, the support of X foruntreated subjects is the range of X values that occurs foruntreated subjects (T = 0).

If the supports of X for T = 0 and T = 1 are identical, the studyis perfectly balanced for X and we do not need to worry about Xbeing a confounder.

But if the study did not control for X , it is unlikely that thesupports are identical.

25 / 44

Common support

As an example, suppose we are interested in estimating the effectof a special program for high risk pre-schoolers. It is hoped that theprogram improves the childrens’ reading scores in the third grade.

Suppose the program is made available to all children in aparticular neighborhood. Some of the children are enrolled in theprogram and some are not by their parents’ choice. Enrollment inthe program is the treatment variable for this analysis.

Another major factor that is associated with the reading score isthe income of the child’s family.

26 / 44

Common support

It is unlikely that the treatment and non-treatment groups will beperfectly balanced for family income. For example, the two incomedistributions may look like this:

5 10 15 20 25 30Income/1000

0

20

40

60

80

100

120

140Fr

equency

UntreatedTreated

In this case, the common support is from around $14,000 to$21,000, and more than half the sample lies in the commonsupport.

27 / 44

Common support

The following case is more extreme:

0 5 10 15 20 25 30 35Income/1000

0

20

40

60

80

100

120

140

Frequency

UntreatedTreated

There is a small region of common support, containing only around10% of the sample.

28 / 44

Common support

If the common support is small, little can be done to reliablyprotect the analysis from strong biases.

In the first example, where there is a substantial common support,several options are available to us.

29 / 44

Stratification

Stratification can be used to protect against biases in observationalstudies when there is substantial common support for a potentialconfounder.

The details vary depending on the particular statistical analysisbeing used (e.g. Z-test, χ2 test, . . .), but there is a basic strategythat is followed.

Continuing with the previous example, suppose we intend tocopmpare the mean test scores in the treated and untreatedsamples using a Z-test.

If we are worried about a confounding effect of income, instead ofdoing one overall test as usual, we could divide the sample into 5strata based on parent income. Within each income stratum, theincome values are relatively similar, so the potential forconfounding is minimized.

30 / 44

Stratification

Say we cut the income into 20% slices based on the marginalincome distribution. Then we can estimate the treatment effectD1, . . . ,D5 for the five slices separately, using

Di = Y(i)T − Y

(i)U ,

where Y(i)T and Y

(i)U are the means of treated responses, and of

untreated responses, in the i th stratum.

We can then form a weighted average

D = w1D1 + · · ·+ w5D5,

where w1 + · · ·+ w5 = 1 are weights.

This weighted average estimates the overall treatment effect, withthe confounding effect of income mostly removed.

31 / 44

StratificationTo use the statified mean D for statistical inference, we need todetermine its variance.

Let Vi = var(Di ) denote the variances for the stratum-levelestimates. We can then obtain

var(D) = w21V1 + · · ·+ w2

5V5.

This allows us to form an overall Z-score

D/√

var(D)

to test whether the treatment effect is nonzero, and to formconfidence intervals

D ± 2√var(D)

for the population treatment effect.32 / 44

Stratification

Since Di is the difference between two sample means, its variance is

Vi = σ2(i)T /n

(i)T + σ

2(i)U /n

(i)U ,

where σ2(i)T , and n

(i)T are the sample variance and sample size for

the treated cases in the i th stratum, and σ2(i)U and n

(i)U are the

sample variance and sample size for the untreated cases in the i th

stratum.

33 / 44

Stratification

However we choose the weights wi , the bias due to confoundingwill be mostly removed.

But we also want to have the least variance in D (which gives thegreatest power, etc.).

These weights are commonly used:

wi = V−1i /

∑j

V−1j .

Exercise: Using calculus, and focusing on the case of two strata forsimplicity, show that this formula for wi minimizes the variance ofD.

34 / 44

Stratification

Here is what we get for a simulated data set with true treatmenteffect 0.4, and using wi = V−1

i /∑

j V−1j .

nT nU Y(i)T Y

(i)U D

8 112 1.14 1.28 -0.1334 86 1.79 1.58 0.2270 50 1.95 1.63 0.3286 34 2.32 2.00 0.32

102 18 2.58 1.65 0.93

The overall estimated treatment effect is 0.38 with a Z-statistic of3.9. The unstratified treatment effect estimate and Z-statistic are0.71 and 8.1, respectively. Thus the treatment is clearly beneficialeven after adjusting for parent income, but not nearly as stronglyas it appeared to be when we did not adjust for the confoundingeffect of income.

35 / 44

MatchingSuppose Yi is the outcome for a treated unit (so Ti = 1). In anideal comparative study, we would compare Yi to the outcome for aunit that is identical to it in every way, except for being untreated.

In practice, we can only work with measured covariates that arepotential confounders. Suppose we have two such covariates, X1

and X2. We can assess the similarity between treated case i anduntreated case j using

Dij =√

(X1i − X1j)2 + (X2i − X2j)2.

The matching estimate of the treatment effect is formed byselecting a number M of matches to use, and then forming theaverage outcome of the M untreated units that are closest totreated unit i . Denote this average by Y ∗

i .

Similarly, for an untreated case j we can form the average outcomefor the M closest treated cases, denoted by Y ∗

j .36 / 44

Matching

Matching each treated unit to the three closest control units (left),match each control unit to the three closest treated units (right):

22 26 30 34 38Parent income ($1000)

10

11

12

13

14

Pare

nt

educa

tion (

years

)

TreatedControl

22 26 30 34 38Parent income ($1000)

10

11

12

13

14

Pare

nt

educa

tion (

years

)

TreatedControl

37 / 44

Matching

The mathcing estimate of the treatment effect is:

D = (∑Ti=1

Yi − Y ∗i +

∑Ti=0

Y ∗i − Yi )/(nT + nU).

It is possible to get an analytic formula for the variance of D, butwe won’t get into that here.

38 / 44

Balance and representativeness

All research studies aim to generalize from the sample of analyzedunits to the population from which the units were drawn.

In order to generalize we should study a representative sample –one that is reasonably similar to a random sample from thepopulation of interest.

It is important to be aware that if extreme selection is used tocreate balance, it may be at the expense of representativeness.

For example, in the study of the effectiveness of the pre-schoolprogram, suppose our population of interest is 50% Hispanic. Wecan create a balanced sample that has no Hispanic children ineither the treatment or control group, but we cannot be confidentthat our findings would generalize to a population that contains asubstantial fraction of Hispanic children.

39 / 44

Sensitivity analysis

Earlier we looked at an example where the mean outcome is

E (Y |T ,S) = αT + βS ,

where αT (T = 0, 1) is the treatment effect and β1 is the effectdue to smoking, which acts as a confounder. We saw that if wedon’t control for smoking, the estimated treatment effect isexpected to be

α1 − α0 + β1(p1 − p0),

where p1, p0 are the proportions of smokers in the treated anduntreated groups, respectively.

Suppose we carry out such an analysis and obtain a Z-score of 4,giving a two-sided p-value of 6.3× 10−5.

What would it take for confounding to reduce this Z-score to anuninteresting value (one that is below 2 in magnitude)?

40 / 44

Sensitivity analysisSuppose we use a two-sample Z-test to compare the treated anduntreated samples:

√n

YT − YU√σ2T/qT + σ2U/qU

where n is the total sample size, YT ,YU are the sample meanoutcomes for treated and untreated subjects, nT , nU are thenumbers of treated and untreated subjects, and qT = nT/n,qU = nU/n are the proportions of treated and untreated subjectsin the sample.

Let

s2 ≡ σ2T/qT + σ2U/qU ,

and treat this as a fixed quantity.41 / 44

Sensitivity analysisIf we further assume that σT = σU = σ, then

s2 = σ2(1/qT + 1/qU) = σ2f ,

where f = 1/qT + 1/qU .

Thus the Z-score becomes

√n(YT − YU)/(σ

√f ).

Above we saw that E (YT − YU) = α1 − α0 + β1(p1 − p0), thus

√n(α1 − α0 + β1(p1 − p0))/(σ

√f ) =

√n(α1 − α0)/(σ

√f ) +

√nβ1(p1 − p0)/(σ

√f )

≈ 4.

42 / 44

Sensitivity analysisThe Z-score based on the direct treatment effect is√n(α1 − α0)/(σ

√f ). For this to be greater than 2, we must have

the Z-score contribution from confounding be less than 2:

√nβ1(p1 − p0)/(σ

√f ) < 2,

so

(β1/σ)(p1 − p0) ≈ 2√

f /n.

The value of β1/σ is the amount that smoking changes theexpected response in SD units. We might have some idea aboutthe plausible size of this and some of the other quantities. Forexample, suppose n = 100 and f = 4 (corresponding to equalnumbers of treated and untreated subjects). Then

(β1/σ)(p1 − p0) ≈ 0.4. 43 / 44

Sensitivity analysis

Since |p1 − p0| < 1, we must have β1/σ > 0.4 (assuming that thedirection β1 ≥ 0 is known, which is typical). This is already a largeeffect, and if |p1 − p0| � 1 which is expected, the effect wouldhave to be even larger.

Thus it seems unlikely that confounding by smoking could reduceour Z-score to an insignificant value in this example.

44 / 44