53
Economics 105: Statistics Any questions? No GH due Friday. For next couple classes, please read first 4 sections of Chapter 13 and Freakonomics, Chapter 5 (copy is in P:\economics\Eco 105 (Statistics) Foley\ freakonomics Ch_5.pdf)

Economics 105: Statistics

  • Upload
    nuru

  • View
    48

  • Download
    2

Embed Size (px)

DESCRIPTION

Economics 105: Statistics. Any questions? No GH due Friday. For next couple classes, please r ead first 4 sections of Chapter 13 and Freakonomics , Chapter 5 (copy is in P:\economics\Eco 105 (Statistics) Foley\ freakonomics Ch_5.pdf). Brief Introduction to Research Design. - PowerPoint PPT Presentation

Citation preview

Page 1: Economics 105: Statistics

Economics 105: Statistics• Any questions?• No GH due Friday. For next couple classes, please read first 4 sections of Chapter 13 and Freakonomics, Chapter 5 (copy is in P:\economics\Eco 105 (Statistics) Foley\freakonomics Ch_5.pdf)

Page 2: Economics 105: Statistics

Brief Introduction to Research Design

Design Notation

Internal Validity

Experimental Design

Page 3: Economics 105: Statistics

Design Notation• Observations or measures are indicated with an “O”• Treatments or programs with an “X”• Groups are shown by the number of rows• Assignment to group is by “R,N,C”

– Random assignment to groups– Nonequivalent assignment to groups– Cutoff assignment to groups

• Time

Page 4: Economics 105: Statistics

Design Notation Example

R O1,2 X O1,2

R O1,2 O1,2

Os indicate differentwaves of

measurement.

Vertical alignmentof Os shows that

pretest and posttestare measured at same time.

X is the treatment.There are twolines, one foreach group.

R indicates the groups

are randomly assigned.

Subscriptsindicate

subsets ofmeasures.

Page 5: Economics 105: Statistics

Types of DesignsRandom assignment?

Control group or multiple measures?

No

Yes

Yes

Randomized(true experiment)

Quasi-experiment

No

Nonexperiment

Page 6: Economics 105: Statistics

Non-Experimental Designs

X O

O X O

X O O

Post-test only (case study)

Single-group, pre-test, post-test

Two-group, post-test only(static group comparison)

Page 7: Economics 105: Statistics

Experimental DesignsR O1 X O1,2

R O1 O1,2

R X O1,2

R O1,2

• Pretest-Posttest Randomized Experiment Design

• If continuous measures, use t-test

• If categorical outcome, use chi-squared test

• Posttest only Randomized Experiment Design

• Less common due to lack of pretest

• Probabilistic equivalence between groups

Page 8: Economics 105: Statistics

Experimental DesignsR O X OR O OR X OR O

• Advantages• Information is available on the effect of treatment

(independent variable), the effect of pretesting alone, possible interaction of pretesting & treatment, and the effectiveness of randomization

• Disadvantages• Costly and more complex to implement

Solomon Four-Group Design

Page 9: Economics 105: Statistics

Establishing Cause and Effect

Single-Group Threats

Multiple-Group Threats

“Social” Interaction Threats

• Internal validity is the approximate truth about inferences regarding cause-effect relationships.

• “Internal” means internal to the study, not “external”, that is, not talking about generalizing the results yet.

Internal Validity

Page 10: Economics 105: Statistics

Threats to Internal ValidityR X

OR

OHistory

MaturationTesting

InstrumentationMortality

Regression to the meanSelection

Selection-historySelection- maturation

Selection- testingSelection- instrumentation

Selection- mortality*Selection- regressionDiffusion or imitation*

Compensatory equalization*Compensatory rivalry*

Resentful demoralization*

Single-Group

Multiple-Group

Social Interaction

Page 11: Economics 105: Statistics

Single-Group Threatsto Internal Validity

Page 12: Economics 105: Statistics

Administerprogram

Measureoutcomes

X O

Two designs:

Administerprogram

Measureoutcomes

X O

Measurebaseline

O

Post-test only a single group

What is a “single-group” threat?

Page 13: Economics 105: Statistics

• Diabetes educational program for newly diagnosed adolescents in a clinic

• Pre-post, single group design• Measures (O) are paper-pencil, standardized

tests of diabetes knowledge (e.g. disease characteristics, management strategies)

Example

Page 14: Economics 105: Statistics

• Any other event that occurs between pretest and posttest

• For example, adolescents learn about diabetes by watching The Health Channel

Program Posttest

X O

Pretest

O

History Threat

Page 15: Economics 105: Statistics

• Normal growth between pretest and posttest.• They would have learned these concepts anyway,

even without program.

Program Posttest

X O

Pretest

O

Maturation Threat

Page 16: Economics 105: Statistics

• The effect on the posttest of taking the pretest• May have “primed” the kids or they may have

learned from the test, not the program• Can only occur in a pre-post design

Program Posttest

X O

Pretest

O

Testing Threat

Page 17: Economics 105: Statistics

• Any change in the test from pretest and posttest• So outcome changes could be due to different

forms of the test, not due to program• May do this to control for “testing” threat, but

may introduce “instrumentation” threat

Program Posttest

X O

Pretest

O

Instrumentation Threat

Page 18: Economics 105: Statistics

• Nonrandom dropout between pretest and posttest• For example, kids “challenged” out of program by

parents or clinicians• Attrition

Program Posttest

X O

Pretest

O

Mortality Threat

Page 19: Economics 105: Statistics

• Group is a nonrandom subgroup of population.• For example, mostly low literacy kids will appear

to improve because of regression to the mean.• Example: height

Program Posttest

X O

Pretest

O

Regression Threat

Page 20: Economics 105: Statistics

When you select a sample from

the low end of a distribution ...

the group will do better on a

subsequent measure.

The group mean on the first measure

appears to “regress toward the mean” of

the population.

Selectedgroup’smean

Overallmean

Regression to the mean

Overallmean

Regression to the Meanpre-test scores ~ N

post-test scores ~ N & assuming no effect of treatment pgm

Page 21: Economics 105: Statistics

Regression to the Mean

Page 22: Economics 105: Statistics
Page 23: Economics 105: Statistics

Regression to the Mean• How to Reduce the effects of RTM

(Barnett, et al., International Journal of Epidemiology, 2005)

1. When designing the study, randomly assign subjects to treatment and control (placebo) groups. Then effects of RTM on responses should be same across groups.

2. Select subjects based on multiple measurements

• RTM increases with larger variance (see graphs) so subjects can be selected using the average of 2 or more baseline measurements.

Page 24: Economics 105: Statistics

Multiple-Group Threats to Internal Validity

Page 25: Economics 105: Statistics

• When you move from single to multiple group research the big concern is whether the groups are comparable.

• Usually this has to do with how you assign units (e.g., persons) to the groups (or select them into groups).

• We call this issue selection or selection bias.

The Central Issue

Page 26: Economics 105: Statistics

Administerprogram

Measureoutcomes

Measurebaseline

Alternativeexplanations

Alternativeexplanations

X OOOO

Do not administerprogram

Measureoutcomes

Measurebaseline

The Multiple Group Case

Page 27: Economics 105: Statistics

• Diabetes education for adolescents

• Pre-post comparison group design

• Measures (O) are standardized tests of diabetes knowledge

Example

Page 28: Economics 105: Statistics

• Any other event that occurs between pretest and posttest that the groups experience differently.

• For example, kids in one group pick up more diabetes concepts because they watch a special show on Oprah related to diabetes.

X OOOO

Selection-History Threat

Page 29: Economics 105: Statistics

• Differential rates of normal growth between pretest and posttest for the groups.

• They are learning at different rates, even without program.

X OOOO

Selection-Maturation Threat

Page 30: Economics 105: Statistics

• Differential effect on the posttest of taking the pretest.

• The test may have “primed” the kids differently in each group or they may have learned differentially from the test, not the program.

X OOOO

Selection-Testing Threat

Page 31: Economics 105: Statistics

• Any differential change in the test used for each group from pretest and posttest

• For example, change due to different forms of test being given differentially to each group, not due to program

X OOOO

Selection-Instrumentation Threat

Page 32: Economics 105: Statistics

• Differential nonrandom dropout between pretest and posttest.

• For example, kids drop out of the study at different rates for each group.

• Differential attrition

X OOOO

Selection-Mortality Threat

Page 33: Economics 105: Statistics

• Different rates of regression to the mean because groups differ in extremity.

• For example, program kids are disproportionately lower scorers and consequently have greater regression to the mean.

X OOOO

Selection-Regression Threat

Page 34: Economics 105: Statistics

“Social Interaction” Threats to Internal Validity

Page 35: Economics 105: Statistics

• All are related to social pressures in the research context, which can lead to posttest differences that are not directly caused by the treatment itself.

• Most of these can be minimized by isolating the two groups from each other, but this leads to other problems (for example, hard to randomly assign and then isolate, or may reduce generalizability).

What Are “Social” Threats?

Page 36: Economics 105: Statistics

• Controls might learn about the treatment from treated people (for example, kids in the diabetes educational group and control group share the same hospital cafeteria and talk with one another).

Diffusion or Imitation of Treatment

Page 37: Economics 105: Statistics

• Administrators give a compensating treatment to controls.

• Researchers feel badly and give control group kids a video to watch pertaining to diabetes. Contaminates the study!

=

Compensatory Equalization of Treatment

Page 38: Economics 105: Statistics

• Controls compete to keep up with treatment group.

Compensatory Rivalry

Page 39: Economics 105: Statistics

• Controls "give up" or get discouraged

• Likely to exaggerate the posttest differences, making your program look more effective than it really is

Resentful Demoralization

Page 40: Economics 105: Statistics

What is a Clinical Trial?• “A prospective study comparing the effect and

value of intervention(s) against a control in human beings.”

• Prospective means “over time”; vs. retrospective• It is attempting to change the natural course of a

disease• It is NOT a study of people who are on drug X

versus people who are not

• http://www.clinicaltrials.gov/info/resources

Page 41: Economics 105: Statistics

Model of Two-Group Randomized Clinical Trial

Page 42: Economics 105: Statistics

What are the characteristics of a Clinical Trial?• Begins with a primary research question, and the trial

design flows from this question (constrained by practicalities)

• Everything must be exhaustively defined in advance (to prevent accusations of fishing for a positive finding)

• The hypothesis (“-es”)• Population to be studied• inclusion criteria• exclusion criteria• contraindications to therapy• indications to therapy• Treatment strategy (treatment, exact dosage, dosage

schedule, etc)• The outcome(s)

Page 43: Economics 105: Statistics

Beta-Blocker Heart Attack Trial (BHAT)• Published in Journal of the American Medical AssociationJAMA 1982; 247: 1701 - 1714JAMA 1983; 250: 2814 – 2819• Up until about 25 years ago, the treatment of myocardial

infarction consisted of bed rest, alleviation of symptomatic pain, possible administration of early antiarrhythmics

• But a third of people who have a heart attack die from it ‘suddenly’

• In 1976, NIH sponsored a conference to discuss potential agents to be used in either a primary or secondary prevention setting to reduce sudden death, for which there was no treatment.

• The conference made an official recommendation to do a clinical trial.

Page 44: Economics 105: Statistics

Beta-Blocker Heart Attack Trial (BHAT)

• Primary Research Question• To test in a multicenter, randomized, double-blind,

placebo, controlled trial, whether the daily administration of propranolol to patients who had had at least one documented MI would results in a significant reduction in all-cause mortality during 2 to 4 years of follow-up (expected mean follow-up = 36 months).

Page 45: Economics 105: Statistics

Beta-Blocker Heart Attack Trial (BHAT)• Inclusion criteria

• Men/Women• Aged 30 to 69 yrs• Documented (defined) MI within 5 to 21 days of

randomization• Exclusion criteria

• Contraindication to propranolol (e.g., asthma, severe bradycardia)

• Likely to be prescribed propranolol (e.g., for severe angina)

• Unlikely to be a compliant participant• Likely to die of noncardiac cause (e.g., cancer)

• What do these do to generalizability?

Page 46: Economics 105: Statistics

BHAT Design and Conduct

1916 Patients - Propranolol

Randomized 3,837 Participants

Screened 16,400 Patients

Time

Treat and Collect Follow-up Data

1921 Patients - Placebo

Follow-up Time

Mean 2 yrs (trial stopped early)

138 Deaths

188 Deaths

Page 47: Economics 105: Statistics

Beta-Blocker Heart Attack Trial (BHAT)• Results

• BHAT (and similar trials) demonstrated great benefitin reducing all-cause mortality and cardiac-specific mortality (including sudden death)in three-quarters of Post-MI Patients (1/4 had contraindication to propranolol)

• Relevance today?• Beta-blockers still should be given post-MI• What happened after BHAT is illustrative of what often

happens a clinical trial result is published • Results reported in 1981 (short report in JAMA)

In 1987, only 36% of post-MI patients on a beta-blocker In 1989, 40% In 1992, 63% In 1993, only 33% of post-MI women

Page 48: Economics 105: Statistics

Example: Job Corps• What is Job Corps? http://jobcorps.doleta.gov/

• January 5, 2006 Thursday Late Edition – Final

SECTION: Section C; Column 1; Business/Financial Desk; ECONOMIC SCENE; Pg. 3

HEADLINE: New (and Sometimes Conflicting) Data on the Value to Society of the Job Corps

BYLINE: By Alan B. Krueger.

Alan B. Krueger is the Bendheim professor of economics and public affairs at Princeton University. His Web site is www.krueger.princeton.edu.

He delivered the 2005 Cornelson Lecture in the Department of Economics here at Davidson (that’s the big econ lecture each year).

Page 49: Economics 105: Statistics

Example: Job Corps• Quotations from “New (and Sometimes Conflicting) Data on the Value

to Society of the Job Corps” by Alan B. Krueger.

• Since 1993, Mathematica Policy Research Inc. has evaluated the performance of the Job Corps for the Department of Labor.

• Its evaluation is based on one of the most rigorous research designs ever used for a government program. From late 1994 to December 1995, some 9,409 applicants to the Job Corps were randomly selected to be admitted to the program and another 6,000 were randomly selected for a control group that was excluded from the Job Corps.

• Those admitted to the program had a lower crime rate, higher literacy scores and higher earnings than the control group.

Page 50: Economics 105: Statistics

RCT for Credit Card Offers

Source: Agarwal, et al. (2010), Journal of Money, Credit & Banking, 42 (4)

Page 51: Economics 105: Statistics

RCT for Education in India

Source: Banerjee, et al. (2007), Quarterly Journal of Economics

Page 52: Economics 105: Statistics

RCT for Education in India

Source: Agarwal, et al. (2010), Journal of Money, Credit & Banking, 42 (4)