2014 04 28 and 30

Ch. 13: Experiments and

Quasi-Experiments

Econ 141 Spring 2014

Lecture: April 28 & 30, 2014

Bart Hobijn

4/28&30/2014 Econ 141, Spring 2014 1

The views expressed in these lecture notes are solely those of the instructor and do not necessarily

reflect those of the UC Berkeley, or other institutions with which he is affiliated.

Sources of exogenous variation

Chapter 12 showed that successful identification of causation requires exogenous variation in the

explanatory variable in the regression.

Ch.13 introduces two types of such variation

1. Experiments

Situation in which exogenous variation is explicitly

generated by the researcher.

2. Quasi-experiments

Situation in which particular circumstances generated

exogenous variation in the data.

Practical examples of experiments and quasi experiments.

4/28&30/2014 Econ 141, Spring 2014 2

Experiment setup

Estimate causal effect of treatment on

outcome (Jargon comes from medical sciences)

Potential outcome Outcome, , for an individual under potential treatment, . Causal effect is difference in outcome if treatment is

received and when it is not received.

Average treatment (causal) effect

Average causal effect across population of interest 4/28&30/2014 Econ 141, Spring 2014 3

Difference estimator

Regression model

= 0 + 1 + +

outcome

treatment

1 average treatment effect Parameter that we want to consistently estimate

control variables

effect of control variables not necessarily consistently estimated

residual of the regression 4/28&30/2014 Econ 141, Spring 2014 4

Randomized controlled experiment

Researcher controls whether an individual is subjected to treatment.

Treatment can be randomly assigned to participants in experiment.

Fully random or

Based on observables to better approximate population.

Treatment assigned to assure exogenous variation in .

4/28&30/2014 Econ 141, Spring 2014 5

Quasi controlled experiment

Whether an individual is subjected to treatment or not is not determined by the

researchers.

Particular circumstances resulted in random assignment of treatment to participants in

experiment.

(Part of) variation in treatment across individuals exogenous.

4/28&30/2014 Econ 141, Spring 2014 6

Five studies as examples

Experiment

Duration Dependence and Labor Market Conditions: Evidence from a Field Experiment

Kroft, Lange, and Notowidigdo (QJE, 2013)

Quasi-experiment

Household Expenditure and the Income Tax Rebates of 2001 Johnson, Parker, and Souleles (AER, 2006)

Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime

Levitt (AER, 1996), comment by McCrary (AER, 2002)

Minimum Wages and Employment: A Case Study of the Fast Food Industry in New Jersey and Pennsylvania

Card and Krueger (AER, 1994)

Did Securitization Lead to Lax Screening? Evidence from Subprime Loans.

Keys, Mukherjee, Seru, and Vig (QJE, 2010)

4/28&30/2014 Econ 141, Spring 2014 7

Focus on same points across papers

1. Research question

2. Regression specification

3. Potential threats to internal validation

4. Instrument/experiment to resolve threats

5. OLS and IV results

6. Hypothesis tests

7. Interpretation

4/28&30/2014 Econ 141, Spring 2014 8

Experiment

Duration Dependence and Labor Market Conditions: Evidence from a Field Experiment Kroft, Lange, and Notowidigdo (QJE, 2013)

4/28&30/2014 Econ 141, Spring 2014 9

Duration dependence of finding job

4/28&30/2014 Econ 141, Spring 2014 10

Causes of duration dependence

Sample selection / unobserved heterogeneity The best workers get hired first and contribute to high job-

finding rate of unemployed at short durations. The pool of

unemployed workers at longer durations just consist of ones

that are less likely to get hired.

Pure duration dependence The longer an individual worker remains unemployed the

less likely she or he is to find a job in subsequent months.

4/28&30/2014 Econ 141, Spring 2014 11

Research question

Sample selection / unobserved heterogeneity The best workers get hired first and contribute to high job-

finding rate of unemployed at short durations. The pool of

unemployed workers at longer durations just consist of ones

that are less likely to get hired.

Pure duration dependence The longer an individual worker remains unemployed the

less likely she or he is to find a job in subsequent months.

Do potential employers hold the time someone has been

out of a job against them and are they less likely to call

back those who have been unemployed longer for an

interview when they apply for a job?

4/28&30/2014 Econ 141, Spring 2014 12

Regression specification

Linear probability model

, = 0 + 1, + 2 log , + , + ,

individual, city (MSA) person lives in

, = 1 if gets called back for interview after applying for a job. Is 0 otherwise

, Dummy if employed at other job at time of application.

, Duration of unemployment if not employed at other job.

, Set of conditioning variables

4/28&30/2014 Econ 141, Spring 2014 13

Regression specification


, = 0 + 1, + 2 log , + , + ,






4/28&30/2014 Econ 141, Spring 2014 14

Research question

boils down to asking

whether < or not.

Why not apply OLS?


, = 0 + 1, + 2 log , + , + ,






4/28&30/2014 Econ 141, Spring 2014 15

Unobserved Heterogeneity:

Workers with undesirable unobserved

characteristics will be unemployed

longer and be less likely to be called

back for an interview when they apply

for a job.

cov , , , < 0

Threat to internal validation

Experiment is a solution


, = 0 + 1, + 2 log , + , + ,






4/28&30/2014 Econ 141, Spring 2014 16

Send out a set of, otherwise identical,

resumes to apply for job openings where the

only difference is the treatments , and log .

Estimate whether there is a significant

difference in call back rates as a function of

the duration of unemployment, .

Description of treatments

4/28&30/2014 Econ 141, Spring 2014 17

Description of sample

4/28&30/2014 Econ 141, Spring 2014 18

Table describes the

composition of the

sample of 12,054

resumes that

researchers sent

out, the call back

rate they got, and

the types of jobs

they applied for.

Show treatment is randomly assigned

4/28&30/2014 Econ 141, Spring 2014 19

Experiment is a solution


, = 0 + 1, + 2 log , + , + ,






4/28&30/2014 Econ 141, Spring 2014 20

Because this is an experiment we have

generated all variation in , and , as exogenously.

Result is that we can estimate equation using

OLS for the data we obtain from the

experiment.

Table with main result

4/28&30/2014 Econ 141, Spring 2014 21

Table with main result

4/28&30/2014 Econ 141, Spring 2014 22

Figure illustrates the point

4/28&30/2014 Econ 141, Spring 2014 23

Interpretation

Very convincing evidence that employers are less likely to call back persons who have

been unemployed for more than 6 moths for

an interview when they apply for a job.

Call back rate drops by about 3 percentage points.

4/28&30/2014 Econ 141, Spring 2014 24

Quasi-Experiment

Household Expenditure and the Income Tax Rebates of 2001

Johnson, Parker, and Souleles (AER, 2006)

4/28&30/2014 Econ 141, Spring 2014 25

Research question

Permanent Income Hypothesis (PIH) the hypothesis states that a change in permanent income, rather than a change in temporary income, affects the choices

that determine a consumer's consumption patterns. The key

conclusion of this theory is that transitory, temporary changes

in income have little effect on consumer spending behavior,

whereas permanent changes can have large effects on

consumer spending behavior.

4/28&30/2014 Econ 141, Spring 2014 26

Research question

Permanent Income Hypothesis (PIH) the hypothesis states that a change in permanent income, rather than a change in temporary income, affects the choices

that determine a consumer's consumption patterns. The key

conclusion of this theory is that transitory, temporary changes

in income have little effect on consumer spending behavior,

whereas permanent changes can have large effects on

consumer spending behavior.

Did consumers change their consumption level upon receipt of

the one-time 2001 federal income tax rebates?

4/28&30/2014 Econ 141, Spring 2014 27

Regression equation

,+1 , = 0,,

12

=1

+ 1 + 2,+1 + ,+1

, Household s consumption in month

, Month dummy for which month time falls in

Set of conditioning variables

,+1 Size of the one-time tax rebate a household r receives at time + 1.

4/28&30/2014 Econ 141, Spring 2014 28

Permanent Income Hypothesis

,+1 , = 0,,

12

=1

+ 1 + 2,+1 + ,+1





4/28&30/2014 Econ 141, Spring 2014 29

Permanent Income Hypothesis

For households that satisfy the PIH the one-time

tax rebate which affects their current income but

barely their permanent income it should be the

case that they do not change their consumption

in response to the tax rebate. That is, the PIH

implies the null hypothesis : = .

Why OLS might be inconsistent

,+1 , = 0,,

12

=1

+ 1 + 2,+1 + ,+1





4/28&30/2014 Econ 141, Spring 2014 30

Observed and unobserved characteristics that affect the

level of consumption, like the level of different types of

income, a households preference for leisure, etc., might not only affect the level of consumption (and the size of

its change) but also the size of the tax rebate a household

receives.

If that is the case then cov ,+1, ,+1 0

and OLS not consistent.

Timing of Rebate Instrumental Variable

Timing of month in which households received the tax rebate depended on their social security

number.

This choice of timing meant random timing of rebate across households.

Let ,+1 = 1 if household received tax

rebate in month + 1 and 0 otherwise.

,+1 is positively correlated with ,+1 (valid)

and random across households (exogenous).

4/28&30/2014 Econ 141, Spring 2014 31

Table with main results from OLS

4/28&30/2014 Econ 141, Spring 2014 32

Reduced form regression

4/28&30/2014 Econ 141, Spring 2014 33

Putting instrument

in as regressor in IV regression rather

than fitted values of

endogenous

variables is called reduced form

regression.

It always has a

better fit than the IV

regression

Table with main results from OLS

4/28&30/2014 Econ 141, Spring 2014 34

Note the footnote to this table

4/28&30/2014 Econ 141, Spring 2014 35

PIH rejected

4/28&30/2014 Econ 141, Spring 2014 36

OLS and 2SLS give similar results

4/28&30/2014 Econ 141, Spring 2014 37

Test whether OLS

estimates are valid

using Hausman test

Problem set

Additional results in paper

Estimate the timing of the spending response with respect to the rebate.

Show that it is mainly low income households and those with a low level of liquid assets

whose consumption responds most to the tax

rebates.

Replication files with STATA do file to generate the results are on bSpace.

Problem 3 on Problem set on Chapters 12 and 13.

4/28&30/2014 Econ 141, Spring 2014 38

Interpretation

Spending of a large fraction of U.S. households responds to temporary changes

in income.

Important for understanding

Permanent income hypothesis

Importance of liquidity constraints

Aggregate demand effect of temporary fiscal stimulus.

4/28&30/2014 Econ 141, Spring 2014 39

Quasi-Experiment

Using Electoral Cycles in Police Hiring to Estimate the Effect of Police

on Crime

Levitt (AER, 1996), comment by McCrary (AER, 2002)

4/28&30/2014 Econ 141, Spring 2014 40

Crime rates peaked in early 90s

4/28&30/2014 Econ 141, Spring 2014 41

Research question

Does putting more cops on the street reduce

the crime rate in a city?

4/28&30/2014 Econ 141, Spring 2014 42

Regression equation

4/28&30/2014 Econ 141, Spring 2014 43

Why OLS might be a problem

4/28&30/2014 Econ 141, Spring 2014 44

Cities are likely to increase the size of their police force

as a direct response to an increase in crime.

So cov ln , > .

OLS inconsistent and likely to understate the degree to

which increasing the police force reduces crime.

Election cycles also drive changes

in police force Timing of Mayoral and Gubernatorial

elections is determined by law and not

affected by changes in crime rates.

Use election-driven changes in size of police forces as exogenous variation in ln to identify causal effect of size of police force on

crime.

4/28&30/2014 Econ 141, Spring 2014 45

1st stage regression to show relevance

4/28&30/2014 Econ 141, Spring 2014 46

First-stage regression

Regress endogenous variable on instruments

and all the explanatory variables that are also

in the second-stage regression.


4/28&30/2014 Econ 141, Spring 2014 47


4/28&30/2014 Econ 141, Spring 2014 48


4/28&30/2014 Econ 141, Spring 2014 49

Columns (4) and (5) reduced forms

4/28&30/2014 Econ 141, Spring 2014 50

Columns (4) and (5) reduced forms

4/28&30/2014 Econ 141, Spring 2014 51

2nd stage regression:

police reduces crime

4/28&30/2014 Econ 141, Spring 2014 52

2nd stage regression:

police reduces crime

4/28&30/2014 Econ 141, Spring 2014 53

Levitt did weighted least squares

4/28&30/2014 Econ 141, Spring 2014 54

But he reversed the weights

4/28&30/2014 Econ 141, Spring 2014 55

McCrary (2002) points out that Levitt made a programming error and gave unreliable

observations more weight rather than less.

Main results not robust to correction of this error.

McCrarys correction table

4/28&30/2014 Econ 141, Spring 2014 56

Heteroskedasticity matters!

4/28&30/2014 Econ 141, Spring 2014 57

Quasi-experiment

Minimum Wages and Employment: A Case Study of the Fast Food Industry in New Jersey and

Pennsylvania

Card and Krueger (AER, 1994)

4/28&30/2014 Econ 141, Spring 2014 58

Minimum wage debate continues

WASHINGTONPresident Barack Obama's proposal Tuesday to raise the federal minimum wage is likely to

rekindle debates over whether the measure helps or hurts

low-income workers.

White House officials say the move to boost the wage to

$9 an hour, from $7.25, is aimed at addressing poverty

and helping low-income Americans.

But the proposal likely will be opposed by Republicans

and business groups, which have traditionally said raising

the minimum wage discourages companies from hiring

low-skilled workers.

Wall Street Journal

4/28&30/2014 Econ 141, Spring 2014 59

Research question

How does raising the minimum wage affect

labor demand (employment levels)?

Quasi experiment On April 1, 1992, New Jersey's minimum wage rose from $4.25 to $5.05 per hour. To evaluate the impact of the law we surveyed 410fast-food

restaurants in New Jersey and eastern Pennsylvania before and after the

rise. Comparison of employment growth at stores in New Jersey and

Pennsylvania (where the minimum wage was constant) provide simple

estimates of the effect of the higher minimum wage.

4/28&30/2014 Econ 141, Spring 2014 60

Regression equation

4/28&30/2014 Econ 141, Spring 2014 61

Regression equation

4/28&30/2014 Econ 141, Spring 2014 62

Sample of fast food restaurants

4/28&30/2014 Econ 141, Spring 2014 63

Fast food restaurants in study

4/28&30/2014 Econ 141, Spring 2014 64

NJ & PA restaurants similar

4/28&30/2014 Econ 141, Spring 2014 65

NJ & PA restaurants similar

4/28&30/2014 Econ 141, Spring 2014 66

Pre-treatment wages similar

4/28&30/2014 Econ 141, Spring 2014 67

Post-treatment wages different

4/28&30/2014 Econ 141, Spring 2014 68

Raising minimum wage

increased employment?

4/28&30/2014 Econ 141, Spring 2014 69

Puzzling result questioned

Internal validity:

Did NJ restaurants already adjust their hiring before the first wave of survey?

Was there another NJ vs PA state-specific shock/policy that affected relative demand for fast

food?

Were restaurants sampled in both states really similar?

External validity:

How does NJ/PA fast food to U.S. labor market?

4/28&30/2014 Econ 141, Spring 2014 70

Quasi-experiment

Did Securitization Lead to Lax Screening? Evidence from Subprime Loans.

Keys, Mukherjee, Seru, and Vig (QJE, 2010)

4/28&30/2014 Econ 141, Spring 2014 71

Mortgage crisis

2008 financial crisis was largely due to default

(delinquency) rates on mortgages that were

part of mortgage backed securities increasing.

Research question:

Did financial institutions lower their standards

on the mortgages they issued when they knew

they could offload them into mortgage backed

securities?

4/28&30/2014 Econ 141, Spring 2014 72

Regression discontinuity

Discontinuity is treatment:

Loans with a credit score above 620 were

much easier securitized and offloaded.

Research question

Was there more scrutiny on loans with a score

just below 620 than on loans just above?

Regression discontinuity analysis

4/28&30/2014 Econ 141, Spring 2014 73

Regression discontinuity

Discontinuity is treatment:

Loans with a credit score above 620 were

much easier securitized and offloaded.

Research question

Was there more scrutiny on loans with a score

just below 620 than on loans just above?

Regression discontinuity analysis

4/28&30/2014 Econ 141, Spring 2014 74

Degree of scrutiny determined by amount of

background documentation on loan applicant

and on loan

Low documentation

vs

Full documentation

4/28&30/2014 Econ 141, Spring 2014 75

Regression discontinuity equation

4/28&30/2014 Econ 141, Spring 2014 76

Picture says more than regression

4/28&30/2014 Econ 141, Spring 2014 77

Regression confirms picture

4/28&30/2014 Econ 141, Spring 2014 78

Interpretation

Size of discontinuity suggests that financial firms were less diligent with loans they could

more easily offload and securitize.

For loans they were more likely to hold on their own book they required more

documentation.

4/28&30/2014 Econ 141, Spring 2014 79

Where to go after this class

This class taught main statistical theory behind

regression analysis. Logical topics beyond what

was covered are

Alternative estimation methods Method of Moments, Generalized Method of Moments, Maximum Likelihood, Bayesian

Estimation.

Time series analysis Observations correlated over time

Extended panel data techniques

Find applications Labor Economics, Industrial Organization, Macroeconomics, Development Economics, ect.

4/28&30/2014 Econ 141, Spring 2014 80

Documents

2014 04 28 and 30