Further Remarks on Causality and Structural Equations ...jenni.uchicago.edu/econ312/Slides/HB1-Econ-Eval_STATIC_2019-05-15a_sjs.pdfPolicy Evaluation The essential contrast between

Policy Evaluation

Further Remarks on Causality and Structural

Equations: Excerpt from Econometric

Evaluation of Social Programs Part I

James J. Heckman and Edward J. Vytlacil

Econ 312, Spring 2019

Heckman and Vytlacil Econometric Evaluation

Policy Evaluation

Policy Evaluation

• Evaluating policy is a central problem in economics.

• Requires economist to construct counterfactuals.


Policy Evaluation

• A model of counterfactuals is more widely accepted the morewidely accepted are its ingredients:

(1) the rules used to derive a model, including whether or not therules of logic and mathematics are followed;

(2) its agreement with other theories; and

(3) its agreement with the evidence.

• Models are of hypothetical worlds obtained byvarying — hypothetically — the factors determining outcomes.

• Frisch: “Causality is in the mind.”


Policy Evaluation

• Epidemiological and statistical models are incomplete: they donot specify the sources of randomness generatingvariability among agents.

• They do not specify why observationally identical people makedifferent choices and have different outcomes given the samechoice.

• They do not distinguish what is in the agent’s information setfrom what is in the observing statistician’s information set.

• This distinction is fundamental in justifying the properties ofany estimator for solving selection and evaluation problems.


Policy Evaluation

• They are also incomplete because they are recursive.

• They do not allow for simultaneity in choices of outcomes oftreatment that are at the heart of game theory and models ofsocial interactions.


Policy Evaluation

• The goal of the econometric literature, like the goal of allscience, is to model phenomena at a deeper level, to understandthe causes producing the effects so that one can use empiricalversions of the models to forecast the effects of interventionsnever previously experienced, to calculate a variety of policycounterfactuals, and to use economic theory to guide thechoices of estimators and the interpretation of the evidence.

• These activities require development of a more elaborate theorythan is envisioned in the current literature on causal inferencein epidemiology and statistics.


Policy Evaluation

• The recent literature sometimes contrasts structural and causalmodels.

• The contrast is not sharp because the term “structural model”is often not precisely defined.

• There are multiple meanings for this term.


Policy Evaluation

• The essential contrast between causal models and expliciteconomic models as currently formulated is in the range ofquestions that they are designed to answer.

• Causal models as formulated in statistics and in theeconometric treatment effect literature are typically black-boxdevices designed to investigate the impact of“treatment” — which are often complex packages ofinterventions — on some observed set of outcomes in a givenenvironment.


Policy Evaluation

• Explicit economic models go into the black box to explore themechanism(s) producing the effects: the causes of effects .

• Holland (1986): understanding the “effects of causes” (thegoal of the treatment effect literature).

• Understanding the “causes of effects” (the goal of theliterature building explicit economic models).

• By focusing on one narrow black-box question, the treatmenteffect and natural experiment literatures avoid many of theproblems confronted in the econometrics literature that buildsexplicit economic models.


Policy Evaluation

• At the same time, it produces parameters that are more limitedin application.


Policy Evaluation

• By not being explicit about the contents of the black-box(understanding the causes of effects), it ties its hands in usinginformation about basic behavioral parameters obtained fromother studies as well as economic intuition to supplementavailable information in the data in hand (limits abduction).

• Lacks the ability to provide explanations for estimated “effects”grounded in economics or to conduct welfare economics.

• When the components of treatments vary across studies,knowledge does not accumulate across treatment effect studies,whereas it does accumulate across studies estimating commonbehavioral or technological parameters.


Policy Evaluation

Policy Evaluation: Notation and Definition of IndividualLevel Treatment Effects

• To evaluate is to value and to compare values amongpossible outcomes.

• Two distinct tasks.

• We define outcomes corresponding to state (policy, treatment)s for an agent characterized by ω as Y (s, ω), ω ∈ Ω.

• The agent can be a household, a firm, or a country.


Policy Evaluation

• One can think of Ω as a universe of agents each characterizedby an element ω.

• The ω encompasses all features of agents that affect Youtcomes.

• Y (·, ·) may be generated from a scientific or economic theory.

• It may be vector valued.

• The Y (s, ω) are outcomes realized after treatments are chosen.

• In advance of treatment, agents will not know the Y (s, ω) butmay make forecasts about them.


Policy Evaluation

• Let S be the set of possible treatments with elements denotedby s.

• For simplicity, assume that this set is the same for all ω.

• For each ω, we obtain a collection of possible outcomes givenby Y (s, ω)s∈S .

• The set S may be finite (e.g., there may be J states),countable, or may be defined on the continuum (e.g.,S = [0, 1]), so that there are an uncountable number of states.


Policy Evaluation

• For example, if S = 0, 1, there are two treatments, one ofwhich may be a no-treatment state — e.g., Y (0, ω).

• This is the outcome for an agent ω not getting a treatment likea drug, schooling, or access to a new technology, while Y (1, ω)is the outcome in treatment state 1 for agent ω getting thedrug, schooling, or access.


Policy Evaluation

• A prototypical policy evaluation problem.

• Country can adopt a policy (e.g., democracy).

• Choice Indicator:• D = 1 if it adopts.• D = 0 if not.


Policy Evaluation

• Two outcomes (Y0(ω),Y1(ω)), ω ∈ Ω

• Y0(ω) if country does not adopt• Y1(ω) if country adopts

• Marshallian ceteris paribus causal effect:

Y1(ω)− Y0(ω)


Policy Evaluation

Figure 1: Extended Roy economy for policy adoptionDistribution of gains and treatment parameters

-5 -4 -3 -2 -1 0 1 2 3 4 50

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

Gain

TT=2.517

AM C=1.5* ATE=0.2

TUT=-0.595

Figure 1. Extended Roy Economy for Policy Adoption Distribution of Gains and Treatment Parameters

*C = Marginal Return

Suppose that a country has to choose whether to implement a policy. Under the policy, the GDP would be Y1.Without the policy,the GDP of the country would be Y0. For sake of simplicity, suppose that

Y1 = μ1 + U1

Y0 = μ0 + U0

where U0 and U1 are unobserved components of the aggregate output. The error terms (U0, U1) are dependent in a general way. Letδ denote the additional GDP due to the policy, i.e. δ = μ1−μ0. We assume δ > 0. Let C denote the cost of implementing the policy.We assume that the cost is a fixed parameter C. We relax this assumption below. The country’s decision can be represented as:

D =

½1 if Y1 − Y0 − C > 00 if Y1 − Y0 − C ≤ 0,

so the country decides to implement the policy (D = 1) if the net gains coming from it are positive. Therefore, we can define theprobabily of adopting the policy in terms of the propensity score

Pr(D = 1) = P (Y1 − Y0 − C > 0)

We assume that (U1, U0) ∼ N (0,Σ), Σ =∙1 −0.5−0.5 1

¸, μ0 = 0.67, δ = 0.2 and C = 1.5.

Gain=Y1 − Y0


Policy Evaluation

Figure 1 Legend

Suppose that a country has to choose whether to implement a policy. Under thepolicy, the GDP would be Y1. Without the policy, the GDP of the countrywould be Y0. For the sake of simplicity, suppose that

Y1 = µ1 + U1

Y0 = µ0 + U0

where U0 and U1 are unobserved components of the aggregate output. Theerror terms (U0,U1) are dependent in a general way. Let δ denote the additionalGDP due to the policy, i.e. δ = µ1 − µ0. We assume δ > 0. Let C denote thecost of implementing the policy. We assume that the cost is a fixed parameterC .


Policy Evaluation

Figure 1 Legend

The country’s decision can be represented as an Extended RoyModel:

D =

1 if Y1 − Y0 − C > 00 if Y1 − Y0 − C ≤ 0,

so the country decides to implement the policy (D = 1) if the netgains coming from it are positive. Therefore, we can define theprobability of adopting the policy in terms of the “propensity score”or probability of selection:

Pr(D = 1) = P(Y1 − Y0 − C > 0).

Assume that (U1,U0) ∼ N (0,Σ), Σ =

[1 −0.5−0.5 1

], µ0 = 0.67,

δ = 0.2, and C = 1.5.


Policy Evaluation

• More generally, define outcomes corresponding to state (policy,treatment) s for an “agent” characterized by ω as Y (s, ω),ω ∈ Ω = [0, 1], s ∈ S, set of possible treatments.

• The agent can be any economic agent such as a household, afirm, or a country.

• The Y (s, ω) are ex post outcomes realized after treatments arechosen.


Policy Evaluation

• The individual treatment effect for agent ω.

Y (s, ω)− Y (s ′, ω) , s 6= s ′, s, s ′ ∈ S , (1)

Individual level causal effect.

• Comparisons can also be made in terms of utilities R (Y (s, ω)).

• R (Y (s, ω) , ω) > R (Y (s ′, ω) , ω) if s is preferred to s ′.

• The difference in subjective outcomes is[R (Y (s, ω) , ω)− R (Y (s ′, ω) , ω)].

• Another possible definition of a treatment effect.

• Holding ω fixed holds all features of the person fixed except thetreatment assigned, s.


Policy Evaluation

• An essential question,

“What question is the analysis supposed to answer?”

is often the big unanswered question in the recent policyevaluation literature.

• The question is usually unanswered because it is unasked inmuch of the modern treatment effect literature which seeks toestimate “an effect” without telling you which effect or why itis interesting to know it.


Policy Evaluation

Adding Uncertainty

• The Y (s, ω) are outcomes realized after treatments are chosen(ex post).

• In advance of treatment, agents may not know the Y (s, ω) butmay make forecasts about them.

• These forecasts may influence their decisions to participate inthe program or may influence the agents who make decisionsabout whether or not an individual participates in the program.


Policy Evaluation

Treatment May Be Complex

• Each “state” (treatment) may consist of a compound ofsubcomponent states.

• In this case, one can define s itself as a vector (e.g.,s = (s1, s2, . . . , sK ) for K components) corresponding to thedifferent components that comprise treatment.

• Thus a job training program typically consists of a package oftreatments.

• We might be interested in the package of one (or more) of itscomponents.


Policy Evaluation

• The outcomes may be time subscripted as well, Yt (s, ω)corresponding to outcomes of treatment measured at differenttimes.

• The index set for t may be the integers, corresponding todiscrete time, or an interval, corresponding to continuous time.

• The Yt (s, ω) are realized or ex post (after treatment)outcomes.


Policy Evaluation

What Exactly Is Treatment?

• The term “treatment” is used in multiple ways in this literatureand this ambiguity is sometimes a source of confusion.

• In its most common usage:

• A treatment assignment mechanism is a rule τ : Ω→ Swhich assigns treatment to each ω.

• The consequences of the assignment are the outcomes Y (s, ω),s ∈ S, ω ∈ Ω.

• The collection of possible assignment rules is T where τ ∈ T .

• Two aspects of a policy.

• The policy selects who gets what.

• More precisely, it selects individuals ω and specifies thetreatment s ∈ S received.


Policy Evaluation

• More nuanced definition of treatment assignment that explicitlyrecognizes the element of choice by agent ω in producing thetreatment assignment rule.

• Participation in treatment is usually a choice made by agents.

• Under a more comprehensive definition of treatment, agents areassigned incentives like taxes, subsidies, endowments andeligibility that affect their choices, but the agent chooses thetreatment selected.

• Agent preferences, program delivery systems, aggregateproduction technologies, market structures, and the like mightall affect the choice of treatment.

• The treatment choice mechanism may involve multiple actorsand multiple decisions that result in an assignment of ω to s.


Policy Evaluation

• For example, s can be schooling while Y (s, ω) is earnings givenschooling for agent ω.

• A policy may be a set of payments that encourage schooling, asin the Progressa program in Mexico, and the treatment in thatcase is choice of schooling with its consequences for earnings.


Policy Evaluation

• Specify assignment rules a ∈ A: map individuals ω intoconstraints (benefits) b ∈ B under different mechanisms.

• A constraint assignment mechanism a is a map

a : Ω→ B

defined over the space of agents.

• The constraints may include endowments, eligibility, taxes,subsidies and the like that affect agent choices of treatment.

• The map a defines the rule used to assign b ∈ B.

• Formally, the probability system for the model withoutrandomization is (Ω, σ (Ω) ,F) where Ω is the probabilityspace, σ (Ω) is the σ-algebra associated with Ω and F is themeasure on the space.


Policy Evaluation

• When we account for randomization we need to extend Ω toΩ′ = Ω×Ψ, where Ψ is the new probability space induced bythe randomization, and we define a system (Ω′, σ (Ω′) ,F ′).


Policy Evaluation

• Some policies may have the same overall effect on theaggregate distribution of b, but may treat given individualsdifferently.

• Under an anonymity postulate, some would judge such policiesas equivalent in terms of the constraints (benefits) offered,even though associated outcomes for individuals and aggregatesmay be different.

• Another definition of equivalent policies is in terms of thedistribution of aggregate outcomes associated with thetreatments.

• This lecture characterizes policies at the individual levelrecognizing that sets of A that are characterized by someaggregate distribution over elements of b ∈ B may be whatothers mean by a policy.


Policy Evaluation

• Given b ∈ B allocated by constraint assignment mechanisma ∈ A, agents pick treatments.

• Treatment assignment mechanism: τ : Ω×A× B → S.

• Map taking agent ω ∈ Ω facing constraints b ∈ B assigned bymechanism a ∈ A into a treatment s ∈ S.

• Note: including B in the domain of definition of τ is redundantsince the map a : Ω→ B selects an element b ∈ B.

• I make b explicit to remind the reader that agents are makingchoices under constraints.

• In settings with choice, τ is the choice rule used by agentswhere τ ∈ T , a set of possible choice rules.

• It is conventional to assume a unique τ ∈ T is selected by therelevant decision makers.

• Not required in our definition.


Policy Evaluation

• A policy regime p ∈ P is a pair (a, τ) ∈ A× T that mapsagents denoted by ω into elements of s.

• In this notation, P = A× T .


Policy Evaluation

• Incorporating choice into the analysis of treatment effects is anessential and distinctive ingredient of the econometric approachto the evaluation of social programs.

• The traditional treatment-control analysis in statistics equatesmechanisms a and τ (e.g., “Rubin model”).

• An assignment in that literature is an assignment to treatment,not an assignment of incentives and eligibility for treatmentwith the agent making treatment choices.

• This approach has only one assignment mechanism and treatsnoncompliance with it as a problem rather than as a source ofinformation on agent preferences, as in the econometricapproach.


Policy Evaluation

• Under “full compliance”, a : Ω→ S and a = τ , whereB = S.


Policy Evaluation

• Policy invariance is a key assumption for the study of policyevaluation.

• It allows analysts to characterize outcomes without specifyinghow those outcomes are obtained.

• Policy invariance has two aspects.

• The first aspect is that, for a given b ∈ B (incentive schedule),the mechanism a ∈ A by which ω is assigned a b (e.g. randomassignment, coercion at the point of a gun, etc.) and theincentive b ∈ B are assumed to be irrelevant for the values ofrealized outcomes for each s that is selected.

• Second: for a given s for agent ω, the mechanism τ by whichassigns ω to s under assignment mechanism a ∈ A is irrelevantfor the values assumed by realized outcomes.


Policy Evaluation

• Both assumptions define what we mean by policy invariance.


Policy Evaluation

• Policy invariance allows us to describe outcomes by Y (s, ω)and ignore features of the policy and choice environment indefining outcomes.

• If have to account for the effects of incentives and assignmentmechanisms on outcomes, must work with Y (s, ω, a, b, τ)instead of Y (s, ω).

• The following policy invariance assumptions justify collapsingthese arguments of Y (·) down: Y (s, ω).


Policy Evaluation

• Policy invariance for objective outcomes:

PI-1 1

For any two constraint assignment mechanisms a, a′ ∈ A andincentives b, b′ ∈ B, with a(ω) = b and a′(ω) = b′, and for allω ∈ Ω, Y (s, ω, a, b, τ) = Y (s, ω, a′, b′, τ), for alls ∈ Sτ(a,b)(ω) ∩ Sτ(a′,b′)(ω) for assignment rule τ where Sτ(a,b)(ω) isthe image set for τ (a, b).

• For simplicity assume Sτ(a,b)(ω) = Sτ(a,b) for all ω ∈ Ω.


Policy Evaluation

• This assumption says that for the same treatment s and agentω, different constraint assignment mechanisms a and a′ andassociated constraint assignments b and b′ produce the sameoutcome.

• “Many ways to skin a cat.”


Policy Evaluation

• A second invariance assumption invoked in the literature:

• For a fixed a and b, the outcomes are the same independent ofthe treatment assignment mechanism:

PI-2 2

For each constraint assignment a ∈ A, b ∈ B and all ω ∈ Ω,Y (s, ω, a, b, τ) = Y (s, ω, a, b, τ ′) for all τ and τ ′ ∈ T withs ∈ Sτ(a,b) ∩ Sτ ′(a,b), where Sτ(a,b) is the image set of τ for a givenpair (a, b).

• Again, exclude the possibility of ω-specific image sets Sτ(a,b)and Sτ ′(a,b).


Policy Evaluation

• If treatment effects based on subjective evaluations are alsoconsidered, we need to broaden invariance assumptions 1 and 2to produce invariance in rewards for certain policies andassignment mechanisms.

• It would be unreasonable to claim that utilities R (·) do notrespond to incentives.

• Suppose, instead, that we examine subsets of constraintassignment mechanisms a ∈ A that give the same incentives(elements b ∈ B) to agents, but are conferred by differentdelivery systems, a.


Policy Evaluation

• For each ω ∈ Ω, define the set of mechanisms delivering thesame incentive or constraint b as Ab (ω):

Ab (ω) = a | a ∈ A, a(ω) = b , ω ∈ Ω.

The set of delivery mechanisms that deliver b may vary amongthe ω.

• Let R (s, ω, a, b, τ) represent the reward to agent ω from atreatment s with incentive b allocated by mechanism a with anassignment to treatment mechanism τ .


Policy Evaluation

PI-3 3

For any two constraint assignment mechanisms a, a′ ∈ A andincentives b, b′ ∈ B with a(ω) = b and a′(ω) = b′, and for allω ∈ Ω, Y (s, ω, a, b, τ) = Y (s, ω, a′, b′, τ) for alls ∈ Sτ(a,b)(ω) ∩ Sτ(a′,b′)(ω) for assignment rule τ , where Sτ(a,b)(ω)is the image set of τ (a, b) and for simplicity we assume thatSτ(a,b)(ω) = Sτ(a,b) for all ω ∈ Ω. In addition, for any mechanismsa, a′ ∈ Ab (ω), producing the same b ∈ B under the same conditionspostulated in the preceding sentence, and for all ω,R (s, ω, a, b, τ) = R (s, ω, a′, b, τ) .


Policy Evaluation

• This assumption says, for example, that utilities are notaffected by randomization or the mechanism of assignment ofconstraints.

• Corresponding to 2 we have a policy invariance assumption forthe utilities with respect to the mechanism of assignment:


Policy Evaluation

PI-4 4

For each pair (a, b) and all ω ∈ Ω,

Y (s, ω, a, b, τ) = Y (s, ω, a, b, τ ′)

R (s, ω, a, b, τ) = R (s, ω, a, b, τ ′)

for all τ, τ ′ ∈ T and s ∈ Sτ(a,b) ∩ Sτ ′(a,b).


Policy Evaluation

• This assumption rules out general equilibrium effects, socialexternalities in consumption, etc. in both subjective andobjective outcomes.


Documents

Further Remarks on Causality and Structural Equations ...jenni.uchicago.edu/econ312/Slides/HB1-Econ-Eval_STATIC_2019-05-15a_sjs.pdfPolicy Evaluation The essential contrast between