Game Theory and Empirics: What Can Go Wrong

Using Formal Theory to Identify and Characterize Biases

• James Fearon, “Signaling versus the Balance of Power and Interests”

• Kenneth Schultz, “Looking for Audience Costs”

• Alastair Smith, “To Intervene or Not to Intervene: A Biased Decision”

“Signaling versus the Balance of Power and Interests”

• Question: How do observable indicators of the balance of interests and capabilities influence the success of immediate extended deterrence?

• Intuition: The more the balance of interests and the balance of capabilities favor the defender, the more likely the defender’s deterrent threat will succeed.

• Findings: – Some indicators of defender interests (alliance, contiguity)

predict immediate deterrence failure.

– Past deterrent threat also predicts failure.

– Inconsistent evidence on balance of capabilities.

The Theoretical Model

Prior beliefs:

[ , ]

[ ,0]C C

D

w U j a

w U k

, Audience costs

, Value of stakes

, Value of war

C D

C D

C D

r r

a a

w w

The Huth Deterrence Data

• Population: 58 cases of extended immediate deterrence– A challenger has made a threat against a protégé– A defender has made a deterrent threat to protect

the protégé

• Data set records– Success or failure: Did challenger back down in

the face of the defender’s threat?– Fight or not fight: If challenger did not back down,

did defender make good on its threat?

Relationship of Data to Model

Immediatedeterrence

Possible Selection Effects

• The decision to challenge is conditioned on– the challenger’s type

• unobserved by analysts and the defender

– the challenger’s prior beliefs about the defender’s type, esp. aD

• Observed, at best, with error by analysts• common knowledge to participants

– all other payoffs, including the stakes for the challenger, aC

• observed with error by analysts• common knowledge to participants

Balance of Interests

Hypothesis: Observable indicators of the defender’s interest in the protégé (which should increase aD) will predict general deterrence success but immediate deterrence failure.

Rationale: • The challenger will only threaten a high-value

protégé if it is highly motivated—i.e., aC is high. • As aD increases, probability of challenge goes down,

but probability that challenger will act given a threat and mobilization goes up.

A Note on BiasIs the estimated effect of alliance ties on deterrence

outcomes biased?• It is an accurate measure of the empirical association

between these variables.• It is biased estimate of the effect of alliance ties on the

defender’s payoffs.– Table 6 seems to suggest that alliance ties decrease the

credibility of the defender’s threat by decreasing aD or wD.– With the model, we see that, if alliance ties increase those

parameters, the challenger’s self-selection on aC can reverse the sign in the observed relationship conditional on a challenge.

– The bias arises because aC is an omitted variable that (conditional on a threat) is positively correlated with alliance ties but negatively correlated with deterrence success.

Past Deterrent Threats

Hypothesis: A past deterrence crisis between the two states decreases the probability of immediate deterrence success.

Rationale: • If the defender made a previous deterrent threat or

was previously a challenger, its type distribution going into the current crisis will have higher mean than if there was no prior crisis.

• As before, the more resolved the defender is initially expected to be, the more resolved the challenger must be in order to make a threat.

The Balance of Capabilities

Hypothesis: The more the ex ante balance of capabilities favors the defender (increasing wD, decreasing wC), the more likely it is that a deterrent threat will succeed.

Rationale:• Weak challengers only threaten strong defenders

when they have a high value for the issue at stake. This should predict immediate deterrence failure.

• But: Weak challengers threaten on matters that they initially believe the defender does not care about. This means that a deterrent threat should lead to updating and immediate deterrence success.

Evidence

• Deterrence success more likely when– short-run and immediate balance of capabilities favors

defender– major power defender vs. minor power challenger– defender possesses nuclear weapons (post 1945)

• Informal evidence that challenges in the face of large power asymmetries are “limited probes” on issues thought to be peripheral for stronger power.

• What we would like to see: Evidence that observable indicators of defender interests are negatively correlated with defender power advantage

Summary

• The fact that challengers are selecting into crises based on variables that we cannot directly observe (esp., aC, aD) creates selection effects that complicate making inferences about – the effects of observable variables on game payoffs

– whether our framework for understanding these kinds of interactions is correct

• This paper uses intuition from a formal model to think about the likely direction of the biases induced.

“Looking for Audience Costs”• Questions:

– Do state leaders incur domestic “audience costs” for backing down after making a threat?

– Do democratic leaders incur higher costs than non-democratic leaders?

• Possible testing strategies– Indirect: Look for a correlation between variables thought

to increase audience costs, e.g., democracy, and outcomes associated with higher audience costs, e.g., acquiescence by targets (Schultz 1999).

– Direct: Look for whether leaders who backed down in a crisis face a higher risk of removal than those who did not.

• Can you make inferences about the distribution of audience costs (mean, covariates) by examining the sample of costs that are actually incurred?

The Theoretical Model

C

C

T

T

(1,0)

(0,1)

(– ac ,1)

(1, – at ) (– wc , – wt )

ChallengeStatus Quo

Concede Resist

Back Down

Back Down

Stand Firm

Stand Firm

ac, at known

~ [0, ] with c.d.f. i iw F

Equilibrium Strategies

wc

wt

kc bc

kt

bt

CH/SFRS/SF

War

CH/BDRS/SF

ChallengerBacks Down

SQ/BDRS/SF

Status Quo

CH/SFRS/BDTarget

Backs Down

CH/BDRS/BD

ChallengerBacks Down

SQ/BDRS/BD

Status Quo

CH/SFCD/BDTarget

Concedes

CH/BDCD/BDTarget

Concedes

SQ/BDCD/BD

Status Quo

Challenger’saudience costsobserved

Target’saudience costsobserved

Strategic Censoring

1. bt is always decreasing in ac probability of resistance decreases with the audience costs generated by the challenge

2. kc is nondecreasing in ac probability that the challenger stands firm is nondecreasing in the audience costs it generated

Therefore, increase in ac lowers the probability that ac is actually incurred.

wc

wt

kc bc

kt

bt

Challenger’saudience costsobserved

Target’saudience costsobserved

Increasing the Challenger’s Costs

wc

wt

kc bc

kt

bt Challenger’saudience costsobserved

Target’saudience costsobserved

Increasing the Challenger’s Costs

Strategic Censoring

3. kc is nonincreasing in at probability that the challenger stands firm is nonincreasing in the audience costs generated by the target

4. kt is increasing at probability that the target stands firm is increasing in the audience costs it generated.

Therefore, increase in at lowers the probability that at is actually incurred.

wc

wt

kc bc

kt

bt

Challenger’saudience costsobserved

Target’saudience costsobserved

Increasing the Target’s Costs

wc

wt

kc bc

kt

bt

Challenger’saudience costsobserved

Target’saudience costsobserved

Increasing the Target’s Costs

A Monte Carlo Demonstration

Assume audience costs are generated as follows:

ccccc DEMa ln

ttttt DEMa ln

ccccc DEMw ln

ttttt DEMw ln

2with , ~ (0, ) and known to participants.c t N

Assume war costs are generated as follows:

2with , ~ (0, ) and private information.c t N

A Monte Carlo Demonstration

1. Assume c = t = 0, c = t = 1, = 1• audience costs same for targets and challengers• ln a for democracies ~ N(1,1)• ln a for non-democracies ~ N(0,1)

2. Generate 100 data sets of simulated data• each set contains 50,000 dyad observations• each state has 30 percent chance of being democratic• for each observation, generate shocks and determine

equilibrium outcome from the model

3. Compare summary statistics of actual and incurred audience costs (logged).

Population Mean Std. Deviation

Full population

All 0.30 1.10

Democracies 1.00 1.00

Nondemocracies 0.00 1.00

Challengers that back down

All -0.33 0.89

Democracies 0.21 0.78

Nondemocracies -0.46 0.86

Targets that back down

All -0.52 0.85

Democracies -0.018 0.71

Nondemocracies -0.62 0.84

Summary

• Efforts to find direct evidence of audience costs hampered by– partial observability: we can only measure them when they

are incurred

– strategic selection: the probability that they are incurred is a function of their value

• These problems tend to – lower the mean of observed audience costs

– attenuate the effects of covariates on audience costs

• What is to be done?

“To Intervene or Not to Intervene: A Biased Decision”

• Questions:– Do states condition their behavior in crises on the

expectation that allies will join in the event of war?

– How reliable are alliance?

• Fact: Previous research finds that states come to aid of allies in war only about 25 percent of the time.

• If crisis behavior is conditioned on expectations of alliance reliability, then our estimates of reliability and its covariates can be biased.

The Theoretical Model A

AttackStatus Quo

B

C (an ally of B)

Acquiesce Resist

HonorAlliance

Stay Out

SQ

ACQ

INT BI-WAR

Modeling Reliability

• C’s decision to intervene:

• Can we estimate c from observational data on whether C intervenes conditional on a war between A and B?

*

1 if * 0

0 if * 0

c c c c c c

cc

c

y X e Z e

yy

y

Sources of Selection Bias

1. B’s decision to resist is conditioned on C’s probability of honoring the alliance

• B is more likely to resist if C is more likely to intervene• If B has information about C that we cannot control for,

then this will bias upward our estimate of C’s reliability

2. A’s decision to attack is conditioned on C’s probability of honoring the alliance

• A is less likely to attack if C is more likely to intervene• If A has information about C that we cannot control for,

then this will bias downward our estimate of C’s reliability

How B’s Decision Introduces Bias

• B’s decision to resist:

• B’s decision to resist is conditioned on C’s reliability if

,1 ,2*

1 if * 0

0 if * 0

b b b c b b b b

bb

b

y X Z e Z e

yy

y

,2 0, or

( , ) 0b

b cC e e

Note: Does this make sense?

How B’s Decision Introduces Bias

• If B resists,

• If then

• Hence, in the sample of cases in which B has resisted, ec is not mean zero nor uncorrelated with Xc.

( )( )

( )b

c b bb

ZE e e Z

Z

( , ) (0,0, )b ce e BN

* 0b b by e Z

A Test for Bias

• B does not condition on C if

• Smith shows that, if these conditions are met, then the distribution of Xc is expected to be the same for all outcomes of the game.

• Hence, we can determine the distribution of reliability in the sample of cases that became wars, and test the null hypothesis that the distribution of reliability is the same in the sample of SQ and ACQ outcomes.

,2 0, and

( , ) 0b

b cC e e

Data

• Observations on 1772 dyads with outcomes– SQ (no force used by either side): 975

– ACQ (force by one side only): 355

– War: 366 (Bigwar: 156)

• For each war, every ally of every participant in the war is an observation– Dependent variable: honor alliance or not

– Independent variables: age of alliance, type of alliance, regime type, capabilities of C, expected utility indicators

ˆ( ) ( )c ch BC X Alliance reliability:

Summary

• Countries do condition behavior on expected reliability of allies.

• Average predicted probability of intervention at least .07 higher in cases in which B resists than in cases in which it does not.

• Evidence of bias, but what direction?

• What is to be done?