74
Finding Optimal Mixed Finding Optimal Mixed Strategies to Commit to in Security Games Vincent Conitzer Departments of Computer Science and Economics Departments of Computer Science and Economics Duke University Co-authors on various parts: @Duke: Dmytro (Dima) Korzhyk, Josh Letchford, Kamesh Munagala, Ron Parr @USC: Zhengyu Yin, Chris Kiekintveld, Milind Tambe @CMU: Tuomas Sandholm

Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

  • Upload
    ngotruc

  • View
    268

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Finding Optimal MixedFinding Optimal Mixed Strategies to Commit to in g

Security GamesVincent Conitzer

Departments of Computer Science and EconomicsDepartments of Computer Science and EconomicsDuke University

Co-authors on various parts:@Duke: Dmytro (Dima) Korzhyk, Josh Letchford,

Kamesh Munagala, Ron Parr@USC: Zhengyu Yin, Chris Kiekintveld, Milind Tambe

@CMU: Tuomas Sandholm

Page 2: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to
Page 3: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

What is game theory?• Game theory studies settings where multiple parties

(agents) each have( g )– different preferences (utility functions),

– different actions that they can take– different actions that they can take

• Each agent’s utility (potentially) depends on all agents’ iactions

– What is optimal for one agent depends on what other agents do

• Very circular!

• Game theory studies how agents can rationally form y g ybeliefs over what other agents will do, and (hence) how agents should actagents should act– Useful for acting as well as predicting behavior of others

Page 4: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Penalty kick example

probability .7

probability .3

action

probability 1

Is this a action

probability .6“rational” outcome? If not, what

action

probability .4 is?

Page 5: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Rock-paper-scissorsColumn player akaColumn player aka. player 2 chooses a

column

0, 0 -1, 1 1, -11, -1 0, 0 -1, 1

Row playeraka. player 1

chooses a row , , ,-1, 1 1, -1 0, 0

c ooses a o

A row or column is , , ,called an action or

(pure) strategyRow player’s utility is always listed first, column player’s secondp y y y , p y

Zero-sum game: the utilities in each entry sum to 0 (or a constant)Three-player game would be a 3D table with 3 utilities per entry, etc.

Page 6: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Matching pennies (~penalty kick)

L R

1, -1 -1, 1L

-1, 1 1, -1R

Page 7: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

“Chicken”• Two players drive cars towards each other

• If one player goes straight that player wins• If one player goes straight, that player wins

• If both go straight, they both die

S D

D S

0 0 1 1D S

0, 0 -1, 1D not zero-sum

1, -1 -5, -5S

Page 8: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

How to play matching pennies

L RThem

1, -1 -1, 1LUs

-1, 1 1, -1RUs

• Assume opponent knows our strategy…– hopeless?

• … but we can use randomization

• If we play L 60% R 40%If we play L 60%, R 40%...

• … opponent will play R…

t 6*( 1) 4*(1) 2• … we get .6*(-1) + .4*(1) = -.2

• What’s optimal for us? What about rock-paper-scissors?

Page 9: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Matching pennies with a sensitive target

L RThem

1, -1 -1, 1LUs

-2, 2 1, -1RUs

• If we play 50% L, 50% R, opponent will attack L

– We get .5*(1) + .5*(-2) = -.5g ( ) ( )

• What if we play 55% L, 45% R?

• Opponent has choice between• Opponent has choice between

– L: gives them .55*(-1) + .45*(2) = .35

R i th 55*(1) 45*( 1) 1– R: gives them .55*(1) + .45*(-1) = .1

• We get -.35 > -.5

Page 10: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Matching pennies with a sensitive target

L RThem

1, -1 -1, 1LUs

-2, 2 1, -1RUs

• What if we play 60% L, 40% R?

• Opponent has choice betweenOpponent has choice between

– L: gives them .6*(-1) + .4*(2) = .2

R: gives them 6*(1) + 4*( 1) = 2– R: gives them .6 (1) + .4 (-1) = .2

• We get -.2 either way

• This is the maximin strategy

– Maximizes our minimum utility

Page 11: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Let’s change roles

L RThem

1, -1 -1, 1LUs

-2, 2 1, -1RUs

• Suppose we know their strategy

• If they play 50% L, 50% R, von Neumann’s minimax y p y , ,

– We play L, we get .5*(1)+.5*(-1) = 0

• If they play 40% L 60% R

theorem [1927]: maximin value = minimax value

(~LP duality)If they play 40% L, 60% R,

– If we play L, we get .4*(1)+.6*(-1) = -.2

If we play R we get 4*( 2)+ 6*(1) = 2

( y)

– If we play R, we get .4 (-2)+.6 (1) = -.2

• This is the minimax strategy

Page 12: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Minimax theorem falls apart in nonzero-sum games

D S

0 0 -1 1D

D S

0, 0 1, 11 -1 -5 -5

D

S 1, 1 5, 5S

• Let’s say we play SLet s say we play S

• Most they could hurt us is by playing S as well

• But that is not rational for them

• If we can commit to S they will play DIf we can commit to S, they will play D– Commitment advantage

Page 13: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Nash equilibrium [Nash 1950]q [ ]

• A profile (= strategy for each player) so that no player wants to deviateplayer wants to deviate

D S

0, 0 -1, 1D

1, -1 -5, -5S

• This game has another Nash equilibrium in g qmixed strategies – both play D with 80%

Page 14: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

The presentation game

Put effort into Do not put effort into

Presenter

Pay attention

Put effort into presentation (E)

Do not put effort into presentation (NE)

Pay attention (A) 2, 2 -8, -7Audience

Do not pay attention (NA) 0, -1 0, 0

• Pure-strategy Nash equilibria: (A, E), (NA, NE)

• Mixed-strategy Nash equilibrium:Mixed strategy Nash equilibrium:

((1/10 A, 9/10 NA), (4/5 E, 1/5 NE))

– Utility 0 for audience, -7/10 for presentery , p

– Can see that some equilibria are strictly better for both players than other equilibria, i.e. some equilibria Pareto-dominate other equilibria

Page 15: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Properties of Nash equilibrium in two-player games

• In zero-sum games, same thing as maximin/minimax strategiesmaximin/minimax strategies

• Any (finite) game has at least one Nash equilibrium [Nash 1950]

• PPAD complete to compute one Nash equilibrium• PPAD-complete to compute one Nash equilibrium [Daskalakis, Goldberg, Papadimitriou 2006; Chen & Deng, 2006]

• NP-hard & inapproximable to compute the “best” Nash equilibrium [Gilboa & Zemel 1989; Conitzer & Sandholm 2008]q

Page 16: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Nash isn’t optimal if one player can commit

2, 1 4, 0U i N h

1, 0 3, 1Unique Nash equilibrium

• Suppose the game is played as follows:– Player 1 commits to playing one of the rows,

– Player 2 observes the commitment and then chooses a columnPlayer 2 observes the commitment and then chooses a column

• Optimal strategy for player 1: commit to Down

Page 17: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Commitment as an i fextensive-form game

Player 1

• For the case of committing to a pure strategy:

Player 1

Up Down

Player 2 Player 2

Left Left RightRight

2, 1 4, 0 1, 0 3, 1

Page 18: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Commitment to mixed strategiesg

2, 1 4, 0.49.5 , ,

1, 0 3, 1.51.5

• Assume follower breaks ties in leader’s favor– In generic games this is the unique SPNE outcome of the extensive-

form game [von Stengel & Zamir 2010]

– We will also refer to this as a Stackelberg strategy

Page 19: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Commitment as an i fextensive-form game…

• for the case of committing to a mixed strategy:Player 1

… for the case of committing to a mixed strategy:

(1,0) (=Up)

(0,1) (=Down)

(.5,.5)

… …Player 2

Left Left RightRight Left Right

2, 1 4, 0 1, 0 3, 11.5, .5 3.5, .5

• Economist: Just an extensive form game nothing new here• Economist: Just an extensive-form game, nothing new here

• Computer scientist: Infinite-size game! Representation matters

Page 20: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Computing the optimal mixed strategy to commit to

[C it & S dh l 2006 St l & Z i 2010][Conitzer & Sandholm 2006, von Stengel & Zamir 2010]

• Separate LP for every possible follower’s action t*Leader utility

Distributional constraint

Follower optimality

• Choose t* for which the LP is feasible and has the highest objective The leader plays thehighest objective. The leader plays the corresponding strategy <ps>.

Slide 7

Page 21: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Easy polynomial-time algorithm for two playersfor two players

[Conitzer & Sandholm 2006; von Stengel & Zamir 2010]

• For every column t separately, we solve separately for the best mixed row strategy (defined by ps) that induces player 2 to play t

• maximize Σ p u (s t)• maximize Σs ps u1(s, t)

• subject to

for any t’, Σs ps u2(s, t) ≥ Σs ps u2(s, t’)

Σ p = 1Σs ps 1

• (May be infeasible)

• Pick the t that is best for player 1

Page 22: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

VisualizationVisualization

L C RL C R

U 0,1 1,0 0,0 (0,1,0) = MM 4,0 0,1 0,0D 0,0 1,0 1,1

( , , )

C

RL R

(1,0,0) = U (0,0,1) = D

Page 23: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Observations about commitment to a mixed strategy in a two-player game

• Coincides with minimax strategies in zero-sumCoincides with minimax strategies in zero sum games

• Leader’s payoff always at least as good as in any Nash equilibrium (see [von Stengel & Zamir 2010])q ( [ g ])– Can simply commit to the Nash equilibrium strategy

– Follower breaks ties in your favor

– Actually at least as good as any correlated equilibrium– Close relationship to LP for correlated equilibrium [Conitzer 2010 draft]

• No equilibrium selection problem• No equilibrium selection problem

• Natural notion of approximation

Page 24: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

(a particular kind of) Bayesian games(a particular kind of) Bayesian games

l d tilitifollower utilities follower utilities

2 4 1 0 1 0

leader utilitiesf

(type 1)f

(type 2)

2 4

1 3

1 0

0 1

1 0

1 3probability .6 probability .4

Page 25: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Multiple types visualizationMultiple types - visualization(0 1 0)(0,1,0)

CCombined

C

R(0,1,0)

(1,0,0)L

(0,0,1)

(0,1,0)(1,0,0) (0,0,1)

L R (R,C)

(1,0,0) C (0,0,1)

Page 26: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to
Page 27: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

LAX techniques [Paruchuri et al. 2008, Pita et al. 2009]

• Uses Bayesian games framework

• Mixed integer programming formulation for solving Bayesian games optimallysolving Bayesian games optimally– Much faster than converting game to normal

form, solving that

Page 28: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

(In)approximability[Letchford Conitzer Munagala 2009][Letchford, Conitzer, Munagala 2009]

• (#types)-approximation: pick one type uniformly at random, optimize for it using LP approach– … or (deterministic) optimize for every type separately, pick best

• Can’t do any better in polynomial time unless P=NPCan t do any better in polynomial time, unless P NP– Reduction from INDEPENDENT-SET

• For adversarially chosen types, cannot decide in polynomial y y ytime whether it is possible to guarantee positive utility, unless P=NPunless P NP– Again, a MIP formulation can be given

Page 29: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Reduction from independent setReduction from independent set

1 2 3leader utilities

A Bal

1 1 0al

2 1 0al

3 1 0f ll l f ll l f ll l

A B A B A B

follower utilities(type 1)

follower utilities(type 2)

follower utilities(type 3)

A Bal

1 3 1al

2 0 10

A Bal

1 0 10al

2 3 1

A Bal

1 0 1al

2 0 10l

al3 0 1

l

al3 0 10

l

al3 3 1

Page 30: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Switching topics: Learningg p g

• Single follower typeSingle follower type

• Unknown follower payoffs

• Repeated play: commit to mixed strategy, see follower’s (myopic) response

L RU 1 ? 3 ?U 1,? 3,?D 2 ? 4 ?D 2,? 4,?

Page 31: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

VisualizationVisualization

L C RL C R

U 0,1 1,0 0,0 (0,1,0) = MM 4,0 0,1 0,0D 0,0 1,0 1,1

( , , )

C

RL R

(1,0,0) = U (0,0,1) = D

Page 32: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

SamplingSampling

C (0,1,0)

L R

(1 0 0) (0 0 1)(1,0,0) (0,0,1)

Page 33: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Three main techniques in qthe learning algorithm

• Find one point in each region (usingFind one point in each region (using random sampling)

• Find a point on an unknown hyperplane

• Starting from a point on an unknown hyperplane, determine the hyperplanehyperplane, determine the hyperplane completely

Page 34: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Finding a point on an unknown hyperplane

Intermediate stateStep 1. Sample in the overlapping region

Step 2. Connect the new point to the pointp p pin the region that doesn’t match

CStep 3. Binary search along this lineL R

L R

R or L

Region: R

Page 35: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Determining the hyperplaneDetermining the hyperplane

Intermediate stateStep 1. Sample a regular d-simplexcentered at the point

Step 2. Connect d lines between points onopposing sides

CStep 3. Binary search along these lines

Step 4. Determine hyperplane (and update

L R

L R

Step 4. Determine hyperplane (and update the region estimates with this information)

R or L

Page 36: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Bound on number of samples

Theorem. Finding all of the hyperplanes necessary to compute the optimal mixed strategy to commit to requires O(Fk log(k) + dLk2) samples

– F depends on the size of the smallest region

L depends on desired precision– L depends on desired precision

– k is the number of follower actions

– d is the number of leader actions

Page 37: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Discussion about appropriateness of leadership model in security applications

• Mixed strategy not actually communicated

Ob bili f i d i ?• Observability of mixed strategies?– Imperfect observation?p

• Does it matter much (close to zero-sum anyway)?

• Modeling follower payoffs?– Sensitivity to modeling mistakes 2 1 4 0Sensitivity to modeling mistakes

• Human players… [Pita et al. 2009]

2, 1 4, 0

1, 0 3, 1, ,

Page 38: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Computing optimal strategies to commit to in t i fextensive-form games [Letchford & Conitzer 2010]

ChanceNo Chance

Imperfect InfoPerfect Info.

NP-hard

Imperfect Info.Perfect Info.

Pure Mixed

NP-hard

Tree DAG Tree DAG

LeftTwo Players Two PlayersThree+ Players Three+ Players

P NP-hard

No Restrictions No Restrictions RestrictionsRestrictionsNP-hardNP-hard

P PNP-hard ?

Page 39: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

A problem for scaling to (some) l li tireal applications

• So far, we have assumed that we can ,enumerate all the defender pure strategies

• Not feasible in some applicationsF d l Ai M h l [T i t l 2009]– Federal Air Marshals [Tsai et al. 2009]

– Protecting a city [Tsai et al. 2010]g y [ ]

– …

• Problem: each possible allocation of resources is a pure strategyresources is a pure strategy– Combinatorial explosion

Page 40: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Security resource allocation games[Ki ki t ld t l 2009]

• Set of targets T

[Kiekintveld et al. 2009]g

• Set of security resources available to the defender (leader)

• Set of schedules• Set of schedules

• Resource can be assigned to one of the schedules in

• Attacker (follower) chooses one target to attack

• Utilities: if the attacked target is defended,

otherwise

• st1

1

s1

s2

t2t3

22

s3

t5t4 Slide 8

Page 41: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Applications and previous workApplications and previous work

• Security checkpoints in airports (i l t d t LAX) [P h i t l(implemented at LAX) [Paruchuri et al. 2008, Pita et al. 2009]008, ta et a 009]

• Federal air marshal service [Tsai et al. 2009]2009]

Slide 9

Page 42: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Compact LPs approachCompact LPs approach

• Motivation: exponential number of pure strategies for the defender so thestrategies for the defender, so the standard LP is exponential in sizep

• Instead, we will find the (marginal) b bilit f b iprobability cs of resource being

assigned to schedule sg

Slide 10

Page 43: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Compact LPCo pac• Cf. ERASER-C algorithm by Kiekintveld et al. [2009]

• Separate LP for every possible t* attacked:

f d iliDefender utility

Marginal probability

Distributional constraints

Marginal probability of t* being defended

Distributional constraints

Attacker optimality

Slide 11

Page 44: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Counter-example to the compact LP2

.5 .5

5 tt

1

.5 tt

.5 t t

• LP suggests that we can cover every target with probability 1…

b t in fact e can co er at most 3• … but in fact we can cover at most 3 targets at a time

Slide 12

Page 45: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Schedules of size 1Schedules of size 1

• Kiekintveld et al. prove that in this case, there exists a mixed strategy with thethere exists a mixed strategy with the given marginal probabilities

• How can we find it?

1 t1

.7

2 t2

.1.3

.2

t3

.7

Slide 13

Page 46: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Birkhoff-von Neumann theorem• Every doubly stochastic n x n matrix can be

represented as a convex combination of n x n permutation matrices .1 .4 .5

.3 .5 .2

.6 .1 .3

1 0 00 0 1= .1

0 1 00 0 1+.1

0 0 10 1 0+.5

0 1 01 0 0+.3

• Decomposition can be found in polynomial time O(n4.5)

0 1 0 1 0 0 1 0 0 0 0 1

Decomposition can be found in polynomial time O(n ), and the size is O(n2) [Dulmage and Halperin, 1955]

C b t d d t t l d bl b t h ti• Can be extended to rectangular doubly substochasticmatrices Slide 14

Page 47: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Computing the probabilities for each pure strategy

1 t1.7

.1 .2 t1 t2 t3

2t2

.7

.3 1 .7 .2 .1

2 0 .3 .7

t3

.1 .2.2 .50 0 10 1 0

0 1 00 0 1

1 0 00 1 0

1 0 00 0 1

Page 48: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Summary of resultsy[Korzhyk, Conitzer, Parr 2010]

HomogeneousR

HeterogeneousResources resources

Size 1 P P(BvN theorem)

dule

s

(BvN theorem)

Size ≤2, bipartite P(BvN theorem)

NP-hard(SAT)

Sche Size ≤2 P

(constraint generation)NP-hard

NP hardSize ≥3 NP-hardNP-hard

(3-COVER)

Slide 16

Page 49: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Is it right to play Stackelberg?• Typical argument: attacker can observe

realizations of our distribution over time before executing an attack learn thebefore executing an attack, learn the distribution

• Is this accurate?

• We show that under certain conditions, it is “safe” to play the Stackelberg strategyis safe to play the Stackelberg strategy [Yin et al. 2010]

Page 50: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Every Stackelberg strategy is also a Nash strategy in security games

• Theorem: If any subset of any schedule is also a sched le then e er Stackelbergalso a schedule, then every Stackelberg strategy is also part of a Nash equilibrium

Set of defender strategies

gy p q

Nash = Minimax

Set of defender strategies

Stackelberg

Page 51: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

So how do we know we’re playing the “right” equilibrium?

T t t t tt• Turns out not to matter:

• Theorem. Security games satisfy the y g yinterchange property:

if <c1,a1> and <c2,a2> are NE profiles, then <c1,a2> and <c2,a1> are also NE profiles 1, 2 2, 1 p– Doesn’t hold in general games (e.g., chicken)

• Proof analyzes a related zero-sum game– Two-player zero-sum games always have the p y g y

interchange property

Page 52: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Interchange property in security gamesInterchange property in security games• There is a 1:1 equivalence between NE

profiles in general-sum and zero-sum games.

Interchange property of NE in zero sum• Interchange property of NE in zero-sum games: if <c1,a1> and <c2,a2> are NE profiles, then <c1,a2> and <c2,a1> are also NE profiles. This property doesn’t hold in general gamesThis property doesn t hold in general games.

• Interchange property carries over to general-sum security games because of the above equivalenceequivalence.

Page 53: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

ConsequenceConsequence• When the defender is uncertain whether her

strategy is known to the attacker or not, it is safe to play an SSE strategysafe to play an SSE strategy.

• If the attacker somehow learns the defender’s strategy, the defender gets optimal utility.

If th tt k d t l th d f d ’• If the attacker does not learn the defender’s strategy, the SSE strategy is as good as any other NE strategy because of the interchange propertyproperty.

Page 54: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Conclusion• Desire to address general-sum games in security

• Optimal mixed strategies to commit to (“Stackelberg strategies”) have certain conceptual & algorithmicstrategies ) have certain conceptual & algorithmic advantages over (say) Nash equilibrium

• Computational challenges remain: Many games have exponential strategy spaceshave exponential strategy spaces

• Also raises & forces close examination of fundamental game-theoretic questions

Th k f tt ti !Thank you for your attention!

Page 55: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Rock-paper-scissors – Seinfeld variant

MICKEY: All right, rock beats paper!(Mickey smacks Kramer's hand for losing)KRAMER I th ht d kKRAMER: I thought paper covered rock.

MICKEY: Nah, rock flies right through paper.KRAMER: What beats rock?

MICKEY: (looks at hand) Nothing beats rockMICKEY: (looks at hand) Nothing beats rock.

0 0 1 1 1 10, 0 1, -1 1, -1-1, 1 0, 0 -1, 1-1, 1 1, -1 0, 0

Page 56: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Dominancef• Player i’s strategy si strictly dominates si’ if

– for any s-i, ui(si , s-i) > ui(si’, s-i) i i i i i i i

• si weakly dominates si’ if – for any s i ui(si s i) ≥ ui(si’ s i); and

-i = “the player(s) other than i”

for any s-i, ui(si , s-i) ≥ ui(si , s-i); and– for some s-i, ui(si , s-i) > ui(si’, s-i)

0 0 1 1 1 10, 0 1, -1 1, -1strict dominance

-1, 1 0, 0 -1, 1weak dominance

-1, 1 1, -1 0, 0

Page 57: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Prisoner’s Dilemma• Pair of criminals has been caught• Pair of criminals has been caught

• District attorney has evidence to convict them of a minor crime (1 year in jail); knows that they committed a major crime together (3 years in jail) but cannot prove itg ( y j )

• Offers them a deal:If both confess to the major crime they each get a 1 year reduction– If both confess to the major crime, they each get a 1 year reduction

– If only one confesses, that one gets 3 years reduction

confess don’t confess

f -2, -2 0, -3confess

-3, 0 -1, -1don’t confess

Page 58: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

“Should I buy an SUV?” purchasing + gas cost accident cost

cost: 5 cost: 5 cost: 5

cost: 3 cost: 8 cost: 2

cost: 5 cost: 5

-10, -10 -7, -11

-11, -7 -8, -8

Page 59: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

“2/3 of the average” game• Everyone writes down a number between 0 and

100100

• Person closest to 2/3 of the average winsg

• Example:– A says 50

– B says 10B says 10

– C says 90

– Average(50, 10, 90) = 50

– 2/3 of average = 33.33g

– A is closest (|50-33.33| = 16.67), so A wins

Page 60: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Iterated dominance

• Iterated dominance: remove (strictly/weakly) te ated do a ce e o e (st ct y/ ea y)dominated strategy, repeat

• Iterated strict dominance on Seinfeld’s RPS:

0, 0 1, -1 1, -1

1 1 0 0 1 10, 0 1, -1

-1, 1 0, 0 -1, 1

-1, 1 1, -1 0, 0-1, 1 0, 0

, , ,

Page 61: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Iterated dominance: path (in)dependenceIterated weak dominance is path-dependent:

sequence of eliminations may determine whichsequence of eliminations may determine which solution we get (if any)

(whether or not dominance by mixed strategies allowed)

0, 1 0, 0

1, 0 1, 0

0, 1 0, 0

1, 0 1, 0

0, 1 0, 0

1, 0 1, 0

0, 0 0, 1 0, 0 0, 1 0, 0 0, 1

Iterated strict dominance is path-independent: elimination process will always terminate at the same point

( h th t d i b i d t t i ll d)(whether or not dominance by mixed strategies allowed)

Page 62: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

“2/3 of the average” game revisited

100

dominated

(2/3)*100dominated after removal of

(originally) dominated strategies(2/3)*(2/3)*100

(originally) dominated strategies

00

Page 63: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Mixed strategies• Mixed strategy for player i = probability

distribution over player i’s (pure) strategies

• E g 1/3 1/3 1/3• E.g. 1/3 , 1/3 , 1/3

• Example of dominance by a mixed strategy:p y gy

3, 0 0, 01/2 , ,0, 0 3, 01/2 0, 0 3, 01, 0 1, 0

1/2

1, 0 1, 0

Page 64: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Checking for dominance by mixed strategies • Linear program for checking whether strategy si*

is strictly dominated by a mixed strategy:• maximize ε• such that:such that:

– for any s-i, Σsipsi

ui(si, s-i) ≥ ui(si*, s-i) + ε– Σsi

psi= 1Σsi

psi 1

• Linear program for checking whether strategy s *• Linear program for checking whether strategy siis weakly dominated by a mixed strategy:

• maximize Σ (Σ p u (s s )) u (s * s )• maximize Σs-i(Σsi

psiui(si, s-i)) - ui(si , s-i)

• such that: f Σ ( ) ≥ ( * )– for any s-i, Σsi

psiui(si, s-i) ≥ ui(si*, s-i)

– Σsipsi

= 1

Page 65: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

The presentation game

Put effort into Do not put effort into

Presenter

Pay attention

Put effort into presentation (E)

Do not put effort into presentation (NE)

Pay attention (A) 4, 4 -16, -14Audience

Do not pay attention (NA) 0, -2 0, 0

• Pure-strategy Nash equilibria: (A, E), (NA, NE)

• Mixed-strategy Nash equilibrium:Mixed strategy Nash equilibrium:

((1/10 A, 9/10 NA), (4/5 E, 1/5 NE))

– Utility 0 for audience, -14/10 for presentery , p

– Can see that some equilibria are strictly better for both players than other equilibria, i.e. some equilibria Pareto-dominate other equilibria

Page 66: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

A poker-like gameA poker like game

“nature”

1 gets King 1 gets Jack

bet betstay stay

player 1player 10, 0 0, 0 1, -1 1, -1

cc cf fc ff

bbbet betstay stay

call fold call fold call fold call fold

player 2 player 2.5, -.5 1.5, -1.5 0, 0 1, -1

-.5, .5 -.5, .5 1, -1 1, -1sb

ss

bs

2 1 1 1 -2 -11 1

0, 0 1, -1 0, 0 1, -1ss

Page 67: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

A poker-like gameA poker like game“nature”

2/31 gets King 1 gets Jack

player 1player 10, 0 0, 0 1, -1 1, -1

cc cf fc ff

bb

2/3 1/3

1/3bet betstay stay

ll f ld ll f ld ll f ld ll f ld

player 2 player 2.5, -.5 1.5, -1.5 0, 0 1, -1

-.5, .5 -.5, .5 1, -1 1, -1sb

bs2/3

call fold call fold call fold call fold

2 1 1 1 -2 -11 1

0, 0 1, -1 0, 0 1, -1ss

• To make player 1 indifferent between bb and bs, we need:

utility for bb = 0*P(cc)+1*(1-P(cc)) = .5*P(cc)+0*(1-P(cc)) = utility for bsy ( ) ( ( )) ( ) ( ( )) y

That is, P(cc) = 2/3

• To make player 2 indifferent between cc and fc, we need:

utility for cc = 0*P(bb)+(-.5)*(1-P(bb)) = -1*P(bb)+0*(1-P(bb)) = utility for fc

That is, P(bb) = 1/3

Page 68: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Rock-paper-scissors

0 0 1 1 1 10, 0 -1, 1 1, -11 1 0 0 1 11, -1 0, 0 -1, 1-1, 1 1, -1 0, 0

• Any pure-strategy Nash equilibria?

• But it has a mixed-strategy Nash equilibrium:

Both players put probability 1/3 on each action

• If the other player does this, every action will give you expected utility 0

– Might as well randomize

Page 69: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Nash equilibria of “chicken”S DS D

SD S

D S

0, 0 -1, 1D

1, -1 -5, -5S• (D, S) and (S, D) are Nash equilibria

– They are pure-strategy Nash equilibria: nobody randomizes

– They are also strict Nash equilibria: changing your strategy will make you t i tl ffstrictly worse off

• No other pure-strategy Nash equilibria

Page 70: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Nash equilibria of “chicken”…D S

0, 0 -1, 1D

D S, ,

1, -1 -5, -5D

S , ,S• Is there a Nash equilibrium that uses mixed strategies? Say, where player 1 uses a mixed

strategy?

• If a mixed strategy is a best response, then all of the pure strategies that it randomizes over must also be best responses

• So we need to make player 1 indifferent between D and S• So we need to make player 1 indifferent between D and S

• Player 1’s utility for playing D = -pcS

• Player 1’s utility for playing S = pcD - 5pc

S = 1 - 6pcSPlayer 1 s utility for playing S p D 5p S 1 6p S

• So we need -pcS = 1 - 6pc

S which means pcS = 1/5

• Then, player 2 needs to be indifferent as wellp y

• Mixed-strategy Nash equilibrium: ((4/5 D, 1/5 S), (4/5 D, 1/5 S))

– People may die! Expected utility -1/5 for each player

Page 71: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Ranges for the follower payoffsRanges for the follower payoffs

• Suppose we just know a range within which each follower payoff lieseach follower payoff lies

L RU 1, [0,3] 2, 3D 0, [1,3] 1, [1,2]

• NP-hard if payoffs are adversarially drawn– We do not know about (in)approximability…

– except for a richer variant… except for a richer variant

Page 72: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Extension of the BvN theorem• Every m x n doubly substochastic matrix

can be represented as a convex combination of m x n matrices withcombination of m x n matrices with elements from {0, 1} such that every row and column contains “1” in at most one

ll .1 .4 .5cell. .1 .4 .5.3 .5 .2.6 .1 .3

1 0 00 0 1= 1

0 1 00 0 1+ 1

0 0 10 1 0+ 5

0 1 01 0 0+ 30 0 1

0 1 0

=.1 0 0 11 0 0

+.1 0 1 01 0 0

+.5 1 0 00 0 1

+.3

Page 73: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

[backup] Will compact LP work for homogeneous resources?

• Suppose that every resource can be assigned to any scheduleassigned to any schedule.

• We can still find a counter-example for pthis case: t

5 5 t5

t t

.5 .5

t t

.5 .5

.5 .5

r rr3 homogeneous resources

Page 74: Finding Optimal MixedFinding Optimal Mixed Strategies …dimacs.rutgers.edu/Workshops/DecisionMaking/slides/conitzer.pdf · Finding Optimal MixedFinding Optimal Mixed Strategies to

Stackelberg games in extensive formg g

(1,3)(2,2)(2.5,1)

Player 2

(2.5,1)(2,2)(3,0)(1,3)(0,1)

Player 1 Player 1

50% 50%50% 50%

2, 1 (0, 1) (2, 2)(1, 3) (3, 0)

Pure strategy commitmentMixed strategy commitmentSubgame Perfect Nash Equilibrium