Sparse Binary Zero Sum Games (ACML2014)

Sparse Binary Zero-Sum Games[ACML 2014]

David Auger1 Jialin Liu2 Sylvie Ruette3 David L. St-Pierre4

Olivier Teytaud2

1AlCAAP, Laboratoire PRiSM, Universite de Versailles Saint Quentin-en-Yvelines, France

2TAO, INRIA-CNRS-LRI, Universite Paris-Sud, France

3Laboratoire de Mathematiques, CNRS, Universite Paris-Sud, France

4Universite du Quebec a Trois-Rivieres, Canada

Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 1 / 26

Thanks to reviewers for very fruitful comments.


Introduction

Two-person zero-sum game MK×K

Nash Equilibrium → O(K 2α) with α > 3

If the Nash is sparse → k × k submatrix

→ O(k3kK logK ) with probability 1− δ (provable)


Introduction

Two-person zero-sum game MK×K

Nash Equilibrium → O(K 2α) with α > 3

If the Nash is sparse → k × k submatrix

→ O(k3kK logK ) with probability 1− δ (provable)


Zero-sum matrix games

Game defined by matrix M

I choose (privately) i

Simultaneously, you choose j

I earn Mi ,j

You earn −Mi ,j

So this is zero-sum.

Or you earn 1−Mi ,j (so this is 1-sum, equivalent).


Ok, I earn Mi ,j , you earn −Mi ,j




Nash Equilibrium

Nash Equilibrium (NE)

Zero-sum matrix game M

My strategy = probability distrib. on rows = x

Your strategy = probability distrib. on cols = y

Expected reward = xTMy

There exists x∗, y∗ such that ∀x , y ,

xTMy∗ ≤ x∗TMy∗ ≤ x∗TMy .

(x∗, y∗) is a Nash Equilibrium (no unicity).



Nash: Ok I play i with probability x∗i

How to compute x*?



Nash: Ok I play i with probability x∗i

How to compute x*?


Solving Nash

Solution 1: Linear Programming (LP)

1 M ← M + C so that it is positive (without loss of generality)

2 LP: find 0 ≤ u minimizing∑iui such that (MT ) · u ≥ 1

3 x∗ = u/∑iui

=⇒ classical, provably exact, polynomial time


Solving Nash

Solution 2: Approximate Nash Equilibrium

Approximate ε-NE

(x∗, y∗) such that

xTMy∗ − ε ≤ x∗TMy∗ ≤ x∗TMy + ε.


Solution 1: LP (comp. expensive)



Solution 1: LP (comp. expensive)



Computing approximate Nash Equilibrium

Assuming the matrix is of size K × K ...

LP (see reduction from Nash to linear programming in[Von Stengel (2002)]): O(K 2α) with 3 < α ≤ 4

[Grigoriadis and Khachiyan(1995)]:

ε-Nash with expected time O(K log(K)ε2 ), i.e. less than the size of the

matrix!Parallel : O( log2(K)

ε2 ) if using Klog(K) processors

Other algorithms: similar complexity, approximate solution + fixedtime with probability 1− δ

EXP3 ([Auer et al.(1995)])Inf ([Audibert and Bubeck(2009)])


Computing approximate Nash Equilibrium

Assuming the matrix is of size K × K ...

LP (see reduction from Nash to linear programming in[Von Stengel (2002)]): O(K 2α) with 3 < α ≤ 4

[Grigoriadis and Khachiyan(1995)]:

ε-Nash with expected time O(K log(K)ε2 ), i.e. less than the size of the

matrix!Parallel : O( log2(K)

ε2 ) if using Klog(K) processors

Other algorithms: similar complexity, approximate solution + fixedtime with probability 1− δ

EXP3 ([Auer et al.(1995)])Inf ([Audibert and Bubeck(2009)])


Other tools 1: Hadamard determinant

Hadamard determinant bound([Hadamard(1893)], [Brenner and Cummings(1972)])

Given matrix Mk×k with coefficients in {−1, 0, 1}, then M has

determinant at most kk2 , i.e.

| detM| ≤ kk2 .


Other tools 2: Linear programming

Solve

min ax

Mx ≤ c

x ∈ Rd

If there is a finite optimum, then there is a finite optimum x suchthat, for some E with |E | = d ,

∀i ∈ E , Mix = cithe Mi for i in E are linear independent(=⇒ i.e. d lin. indep. constraints are active)


Why is this relevant ?

Nash = solution of linear programming problem

x∗: Nash Equilibrium of MK×K

Let us assume that x∗ is unique and has at most k non-zerocomponents (sparsity)

⇒ x∗ = also NE of a k × k submatrix: M ′k×k⇒ x∗ = solution of LP in dimension k⇒ x∗ = solution of k lin. eq. with coefficients in {−1, 0, 1}⇒ x∗ = inv-matrix ∗ vector⇒ x∗ = obtained by “cofactors / det matrix”

⇒ x∗ has denominator at most kk2


Why is this relevant ?

Nash = solution of linear programming problem

x∗: Nash Equilibrium of MK×K

Let us assume that x∗ is unique and has at most k non-zerocomponents (sparsity)⇒ x∗ = also NE of a k × k submatrix: M ′k×k⇒ x∗ = solution of LP in dimension k⇒ x∗ = solution of k lin. eq. with coefficients in {−1, 0, 1}⇒ x∗ = inv-matrix ∗ vector⇒ x∗ = obtained by “cofactors / det matrix”

⇒ x∗ has denominator at most kk2


How to realise ?

Under assumption that the Nash is sparse

x∗ is rational with “small” denominator

So let us compute an ε-Nash (sublinear time!)

And let us compute its closest approximation with “smalldenominator” (Hadamard)

variants for ε-Nash =⇒ exact Nash

Rounding: switch to closest approximation

Truncation: remove small components and work on the remainingsubmatrix (exact solving)


Evil in the details

||y − y∗||∞ ≥ ε does not imply V (y) ≥ V (y∗) + ε;

indeed V (y) ≥ V (y∗) + ||y−y∗||∞k

k2

Results : (if Grigoriadis)

For a K × K matrix with Nash k-sparseExact solution in time O(poly(k) + (K logK )k3k) withtruncation-algorithm


Experimental results: two card games

Previous results: ingaming of Urban Rivals

New results: metagaming of Pokemon


Ingaming results (Urban Rivals)

Previous work: [Flory and Teytaud(2011)], implementation ofTruncated-EXP3, without proof

Urban Rivals AI= Monte Carlo Tree Search([Coulom (2006)]),using zero-sum matrix gamesas a key component


Ingaming results (Urban Rivals)

Previous work: [Flory and Teytaud(2011)], implementation ofTruncated-EXP3, without proof

Results don’t look impressive (∼ 56%), but the game is highlyrandomized =⇒ Reaching 55% is far from being negligible


New experiments

Test on Pokemon Deck choice (“metagaming”)

Based on EXP3+truncation

Various versions of EXP3 (6= parameters)

Code available https://www.lri.fr/~teytaud/games.html


https://www.lri.fr/~teytaud/games.html

New experiments

With a poorly tuned EXP3 : truncation brings a huge improvement

100 101 102 103 1040.45

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

TEXP3 vs EXP3

100 101 102 103 1040.45

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

TEXP3 vs UniformEXP3 vs Uniform

Figure: Performance in terms of budget T with a poorly tuned EXP3 for thegame of Pokeman using 2 cards.


New experiments

With a well-tuned EXP3, truncation brings a significant improvement

100 101 102 103 1040.5

0.51

0.52

0.53

0.54

0.55

0.56

0.57

0.58

TEXP3 vs EXP3

100 101 102 103 1040.45

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

TEXP3 vs UniformEXP3 vs Uniform

Figure: Performance in terms of budget T with a well-tuned EXP3 for the gameof Pokeman using 2 cards.


Conclusions & further work

Proved small improvement, experimentally big improvement.Improving the bound ?

We don’t know k (sparsity level). Adaptive algorithms ?

Proved only with unique Nash (x∗, y∗). Necessary ?


Jean-Yeves Audibert and Sebastien Bubeck.

Minimax policies for adversarial and stochastic bandits.In 22th annual conference on learning theory, 2009.

Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E. Schapire.

Gambling in a rigged casino: the adversarial multi-armed bandit problem.In Proceedings of the 36th Annual Symposium on Foundations of Computer Science. IEEE Computer Society Press, 1995.

Remi Coulom (2006).

Efficient selectivity and backup operators in Monte-Carlo tree search.In Computers and games, 2006.

Joel Brenner and Larry Cummings.

The Hadamard maximum determinant problem.In Amer. Math. Monthly, 1972.

Sebastien Flory and Olivier Teytaud.

Upper confidence trees with short term partial information.In Procedings of EvoGames, 2011.

Michael D. Grigoriadis and Leonid G. Khachiyan.

A sublinear-time randomized approximation algorithm for matrix games.In Operations Research Letters, 1995.

Jacques Hadamard.

Resolution d’une question relative aux determinants.In Bull. Sci. Math., 1893.

Bernhard Von Stengel.

Computing equilibria for two-person games.In Handbook of game theory with economic applications, 2002.


Thank you for your attention !

David Auger

David L. St-Pierre

Sylvie Ruette

Olivier Teytaud


[ACML 2014]

Sparse Binary Zero-Sum Games

D. Auger J. Liu S. Ruette D. L. St-Pierre O. Teytaud


Presentations & Public Speaking

Sparse Binary Zero Sum Games (ACML2014)