Computational EconomicsοΌAlgorithmic Game TheoryοΌ
Xiang-Yang Li, Lan ZhangSchool of Computer Science and Technology, USTC
Professor, CS Department2021
Graphical Games:Equilibrium and Extensions
Β» 2.3.1 Graphical Games
β’ compact representation
Β» 2.3.2 Computing Nash Equilibria
β’ TreeProp
β’ NashProp : a distributed, message-passing algorithm
Β» 2.3.3 Correlated Equilibria
β’ definition and motivation
β’ representation
β’ computation
2.3 Graphical Games
3
Β» 2.3.1 Graphical Games
β’ compact representation
Β» 2.3.2 Computing Nash Equilibria
β’ TreeProp
β’ NashProp : a distributed, message-passing algorithm
Β» 2.3.3 Correlated Equilibria
β’ definition and motivation
β’ representation
β’ computation
2.3 Graphical Games
4
Β» Large population games with limit playersβ interaction.
Β» Game Theory:
β’ provide sound, rigorous mathematical formulation.
β’ limited attention to problem representation: commonly βflatβ, large-size representation that donβt exploit βstructureβ.
β’ graph-based representation introduced to model interaction.
Β» Multiplayer games
Multiplayer Games
Algorithms for computing equilibria in large population games with structured interactions
Β» A multiplayer game consists of π players, each with a finite set of pure strategies or actions available to them, along with a specification of the payoffs to each player.
Β» Example 1: Pollution game
β’ This game is the extension of Prisonerβs Dilemma to the case of many players.
Β» Example 2: Tragedy of the commons.
Multiplayer Game Example
Different geographic locations
Β» A multiplayer game consists of n players, each with a finite set of pure strategies or actions available to them, along with a specification of the payoffs to each player.
Β» ππ β {0, 1} : the action chosen by player π, and the joint action Τ¦π β {0, 1}π
Β» ππ β [0, 1] : the payoff to player π, indexed by the joint action Τ¦π.
Β» The actions 0 and 1 are the pure strategies of each player:
β’ a mixed strategy for player π is given by the probability ππ β [0, 1] that the player will play 0
Β» For any joint mixed strategy, given by a produce distribution Τ¦π, we define the expected payoff to player π as :
ππ Τ¦π = πΈπ~ Τ¦π[ππ( Τ¦π)]
Multiplayer Game Normal Form
7
Β» The normal form representation of Multiplayer Game with large player population is not good enough.
Β» Complexity: the number of parameters that must be specified grows exponentially with the size of the population.
β’ Assuming π players and 2 actions, as we have here, leads to the need for: π matrices ππ(one for each player) each of size 2π
Β» Conceptual: the normal form may fail to capture structurethat is present in the strategic interaction:
β’ Strong Influence
β’ Weakly Influence
Issues with Normal Form
8
Β» It is assumed that the payoff ππ for player π is a function of all the components ππ , (π = 1,β¦ , π) in the joint action
vector Τ¦π
Β» However, the payoff for player π may be dependent only on the actions of a subset of player π(π)
β’ Conditional independence payoff assumption
Β» Perhaps: the payoff of each player are determined by the actions of only a small subpopulation.
β’ Strong influence
β’ The environmental pollution of a country is more affected by neighboring countries.
Structure in Multiplayer Game
9
Β» Graphical Model : models of structed probabilistic interaction.
β’ π points represent π players, for which naΓ―ve representation is size : 2π
β’ representation size mostly a function of the βdegree of local interactionβ among points or players.
Β» Graph Model allows:
β’ easy interpretation
β’ compact representation
β’ efficient computation in some cases
Graphical Model
Β» Graphical Games adopt a simple Graph Model.
Β» A graphical game is described by an undirected graph πΊ:
β’ players ~ vertices
Β» A player or vertex π has payoffs that are entirely specified by the actions of π and those of its neighbor set in πΊ
β’ Strong Influence along neighbors
Β» Now the payoff to player π is a function only of the actions of π and its neighbors:
β’ Rather than the actions of the entire population
β’ Reduce Complexity.
Graphical Game & Graph Model
11
Β» Graphical games can capture and exploit locality or sparsity of direct influences.
Β» Graphical Games:
β’ borrow representational ideas from graphical models.
β’ intuitive graph interpretation: a playerβs payoff is only a function of its neighborhood.
β’ example : geography, networks
β’ analogy to probabilistic graphical model : special structure.
Graphical Games
Graphical Games Formal Definition
Β» A graphical game is a pair (πΊ,β³), where πΊ is an undirected graph over the vertices {1,β¦ , π}, and β³ is a set of π local game matrices. For any joint action Τ¦π, the
local game matrix ππ β β³ specifies the payoff ππ(ππ) for
player π, which depends only on the actions by the players in π(π).
Graphical Games Definition
Β» πΊ : undirected graph representing the local interaction
Β» π : a set of π local game matrices
Β» Player πβs payoff ππ Τ¦π is only a function of its neighborhood π(π): implies conditional independence payoff assumption.
Β» Local payoff matrix ππβ²: ππ Τ¦π = ππ
β² Τ¦π π π
Β» Graphical game complexity:
max degree of local interaction π = πππ₯π π π βͺ πrepresentation size π(π2π) (exponential in max degree)
Β» Nash equilibrium(NE): A Nash equilibrium for the game is a mixed strategy Τ¦π such that for any player π, and for any value ππβ² β [0, 1]
ππ Τ¦π β₯ ππ( Τ¦π[π: ππβ²])
We say that ππ is a best response to the rest of Τ¦π
Β» π-Nash equilibrium: An π-Nash equilibrium is a mixed strategy Τ¦π such for any player π, and for any any value ππ
β² β [0, 1]ππ Τ¦π + π β₯ ππ( Τ¦π[π: ππ
β²])
We say that ππ is an Ξ΅-best response to the rest of Τ¦π
Equilibrium Definition
15
Β» 2.3.1 Graphical Games
β’ compact representation
Β» 2.3.2 Computing Nash Equilibria
β’ TreeProp
β’ NashProp
Β» 2.3.3 Correlated Equilibria
β’ definition and motivation
β’ representation
β’ computation
2.3 Graphical Games
16
Β» TreeNash is a tree algorithm, which runs in polynomial time and provably computes approximations of all equilibria.
Β» Capital letters: π, π πππ π to denote player/vertex to distinguish them from their chosen actions/values(lower letter π’, π£, π€).
Β» Local game matrix: ππ to denote the local game matrix for the player/vertex π.
TreeNash Notation and Concept
Β» If G is a tree, we choose an arbitrary vertex as the root (which we visualize as being at the bottom, with the leaves at the top).
Β» Upstream: any vertex on a path from π to any leaf will be called upstreamfrom π:
β’ πππΊ π to denote the set of all vertices in πΊ that are upstream from π
Β» Downstream: any vertex on the path from a vertex π to the root will be called downstream from π.
TreeNash Notation and Concept
Leaf/Parent
Root/Child
upstreamdownstream
Β» Suppose that π is the child of π in πΊ.
Β» So, we let πΊπ denote the subgraph induced by the vertices in πππΊ(π)
the subtree of πΊ rooted at π
Β» Let: π£ β [0, 1] is a mixed strategy for player/vertex π: β³π=π£
π will be the subset of payoff matrices in β³ corresponding to the vertices in πππΊ(π)
TreeProp NE
Β» An NE for the graphical game (πΊπ,β³π=π£
π ) is a conditional equilibrium βupstreamβ from π β that is an equilibrium for πΊπ given that π plays π£.
Β» Here we are simply exploiting the fact that since πΊ is a tree, fixing a mixed strategy π£ for the play of π isolates πΊπ
from the rest of πΊ.
TreeProp NE
Β» Now suppose that vertex π has πparents π1, π2, β¦ , ππ and the single child π.
Β» The data structures sent from each ππto π, on the downstream pass of TreeNash , and in turn from π to π :
β’ Each parent ππ will send to π a binary-valued βtableβ π(π£, π’π)
TreeProp Downstream: π(π£, π’)
π(π£, π’π)
Β» The table is indexed by the continuumof possible values for the mixed strategies π£ β [0, 1] of π and π’π β[0, 1] of π1, π2, β¦ , ππ
Β» The semantics of this table: for any pair (π£, π’π), π(π£, π’π) will be 1 if and only if there exists an NE for
(πΊππ ,β³π=π£ππ ) in which π1 = π’π
TreeProp Downstream: π(π£, π’)
π(π£, π’π)
Β» The data structures sent from π to π, on the downstream pass of TreeNash
Β» π(π€, π£) = 1βΊ there exists a vector of mixed strategies π’ =(π’1, β¦ , π’π) (called a witness) for the
parents π = (π1, β¦ , ππ) of π such that:
π π£, π’π = 1 for all 1 β€ π β€ π
π = π£ is a best response to π =π’,π = π€.
TreeProp Downstream: π(π€, π£)
π(π€, π£)
Β» There may be more than one witness for π (π€, π£) = 1:
β’ π will also keep a list of the witnesses π’ for each pair (π€, π£) for which π (π€, π£) = 1
β’ These witness lists will be used on the upstream pass.
Β» The downstream pass of the algorithm terminates at the root π.
TreeProp Downstream
Β» Begin: Root node W choose any witness (π1, β¦ , ππ) to π(π€) = 1, and then passing both w and π£π to each parent ππ.
β’ The interpretation is that π will play π€, and is βinstructingβ ππ to play π£π
β’ if a vertex π receives a value π£to play from its downstream neighbor π, and the value π€that π will play, then it must be that π (π€, π£) = 1.
TreeProp Upstream
π(π€, π£)
Β» π chooses a witness π’ to π (π€, π£) = 1, and passes each parent ππ their value π’π as well as π£
Β» Note that the semantics of π (π€, π£) =1 ensure that π = π£ is a best response
to π = π’,π = π€.
TreeProp Upstream
π(π£, π’π)
Β» Downstream Pass:
β’ each node π receives π(π£, π’π)from each ππ;
β’ node π computes π(π€, π£) and witness lists for each π(π€, π£) = 1
Β» Upstream Pass:
β’ each node π receives value (π€, π£) from node W, s.t. π(π€, π£) =1
β’ node π pick witness π’ =(π’1, π’2, β¦ , π’π) for π(π€, π£), pass (π£, π’π) to node ππ.
TreeProp Algorithm Flow
TreeProp Algorithm Flow
TreeProp Algorithm Flow
Β» Dynamic programming algorithm
Β» Table-Passing phase
β’ T(w, v) represent there exists a NE βupstreamβ in which π plays π£ and π is βclampedβ to π€.
Β» Assignment-Passing phase
β’ assign NE mixed strategy to root. - downstream
β’ recursively find assignment for immediate βupstreamβ neighbors consistent with tables.
TreeProp
Β» Representation results
β’ for Ο΅ β NE need Ο β size grid for tables polynomial in model size and 1/Ο΅
Β» Lemma: Let (πΊ,β³) be any graphical game in which πΊ is a tree. Algorithm TreeNash computes a Nash equilibrium for (πΊ,β³). Furthermore, the tables and witness lists computed by the algorithm represent all Nash equilibria of (πΊ,β³).
TreeProp
Β» The actions (π’, π£, π€ β¦ ) are continuous variables β π(π€, π£) be represented compactly?
TreeProp: A Slight Issue
Β» Discretization of the action space
Β» Player I can now only play action ππ β {1, π, 2π, β¦ , 1}
Β» Algorithm takes an extra input parameter π
Β» At each node the π-best response is computed(π =π(π/π))
Β» Lemma: Approximate TreeNash computes a π-NE for the game (πΊ,π) in time polynomial in the representation of (πΊ,π)
Approximate TreeNash
Β» However its complexity is exponential in the number of vertices of πΊ.
Β» Computing an exact equilibrium in time polynomial in the size of the tree remains an open issue.
Exact TreeNash
Β» A distributed, message-passing algorithm: natural extension of TreePropto arbitrary graphs.
Β» each π does this computation once for each of its neighbors:
β’ each time βpretendingβ that this neighbor plays the role of the downstream neighbor in TreeNash;
β’ the remaining neighbors play the roles of the upstream ππ.
NashProp for Arbitrary Graph
Β» Table-Passing phase:
β’ each π does this computation once for each of its neighbors
β’ the remaining neighbors play the roles of the upstream ππ
β’ Initialization: π 0 π€, π£ = 1 for all (π€, π£)
β’ Induction: π π + 1 π€, π£ =1, πππ β π’:Β» π π π£, π’π = 1 πππ πππ πΒ» π = π£ a best response for
π = π€,π = π’
Table-Passing
Β» Table-Passing phase always converges.
Β» ALL NE preserved.
Β» For discretization scheme:
β’ tables converge quickly(number of rounds polynomial in model size).
β’ each round takes polynomial in model size(for fixed grid size).
Convergence of Table-Passing
Β» NEs preserved but search still needed
Β» More 0βs in tables can lead to significantly reduced search space.
Β» Many heuristics possible
β’ Backtracking local search
Β» Discretization scheme: Computation time per round polynomial in model size (for fixed grid size).
β’ Discretization scheme leads to constraint satisfaction problem (CSP) formulations.
β’ NashProp is a particular instantiation of arc-consistency followed by backtracking local search in a particular CSP.
Assignment-Passing
Β» Converging first phase with
β’ table size for π β ππΈ polynomial in the size of the model.
β’ running time also polynomial (for fixed k).
Β» Second phase is backtracking local search.
Β» For both phases: each round polynomial in the size of the model (for fixed k)
NashProp Summary
Β» Exact NE Computation
All NE in trees: exponential in representation size [Kearns, Littman and Singh, 2001]; Single NE in trees: polynomial for 2-action [Littman, Kearns and Singh, 2002], m-action open!; Single NE in loopy graphs: continuation-method heuristic [Blum, Shelton and Koller, 2003]
Β» Other approximation heuristics
CSP formulation: Cluster [Kearns, Littman and Singh, 2001] and junction- tree [Vickrey and Koller, 2002]; Gradient ascent and βhybridβ approaches [Vickrey and Koller, 2002]
Β» Some results on computing NE for torus-like GG [Daskalakis and Papadimitriou, 2004]
Related Work
Β» NashProp for computing Nash equilibria in graphical games :
β’ Distributed, message-passing: Avoids centralized computation
β’ Runs directly on game graph: Avoids operating on βhyper-graphsβ
β’ Generalizes approximation algorithm for trees
β’ Strong theoretical guarantees for table-passing phase
β’ Assignment passing phase: Simple implementations experimentally sufficient and effective in arbitrary graphs
β’ Promising, effective heuristic in loopy graphs (as loopy belief propagation)
Summary
Β» 2.6.1 Graphical Games
β’ compact representation
Β» 2.6.2 Computing Nash Equilibria
β’ TreeProp
β’ NashProp : a distributed, message-passing algorithm
Β» 2.6.3 Correlated Equilibria
β’ definition and motivation
β’ representation
β’ computation
2.6 Graphical Games
43
Β» The game we consider is when two players drive up to the same intersection at the same time. If both attempt to cross, the result is a fatal traffic accident. The game can be modeled by a payoff matrix where crossing successfully has a payoff of 1, not crossing pays 0, while an accident costs β100.
Example: Traffic Light
Β» This game has three Nash equilibria:
β’ two correspond to letting only one car cross.
β’ the third is a mixed equilibrium where both players cross
with an extremely small probability π =1
101, and with π2
probability they crash.
Β» In a correlated equilibrium a coordinator can choose strategies for both players
β’ For example, the coordinator can randomly let one of the two players cross with any probability.
β’ The player who is told to stop has 0 payoff, but he knows that attempting to cross will cause a traffic accident.
Example: Traffic Light
Β» Same setting as before: Games with greedy players.
Β» Mathematical formulation: Normal-form games :
β’ A set of players {1,β¦ , π}, each with a set of actions or pure strategies π΄ = {1,β¦ ,π}, joint-action (π1, β¦ , ππ) β π΄
π
β’ Payoff matrix ππ: player iβs payoff ππ(π1, β¦ , ππ)
Β» Correlated equilibrium (CE): A joint probability distribution π(π1, β¦ , ππ) such that:
β’ Every player individually receives βsuggestionβ from P.
β’ Knowing P , players are happy with βsuggestionβ
Β» NE as a special case: P - a product distribution; always exists!
Correlated Equilibria
Correlated Equilibria
Β» Definition: A correlated equilibrium (CE) for a two-actionnormal form game is distribution π( Τ¦π) over actions satisfying:
βπ β 1, β¦ , π , βπ β 0,1 : πΈπ~πππ=π[ππ( Τ¦π)] β₯ πΈπ~πππ=π
[ππ( Τ¦π[π: βπ])]
Β» Representation
β’ P(a1, β¦ , an) exponential in number of players π
β’ Does succinct graphical game representation? β succinct CE representation? (Intuition: interaction due to game should govern correlations)
β’ Yes (under a reasonable equivalence class)
Β» Computation
β’ Normal-form games: CE computable via LP (with variables P(a1, β¦ , an) ).
β’ Does succinct graphical game computation? β efficient CE algorithm?
β’ Yes (for some interesting subclass of graphical games)
CE in Graphical Games
Β» First, we argue that it is not necessary to model all the correlations that might arise in a CE, but only those required to represent all of the possible (expected payoff) outcomes for the players.
Β» Second, we need show that the remaining correlations can be represented by a Markov network.
β’ the interactions between vertices in the graphical game are entirely strategic and given by local payoff matrices
β’ the interactions in the associated Markov network are entirely probabilistic and given by local potential functions
Representation - succinctly
Β» Representation issue: In general, just considering arbitrary CE does not help representationally
β’ Want to preserve succinctness of GG in CE representation as well
Β» Keypoint: there is a natural subclass of the set of all CE of a graphical game, based on expected payoff equivalence, whose representation size is linearly related to the representation size of the graphical game.
CE Equivalence
EPE & LNE Definition
Β» EPE: two distribution π and π over joint actions Τ¦π are expected payoff equivalent, denote π β‘πΈπ π, if π and πyield the same expected payoff vector: for each π, πΈπ~π ππ Τ¦π = πΈπ~π ππ Τ¦π
β’ Expected payoff equivalence of two distributions is dependent upon the reward matrices of a graphical game
Β» LNE: For a graph πΊ, two distribution π and π over joint actions Τ¦π are local neighborhood equivalent with respect to πΊ, denote π β‘πΏπ π, if for all players π, and for all setting Τ¦ππ
of π(π), π Τ¦ππ = π( Τ¦ππ)
β’ the marginal distributions over all local neighborhoods is dependent up on the context of graph πΊ.
EPE & LNE Equation Lemma
Thus local neighborhood equivalence implies payoff equivalence, but the converse is not true in general. We now establish that local neighborhood
equivalence also preserves CE.
Β» Lemma: For all graphs πΊ, for all joint distributions π and πon actions, and for all graphical games with graph πΊ, if π β‘πΏπ π then π β‘πΈπ π. Furthermore, for any graph πΊ and joint distributions π and π, there exist payoff matrices β³such that for the graphical (πΊ,β³), if π β’πΏπ π then π β’πΈπ π.
LNE β EPE
Efficient CE Representation Lemma
Β» Lemma: For any graphical game (πΊ,β³), if π is a CE for (πΊ,β³) and π β‘πΏπ π then π is a CE for (πΊ,β³).
Β» Now we can concisely represent, in a single model, all CE up to local neighborhood (and therefore payoff) equivalence.
Correlated Equilibria & Markov Nets
Β» Graphical games provide a concise language for expressing local interaction in game theory
Β» Markov networks exploit undirected graphs for expressing local interaction in probability distributions
Β» Markov networks are a natural and powerful language for expressing the CE of a graphical game:
β’ there is a close relationship between the graph of the game and its associated Markov network graph
Local Markov Nets Definition
Β» A local Markov network is a pair M β‘ (G,Ξ¨), where
β’ πΊ is an undirected graph on vertices {1, β¦ , π}
β’ Ξ¨ is a set of potential functions, one for each local neighborhood π(π), mapping binary assignments of values of π(π) to the range [0,β):
Ξ¨ β‘ {ππ: π = 1,β¦ , π; ππ: { Τ¦ππ} β [0,β)},
where { Τ¦ππ}is the set of all 2|π(π)| settings to π(π)
Β» Compact representation of joint probability distributions
Β» Graph G = (V,E)
β’ vertices V correspond to random variables
β’ potential function for each neighborhood of G
βπ, ππ: Τ¦ππ β [0,+β)
Β» Local Markov network: M β‘ (G,Ξ¨)
Β» A local Markov network π defines a probability distribution π as
π Τ¦π =1
πΰ·
π=1
π
ππ( Τ¦ππ)
Β» Again, representation size exponential in size of largest neighborhood
Local Markov Networks
CE Representation Lemma
Β» Lemma: for all graphs πΊ, and for all joint distribution π over joint actions, there exist a distribution π that is representable as a local Markov network with graph πΊ such that Q β‘πΏπ πwith respect to πΊ.
Β» Lemma: for all graphical games (πΊ,β³), and for any correlated equilibrium π of (πΊ,β³), there exist a distribution π such that:
β’ π is also correlated equilibrium for (πΊ,β³)
β’ π gives all players the same expected payoffs as P: Q β‘πΈπ π
β’ π can be represented as a local Markov network with graph πΊ.
Β» Efficient CE Representation Lemma :
For every CE π for a graphical game with graph πΊ, there exists a CE π for the game with the properties
β’ π is a local Markov network with graph πΊ
β’ Q is expected payoff equivalent to P
Β» Proof idea/sketch: Maximum entropy (ME) distribution consistent with local neighborhood distributions of P satisfies those conditions
Β» Implication: Qualitative probabilistic properties of CEs in GGs
CE Representation Lemma
Β» CE conditions correspond to linear inequalities in the joint distribution values π(π1, β¦ , ππ)
Β» Distribution constraints also linear in π(π1, β¦ , ππ)
Β» Known result: We can compute a single exact CE for a game in normal-form in time polynomial in the representation size of the game (π(2π)) by using linear programming (LP)
Β» Can we preserve succinctness of GG representation in CE computation?
Computation: Normal-form Games
Β» Variables: {ππ( Τ¦ππ)} (local neighborhood marginals)
Β» Global CE constraints (can be a large set)
β’ Best response: linear in {ππ( Τ¦ππ)}
β’ Global consistency: ensure {ππ( Τ¦ππ)} correspond to some
proper global joint probability distribution P (a1, . . . , an)
Β» Local CE constraints (polynomial number of linear constraints)
β’ Best response
β’ Local Marginal Distribution
β’ Intersection Consistency
Β» In general, local consistency does not imply global consistency, BUT ...
Constraints
Β» Local consistency sufficient for global consistency if the game graph is a tree (for example)
Β» Efficient Tree Algorithm :
β’ There exists an algorithm (based on LP) that finds a CE π for tree graphical games with graph πΊ in time polynomial in the representation size of the game
Β» Can be extended to bounded tree-width graphical games
Β» π is also a local Markov network with graph πΊ.
Β» Can sample uniformly from set of CE in polynomial-time for bounded tree-width GG
Constraints
Linear Programming
Β» Variables: for every player I and assignment Τ¦ππ, there is a variable π( Τ¦ππ)
Β» LP Constraints:β’ CE Constraints: for all players π and actions π, πβ²
ππ:πππ=π
π( Τ¦ππ)ππ( Τ¦ππ) β₯
ππ:πππ=π
π Τ¦ππ ππ[ Τ¦ππ[π: πβ²]]
β’ Neighborhood Marginal Constraints: for all players π:β Τ¦ππ: π Τ¦ππ β₯ 0,
ππ
π Τ¦ππ = 1
β’ Intersection Consistency Constraints: for all players π and π, and for any assignment Τ¦π¦ππ to the intersection π(π) β© π(π) :
π Τ¦πππ β‘
ππ:πππ=π¦ππ
π Τ¦ππ =
ππ:πππ=π¦ππ
ππ Τ¦ππ β‘ ππ( Τ¦πππ)
Algorithm Result Lemma
Β» Lemma: for all tree graphical games (πΊ,β³), any solution to the linear constraints given above is a correlated equilibrium for (πΊ,β³)
Β» CE Representation Theorem
β’ Every CE for a graphical game can be characterized by another achieving the same expected payoffs vector and which can be represented in size polynomial in the representation size of the game
Β» Efficient Tree Algorithm (Generalizes normal-form games algorithm)
β’ For all graphical games with graphs in a certain class, containing trees and full-graphs as βcanonicalβ examples, we can compute a CE for the game in time polynomial in the representation size of the game
CE in GG: Summary
Β» Efficient Algorithms for Exact Nash Computation in Trees.
Β» Strategy-Proof Algorithms for Distributed Nash Computation.
Β» Cooperative, Behavioral, and Other Equilibrium Notions.
Open Problems/Areas