Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
The optimizing agent The transported density Symmetric N-player games
A Soft Intro to Mean Field Games
Barnabe Monnot
December 11, 2015
The optimizing agent The transported density Symmetric N-player games
What we are looking at
Game theory is complicated enough with only two players, butwhen the number explodes...
I Use ’population’ arguments (congestion games...)
I Use ’algorithmic’ arguments (learning...)
I Play on networks (local effects...)
Many connections between one another.
Today we are looking at way to move to the continuous usingPDEs and mean field arguments.
The optimizing agent The transported density Symmetric N-player games
If you like pina coladas...
The optimizing agent The transported density Symmetric N-player games
The Beast
MFG “basic” system∂u∂t
+ ν∆u + H(x ,∇u) + V [m] = 0∂m∂t− ν∆m + div (m∇pH(x ,∇u)) = 0
m|t=0 = m0, u|t=T = g [mT ]
The optimizing agent The transported density Symmetric N-player games
Why it is worth looking at
Very general model, extends far beyond economics. The previoussystem admits as particular cases famous equations such as theEuler equations or the Vlasov ones.
Piggybacks on very active fields of research, such as optimaltransport or stochastic control.
Has found a good deal of numerical methods for solving thesetypes of games, taking it out of the theoretical forest (pedestriancrossing animation).
The optimizing agent The transported density Symmetric N-player games
Derivation of the MFG system
We will derive The Beast from a simple model. Two steps:
I Agents optimize their choices given the density of otherplayers (our m) ⇒ control in feedback, gives backwardequation.
I The densities are transported by these choices ⇒ forwardequation.
The optimizing agent The transported density Symmetric N-player games
Plan
I The optimizing agent
I The transported density
I Symmetric N-player games
The optimizing agent The transported density Symmetric N-player games
The agent’s problem
We have a nice space Y (take Rd , or call it the beach), with adistance d . Look at time interval [0,T ].
For all t ∈ [0,T ], suppose the density of the players on Y , mt isgiven.
The agent controls (using α ∈ A) the stochastic equation
dXt = αtdt +√
2νdBt
where Bt is a Brownian motion in Rd .
The optimizing agent The transported density Symmetric N-player games
The agent’s problem (cont.)
The agent solves the following stochastic control probleminfα E
[ ∫ T0
(L(Xt , αt) + V [mt ](Xt)
)dt + g [mT ](XT )
]dXt = αtdt +
√2νdBt
X0 = x
Define Hamiltonian H
H(x , p) = minα∈A{L(x , α) + α · p}
Hamilton-Jacobi-Bellman
The value function u satisfies{∂u∂t + ν∆u + H(x ,∇u) + V [m] = 0
u|t=T (x) = g [mT ](x)
The optimizing agent The transported density Symmetric N-player games
A closer look at the V
V is the cost incurred from the density of other players. Can give ageneral example:
V [m](x) = V0(x)︸ ︷︷ ︸I like the bar
+
I don’t like people︷ ︸︸ ︷f (x ,m(x)) +
∫YW (x , y)m(dy)︸ ︷︷ ︸
But not too alone either
BUT
I With congestion term f (x ,m(x)), no existence of equilibrium(need V [m] ∈ C (Y ))
I With distance term∫Y W (x , y)m(dy), no uniqueness (need
V [m] to be monotonous in m, e/gV [m](x ,m(x)) ≥ V [m](y ,m(y)) if m(x) ≥ m(y))
The optimizing agent The transported density Symmetric N-player games
Plan
I The optimizing agent
I The transported density
I Symmetric N-player games
The optimizing agent The transported density Symmetric N-player games
Movement of agents
So far we have considered the measure m as given, but if agentsmove, m does too.
Agents find an optimal control α(x) such that
∇pH(x , p) = α(x)
This drift is their motion.
The optimizing agent The transported density Symmetric N-player games
Transport of the measure
Now, for all t, mt = L(Xt), with m0 = L(X0) given. So mt is thesolution of
∫Y φ dmt =
∫Y E[φ(X x
t )] dm0(x)
dXt = αtdt +√
2νdBt
m0 = L(X0)
φ is a test function (∈ C∞c (Y ))
Fokker-Planck equation
mt is governed by{∂m∂t − ν∆m + div (mα) = 0
m|t=0 = m0
The optimizing agent The transported density Symmetric N-player games
The Beast, dissected
Clear where this system is coming from?∂u∂t
+ ν∆u + H(x ,∇u) + V [m] = 0∂m∂t− ν∆m + div (m∇pH(x ,∇u)) = 0
m|t=0 = m0, u|t=T = g [mT ]
The optimizing agent The transported density Symmetric N-player games
The linear-quadratic case
Let L(x , α) = |α|22 , then
H(x , p) = minα
|α|2
2+ p · α = −|p|
2
2
The MFG system now reads∂u∂t + ν∆u− |∇u|
2
2 + V [m] = 0∂m∂t − ν∆m − div (m∇u) = 0
m|t=0 = m0, u|t=T = g [mT ]
The optimizing agent The transported density Symmetric N-player games
Plan
I The optimizing agent
I The transported density
I Symmetric N-player games
The optimizing agent The transported density Symmetric N-player games
Symmetric N-player games
We can look at games with N players and see what happens whenN → +∞.
We assume all players are identical, choose an action xi ∈ Y . Costfunction for player i is
Ji (xi , x−i ) = F (xi , x−i )
F is symmetric in the sense that for all permutations σ on N − 1,F (xi , x−i ) = F (xi , (xσ(j))j 6=i ).
If N → +∞, then the number of variables in F also goes toinfinity...
The optimizing agent The transported density Symmetric N-player games
Ashes to ashes, points to measures
The trick: map positions x = (x1, ..., xN) ⊂ Y of N players to theempirical measure mN
x ∈ P(Y )
mNx =
1
N
∑i∈N
δxi
Then if x = y up to a permutation, mNx = mN
y .
(Some like to think of x and y as belonging to the quotient spaceY /GN)
The optimizing agent The transported density Symmetric N-player games
Distances on P(Y )
Who says limits says distances, and we have a couple options forour space of measures P(Y ).
Let m, m in P(Y ). Define the 1-Wasserstein distance by
W1(m, m) =
{inf∫Y×Y d(x , y) γ(dx , dy)
γ ∈ Π(m, m)
Π(m, m) is the set of couplings between m and m, i/e jointdistributions whose marginals are m and m.
Connections with optimal transport here...
The optimizing agent The transported density Symmetric N-player games
Deblais et remblais
Problem: X and Y have the same volume, find the cheapesttransportation plan between them.
Figure: Pulled directly from Topics on Optimal Transportation, C. Villani
The optimizing agent The transported density Symmetric N-player games
W1 for empirical measures
To nail our transition from Y to P(Y ), we show our distancestranslate naturally.
Take x and y in Y N , define d a distance such that
d(x , y) = minσ∈GN
1
N
∑i∈N
d(xi , yσ(i))
Then W1(mNx ,m
Ny ) = d(x , y) (uses Birkhoff’s theorem on
bistochastic matrices).
The optimizing agent The transported density Symmetric N-player games
Looking back
I We started from a space Y of players’ positions.
I We moved from Y to P(Y ) by using empirical measures.
I We defined a distance on the space P(Y ), the 1-Wassersteindistance.
I This distance metrizes the weak topology on P(Y ), i/e ifmn ⇀
∗ m⇒W1(mn,m)→ 0.
⇒ We can take limits in P(Y )!
The optimizing agent The transported density Symmetric N-player games
The limit of functions with N variables
Let FN : Y N −→ R symmetric, FN(x) = FN(mNx ). Under nice
assumptions, there is a function F ∈ C (P(Y )) such that:
supx∈Y N
|FN(mNx )− F (mN
x )| −→N→∞ 0
The optimizing agent The transported density Symmetric N-player games
Back to games
We now have a strong conceptual background to look at limits ofgames.
Assume each player has cost function of the form FNi (x1, ..., xN).
Our previous theorem says we can send N to infinity.
Asymptotically, players’ costs will be close to
F(xi ,
1
N − 1
∑j 6=i
δxj
)for some F : Y × P(Y )→ R.
The optimizing agent The transported density Symmetric N-player games
Nash equilibrium in mean field
We arrive at our mean field theorem.
Mean field equation
Assume that for all N, XN = (xN1 , ..., xNN ) is a Nash equilibrium for
the game (FN1 , ...F
NN ). Then, up to a subsequence, the sequence of
empirical measures (mN) converges to a measure m such that∫YF (y , m) dm(y) = inf
m∈P(Y )
∫YF (y , m) dm(y)
We did the reasoning for pure strategies, can extend to mixed⇒ guarantees existence of NE.
The optimizing agent The transported density Symmetric N-player games
Where to go
Once we have all this, we can tie up the two parts (derivation ofthe MFG system and symmetric N-player games).Assume N players, each one controls system in the form∫ T
0Li (x1(t), ..., xN(t), αi (t))dt + gi (x1(T ), ..., xN(T ))
Can assume
I Li (x1(t), ..., xN(t), αi (t)) = |α|22 + F
(1
N−1∑
j 6=i δxj
)I gi (x1(T ), ..., xN(T )) = g
(xi ,
1N−1
∑j 6=i δxj
)
The optimizing agent The transported density Symmetric N-player games
Nash equilibrium
We assume there is a map UNi such that
UNi (xi , t, (xj)j 6=i ) = UN(xi , t, (xj)j 6=i )
and it satisfies a bunch of HJB equations∂UNi
∂t + 12
∣∣∣∇xiUNi
∣∣∣2 − F(
1N−1
∑j 6=i δxj
)+∑
j 6=i ∇xjUNj · ∇xjU
Ni = 0
UNi = gi , t = T
In other words, we have a direction for the player to follow, andthe optimal control αi = −∇xiU
Ni is a Nash equilibrium (can prove
that).
The optimizing agent The transported density Symmetric N-player games
The optimizing agent The transported density Symmetric N-player games
Suspended disbelief
We know we can push N to infinity and the symmetric functionUN will have a limit U, but we need to do it in the space ofprobabilities P(Y ).
Using estimates on ∇xiUN , we can switch to that space (actually
using the 2-Wasserstein distance...)
We then give ourselves the initial measure m0 and push it forwardusing the obtained U (and its derivatives).
The optimizing agent The transported density Symmetric N-player games
Resulting system
With a bit more work we obtain:−∂u
∂t+ 1
2|Du|2 = F (m)
∂m∂t− div (mDu) = 0
m|t=0 = m0, u(x ,T ) = g(x ,m(T ))
The optimizing agent The transported density Symmetric N-player games
References
Good references out there, I used:
I P.L Lions videos at the College de France (some are inEnglish)
I P. Cardaliaguet’s notes (online)
I Lecture notes from class taught by G. Carlier in Dauphine
Cocorico!
Thank you!