Partial gradient flows in mean field games and statistical ... · Outline 1 Gradient flows General introduction Gradient flows examples 2 Vaccination (mean field) games 3 Computing

Partial gradient flows in mean field games andstatistical learning by GANs

Gabriel Turinici

CEREMADE, Universite Paris DauphineInstitut Universitaire de France

Groupe de Travail Statistique NumeriqueParis Dauphine May 20, 2019

Gabriel Turinici (CEREMADE & IUF) Equilibrium flows and GAN flows Paris Dauphine, May 20, 2019 1 / 30

Outline

1 Gradient flowsGeneral introductionGradient flows examples

2 Vaccination (mean field) games

3 Computing the equilibrium

4 MFG numerical schemes on metric spaces: theoretical results

5 GAN and equilibrium flows

6 Perspectives


Gradient flows: theory• F : Rd → R = a smooth convex function, x ∈ Rd ; gradient flow from x= a curve (xt)t≥0: x ′t = −∇F (xt) for t > 0, x0 = x .• Polish metric space (X , d), functional F : (X , d)→ R ∪ +∞:non-trivial defintion, huge litterature (cf. books by Ambrosio et al. ,Villani, Santambroggio).

• Euclidian space (under some regularity assumptions):

ddt F (xt) = 〈∇F (xt), x ′t〉 ≥ −

∣∣x ′t ∣∣ · |∇F | (xt) ≥ −12∣∣x ′t ∣∣2 − 1

2 |∇F |2 (xt),

or equivalently ddt F (xt) + 1

2∣∣x ′t ∣∣2 + 1

2 |∇F |2 (xt) ≥ 0,

with equality only if x ′t = −∇F (xt).• Conclusion: d

dt F (xt) + 12 |x′t |

2 + 12 |∇F |2 (xt) ≤ 0 a.e. is equivalent with

x ′t = −∇F (xt).


Gradient flows: theory

• Euclidian space formulation: ddt F (xt) + 1

2 |x′t |

2 + 12 |∇F |2 (xt) ≤ 0 a.e.

• the (local metric) slope of F at x :|∇F | (x) = lim sup

z→x[F (x)−F (z)]+

d(x ,z) = max

lim supz→x

F (x)−F (z)d(x ,z) , 0

.

Definition: let I an interval in R. A function f : I → (X , d) is absolutelycontinuous on I if for any ε > 0, there exists δ > 0 such that whenever afinite sequence of pairwise disjoint sub-intervals (xk , yk) of I withxk , yk ∈ I satisfies

∑k(yk − xk) < δ then

∑k d(f (yk)− f (xk)) < ε.

• the metric derivative of x at t: |x ′t | = limh→0d(xt+h,xt )|h| , exists a.e. as

soon as t 7→ xt is absolutely continuous. Moreover |x ′| ∈ L1(0, 1).


Gradient flows: theory

• EDI ∇-flow (pointwise): ddt F (xt) + 1

2 |x′t |

2 + 12 |∇F |2 (xt) ≤ 0 a.e.

• EDI ∇-flow from x : an absolutely continuous curve such that:

∀s ≥ 0, F (xs) + 12

∫ s

0

∣∣x ′r ∣∣ dr + 12

∫ s

0|∇F |2 (xr ) dr ≤ F (x),

a.e. t > 0, ∀s ≥ t, F (xs) + 12

∫ s

t

∣∣x ′r ∣∣ dr + 12

∫ s

t|∇F |2 (xr )dr ≤ F (xt).

• EVI form for λ-convex (i.e., when smooth F ′′ ≥ λId ...) functionals:F (xt) + 1

2ddt d2(xt , y) + λ

2 d2(xt , y) ≤ F (y), ∀y , a.e. t ≥ 0.


Gradient flows examples: heat flow (Fokker-Planck)

X= P2(R) (the set of probability measures on (R,B(R)) with finitesecond-order moment, endowed with the Wasserstein distance W2)Consider for σ ∈ R F : P2(R)→ R ∪ +∞:F (ν) =

∫R V (x)ρ(x) + σ2

2∫R ρ(x) log(ρ(x))dx , if ν dx , ν = ρ(x)dx

F (ν) = +∞, if ν / dx .For smooth V , the gradient flow t 7→ ν(t) ∈ P2(R) of F satisfiesν(t) = ρ(t, ·)dx and:

∂ρ

∂t (t, x) = ∂

∂x [V ′(x)ρ(t, x)] + σ2

2∂2ρ

∂x2 (t, x), (1)

i.e., Fokker-Planck of the SDE: dX (t) = −V ′(X (t))dt + σdW (t).

Remark: also a L2 flow (term∫|∇ρ|2)...


Gradient flows examples: heat flow (Fokker-Planck)

-5 -4 -3 -2 -1 0 1 2 3 4 5

x

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

;0(x

)

initial data ;0(x)

Figure: Initial data for the heat flow (FP) model and its evolution (VIDEO).


Gradient flows examples : a 1D Patlak-Keller-Segel model

• the (modified) Patlak–Keller–Segel system (Perthame-Calvez-SharifiTabar 2007, Blanchet-Calvez-Carrillo 2008), is a PDE model fordiffusion-aggregation competition in biological applications (chemotaxis).

• Free energy functional:

G[ρ] =∫ρ(t, x) log(ρ(t, x)) dx + χ

π

∫ ∫ρ(t, x)ρ(t, y) log |x − y |dxdy

• the resulting Patlak-Keller-Segel equation:

∂ρ∂t = ∆ρ−∇(χρ∇c), t > O, x ∈ Ω ⊂ Rd (2)

c = − 1dπ log |z | ? ρ (3)

ρ = cell density, c = concentration of chemo-attractant, χ = sensitivityof the cells to the chemo-attractant.


Gradient flows examples: 1D Patlak-Keller-Segel model

-4 -2 0 2 4x

0

0.2

0.4

0.6

0.8

;E

VIE

(x)

t=0

;EVIE(x)

-5 0 5 10x

0

0.5

1

1.5

2

2.5

;E

VIE

(x)

t=0

;EVIE(x)

Figure: Initial data for the PKS model: χ = π (left), χ = 1.9π (right) and its evolution(VIDEO T = 2). Implementation : G. Legendre; ∇-flow JKO PKS code : courtesy A. Blanchet.


Gradient flows: the JKO scheme

• Jordan, Kinderlehrer and Otto ’98, (JKO) numerical scheme: time step= τ > 0, x τ0 = x ∈ X , by recurrence x τn+1 = a minimizer of the functional

x 7→ PJKOF (x ; x τn , τ) := 1

2τ d2(x τn , x) + F (x). (4)

• If X= Hilbert, F = smooth, JKO = implicit Euler (IE) scheme, i.e.,xτ

n+1−xτn

τ = −∇F (x τn+1).• JKO scheme was initially used theoretically to prove the existence of agradient flow

• similar schemes (VIM, EVIE) were proposed to compute numerically thesolution at second order in time (Legendre, G.T. 2017)


Theoretical results for the JKO scheme

Consider division ∆ = [0, τ, 2τ, ....T = Nτ ] of [0,T ].

Short computation: x τn+1 = argminx1

2τ d2(x τn , x) + F (x). Thus1

2τ d2(x τn , x τn+1) + F (x τn+1) ≤ F (x τn )

Summing over n this gives a bound∑

n1

2τ d2(x τn , x τn+1) ≤ F (x0)− F (x τN)

In fact on a time step:

ddt

(F (x τn+t

n ) + d2(x , x τn+tn )

2t

)= −d2(x , x τn+t

n )2t2

This is a key argument to prove compactness of the numerical curves x τnwhen the size of divisions |∆| → 0.


Outline






6 Perspectives


Disclaimer:

What follows is a THEORETICAL epidemiological investigation. It is notmeant to be used directly for health-related decisions; if in need to takesuch a decision please seek professional medical advice.


Vaccine scares: MFG modelsInfluenza A (H1N1) (flu) (2009-10)• At 15/06/2010 flu (H1N1): 18.156 deaths in 213 countries (WHO)• France: 1334 severe forms (out of 7.7M-14.7M people infected)

Countries Official target coverage Effective rate of vaccinationGermany 100 % 10%Belgium 100 % 6 %

Spain 40 % < 4%France 70 - 75 % 8.5 %Italy 40 % 1.4 %

Previous vaccine scares (some have been disproved since):• France: hepatitis B vaccines cause multiple sclerosis• US: mercury additives are responsible for the rise in autism• UK: the whooping cough (1970s), the measles-mumps-rubella (MMR)

(1990s).


Individual dynamics

Susceptible Infected Recovered

Vaccinated

−dV

−βSIdt −γIdt

Global dynamics : con-tinuous time determinis-tic ODE; is the masterequation of the individ-ual dynamics.

Susceptible Infected Recovered

Vaccinated

...

rate βI rate γ

Individual dynamics:continuous time Markovjumps between ’Sus-ceptible’, ’Infected’,’Recovered’ and’Vaccinated’ classes.


Summary for MFG vaccination models

• cost has the structure C(individual , societal); it is to be optimized withrespect to the ’individual ’ strategy, the ’societal ’ remains fixed, i.e.individual 7→ C(individual , societal);

• Nash / MFG equilibrium when ’individual ’ is unilaterally optimal and’societal ’= ’individual ’ (similar to a fixed point);

• benevolent planner approach: minimize individual 7→C(individual , individual);

• mean field framework suggests some regularity with respect to ’societal’argument;

• in general neither a benevolent planner game (cost of anarchy ...), nor azero-sum game.


Outline






6 Perspectives


Computing the equilibrium: semi-explicit schemes onmetric spaces

• How to find and equilibrium ?• JKO converges to a ”benevolent planner” perspective.• semi-explicit numerical scheme for C(individual = ξI , global = ξG):Algorithm: set ξk = ξG

k = ξIk , and

ξk+1 ∈ argminη∈ΣN+1

dist(η, ξk)2

2∆τ + C(η, ξk).

• related to ”best reply” (MFG: cf. G. Carlier, A. Blanchet,...) and”fictitious play” (MFG: cf. P. Cardialiaguet et al.) learning methods ingame theory.


Computing MFG equilibrium

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

time

0

1

2

3

4

5

6

7

8

9

9(t)

#10 -3 initial strategy 9 , iter=0/1000 nonvacmass=100%

9(t)

Figure: Notation: ξτ (t) is a time-dependent probability law over the possible vaccination timesindexed by variable t. Initial data ξτ=0(t) (uniform) and iterations (VIDEO) of the vaccinationMFG equilibrium strategy ξτ . Case: short persistence.

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

time

0

1

2

3

4

5

6

7

8

9

9

#10 -3


Outline






6 Perspectives


MFG numerical schemes on metric spaces: theoreticalresults (GT ’17)

• Question: explicit ∆τ → 0 has a meaning ?

• Hilbert space: ∂τξ(τ, t) +∇1C(ξ(τ, t), ξ(τ, t)) = 0; metric spaceequivalent ?

• give a meaning in a metric space to:∂τξ(τ, t) +∇1C(ξ(τ, t), ξ(τ, t)) = 0;• literature: ∇-flows for E (t, x): Ferreira-Valencia-Guevara ’15,Rossi-Mielke-Savare ’08, C. Jun ’12, Kopfer-Sturm ’16


Theoretical results for the equilibrium flows

Consider division ∆ = [0, τ, 2τ, ....T = Nτ ] of [0,T ].

Short computation:

ξk+1 ∈ argminηdist(η, ξk)2

2∆τ + C(η, ξk).

Thus 12τ d2(ξk+1, ξk) + C(ξk+1, ξk) ≤ C(ξk , ξk)

Summing over k this gives:∑k

12τ d2(ξk+1, ξk) ≤ −

∑k C(ξk+1, ξk)− C(ξk , ξk)

Bound when |∆| → 0?


MFG numerical schemes on metric spaces: theoreticalresults (GT ’17)For x = (xt)t∈[0,T ] an absolutely continuous curve define:

Υ(x , C, a, b) = lim inf|tk+1−tk |→0

∑kC(xtk+1 , xtk )− C(xtk , xtk ). (5)

• EDI (pointwise) formulation:∀τ1 ≥ 0 : Υ(ξ, C, 0, τ1) + 1

2∫ τ1

0 |x ′r | dr + 12∫ τ1

0 |∇1C|2 (xr , xr ) dr ≤ 0a.e.τ1 > 0,∀τ2 ≥ τ1 :Υ(ξ, C, τ1, τ2) + 1

2∫ τ2τ1|x ′r | dr + 1

2∫ τ2τ1|∇1C|2 (xr , xr ) dr ≤ 0.

does not use convexity but uses regularity hypothesis for C.• EVI formulationC(ξτ , ξτ ) + 1

2d

dτ d2(ξτ , y) + λ2 d2(ξτ , y) ≤ C(y , ξτ ), ∀y , a.e. τ ≥ 0.

does not use much regularity uses λ-convexity.

• both are the limit of numerical schemes (under hyp.)


Theoretical results for the equilibrium flows

Bound when |∆| → 0 for∑

∆ C(ξk+1, ξk)− C(ξk , ξk) ?Finer division: what changes from x ,y when add z between ?C(y , x)− C(x , x)− [C(z , x)− C(x , x) + C(y , z)− C(z , z)]arg1

arg2

x

x

z

z

y

y

-

+

-

+ -

+

arg1

arg2

x

x

z

z

y

y

+

-

-

+

Requirement: C(x , y) + C(z , z)− C(z , y)− C(x , z) of order 2 ind(x , z) + d(z , y), formulated as:|C(x , y) + C(z , z)− C(z , y)− C(x , z)| ≤ CLd(x , z) · d(z , y), ok in bi-linearsetting.Right term only need to be of order O(d1+ε) or even an equivalent of uniform continuity ... asin Riemann sums.


Outline






6 Perspectives


Generative Adversarial Networks and equilibrium flows

~z ~xsyntheticG(~z)

generator / actor

pθ(~z)

~xrealpdata(~x)

~x real?D(~x)

discriminator / critic

Image credits : adapted from Petar Velickovic

• Notations: actor /generator θ → g(θ)→ Pθ• critic / discriminator (Wasserstein GAN formulation of Martin Arjovsky,Soumith Chintala, and Leon Bottou) : µ→ fµ• Functional 〈fµ,Pθ − Pr 〉 (1-Waserstein distance in dual form) to bemaximized with respect to µ in order to find (an approximation of)dW1(Pθ,Pr ). Then this is minimized with respect to θ.• WGAN formulation: nc steps of the SGD (or other stochastic descentlike Adam, RMSProp, ...) for µ and 1 step for θ.


Generative Adversarial Networks and equilibrium flows• F (µ, θ) = 〈fµ,Pθ − Pr 〉 =

∫fµ(x)Pθ(dx)−

∫fµ(x)Pr (dx)

• next critic µn+1 = arg minµ d(µ,µn)2

2τ − F (µ, θn)• next actor θn+1 = arg minθ d(θ,θn)2

2τ + F (µn+1, θ)• In particular d(µn+1,µn)2

2τ − F (µn+1, θn) ≤ −F (µn, θn)d(θn+1,θn)2

2τ + F (µn+1, θn+1) ≤ F (µn+1, θn)• We are accumulating the index:F (µn+1, θn)− F (µn, θn) + F (µn+1, θn)− F (µn+1, θn+1)• difference w/r to introducing a new point (µ′, θ′) between (µn, θn), and(µn+1, θn+1)

θ

µ

θn

µn

θn+1

µn+1

+- +

-

θ

µ

θn

µn

θn+1

µn+1

θ′

µ′

2- 2+

2+ 2-


Generative Adversarial Networks and equilibrium flows

• same if several steps are taken for the critic, as the same quantityaccumulates F (µn+1, θn)− F (µn, θn)• Assumption: similar to previously, i.e.,|F (µ2, θ1) + F (µ1, θ2)− F (µ1, θ1)− F (µ2, θ2)| ≤ CL · d(µ1, µ2) · d(θ1, θ2),ok in our bi-linear setting (distances Lip(·)× dW1).


Outline






6 Perspectives


Perspectives

• SGD, discretization, ...


Documents

Partial gradient flows in mean field games and statistical ... · Outline 1 Gradient flows General introduction Gradient flows examples 2 Vaccination (mean field) games 3 Computing