Embed Size (px)
Citation preview
Variational Bayes and VariationalMessage Passing
Mohammad Emtiyaz Khan
CS,UBC
Variational Bayes and Variational Message Passing – p.1/16
Variational Inference
Find a tractable distribution Q(H) that closely approximatesthe true posterior distribution P (H|V ).
log P (V ) =∑
H
Q(H) log P (V )
=∑
H
Q(H) logP (H,V )
P (H|V )
=∑
H
Q(H) log
[P (H,V )
Q(H)
Q(H)
P (H|V )
]
=∑
H
Q(H) logP (H,V )
Q(H)︸ ︷︷ ︸
L(Q)
+∑
H
−Q(H) logP (H|V )
Q(H)︸ ︷︷ ︸
KL(Q||P )
Variational Bayes and Variational Message Passing – p.2/16
Variational Inference
log P (V ) = L(Q) + KL(Q||P ) (1)
L(Q) =∑
H
Q(H) logP (H,V )
Q(H)(2)
KL(Q||P ) = −∑
H
Q(H) logP (H|V )
Q(H)(3)
Find Q(H) that maximizes lower bound L(Q) (andhence minimizes KL divergence).
For Q(H) = P (H|V ), KL vanishes to zero, but P (H|V )is intractable (that’s why variational approach).
Trick : Consider a restricted class of Q(H), and thenfind the member which minimizes the KL divergence.
Variational Bayes and Variational Message Passing – p.3/16
Factorized Distributions
Q(H) =∏
i
Qi(Hi) (4)
Substituting this in the expression for lower bound,
L(Q) =∑
H
∏
i
Qi(Hi) logP (H,V )
∏
i Qi(Hi)(Outline)
=∑
H
∏
i
Qi(Hi) log P (H,V )−∑
H
∏
i
Qi(Hi)∑
i
log Qi(Hi)
=∑
H
∏
i
Qi(Hi) log P (H,V )−∑
i
∑
Hi
∏
i
Qi(Hi) log Qi(Hi)
=∑
H
∏
i
Qi(Hi) log P (H,V ) +∑
i
H(Qi)
Variational Bayes and Variational Message Passing – p.4/16
Factorized Distributions
Now separate out all the terms in one factor Qj.
L(Q) =∑
Hj
Qj(Hj)〈log P (H,V )〉∼Qj(Hj)︸ ︷︷ ︸
log Q∗
j (Hj)
+ H(Qi) +∑
i6=j
H(Qi)
= −KL(Qj ||Q∗j) + terms not in Qj (5)
This bound is maximized wrt Qj when
log Qj(Hj) = log Q∗j(Hj) = 〈log P (H,V )〉∼Qj(Hj) + c (6)
Now iterate, guaranteed convergence ...
Variational Bayes and Variational Message Passing – p.5/16
Variational Bayes for Bayesian Networks
log Q∗j(Hj) = 〈log P (H,V )〉∼Qj(Hj) + c
=∑
i
〈log P (Xi|pai)〉∼Qj(Hj)+ c
= 〈log P (Hj |paj)〉∼Qj(Hj)
+∑
k∈chj
〈log P (Xk|paj)〉∼Qj(Hj) + c
Variational Bayes and Variational Message Passing – p.6/16
Exponential-Conjugate Models
P (Y |θ) = exp[φTY (θ)u(Y ) + f(Y ) + g(θ)] (7)
u(Y ) = Natural statistics (8)
φY (θ) = Natural Parameter vector (9)
g(θ) = Constant of integration (10)
Example I: Bernoulli Distribution
p(x|µ) = µx(1− µ)1−x (11)
log p(x|µ) = x log µ + (1− x) log(1− µ) (12)
= logµ
(1− µ)︸ ︷︷ ︸
φ(µ)
x︸︷︷︸
u(x)
+ log(1− µ)︸ ︷︷ ︸
g(µ)
(13)
Variational Bayes and Variational Message Passing – p.7/16
Exponential-Conjugate Models
P (Y |θ) = exp[φTY (θ)u(Y ) + f(Y ) + g(θ)] (14)
P (Y |φ) = exp[φT u(Y ) + f(Y ) + g̃(φ)](Re-parametrization)
Property I: 〈u(Y )〉P (Y |θ) = −dg̃(φ)dφ
log p(x|µ) = logµ
(1− µ)︸ ︷︷ ︸
φ(µ)
x︸︷︷︸
u(x)
+ log(1− µ)︸ ︷︷ ︸
g(µ)
(15)
φ = logµ
(1− µ)⇒ µ =
eφ
1 + eφ(16)
g(µ) = log(1− µ) = − log(1 + eφ) = g̃(φ) (17)
E(x) = 〈u(Y )〉 = eφ(1 + eφ)−1 = µ (18)
Variational Bayes and Variational Message Passing – p.8/16
Exponential-Conjugate Models
P (Y |θ) = exp[φTY (θ)u(Y ) + f(Y ) + g(θ)] (19)
Example II: Gaussian Distribution θ → Y → X ← β
p(Y |θ) = (2π)−1/2 exp−1
2(Y −θ)2
log p(Y |θ) = [θ,−1/2]︸ ︷︷ ︸
φY (θ)
[
Y
Y 2
]
︸ ︷︷ ︸
uY (Y )
−1
2θ2
︸︷︷︸
gY (θ)
−1
2log(2π)
︸ ︷︷ ︸
fY (Y )
p(X|Y, β) = (2π)−1/2β1/2 exp−β
2(X−Y )2
log p(X|Y, β) = [βY,−β/2]︸ ︷︷ ︸
φX(Y,β)
[
X
X2
]
︸ ︷︷ ︸
uX(X)
+−1
2(βY 2 + log β)
︸ ︷︷ ︸
gX(Y,β)
−1
2log(2π)
︸ ︷︷ ︸
fX(X)
Variational Bayes and Variational Message Passing – p.9/16
Exponential-Conjugate Models
Property II: Multi-linearity θ → Y → X ← β
log p(X|Y, β) = [βY,−β/2]︸ ︷︷ ︸
φX(Y,β)
[
X
X2
]
︸ ︷︷ ︸
uX(X)
+−1
2(βY 2 + log β)
︸ ︷︷ ︸
gX(Y,β)
−1
2log(2π)
︸ ︷︷ ︸
fX(X)
= [βX,−β/2]︸ ︷︷ ︸
φXY (X,β)
[
Y
Y 2
]
︸ ︷︷ ︸
uY (Y )
+−1
2(βX2 + log β)
︸ ︷︷ ︸
gXY (X,β)
−1
2log(2π)
︸ ︷︷ ︸
fY (Y )
log p(Y |θ) = [θ,−1/2]︸ ︷︷ ︸
φY (θ)
[
Y
Y 2
]
︸ ︷︷ ︸
uY (Y )
−1
2θ2
︸︷︷︸
gY (θ)
−1
2log(2π)
︸ ︷︷ ︸
fY (Y )
Variational Bayes and Variational Message Passing – p.10/16
Exponential-Conjugate Models
Consider Y node and it’s children in θ → Y → X ← β,
log P (Y |θ) = φTY (θ)uY (Y ) + fY (Y ) + gY (θ)
log P (X|Y, β) = φTX(Y, β)uX(X) + fX(X) + gX(Y, β)
= φTXY (X, β)uY (Y ) + gXY (Y, β)
Recall that,
log Q∗Y (Y ) = 〈log P (Y |θ)〉∼QY (Y ) + 〈log P (X|Y, β)〉∼QY (Y ) + c
= 〈φTY (θ)uY (Y ) + fY (Y ) + gY (θ)〉∼QY (Y )
+〈φTXY (X, β)uY (Y ) + gXY (Y, β)〉∼QY (Y ) + c
= 〈φTY (θ) + φT
XY (X, β)〉∼QY (Y )uY (Y ) + fY (Y ) + c1
Variational Bayes and Variational Message Passing – p.11/16
Exponential-Conjugate Models
log Q∗Y (Y ) = 〈φT
Y (θ) + φTXY (X, β)〉∼QY (Y )uY (Y ) + fY (Y ) + c1
Finally,
〈φTY (θ)〉 = [θ,−1/2]
〈φTXY (X, β)〉 = 〈[βX,−β/2]〉
Later is found using the property I (explain).
Variational Bayes and Variational Message Passing – p.12/16
Back to Bayesian Networks
Take each node, write the expression as a function ofnatural statistics of that node.
log Q∗Y (Y )
= 〈log P (Y |paY )〉∼QY (Y ) +∑
k∈chj
〈log P (Xk|paj)〉∼QY (Y ) + c
=
〈φTY (θ) +
∑
k∈chj
φTXY (X, β)〉∼QY (Y )
uY (Y ) + fY (Y ) + c1
The compute the expectation of natural statistics of eachchildren node, and use that to find the quantity in bracket.
Variational Bayes and Variational Message Passing – p.13/16
Variational Message Passing
Message from a parent node Y to a child node X:
mY →X = 〈uY 〉 (20)
Message from a child node X to a parent node Y:
mX→Y = φ̃XY (〈uX〉, {mi→X}i∈cpY) (21)
Node Y update it’s posterior Q∗Y :
φ∗Y = φ̃Y ({mi→Y }i∈paY
) +∑
j∈chY
mj→Y (22)
Variational Bayes and Variational Message Passing – p.14/16
Variational Message Passing
Variational Bayes and Variational Message Passing – p.15/16
Discussion
Initialization and message passing schedule.
Calculation of Lower Bound
Allowable Model
VIBES
Variational Bayes and Variational Message Passing – p.16/16
