Upload
yuki-soma
View
434
Download
2
Embed Size (px)
Citation preview
PRML 10.1 ~ 10.3 7/31
Yuki Soma
10.1 Variational Inference . . . . . . . . . . . . . . . . . . . . . . 462 ◦ 10.1.1 Factorized distributions . . . . . . . . . . . . . . . . . . . . 464 ◦ 10.1.2 Properties of factorized approximations . . . . . . . . . . . 466 ◦ 10.1.3 Example: The univariate Gaussian . . . . . . . . . . . . . . 470 ◦ 10.1.4 Model comparison . . . . . . . . . . . . . . . . . . . . . . 473
10.2 Illustration: Variational Mixture of Gaussians . . . 474 ◦ 10.2.1 Variational distribution . . . . . . . . . . . . . . . . . . . . 475 ◦ 10.2.2 Variational lower bound . . . . . . . . . . . . . . . . . . . 481 ◦ 10.2.3 Predictive density . . . . . . . . . . . . . . . . . . . . . . . 482 ◦ 10.2.4 Determining the number of components . . . . . . . . . . . 483 ◦ 10.2.5 Induced factorizations . . . . . . . . . . . . . . . . . . . . 485
10.3 Variational Linear Regression . . . . . . . . .. . . . . . 486 ◦ 10.3.1 Variational distribution . . . . . . . . . . . . . . . . . . . . 486 ◦ 10.3.2 Predictive distribution . . . . . . . . . . . . . . . . . . . . 488 ◦ 10.3.3 Lower bound . . . . . . . . . . . . . . . . . . . . . . . . . 489
𝐗 𝐙 𝑝(𝐙|𝐗) 𝐙
◦
◦ 11 MCMC
◦
◦ EP 10.7
𝐙
1.
◦ 𝑞 𝐙 = 𝑞(𝐙|𝛚) 𝛚
2. 𝐙
◦
◦ 𝑞𝑖 𝐙𝑖 = 𝑞𝑖
◦
◦ 𝐙
𝐗
𝑝(𝐗, 𝐙)
𝑝(𝐙|𝐗) 𝑝(𝐗)
◦
ℒ(𝑞) 𝐾𝐿(𝑞| 𝑝 = 0 𝑞 𝐙 = 𝑝(𝐙|𝐗)
10.5 ℒ(𝑞)
𝒒𝒊
10.5 ℒ(𝑞)
KL
𝑞𝑗 = 𝑝 𝐗, 𝐙𝐣
ℒ(𝑞) 𝑞𝑗∗ 𝑗 = 1,… ,𝑀
𝑞𝑖 𝑖 ≠ 𝑗
1. 𝑞𝑗
2.
foreach 𝑞𝑖 :
𝑞𝑖 𝑞𝑗 𝑞𝑖
𝑞𝑖
◦ ℒ(𝑞) 𝑞𝑖
𝐙
KL
KL 𝑝(𝐙) 0 𝑞(𝐙)0
KL 𝑞(𝐙) 𝑝(𝐙)
KL 1
1
𝑝(𝑚) 𝑚 ◦ 𝑝(𝑚|𝐗)
𝑞 𝐙,𝑚 = 𝑞 𝐙 𝑞(𝑚) ◦ 𝐙
⇒ 𝑞 𝐙,𝑚 = 𝑞 𝐙|𝑚 𝑞(𝑚)
10.10
ln 𝑝 𝐗 = ℒ − 𝑞 𝑍 𝑚 𝑞 𝑚 ln𝑝 𝑍,𝑚 𝑋
𝑞 𝑍 𝑚 𝑞 𝑚𝐙𝑚
◦ ℒ = 𝑞 𝑍 𝑚 𝑞 𝑚 ln𝑝(𝑍,𝑋,𝑚)
𝑞 𝑍 𝑚 𝑞 𝑚𝐙𝑚
ℒ 𝑞(𝑚)
◦ 𝑞(𝑚) ∝ 𝑝(𝑚)𝑒ℒ𝑚 (10.36)
ℒ𝑚 = 𝑞 𝑍 𝑚 𝑞 𝑚 ln𝑝(𝑍,𝑋|𝑚)
𝑞 𝑍 𝑚𝐙
ℒ𝑚 𝑞(𝑍|𝑚) (10.36)𝑞(𝑚)
◦ 𝐾
𝐾
𝛑 1
◦ 𝐳𝑖 1-of-𝐾 𝐾
𝐙
𝜋
𝜇, Λ
◦ 𝐦0 = 𝟎
◦ 𝑝 𝑞
◦ PRML
1. E 𝑧𝑛𝑘 = 𝑟𝑛𝑘
2. 𝑞∗(𝜋, 𝜇, Λ)
3. 𝑞∗ 𝐙 ◦ 𝑟𝑛𝑘
4. 2.
0 ◦
𝛼0 < 1 ◦ 𝛼0 = 10−3
◦
◦
◦
◦
t
10.81 ◦ 𝑁
𝑞 𝜋 𝑞(𝜇, Λ)
𝐾 𝐾!
K
ln 𝐾!10.22
𝜋
𝜋 𝑞 …
0
◦ RVM 7.22
3.3
𝛼, 𝛽
◦ 𝛽
10.6
…
10.9 𝛼
𝛼
𝑞