View
215
Download
0
Category
Preview:
Citation preview
Tokyo Institute of Technology, Japan
Yu Nishiyama and Sumio Watanabe
Theoretical Analysis of Accuracy of Gaussian Belief Propagation
Background
Belief propagation (BP)
The algorithm which computesmarginal distributions efficiently
),,,,()( 21
1
dxxx
ii xxxpxpdi
Marginal distribution:
requires huge computational cost.
1d
Variety of Research Areas
(i) Probabilistic inference for AI
(ii) Error correcting code (LDPC, Turbo codes)
(iii) Code division multiple access (CDMA)
ex.
(iv) Probabilistic image processing
000101 000111 000101correctingnoise
degradeimage
restoredimage
restore
Properties of BP & Loopy BP (LBP)
Tree-structured target distribution
Exact marginal probabilities
Loop-structured target distribution
Convergence?Approximate marginal probabilities
Y. Weiss,”Correctness of belief propagation in graphical models witharbitrary topology”, Neural Computation 13(10), 2173-2200, 2001.
T. Heskes, ”On the Uniqueness of Loopy Belief Propagation Fixed Points”,Neural Computation 16(11), 2379-2414, 2004.
Ex.
PurposeWe analytically clarify the accuracy of LBP when the target distribution is a Gaussian distribution.
What is the conditions for LBP convergence?
How close is the LBP solution to the true marginal distributions?
K. Tanaka, H. Shouno, M. Okada, “Accuracy of the Bethe approximation for hyperparameter estimation in probabilistic image processing”,J.phys. A, Math. Gen., vol.37, no.36, pp.8675-8696, 2004.
In Probabilistic image processing
Table of Contents
・ BP ＆ LBP
・ Gaussian Distribution
・Main Results
(i) Single Loop
(ii) Graphs with Multi-loops
・ Conclusion
Graphical Models
Bij
jiij xxWZ
p}{
),(1
)(x
Target distribution
1x2x
4x 5x
6x
12W
23W
34W
35W56W
Marginal distribution
3\ }{
33 ),(1
)(x Bij
jiij xxWZ
xpx
53 ,\ }{
5335 ),(1
),(xx Bij
jiij xxWZ
xxpx
3x
BP ＆ LBP
ix jxijW
)()()(
iiNk
ikii xMxp
)()(),(
\)(\)(jjk
ijNkij
jiNkiikjiij xMWxMxxp
Marginal distribution
ix
1kx
2kx
)(iN)(iN )( jN
1kx 1k
x
How are messages decided?)( jji xM
)( jji xM Messages are decided by the fixed-points
of a message update rule:
.)(),()(}\{)(
)()1(
jiNk
itikji
xijj
tji xMxxWxM
i
1x 2x
4x 3x
)(21
tM
)(23
tM
)1(42
tM
)(43
tM
)(41
tM )1(24
tM
)1(32
tM
)1(34
tM
)(21
tM
)(24
tM
)(34
tM
)1(23
tM )(32
tM
)1(43
tM
)1(21
tM
)(14
tM )1(41
tM
)(12
tM )1(12
tM
)(41
tM )(42
tM )(42
tM
)(43
tM
)1(14
tM
)(23
tM
)(24
tM If it converges 1x 2x
4x 3x
*12M*
21M
*32M
*23M
*43M*
34M
*41M
*14M *
24M
*42M
a fixed-point
Gaussian Distribution
jxix
mx
lx)(tjiM
)(tijM
)(tilM
)(timM
)(til
)(tij
)(tim
)(tji
Messages: ),0()( )()( tjij
tji NxM
}\{)(
)(
2)1(
~~
jiNk
tikii
ijii
tji s
ss
Update rule:
Target distribution: ),( SN 0 ( Inverse covariance matrix):S
}\{)(
)()1( )(),()(jiNk
itikji
xijj
tji xMxxWxM
i
Fixed-Points of Messages 1x
2x
3x
dx
Single loop
When a Gaussian distribution forms a single loop, the fixed-points of messages are given by
,2 1,1
2,12,11,1,*1
ii
iiiiiiiiii
Dss),,0()( *
11*
1 iiiii NxM
Theorem1
,det4)1()(det 123122 SsssSD d
d
,2 1,1
2,12,11,1,*1
ii
iiiiiiiiii
Dss),,0()( *
11*
1 iiiii NxM
where }{ , ji are the cofactors.
LBP Solution
Theorem 2
The solution of LBP is given by
,det
4)1(1
det 12312*
S
sssS dd
iii
),,0( **
ii Np
),,( *1,
*1, iiii SNp 0 ,
1,1
1,1,
1,,
1,
*1,
ii
iiii
iiii
ii
ii Es
sE
S
where .2
det1,1,1,
iiiiii s
DSE
Intuitive Understanding
S
sssS dd
iii det
4)1(1
det 12312*
LBP Solution True
ii
S
det
12s 12s1x 2x
4x
23s41s1x 2x
4x
41s023 s
Loop Tree
Loopy Belief Propagation Belief Propagation
3x3x34s 34s
Accuracy of LBP
.det
4)1( 12312
S
sss dd
The Kullback-Leibler (KL) distances are calculated as
,1log2
11
2
1
2
1)||( * ii ppKL
where is given by
Solution of LBP True marginaldensity
.1det*
ii
i
SConvergence condition is 1 since
Theorem 3
},,,1{ di
Graphs with Multi-Loops
1x2x
3x
4x 5x
6x
Multi-loops
How about the graphshaving arbitrary structures?
We clarify the LBP solution at small covariances.
We derive the expansions ,sw. r. t. where inverse .diagonaloffdiagonal sSSS covariance matrix is
A Fixed-Point of Inverse Variances
A fixed-point of inverse variances satisfies the following system of equations:
,4
12
2||22
iNj ij
ji
i
iii
sssN }.,,1{ di
The solution of the system is expanded as
),()( 422
)(1
sOss
sss
jj
jid
ijiii
}.,,1{ di
Theorem 4
Comparison with true inverse variances
Expansions of LBP solution are
),()( 422
)(1
sOss
sss
jj
jid
ijiii
True inverse variances are
}.,,1{ di
),(det 432
2
)(1
sOsss
ss
Si
jj
jid
ijii
ii
}.,,1{ di ],})(){()([3
3131iioiidod
iii SStrSStrs
Accuracy of LBP
Theorem 5
The Kullback-Leibler (KL) distances are expanded as
),(4
)||( 762
2* sOs
sppKL
ii
iii
Solution of LBP True marginaldensity
where
],})(){()([3
3131iioiidod
iii SStrSStrs
}{ i are
},,,1{ di
}.,,1{ di
Conclusion We analytically clarified the accuracy of
LBP in a Gaussian distribution.
(i) For a single loop, we revealed the parameter that determines the accuracy of LBP and the condition
that tells us when LBP converges. (ii) For arbitrary structures, we revealed the
expansions of LBP solution at small covariances and the accuracy.
These fundamental results contribute to understanding the theoretical properties underlying LBP.
Recommended