Physical Fluctuomatics 7th~10th Belief propagation

Physics Fluctuomatics (Tohoku University) 1

Physical Fluctuomatics7th~10th Belief propagation

Kazuyuki TanakaGraduate School of Information Sciences, Tohoku University

[email protected]://www.smapip.is.tohoku.ac.jp/~kazu/


Textbooks

Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 8.Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Co., Ltd., October 2009 (in Japanese), Chapters 6-9.


What is an important point in computational complexity?

How should we treat the calculation of the summation over 2N configuration?

FT, FT, FT,

211 2

,,,x x x

NN

xxxf

}}

} ;,,,

F){or Tfor(

F){or Tfor( F){or Tfor(

;0

21

2

1

L

N

xxxfaax

xx

a

N fold loops

If it takes 1 second in the case of N=10, it takes 17 minutes in N=20, 12 days in N=30 and 34 years in N=40.

Markov Chain Monte Carlo MethodBelief Propagation Method This Talk


Probabilistic Model and Belief Propagation

Probabilistic Information Processing

Probabilistic Models

Bayes Formulas

Belief Propagation

J. Pearl: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan Kaufmann, 1988).C. Berrou and A. Glavieux: Near optimum error correcting coding and decoding: Turbo-codes, IEEE Trans. Comm., 44 (1996).

Bayesian Networks


Mathematical Formulation of Belief Propagation

Similarity of Mathematical Structures between Mean Field Theory and Bepief PropagationY. Kabashima and D. Saad, Belief propagation vs. TAP for decoding corrupted messages, Europhys. Lett. 44 (1998). M. Opper and D. Saad (eds), Advanced Mean Field Methods ---Theory and 　 Practice (MIT Press, 2001).

Generalization of Belief PropagationS. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energyapproximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005).

Interpretations of Belief Propagation based on Information GeometryS. Ikeda, T. Tanaka and S. Amari: Stochastic reasoning, free energy, and information geometry, Neural Computation, 16 (2004).


Generalized Extensions of Belief Propagation based on Cluster Variation Method

Generalized Belief PropagationJ. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005).

Key Technology is the cluster variation method in Statistical PhysicsR. Kikuchi: A theory of cooperative phenomena, Phys. Rev., 81 (1951).T. Morita: Cluster variation method of cooperative phenomena and its generalization I, J. Phys. Soc. Jpn, 12 (1957).


Belief Propagation in Statistical Physics

In graphical models with tree graphical structures, Bethe approximation is equivalent to Transfer Matrix Method in Statistical Physics and give us exact results for computations of statistical quantities.

In Graphical Models with Cycles, Belief Propagation is equivalent to Bethe approximation or Cluster Variation Method.

Bethe Approximation

Trandfer Matrix Method

(Tree Structures)Belief Propagation

Cluster Variation Method(Kikuchi Approximation)

Generalized Belief Propagation


Applications of Belief PropagationsImage ProcessingK. Tanaka: Statistical-mechanical approach to image processing (Topical Review), J. Phys. A, 35 (2002).A. S. Willsky: Multiresolution Markov Models for Signal and Image Processing, Proceedings of IEEE, 90 (2002).

Low Density Parity Check CodesY. Kabashima and D. Saad: Statistical mechanics of 　 low-density parity-check codes (Topical Review), J. Phys. A, 37 (2004). S. Ikeda, T. Tanaka and S. Amari: Information geometry of turbo and low-density parity-check codes, IEEE Transactions on Information Theory, 50 (2004).

CDMA Multiuser Detection AlgorithmY. Kabashima: A CDMA multiuser detection algorithm on the basis of belief propagation, J. Phys. A, 36 (2003).T. Tanaka and M. Okada: Approximate Belief propagation, density evolution, and statistical neurodynamics for CDMA multiuser detection, IEEE Transactions on Information Theory, 51 (2005).

Satisfability ProblemO. C. Martin, R. Monasson, R. Zecchina: Statistical mechanics methods and phase transitions in optimization problems, Theoretical Computer Science, 265 　 (2001).M. Mezard, G. Parisi, R. Zecchina: Analytic and algorithmic solution of random satisfability problems, Science, 297 (2002).


Strategy of Approximate Algorithm in Probabilistic Information Processing

It is very hard to compute marginal probabilities exactly except some tractable cases.

What is the tractable cases in which marginal probabilities can be computed exactly?Is it possible to use such algorithms for tractable cases to compute marginal probabilities in intractable cases?

FT, FT, FT,

21112 3

,,,x x x

LL

xxxPxP


Graphical Representations of Tractable Probabilistic Models

A B C D E

),(),(),(),( EDWDCWCBWBAW DECDBCAB

),( BAWAB ),( CBWBC ),( DCWCD ),( EDWDE

A B C D E


B C DX X X=

=



A B C D EA B C D E

A BA B C D E

B C D EX



A B C D EA B C D E

A BA B C D E

B C D EX

A B

B C D E A

B C D E



A B C D EA B C D E

A BA B C D E

B C D EX

A B

B C D E A

B C D E

A B



A B C D EA B C D E

A BA B C D E

B C D EX

A B

B C D E A

B C D E

A B

A B C D EB C D E



A B C D EB C D E



B C D E

C D EX

A B C D EB C D E

A B C



B C D E

C D EX

C D E B

C D E

A B C D EB C D E

A B C

A B C X



B C D E

C D EX

C D E B

C D E

B C

A B C D EB C D E

A B C

A B C X



B C D E

C D EX

C D E B

C D E

B C

A B C D EB C D E

A B C

A B C

B C D EC D E

X



A B C D EA B C D E



A B C D EB C D E

A B C D EA B C D E



A B C D EB C D E

B C D EC D E

A B C D EA B C D E



A B C D EB C D E

B C D EC D E

A B C D EA B C D E

C D ED E



A B C D EB C D E

B C D EC D E

A B C D EA B C D E

C D ED E

D EE



A B C E E

),(),(),(),(),( FEWEDWECWCBWCAW EFDECEBCAC


C C DX X X=

=

F

),( FEWEF

EX

A

B

EC

D

F



A

B

ECA B C D E F

D

F



A

B

ECA B C D E F

D

F

A

B

EC

D

F

B C D E F

A

A

C

A

C



A

B

ECA B C D E F

D

F

A

B

EC

C D E F

D

FA

B

EC

B C D E F

D

F

B

CBB

C



A

B

ECA B C D E F

D

F

A

B

EC

C D E F

D

FA

B

EC

B C D E F

D

F

ECD E F

D

F



A

B

ECA B C D E F

D

F

A

B

EC

C D E F

D

FA

B

EC

B C D E F

D

F

ECD E F

D

F

ECE F

D

F



A

B

ECA B C D E F

D

F

A

B

EC

C D E F

D

FA

B

EC

B C D E F

D

F

ECD E F

D

F

ECE F

D

F

EF F



Graphical Representation of Marginal Probability in terms of Messages A

B

ECA B C D F

E}Pr{

D

F




B

ECA B C D F

E}Pr{

D

F

A

B

ECA B C E

D E

FF

D

＝




B

ECA B C D F

E}Pr{

D

F

A

B

ECA B C E

D E

FF

D

＝

＝ ECE

D E

F




B

ECA B C D F

E}Pr{

D

F

A

B

ECA B C E

D E

FF

D

＝

＝ ECE

D E

FEC

D

F

＝




B

ECA B D F

EC },Pr{

D

F

A

CA E

D E

FF

D

＝

＝ ECE

D E

F＝

ECB

CB

A

C B

CA

B

EC

D

F



Graphical Representation of Marginal Probability in terms of Messages

},Pr{ ECA

B

EC

D

F

＝}Pr{E EC

D

F

＝

EC

A

B

ECC

EC

D

F

A

B

EC

D

F

C

C

ECE },Pr{}Pr{

Recursion Formulas for Messages




A

B

ECC

EC

EC

D

FE

E

FEC

D

FE

E

D

EC

D

F

E

EC

A

B

ECC

A

C

A

B

ECC

B

C

A

A

C

A

C B

CBB

C

E

DE

DD

E

FE

FF

A

B

EC

D

FStep 1

Step 2

Step 3




Step 1

Step 2

Step 3

A

B

EC

D

F

A

B

EC

D

F

A

B

EC

D

F},Pr{ EC

A

B

EC

D

F＝

}Pr{E EC

D

F＝

}Pr{BB

C

＝

},Pr{ CAA

B

EC＝


Belief Propagation

ab

1

cd

2

3 4

56

2221}2,1{11

21

,,,,,,,,,,

xWxWxxWxWxWxxP

CBA dcbadcba

a3xc5x

b4xd6x

Probabilistic Models with no Cycles


Belief Propagation

1

2 21}2,1{ , xxW

b41

1, xWB b

1

a3

1, xWA a

d

26

2, xWC c

c5

2

2, xWD d

2221}2,1{

11

21

,,,,,

,,,,,

xWxWxxWxWxW

xxP

C

BA

dcba

dcba

ab

1

cd

2

3 4

56

Probabilistic Model on Tree Graph



ab

1

cd

2

3 4

56

22522621}2,1{114113

2121}2,1{

,

,,,,,,

xMxMxxWxMxM

xxPxxP

dc,b,a,

dcba

a

a 1113 , xWxM A b

b 1114 , xWxM B

c

c 2225 , xWxM C d

D xdWxM 2226 ,


Belief Propagation

ab

1

cd

2

3 4

56

1

1

11411321}2,1{

1121}2,1{221

,

,,,

x

x a bBA

xMxMxxW

xWxWxxWxM ba


a

a 1113 , xWxM A

b

b 1114 , xWxM B


Belief Propagation for Probabilistic Model on Tree Graph

},{

},{ ,1Prji

jiji xxWZ

xX

No Cycles!!

1X

2X 3X

1kX

kX

2kX

3kX

1kX


Belief Propagation for Probabilistic Model on Square Grid Graph

E: Set of all the links

Eji

jiijL xxWxxxPxP},{

21 ,,,,






Marginal Probability

1 3 4

,,,,, 432122x x x x

NN

xxxxxPxP



1 3 4x x x xN

2

1 3 4

,,,,, 432122x x x x

NN

xxxxxPxP



1 3 4x x x xN

2 2

1 3 4

,,,,, 432122x x x x

NN

xxxxxPxP



3 4

,,,,,, 432121}2,1{x x x

NN

xxxxxPxxP



3 4x x xN

3 4

,,,,,, 432121}2,1{x x x

NN

xxxxxPxxP

1 2



3 4x x xN

3 4

,,,,,, 432121}2,1{x x x

NN

xxxxxPxxP

1 2 1 2



1

21}2,1{22 ,x

xxPxP



1

21}2,1{22 ,x

xxPxP

14

5

3

2

6

8

7



21 7

6

8

1

21}2,1{22 ,x

xxPxP

14

5

3

2

6

8

7



21 7

6

8

1

21}2,1{22 ,x

xxPxP

Message Update Rule

14

5

3

2

6

8

7

3

2 1

5

41x

12



21

3

4

5

3

2 1

5

413M

14M

15M

1x1

21M2

1 2

1

1151141132112

1151141132112

221 ,

,

z z

z

zMzMzMzzW

zMzMzMxzWxM

MM

Fixed Point Equations for Messages


Fixed Point Equation and Iterative Method

Fixed Point Equation ** MM




Iterative Method

0

xy

)(xy

y

x*M




Iterative Method

0M0

xy

)(xy

y

x*M




Iterative Method

01 MM

0M

1M

0

xy

)(xy

y

x*M




Iterative Method

12

01

MM

MM

0M1M

1M

0

xy

)(xy

y

x*M




Iterative Method

12

01

MM

MM

0M1M

1M

0

xy

)(xy

y

x*M

2M




Iterative Method

23

12

01

MM

MM

MM

0M1M

1M

0

xy

)(xy

y

x*M

2M



Four Kinds of Update Rule with Three Inputs and One Output


Interpretation of Belief Propagation based on Information Theory

0ln)(

x

xxx P

QQPQD

x

xx 1)( ,0 QQ

ZQF

ZQQxxWQPQD

QF

Ejijiij

ln][

lnln)(,ln)(]|[

][

},{

xxxxx

0 PQDPQ xx

ZPFQQFQ

ln][1][min

x

x

x Eji

jiij xxWZ},{

,

Eji

jiijV xxWZ

xxxP},{

||21 ,1,,,

Free Energy

Kullback-Leibler Divergence



ZQFPQD ln

xxxx

xxxx

xxx

xx

xx xx

xx

QQWQ

QQWQ

QQxWQQF

E

E

E

ln)(ln

ln)(ln)(

ln)(ln)(

\

0ln)(

x

xxx P

QQPQD

Free EnergyKL Divergence

E

WZ

P

xx 1

x\x

x

x

)(

)(

Q

Q



ZQFPQD ln

Ejijjiiijij

Viii

Ejiijij

Ejiijij

QQQQQQ

QQ

WQ

QQ

WQQF

},{

},{

},{

lnln,ln,

ln

,ln,

ln)(

,ln,

xxx

Bethe Free Energy


E

WZ

P

xx 1

x\x

xx )()( QQ

ix

ii QxQ\x

x)()(



FPQDQQ

minargminarg

,iji QQ

ZQQFPQD iji ln,Bethe

ijiQQQ

QQFPQDiji

,minargminarg Bethe,

1,

iji QQ

Ejijjiiijij

Viii

Ejiijijiji

QQQQQQ

QQWQQQF

},{

},{Bethe

lnln,ln,

ln,ln,,



Ejiijij

Viii

Vi ijijiji

ijiiji

QQ

QQ

QQFQQL

},{

,

BetheBethe

1,1

,

,,

1, ,,,minarg Bethe,

ijiijiijiQQ

QQQQQQFiji

Lagrange Multipliers to ensure the constraints



Bijijij

Viii

Vi ijijiji

Ejijjiiijij

Viii

Ejiijij

Ejiijij

Viii

Vi ijijijiijiiji

QQQQ

QQQQQQ

QQWQ

QQ

QQQQFQQL

1,1,

lnln,ln,

ln,ln,

1,1

,,,

,

},{

},{

},{

,BetheBethe

0,Bethe

ijiii

QQLxQ

Extremum Condition

0,, Bethe

ijijiij

QQLxxQ

Interpretation of Bethe Approximation (7)

FGfP yxyxyx g ,,,

ikikiiii x

ixQ )(

1||1exp },{,

)()(exp

,,

},{,},{, jjijijii

jiijjiij

aa

xxWxxQ

Extremum Condition 0,Bethe

ijiii

QQLxQ 0,

, Bethe

iji

jiij

QQLxxQ

73Physics Fluctuomatics (Tohoku

University)

ik

iiki

ii xMZ

xQ 1

}{\}{\,1,

ijljjljiij

jikiik

ijjiij xMxxWxM

ZxxQ

)()(exp\

},{, ijik

ikijii xMx



FGfP yxyxyx g ,,,14 2

5

13M

14M

15M

12M

3

26M14

5

13M

14M

15M

12W3

2

6

27M

8

7

28M

115114

1131121

111

xMxM

xMxMZ

xQ

2282272262112

11511411312

2112

,

1,

xMxMxMxxW

xMxMxMZ

xxQ


ijiii

QQLxQ 0,

, Bethe

iji

jiij

QQLxxQ



FGfP yxyxyx g ,,,14 2

5

13M

14M

15M

12M

3

26M14

5

13M

14M

15M

12W3

2

6

27M

8

7

28M

,121 QQ

115114

1131121

111

xMxM

xMxMZ

xQ

2282272262112

11511411312

2112

,

1,

xMxMxMxxW

xMxMxMZ

xxQ

1514

1312

21

,

MM

MW

M

Message Update Rule



15141312

15141312

21 ,

,

MMMW

MMMWM

1

3

4 2

5

13M

14M

15M

21M

14

5

3

2

6

8

7

2a

14 2

5

3

=

Message Passing Rule of Belief Propagation

Physics Fluctuomatics (Tohoku University)

Graphical Representations for Probabilistic Models

Probability distribution with two random variables is assigned to a edge.

),(},Pr{ 21}2,1{2211 xxfxXxX 1 2

)(}Pr{ 1111 xfxX 1 Node

Edge

Probability distribution with one random variable is assigned to a graph with one node.

),,(},,Pr{ 321}3,2,1{332211 xxxfxXxXxX 31

2Hyper-edge

77

Physics Fluctuomatics (Tohoku University)

Hyper-graph

Bayesian Network and Graphical Model

),(),(},,Pr{ 32}3,2{21}2,1{332211 xxfxxfxXxXxX

321

4

),(),(),(},,Pr{

13}1,3{32}3,2{21}2,1{

332211

xxfxxfxxfxXxXxX

Tree

31

2Cycle

More practical probabilistic models are expressed in terms of a product of functions and is assigned to chain, tree, cycle or hyper-graph representation.

1 2 3Chain

),(),(),(},,,Pr{

41}4,1{31}3,1{21}2,1{

44332211

xxfxxfxxfxXxXxXxX

312

45),,(),,(

},,,,Pr{

543}5,4,3{321}3,2,1{

5544332211

xxxfxxxfxXxXxXxXxX

78



),,(),,(),,(),,( IHEWGFDWEDCWCBAW EHIDFGCDEABC

),,( CBAWABC ),,( EDCWCDE

X=C

D

EA

CB

D

F GX

E H

I

=

A

B C

D

E

F G

H

I

),,( IHEWEHI),,( GEDWDEG

X

H IF GA B



A B F G H I

EDC },,Pr{

A

B C

D

E

F G

H

I

C

D

E A

B C F G

D

E H

I

C

D

E A

B C F G

D

E H

I

A

B C

D

E

F G

H

I

=

= x x x =

D E H IF GA B



A B D E F G H I

C}Pr{

A

B C

D

E

F G

H

I

C

D

EA

B C

C

D

EA

B C F G

D

E H

I=

= x =

A

B C

D

E



},,Pr{ EDCC

D

E A

B C F G

D

E H

Ix x x

C

D

EA

B C x}Pr{C

C

D

E

F G

D

E H

IA

B Cx x xC

D

EA

B C x D E

C

D

E

F G

D

E H

Ix x

C

D

ED E

D E

C

D

E

F G

H

I

D E

EDCC },,Pr{}Pr{


Belief Propagation on Hypergraph Representations in terms of Cactus Tree

C

D

ED E

C

D

E

F G

H

I

F G

D

E H

I

A

B C

F G

D

E H

I

F G

H I

A B A

B C

A

B C

D

E}Pr{C

A

B C

D

E

F G

H

I

Update Flow of Messages in computing the marginal probability Pr{C}


Interpretation of Belief Propagationfor Hypergraphs based on Information Theory

or :, VEE

We consider hypergraphs which satisfy

Cactus Tree

HypergraphV: Set of all the nodesE: Set of all the hyperedges



0ln)(

x

xxx P

QQPQD

x

xx 1)( ,0 QQ

ZQF

ZQQWQPQD

QF

E

ln][

lnln)(ln)(]|[

][

xxxxxx

0 PQDPQ xx

ZPFQQFQ

ln][1][min

x

x

x

xE

WZ

E

V xWZ

xxxP

1,,, ||21

Free Energy

Kullback-Leibler Divergence



ZQFPQD ln

xxxx

xxxx

xxx

xx

xx xx

xx

QQWQ

QQWQ

QQxWQQF

E

E

E

ln)(ln

ln)(ln)(

ln)(ln)(

\

0ln)(

x

xxx P

QQPQD


E

WZ

P

xx 1

x\x

x

x

)(

)(

Q

Q

E i xiiii

Viii

E

E

i

xQxQQQ

QQ

WQ

QQWQQF

lnln

ln

ln

ln)(ln

x

x

xx

xx

xx

xxxx



ZQFPQD ln


E

WZ

P

xx 1

x\x

xx )()( QQ

ix

ii QxQ\x

x)()(

Bethe Free Energy



FPQDQQ

minargminarg

ix

ii QxQ\

x

x

ZQQFPQD iji ln,Bethe

QQFPQD iQQQ i

,minargminarg Bethe,

1

x

xQxQix

ii

E i xiiii

Viii

Ei

i

xQxQQQ

QQWQQQF

lnln

lnln,Bethe

x

x

xx

xx



EVi xiii

Vi i xiii

ii

QxQ

QxQ

QQFQQL

i

i

11

,,

\,

BetheBethe

x

x

x

x

1 ,,minarg\

Bethe,

xx

xx QxQQxQQQFiii x

iix

iiiQQ

Lagrange Multipliers to ensure the constraints



Eij

Vi xiii

Vi i x xiiii

E Vi xii

Viii

xii

E

EVi xiii

Vi i x xiiiiii

QxQQxQx

QQQQ

xQxQWQ

QxQ

QxQxQQFQQL

ii i

i

i

i

i i

11

lnln

lnln

11

,,

\,

\,BetheBethe

xx

x

x

x

x

xx

xx

xx

x

x

0,Bethe

QQLxQ iii

Extremum Condition

0,Bethe

QQLQ ix



i

iiii

ii xMZ

xQ

\1

j jjjj xMxW

ZxQ

\\

1


QQLxQ iii 0,

, Bethe

QQL

xxQ ijiij

ix

ii QxQ\

x

x

ij jjjj

iiii xMxW

ZZxM

\ \\\


Summary

Belief Propagation and Message Passing RuleInterpretation of Belief Propagation in the stand point of Information Theory

Future Talks

11th Probabilistic image processing by means of physical models 12th Bayesian network and belief propagation in statistical inference


Practice 9-1

a b c d

XY ya,b,c,d,xPyxP ,,

a

AXA xaWxM , b

BXB xbWxM ,

c

CYC ycWyM , d

DYD ydWyM ,

ydWycWyxWxbWxaWya,b,c,d,xP DCXYBA ,,,,,,

We consider a probability distribution P(a,b,c,d,x,y) defined by

Show that marginal Probability

is expressed by

yMyMyxWxMxMyxP DCXYXBXAXY 22,,


Practice 9-2

1151141131121

111 xMxMxMxMZ

xQ

228227226211211511411312

2112 ,1, xMxMxMxxWxMxMxMZ

xxQ

2

228227226211212

1112 ,

x

xMxMxMxxWZZxM

By substituting

2

211211 ,x

xxQxQto 　　　　　　　　　　　　 , derive the following equation.


Practice 9-3

Make a program to solve the nonlinear equation x=tanh(Cx) for various values of C. Obtain the solutions for C=0.5, 1.0, 2.0 numerically. Discuss how the iterative procedures converge to the fixed points of the equations in the cases of C=0.5, 1.0, 2.0 by drawing the graphs of y=tanh(Cx) and y=x.

23

12

01

tanhtanhtanh

CxxCxxCxx

Documents

Physical Fluctuomatics 7th~10th Belief propagation