Dynamics and its stability of Boltzmann-machine learning algorithm for gray scale image restoration

J. Inoue (Hokkaido Univ.) and K. Tanaka (Tohoku Univ.)

The 3rd International Symposium on Slow Dynamics in Complex Systems in Sendai November 2003

Plan of this talk Bayesian image restoration and hyper-parameter

estimation Boltzmann-machine learning algorithm for the hyper-

parameter estimation Dynamic behavior of the BML algorithm Stability of the solution Concluding remarks

Bayesian image restoration

Original Corrupted

We treat images and the degrading process as spin systems

Definitions of the model by spin systems

Original

2exp[ ( / 2 ) ( ) ]({ })

s

i jijs

s

NP

Z

0,1, 2,....,Q 1i

Corrupted

2 2exp[ (1/ 2 ) ( ) ]({ } |{ })

2i i

ai

a

aP

,s a : Hyper-parameters (true value)

{ }

{ }

Bayesian approach and MPM estimation2 2

2 2

( )

{ }

Q 1

0

exp[ ( / 2 ) ( ) ( ) ]({ } |{ })

exp[ ( / 2 ) ( ) ( ) ]

(arg max ({ }|{ }))

2 1 2 1( ) [ ( ) ( )]2 2

i i

i j i iij i

i j i iij i

MPM

i

k

NP

tr N

tr P

k kx k

h

x x

h

( )

2(1/ ) ( )MPM

iiiD N

takes its minimum at 1 2, 1/ 2s sT h h a

[Inoue and Carlucci (2001)]

Maximization of the marginal likelihood via Boltzmann-machine learning algorithm

2

2 2

,

2exp[ ( / 2 ) ( ) ( ) ]({ , }) log

exp[ ( / 2 ) ( ) , exp[ ( ) ]

i j i iij i

L

i j L i iij i

h

NK tr

Z Z

Z tr N Z tr

h

h

, ({ , })hK takes its maximum at ,s sh h on average

We evaluate the data-averaged BML algorithm at the mean-field level :

{ ,,

,

}

{ , }

[ ({ , })]

[ ({ , })]

h

hh h

K

Kh

[Inoue and Tanaka (2003)]

Dynamic behavior of the hyper-parameters

1 0.75, 0.5,Q 3, 1.0s s sh

{ , } {, , , }[ ({ , }] , [ ({ , }]h hh hhK K

are integrated numerically

Analysis of the stabilityExpand the BML equations around * * *( , )hs

2( ) ( ), ( , )d O hdt

s A s s s

and check the sign of eigenvalues of the Hessian A

1 0.75, 0.5s sh

The solution1 1,s sh h

is asymptotically stable

True hyper-parameter dependence of the stability

The solution of the BML algorithm is asymptotically stable as long as the solution is identical to the true value of the hyper-parameters

0.5sh (fixed)

1 0.75s (fixed)

Behavior of the BML algorithm around the solution

1 2( , ) ( , ) ,11 12 ( , )s s s sh h

s sh C h

Trajectories in the hyper-parameter space

1 0.75, 0.5s sh

(around the solution)

Concluding remarks We investigated dynamic behavior and its stability

of the BML algorithm for gray scale image restoration

We derived the data-averaged BML equations The solution is asymptotically stable as long as the

solution is identical to the true value of the hyper-parameters

More details of the present study are available athttp://chaosweb.complex.eng.hokudai.ac.jp/~j_inoue/

Send email : j_inoue@complex.eng.hokudai.ac.jp