An Introduction to Complex-Valued Recurrent Correlation ...valle/PDFfiles/Talk_IJCNN2014.pdf · Introduction Complex-Valued Hopﬁeld Networks Low storage capacity! Hopﬁeld Network

An Introduction to Complex-Valued RecurrentCorrelation Neural Networks

Marcos Eduardo Valle

Department of Applied MathematicsInstitute of Mathematics, Statistics, and Scientific Computing

University of Campinas - Brazil

July 11, 2014

Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 1 / 23

Introduction

Complex-Valued Hopfield NetworksLow storage capacity!

Hopfield Network

Low storage capacity!

Complex-valuedRecurrent CorrelationNeural NetworkHigh storage capacity

Recurrent Correlation NetworksHigh storage capacity!


Introduction


Hopfield NetworkLow storage capacity!




Introduction



(Chiueh&Goodman,1991)

��




Introduction




��

(Jankowski et al.,1996)

OO




Introduction


))



��

(Jankowski et al.,1996)

OO



55


Outline

1 Introduction

2 Some Complex-Valued Dynamic Networks

3 Review the Bipolar Recurrent Correlation Neural Networks

4 Complex-Valued Recurrent Correlation Neural Networks

5 Computational Experiments

6 Concluding Remarks


Some Complex-Valued Dynamic Networks

Many complex-valued dynamic neural networks (CV-DNNs) aredefined as follows:

Complex-Valued Dynamic Neural Networks (CV-DNNs)

Given a complex-valued input z(0) = [z1(0), . . . , zn(0)]T , compute

zj(t + 1) = ϕ

(n∑

k=1

mjkzk (t)

), ∀j = 1, . . . ,n,

whereϕ is a complex-valued activation function.M = (mjk ) ∈ Cn×n is the synaptic weight matrix,


Recording Recipe

Given a set of vectors u1, . . . ,up, M is usually defined byThe outer product rule (or hebbian learning), i.e.,

M = UU∗.

The generalized-inverse learning (or projection rule), i.e.,

M = UU†,

whereU = [u1, . . . ,up],U∗ is the adoint (or hermitian conjungate) of U,U† is the pseudo-inverse (or generalized inverse) of U.


Some Complex-Valued Activation Functions

Definition (Complex-Signum Activation Function)

csgnK (z) =

1, 0 ≤ Arg(z) < θK ,

eiθK , θK ≤ Arg(z) < 2θK ,...

...ei(K−1)θK , (K − 1)θK ≤ Arg(z) < 2π,

for some integer K and θK = 2πK is called angular size.

Proposition (Alternative Representation of csgn)

csgnK (z) = rK ◦ φK ◦ qK (z),

where φK denotes the floor function,

qK (z) =Arg(z)θK

, and rK (x) = eixθK .


Definition (Complex-Sigmoid Activation Function)

csgmK (z) = rK ◦mK ◦ qK (z),

where

mK (x) = mod

([(K∑κ=1

11− e−(x−κ)/ε

)− 1

2

],K

).

RemarkThe csgmK has an aditional parameter ε:

The csgmK → csgnK as ε→ 0.Tanaka and Aihara suggested the value ε = 0.2.


For ε = 0.2, mK (x) ≈ x :

0

1

2

3

4

5

6

7

8

0 1 2 3 4 5 6 7 8

mK

x


Definition (Continuous-valued Activation Function)

σ(z) = rK ◦ ι ◦ qK (z),

where ι denotes the identity function. Alternatively,

σ(z) =z|z|.

RemarkWe consier σ because it is simpler than csgn and csgm.


From Hopfield to RCNNs

Consider U = {u1, . . . ,up} ⊆ {−1,+1}n.Given x(0) = [x1(0), x2(0), . . . , xn(0)]T ∈ Bn, define

xi(t + 1) = sgn

n∑j=1

mijxj(t)

, ∀i = 1, . . . ,n,

where

mij =

p∑ξ=1

uξi uξj .




xi(t + 1) = sgn

n∑j=1

p∑ξ=1

uξi uξj

xj(t)

, ∀i = 1, . . . ,n.




xi(t + 1) = sgn

p∑ξ=1

n∑j=1

xj(t)uξj

uξi

, ∀i = 1, . . . ,n.




xi(t + 1) = sgn

p∑ξ=1

⟨uξ,x(t)

⟩uξi

, ∀i = 1, . . . ,n.




xi(t + 1) = sgn

p∑ξ=1

f(⟨

uξ,x(t)⟩)

uξi

, ∀i = 1, . . . ,n,

where f : [−n,n]→ R is a continuous nondecreasing function.


Recurrent Correlation Neural Networks

Definition (Recurrent Correlation Neural Networks (RCNNs))


xi(t + 1) = sgn

p∑ξ=1

wξ(t)uξi

, ∀i = 1, . . . ,n,

wherewξ(t) = f

(⟨uξ,x(t)

⟩), ∀ξ = 1, . . . ,p,

for some (fixed) continuous nondecreasing function f : [−n,n]→ R.

PropositionThe sequence produced by a RCNN always converges.


Complex-Valued Recurrent Correlation NNs

Definition (Complex-Valued RCNNs)

Consider U = {u1, . . . ,up} ⊆ Sn, S = {z ∈ C : |z| = 1}.Given z(0) = [z1(0), z2(0), . . . , zn(0)]T ∈ Sn, define

zi(t + 1) = σ

p∑ξ=1

wξ(t)uξi

, ∀i = 1, . . . ,n,

wherewξ(t) = f

(<{⟨

uξ, z(t)⟩})

, ∀ξ = 1, . . . ,p,

for some (fixed) continuous nondecreasing function f : [−n,n]→ R.

Theorem (Convergence of the CV-RCNNs)The sequence produced by a CV-RCNN always converges.


Examples:

The correlation CV-RCNN is obtained by considering

fc(x) = x/n.

The exponential CV-RCNN is obtained considering

fe(x) = eαx/n, α > 0.

The high-order CV-RCNN is obtained by considering

fh(x) = (1 + x/n)q, q > 1 is an integer.

The potential-function CV-RCNN is obtained by considering

fp(x) = 1/(1− x/n)L, L ≥ 1.


Experiments: Exponential CV-RCNN

0

2

4

6

8

10

12

14

16

0 2 4 6 8 10 12 14 16

Out

put E

rror

Input Error

alpha=1alpha=3alpha=5



Experiments: High-order CV-RCNN

0

2

4

6

8

10

12

14

16

0 2 4 6 8 10 12 14 16

Out

put E

rror

Input Error

q=2q=3q=5

q=10q=20q=30


Experiments: Potential-function CV-RCNN

0

2

4

6

8

10

12

14

16

0 2 4 6 8 10 12 14 16

Out

put E

rror

Input Error

L=1L=3L=5

L=10L=20L=30


Comparison between the four CV-RCNNs

Parameters: q = 10,L = 10, and α = 10.

0

2

4

6

8

10

12

14

16

0 2 4 6 8 10 12 14 16

Out

put E

rror

Input Error

CorrelationHigh-Order

PotentialExponential


General remarks:

The high-order CV-RCNN and the exponential CV-RCNN failed toperfectly recall a fundamental vector for small values of theparameters q and α, i.e., for q, α ≤ 5.The error correction capability of the high-order and exponentialCV-RCNNs have similar dependence on the parameters.The vector recalled by the potential-function CV-RCNN is alwaysvery similar to the desired fundamental vector u1.The correlation CV-RCNN failed to retrieve the fundamental vectoru1 in many steps.The high-order, potential-function, and exponential CV-RCNNs,besides apparently giving perfect recall of undistorted vectors,exhibited similar error capability for large values of theirparameters, i.e., for q,L, α ≥ 10.

In the following, we focus on the exponential CV-RCNN.


On the Effect of the Activation Function

Let us compare the effect of the three activation functions σ, csgn, andcsgm in the exponential CV-RCNN with α = 10.

For p = 12, the three variations of the exponential CV-RCNNalways recalled one of the fundamental vectors.We increased considerably the number of fundamental vectorsfrom 12 to 4096.


Probability of Successful Recall

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

Rec

all P

roba

bilit

y

Epsilon

csgmcsgn

sigma


Recall Phase of the Exponential CV-RCNN

Number of times the exponential CV-RCNN recalls the vector that iscloser to the input vector using the Euclidean distance.

0

0.2

0.4

0.6

0.8

1

12 4096



Storage Capacity of the Exponential CV-RCNN

101

102

103

2 4 6 8 10 12 14 16 18 20

Avera

ge o

f th

e E

stim

ate

d S

tora

ge C

ap

aci

ty

n

alpha=3 alpha=4


Straight lines:

Acn.

(least-squares).α A c3 3.43 1.054 4.41 1.125 5.39 1.226 4.18 1.397 3.50 1.58

The storage capacity of the exponential CV-RCNN visually scalesexponentially with the length n of the vectors.


Concluding Remarks

We generalized the bipolar RCNNs using complex.A CV-RCNNs is characterized by a continuous non-decreasingfunction f .The sequence produced by a CV-RCNN always converges to astationary state.Preliminary computational experiments revealed that the storagecapacity of the exponential CV-RCNN scales exponentially withthe length of the stored vectors.

A detailed account on CV-RCNNs have been submitted for publicationon IEEE TNNLS.

Thank you!


Documents

An Introduction to Complex-Valued Recurrent Correlation ...valle/PDFfiles/Talk_IJCNN2014.pdf · Introduction Complex-Valued Hopﬁeld Networks Low storage capacity! Hopﬁeld Network