An Introduction to Complex-Valued Recurrent Correlation ...valle/PDFfiles/Talk_IJCNN2014.pdf ·...

An Introduction to Complex-Valued RecurrentCorrelation Neural Networks

Marcos Eduardo Valle

Department of Applied MathematicsInstitute of Mathematics, Statistics, and Scientific Computing

University of Campinas - Brazil

July 11, 2014

Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 1 / 23

Introduction

Complex-Valued Hopfield NetworksLow storage capacity!

Hopfield Network

Low storage capacity!

Complex-valuedRecurrent CorrelationNeural NetworkHigh storage capacity

Recurrent Correlation NetworksHigh storage capacity!

Introduction

Hopfield NetworkLow storage capacity!

Introduction

(Chiueh&Goodman,1991)

��

Introduction

��

(Jankowski et al.,1996)

Introduction

��

(Jankowski et al.,1996)

Outline

1 Introduction

2 Some Complex-Valued Dynamic Networks

3 Review the Bipolar Recurrent Correlation Neural Networks

4 Complex-Valued Recurrent Correlation Neural Networks

5 Computational Experiments

6 Concluding Remarks

Some Complex-Valued Dynamic Networks

Many complex-valued dynamic neural networks (CV-DNNs) aredefined as follows:

Complex-Valued Dynamic Neural Networks (CV-DNNs)

Given a complex-valued input z(0) = [z1(0), . . . , zn(0)]T , compute

zj(t + 1) = ϕ

mjkzk (t)

), ∀j = 1, . . . ,n,

whereϕ is a complex-valued activation function.M = (mjk ) ∈ Cn×n is the synaptic weight matrix,

Recording Recipe

Given a set of vectors u1, . . . ,up, M is usually defined byThe outer product rule (or hebbian learning), i.e.,

M = UU∗.

The generalized-inverse learning (or projection rule), i.e.,

M = UU†,

whereU = [u1, . . . ,up],U∗ is the adoint (or hermitian conjungate) of U,U† is the pseudo-inverse (or generalized inverse) of U.

Some Complex-Valued Activation Functions

Definition (Complex-Signum Activation Function)

csgnK (z) =

1, 0 ≤ Arg(z) < θK ,

eiθK , θK ≤ Arg(z) < 2θK ,...

...ei(K−1)θK , (K − 1)θK ≤ Arg(z) < 2π,

for some integer K and θK = 2πK is called angular size.

Proposition (Alternative Representation of csgn)

csgnK (z) = rK ◦ φK ◦ qK (z),

where φK denotes the floor function,

qK (z) =Arg(z)θK

, and rK (x) = eixθK .

Definition (Complex-Sigmoid Activation Function)

csgmK (z) = rK ◦mK ◦ qK (z),

mK (x) = mod

([(K∑κ=1

11− e−(x−κ)/ε

)− 1

RemarkThe csgmK has an aditional parameter ε:

The csgmK → csgnK as ε→ 0.Tanaka and Aihara suggested the value ε = 0.2.

For ε = 0.2, mK (x) ≈ x :

0 1 2 3 4 5 6 7 8

Definition (Continuous-valued Activation Function)

σ(z) = rK ◦ ι ◦ qK (z),

where ι denotes the identity function. Alternatively,

σ(z) =z|z|.

RemarkWe consier σ because it is simpler than csgn and csgm.

From Hopfield to RCNNs

Consider U = {u1, . . . ,up} ⊆ {−1,+1}n.Given x(0) = [x1(0), x2(0), . . . , xn(0)]T ∈ Bn, define

xi(t + 1) = sgn

n∑j=1

mijxj(t)

, ∀i = 1, . . . ,n,

p∑ξ=1

uξi uξj .

xi(t + 1) = sgn

n∑j=1

p∑ξ=1

uξi uξj

, ∀i = 1, . . . ,n.

xi(t + 1) = sgn

p∑ξ=1

n∑j=1

xj(t)uξj

, ∀i = 1, . . . ,n.

xi(t + 1) = sgn

p∑ξ=1

⟨uξ,x(t)

⟩uξi

, ∀i = 1, . . . ,n.

xi(t + 1) = sgn

p∑ξ=1

uξ,x(t)⟩)

, ∀i = 1, . . . ,n,

where f : [−n,n]→ R is a continuous nondecreasing function.

Recurrent Correlation Neural Networks

Definition (Recurrent Correlation Neural Networks (RCNNs))

xi(t + 1) = sgn

p∑ξ=1

wξ(t)uξi

, ∀i = 1, . . . ,n,

wherewξ(t) = f

(⟨uξ,x(t)

⟩), ∀ξ = 1, . . . ,p,

for some (fixed) continuous nondecreasing function f : [−n,n]→ R.

PropositionThe sequence produced by a RCNN always converges.

Complex-Valued Recurrent Correlation NNs

Definition (Complex-Valued RCNNs)

Consider U = {u1, . . . ,up} ⊆ Sn, S = {z ∈ C : |z| = 1}.Given z(0) = [z1(0), z2(0), . . . , zn(0)]T ∈ Sn, define

zi(t + 1) = σ

p∑ξ=1

wξ(t)uξi

, ∀i = 1, . . . ,n,

wherewξ(t) = f

(<{⟨

uξ, z(t)⟩})

, ∀ξ = 1, . . . ,p,

for some (fixed) continuous nondecreasing function f : [−n,n]→ R.

Theorem (Convergence of the CV-RCNNs)The sequence produced by a CV-RCNN always converges.

Examples:

The correlation CV-RCNN is obtained by considering

fc(x) = x/n.

The exponential CV-RCNN is obtained considering

fe(x) = eαx/n, α > 0.

The high-order CV-RCNN is obtained by considering

fh(x) = (1 + x/n)q, q > 1 is an integer.

The potential-function CV-RCNN is obtained by considering

fp(x) = 1/(1− x/n)L, L ≥ 1.

Experiments: Exponential CV-RCNN

0 2 4 6 8 10 12 14 16

Input Error

alpha=1alpha=3alpha=5

Experiments: High-order CV-RCNN

0 2 4 6 8 10 12 14 16

Input Error

q=2q=3q=5

q=10q=20q=30

Experiments: Potential-function CV-RCNN

0 2 4 6 8 10 12 14 16

Input Error

L=1L=3L=5

L=10L=20L=30

Comparison between the four CV-RCNNs

Parameters: q = 10,L = 10, and α = 10.

0 2 4 6 8 10 12 14 16

Input Error

CorrelationHigh-Order

PotentialExponential

General remarks:

The high-order CV-RCNN and the exponential CV-RCNN failed toperfectly recall a fundamental vector for small values of theparameters q and α, i.e., for q, α ≤ 5.The error correction capability of the high-order and exponentialCV-RCNNs have similar dependence on the parameters.The vector recalled by the potential-function CV-RCNN is alwaysvery similar to the desired fundamental vector u1.The correlation CV-RCNN failed to retrieve the fundamental vectoru1 in many steps.The high-order, potential-function, and exponential CV-RCNNs,besides apparently giving perfect recall of undistorted vectors,exhibited similar error capability for large values of theirparameters, i.e., for q,L, α ≥ 10.

In the following, we focus on the exponential CV-RCNN.

On the Effect of the Activation Function

Let us compare the effect of the three activation functions σ, csgn, andcsgm in the exponential CV-RCNN with α = 10.

For p = 12, the three variations of the exponential CV-RCNNalways recalled one of the fundamental vectors.We increased considerably the number of fundamental vectorsfrom 12 to 4096.

Probability of Successful Recall

0 0.2 0.4 0.6 0.8 1 1.2

Epsilon

csgmcsgn

Recall Phase of the Exponential CV-RCNN

Number of times the exponential CV-RCNN recalls the vector that iscloser to the input vector using the Euclidean distance.

12 4096

Storage Capacity of the Exponential CV-RCNN

2 4 6 8 10 12 14 16 18 20

alpha=3 alpha=4

Straight lines:

(least-squares).α A c3 3.43 1.054 4.41 1.125 5.39 1.226 4.18 1.397 3.50 1.58

The storage capacity of the exponential CV-RCNN visually scalesexponentially with the length n of the vectors.

Concluding Remarks

We generalized the bipolar RCNNs using complex.A CV-RCNNs is characterized by a continuous non-decreasingfunction f .The sequence produced by a CV-RCNN always converges to astationary state.Preliminary computational experiments revealed that the storagecapacity of the exponential CV-RCNN scales exponentially withthe length of the stored vectors.

A detailed account on CV-RCNNs have been submitted for publicationon IEEE TNNLS.

Thank you!

An Introduction to Complex-Valued Recurrent Correlation ...valle/PDFfiles/Talk_IJCNN2014.pdf ·...

Documents

Complex-Valued Generalized Hebbian Algorithm and Its ... · Complex-Valued Generalized Hebbian Algorithm and Its Applications to Sensor Array Signal Processing Yanwu Zhang Principal

ϕ-contractive multivalued mappings in complex valued metric … · φ-contractive multivalued mappings in complex valued metric spaces and remarks on some recent papers Vishal Joshi1,

Review of complex numbers, complex valued functions, 1 ... · Review of complex numbers, complex valued functions, and contour integration: 1) Find your calculus book and review things

Complex Valued Nonlinear Adaptive Filters

Complex Variables - Department of Physicsnearing/mathmethods/complex_variables.pdf · The applications of ... Extend the idea of di erentiation to complex-valued functions of complex

2D Complex-Valued Eikonal Equation

Complex-Valued Matrix Derivatives - Assetsassets.cambridge.org/97805211/92644/frontmatter/... · 2011. 2. 4. · Complex-Valued Matrix Derivatives In this complete introduction to

A novel construction of complex-valued Gaussian processes ...web.mit.edu/jianshucaogroup/pdfdir/JCP138-224104-2013.pdfA novel construction of complex-valued Gaussian processes with

Fast Recall for Complex-Valued Hopfield Neural Networks ...downloads.hindawi.com/journals/cin/2017/4894278.pdf · ResearchArticle Fast Recall for Complex-Valued Hopfield Neural Networks

Complex-valued Linear Layers for Deep Neural Network-based ... · 4/8/2016 · Complex-valued Linear Layers for Deep Neural Network-based Acoustic Models for Speech Recognition Author:

Analysis on radial complex-valued functions of p-adic ...€¦ · Analysis on radial complex-valued functions of p-adic arguments Anatoly N. Kochubei Institute of Mathematics, National

A Low-complexity Complex-valued Activation Function for

Complex-valued wavelet lifting and applicationsstats.lse.ac.uk/fryzlewicz/cplxlift/cplxlift.pdf · Complex-valued wavelet lifting and applications Jean Hamilton HEDS, ScHARR, University

Complex Numbers and Complex-Valued Functions

Bayesian Modeling of Complex-valued fMRI Signals for Brain ... · Bayesian Modeling of Complex-valued fMRI Signals for Brain Activation Cheng-Han Yu, Raquel Prado, Hernando Ombao,

Complex-valued wavelet lifting and applications

Extended Hopﬁeld Network for Sequence Learning ...lasa.epfl.ch/publications/uploadedFiles/icann05_maurer_billard.pdf · Extended Hopﬁeld Network for Sequence Learning: Application

On the appropriateness of complex-valued neural networks ... · Lukas Drude, Complex-valued Networks Motivation • Quite a few older papers on CVNNs, even on real-valued tasks •

Image Segmentation by Complex-Valued Units

complex-valued signals and systems – basic principles and