Hypercubes and

Neural Networks

bill wolfe

10/23/2005

Modeling

Simple Neural Model

• ai Activation

• ei External input

• wij Connection Strength

Assume: wij = wji (“symmetric” network)

W = (wij) is a symmetric matrix

ai ajwij

ei ej

Net Input

eaWnet

i

j

jiji eawnet ai

aj

wij

Vector Format:

Dynamics

• Basic idea:

ai

neti > 0

ai

neti < 0

ii

ii

anet

anet

0

0

netdt

adnet

dt

dai

i eaW

Energy

aeaWaE TT 21

net

netnet

ewew

aEaE

E

n

j

nnj

j

j

n

,...,

,...,

/,....,/

1

11

1

netE

Lower Energy

• da/dt = net = -grad(E) seeks lower energy

net

Energy

a

Problem: Divergence

Energy

net a

A Fix: Saturation

))(1( iiii

aanetdt

da

corner-seeking

lower energy

10 ia

Keeps the activation vector inside the hypercube boundaries

a

Energy

0 1

))(1( iiii

aanetdt

da

corner-seeking

lower energy

Encourages convergence to corners

Summary: The Neural Model

))(1( iiii

aanetdt

da

i

j

jiji eawnet

ai Activation ei External Inputwij Connection StrengthW (wij = wji) Symmetric

10 ia

Example: Inhibitory Networks

• Completely inhibitory– wij = -1 for all i,j– k-winner

• Inhibitory Grid– neighborhood inhibition

Traveling Salesman Problem

• Classic combinatorial optimization problem

• Find the shortest “tour” through n cities

• n!/2n distinct tours

D

D

AE

B

C

AE

B

C

ABCED

ABECD

TSP solution for 15,000 cities in Germany

TSP

50 City Example

Random

Nearest-City

2-OPT

http://www.jstor.org/view/0030364x/ap010105/01a00060/0

An Effective Heuristic for the Traveling Salesman Problem

S. Lin and B. W. Kernighan

Operations Research, 1973

Centroid

Monotonic

Neural Network Approach

D

C

B

A1 2 3 4

time stops

cities neuron

Tours – Permutation Matrices

D

C

B

A

tour: CDBA

permutation matrices correspond to the “feasible” states.

Not Allowed

D

C

B

A

Only one city per time stopOnly one time stop per city

Inhibitory rows and columns

inhibitory

Distance Connections:

Inhibit the neighboring cities in proportion to their distances.

D

C

B

A-dAC

-dBC

-dDC

D

A

B

C

D

C

B

A-dAC

-dBC

-dDC

putting it all together:

Research Questions

• Which architecture is best?• Does the network produce:

– feasible solutions?– high quality solutions?– optimal solutions?

• How do the initial activations affect network performance?

• Is the network similar to “nearest city” or any other traditional heuristic?

• How does the particular city configuration affect network performance?

• Is there a better way to understand the nonlinear dynamics?

A

B

C

D

E

F

G

1 2 3 4 5 6 7

typical state of the network before convergence

“Fuzzy Readout”

A

B

C

D

E

F

G

1 2 3 4 5 6 7

à GAECBFD

A

B

C

D

E

F

G

Neural ActivationsFuzzy Tour

Initial Phase

Neural ActivationsFuzzy Tour

Monotonic Phase

Neural ActivationsFuzzy Tour

Nearest-City Phase

24

23

22

21

20

19

18

17

16

15

14

13

12

11

10

9

8

7

6

to

ur

len

gt

h

10009008007006005004003002001000iteration

Fuzzy Tour Lengths

centroidphase

monotonicphase

nearest-cityphase

monotonic (19.04)

centroid (9.76)nc-worst (9.13)

nc-best (7.66)2opt (6.94)

Fuzzy Tour Lengthstour length

iteration

12

11

10

9

8

7

6

5

4

3

2

tour length

70656055504540353025201510# cities

average of 50 runs per problem size

centroid

nc-w

nc-bneur

2-opt

Average Results for n=10 to n=70 cities

(50 random runs per n)

# cities

DEMO 2

Applet by Darrell Longhttp://hawk.cs.csuci.edu/william.wolfe/TSP001/TSP1.html

Conclusions

• Neurons stimulate intriguing computational models.

• The models are complex, nonlinear, and difficult to analyze.

• The interaction of many simple processing units is difficult to visualize.

• The Neural Model for the TSP mimics some of the properties of the nearest-city heuristic.

• Much work to be done to understand these models.

a1 a2

a3

w12

w23w13

3 Neuron Example

000

111

100110

010

001 011

a1

a2

a3

Brain State: <a1, a2, a3>

000

111

100110

010

001 011

a1

a2

a3

“Thinking”

Binary Model

aj = 0 or 1

Neurons are either “on” or “off”

Binary Stability

aj = 1 and Netj >=0

Or

aj = 0 and Netj <=0

Hypercubes

n=0

n=1

n=2

4-Cube

4-Cube

5-Cube

5-Cube

5-Cube

http://www1.tip.nl/~t515027/hypercube.html

Hypercube Computer Game

00

01 11

100

1

2

3

2-Cube

0123

0 1 2 3

0110

0110

1001

1001

Q2 =Adjacency Matrix:

Hypercube Graph

I

QQ

nn

1

1nQ

I

Recursive Definition

Theorem 1: If v is an eigenvector of Qn-1 with eigenvalue x then the concatenated vectors [v,v] and [v,-v] are eigenvectors of Qn with eigenvalues x+1 and x-1 respectively.

Eigenvectors of the Adjacency Matrix

v

vQn

I

Qn 1

1nQ

I

v

v

vv

vv

vQv

vvQ

v

v

n

n)1(

1

1

Proof

= +1

11

= -1

1-1

= +2

1111

= 0

11-1-1

= 0

1-11-1

= -2

1-1-11

= +3

11111111

= +1

1111-1-1-1-1

= +1

11-1-111-1-1

= -1

11-1-1-1-111

= +1

1-11-11-11-1

= -1

1-11-1-11-11

= -1

1-1-111-1-11

= -3

1-1-11-111-1

+1

+1

+1 +1 +1 +1

+1

-1

-1

-1-1

-1

-1 -1

n=0

n=1

n=2

n=3

Generating Eigenvectors and Eigenvalues

Walsh Functions for n=1, 2, 3

1

1

1

1

-1

-1

-1

-1

000

001

010

011

100

101

110

111

eigenvector binary number

000

100 110

010

111000

001 011

x

y

z

x

=-1k=1

=-1k=1

=+1k=2

=+1k=2

=+1k=2

=-3k=0

=-1k=1

y

z n=3

n=3

Theorem 3: Let k be the number of +1 choices in the recursive construction of the eigenvectors of the n-cube. Then for k not equal to n each Walsh state has 2n-k-1 non adjacent subcubes of dimension k that are labeled +1 on their vertices, and 2n-k-1 non adjacent subcubes of dimension k that are labeled -1 on their vertices. If k = n then all the vertices are labeled +1. (Note: Here, "non adjacent" means the subcubes do not share any edges or vertices and there are no edges between the subcubes).

n=5k=3

reduced graph

n=5k=2

reduced graph

Schamtice of the 5-cube Schamtice of the 5-cube

n=5, k= 3 n=5, k= 2

-1

-1

-1

-1

-1

-1

-1

-1

-1

-1

-1 -1

Inhibitory Hypercube

Theorem 5: Each Walsh state with positive, zero, or negative eigenvalue is an unstable, weakly stable, or strongly stable state of the inhibitory hypercube network, respectively.