View
230
Download
0
Category
Preview:
Citation preview
8/11/2019 Connectionist models and their properties.pdf
1/50
CO GN IT IVE SCIENCE 6 , 205 -25 4 1982)
C nnect i n ist M od els and T h ei r Prop erties
J A F E L D M A N A N D D H B A L L A R D
C o m p u t e r S c ie n c e D e p a r t m e n t
Univers i t y o f Roches ter
Roches ter N Y 14627
M uch o f t he p rogress in the f i e lds const it u ting cogn i t ive sc ience has be en ba sed
upon t he use o f exp l ic i t in f o rm at ion p rocess ing m odels a lmo st exc lus ive ly
p a t t e r n e d a f t e r c o n v e n t i o n a l s e r i a l c o m p u t e r s . A n e x t e n s i o n o f th e s e i d e a s t o
m a s s iv e ly p a r a l l e l c o n n e c t i a n i s t m o d e l s a p p e a r s t o o f f e r a n u m b e r o f a d v a n -
t ages . A f t e r a p re l imina ry d iscuss ion t h is pap er in t roduces a ge ne ra l connec-
t ion is t m od e l and cons iders h ow i t m igh t be used in cog n i t i ve sc ience . Am on g
t he issues add resse d are : s t ab i l i ty and no ise -sens it iv it y d is t r ibu t ed dec is ion -
m a k i n g t im e a n d s e q u e n e p r o b le m s a n d t h e re p r e s e n t a t io n o f c o m p l e x
concepts.
1 . I N T R O D U C T I O N
M u c h o f t h e p r o g r e s s i n t h e f i e l d s c o n s t i t u t i n g c o g n i t i v e s c i e n c e h a s b e e n
b a s ed u p o n t h e u s e o f c o n c r e t e i n f o r m a t i o n p r o c e ss i n g m o d e l s I P M ) ,
a l m o s t e x c lu s iv e ly p a t t e r n e d a f t e r c o n v e n t i o n a l s e q u e n t i a l c o m p u t e r s .
T h e r e a r e s ev e ra l r e a s o n s f o r t ry i n g t o e x t e n d I P M t o c a se s w h e r e th e c o m -
p u t a t i o n s a r e c ar r ie d o u t b y a p a r a ll e l c o m p u t a t i o n a l e n g i n e w i th p e r h a p s
b i l l i o n s o f a c t i v e u n i t s . A s a n i n t r o d u c t i o n , w e w i l l a t t e m p t t o m o t i v a t e t h e
c u r r e n t i n t e r e s t i n m a s s i v e l y p a r a l l e l m o d e l s f r o m f o u r d i f f e r e n t p e r s p e c -
t iv e s: a n a t o m y , c o m p u t a t i o n a l c o m p l e x i t y , t e c h n o l o g y , a n d t h e r o l e o f f o r -
m a l l a n g u a g e s i n s c i e n c e . I t is t h e l a s t o f t h e s e w h i c h is o f p r i m a r y c o n c e r n
h e re . W e w ill fo c u s u p o n a p a r t i c u l a r f o r m a l i s m , c o n n e c t i o n i s t m o d e l s
C M ) , w h i c h is b a s e d e x p l ic i tl y o n a n a b s t r a c t i o n o f o u r c u r r e n t u n d e r s t a n d -
i n g o f t h e i n f o r m a t i o n p r o c e s s i n g p r o p e r t i e s o f n e u r o n s .
A n i m a l b r a in s d o n o t c o m p u t e l ik e a c o n v e n t i o n a l c o m p u t e r . C o m -
p a r a t i v e l y s lo w m i l l is e c o n d) n e u r a l c o m p u t i n g e l e m e n t s w i t h c o m p l e x ,
p a r a ll e l c o n n e c t i o n s f o r m a s t ru c t u r e w h i c h is d r a m a t i c a l l y d i f f e r e n t f r o m a
h i g h -s p e e d , p r e d o m i n a n t l y s er ia l m a c h i n e . M u c h o f c u r r e n t r e s e a rc h in t h e
n e u r o s c i e n c e s is c o n c e r n e d w i t h t r a c in g o u t t h e se c o n n e c t i o n s a n d w i t h d is -
c o v e ri n g h o w t h e y t r a n s f e r i n f o r m a t i o n . O n e p u r p o s e o f t h i s p a p e r is t o
s u g g e s t h o w c o n n e c t i o n i s t t h e o r i e s o f t h e b r a i n c a n b e u s e d t o p r o d u c e
2 0 5
8/11/2019 Connectionist models and their properties.pdf
2/50
2 6 F E L D M A N A N D B A LL AR D
t e s t a b l e , d e t a i l e d m o d e l s o f i n t e r e s t i n g b e h a v i o r s . T h e d i s t r i b u t e d n a t u r e o f
i n f o r m a t i o n p r o c e s si n g in t h e b r a i n is n o t a n e w d i s c o v e r y . T h e t r a d i t i o n a l
v ie w (w h i c h w e s h a r e d ) is t h a t c o n v e n t i o n a l c o m p u t e r s a n d l a n g u a g e s w e r e
T u r i n g u n i v e rs a l a n d c o u l d b e m a d e t o s i m u l a t e a n y p a r a l le l is m ( o r a n a l o g
v a lu e s ) w h i c h m i g h t b e r e q u i r e d . C o n t e m p o r a r y c o m p u t e r s c ie n c e h a s s h a r p -
e n e d o u r n o t io n s o f w h a t is c o m p u t a b l e t o i n cl ud e b o u n d s o n t im e , s t o r-
a g e , a n d o t h e r r e s o u r c e s . I t d o e s n o t s e e m u n r e a s o n a b l e t o r e q u i r e t h a t
c o m p u t a t i o n a l m o d e l s i n c o g n i t i v e s c i e n c e b e a t l e a s t p l a u s i b l e i n t h e i r
p o s t u l a t e d r e s o u r c e r e q u i r e m e n t s .
T h e c r i ti c a l r e s o u r c e th a t is m o s t o b v i o u s is t i m e . N e u r o n s w h o s e b a s i c
c o m p u t a t i o n a l s p e e d i s a f e w m i l l i s e c o n d s m u s t b e m a d e t o a c c o u n t f o r
c o m p l e x b e h a v i o r s w h i c h a r e c a r r ie d o u t i n a f e w h u n d r e d m i l l is e c o n d s
( P o s n e r , 1 9 7 8 ) . T h i s m e a n s t h a t
entire complex behaviors are carried out in
less than a hundred time steps C u r r e n t A I a n d s i m u l a t i o n p r o g r a m s r e q u i r e
m i l l io n s o f t im e st e p s. I t m a y a p p e a r t h a t t h e p r o b l e m p o s e d h e r e is i n h e r -
e n t l y u n s o l v a b l e a n d t h a t t h e r e i s a n e r r o r i n o u r f o r m u l a t i o n . B u t r e c e n t
r e su l ts i n c o m p u t a t i o n a l c o m p l e x i t y t h e o r y ( J a ' J a ' , 1 98 0) s u g ge s t t h a t n e t -
w o r k s o f a c ti v e c o m p u t i n g e l e m e n t s c a n c a r r y o u t a t l e as t s i m p l e c o m p u -
t a t i o n s i n t h e r e q u i r e d t i m e r a n g e . I n s u b s e q u e n t s e c t i o n s w e p r e s e n t f a s t
s o l u ti o n s t o a v a r i e ty o f r e le v a n t c o m p u t i n g p r o b l e m s . T h e s e s o l u t io n s i n -
v o l v e u s in g m a s s i v e n u m b e r s o f u n i ts a n d c o n n e c t i o n s , a n d w e a ls o a d d r e s s
t h e q u e s t i o n s o f l i m i t a t i o n s o n t h e s e r e s o u r c e s .
A n o t h e r r e c e n t d e v e l o p m e n t i s t h e f e a si b il it y o f b u i ld i n g p a r a ll e l c o m -
p u t e r s . T h e r e is c u r r e n t l y t h e c a p a b i l i t y to p r o d u c e c h i p s w i t h 1 0 0 ,0 0 0 g at e s
a t a r e p r o d u c t i o n c o s t o f a f ew c e n ts e a c h , a n d t h e t e c h n o l o g y t o g o to
1 ,0 0 0 ,0 0 0 g a t e s / c h i p a p p e a r s t o b e in h a n d . T h i s h a s tw o i m p o r t a n t c o n s e -
q u e n c e s f o r t h e s t u d y o f C M . T h e o b v i o u s c o n s e q u e n c e is t h a t i t is n o w f e a-
s ib le t o f a b r i c a t e m a s s i v e ly p ar a ll e l c o m p u t e r s , a l t h o u g h n o o n e h a s y e t d o n e
s o ( F a h l m a n , 1 98 0; H i l li s , 1 9 8 1) . T h e s e c o n d c o n s e q u e n c e o f th i s d e v e l o p -
m e n t i s t h e r e n e w e d i n t e r e s t i n t h e b a s i c p r o p e r t i e s o f h i g h l y p a r a l l e l c o m -
p u t a t i o n . A m a j o r r e a s o n w h y th e r e a r e n ' t y e t a n y o f t h e s e C M m a c h i n e s i s
t h a t w e d o n o t y e t k n o w h o w t o d e si g n , a s se m b l e , t e st , o r p r o g r a m s u c h
e n g in e s. A n i m p o r t a n t m o t i v a t i o n f o r t h e c a r e f u l s t u d y o f C M is t h e h o p e
t h a t w e w i ll l e a r n m o r e a b o u t h o w t o d o p a r a l l e l c o m p u t i n g , b u t w e w i ll s a y
n o m o r e a b o u t t h a t i n t h i s p a p e r .
T h e m o s t i m p o r t a n t r e a s o n f o r a se r io u s c o n c e r n i n c o g n it iv e sc ie n c e
f o r C M is t h a t t h e y m i g h t l e a d t o b e t t e r s c i e n c e. I t is o b v i o u s t h a t t h e c h o i c e
o f t ec h n i c a l la n g u a g e t h a t is u se d f o r e x p r e s s in g h y p o t h e s e s h a s a p r o f o u n d
i n f lu e n c e o n t h e f o r m i n w h i ch t h e o r i e s a r e f o r m u l a t e d a n d e x p e r i m e n t s
u n d e r t a k e n . A r t i f i c i a l i n t e ll i g e n c e a n d a r t i c u l a t i n g c o g n i t i v e sc i en c e s h a v e
m a d e g r e a t p r o g r e s s b y e m p l o y i n g m o d e l s b a s e d o n c o n v e n t i o n a l d i g i t a l
c o m p u t e r s a s t h e o r i es o f i n te l li g e nt b e h a v i o r . B u t a n u m b e r o f c r u c ia l
p h e n o m e n a s u c h a s a s s o c i a t i v e m e m o r y , p r i m i n g , p e r c e p t u a l r i v a l r y , a n d
8/11/2019 Connectionist models and their properties.pdf
3/50
CO NNE C TI ONI S T MO DE LS A N D THE IR P ROP ERTIE S 2 7
the remarkable recovery ability of animals have not yielded to this treat-
ment. A major goal of this paper is to lay a foundation for the systematic
use of massively parallel connectionist models in the cognitive sciences, even
where these are not yet reducible to physiology or silicon.
Over the past few years, a number of investigators in different fields
have begun to employ highly parallel models (idiosyncratically) in their
work. The general idea has been advocated for animal models by Arbib
(1979) and for cognitive models by Anderson (Anderson et al., 1977) and
Ratcliff (1978). Parallel search of semantic memory and various spreading
act ivat ion theories have become common (though not quite consistent)
parts of information processing modeling. In machine perception research,
massively parallel, cooperative computational theories have become a
dominant paradigm (Marr & Poggio, 1976; Rosenfeld et al., 1976) and
many of our examples come from our own work in this area (Ballard, 1981;
Sabbah, 1981). Scientists looking at performance errors and other non-
repeatable behaviors have not found conventional IPM to be an adequate
framework for their efforts. Norman (1981) has recently summarized argu-
ments from cognitive psychology, and Kinsbourne and Hicks (1979) have
been led to a similar view from a different perspective. It appears to us that
all of these efforts could fit within the CM paradigm outlined here.
One of the most interesting recent studies employing CM techniques is
the partial theory of reading developed in (McClelland & Rumelhart, 1981).
They were concerned with the word superiority effect and related questions
in the perception of printed words, and had a large body of experimental
data to explain. One major finding is that the presence of a printed letter in a
brief display is easier to determine when the letter is presented in the context
of a word than when it is presented alone. The model they developed (cf.
Figure 1) explicitly represents three levels of processing: visual features of
printed letters, letters, and words. The model assumes that there are positive
and negative (circular tipped) connections from visual features to the letters
that they can (respectively, cannot) be part of. The connections between let-
ters and words can go in either direction and embody the constraints of
English. The model assumes that many units can be simultaneously active,
that units form algebraic sums of their inputs and output values propor-
tionally. The activity of a unit is bounded from above and below, has some
memory, and decays with time. All of these features, and several more, are
captured in the abstract unit described in Section 2.
This idea of simultaneously evaluating many hypotheses (here words)
has been successfully used in machine perception for some time (Hanson &
Riseman, 1978). What has occurred to us relatively recently is that this is a
natural mode of computation for widely interconnected networks of active
elements like those envisioned in connectionist models. The generalization
of these ideas to the connectionist view of brain and behavior is that all im-
8/11/2019 Connectionist models and their properties.pdf
4/50
208 FELDMAN AN D BALLARD
Figu re 1 . A f e w o f th e n e ig h o r s o f t h e n o d e f o r t h e l e t t e r t i n t he f i rs t p o s i t i o n in a w o r d ,
and the i r i n te rconnec t ions (McC le l land & Rume lha r t , 1981 ) .
portant encodings in the brain are in terms of the relative strengths of
synaptic connections. The fundamental premise of connectionism is that in-
dividual neurons do not transmit large amounts of symbolic information
Instead they compute by being appropriately connected to large numbers of
similar units. This is in sharp contrast to the conventional computer model
of intelligence prevalent in computer science and cognitive psychology.
The fundamental distinction between the conventional and connec-
tionist comput ing models can be conveyed by the following example. When
one sees an apple and says the phrase wormy app le, some information
must be transferred, however indirectly, from the visual system to the
speech system. Either a sequence of special symbols that denote a wormy
apple is transmitted to the speech system, or there are special connections to
the speech command area for the words. Figure 2 is a graphic presentation
of the two alternatives. The path on the right described by double-lined
arrows depicts the situation (as in a computer) where the information that a
wormy apple has been seen is encoded by the visual system and sent as an
abstract message (perhaps frequency-coded) to a general receiver in the
speech system which decodes the message and initiates the appropriate
speech act. Notice that a complex message would presumably have to be
transmitted sequentially on this channel, and that each end would have to
8/11/2019 Connectionist models and their properties.pdf
5/50
CONNECTIONISTMODELSAND THEIRPROPERTIES 209
der
~ o d e r
Figure 2 Connectionism s sym bolicencoding
s s u m e s s o m e gen eral encoding
Assumes ndividualconnections
l ea r n th e c o m m o n c o d e f o r e v e r y n e w c o n c e p t . N o o n e h a s y et p r o d u c e d a
b i o l o g ic a l ly a n d c o m p u t a t i o n a l l y p l a u s ib l e r e a l i z at i o n o f t h is c o n v e n t i o n a l
c o m p u t e r m o d e l .
T h e o n l y a lt e r n a t iv e t h a t w e h a v e b e e n a b l e t o u n c o v e r is d e s c r i b ed b y
t h e p a t h w i t h s i n g l e - w i d t h a r r o w s . T h i s s u g g e s t s t h a t t h e r e a r e ( i n d i r e c t )
l i n k s f r o m t h e u n i t s ( c e l l s , c o l u m n s , c e n t e r s , o r w h a t - h a v e - y o u ) t h a t
r e c o g n i z e a n a p p l e t o s o m e u n i ts r e s p o n s i b l e f o r s p e a k in g t h e w o r d . T h e
c o n n e c t i o n i s t m o d e l r e q u i r e s o n l y v e r y s im p l e m e s sa g e s ( e .g . s t i m u l u s
s t r e n g t h ) t o c r o s s a c h a n n e l b u t p u t s s t r o n g d e m a n d s o n t h e a v a i l a b i l i t y o f
t h e r ig h t c o n n e c t i o n s . Q u e s t i o n s c o n c e r n i n g t h e le a r n in g a n d r e i n f o r c e m e n t
o f c o n n e c t i o n s a r e a d d r e s s e d i n F e l d m a n , ( 1 98 1 b ).
F o r a n u m b e r o f r e a s o n s ( i n c l u d i n g r e d u n d a n c y f o r r e l i a b i l i t y ) , i t i s
h i g h ly u n l i k e ly t h a t t h e r e i s e x a c t ly o n e n e u r o n f o r e a c h c o n c e p t , b u t t h e
p o i n t o f v ie w t a k e n h e r e is t h a t t h e a c t i v it y o f a sm a l l n u m b e r o f n e u r o n s
( s ay 10) e n c o d e s a c o n c e p t l i k e a p p l e . A n a l t e r n a t i v e v ie w ( H i n t o n & A n d e r -
s o n , 1 98 1) is t h a t c o n c e p t s a r e r e p r e s e n t e d b y a p a t t e r n o f a c t i v i t y i n a
m u c h l a rg e r s et o f n e u r o n s ( s a y 1 ,0 00 ) w h i c h a l s o r e p r e s e n t m a n y o t h e r c o n -
c e p ts . W e h a v e n o t s e en h o w t o c a r r y o u t a p r o g r a m o f s p e ci fi c m o d e l i n g i n
t e r m s o f t h e se d i f fu s e m o d e l s . O n e o f t h e m a j o r p r o b l e m s w i t h d i f fu s e
8/11/2019 Connectionist models and their properties.pdf
6/50
21 FELDMAN AND BALLARD
m o d e l s a s a p a ra l le l c o m p u t a t i o n s c h e m e is c r o s s -t a lk a m o n g c o n c e p t s . F o r
e x a m p l e , i f c o n c e p t s u s i n g u n i t s 1 0 , 2 0 , 3 0 . . . . ) a n d 5 , 15 , 2 5 . . . . ) w e r e
s i m u l t a n e o u s l y a c t i v a t e d , m a n y o t h e r c o n c e p t s , e . g . , 2 0 , 2 5 , 3 0 , 3 5 . . . . )
w o u l d b e a c t i v e a s w e ll . I n t h e e x a m p l e o f F i g u r e 2 , t h is m e a n s t h a t d i f f u s e
m o d e l s w o u l d b e m o r e l i ke t h e s h a r e d s e q u e n t ia l c h a n n e l . A l t h o u g h a si n gl e
c o n c e p t c o u l d b e t r a n s m i t t e d i n p a ra l le l , c o m p l e x c o n c e p t s w o u l d h a v e t o
g o o n e a t a t i m e . S i m u l t a n e o u s l y t r a n s m i t t i n g m u l t ip l e c o n c e p t s th a t s h a r e d
u n i t s w o u l d c a u s e c r o s s - t a l k . I t i s s t i l l t r u e i n o u r C M t h a t m a n y r e l a t e d
u n i t s w i l l b e t r i g g e r e d b y s p r e a d i n g a c t i v a t i o n , b u t t h e r e p r e s e n t a t i o n o f
e a c h c o n c e p t i s t a k e n t o b e c o m p a c t .
M o s t c o g n i t i v e s c ie n t is ts b e l i e v e t h a t t h e b r a i n a p p e a r s t o b e m a s s i v e l y
p a r a l le l a n d t h a t s u c h s t r u c t u r e s c a n c o m p u t e s p e c ia l f u n c t i o n s v e r y w e l l.
B u t m a s s i v e l y p a r a l l e l s t r u c t u r e s d o n o t s e e m t o b e u s a b l e f o r g e n e r a l p u r -
p o s e c o m p u t i n g a n d t h e r e is n o t n e a r l y a s m u c h k n o w l e d g e o f h o w t o c o n -
s t ru c t a n d a n a l y z e s u c h m o d e l s . T h e c o m m o n b e l i e f w h i c h m a y w e ll b e
r ig h t ) is t h a t t h e r e a r e o n e o r m o r e i n t e r m e d i a t e l e ve ls o f c o m p u t a t i o n a l
o r g a n i z a t i o n l a y e r e d o n th e n e u r o n a l s t r u c t u r e , a n d t h a t t h e o r i e s o f i n te l li -
g e n t b e h a v i o r s h o u l d b e d e s c r i b e d i n t e r m s o f t h e s e h ig h e r - l e v e l l a n g u a g e s ,
s u ch a s P r o d u c t i o n S y s t e m s , P r e d i c a t e C a l c u l u s, o r L I S P . W e h a v e n o t s e e n
a r e d u c t i o n i n t e r p re t e r , if y o u w i ll) o f a n y h i g h e r f o r m a l i s m w h i c h h a s
p l a u s i b l e r e s o u r c e r e q u i r e m e n t s , a n d t h is is a p r o b l e m w e l l w o r t h p u r s u i n g .
O u r a t t e m p t s t o d e v e l o p c o g n i t iv e s c ie n c e m o d e l s d i r e c tl y in n e u r a l
t e r m s m i g h t f a il f o r o n e o f t w o r e a s o n s . I t m a y b e t h a t t h e r e r e a l l y is a n i n -
t e r p r e t e d s y m b o l s y st e m i n a n im a l b r a i n s . I n t hi s ca s e w e w o u l d h o p e t h a t
o u r e f f o r t s w o u l d b r e a k d o w n i n a w a y t h a t c o u l d s h e d l i g h t o n t h e n a t u r e
o f t h is s y m b o l s y s t em . T h e o t h e r p o s s ib i li ty is t h a t C M t e c h n i q u e s a r e
d i r e ct l y a p p l ic a b l e b u t w e a r e u n a b l e t o f ig u r e o u t h o w t o m o d e l s o m e i m -
p o r t a n t c a p a c i t y , e .g . , p l an n i n g . O u r p r o g r a m is t o c o n t i n u e t h e C M a t t a c k
o n p r o b l e m s o f in c r ea s in g d i f f i c u l ty a n d t o i n d u c e s o m e o f y o u t o j o i n u s )
u n t il w e e n c o u n t e r o n e t h a t is i n t r a c t a b l e in o u r t e r m s . T h e r e a r e a n u m b e r
o f p r o b l e m s t h a t a r e k n o w n t o b e d i f f i c u lt f o r sy s t em s w i t h o u t a n i n t e r p r e t e d
s y m b o l i c r e p r e s e n t a t i o n , i n c l ud i n g c o m p l e x c o n c e p t s , le a r n i n g , a n d n a t u r a l
l a n g u a g e u n d e r s t a n d i n g . T h e c u r r e n t p a p e r is m a i n l y c o n c e r n e d w i t h la y in g
o u t t h e f o r m a l i s m a n d s h o w i n g h o w i t a p p l i e s i n t h e e a s y c a s es , b u t w e d o
a d d r e s s t he p r o b l e m o f c o m p l e x c o n c e p t s in S e c t io n 4 . W e h a v e m a d e s o m e
p r o g r e s s o n t h e p r o b l e m o f l e a rn i n g i n C M s y s te m s F e l d m a n , 1 98 1b ) a n d
a r e b e g i n n i n g t o w o r k s e r io u s l y o n n a t u r a l l a n g u a g e p r o c e s si n g a n d o n
h i g h er -l e v el v i si o n . O u r e f f o r t s o n p l a n n i n g a n d l o n g - t e r m m e m o r y r e o r g a -
n i z a t io n h a v e n o t a d v a n c e d s i g n if i c an t l y b e y o n d t h e d i s cu r s iv e p r e s e n t a t i o n
i n F e l d m a n , 1 9 8 0) .
W e w i l l c e r t a i n l y n o t g e t v e r y f a r i n t h i s p r o g r a m w i t h o u t d e v e l o p i n g
s o m e s y s t e m a t ic m e t h o d s o f a tt a c k i n g C M t a sk s a n d s o m e b u i ld i n g - b l o c k
c i r c u i t s w h o s e p r o p e r t i e s w e u n d e r s t a n d . A f i r s t s t e p t o w a r d s a s y s t e m a t i c
d e v e l o p m e n t o f C M is t o d e f i n e a n a b s t r a c t c o m p u t i n g u n i t . O u r u n i t is
8/11/2019 Connectionist models and their properties.pdf
7/50
C O N N E C T I O N I S T M O D E L S A N D T H EIR P RO P ER TIE S 2
r a t h e r m o r e g e n e r a l t h a n p r e v i o u s p r o p o s a l s a n d is i n t e n d e d t o c a p t u r e t h e
c u r r e n t u n d e r s t a n d i n g o f t h e i n f o r m a t i o n p r o c e s s i n g c a p a b i l i t i e s o f
n e u r o n s . S o m e u s e f u l s p e ci a l c a se s o f o u r g e n e r a l d e f i n i t io n a n d s o m e p r o p -
e r ti es o f v e r y s i m p l e n e t w o r k s a r e d e v e l o p e d i n S e c t i o n 2 . A m o n g t h e k e y
i d e a s a r e l o c a l m e m o r y , n o n - h o m o g e n e o u s a n d n o n - l i n e a r f u n c t i o n s , a n d
t h e n o t i o n s o f m u t u a l i n h i b i ti o n a n d s t ab l e c o a li t io n s .
A m a j o r p u r p o s e o f t h e re s t o f t h e p a p e r is t o d e s c r ib e b u i ld i n g b l o c k s
w h i c h w e h a v e f o u n d u s e f u l i n c o n s t r u c t i n g C M s o l u t i o n s t o v a r i o u s t a s k s .
T h e c o n s t r u c t i o n s a r e i n t e n d e d t o b e u s e d t o m a k e s p e c if ic m o d e l s b u t t h e
e x a m p l e s i n t h is p a p e r a r e o n l y s u g g es ti v e. W e p r e s e n t a n u m b e r o f C M
s o l u t i o n s t o g e n e r a l p r o b l e m s a r i s i n g i n i n t e l l i g e n t b e h a v i o r , b u t
we are not
suggesting that any o f these are necessarily employed by nature. O u r n o t i o n
o f a n a d e q u a t e m o d e l is o n e t h a t a c c o u n t s f o r
all
o f t h e e s t a b l i s h e d r e l e v a n t
f i n d i n g s a n d t h i s is n o t a ta s k t o b e u n d e r t a k e n l ig h t ly . W e a r e d e v e l o p i n g
s o m e p r e l i m i n a r y s k e t c h e s (B a l l a r d & S a b b a h , 1 9 81 ; S a b b a h , 1 9 81 ) f o r a
s e r io u s m o d e l o f l o w a n d i n t e r m e d i a t e l e ve l v i si o n . A s w e d e v e l o p v a r i o u s
b u i ld i n g b l o c k s a n d t e c h n i q u e s w e w ill al so b e t r y i n g t o b u r y s o m e o f th e
c o n t a m i n a t e d d e b ri s o f p as t n e u r a l m o d e l i n g e f f o r ts . M a n y o f o u r c o n s t ru c -
t io n s a re i n te n d e d a s an s w e rs t o k n o w n h a r d p r o b l e m s in C M c o m p u t a t i o n .
A m o n g t h e i s s u e s a d d r e s s e d a r e : s t a b i l i t y a n d n o i s e - s e n s i t i v i t y , d i s t r i b u t e d
d e c i s i o n - m a k i n g , t i m e a n d s e q u e n c e p r o b l e m s , a n d t h e r e p r e s e n t a t i o n o f
c o m p l e x c o n c e p t s . T h e c r u c ia l q u e s ti o n s o f l e a rn i n g a n d c h a n g e i n C M
s y s t e m s a r e d i s c u s s e d e ls e w h e r e ( F e l d m a n , 1 9 8 1b ) .
2 . N E U R O N - L I K E C O M P U T I N G U N I T S
A s p a r t o f o u r e f f o r t to d e v e l o p a g en e r a ll y u s e fu l f r a m e w o r k f o r c o n n e c -
t io n i s t th e o r i e s , w e h a v e d e v e l o p e d a s t a n d a r d m o d e l o f t h e i n d i v id u a l u n i t .
I t w ill t u r n o u t t h a t a u n i t m a y b e u s e d t o m o d e l a n y t h i n g f r o m a s m a l l
p a r t o f a n e u r o n t o t h e e x t e r n a l f u n c t i o n a l i t y o f a m a j o r s u b s y s t e m . B u t t h e
b a si c n o t i o n o f u n i t is m e a n t t o lo o s e l y c o r r e s p o n d t o a n i n f o r m a t i o n p r o -
c es si ng m o d e l o f o u r c u r r e n t u n d e r s t a n d i n g o f n e u r o n s . T h e p a r t i c u la r
d e f i n i t io n s h e r e w e r e c h o s e n t o m a k e it e a s y to s p e c i f y d e t a il e d e x a m p l e s o f
r e la t iv e l y c o m p l e x b e h a v i o r s . T h e r e i s n o a t t e m p t t o b e m i n i m a l o r m a t h e -
m a t i c a l l y e l e g a n t. T h e v a r i o u s n u m e r i c a l v a l ue s a p p e a r i n g i n t h e d e f i n i ti o n s
a r e a r b i t r a r y , b u t f i x ed f in i t e b o u n d s p l a y a c r u c i a l r o l e i n t h e d e v e l o p m e n t .
T h e p r e s e n t a t i o n o f t h e d e f i n i t i o n s w i ll b e in s ta g e s, a c c o m p a n i e d b y e x-
a m p l e s . A c o m p a c t t e c h n i c a l s p e c i f ic a t i o n f o r r e f e r e n c e p u r p o s e s is i n c l u d e d
a s A p p e n d i x A . E a c h u n i t w ill b e c h a r a c t e r i z e d b y a s m a l l n u m b e r o f d is -
c r e t e s t a t e s p l u s :
p - - a c o n t i n u o u s v a l u e i n [ - 10 , 10], ca l le d
potential
( a c c u r a c y o f s e v e r a l d i g it s )
v - - a n output value i n t e g e r s O _ v _ 9
i - - a v e c t o r o f inputs i , . . . . . in
8/11/2019 Connectionist models and their properties.pdf
8/50
2 2 FELDMAN AND BALLARD
P U n i ts
F o r s o m e a p p l i c a t i o n s , w e w i l l b e a b l e t o u s e a p a r t i c u l a r l y s i m p l e k i n d o f
u n i t w h o s e o u t p u t v is p r o p o r t i o n a l t o i ts p o t e n t i a l p ( r o u n d e d ) w h e n p > 0
a n d w h i ch h a s o n l y o n e s t at e . I n o t h e r w o r d s
p - - p + /3
Z; w kik
V -- if p> O then r o u n d ( p - 0 ) e /s e 0
[0_~ wk_< 1]
[v = 0 . . .91
w h e r e / ~ , 0 a r e c o n s t a n t s a n d w ~ a r e w e i g h t s o n t h e i n p u t v a l u e s . T h e w e i g h t s
a r e th e s o l e l o c u s o f c h a n g e w i t h e x p e r i e n c e i n t h e c u r r e n t m o d e l . M o s t
o f t e n , t h e p o t e n t i a l a n d o u t p u t o f a u n i t w il l b e e n c o d i n g i ts confidence a n d
w e w ill s o m e t i m e s u s e t hi s t e r m . T h e - - n o t a t i o n is b o r r o w e d f r o m th e
a s si g nm e n t s t a te m e n t o f p r o g r a m m i n g l an g u a ge s . T h i s n o t a t i o n c o v e r s b o t h
c o n t i n u o u s a n d d i s cr e te t i m e f o r m u l a t i o n s a n d a l l o w s u s t o t a lk a b o u t s o m e
i ss ue s w i t h o u t a n y e x p l i c it m e n t i o n o f ti m e . O f c o u r s e , c e r t a i n o t h e r q u e s -
t io n s w ill i n h e r e n t l y in v o l v e t i m e a n d c o m p u t e r s i m u l a t i o n o f a n y n e t w o r k
o f u n i t s w i l l r a i s e d e l i c a t e q u e s t i o n s o f d i s c r e t i z i n g t i m e .
T h e r e s t r i c t i o n t h a t o u t p u t t a k e o n s m a l l i n t e g e r v a l u e s i s c e n t r a l t o
o u r e n t e r p r is e . T h e f i r i n g f r e q u e n c ie s o f n e u r o n s r a n g e f r o m a f e w to a f e w
h u n d r e d i m p u ls e s p e r s e c o n d . I n t h e 1 / 1 0 s e c o n d n e e d e d f o r b a s ic m e n t a l
e v e n t s , t h e r e c a n o n l y b e a l i m i t e d a m o u n t o f i n f o r m a t i o n e n c o d e d i n f r e -
q u e n c ie s . T h e t en o u t p u t v a l ue s ar e a n a t t e m p t t o c a p t u r e t h is i d ea . A m o r e
a c c u r a t e r e n d e r i n g o f n e u r a l e v e n t s w o u l d b e t o a l l o w 1 0 0 d i s c r e t e v a l u e s
w i t h n o i se o n t r a n s m i s s i o n ( c f. S e j n o w s k i , 1 9 77 ). T r a n s m i s s i o n t i m e is
a s s u m e d t o b e n eg l ig i b le ; d e l a y u n i t s c a n b e a d d e d w h e n t r a n s i t t i m e n e e d s
t o b e t a k e n i n t o a c c o u n t .
T h e p - u n i t is s o m e w h a t l ik e c la s si ca l l i n e a r t h r e s h o l d e l e m e n t s ( M i n s k y
& P a p e r t , 1 9 7 2 ) , b u t t h e r e a r e s e v e r a l d i f f e r e n c e s . T h e p o t e n t i a l , p , i s a
c r u d e f o r m o f m e m o r y a n d i s a n a b s t r a c t i o n o f th e in s t a n t a n e o u s m e m b r a n e
p o t e n t i a l t h a t c h a r a c t e r i z e s n e u r o n s ; i t g r e a t l y r e d u c e s t h e n o i s e s e n s it i v it y o f
o u r n e t w o r k s . W i t h o u t l o c a l m e m o r y i n t h e u n i t , o n e m u s t g u a r a n t e e t h a t
a ll t h e i n pu t s r e q u i r e d f o r a c o m p u t a t i o n a p p e a r s i m u l t a n e o u s l y a t th e u n i t .
O n e p r o b l e m w i t h t h e d e f i n i t i o n a b o v e o f a p - u n i t i s t h a t i t s p o t e n t i a l
d o e s n o t d e c a y i n t h e a b s e n c e o f i n p u t . T h i s d e c a y i s b o t h a p h y s i c a l p r o p -
e r t y o f n e u r o n s a n d a n i m p o r t a n t c o m p u t a t i o n a l f e a tu r e f o r o u r h ig h ly
p a r a l l e l m o d e l s . O n e c o m p u t a t i o n a l t r i c k t o s o l v e t h i s i s t o h a v e a n i n -
h i b i t o r y c o n n e c t i o n f r o m t h e u n i t b a c k t o i t se lf . I n f o r m a l l y , w e i d e n t i f y t h e
n e g a t i v e s e lf f e e d b a c k w i t h a n e x p o n e n t i a l d e c a y in p o t e n t i a l w h i c h i s
m a t h e m a t i c a l l y e q u i v a l e n t . W i t h t h is a d d i t i o n , p - u n it s c a n b e u s e d f o r
m a n y C M t as k s o f i n t e r m e d i a t e d i f f ic u l t y . T h e I n t e r a c t i v e A c t i v a t i o n
m o d e l s o f M c C l e l l a n d a n d R u m e l h a r t c a n b e d e s c r i b e d n a t u r a l l y w i t h
p - u n it s , a n d s o m e o f o u r o w n w o r k ( B a l l a rd , 1 98 1) a n d t h a t o f o t h e r s ( M a r r
8/11/2019 Connectionist models and their properties.pdf
9/50
C ON N EC T I ON I ST M OD ELS A N D T H EI R PR OPER TIES 21 3
& P o g g i o , 1 97 6) ca n b e d o n e w i th p - u n i t s . B u t th e r e a r e a n u m b e r o f a d d i -
t io n a l f e a t u r e s w h ic h w e h a v e f o u n d v a l u a b l e in m o r e c o m p l e x m o d e l i n g
t a s k s .
D i s j u n c t iv e F i r in g o n d i t io n s a n d o n j u n c t iv e o n n e c t i o n s
I t i s b o t h c o m p u t a t i o n a l l y e f f i c i e n t a n d b i o l o g i c a l l y r e a l i s t i c t o a l l o w a u n i t
t o r e s p o n d t o o n e o f a n u m b e r o f a l te r n a t i v e c o n d i t i o n s . O n e w a y t o vi e w
t hi s is t o i m a g i n e t h e u n it h a v i n g d e n d r i t e s e a c h o f w h i c h d e p ic t s a n a lt e r-
n a t i v e e n a b l in g c o n d i t in n ( F ig u r e 3 ). F o r e x a m p l e , o n e c o u l d e x t e n d t h e n e t-
w o r k o f F i g u r e 1 t o a l lo w f o r s ev e r a l d i f f e r e n t t y p e f o n t s a c t i v a t in g t h e s a m e
l e tt e r n o d e , w i t h t h e h i g h e r c o n n e c t i o n s u n c h a n g e d . B i o l o g i c a l ly , t h e f ir i n g
o f a n e u r o n d e p e n d s , in m a n y c as e s , o n lo c al s p a t i o - t e m p o r a l s u m m a t i o n
i n v o l v in g o n l y a s m a l l p a r t o f t h e n e u r o n ' s s u r f a c e . S o - c a ll e d d e n d r i t ic
s p i k e s t r a n s m i t t h e a c t i v a t i o n t o t h e r e s t o f t h e c e l l .
i
i
is
i 8
i7
F i g u r e 3 C o n j u n c t i v e c o n n e c t i o n s a n d d i s ju n c t i v e i n p u t s i t e s
I n te r m s o f o u r f o r m a l i s m , t h i s c o u l d b e d e s c r ib e d in a v a r i e t y o f
w a y s . O n e o f t h e s i m p l e st is t o d e f i n e th e p o t e n t i a l i n t e r m s o f t h e m a x i m u m
o f t h e s e p a r a te c o m p u t a t i o n s , e . g . ,
p - - p + /~ M ax ( i , + i2 - ~ , i3 + i , - ~ , is + i6 - i7 - )
w h e r e / 3 is a s c al e c o n s t a n t a s in t h e p - u n i t a n d i s a c o n s t a n t c h o s e n ( u s u a l l y
> 10) t o s u p p r e s s n o i s e a n d r e q u i r e t h e p r e s e n c e o f m u l t i p l e a c t i v e i n p u t s
( S a b b a h , 1 9 81 ). T h e m i n u s s i gn a s s o c i a t e d w i th i , c o r r e s p o n d s t o i ts b e i n g
a n i n h i b i t o r y i n p u t .
I t d o e s n o t s e e m u n r e a s o n a b l e ( g iv e n c u r r e n t d a t a , K u f f l e r & N ic h o l ls ,
1 97 6) to m o d e l t h e f i r i n g r a te o f s o m e u n i t s a s th e m a x i m u m o f t h e r a te s a t
its a ct iv e si te s. U n i t s w h o s e p o t e n t i a l is c h a n g e d a c c o r d i n g t o th e m a x i m u m
o f a s e t o f a l g e b r a i c s u m s w il l o c c u r f r e q u e n t l y i n o u r s p e c i fi c m o d e l s . O n e
a d v a n t a g e o f k e e p i n g th e p r o c e s s i n g p o w e r o f o u r a b s t r a c t u n i t c lo s e t o th a t
o f a n e u r o n is t h a t i t h e l ps i n f o r m o u r c o u n t i n g a r g u m e n t s . W h e n w e a t -
8/11/2019 Connectionist models and their properties.pdf
10/50
2 4 F E L D M A N A N D B AL LA RD
t e m p t t o m o d e l a p a r t i c u l a r f u n c t i o n ( e . g ., s t e r e o p s i s) , w e e x p e c t t o r e q u i r e
t h a t t h e n u m b e r o f u n i t s a n d c o n n e c t i o n s a s w e ll a s t h e e x e c u t i o n t im e r e-
q u i r e d b y t h e m o d e l a r e p l a u s i b l e .
T h e m a x - o f - s u m u n it is t h e c o n t i n u o u s a n a l o g o f a lo g ic a l O R - o f - A N D
( d i sj u n c ti v e n o r m a l f o r m ) u n i t a n d w e w ill s o m e t i m e s u s e t h e l a t t e r as a n a p -
p r o x i m a t e v e r si o n o f t h e f o r m e r . T h e O R - o f - A N D u n i t c o r re s p o n d i n g t o
F ig u re 3 i s :
p - p + ot O R ( i , i2 , i3 i , , i s i~ (no t i , ) )
T h i s f o r m u l a t i o n s tr es se s t h e i m p o r t a n c e t h a t n e a r b y s p a t i a l c o n n e c t i o n s ll
b e f i r i n g b e f o r e t h e p o t e n t i a l i s a f f e c t e d . H e n c e , i n t h e a b o v e e x a m p l e , i 3
a n d i4 m a k e a
conjunctive connection
w i t h t h e u n it . T h e e f f e c t o f a c o n j u n c -
t iv e c o n n e c t i o n c a n a l w a y s b e s im u l a t e d w i th m o r e u n i t s b u t t h e n u m b e r o f
e x t r a u n i t s m a y b e v e r y l a r g e .
Q-Uni ts and Compound Uni ts
A n o t h e r u s e f u l sp e c ia l c as e ar is e s w h e n o n e s u p p r e s s e s t h e n u m e r i c a l p o t e n -
t ia l , p , a n d r e li es u p o n a f i n i t e - s t a t e s e t { q } f o r m o d e l i n g . I f w e a l s o i d e n t i f y
e a c h i n p u t o f i w i t h a s e p a r a t e n a m e d i n p u t s i g n a l , w e c a n g e t cl a ss i ca l f i n it e
a u t o m a t a . A s i m p le e x a m p l e w o u l d b e a u n i t t h a t c o u l d b e s t a r t e d o r s t o p p e d
f r o m f i r i n g .
O n e c o u l d d e s c r ib e t h e b e h a v i o r o f t h i s u n i t b y a t a b l e , w i th r o w s c o r -
r e s p o n d i n g t o s t a t e s i n { q } a n d c o l u m n s t o p o s s i b l e i n p u t s , e . g . ,
i~ (s ta rt ) i2 s top)
F i r i n g F i r i n g N u l l
N u l l N u l li r i n g
O n e w o u l d a l s o h a v e t o s p e c if y a n o u t p u t f u n c t i o n , g i v i n g o u t p u t v a l u e s r e -
q u i r e d b y t h e r e s t o f t h e n e t w o r k , e . g . ,
v - - i f q = F i r i n g t h e n 6 e ls e 0.
T h i s c o u l d a l s o b e a d d e d t o t h e t a b l e a b o v e . A n e q u i v a l e n t n o t a t i o n w o u l d
b e t r a n s i ti o n n e t w o r k s w i t h s ta t es a s n o d e s a n d i n p u t s a n d o u t p u t s o n t h e
a r c s .
I n o r d e r t o b u i l d m o d e l s o f i n t e r e s t i n g b e h a v i o r s w e w i l l n e e d t o
e m p l o y m a n y o f t h e s a m e t e c h n iq u e s u s e d b y d e si gn e rs o f c o m p l e x c o m -
p u t e r s a n d p r o g r a m s . O n e o f th e m o s t p o w e r f u l t e c h n i q u e s w i ll b e e n c a p s u -
8/11/2019 Connectionist models and their properties.pdf
11/50
CONNECTIONISTMODELSAN D THEIRPROPERTIES 215
lation and abst raction o f a subnetwork by an individual unit. For example,
a system that had separate motor abilities for turning eft and turning right
(e.g., fins) could use two start-stop units to model a turn-unit, as shown in
Figure 4.
l e f t start ~ c u s e s
s t o p ~ ~ ~ m ot ion
t o left
~ ~ s t e c u s e s
righ t rt . ~ -~
m o t i o n
~ . s t o p ~ t o r ig h t
Figure 4. A Turn Unit .
Note that the compound unit here has two distinct outputs, where
basic units have only one (which can branch, of course). In general, com-
pound units will differ from basic ones only in that they can have several
distinct outputs.
The main point of this example is that the turn-unit can be described
abstractly, independent of the details of how it is built. For example, using
the tabular conventions described above,
e f t R i g h t V a l u e s O u t p u t
a gauche a gauche ad ro i t v, =7 , v2=O
ad ro it a gauche ad ro it v, =0, v2=8
where the right-going output being larger than the left could mean that we
have a right-finned robot . There is a great deal more that must be said about
the use of states and symbolic input names, about multiple simultaneous in-
puts, etc., but the idea of describing the external behavior of a system only
in enough detail for the task at hand is of great importance. This is one of
the few ways known of coping with the complexity of the magnitude needed
for serious modeling of biological functions . It is not strictly necessary that
the same formalism be used at each level of functional abstraction and, in
the long run, we may need to employ a wide range of models. For example,
for certain purposes one might like to expand our units in terms of compar t-
mental models of neurons like those of (Perkel, 1979). The advantage of
keeping within the same formalism is that we preserve intuition, mathe-
matics, and the ability to use existing simulation programs. With sufficient
care, we can use the units defined above to represent large subsystems with-
8/11/2019 Connectionist models and their properties.pdf
12/50
2 6 F E L D M A N A N D B AL LA RD
o u t g iv i n g u p t h e n o t i o n t h a t e a c h u n i t c a n s t a n d f o r a n a b s t r a c t n e u r o n . T h e
c r u c i a l p o i n t i s t h a t a s u b s y s t e m m u s t b e e l a b o r a t e d i n t o it s n e u r o n - l e v e l
u n i ts f o r t i m i n g a n d s i ze c a l c u l a ti o n s , b u t c a n h o p e f u l l y ) b e d e s c r i b e d m u c h
m o r e s i m p l y w h e n o n l y i ts e f f e c ts o n o t h e r s u b s y s t e m s a r e o f d i re c t c o n c e r n .
U n i t s E m p l o y i n g p a n d q
I t w i ll a l r e a d y h a v e o c c u r r e d t o t h e r e a d e r t h a t a n u m e r i c a l v a l u e , l ik e o u r p ,
w o u l d b e u s e f u l f o r m o d e l i n g t h e a m o u n t o f t u r n i n g t o th e le f t o r r ig h t i n
t h e l a s t e x a m p l e . I t a p p e a r s t o b e g e n e r a l l y t r u e t h a t a s in g l e n u m e r i c a l v a l u e
a n d a s m a l l se t o f d i s c r e te s t at e s c o m b i n e t o p r o v i d e a p o w e r f u l y e t t r a c t a b l e
m o d e l i n g u n i t . T h i s is o n e r e a s o n t h a t t h e c u r r e n t d e f i n i ti o n s w e r e c h o s e n .
A n o t h e r r e a s o n is t h a t t h e m i x e d u n i t s e e m s t o b e a p a r t i c u l a r l y c o n v e n i e n t
w a y o f m o d e l in g t h e i n f o r m a t i o n p r o c es s in g b e h a v i o r o f n e u r o n s , a s g e n e r -
a l ly d e s c r ib e d . T h e d i s c r e te s t a te s e n a b l e o n e t o m o d e l t h e e f f e c t s i n n e u r o n s
o f p o l y p e p t id e m o d u l a t o r s , a b n o r m a l c h e m i c a l e n v i r o n m e n t s , f a ti g u e, e tc .
A l t h o u g h t h e s e e f f e c ts a r e o f t e n c o n t i n u o u s f u n c t i o n s o f un i t p a r a m e t e r s ,
t h e r e a r e s e v e r a l a d v a n t a g e s t o u s i n g d i s c r e t e s t a t e s in o u r m o d e l s . S c i e n ti s ts
a n d l a y m e n al ik e o f t e n g i v e d i st in c t n a m e s e . g . , c o o l , w a r m , h o t ) t o p a r a m -
e t e r r a n g e s t h a t t h e y w a n t t o t r e a t d i f f e r e n t l y . W e a l s o c a n e x p l o i t a l a r g e
l i te r a t u re o n u n d e r s t a n d i n g l o o s e l y - c o u p l e d s y s t e m s a s f i n it e - st a te m a c h i n e s
S u n s h i n e , 1 97 9). I t is a l s o t r a d i t i o n a l t o b r e a k u p a f u n c t i o n i n t o s e p a r a t e
r a n g e s w h e n i t is s i m p l e r t o d e s c r ib e t h a t w a y . W e h a v e a l r e a d y e m p l o y e d a l l
o f th e s e u se s o f d is c r et e s t a te s i n o u r d e t a il e d w o r k F e l d m a n , 1 9 81 b ; S a b b a h ,
1 98 1). O n e e x a m p l e o f a u n it e m p l o y i n g b o t h p a n d q n o n - t r i v i a l l y is t h e
f o l lo w i n g c r u d e n e u r o n m o d e l . T h i s m o d e l is c o n c e r n e d w i th s a t u r a t i o n a n d
a s s u m e s t h a t t h e o u t p u t s t r e n g t h , v , is s o m e t h i n g l ik e a v e r a g e f i r i n g f re -
q u e n c y . I t is n o t a m o d e l o f in d i v i d u a l a c t io n p o t e n t i a l s a n d r e f r a c t o r y
p e r i o d s .
W e s u p p o s e t h e d is t in c t s t a t e s o f t h e u n i t q e { n o r m a l , r e c o v e r } . I n
norm l
s t a t e t h e u n i t b e h a v e s l i k e a p - u n i t , b u t w h i l e i t i s
recovering
i t ig-
n o r e s i n p u t s . T h e f o l lo w i n g t a b l e c a p t u r e s a l m o s t a l l o f t hi s b e h a v i o r .
i n c o m p l e t e )
n o r m a l
r e c o v e r
1 < p < 9 p > 9 O u t p u t V a l u e
p - - p + E i p - - - p / v - -c t p - / 9
r e c o v e r
n o r m a l < i m p o s s i b le > v - - O
H e r e w e h a v e t he c h a n g e f r o m o n e s t a te t o th e o t h e r d e p e n d i n g o n t h e
v a l u e o f t h e p o t e n t i a l , p , r a t h e r t h a n o n s p e c i f ic i n p u t s . T h e r e c o v e r i n g s t a t e
is a l s o c h a r a c t e r i z e d b y t h e p o t e n t i a l b e i n g s e t n e g a t i v e . T h e u n s p e c i f i e d
is su e is w h a t d e t e r m i n e s t h e d u r a t i o n o f th e r e c o v e r i n g s t a t e - - t h e r e a r e
8/11/2019 Connectionist models and their properties.pdf
13/50
CONNECTIONIST MODELS A N D THEIR PROPERTIES 2 7
s e v e r a l p o s s i b i l it i e s . O n e is a n e x p l i c i t d i s h a b i t u a t i o n s i g n a l l i k e t h o s e i n
K a n d e l s e x p e r i m e n t s ( K a n d e l , 1 97 6) . A n o t h e r w o u l d b e t o h a v e th e u n i t
s u m i n p u t s i n t h e r e c o v e r i n g s ta t e a s w e ll . T h e r e a d e r m i g h t w a n t t o c o n -
s i d e r h o w t o a d d t h i s t o t h e t a b l e .
T h e t h i r d p o s s i b il i ty , w h i c h w e w i ll u s e f r e q u e n t l y , is t o a s s u m e t h a t
t h e p o t e n t i a l , p , d e c a y s t o w a r d z e r o ( f r o m b o t h d i r e c t i o n ~ ) u n l e s s e x p l i c i t l y
c h a n g e d . T h i s i m p l i c it d e c a y p - - p 0 e -kf c a n b e m o d e l e d b y s e l f i n h i b i t i o n ; t h e
d e c a y c o n s t a n t , k , d e t e r m i n e s t h e l e n g th o f t h e r e c o v e r y p e r i o d .
T h e g e n e r al d e f i n i t i o n o f o u r a b s t r a c t n e u r a l c o m p u t i n g u n i t is j u s t a
f o r m a l i z a t i o n o f th e i d e as p r e s e n t e d a b o v e . T o th e p r e v i o u s n o t i o n s o f p , v ,
a n d i w e f o r m a l l y a d d
{ q } - - a s e t o f
discrete s tates
< 10
a n d f u n c t i o n s f r o m o l d t o n e w v a l u e s o f t h es e
p - - f ( i , p , q )
q - - g ( i , p , q )
v - - h ( i , p , q )
w h i ch w e a s su m e , f o r n o w , t o c o m p u t e c o n t i n u o u s l y . T h e f o r m o f t h e f , g ,
a n d h f u n c t i o n s w il l v a r y , b u t w i ll g e n e r a l l y b e r e s t r i c t e d t o c o n d i t i o n a l s a n d
s i m p l e f u n c t i o n s . T h e r e a r e b o t h b i o l o g i c a l a n d c o m p u t a t i o n a l r e a s o n s f o r
a l l o w i n g u n i ts t o r e s p o n d ( f o r e x a m p l e ) l o g a r i t h m i c a l l y t o t h e i r i n p u t s a n d
w e h a v e a l re a d y s ee n i m p o r t a n t u s e s o f th e m a x i m u m f u n c t i o n .
T h e o n l y o t h e r n o t i o n t h a t w e w ill n e e d is m o d i f i e r s a s s o c ia t e d w i t h
t h e i n p u t s o f a u n i t . W e e l a b o r a t e t h e i n p u t v e c t o r i i n t e r m s o f r e c e i v e d
v a l u e s , w e i g h t s , a n d m o d i f i e r s :
V j , b = r~-w~..mj j = 1 . . . . n
w h e r e r j i s t h e value r ec e iv e d f r o m a p re d e c e s so r [ r = 0 . . . 9 ] ; w j is a
c h a n g e a b l e
weight
u n s ig n e d [0 _< w~_< 1] ( a c c u r a c y o f s e v e ra l d ig i t s ) ; a n d m j
i s a s y n a p t o - s y n a p t i c m o d i f i e r wh ic h i s e i t h e r 0 o r 1 .
T h e w e i gh t s a r e th e o n l y t h i n g i n th e s y s t e m w h i c h c a n c h a n g e w i th e x -
p e r i en c e . T h e y a r e u n s i g n e d b e c a u s e w e d o n o t w a n t a c o n n e c t i o n to c h a n g e
f r o m e x c i t a t o r y to i n h i b i t o r y . T h e m o d i f i e r o r g a te s im p l i fi e s m a n y o f o u r
d e t a i l e d m o d e l s . L e a r n i n g a n d c h a n g e w i ll n o t b e t r e a t e d t e c h n i c a l l y i n t h is
p a p e r , b u t t h e d e f i n i t i o n s a r e i n c l u d e d i n t h e A p p e n d i x f o r c o m p l e t e n e s s
( F e l d m a n , 1 9 8 1 b ) .
W e c o n c l u d e t h is s e c t io n w i t h s o m e p r e l i m i n a r y e x a m p l e s o f n e t w o r k s
o f o u r u n i t s , i l l u s tr a t i n g t h e k e y id e a o f m u t u a l ( l a te r a l ) i n h i b i t i o n ( F i g . 5 ).
M u t u a l i n h i b i ti o n i s w i d e s p r e a d i n n a t u r e a n d h a s b e e n o n e o f t h e b as ic
c o m p u t a t i o n a l s c h e m e s u s e d i n m o d e l i n g . W e w i l l p r e s e n t t w o e x a m p l e s o f
8/11/2019 Connectionist models and their properties.pdf
14/50
2 8 FELDMAN AND BALLARD
h o w i t w o r k s t o h e l p a id i n i n t u i t i o n a s w e l l a s t o i l lu s t r a t e t h e n o t a t i o n . T h e
b a s ic s i t u a t io n is sy m m e t r i c c o n f i g u r a t i o n s o f p - u n i t s w h i c h m u t u a l l y i n-
h i b i t o n e a n o t h e r . T i m e is b r o k e n i n t o d i s c r e t e i n t e r v a ls f o r th e s e e x a m p l e s .
T h e e x a m p l e s a r e t o o s i m p l e t o b e r e a l i s t i c , b u t d o c o n t a i n i d e a s w h i c h w e
w i l l e m p l o y r e p e a t e d l y .
T w o P U n i t s S y m m e t r i c a l l y C o n n e c t e d
S u pp o se w, = 1 , w2 = - .5
p( t + 1) = p( t ) + r , - ( .5)r2
v = r o u n d ( p ) [ 0 . . . 9 ]
rs = r e c e iv e d
R e f e r r i n g t o F i g u r e 5 a , s u p p o s e t h e i n i ti a l i n p u t t o t h e u n i t A . 1 is 6 , t h e n 2
p e r t i m e s t e p , a n d t h e i n it ia l i n p u t t o B . I is 5 , t h e n 2 p e r t i m e s t e p . A t e a c h
t i m e s t e p , e a c h u n i t c h a n g e s i t s p o t e n t i a l b y a d d i n g t h e e x t e r n a l v a l u e ( r , )
a n d s u b s t r a c t i n g h a l f t h e o u t p u t v a l u e o f i ts r iv a l . T h i s s y s t e m w i ll s t a b il i ze
t o t h e s id e o f t h e l a rg e r o f t w o i n s t a n t a n e o u s i n p u t s.
T w o S y m m e t r i c C o a l i t i o n s o f 2 U n i t s
w l = l
W ~ = . 5
W3 = .5
p( t + 1) = p( t ) + r , + .5( r , - r3)
v = r o u n d ( p )
A , C s t a r t a t 6 , B , D a t 5 ;
A , B , C , D h a v e n o e x t e r n a l i n p u t f o r t > 1
T h e c o n n e c t i o n s f o r t hi s s y st e m a r e s h o w n in F i g u r e 5 b . T h i s s y s te m
c o n v e r g e s f a st e r t h a n t h e p r e v i o u s e x a m p l e . T h e i d e a h e r e is t h a t u n i ts A a n d
C f o r m a c o a l i t i o n w i th m u t u a l ly r e i n fo r c i n g c o n n e c t i o n s . T h e c o m p e t i n g
u n i t s a r e A v s . B a n d C v s . D . T h e l a s t e x a m p l e is t h e s m a l l e s t n e t w o r k d e -
p i ct in g w h a t w e b e l ie v e t o b e t h e b a s ic m o d e o f o p e r a t i o n i n c o n n e c t i o n i s t
s y s t e m s . T h e f a s t e r c o n v e r g e n c e is n o t a n a r t i f a c t ; t h e
positive feedb ck
a m o n g m e m b e r s o f a c o a l i t io n w i ll g e n e r a l l y l e ad t o f a s t e r c o n v e r g e n c e t h a n
in s e p a r a t e c o m p e t i t i o n s . I t is t h e a m o u n t o f p o s i ti v e f e e d b a c k r a t h e r t h a n
j u s t t h e s iz e o f t h e c o a l i t io n t h a t d e t e r m i n e s t h e r a t e o f c o n v e r g e n c e ( F el d -
m a n & B a l l a r d , 1 98 2). I n t e r m s o f F i g u r e 1, t h is c o u l d r e p r e s e n t t h e b e h a v i o r
o f t h e ri va l l et te r s A a n d T in c o n j u n c t i o n w i t h t h e ri v al w o r d s A B L E a n d
T R A P , in th e a b s e n c e o f o t h e r a c t i v e n o d e s .
C o m p e t i n g c o a l i ti o n s o f u n i t s w i ll b e th e o r g a n i z i n g p r i n c ip l e b e h i n d
m o s t o f o u r m o d e l s . C o n s i d e r t h e tw o a l t e r n a ti v e r e ad i n g s o f th e N e c k e r
8/11/2019 Connectionist models and their properties.pdf
15/50
. Q
Q Q .
u u
0 0
o .
~ ~ o
0
z
u
E
E
E
0
C
o
M.
2 9
8/11/2019 Connectionist models and their properties.pdf
16/50
2 2 F E LD MA N A N D B A LL A RD
c u b e s h o w n i n F i g u r e 6. A t e a c h l ev e l o f v i s u a l p r o c e s s i n g , t h e r e a r e m u t u a l l y
c o n t r a d i c t o r y u n it s r e p r e s e n t i n g a l t e r n a t i v e p o s s ib i li ti e s. T h e d a s h e d l in e s
d e n o t e th e b o u n d a r i e s o f c o a li t io n s w h i c h e m b o d y t h e a l t e rn a t i v e in t e r p r e -
t a t io n s o f t h e im a g e . A n u m b e r o f i n te r e s ti n g p h e n o m e n a ( e .g . , p r i m i n g ,
p e r c e p t u a l r i v a l r y , f i ll in g , s u b j e c t i v e c o n t o u r ) f i n d n a t u r a l e x p r e s s i o n in th i s
f o r m a l i s m . W e a re e n g a g e d i n an o n g o i n g e f f o r t (B a l l a rd , 1 98 1; S a b b a h ,
1 9 8 1 ) t o m o d e l a s m u c h o f v i s u a l p r o c e s s i n g a s p o s s i b l e w i t h i n t h e c o n n e c -
t i o n is t f r a m e w o r k . T h e n e x t s e c t io n d e s c r ib e s in s o m e d e t a i l a v a r i e t y o f
s i m p l e n e t w o r k s w h i c h w e h a v e f o u n d t o b e u s e f u l i n t h i s e f f o r t .
3 . N E T W O R K S O F U N I T S
T h e m a i n r e s tr i c ti o n i m p o s e d b y t h e c o n n e c t i o n i s t p a r a d i g m is t h a t n o s y m -
b o l i c i n f o r m a t i o n is p a s s e d f r o m u n i t to u n i t . T h i s r e s t r ic t i o n m a k e s i t d i f f i -
c u lt t o e m p l o y s t a n d a r d c o m p u t a t i o n a l d e v i c es l ik e p a r a m e t e r i z e d f u n c t i o n s .
I n t h is s e c t io n , w e p r e s e n t c o n n e c t i o n i s t s o l u t i o n s t o a v a r i e t y o f c o m p u t a -
t i o n a l p r o b l e m s . T h e s e c t i o n s a d d r e s s t w o p r i n c i p a l i s s u e s . O n e i s : C a n t h e
n e t w o r k s b e c o n n e c t e d u p i n a w a y t h a t i s s u f f i c i e n t t o r e p r e s e n t t h e p r o b -
l e m a t h a n d ? T h e o t h e r i s : G i v e n t h e s e c o n n e c t i o n s , h o w c a n t h e n e t w o r k s
e x h i b i t a p p r o p r i a t e d y n a m i c b e h a v i o r , s u c h a s m a k i n g a d e c i s i o n a t a n
a p p r o p r i a t e t i m e ?
U s i n g a U n i t t o e p r e s e n t a V a l u e
O n e k e y to m a n y o f o u r c o n s t r u c t i o n s i s t h e d e d i c a t i o n o f a s e p a r a t e u n i t t o
e a c h v a lu e o f e a c h p a r a m e t e r o f i n te r e st , w h i c h w e te r m t h e u n i t / v a l u e p r i n -
c ip le . W e w ill s h o w h o w t o c o m p u t e u s in g u n i t / v a l u e n e t w o r k s a n d p r e s e n t
a r g u m e n t s t h a t t h e n u m b e r o f u n i ts r e q u i r e d is n o t u n r e a s o n a b l e . I n t h is
r e p r e s e n ta t i o n t h e o u t p u t o f a u n i t m a y b e t h o u g h t o f a s a c o n f i d e n c e m e a -
s u re . S u p p o s e a n e t w o r k o f d e p t h u n i t s e n c o d e s th e d i s t a n c e o f s o m e o b j e c t
f r o m t h e r e t i n a . T h e n i f t h e u n i t r e p r e s e n t i n g d e p t h = 2 s a t u r a t e s , t h e n e t -
w o r k is e x p r e s s i n g c o n f i d e n c e th a t t h e d i s t a n c e is t w o u n i t s . S i m i l a r ly , t h e
G - h i d d e n n o d e in F i g u re 6 e x p r es s e s c o n f i d e n c e in i ts a s s e r ti o n . T h e r e is
m u c h n e u r o p h y s i o l o g i c a l e v i d e n c e t o su g g e st u n i t / v a l u e o r g a n i z a t i o n s i n
l es s a b s t r a c t c o r t i c a l m a p s . E x a m p l e s a r e e d g e s e n s i ti v e u n i t s ( H u b e l &
W i e s e l , 1 9 7 9 ) a n d p e r c e p t u a l c o l o r u n i t s ( Z e k i , 1 9 8 0 ) , w h i c h a r e r e l a t i v e l y
i n se n s it iv e t o i l l u m i n a t i o n s p e c t r a . E x p e r i m e n t s w i th c o r t i c a l m o t o r c o n t r o l
i n t h e m o n k e y a n d c a t ( W u r t z & A l b a n o , 1 98 0) s u g ge s t a u n i t / v a l u e o r g a n i -
z a t i o n . O u r h y p o t h e s i s is t h a t t h e u n i t / v a l u e o r g a n i z a t i o n i s w i d e s p r e a d ,
a n d is a f u n d a m e n t a l d e s i g n p ri n c i p l e .
8/11/2019 Connectionist models and their properties.pdf
17/50
CON NEC TIONIST MO DELS A N D THEIR PROPERTIES 221
H
C
\ \
\
\
J A
\
I /
F i gu r e 6 The Ne cke r Cube
A l t h o u g h m a n y p h y si ca l n e u r o n s d o s e e m t o f o l lo w th e u n i t / v a l u e
r u le a n d r e s p o n d a c c o r d i n g t o t h e r e l i ab i li ty o f a p a r t i c u l a r c o n f i g u r a t i o n ,
t h e r e a r e al so o t h e r n e u r o n s w h o s e o u t p u t r e p r e se n t s th e r a n g e o f s o m e
p a r a m e t e r , a n d a p p a r e n t l y s o m e u n i ts w h o s e f i ri n g f r e q u e n c y re f le c t s b o t h
r a n g e a n d s t re n g t h i n f o r m a t i o n S c i e nt if ic A m e r i c a n , 1 97 9). B o t h o f t h e
l a tt e r t y p e s c a n b e a c c o m m o d a t e d w i t h i n o u r d e f i n i t io n o f a u n i t , b u t w e
w i ll e m p l o y o n l y u n i t / v a l u e n e t w o r k s i n t h e r e m a i n d e r o f t h is p a p e r .
I n t h e u n i t / v a l u e r e p r e s e n ta t i o n , m u c h c o m p u t a t i o n is d o n e b y t a bl e
l o o k - u p . A s a s i m p l e e x a m p l e , l e t u s c o n s i d e r t h e m u l t i p l i c a t io n o f t w o v a r i-
a b l e s , i . e . , z = x y . I n t h e u n i t / v a l u e f o r m a l i s m t h e r e w i l l b e u n i t s f o r
e v e r y
v a l u e o f x a n d y t h a t i s i m p o r t a n t . A p p r o p r i a t e p a i rs o f t h e se w ill m a k e a
c o n j u n c t i v e c o n n e c t i o n w i t h a n o t h e r u n i t c e ll r e p r e s e n t in g a s p e c if i c v a l u e
f o r t h e p r o d u c t . F i g u r e 7 s h o w s t h i s f o r a s m a l l se t o f u n i t s r e p r e s e n t i n g
v a l u es f o r x a n d y . N o t i c e th a t t h e c o n f i d e n c e e x p r e ss e d a s o u t p u t v a l u e )
t h a t a p a r t i c u l a r p r o d u c t i s a n a n s w e r c a n b e a l i n e a r f u n c t i o n o f t h e m a x -
8/11/2019 Connectionist models and their properties.pdf
18/50
2 2 2 F E L D M A N A N D B A LL AR D
i m u m o f t h e s um s o f t h e c o n f i d e n c es o f i ts t w o i n p ut s . A m a j o r p r o b l e m
w i t h f u n c t i o n ta b l e s a n d w i t h C M i n g e n e r a l) is t h e p o t e n t i a l c o m b i n a t o r i a l
e x p l o s i o n i n t h e n u m b e r o f u n i ts r e q u i r e d f o r a c o m p u t a t i o n . A n a i v e a p -
p r o a c h w o u l d d e m a n d N 2 u n i t s to r e p re s e n t a ll p r o d u c t s o f n u m b e r s f r o m 1
t o N . T h e n e t w o r k o f F i g u r e 7 r e q u i r e s m a n y f e w e r u n i ts b e c a u s e e a c h p r o d -
u c t is r e p r e s e n t e d o n l y o n c e , a n o t h e r a d v a n t a g e o f c o n j u n c t i v e c o n n e c t i o n s .
W e c o u l d u s e e v e n f e w e r u n it s b y e x p l o i ti n g p o s i t i o n a l n o t a t i o n a n d r e p l ac -
i ng e ac h o u t p u t c o n n e c t i o n w i th a c o n j u n c t i o n o f o u t p u t s f r o m u n it s r e p re -
s e n t i n g m u l t i p l e s o f l , 10 , 1 00 , e t c . T h e q u e s t i o n o f e f f i c i e n t w a y s o f
b u i l d i n g c o n n e c t i o n n e t w o r k s i s t r e a t e d i n d e t a i l i n S e c t i o n 4 c f . a l s o H i n -
to n , 1 9 8 1 a ; 1 9 8 1 b ) .
z - - - - f x , y ) = x y
x u n i t s
.onit
z u n i t :
F i g u r e 7. M u l t i p l i c a t i o n U n i ts
odifiers and appings
T h e i d ea o f f u n c t i o n t ab l e s F ig . 7) c a n b e e x t e n d e d t h r o u g h t h e u s e o f vari
able mappings. I n o u r d e f i n i t io n o f t h e c o m p u t a t i o n a l u n i t , w e i n c l u d e d a
b i n a r y m o d i f i e r , m , a s a n o p t i o n o n e v e r y c o n n e c t i o n . A s t h e d e f i n i t i o n
s p e c if ie s , i f t h e m o d i f i e r a s s o c i a t e d w i t h a c o n n e c t i o n is z e r o , t h e v a l u e v
s e n t a lo n g t h a t c o n n e c t i o n is i g n o r e d . T h u s t h e m o d i f i e r d e n o t e s i n h i b i t i o n ,
o r b l o c k i n g . T h e r e i s c o n s i d e r a b l e e v i d e n c e i n n a t u r e f o r s y n a p s e s o n s y n a p -
se s K a n d e l , 1 97 6) a n d t h e m o d i f i e r s a d d g r e a t ly t o th e c o m p u t a t i o n a l
s im p l i c it y o f o u r n e t w o r k s . L e t u s st a r t w i t h a n i n it ia l i n f o r m a l e x a m p l e o f
t h e u se o f m o d i f i e r s a n d m a p p i n g s . S u p p o s e t h a t o n e h a s a m o d e l o f g ra s s as
g r e e n e x c e p t i n C a l i f o r n i a w h e r e i t is b r o w n g o l d e n ) , a s s h o w n i n F i g u r e 8 .
8/11/2019 Connectionist models and their properties.pdf
19/50
C ON N EC T I ON I ST M OD E LS A N D T H EIR PR OPER TIES 22 3
F i g u r e 8 G r a s s is G r e e n c o n n e c t i o n m o d i f i e d b y C a l i fo r n i a
H e r e w e c a n s e e t h a t g r a s s a n d g r e e n a re p o t e n t i a l m e m b e r s o f a c o a l i t i o n
c a n r e i n f o r c e o n e a n o t h e r ) e x c e p t w h e n t h e l i n k is b l o c k e d . T h i s u s e is s im i -
l a r t o t h e c a n c e l l a t i o n li n k o f F a h l m a n , 1 97 9) a n d g i v e s a c r u d e id e a o f h o w
c o n t e x t c a n e f f e c t p e r c e p t i o n i n o u r m o d e l s . N o t e t h a t i n F i g u r e 8 w e a r e
u si ng a s h o r t h a n d n o t a t i o n . A m o d i f i e r t o u c h i n g a d o u b l e - e n d e d a r r o w
a c t u a l ly b l o c k s t w o c o n n e c t i o n s . S o m e t i m e s w e a ls o o m i t t h e a r r o w h e a d s
w h e n c o n n e c t i o n i s d o u b l e - e n d e d . )
M a p p i n g s c a n a l so b e u s ed t o s el ec t a m o n g a n u m b e r o f p o s s i b le
v a lu e s . C o n s i d e r t h e e x a m p l e o f t h e r e l a t i o n b e t w e e n d e p t h , p h y s i ca l si ze ,
a n d r e t i n a l si ze o f a ci rc l e. F o r n o w , a s s u m e t h a t t h e c i r cl e is c e n t e r e d o n
a n d o r t h o g o n a l t o t h e l i n e o f s i g h t, t h a t t h e f o c u s is f i x e d , e t c . ) T h e n t h e r e i s
a f i x e d r e l a t i o n b e t w e e n t h e s i ze o f r e t i n a l i m a g e a n d t h e s iz e o f t h e p h y s i c a l
c i r c l e f o r a n y g i v e n d e p t h . T h a t i s , e a c h d e p t h s p e c i f i e s a m pping f r o m
r e t in a l t o p h y s i c a l s iz e s e e F i g . 9) . H e r e w e s u p p o s e t h e s c al es f o r d e p t h a n d
t h e t w o s iz es a r e c h o s e n s o t h a t u n i t d e p t h m e a n s t h e s a m e n u m e r i c a l s iz e . I f
w e k n e w t h e d e p t h o f t h e o b j e c t b y t o u c h , c o n t e x t , o r m a g i c ) w e w o u l d
k n o w i t s p h y s i c a l s i z e . T h e n e t w o r k a b o v e a l l o w s r e t i n a l s i z e 2 t o r e i n f o r c e
p h y s i c a l s iz e 2 w h e n d e p t h = 1 b u t i n h i b i t s t h i s c o n n e c t i o n f o r a ll o t h e r
d e p t h s . S i m i l a r l y , a t d e p t h 3 , w e s h o u l d i n t e r p r e t r e t i n a l si ze 2 as p h y s i c a l
s iz e 8 , a n d i n h i b i t o t h e r i n t e r p r e t a t i o n s . S e v e r a l r e m a r k s a r e in o r d e r . F i r s t ,
n o t i c e t h a t t h is n e t w o r k i m p l e m e n t s a f u n c t i o n p h y s = f r e t , d e p ) th a t m a p s
f r o m r e t in a l s iz e a n d d e p t h t o p h y s i c a l s iz e, p r o v i d i n g a n e x a m p l e o f h o w t o
r e p l a c e f u n c t i o n s w i t h p a r a m e t e r s b y m a p p i n g s . F o r t h e s i m p l e c a s e o f
l o o k i n g a t o n e o b j e c t p e r p e n d i c u l a r t o t h e l i n e o f s i g h t , t h e r e w i l l b e o n e
c o n s i s t e n t c o a l i t i o n o f u n i t s w h i c h w i l l b e s t a b l e . T h e w o r k d o e s s o m e t h i n g
m o r e , a n d t h i s i s c r u c i a l t o o u r e n t e r p r i s e ; t h e n e t w o r k c a n r e p r e s e n t t h e
c o n s i s t e n c y r e l a t i o n R a m o n g t h e t h r e e q u a n t i t i e s : d e p t h , r e t i n a l s i z e , a n d
8/11/2019 Connectionist models and their properties.pdf
20/50
4 FELDMAN AND BALLARD
p h y s i c a l si ze . I t e m b o d i e s n o t o n l y t h e f u n c t i o n f , b u t i t s t w o i n v e r se f u n c -
t i o n s a s w e ll (d e p = f ~ ( r e t , p h y s ) , a n d r e t = f 2 ( p h y s , d e p ) ) . ( T h e n e t w o r k a s
s h o w n d o e s n o t i n c l u d e t h e li n k s f o r f , a n d f~ , b u t t h e s e a r e s i m i l a r t o t h o s e
f o r f . ) M o s t o f S e c t i o n 5 i s d e v o t e d t o l a y i n g o u t n e t w o r k s t h a t e m b o d y
t h e o r i e s o f p a r t i c u l a r v i s u a l c o n s i s t e n c y r e l a t i o n s .
T h e i d e a o f m o d i f i e r s is , in a se n se , c o m p l e m e n t a r y t o t h a t o f c o n -
j u n c t i v e c o n n e c t i o n s . F o r e x a m p l e , t h e n e t w o r k o f F i g u r e 9 c o u l d b e t r an s -
f o r m e d i n t o t h e f o l l o w i n g n e t w o r k ( F ig . 10 ). I n th i s n e t w o r k t h e v a r ia b l e s
f o r p h y s i c a l s iz e , d e p t h , a n d r e t i n a l s iz e a r e a l l g i v e n e q u a l w e i g h t . F o r e x -
a m p l e , p h y s i c a l si ze = 4 a n d d e p t h = 1 m a k e a conjunctive connection w i t h
r e ti n a l s iz e = 4 . E a c h o f t h e v a l u e u n i ts in a c o m p e t i n g r o w c o u l d b e c o n -
n e c t e d t o a ll o f i ts c o m p e t i t o r s b y i n h i b i t o r y l i n ks a n d t h is w o u l d t e n d t o
m a k e t h e n e t w o r k a c t i v a t e o n l y o n e v a l u e i n e a c h c a t e g o r y . T h e g e n e r a l
i ss u e o f r i v a l r y a n d c o a l i t i o n s w ill b e d i sc u s s e d i n t h e n e x t t w o s u b - s e c t i o n s .
W h e n s h o u l d a r e l a t i o n b e i m p l e m e n t e d w i t h m o d i f i e r s a n d w h e n
s h o u l d i t b e im p l e m e n t e d w i th c o n j u n c t i v e c o n n e c t i o n s ? A s i m p l e, n o n -
r i g o r o u s a n s w e r t o t hi s q u e s t i o n c a n b e o b t a i n e d b y e x a m i n i n g t h e s iz e o f
t w o s e t s o f u n i t s: (1 ) t h e n u m b e r o f u n it s t h a t w o u l d h a v e to b e in h i b i t e d b y
m o d i f i e r s ; a n d ( 2 ) t h e n u m b e r o f u n i t s t h a t w o u l d h a v e t o b e r e i n f o r c e d
w i t h c o n j u n c t i v e c o n n e c t i o n s . I f ( 1 ) i s l a r g e r t h a n ( 2 ) , t h e n o n e s h o u l d
c h o o s e m o d i f i e r s ; o t h e rw i s e c h o o s e c o n j u n c t i v e c o n n e c t i o n s . S o m e t i m e s
t h e c h o ic e is o b v i o u s : t o i m p l e m e n t th e b r o w n C a l i f o r n i a n g r a ss e x a m p l e o f
F i g u r e 8 w i th c o n j u n c t i v e c o n n e c t i o n s , o n e w o u l d h a v e t o r e i n f o r c e al l u n it s
r e p r e s e n t i n g p l a c e s t h a t h a d g r e e n g r a s s C l e a r l y i n t h i s c a s e i t is e a s i e r t o
h a n d l e t h e e x c e p t io n w i t h m o d i f i e r s . O n t h e o t h e r h a n d , t h e d e p t h r e l a ti o n
R ( p h y , d e p , r e t ) i s m o r e c h e a p l y i m p l e m e n t e d w i t h c o n j u n c t i v e c o n n e c t i o n s .
S i n c e o u r m o d i f i e r s a r e s t r i c t l y b i n a r y , c o n j u n c t i v e c o n n e c t i o n s h a v e t h e
a d d i ti o n a l a d v a n t a g e o f c o n t i n u o u s m o d u l a t i o n .
T o s ee h o w t h e c o n j u n c t i v e c o n n e c t i o n s t r a te g y w o r k s i n g e n e r a l, s u p -
p o s e a c o n s t r a i n t r e l a t i o n t o b e s a t is f i e d in v o l v e s a v a r i a b l e x , e . g . , f ( x , y , z , w )
= 0 . F o r a p a r t i c u l a r v a l u e o f x , t h e r e w i ll b e tr i p le s o f v a l u e s o f y , z , a n d w
t h a t s a t i s fy th e r e l a t i o n f . E a c h o f th e s e tr i p le s s h o u l d m a k e a c o n j u n c t i v e
c o n n e c t i o n w i t h t h e u n i t r e p r e s e n t i n g t h e x - v a l u e . T h e r e c o u l d a l s o b e 3 - i n -
p u t c o n j u n c t i o n s a t ea c h v a l u e o f y , z , w . E a c h o f t he s e f o u r d i f f e r e n t k in d s
o f c o n j u n c t i v e c o n n e c t i o n s c o r r e s p o n d s t o a n i n t e r p r e t a t i o n o f t h e relation
f ( x , y , z ,w ) = 0 a s a function i . e . , x = f , ( y , z , w ) , y = f2 (x , z , w) , z = f3 (x , y , w ) , o r
w = f , ( x ,y , z ) . O f c o u r s e , t h e s e f u n c t i o n s n e e d n o t b e s i n g le - v a l u e d . T h i s n e t -
w o r k c o n n e c t i o n p a t t e r n c o u l d b e e x t e n d e d t o m o r e t h a n f o u r v a ri a b le s , b u t
h i g h n u m b e r s o f v a r i a b l e s w o u l d t e n d t o i n c r e a s e i ts s e n s it iv i t y t o n o i s y in -
p u t s . H i n t o n h a s s u g g e s te d a s p e c ia l n o t a t i o n f o r th e s i t u a t io n w h e r e a n e t-
w o r k e x a c t l y c a p t u r e s a c o n s i s t e n c y r e la t i o n . T h e m u t u a l l y c o n s i s t e n t v a l u e s
a r e a ll s h o w n t o b e c e n t r a l l y l i n k e d ( F i g . 1 1). T h i s n o t a t i o n p r o v i d e s a n e le -
8/11/2019 Connectionist models and their properties.pdf
21/50
a
u_ e ~
e -
a . .
1 3
0
c :
0
z
e
Q .
a~
14.
m
5
8/11/2019 Connectionist models and their properties.pdf
22/50
0
u ~
~ . ~ . ~
i
~ ~
r ~
o
r
t
o
u
o
>
c
0
U
. ~
0
0
Z
n
2 2 6
8/11/2019 Connectionist models and their properties.pdf
23/50
CONNECTIONIST MODELS A N D THEIR PROPERTIES
F ig u r e 1 N o t a t i o n f o r c o n s i s te n c y r e l a t i o n s
7
g a n t w a y o f p r e s e n ti n g th e i n t e r a c t i o n s a m o n g n e t w o r k s , b u t m u s t b e u s e d
w i th c a r e . W r i t i n g d o w n a t r ia n g l e d i a g r a m d o e s n o t i n su r e t h a t t h e u n d e r -
l y in g m a p p i n g s c a n b e m a d e c o n s i s t e n t o r c o m p u t a t i o n a l l y w e l l - b e h a v e d .
W i n n er T a k e A l l N e t w o r k s a n d R e g u l a te d N e t w o r k s
A v e r y g e n e r a l p r o b l e m t h a t a ri se s in a n y d i s t r i b u t e d c o m p u t i n g s i t u a t io n is
h o w t o g e t t h e e n t i re s y s te m t o m a k e a d e c is i o n ( o r p e r f o r m a c o h e r e n t ac -
t i o n , e t c .) . B i o lo g i c a ll y n e c e s s a r y e x a m p l e s o f t h is b e h a v i o r a b o u n d ; r a n g i n g
f r o m t u r n i n g l e ft o r ri g h t , t h r o u g h f i g h t -o r - f li g h t r e s p o n s e s, t o i n t e r p r e t a -
t i o n s o f a m b i g u o u s w o r d s a n d i m a g e s . D e c i s i o n - m a k i n g is a p a r t i c u l a r l y i m -
p o r t a n t i ss ue f o r t h e c u r r e n t m o d e l b e c a u s e o f it s r e s t ri c t io n s o n i n f o r m a t i o n
f l o w a n d b e c a u s e o f t h e a l m o s t l i n ea r n a t u r e o f t h e p - u n i t s u s e d i n m a n y o f
o u r s p e c i fi c e x a m p l e s . D e c i s i o n - m a k i n g i n t r o d u c e s t h e n o t i o n s o f
stable
states a n d convergence o f n e t w o r k s .
O n e w a y t o d e a l w i t h t h e is su e o f c o h e r e n t d e c is i o ns i n a c o n n e c t i o n i s t
f r a m e w o r k is t o i n t r o d u c e winner-take-all ( W T A ) n e t w o r k s , w h i c h h a v e t h e
p r o p e r t y t h a t o n l y t h e u n i t w i th t h e h i g h es t p o t e n t i a l ( a m o n g a se t o f c o n -
t e n d e r s ) w i ll h a v e o u t p u t a b o v e z e r o a f t e r s o m e s e t t in g t i m e ( F ig . 1 2) . T h e r e
a r e a n u m b e r o f w a ys to c o n s t r u c t W T A n e t w o r k s f r o m t h e u n i ts d e s c ri b e d
a b o v e . F o r o u r p u r p o s e s i t is e n o u g h t o c o n s i d e r o n e e x a m p l e o f a W T A n e t-
w o r k w h i c h w i ll o p e r a t e i n o n e t i m e s t e p f o r a s et o f c o n t e n d e r s e a c h o f
w h o m c a n r e a d t h e p o t e n t i a l o f a l l o f t h e o t h e r s . E a c h u n i t i n t h e n e t w o r k
c o m p u t e s i t s n e w p o t e n t i a l a c c o r d i n g t o t h e r u l e :
p- -i f p >max ij, .1) then else O.
8/11/2019 Connectionist models and their properties.pdf
24/50
8 F E L D M A N A N D B A L LA R D
T h a t i s , e a c h u n i t s e t s i t s e l f t o z e r o i f i t k n o w s o f a h i g h e r i n p u t . T h i s is f a s t
a n d s i m p l e , b u t p r o b a b l y a li tt le t o o c o m p l e x t o b e p la u s i b l e as th e b e h a v i o r
o f a s in g le n e u r o n . T h e r e i s a s t a n d a r d t r ic k ( a p p a r e n t l y w i d e ly u s e d b y
n a t u r e ) t o c o n v e r t t hi s in t o a m o r e p l a u s i b l e s c h e m e . R e p l a c e e a c h u n i t
a b o v e w i th t w o u n it s; o n e c o m p u t e s t h e m a x i m u m o f th e c o m p e t i t o r s in -
p u t s a n d i n h ib i ts t h e o t h e r . T h e c i r cu i t a b o v e c a n b e s tr e n g t h e n e d b y a d d i n g
a re v e r s e i n h i b i t o r y l in k , o r o n e c o u l d u s e a m o d i f i e r o n t h e o u t p u t , e t c . O b -
v i o u s ly o n e c o u l d h a v e a W T A l a y e r t h a t g o t i n p u ts f r o m s o m e s et o f c o m -
p e t i to r s a n d s e tt le d t o a w i n n e r w h e n t r i g g e r e d to d o s o b y s o m e d o w n s t r e a m
n e t w o r k . T h i s is a n e x a c t a n a l o g y o f s t r o b i n g a n o u t p u t b u f f e r in a c o n v e n -
t i o n a l c o m p u t e r .
O n e p r o b l e m w i t h p r e v i o u s n e u r a l m o d e l i n g a t t e m p t s i s t h a t t h e ci r-
c u it s p r o p o s e d w e r e o f t e n u n n a t u r a l l y d e l ic a te ( u n s t a b le ) . S m a l l c h a n g e s i n
p a r a m e t e r v a l u e s w o u l d c a u s e t h e n e t w o r k s t o o s c i l la t e o r c o n v e r g e to i n c o r -
r e c t a n s w e r s . W e w ill h a v e t o b e c a r e f u l n o t t o f a ll i n t o t h i s tr a p , b u t w o u l d
l ik e t o a v o i d d e t a i le d a n a l y s i s o f e a c h p a r t i c u l a r m o d e l f o r d e l i c a c y in t h is
p a p e r . W h a t a p p e a r s t o b e r e q u ir e d a r e s o m e b u il d in g b l o c k s a n d c o m b i n a -
t io n r u l es th a t p r e s e r v e th e d e s ir e d p r o p e r t i e s . F o r e x a m p l e , t h e W T A s u b -
n e t w o r k s o f th e l as t e x a m p l e w i ll n o t o s c i l la t e in t h e a b s e n c e o f o s c i ll a t in g
i n p u ts . T h i s is a l so t ru e o f a n y s y m m e t r i c m u t u a l l y i n h i b i t o r y s u b n e t w o r k .
T h i s is i n tu i ti v e ly c l e a r a n d c o u l d b e p r o v e n r i g o r o u s l y u n d e r a v a r i e t y o f
a s s u m p t i o n s ( cf . G r o s s b e r g , 1 98 0). I f e v e r y u n i t r ec e iv e s i n h i b i ti o n p r o p o r -
t i o n a l t o t h e a c t i v i t y ( p o t e n t i a l ) o f e a c h o f it s r i v a ls , t h e i n s t a n t a n e o u s
l e a d e r w i ll r e c e i v e le s s i n h i b i t i o n a n d t h u s n o t l o s e i ts l e a d u n l e s s t h e i n p u t s
c h a n g e s i g n i f i c a n t l y .
A n o t h e r u s e fu l p ri n c ip l e is t h e e m p l o y m e n t o f l o w e r - b o u n d a n d u p p e r -
b o u n d c e ll s t o k e e p t h e t o t a l a c t iv i t y o f a n e t w o r k w i t h i n b o u n d s ( F ig . 13).
S u p p o s e t h a t w e a d d t w o e x t r a u n i ts , L B a n d U B , t o a n e t w o r k w h i c h h a s
c o o r d i n a t e d o u t p u t . T h e L B c e ll c o m p a r e s t h e t o t a l ( s u m ) a c t i v i ty o f th e
u n i ts o f th e n e t w o r k w i t h a l o w e r b o u n d a n d s e n d s p o s i t i v e a c t i v a t i o n u n i -
f o r m l y t o a l l m e m b e r s i f t h e s u m is t o o l o w . T h e U B c el l i n h i b i ts a l l u n i t s
e q u a l l y i f t h e s u m o f a c t i v i t y is t o o h i g h . N o t i c e t h a t L B a n d U B c a n b e
p a r a m e t e r s s e t f r o m o u t si d e t h e n e t w o r k . U n d e r a w id e r a n g e o f co n d i t io n s
( b u t n o t al l ) , t h e L B - U B a u g m e n t e d n e t w o r k c a n b e d e s i g n e d to p r e s e r v e
o r d e r r e l a t i o n s h i p s a m o n g t h e o u t p u t s v j o f th e o r ig i n a l n e t w o r k w h i le k e e p -
in g t h e s u m b e t w e e n L B a n d U B .
W e w ill o f te n a s s u m e t h a t L B - U B p a i r s a r e u se d t o k e e p th e s u m o f
o u t p u t s f r o m a n e t w o r k w i th in a g iv e n r a n g e . T h i s s a m e m e c h a n i s m a ls o
g o e s f a r t o w a r d s e l i m i n a t in g t h e tw i n p e ri ls o f u n i f o r m s a t u r a t i o n a n d
u n i f o r m s il en c e w h i c h c a n e a s il y a r is e in m u t u a l i n h i b i t io n n e t w o r k s . T h u s
w e will o f t e n b e a b le t o r e a s o n a b o u t t he c o m p u t a t i o n o f a n e t w o r k a s s u m -
i n g t h a t i t s t a y s a c t i v e a n d b o u n d e d .
8/11/2019 Connectionist models and their properties.pdf
25/50
a
-i
a
t-
C~
,.C
0
c ~
L~
C~
_ o
r--
L
O
0~
C
O ~
Z aJ
c ~
-j
0
LL
2 ~ 9
8/11/2019 Connectionist models and their properties.pdf
26/50
2 3 F E L D M A N A N D B A LL AR D
S t a b l e o a l i t i o n s
F o r a m a s s i v e l y p a r a l l e l s y s t e m to a c t u a l l y m a k e a d e c i s io n ( o r d o s o m e -
t h i n g ), t h e r e w ill h a v e t o b e s ta t e s in w h i c h s o m e a c t i v i t y s t r o n g l y d o m i n a t e s .
S u c h s t a bl e , c o n n e c t e d , h i g h c o n f i d e n c e u n i ts a r e t e r m e d s t a b l e c o a l i t i o n s .
A s t a b l e c o a l i ti o n i s o u r a r c h i t e c t u r a l ly - b i a s e d t e r m f o r t h e p s y c h o l o g i c a l
n o t i o n s o f p e r c e p t , a c t io n , e t c . W e h a v e s h o w n s o m e s i m p l e i n s ta n c e s o f
s t a b le c o a l i t i o n s , i n F i g u r e 5 b a n d t h e W T A n e t w o r k . I n t h e d e p t h n e t w o r k s
o f F i g u r e s 9 a n d 10, a s t a b l e c o a l i t i o n w o u l d b e t h r e e u n i t s r e p r e s e n t i n g c o n -
s i s te n t v a l u e s o f r e t in a l s i z e , d e p t h , a n d p h y s i c a l s iz e . B u t t h e g e n e r a l i d e a is
t h a t a v e r y l a r g e c o m p l e x s u b s y s t e m m u s t s t a b i l iz e , e . g . , t o a f i x e d i n t e r p r e -
t a t i o n o f v is u a l i n p u t , a s i n F i g u r e I . T h e w a y w e b e li e v e t h i s t o h a p p e n is
t h r o u g h m u t u a l l y r e i n f o r c i n g c o a l i ti o n s w h i c h d o m i n a t e a ll r iv a l a c t iv i t y
w h e n t h e d e c i s io n is r e q u i r e d . T h e s i m p l e s t c a s e o f t h is is F i g u r e 5 b , w h e r e
t h e t w o u n it s A a n d B f o r m a c o a l i t i o n w h i c h s u p p r e s s e s C a n d D . F o r m a l l y ,
a c o a l it io n w i ll b e c a ll e d s t a b l e w h e n t h e o u t p u t o f a ll i ts m e m b e r s i s n o n
d e c re a s i n g . N o t i c e t h a t a c o a l i t i o n is n o t a p a r t i c u l a r a n a t o m i c a l s t r u c t u r e ,
b u t a n i n s t a n t a n e o u s l y m u t u a l l y r e i n f o r c i n g s et o f u n i ts , in
Recommended