Embed Size (px)
Citation preview
Soft Computing
Lecture 14
Clustering and model ART
2112005 2
Definition
bull Clustering is the process of partitioning a set of objects into subsets based on some measure of similarity (or dissimilarity) between pairs of the objects
2112005 3
Cluster Analysis
bull Goalsndash Organize information about data so that relatively
homogeneous groups (clusters) are formed and describe their unknown properties
ndash Find useful and interesting groupings of samplesndash Find representatives for homogeneous groups
bull Two components of cluster analysisndash The (dis)similarity measure between two data
samplesndash The clustering algorithm
2112005 4
Hierarchy of clusters
2112005 5
Display of hierarchy as tree
2112005 6
Minimum-Distance Clustering
2112005 7
Vehicle Example
Vehicle Top speedkmh
Colour Airresistance
WeightKg
V1 220 red 030 1300V2 230 black 032 1400V3 260 red 029 1500V4 140 gray 035 800V5 155 blue 033 950V6 130 white 040 600V7 100 black 050 3000V8 105 red 060 2500V9 110 gray 055 3500
2112005 8
Vehicle Clusters
100 150 200 250 300500
1000
1500
2000
2500
3000
3500
Top speed [kmh]
We
igh
t [k
g] Sports cars
Medium market cars
Lorries
2112005 9
Terminology
100 150 200 250 300500
1000
1500
2000
2500
3000
3500
Top speed [kmh]
We
igh
t [k
g] Sports cars
Medium market cars
Lorries
Object or data point
feature
feature space
cluster
feature
label
2112005 10
DistanceSimilarity Measures
2112005 11
DistanceSimilarity Measures (2)
2112005 12
DistanceSimilarity Measures (3)
2112005 13
DistanceSimilarity Measures (4)
2112005 14
1048698HierarchydiamsHierarchical clusteringdiamsNon-hierarchical (flat) clustering
1048698Underlying model assumptiondiamsParametric clusteringdiamsNon-parametric clustering
1048698StrictnessdiamsHard clusteringdiamsSoft clustering
Clustering Algorithms
2112005 15
Adaptive Resonance Theory (ART)
bull One of the nice features of human memory is its ability to learn many new things without necessarily forgetting things learned in the past
bull Stephen Grossberg Stability-Plasticity dilemma(1)How can a learning system remain adaptive
(plastic) in response to significant input yet remain stable in response to irrelevant input
(2)How does the system known to switch between its plastic and its stable modes
(3)How can the system retain previously learned information while continuing to learn new things
2112005 16
Basic Concept of ART
bull A key to solving the stability-plasticity dilemma is to add a feedback mechanism between the competitive layer and the input layer of a network
bull Grossberg and Carpenter ART modelbull ART is one of the unsupervised learning modelsbull This kind of model was first established in the
early 1960bull Grossberg introduced the ART in 1976bull GA Carpenter continued the research in ARTbull The original ART model is very complicated
This session only discussed the simplified Carpenter-Grossberg network
2112005 17
Basic Concept of ART (2)
bull ART 1 requires that the input vectors be binarybull ART 2 is suitable for processing analog or gray
scale patternsbull ART gets its name from the particular way in
which learning and recall interplay in the network
bull In physics resonance occurs when a small-amplitude vibration of the proper frequency causes a large-amplitude vibration in an electrical or mechanical system
2112005 18
recognition
comparison
G2
G1
input
gain control
gain control
+
+
+
+
+
+
+
-
-top-downweightsTji
bottom-upweightsBji
attentional subsystem orienting subsystem
F0
F1
F2
Basic Concept of ART (3) The ART network (Carpenter and Grossberg 1988)
2112005 19
41 Basic Concept of ART
bull Bji Forward the output from F1 to F2 for competition
bull Tji Forward the pattern of winner neuron to F1 for comparison
bull G1 To distinguish the feature of input pattern with stored patterns
bull G2 To reset the depressed neurons in F2 (ie reset losers)
bull attentional subsystem to rapidly classify the recognized patterns
bull orienting subsystem to help attentional subsystem learn new patterns
2112005 20
ART-1 Model
input vector
output vector
output layercluster
input layerinput variables
2112005 21
ART-1 Model (2)
bull Input layer input patterns or characteristic vectors Activation function f(x)=x inputs are binary values
bull Output layer representing the clustering of training patterns This is similar to SOFM except that SOFM has the neighborhood concept Initially there is only one output node The number of output nodes increases when learning proceeds When the stability is achieved the learning process stops
2112005 22
ART-1 Model (3)Algorithm
1 Set the network parameter Nout=1
2 Set the initial weighting matrices
3 Input the training vector X
4 Calculate the matching value
5 Find the max matching value in the output nodes
Niw
iw
b
t
1
1]1][[
1]1][[
0
][]][[][
count
i
b
I
iXjiwjnet
][max][ jnetjnetj
2112005 23
ART-1 Model (4) Algorithm (continue)
(6) Calculate the similarity
value
(7) Test the similarity value
If then go to step (8)
Otherwise go to step (9)
(8) Test whether there are output nodes applicable to the rule
If IcountltNout then try the second max matching value in the output nodes
||||
||||
][]][[||||
][||||
X
XwV
iXjiwXw
iXX
t
j
j
i
tt
j
i
)(vigilance V
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 2
Definition
bull Clustering is the process of partitioning a set of objects into subsets based on some measure of similarity (or dissimilarity) between pairs of the objects
2112005 3
Cluster Analysis
bull Goalsndash Organize information about data so that relatively
homogeneous groups (clusters) are formed and describe their unknown properties
ndash Find useful and interesting groupings of samplesndash Find representatives for homogeneous groups
bull Two components of cluster analysisndash The (dis)similarity measure between two data
samplesndash The clustering algorithm
2112005 4
Hierarchy of clusters
2112005 5
Display of hierarchy as tree
2112005 6
Minimum-Distance Clustering
2112005 7
Vehicle Example
Vehicle Top speedkmh
Colour Airresistance
WeightKg
V1 220 red 030 1300V2 230 black 032 1400V3 260 red 029 1500V4 140 gray 035 800V5 155 blue 033 950V6 130 white 040 600V7 100 black 050 3000V8 105 red 060 2500V9 110 gray 055 3500
2112005 8
Vehicle Clusters
100 150 200 250 300500
1000
1500
2000
2500
3000
3500
Top speed [kmh]
We
igh
t [k
g] Sports cars
Medium market cars
Lorries
2112005 9
Terminology
100 150 200 250 300500
1000
1500
2000
2500
3000
3500
Top speed [kmh]
We
igh
t [k
g] Sports cars
Medium market cars
Lorries
Object or data point
feature
feature space
cluster
feature
label
2112005 10
DistanceSimilarity Measures
2112005 11
DistanceSimilarity Measures (2)
2112005 12
DistanceSimilarity Measures (3)
2112005 13
DistanceSimilarity Measures (4)
2112005 14
1048698HierarchydiamsHierarchical clusteringdiamsNon-hierarchical (flat) clustering
1048698Underlying model assumptiondiamsParametric clusteringdiamsNon-parametric clustering
1048698StrictnessdiamsHard clusteringdiamsSoft clustering
Clustering Algorithms
2112005 15
Adaptive Resonance Theory (ART)
bull One of the nice features of human memory is its ability to learn many new things without necessarily forgetting things learned in the past
bull Stephen Grossberg Stability-Plasticity dilemma(1)How can a learning system remain adaptive
(plastic) in response to significant input yet remain stable in response to irrelevant input
(2)How does the system known to switch between its plastic and its stable modes
(3)How can the system retain previously learned information while continuing to learn new things
2112005 16
Basic Concept of ART
bull A key to solving the stability-plasticity dilemma is to add a feedback mechanism between the competitive layer and the input layer of a network
bull Grossberg and Carpenter ART modelbull ART is one of the unsupervised learning modelsbull This kind of model was first established in the
early 1960bull Grossberg introduced the ART in 1976bull GA Carpenter continued the research in ARTbull The original ART model is very complicated
This session only discussed the simplified Carpenter-Grossberg network
2112005 17
Basic Concept of ART (2)
bull ART 1 requires that the input vectors be binarybull ART 2 is suitable for processing analog or gray
scale patternsbull ART gets its name from the particular way in
which learning and recall interplay in the network
bull In physics resonance occurs when a small-amplitude vibration of the proper frequency causes a large-amplitude vibration in an electrical or mechanical system
2112005 18
recognition
comparison
G2
G1
input
gain control
gain control
+
+
+
+
+
+
+
-
-top-downweightsTji
bottom-upweightsBji
attentional subsystem orienting subsystem
F0
F1
F2
Basic Concept of ART (3) The ART network (Carpenter and Grossberg 1988)
2112005 19
41 Basic Concept of ART
bull Bji Forward the output from F1 to F2 for competition
bull Tji Forward the pattern of winner neuron to F1 for comparison
bull G1 To distinguish the feature of input pattern with stored patterns
bull G2 To reset the depressed neurons in F2 (ie reset losers)
bull attentional subsystem to rapidly classify the recognized patterns
bull orienting subsystem to help attentional subsystem learn new patterns
2112005 20
ART-1 Model
input vector
output vector
output layercluster
input layerinput variables
2112005 21
ART-1 Model (2)
bull Input layer input patterns or characteristic vectors Activation function f(x)=x inputs are binary values
bull Output layer representing the clustering of training patterns This is similar to SOFM except that SOFM has the neighborhood concept Initially there is only one output node The number of output nodes increases when learning proceeds When the stability is achieved the learning process stops
2112005 22
ART-1 Model (3)Algorithm
1 Set the network parameter Nout=1
2 Set the initial weighting matrices
3 Input the training vector X
4 Calculate the matching value
5 Find the max matching value in the output nodes
Niw
iw
b
t
1
1]1][[
1]1][[
0
][]][[][
count
i
b
I
iXjiwjnet
][max][ jnetjnetj
2112005 23
ART-1 Model (4) Algorithm (continue)
(6) Calculate the similarity
value
(7) Test the similarity value
If then go to step (8)
Otherwise go to step (9)
(8) Test whether there are output nodes applicable to the rule
If IcountltNout then try the second max matching value in the output nodes
||||
||||
][]][[||||
][||||
X
XwV
iXjiwXw
iXX
t
j
j
i
tt
j
i
)(vigilance V
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 3
Cluster Analysis
bull Goalsndash Organize information about data so that relatively
homogeneous groups (clusters) are formed and describe their unknown properties
ndash Find useful and interesting groupings of samplesndash Find representatives for homogeneous groups
bull Two components of cluster analysisndash The (dis)similarity measure between two data
samplesndash The clustering algorithm
2112005 4
Hierarchy of clusters
2112005 5
Display of hierarchy as tree
2112005 6
Minimum-Distance Clustering
2112005 7
Vehicle Example
Vehicle Top speedkmh
Colour Airresistance
WeightKg
V1 220 red 030 1300V2 230 black 032 1400V3 260 red 029 1500V4 140 gray 035 800V5 155 blue 033 950V6 130 white 040 600V7 100 black 050 3000V8 105 red 060 2500V9 110 gray 055 3500
2112005 8
Vehicle Clusters
100 150 200 250 300500
1000
1500
2000
2500
3000
3500
Top speed [kmh]
We
igh
t [k
g] Sports cars
Medium market cars
Lorries
2112005 9
Terminology
100 150 200 250 300500
1000
1500
2000
2500
3000
3500
Top speed [kmh]
We
igh
t [k
g] Sports cars
Medium market cars
Lorries
Object or data point
feature
feature space
cluster
feature
label
2112005 10
DistanceSimilarity Measures
2112005 11
DistanceSimilarity Measures (2)
2112005 12
DistanceSimilarity Measures (3)
2112005 13
DistanceSimilarity Measures (4)
2112005 14
1048698HierarchydiamsHierarchical clusteringdiamsNon-hierarchical (flat) clustering
1048698Underlying model assumptiondiamsParametric clusteringdiamsNon-parametric clustering
1048698StrictnessdiamsHard clusteringdiamsSoft clustering
Clustering Algorithms
2112005 15
Adaptive Resonance Theory (ART)
bull One of the nice features of human memory is its ability to learn many new things without necessarily forgetting things learned in the past
bull Stephen Grossberg Stability-Plasticity dilemma(1)How can a learning system remain adaptive
(plastic) in response to significant input yet remain stable in response to irrelevant input
(2)How does the system known to switch between its plastic and its stable modes
(3)How can the system retain previously learned information while continuing to learn new things
2112005 16
Basic Concept of ART
bull A key to solving the stability-plasticity dilemma is to add a feedback mechanism between the competitive layer and the input layer of a network
bull Grossberg and Carpenter ART modelbull ART is one of the unsupervised learning modelsbull This kind of model was first established in the
early 1960bull Grossberg introduced the ART in 1976bull GA Carpenter continued the research in ARTbull The original ART model is very complicated
This session only discussed the simplified Carpenter-Grossberg network
2112005 17
Basic Concept of ART (2)
bull ART 1 requires that the input vectors be binarybull ART 2 is suitable for processing analog or gray
scale patternsbull ART gets its name from the particular way in
which learning and recall interplay in the network
bull In physics resonance occurs when a small-amplitude vibration of the proper frequency causes a large-amplitude vibration in an electrical or mechanical system
2112005 18
recognition
comparison
G2
G1
input
gain control
gain control
+
+
+
+
+
+
+
-
-top-downweightsTji
bottom-upweightsBji
attentional subsystem orienting subsystem
F0
F1
F2
Basic Concept of ART (3) The ART network (Carpenter and Grossberg 1988)
2112005 19
41 Basic Concept of ART
bull Bji Forward the output from F1 to F2 for competition
bull Tji Forward the pattern of winner neuron to F1 for comparison
bull G1 To distinguish the feature of input pattern with stored patterns
bull G2 To reset the depressed neurons in F2 (ie reset losers)
bull attentional subsystem to rapidly classify the recognized patterns
bull orienting subsystem to help attentional subsystem learn new patterns
2112005 20
ART-1 Model
input vector
output vector
output layercluster
input layerinput variables
2112005 21
ART-1 Model (2)
bull Input layer input patterns or characteristic vectors Activation function f(x)=x inputs are binary values
bull Output layer representing the clustering of training patterns This is similar to SOFM except that SOFM has the neighborhood concept Initially there is only one output node The number of output nodes increases when learning proceeds When the stability is achieved the learning process stops
2112005 22
ART-1 Model (3)Algorithm
1 Set the network parameter Nout=1
2 Set the initial weighting matrices
3 Input the training vector X
4 Calculate the matching value
5 Find the max matching value in the output nodes
Niw
iw
b
t
1
1]1][[
1]1][[
0
][]][[][
count
i
b
I
iXjiwjnet
][max][ jnetjnetj
2112005 23
ART-1 Model (4) Algorithm (continue)
(6) Calculate the similarity
value
(7) Test the similarity value
If then go to step (8)
Otherwise go to step (9)
(8) Test whether there are output nodes applicable to the rule
If IcountltNout then try the second max matching value in the output nodes
||||
||||
][]][[||||
][||||
X
XwV
iXjiwXw
iXX
t
j
j
i
tt
j
i
)(vigilance V
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 4
Hierarchy of clusters
2112005 5
Display of hierarchy as tree
2112005 6
Minimum-Distance Clustering
2112005 7
Vehicle Example
Vehicle Top speedkmh
Colour Airresistance
WeightKg
V1 220 red 030 1300V2 230 black 032 1400V3 260 red 029 1500V4 140 gray 035 800V5 155 blue 033 950V6 130 white 040 600V7 100 black 050 3000V8 105 red 060 2500V9 110 gray 055 3500
2112005 8
Vehicle Clusters
100 150 200 250 300500
1000
1500
2000
2500
3000
3500
Top speed [kmh]
We
igh
t [k
g] Sports cars
Medium market cars
Lorries
2112005 9
Terminology
100 150 200 250 300500
1000
1500
2000
2500
3000
3500
Top speed [kmh]
We
igh
t [k
g] Sports cars
Medium market cars
Lorries
Object or data point
feature
feature space
cluster
feature
label
2112005 10
DistanceSimilarity Measures
2112005 11
DistanceSimilarity Measures (2)
2112005 12
DistanceSimilarity Measures (3)
2112005 13
DistanceSimilarity Measures (4)
2112005 14
1048698HierarchydiamsHierarchical clusteringdiamsNon-hierarchical (flat) clustering
1048698Underlying model assumptiondiamsParametric clusteringdiamsNon-parametric clustering
1048698StrictnessdiamsHard clusteringdiamsSoft clustering
Clustering Algorithms
2112005 15
Adaptive Resonance Theory (ART)
bull One of the nice features of human memory is its ability to learn many new things without necessarily forgetting things learned in the past
bull Stephen Grossberg Stability-Plasticity dilemma(1)How can a learning system remain adaptive
(plastic) in response to significant input yet remain stable in response to irrelevant input
(2)How does the system known to switch between its plastic and its stable modes
(3)How can the system retain previously learned information while continuing to learn new things
2112005 16
Basic Concept of ART
bull A key to solving the stability-plasticity dilemma is to add a feedback mechanism between the competitive layer and the input layer of a network
bull Grossberg and Carpenter ART modelbull ART is one of the unsupervised learning modelsbull This kind of model was first established in the
early 1960bull Grossberg introduced the ART in 1976bull GA Carpenter continued the research in ARTbull The original ART model is very complicated
This session only discussed the simplified Carpenter-Grossberg network
2112005 17
Basic Concept of ART (2)
bull ART 1 requires that the input vectors be binarybull ART 2 is suitable for processing analog or gray
scale patternsbull ART gets its name from the particular way in
which learning and recall interplay in the network
bull In physics resonance occurs when a small-amplitude vibration of the proper frequency causes a large-amplitude vibration in an electrical or mechanical system
2112005 18
recognition
comparison
G2
G1
input
gain control
gain control
+
+
+
+
+
+
+
-
-top-downweightsTji
bottom-upweightsBji
attentional subsystem orienting subsystem
F0
F1
F2
Basic Concept of ART (3) The ART network (Carpenter and Grossberg 1988)
2112005 19
41 Basic Concept of ART
bull Bji Forward the output from F1 to F2 for competition
bull Tji Forward the pattern of winner neuron to F1 for comparison
bull G1 To distinguish the feature of input pattern with stored patterns
bull G2 To reset the depressed neurons in F2 (ie reset losers)
bull attentional subsystem to rapidly classify the recognized patterns
bull orienting subsystem to help attentional subsystem learn new patterns
2112005 20
ART-1 Model
input vector
output vector
output layercluster
input layerinput variables
2112005 21
ART-1 Model (2)
bull Input layer input patterns or characteristic vectors Activation function f(x)=x inputs are binary values
bull Output layer representing the clustering of training patterns This is similar to SOFM except that SOFM has the neighborhood concept Initially there is only one output node The number of output nodes increases when learning proceeds When the stability is achieved the learning process stops
2112005 22
ART-1 Model (3)Algorithm
1 Set the network parameter Nout=1
2 Set the initial weighting matrices
3 Input the training vector X
4 Calculate the matching value
5 Find the max matching value in the output nodes
Niw
iw
b
t
1
1]1][[
1]1][[
0
][]][[][
count
i
b
I
iXjiwjnet
][max][ jnetjnetj
2112005 23
ART-1 Model (4) Algorithm (continue)
(6) Calculate the similarity
value
(7) Test the similarity value
If then go to step (8)
Otherwise go to step (9)
(8) Test whether there are output nodes applicable to the rule
If IcountltNout then try the second max matching value in the output nodes
||||
||||
][]][[||||
][||||
X
XwV
iXjiwXw
iXX
t
j
j
i
tt
j
i
)(vigilance V
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 5
Display of hierarchy as tree
2112005 6
Minimum-Distance Clustering
2112005 7
Vehicle Example
Vehicle Top speedkmh
Colour Airresistance
WeightKg
V1 220 red 030 1300V2 230 black 032 1400V3 260 red 029 1500V4 140 gray 035 800V5 155 blue 033 950V6 130 white 040 600V7 100 black 050 3000V8 105 red 060 2500V9 110 gray 055 3500
2112005 8
Vehicle Clusters
100 150 200 250 300500
1000
1500
2000
2500
3000
3500
Top speed [kmh]
We
igh
t [k
g] Sports cars
Medium market cars
Lorries
2112005 9
Terminology
100 150 200 250 300500
1000
1500
2000
2500
3000
3500
Top speed [kmh]
We
igh
t [k
g] Sports cars
Medium market cars
Lorries
Object or data point
feature
feature space
cluster
feature
label
2112005 10
DistanceSimilarity Measures
2112005 11
DistanceSimilarity Measures (2)
2112005 12
DistanceSimilarity Measures (3)
2112005 13
DistanceSimilarity Measures (4)
2112005 14
1048698HierarchydiamsHierarchical clusteringdiamsNon-hierarchical (flat) clustering
1048698Underlying model assumptiondiamsParametric clusteringdiamsNon-parametric clustering
1048698StrictnessdiamsHard clusteringdiamsSoft clustering
Clustering Algorithms
2112005 15
Adaptive Resonance Theory (ART)
bull One of the nice features of human memory is its ability to learn many new things without necessarily forgetting things learned in the past
bull Stephen Grossberg Stability-Plasticity dilemma(1)How can a learning system remain adaptive
(plastic) in response to significant input yet remain stable in response to irrelevant input
(2)How does the system known to switch between its plastic and its stable modes
(3)How can the system retain previously learned information while continuing to learn new things
2112005 16
Basic Concept of ART
bull A key to solving the stability-plasticity dilemma is to add a feedback mechanism between the competitive layer and the input layer of a network
bull Grossberg and Carpenter ART modelbull ART is one of the unsupervised learning modelsbull This kind of model was first established in the
early 1960bull Grossberg introduced the ART in 1976bull GA Carpenter continued the research in ARTbull The original ART model is very complicated
This session only discussed the simplified Carpenter-Grossberg network
2112005 17
Basic Concept of ART (2)
bull ART 1 requires that the input vectors be binarybull ART 2 is suitable for processing analog or gray
scale patternsbull ART gets its name from the particular way in
which learning and recall interplay in the network
bull In physics resonance occurs when a small-amplitude vibration of the proper frequency causes a large-amplitude vibration in an electrical or mechanical system
2112005 18
recognition
comparison
G2
G1
input
gain control
gain control
+
+
+
+
+
+
+
-
-top-downweightsTji
bottom-upweightsBji
attentional subsystem orienting subsystem
F0
F1
F2
Basic Concept of ART (3) The ART network (Carpenter and Grossberg 1988)
2112005 19
41 Basic Concept of ART
bull Bji Forward the output from F1 to F2 for competition
bull Tji Forward the pattern of winner neuron to F1 for comparison
bull G1 To distinguish the feature of input pattern with stored patterns
bull G2 To reset the depressed neurons in F2 (ie reset losers)
bull attentional subsystem to rapidly classify the recognized patterns
bull orienting subsystem to help attentional subsystem learn new patterns
2112005 20
ART-1 Model
input vector
output vector
output layercluster
input layerinput variables
2112005 21
ART-1 Model (2)
bull Input layer input patterns or characteristic vectors Activation function f(x)=x inputs are binary values
bull Output layer representing the clustering of training patterns This is similar to SOFM except that SOFM has the neighborhood concept Initially there is only one output node The number of output nodes increases when learning proceeds When the stability is achieved the learning process stops
2112005 22
ART-1 Model (3)Algorithm
1 Set the network parameter Nout=1
2 Set the initial weighting matrices
3 Input the training vector X
4 Calculate the matching value
5 Find the max matching value in the output nodes
Niw
iw
b
t
1
1]1][[
1]1][[
0
][]][[][
count
i
b
I
iXjiwjnet
][max][ jnetjnetj
2112005 23
ART-1 Model (4) Algorithm (continue)
(6) Calculate the similarity
value
(7) Test the similarity value
If then go to step (8)
Otherwise go to step (9)
(8) Test whether there are output nodes applicable to the rule
If IcountltNout then try the second max matching value in the output nodes
||||
||||
][]][[||||
][||||
X
XwV
iXjiwXw
iXX
t
j
j
i
tt
j
i
)(vigilance V
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 6
Minimum-Distance Clustering
2112005 7
Vehicle Example
Vehicle Top speedkmh
Colour Airresistance
WeightKg
V1 220 red 030 1300V2 230 black 032 1400V3 260 red 029 1500V4 140 gray 035 800V5 155 blue 033 950V6 130 white 040 600V7 100 black 050 3000V8 105 red 060 2500V9 110 gray 055 3500
2112005 8
Vehicle Clusters
100 150 200 250 300500
1000
1500
2000
2500
3000
3500
Top speed [kmh]
We
igh
t [k
g] Sports cars
Medium market cars
Lorries
2112005 9
Terminology
100 150 200 250 300500
1000
1500
2000
2500
3000
3500
Top speed [kmh]
We
igh
t [k
g] Sports cars
Medium market cars
Lorries
Object or data point
feature
feature space
cluster
feature
label
2112005 10
DistanceSimilarity Measures
2112005 11
DistanceSimilarity Measures (2)
2112005 12
DistanceSimilarity Measures (3)
2112005 13
DistanceSimilarity Measures (4)
2112005 14
1048698HierarchydiamsHierarchical clusteringdiamsNon-hierarchical (flat) clustering
1048698Underlying model assumptiondiamsParametric clusteringdiamsNon-parametric clustering
1048698StrictnessdiamsHard clusteringdiamsSoft clustering
Clustering Algorithms
2112005 15
Adaptive Resonance Theory (ART)
bull One of the nice features of human memory is its ability to learn many new things without necessarily forgetting things learned in the past
bull Stephen Grossberg Stability-Plasticity dilemma(1)How can a learning system remain adaptive
(plastic) in response to significant input yet remain stable in response to irrelevant input
(2)How does the system known to switch between its plastic and its stable modes
(3)How can the system retain previously learned information while continuing to learn new things
2112005 16
Basic Concept of ART
bull A key to solving the stability-plasticity dilemma is to add a feedback mechanism between the competitive layer and the input layer of a network
bull Grossberg and Carpenter ART modelbull ART is one of the unsupervised learning modelsbull This kind of model was first established in the
early 1960bull Grossberg introduced the ART in 1976bull GA Carpenter continued the research in ARTbull The original ART model is very complicated
This session only discussed the simplified Carpenter-Grossberg network
2112005 17
Basic Concept of ART (2)
bull ART 1 requires that the input vectors be binarybull ART 2 is suitable for processing analog or gray
scale patternsbull ART gets its name from the particular way in
which learning and recall interplay in the network
bull In physics resonance occurs when a small-amplitude vibration of the proper frequency causes a large-amplitude vibration in an electrical or mechanical system
2112005 18
recognition
comparison
G2
G1
input
gain control
gain control
+
+
+
+
+
+
+
-
-top-downweightsTji
bottom-upweightsBji
attentional subsystem orienting subsystem
F0
F1
F2
Basic Concept of ART (3) The ART network (Carpenter and Grossberg 1988)
2112005 19
41 Basic Concept of ART
bull Bji Forward the output from F1 to F2 for competition
bull Tji Forward the pattern of winner neuron to F1 for comparison
bull G1 To distinguish the feature of input pattern with stored patterns
bull G2 To reset the depressed neurons in F2 (ie reset losers)
bull attentional subsystem to rapidly classify the recognized patterns
bull orienting subsystem to help attentional subsystem learn new patterns
2112005 20
ART-1 Model
input vector
output vector
output layercluster
input layerinput variables
2112005 21
ART-1 Model (2)
bull Input layer input patterns or characteristic vectors Activation function f(x)=x inputs are binary values
bull Output layer representing the clustering of training patterns This is similar to SOFM except that SOFM has the neighborhood concept Initially there is only one output node The number of output nodes increases when learning proceeds When the stability is achieved the learning process stops
2112005 22
ART-1 Model (3)Algorithm
1 Set the network parameter Nout=1
2 Set the initial weighting matrices
3 Input the training vector X
4 Calculate the matching value
5 Find the max matching value in the output nodes
Niw
iw
b
t
1
1]1][[
1]1][[
0
][]][[][
count
i
b
I
iXjiwjnet
][max][ jnetjnetj
2112005 23
ART-1 Model (4) Algorithm (continue)
(6) Calculate the similarity
value
(7) Test the similarity value
If then go to step (8)
Otherwise go to step (9)
(8) Test whether there are output nodes applicable to the rule
If IcountltNout then try the second max matching value in the output nodes
||||
||||
][]][[||||
][||||
X
XwV
iXjiwXw
iXX
t
j
j
i
tt
j
i
)(vigilance V
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 7
Vehicle Example
Vehicle Top speedkmh
Colour Airresistance
WeightKg
V1 220 red 030 1300V2 230 black 032 1400V3 260 red 029 1500V4 140 gray 035 800V5 155 blue 033 950V6 130 white 040 600V7 100 black 050 3000V8 105 red 060 2500V9 110 gray 055 3500
2112005 8
Vehicle Clusters
100 150 200 250 300500
1000
1500
2000
2500
3000
3500
Top speed [kmh]
We
igh
t [k
g] Sports cars
Medium market cars
Lorries
2112005 9
Terminology
100 150 200 250 300500
1000
1500
2000
2500
3000
3500
Top speed [kmh]
We
igh
t [k
g] Sports cars
Medium market cars
Lorries
Object or data point
feature
feature space
cluster
feature
label
2112005 10
DistanceSimilarity Measures
2112005 11
DistanceSimilarity Measures (2)
2112005 12
DistanceSimilarity Measures (3)
2112005 13
DistanceSimilarity Measures (4)
2112005 14
1048698HierarchydiamsHierarchical clusteringdiamsNon-hierarchical (flat) clustering
1048698Underlying model assumptiondiamsParametric clusteringdiamsNon-parametric clustering
1048698StrictnessdiamsHard clusteringdiamsSoft clustering
Clustering Algorithms
2112005 15
Adaptive Resonance Theory (ART)
bull One of the nice features of human memory is its ability to learn many new things without necessarily forgetting things learned in the past
bull Stephen Grossberg Stability-Plasticity dilemma(1)How can a learning system remain adaptive
(plastic) in response to significant input yet remain stable in response to irrelevant input
(2)How does the system known to switch between its plastic and its stable modes
(3)How can the system retain previously learned information while continuing to learn new things
2112005 16
Basic Concept of ART
bull A key to solving the stability-plasticity dilemma is to add a feedback mechanism between the competitive layer and the input layer of a network
bull Grossberg and Carpenter ART modelbull ART is one of the unsupervised learning modelsbull This kind of model was first established in the
early 1960bull Grossberg introduced the ART in 1976bull GA Carpenter continued the research in ARTbull The original ART model is very complicated
This session only discussed the simplified Carpenter-Grossberg network
2112005 17
Basic Concept of ART (2)
bull ART 1 requires that the input vectors be binarybull ART 2 is suitable for processing analog or gray
scale patternsbull ART gets its name from the particular way in
which learning and recall interplay in the network
bull In physics resonance occurs when a small-amplitude vibration of the proper frequency causes a large-amplitude vibration in an electrical or mechanical system
2112005 18
recognition
comparison
G2
G1
input
gain control
gain control
+
+
+
+
+
+
+
-
-top-downweightsTji
bottom-upweightsBji
attentional subsystem orienting subsystem
F0
F1
F2
Basic Concept of ART (3) The ART network (Carpenter and Grossberg 1988)
2112005 19
41 Basic Concept of ART
bull Bji Forward the output from F1 to F2 for competition
bull Tji Forward the pattern of winner neuron to F1 for comparison
bull G1 To distinguish the feature of input pattern with stored patterns
bull G2 To reset the depressed neurons in F2 (ie reset losers)
bull attentional subsystem to rapidly classify the recognized patterns
bull orienting subsystem to help attentional subsystem learn new patterns
2112005 20
ART-1 Model
input vector
output vector
output layercluster
input layerinput variables
2112005 21
ART-1 Model (2)
bull Input layer input patterns or characteristic vectors Activation function f(x)=x inputs are binary values
bull Output layer representing the clustering of training patterns This is similar to SOFM except that SOFM has the neighborhood concept Initially there is only one output node The number of output nodes increases when learning proceeds When the stability is achieved the learning process stops
2112005 22
ART-1 Model (3)Algorithm
1 Set the network parameter Nout=1
2 Set the initial weighting matrices
3 Input the training vector X
4 Calculate the matching value
5 Find the max matching value in the output nodes
Niw
iw
b
t
1
1]1][[
1]1][[
0
][]][[][
count
i
b
I
iXjiwjnet
][max][ jnetjnetj
2112005 23
ART-1 Model (4) Algorithm (continue)
(6) Calculate the similarity
value
(7) Test the similarity value
If then go to step (8)
Otherwise go to step (9)
(8) Test whether there are output nodes applicable to the rule
If IcountltNout then try the second max matching value in the output nodes
||||
||||
][]][[||||
][||||
X
XwV
iXjiwXw
iXX
t
j
j
i
tt
j
i
)(vigilance V
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 8
Vehicle Clusters
100 150 200 250 300500
1000
1500
2000
2500
3000
3500
Top speed [kmh]
We
igh
t [k
g] Sports cars
Medium market cars
Lorries
2112005 9
Terminology
100 150 200 250 300500
1000
1500
2000
2500
3000
3500
Top speed [kmh]
We
igh
t [k
g] Sports cars
Medium market cars
Lorries
Object or data point
feature
feature space
cluster
feature
label
2112005 10
DistanceSimilarity Measures
2112005 11
DistanceSimilarity Measures (2)
2112005 12
DistanceSimilarity Measures (3)
2112005 13
DistanceSimilarity Measures (4)
2112005 14
1048698HierarchydiamsHierarchical clusteringdiamsNon-hierarchical (flat) clustering
1048698Underlying model assumptiondiamsParametric clusteringdiamsNon-parametric clustering
1048698StrictnessdiamsHard clusteringdiamsSoft clustering
Clustering Algorithms
2112005 15
Adaptive Resonance Theory (ART)
bull One of the nice features of human memory is its ability to learn many new things without necessarily forgetting things learned in the past
bull Stephen Grossberg Stability-Plasticity dilemma(1)How can a learning system remain adaptive
(plastic) in response to significant input yet remain stable in response to irrelevant input
(2)How does the system known to switch between its plastic and its stable modes
(3)How can the system retain previously learned information while continuing to learn new things
2112005 16
Basic Concept of ART
bull A key to solving the stability-plasticity dilemma is to add a feedback mechanism between the competitive layer and the input layer of a network
bull Grossberg and Carpenter ART modelbull ART is one of the unsupervised learning modelsbull This kind of model was first established in the
early 1960bull Grossberg introduced the ART in 1976bull GA Carpenter continued the research in ARTbull The original ART model is very complicated
This session only discussed the simplified Carpenter-Grossberg network
2112005 17
Basic Concept of ART (2)
bull ART 1 requires that the input vectors be binarybull ART 2 is suitable for processing analog or gray
scale patternsbull ART gets its name from the particular way in
which learning and recall interplay in the network
bull In physics resonance occurs when a small-amplitude vibration of the proper frequency causes a large-amplitude vibration in an electrical or mechanical system
2112005 18
recognition
comparison
G2
G1
input
gain control
gain control
+
+
+
+
+
+
+
-
-top-downweightsTji
bottom-upweightsBji
attentional subsystem orienting subsystem
F0
F1
F2
Basic Concept of ART (3) The ART network (Carpenter and Grossberg 1988)
2112005 19
41 Basic Concept of ART
bull Bji Forward the output from F1 to F2 for competition
bull Tji Forward the pattern of winner neuron to F1 for comparison
bull G1 To distinguish the feature of input pattern with stored patterns
bull G2 To reset the depressed neurons in F2 (ie reset losers)
bull attentional subsystem to rapidly classify the recognized patterns
bull orienting subsystem to help attentional subsystem learn new patterns
2112005 20
ART-1 Model
input vector
output vector
output layercluster
input layerinput variables
2112005 21
ART-1 Model (2)
bull Input layer input patterns or characteristic vectors Activation function f(x)=x inputs are binary values
bull Output layer representing the clustering of training patterns This is similar to SOFM except that SOFM has the neighborhood concept Initially there is only one output node The number of output nodes increases when learning proceeds When the stability is achieved the learning process stops
2112005 22
ART-1 Model (3)Algorithm
1 Set the network parameter Nout=1
2 Set the initial weighting matrices
3 Input the training vector X
4 Calculate the matching value
5 Find the max matching value in the output nodes
Niw
iw
b
t
1
1]1][[
1]1][[
0
][]][[][
count
i
b
I
iXjiwjnet
][max][ jnetjnetj
2112005 23
ART-1 Model (4) Algorithm (continue)
(6) Calculate the similarity
value
(7) Test the similarity value
If then go to step (8)
Otherwise go to step (9)
(8) Test whether there are output nodes applicable to the rule
If IcountltNout then try the second max matching value in the output nodes
||||
||||
][]][[||||
][||||
X
XwV
iXjiwXw
iXX
t
j
j
i
tt
j
i
)(vigilance V
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 9
Terminology
100 150 200 250 300500
1000
1500
2000
2500
3000
3500
Top speed [kmh]
We
igh
t [k
g] Sports cars
Medium market cars
Lorries
Object or data point
feature
feature space
cluster
feature
label
2112005 10
DistanceSimilarity Measures
2112005 11
DistanceSimilarity Measures (2)
2112005 12
DistanceSimilarity Measures (3)
2112005 13
DistanceSimilarity Measures (4)
2112005 14
1048698HierarchydiamsHierarchical clusteringdiamsNon-hierarchical (flat) clustering
1048698Underlying model assumptiondiamsParametric clusteringdiamsNon-parametric clustering
1048698StrictnessdiamsHard clusteringdiamsSoft clustering
Clustering Algorithms
2112005 15
Adaptive Resonance Theory (ART)
bull One of the nice features of human memory is its ability to learn many new things without necessarily forgetting things learned in the past
bull Stephen Grossberg Stability-Plasticity dilemma(1)How can a learning system remain adaptive
(plastic) in response to significant input yet remain stable in response to irrelevant input
(2)How does the system known to switch between its plastic and its stable modes
(3)How can the system retain previously learned information while continuing to learn new things
2112005 16
Basic Concept of ART
bull A key to solving the stability-plasticity dilemma is to add a feedback mechanism between the competitive layer and the input layer of a network
bull Grossberg and Carpenter ART modelbull ART is one of the unsupervised learning modelsbull This kind of model was first established in the
early 1960bull Grossberg introduced the ART in 1976bull GA Carpenter continued the research in ARTbull The original ART model is very complicated
This session only discussed the simplified Carpenter-Grossberg network
2112005 17
Basic Concept of ART (2)
bull ART 1 requires that the input vectors be binarybull ART 2 is suitable for processing analog or gray
scale patternsbull ART gets its name from the particular way in
which learning and recall interplay in the network
bull In physics resonance occurs when a small-amplitude vibration of the proper frequency causes a large-amplitude vibration in an electrical or mechanical system
2112005 18
recognition
comparison
G2
G1
input
gain control
gain control
+
+
+
+
+
+
+
-
-top-downweightsTji
bottom-upweightsBji
attentional subsystem orienting subsystem
F0
F1
F2
Basic Concept of ART (3) The ART network (Carpenter and Grossberg 1988)
2112005 19
41 Basic Concept of ART
bull Bji Forward the output from F1 to F2 for competition
bull Tji Forward the pattern of winner neuron to F1 for comparison
bull G1 To distinguish the feature of input pattern with stored patterns
bull G2 To reset the depressed neurons in F2 (ie reset losers)
bull attentional subsystem to rapidly classify the recognized patterns
bull orienting subsystem to help attentional subsystem learn new patterns
2112005 20
ART-1 Model
input vector
output vector
output layercluster
input layerinput variables
2112005 21
ART-1 Model (2)
bull Input layer input patterns or characteristic vectors Activation function f(x)=x inputs are binary values
bull Output layer representing the clustering of training patterns This is similar to SOFM except that SOFM has the neighborhood concept Initially there is only one output node The number of output nodes increases when learning proceeds When the stability is achieved the learning process stops
2112005 22
ART-1 Model (3)Algorithm
1 Set the network parameter Nout=1
2 Set the initial weighting matrices
3 Input the training vector X
4 Calculate the matching value
5 Find the max matching value in the output nodes
Niw
iw
b
t
1
1]1][[
1]1][[
0
][]][[][
count
i
b
I
iXjiwjnet
][max][ jnetjnetj
2112005 23
ART-1 Model (4) Algorithm (continue)
(6) Calculate the similarity
value
(7) Test the similarity value
If then go to step (8)
Otherwise go to step (9)
(8) Test whether there are output nodes applicable to the rule
If IcountltNout then try the second max matching value in the output nodes
||||
||||
][]][[||||
][||||
X
XwV
iXjiwXw
iXX
t
j
j
i
tt
j
i
)(vigilance V
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 10
DistanceSimilarity Measures
2112005 11
DistanceSimilarity Measures (2)
2112005 12
DistanceSimilarity Measures (3)
2112005 13
DistanceSimilarity Measures (4)
2112005 14
1048698HierarchydiamsHierarchical clusteringdiamsNon-hierarchical (flat) clustering
1048698Underlying model assumptiondiamsParametric clusteringdiamsNon-parametric clustering
1048698StrictnessdiamsHard clusteringdiamsSoft clustering
Clustering Algorithms
2112005 15
Adaptive Resonance Theory (ART)
bull One of the nice features of human memory is its ability to learn many new things without necessarily forgetting things learned in the past
bull Stephen Grossberg Stability-Plasticity dilemma(1)How can a learning system remain adaptive
(plastic) in response to significant input yet remain stable in response to irrelevant input
(2)How does the system known to switch between its plastic and its stable modes
(3)How can the system retain previously learned information while continuing to learn new things
2112005 16
Basic Concept of ART
bull A key to solving the stability-plasticity dilemma is to add a feedback mechanism between the competitive layer and the input layer of a network
bull Grossberg and Carpenter ART modelbull ART is one of the unsupervised learning modelsbull This kind of model was first established in the
early 1960bull Grossberg introduced the ART in 1976bull GA Carpenter continued the research in ARTbull The original ART model is very complicated
This session only discussed the simplified Carpenter-Grossberg network
2112005 17
Basic Concept of ART (2)
bull ART 1 requires that the input vectors be binarybull ART 2 is suitable for processing analog or gray
scale patternsbull ART gets its name from the particular way in
which learning and recall interplay in the network
bull In physics resonance occurs when a small-amplitude vibration of the proper frequency causes a large-amplitude vibration in an electrical or mechanical system
2112005 18
recognition
comparison
G2
G1
input
gain control
gain control
+
+
+
+
+
+
+
-
-top-downweightsTji
bottom-upweightsBji
attentional subsystem orienting subsystem
F0
F1
F2
Basic Concept of ART (3) The ART network (Carpenter and Grossberg 1988)
2112005 19
41 Basic Concept of ART
bull Bji Forward the output from F1 to F2 for competition
bull Tji Forward the pattern of winner neuron to F1 for comparison
bull G1 To distinguish the feature of input pattern with stored patterns
bull G2 To reset the depressed neurons in F2 (ie reset losers)
bull attentional subsystem to rapidly classify the recognized patterns
bull orienting subsystem to help attentional subsystem learn new patterns
2112005 20
ART-1 Model
input vector
output vector
output layercluster
input layerinput variables
2112005 21
ART-1 Model (2)
bull Input layer input patterns or characteristic vectors Activation function f(x)=x inputs are binary values
bull Output layer representing the clustering of training patterns This is similar to SOFM except that SOFM has the neighborhood concept Initially there is only one output node The number of output nodes increases when learning proceeds When the stability is achieved the learning process stops
2112005 22
ART-1 Model (3)Algorithm
1 Set the network parameter Nout=1
2 Set the initial weighting matrices
3 Input the training vector X
4 Calculate the matching value
5 Find the max matching value in the output nodes
Niw
iw
b
t
1
1]1][[
1]1][[
0
][]][[][
count
i
b
I
iXjiwjnet
][max][ jnetjnetj
2112005 23
ART-1 Model (4) Algorithm (continue)
(6) Calculate the similarity
value
(7) Test the similarity value
If then go to step (8)
Otherwise go to step (9)
(8) Test whether there are output nodes applicable to the rule
If IcountltNout then try the second max matching value in the output nodes
||||
||||
][]][[||||
][||||
X
XwV
iXjiwXw
iXX
t
j
j
i
tt
j
i
)(vigilance V
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 11
DistanceSimilarity Measures (2)
2112005 12
DistanceSimilarity Measures (3)
2112005 13
DistanceSimilarity Measures (4)
2112005 14
1048698HierarchydiamsHierarchical clusteringdiamsNon-hierarchical (flat) clustering
1048698Underlying model assumptiondiamsParametric clusteringdiamsNon-parametric clustering
1048698StrictnessdiamsHard clusteringdiamsSoft clustering
Clustering Algorithms
2112005 15
Adaptive Resonance Theory (ART)
bull One of the nice features of human memory is its ability to learn many new things without necessarily forgetting things learned in the past
bull Stephen Grossberg Stability-Plasticity dilemma(1)How can a learning system remain adaptive
(plastic) in response to significant input yet remain stable in response to irrelevant input
(2)How does the system known to switch between its plastic and its stable modes
(3)How can the system retain previously learned information while continuing to learn new things
2112005 16
Basic Concept of ART
bull A key to solving the stability-plasticity dilemma is to add a feedback mechanism between the competitive layer and the input layer of a network
bull Grossberg and Carpenter ART modelbull ART is one of the unsupervised learning modelsbull This kind of model was first established in the
early 1960bull Grossberg introduced the ART in 1976bull GA Carpenter continued the research in ARTbull The original ART model is very complicated
This session only discussed the simplified Carpenter-Grossberg network
2112005 17
Basic Concept of ART (2)
bull ART 1 requires that the input vectors be binarybull ART 2 is suitable for processing analog or gray
scale patternsbull ART gets its name from the particular way in
which learning and recall interplay in the network
bull In physics resonance occurs when a small-amplitude vibration of the proper frequency causes a large-amplitude vibration in an electrical or mechanical system
2112005 18
recognition
comparison
G2
G1
input
gain control
gain control
+
+
+
+
+
+
+
-
-top-downweightsTji
bottom-upweightsBji
attentional subsystem orienting subsystem
F0
F1
F2
Basic Concept of ART (3) The ART network (Carpenter and Grossberg 1988)
2112005 19
41 Basic Concept of ART
bull Bji Forward the output from F1 to F2 for competition
bull Tji Forward the pattern of winner neuron to F1 for comparison
bull G1 To distinguish the feature of input pattern with stored patterns
bull G2 To reset the depressed neurons in F2 (ie reset losers)
bull attentional subsystem to rapidly classify the recognized patterns
bull orienting subsystem to help attentional subsystem learn new patterns
2112005 20
ART-1 Model
input vector
output vector
output layercluster
input layerinput variables
2112005 21
ART-1 Model (2)
bull Input layer input patterns or characteristic vectors Activation function f(x)=x inputs are binary values
bull Output layer representing the clustering of training patterns This is similar to SOFM except that SOFM has the neighborhood concept Initially there is only one output node The number of output nodes increases when learning proceeds When the stability is achieved the learning process stops
2112005 22
ART-1 Model (3)Algorithm
1 Set the network parameter Nout=1
2 Set the initial weighting matrices
3 Input the training vector X
4 Calculate the matching value
5 Find the max matching value in the output nodes
Niw
iw
b
t
1
1]1][[
1]1][[
0
][]][[][
count
i
b
I
iXjiwjnet
][max][ jnetjnetj
2112005 23
ART-1 Model (4) Algorithm (continue)
(6) Calculate the similarity
value
(7) Test the similarity value
If then go to step (8)
Otherwise go to step (9)
(8) Test whether there are output nodes applicable to the rule
If IcountltNout then try the second max matching value in the output nodes
||||
||||
][]][[||||
][||||
X
XwV
iXjiwXw
iXX
t
j
j
i
tt
j
i
)(vigilance V
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 12
DistanceSimilarity Measures (3)
2112005 13
DistanceSimilarity Measures (4)
2112005 14
1048698HierarchydiamsHierarchical clusteringdiamsNon-hierarchical (flat) clustering
1048698Underlying model assumptiondiamsParametric clusteringdiamsNon-parametric clustering
1048698StrictnessdiamsHard clusteringdiamsSoft clustering
Clustering Algorithms
2112005 15
Adaptive Resonance Theory (ART)
bull One of the nice features of human memory is its ability to learn many new things without necessarily forgetting things learned in the past
bull Stephen Grossberg Stability-Plasticity dilemma(1)How can a learning system remain adaptive
(plastic) in response to significant input yet remain stable in response to irrelevant input
(2)How does the system known to switch between its plastic and its stable modes
(3)How can the system retain previously learned information while continuing to learn new things
2112005 16
Basic Concept of ART
bull A key to solving the stability-plasticity dilemma is to add a feedback mechanism between the competitive layer and the input layer of a network
bull Grossberg and Carpenter ART modelbull ART is one of the unsupervised learning modelsbull This kind of model was first established in the
early 1960bull Grossberg introduced the ART in 1976bull GA Carpenter continued the research in ARTbull The original ART model is very complicated
This session only discussed the simplified Carpenter-Grossberg network
2112005 17
Basic Concept of ART (2)
bull ART 1 requires that the input vectors be binarybull ART 2 is suitable for processing analog or gray
scale patternsbull ART gets its name from the particular way in
which learning and recall interplay in the network
bull In physics resonance occurs when a small-amplitude vibration of the proper frequency causes a large-amplitude vibration in an electrical or mechanical system
2112005 18
recognition
comparison
G2
G1
input
gain control
gain control
+
+
+
+
+
+
+
-
-top-downweightsTji
bottom-upweightsBji
attentional subsystem orienting subsystem
F0
F1
F2
Basic Concept of ART (3) The ART network (Carpenter and Grossberg 1988)
2112005 19
41 Basic Concept of ART
bull Bji Forward the output from F1 to F2 for competition
bull Tji Forward the pattern of winner neuron to F1 for comparison
bull G1 To distinguish the feature of input pattern with stored patterns
bull G2 To reset the depressed neurons in F2 (ie reset losers)
bull attentional subsystem to rapidly classify the recognized patterns
bull orienting subsystem to help attentional subsystem learn new patterns
2112005 20
ART-1 Model
input vector
output vector
output layercluster
input layerinput variables
2112005 21
ART-1 Model (2)
bull Input layer input patterns or characteristic vectors Activation function f(x)=x inputs are binary values
bull Output layer representing the clustering of training patterns This is similar to SOFM except that SOFM has the neighborhood concept Initially there is only one output node The number of output nodes increases when learning proceeds When the stability is achieved the learning process stops
2112005 22
ART-1 Model (3)Algorithm
1 Set the network parameter Nout=1
2 Set the initial weighting matrices
3 Input the training vector X
4 Calculate the matching value
5 Find the max matching value in the output nodes
Niw
iw
b
t
1
1]1][[
1]1][[
0
][]][[][
count
i
b
I
iXjiwjnet
][max][ jnetjnetj
2112005 23
ART-1 Model (4) Algorithm (continue)
(6) Calculate the similarity
value
(7) Test the similarity value
If then go to step (8)
Otherwise go to step (9)
(8) Test whether there are output nodes applicable to the rule
If IcountltNout then try the second max matching value in the output nodes
||||
||||
][]][[||||
][||||
X
XwV
iXjiwXw
iXX
t
j
j
i
tt
j
i
)(vigilance V
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 13
DistanceSimilarity Measures (4)
2112005 14
1048698HierarchydiamsHierarchical clusteringdiamsNon-hierarchical (flat) clustering
1048698Underlying model assumptiondiamsParametric clusteringdiamsNon-parametric clustering
1048698StrictnessdiamsHard clusteringdiamsSoft clustering
Clustering Algorithms
2112005 15
Adaptive Resonance Theory (ART)
bull One of the nice features of human memory is its ability to learn many new things without necessarily forgetting things learned in the past
bull Stephen Grossberg Stability-Plasticity dilemma(1)How can a learning system remain adaptive
(plastic) in response to significant input yet remain stable in response to irrelevant input
(2)How does the system known to switch between its plastic and its stable modes
(3)How can the system retain previously learned information while continuing to learn new things
2112005 16
Basic Concept of ART
bull A key to solving the stability-plasticity dilemma is to add a feedback mechanism between the competitive layer and the input layer of a network
bull Grossberg and Carpenter ART modelbull ART is one of the unsupervised learning modelsbull This kind of model was first established in the
early 1960bull Grossberg introduced the ART in 1976bull GA Carpenter continued the research in ARTbull The original ART model is very complicated
This session only discussed the simplified Carpenter-Grossberg network
2112005 17
Basic Concept of ART (2)
bull ART 1 requires that the input vectors be binarybull ART 2 is suitable for processing analog or gray
scale patternsbull ART gets its name from the particular way in
which learning and recall interplay in the network
bull In physics resonance occurs when a small-amplitude vibration of the proper frequency causes a large-amplitude vibration in an electrical or mechanical system
2112005 18
recognition
comparison
G2
G1
input
gain control
gain control
+
+
+
+
+
+
+
-
-top-downweightsTji
bottom-upweightsBji
attentional subsystem orienting subsystem
F0
F1
F2
Basic Concept of ART (3) The ART network (Carpenter and Grossberg 1988)
2112005 19
41 Basic Concept of ART
bull Bji Forward the output from F1 to F2 for competition
bull Tji Forward the pattern of winner neuron to F1 for comparison
bull G1 To distinguish the feature of input pattern with stored patterns
bull G2 To reset the depressed neurons in F2 (ie reset losers)
bull attentional subsystem to rapidly classify the recognized patterns
bull orienting subsystem to help attentional subsystem learn new patterns
2112005 20
ART-1 Model
input vector
output vector
output layercluster
input layerinput variables
2112005 21
ART-1 Model (2)
bull Input layer input patterns or characteristic vectors Activation function f(x)=x inputs are binary values
bull Output layer representing the clustering of training patterns This is similar to SOFM except that SOFM has the neighborhood concept Initially there is only one output node The number of output nodes increases when learning proceeds When the stability is achieved the learning process stops
2112005 22
ART-1 Model (3)Algorithm
1 Set the network parameter Nout=1
2 Set the initial weighting matrices
3 Input the training vector X
4 Calculate the matching value
5 Find the max matching value in the output nodes
Niw
iw
b
t
1
1]1][[
1]1][[
0
][]][[][
count
i
b
I
iXjiwjnet
][max][ jnetjnetj
2112005 23
ART-1 Model (4) Algorithm (continue)
(6) Calculate the similarity
value
(7) Test the similarity value
If then go to step (8)
Otherwise go to step (9)
(8) Test whether there are output nodes applicable to the rule
If IcountltNout then try the second max matching value in the output nodes
||||
||||
][]][[||||
][||||
X
XwV
iXjiwXw
iXX
t
j
j
i
tt
j
i
)(vigilance V
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 14
1048698HierarchydiamsHierarchical clusteringdiamsNon-hierarchical (flat) clustering
1048698Underlying model assumptiondiamsParametric clusteringdiamsNon-parametric clustering
1048698StrictnessdiamsHard clusteringdiamsSoft clustering
Clustering Algorithms
2112005 15
Adaptive Resonance Theory (ART)
bull One of the nice features of human memory is its ability to learn many new things without necessarily forgetting things learned in the past
bull Stephen Grossberg Stability-Plasticity dilemma(1)How can a learning system remain adaptive
(plastic) in response to significant input yet remain stable in response to irrelevant input
(2)How does the system known to switch between its plastic and its stable modes
(3)How can the system retain previously learned information while continuing to learn new things
2112005 16
Basic Concept of ART
bull A key to solving the stability-plasticity dilemma is to add a feedback mechanism between the competitive layer and the input layer of a network
bull Grossberg and Carpenter ART modelbull ART is one of the unsupervised learning modelsbull This kind of model was first established in the
early 1960bull Grossberg introduced the ART in 1976bull GA Carpenter continued the research in ARTbull The original ART model is very complicated
This session only discussed the simplified Carpenter-Grossberg network
2112005 17
Basic Concept of ART (2)
bull ART 1 requires that the input vectors be binarybull ART 2 is suitable for processing analog or gray
scale patternsbull ART gets its name from the particular way in
which learning and recall interplay in the network
bull In physics resonance occurs when a small-amplitude vibration of the proper frequency causes a large-amplitude vibration in an electrical or mechanical system
2112005 18
recognition
comparison
G2
G1
input
gain control
gain control
+
+
+
+
+
+
+
-
-top-downweightsTji
bottom-upweightsBji
attentional subsystem orienting subsystem
F0
F1
F2
Basic Concept of ART (3) The ART network (Carpenter and Grossberg 1988)
2112005 19
41 Basic Concept of ART
bull Bji Forward the output from F1 to F2 for competition
bull Tji Forward the pattern of winner neuron to F1 for comparison
bull G1 To distinguish the feature of input pattern with stored patterns
bull G2 To reset the depressed neurons in F2 (ie reset losers)
bull attentional subsystem to rapidly classify the recognized patterns
bull orienting subsystem to help attentional subsystem learn new patterns
2112005 20
ART-1 Model
input vector
output vector
output layercluster
input layerinput variables
2112005 21
ART-1 Model (2)
bull Input layer input patterns or characteristic vectors Activation function f(x)=x inputs are binary values
bull Output layer representing the clustering of training patterns This is similar to SOFM except that SOFM has the neighborhood concept Initially there is only one output node The number of output nodes increases when learning proceeds When the stability is achieved the learning process stops
2112005 22
ART-1 Model (3)Algorithm
1 Set the network parameter Nout=1
2 Set the initial weighting matrices
3 Input the training vector X
4 Calculate the matching value
5 Find the max matching value in the output nodes
Niw
iw
b
t
1
1]1][[
1]1][[
0
][]][[][
count
i
b
I
iXjiwjnet
][max][ jnetjnetj
2112005 23
ART-1 Model (4) Algorithm (continue)
(6) Calculate the similarity
value
(7) Test the similarity value
If then go to step (8)
Otherwise go to step (9)
(8) Test whether there are output nodes applicable to the rule
If IcountltNout then try the second max matching value in the output nodes
||||
||||
][]][[||||
][||||
X
XwV
iXjiwXw
iXX
t
j
j
i
tt
j
i
)(vigilance V
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 15
Adaptive Resonance Theory (ART)
bull One of the nice features of human memory is its ability to learn many new things without necessarily forgetting things learned in the past
bull Stephen Grossberg Stability-Plasticity dilemma(1)How can a learning system remain adaptive
(plastic) in response to significant input yet remain stable in response to irrelevant input
(2)How does the system known to switch between its plastic and its stable modes
(3)How can the system retain previously learned information while continuing to learn new things
2112005 16
Basic Concept of ART
bull A key to solving the stability-plasticity dilemma is to add a feedback mechanism between the competitive layer and the input layer of a network
bull Grossberg and Carpenter ART modelbull ART is one of the unsupervised learning modelsbull This kind of model was first established in the
early 1960bull Grossberg introduced the ART in 1976bull GA Carpenter continued the research in ARTbull The original ART model is very complicated
This session only discussed the simplified Carpenter-Grossberg network
2112005 17
Basic Concept of ART (2)
bull ART 1 requires that the input vectors be binarybull ART 2 is suitable for processing analog or gray
scale patternsbull ART gets its name from the particular way in
which learning and recall interplay in the network
bull In physics resonance occurs when a small-amplitude vibration of the proper frequency causes a large-amplitude vibration in an electrical or mechanical system
2112005 18
recognition
comparison
G2
G1
input
gain control
gain control
+
+
+
+
+
+
+
-
-top-downweightsTji
bottom-upweightsBji
attentional subsystem orienting subsystem
F0
F1
F2
Basic Concept of ART (3) The ART network (Carpenter and Grossberg 1988)
2112005 19
41 Basic Concept of ART
bull Bji Forward the output from F1 to F2 for competition
bull Tji Forward the pattern of winner neuron to F1 for comparison
bull G1 To distinguish the feature of input pattern with stored patterns
bull G2 To reset the depressed neurons in F2 (ie reset losers)
bull attentional subsystem to rapidly classify the recognized patterns
bull orienting subsystem to help attentional subsystem learn new patterns
2112005 20
ART-1 Model
input vector
output vector
output layercluster
input layerinput variables
2112005 21
ART-1 Model (2)
bull Input layer input patterns or characteristic vectors Activation function f(x)=x inputs are binary values
bull Output layer representing the clustering of training patterns This is similar to SOFM except that SOFM has the neighborhood concept Initially there is only one output node The number of output nodes increases when learning proceeds When the stability is achieved the learning process stops
2112005 22
ART-1 Model (3)Algorithm
1 Set the network parameter Nout=1
2 Set the initial weighting matrices
3 Input the training vector X
4 Calculate the matching value
5 Find the max matching value in the output nodes
Niw
iw
b
t
1
1]1][[
1]1][[
0
][]][[][
count
i
b
I
iXjiwjnet
][max][ jnetjnetj
2112005 23
ART-1 Model (4) Algorithm (continue)
(6) Calculate the similarity
value
(7) Test the similarity value
If then go to step (8)
Otherwise go to step (9)
(8) Test whether there are output nodes applicable to the rule
If IcountltNout then try the second max matching value in the output nodes
||||
||||
][]][[||||
][||||
X
XwV
iXjiwXw
iXX
t
j
j
i
tt
j
i
)(vigilance V
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 16
Basic Concept of ART
bull A key to solving the stability-plasticity dilemma is to add a feedback mechanism between the competitive layer and the input layer of a network
bull Grossberg and Carpenter ART modelbull ART is one of the unsupervised learning modelsbull This kind of model was first established in the
early 1960bull Grossberg introduced the ART in 1976bull GA Carpenter continued the research in ARTbull The original ART model is very complicated
This session only discussed the simplified Carpenter-Grossberg network
2112005 17
Basic Concept of ART (2)
bull ART 1 requires that the input vectors be binarybull ART 2 is suitable for processing analog or gray
scale patternsbull ART gets its name from the particular way in
which learning and recall interplay in the network
bull In physics resonance occurs when a small-amplitude vibration of the proper frequency causes a large-amplitude vibration in an electrical or mechanical system
2112005 18
recognition
comparison
G2
G1
input
gain control
gain control
+
+
+
+
+
+
+
-
-top-downweightsTji
bottom-upweightsBji
attentional subsystem orienting subsystem
F0
F1
F2
Basic Concept of ART (3) The ART network (Carpenter and Grossberg 1988)
2112005 19
41 Basic Concept of ART
bull Bji Forward the output from F1 to F2 for competition
bull Tji Forward the pattern of winner neuron to F1 for comparison
bull G1 To distinguish the feature of input pattern with stored patterns
bull G2 To reset the depressed neurons in F2 (ie reset losers)
bull attentional subsystem to rapidly classify the recognized patterns
bull orienting subsystem to help attentional subsystem learn new patterns
2112005 20
ART-1 Model
input vector
output vector
output layercluster
input layerinput variables
2112005 21
ART-1 Model (2)
bull Input layer input patterns or characteristic vectors Activation function f(x)=x inputs are binary values
bull Output layer representing the clustering of training patterns This is similar to SOFM except that SOFM has the neighborhood concept Initially there is only one output node The number of output nodes increases when learning proceeds When the stability is achieved the learning process stops
2112005 22
ART-1 Model (3)Algorithm
1 Set the network parameter Nout=1
2 Set the initial weighting matrices
3 Input the training vector X
4 Calculate the matching value
5 Find the max matching value in the output nodes
Niw
iw
b
t
1
1]1][[
1]1][[
0
][]][[][
count
i
b
I
iXjiwjnet
][max][ jnetjnetj
2112005 23
ART-1 Model (4) Algorithm (continue)
(6) Calculate the similarity
value
(7) Test the similarity value
If then go to step (8)
Otherwise go to step (9)
(8) Test whether there are output nodes applicable to the rule
If IcountltNout then try the second max matching value in the output nodes
||||
||||
][]][[||||
][||||
X
XwV
iXjiwXw
iXX
t
j
j
i
tt
j
i
)(vigilance V
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 17
Basic Concept of ART (2)
bull ART 1 requires that the input vectors be binarybull ART 2 is suitable for processing analog or gray
scale patternsbull ART gets its name from the particular way in
which learning and recall interplay in the network
bull In physics resonance occurs when a small-amplitude vibration of the proper frequency causes a large-amplitude vibration in an electrical or mechanical system
2112005 18
recognition
comparison
G2
G1
input
gain control
gain control
+
+
+
+
+
+
+
-
-top-downweightsTji
bottom-upweightsBji
attentional subsystem orienting subsystem
F0
F1
F2
Basic Concept of ART (3) The ART network (Carpenter and Grossberg 1988)
2112005 19
41 Basic Concept of ART
bull Bji Forward the output from F1 to F2 for competition
bull Tji Forward the pattern of winner neuron to F1 for comparison
bull G1 To distinguish the feature of input pattern with stored patterns
bull G2 To reset the depressed neurons in F2 (ie reset losers)
bull attentional subsystem to rapidly classify the recognized patterns
bull orienting subsystem to help attentional subsystem learn new patterns
2112005 20
ART-1 Model
input vector
output vector
output layercluster
input layerinput variables
2112005 21
ART-1 Model (2)
bull Input layer input patterns or characteristic vectors Activation function f(x)=x inputs are binary values
bull Output layer representing the clustering of training patterns This is similar to SOFM except that SOFM has the neighborhood concept Initially there is only one output node The number of output nodes increases when learning proceeds When the stability is achieved the learning process stops
2112005 22
ART-1 Model (3)Algorithm
1 Set the network parameter Nout=1
2 Set the initial weighting matrices
3 Input the training vector X
4 Calculate the matching value
5 Find the max matching value in the output nodes
Niw
iw
b
t
1
1]1][[
1]1][[
0
][]][[][
count
i
b
I
iXjiwjnet
][max][ jnetjnetj
2112005 23
ART-1 Model (4) Algorithm (continue)
(6) Calculate the similarity
value
(7) Test the similarity value
If then go to step (8)
Otherwise go to step (9)
(8) Test whether there are output nodes applicable to the rule
If IcountltNout then try the second max matching value in the output nodes
||||
||||
][]][[||||
][||||
X
XwV
iXjiwXw
iXX
t
j
j
i
tt
j
i
)(vigilance V
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 18
recognition
comparison
G2
G1
input
gain control
gain control
+
+
+
+
+
+
+
-
-top-downweightsTji
bottom-upweightsBji
attentional subsystem orienting subsystem
F0
F1
F2
Basic Concept of ART (3) The ART network (Carpenter and Grossberg 1988)
2112005 19
41 Basic Concept of ART
bull Bji Forward the output from F1 to F2 for competition
bull Tji Forward the pattern of winner neuron to F1 for comparison
bull G1 To distinguish the feature of input pattern with stored patterns
bull G2 To reset the depressed neurons in F2 (ie reset losers)
bull attentional subsystem to rapidly classify the recognized patterns
bull orienting subsystem to help attentional subsystem learn new patterns
2112005 20
ART-1 Model
input vector
output vector
output layercluster
input layerinput variables
2112005 21
ART-1 Model (2)
bull Input layer input patterns or characteristic vectors Activation function f(x)=x inputs are binary values
bull Output layer representing the clustering of training patterns This is similar to SOFM except that SOFM has the neighborhood concept Initially there is only one output node The number of output nodes increases when learning proceeds When the stability is achieved the learning process stops
2112005 22
ART-1 Model (3)Algorithm
1 Set the network parameter Nout=1
2 Set the initial weighting matrices
3 Input the training vector X
4 Calculate the matching value
5 Find the max matching value in the output nodes
Niw
iw
b
t
1
1]1][[
1]1][[
0
][]][[][
count
i
b
I
iXjiwjnet
][max][ jnetjnetj
2112005 23
ART-1 Model (4) Algorithm (continue)
(6) Calculate the similarity
value
(7) Test the similarity value
If then go to step (8)
Otherwise go to step (9)
(8) Test whether there are output nodes applicable to the rule
If IcountltNout then try the second max matching value in the output nodes
||||
||||
][]][[||||
][||||
X
XwV
iXjiwXw
iXX
t
j
j
i
tt
j
i
)(vigilance V
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 19
41 Basic Concept of ART
bull Bji Forward the output from F1 to F2 for competition
bull Tji Forward the pattern of winner neuron to F1 for comparison
bull G1 To distinguish the feature of input pattern with stored patterns
bull G2 To reset the depressed neurons in F2 (ie reset losers)
bull attentional subsystem to rapidly classify the recognized patterns
bull orienting subsystem to help attentional subsystem learn new patterns
2112005 20
ART-1 Model
input vector
output vector
output layercluster
input layerinput variables
2112005 21
ART-1 Model (2)
bull Input layer input patterns or characteristic vectors Activation function f(x)=x inputs are binary values
bull Output layer representing the clustering of training patterns This is similar to SOFM except that SOFM has the neighborhood concept Initially there is only one output node The number of output nodes increases when learning proceeds When the stability is achieved the learning process stops
2112005 22
ART-1 Model (3)Algorithm
1 Set the network parameter Nout=1
2 Set the initial weighting matrices
3 Input the training vector X
4 Calculate the matching value
5 Find the max matching value in the output nodes
Niw
iw
b
t
1
1]1][[
1]1][[
0
][]][[][
count
i
b
I
iXjiwjnet
][max][ jnetjnetj
2112005 23
ART-1 Model (4) Algorithm (continue)
(6) Calculate the similarity
value
(7) Test the similarity value
If then go to step (8)
Otherwise go to step (9)
(8) Test whether there are output nodes applicable to the rule
If IcountltNout then try the second max matching value in the output nodes
||||
||||
][]][[||||
][||||
X
XwV
iXjiwXw
iXX
t
j
j
i
tt
j
i
)(vigilance V
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 20
ART-1 Model
input vector
output vector
output layercluster
input layerinput variables
2112005 21
ART-1 Model (2)
bull Input layer input patterns or characteristic vectors Activation function f(x)=x inputs are binary values
bull Output layer representing the clustering of training patterns This is similar to SOFM except that SOFM has the neighborhood concept Initially there is only one output node The number of output nodes increases when learning proceeds When the stability is achieved the learning process stops
2112005 22
ART-1 Model (3)Algorithm
1 Set the network parameter Nout=1
2 Set the initial weighting matrices
3 Input the training vector X
4 Calculate the matching value
5 Find the max matching value in the output nodes
Niw
iw
b
t
1
1]1][[
1]1][[
0
][]][[][
count
i
b
I
iXjiwjnet
][max][ jnetjnetj
2112005 23
ART-1 Model (4) Algorithm (continue)
(6) Calculate the similarity
value
(7) Test the similarity value
If then go to step (8)
Otherwise go to step (9)
(8) Test whether there are output nodes applicable to the rule
If IcountltNout then try the second max matching value in the output nodes
||||
||||
][]][[||||
][||||
X
XwV
iXjiwXw
iXX
t
j
j
i
tt
j
i
)(vigilance V
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 21
ART-1 Model (2)
bull Input layer input patterns or characteristic vectors Activation function f(x)=x inputs are binary values
bull Output layer representing the clustering of training patterns This is similar to SOFM except that SOFM has the neighborhood concept Initially there is only one output node The number of output nodes increases when learning proceeds When the stability is achieved the learning process stops
2112005 22
ART-1 Model (3)Algorithm
1 Set the network parameter Nout=1
2 Set the initial weighting matrices
3 Input the training vector X
4 Calculate the matching value
5 Find the max matching value in the output nodes
Niw
iw
b
t
1
1]1][[
1]1][[
0
][]][[][
count
i
b
I
iXjiwjnet
][max][ jnetjnetj
2112005 23
ART-1 Model (4) Algorithm (continue)
(6) Calculate the similarity
value
(7) Test the similarity value
If then go to step (8)
Otherwise go to step (9)
(8) Test whether there are output nodes applicable to the rule
If IcountltNout then try the second max matching value in the output nodes
||||
||||
][]][[||||
][||||
X
XwV
iXjiwXw
iXX
t
j
j
i
tt
j
i
)(vigilance V
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 22
ART-1 Model (3)Algorithm
1 Set the network parameter Nout=1
2 Set the initial weighting matrices
3 Input the training vector X
4 Calculate the matching value
5 Find the max matching value in the output nodes
Niw
iw
b
t
1
1]1][[
1]1][[
0
][]][[][
count
i
b
I
iXjiwjnet
][max][ jnetjnetj
2112005 23
ART-1 Model (4) Algorithm (continue)
(6) Calculate the similarity
value
(7) Test the similarity value
If then go to step (8)
Otherwise go to step (9)
(8) Test whether there are output nodes applicable to the rule
If IcountltNout then try the second max matching value in the output nodes
||||
||||
][]][[||||
][||||
X
XwV
iXjiwXw
iXX
t
j
j
i
tt
j
i
)(vigilance V
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 23
ART-1 Model (4) Algorithm (continue)
(6) Calculate the similarity
value
(7) Test the similarity value
If then go to step (8)
Otherwise go to step (9)
(8) Test whether there are output nodes applicable to the rule
If IcountltNout then try the second max matching value in the output nodes
||||
||||
][]][[||||
][||||
X
XwV
iXjiwXw
iXX
t
j
j
i
tt
j
i
)(vigilance V
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 24
ART-1 Model (5) Algorithm (continue)
Set Icount=Icount+1 net[j]=0 go to step (5)
otherwise
(a) generate new cluster
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
|w|05
x][i][Nw
x][i][N
matrix weightingnewset
1set
toutb
out
X
w
w
NN
t
outout
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 25
ART-1 Model (6)Algorithm (continue)
(9) Adjust the weighting matrix
(a) adjust the weights
(b) set the output values for output nodes
if j=j then Y[j]=1
else Y[j]=0
(c) go to step (3) (input new vector X)
][]][[05
][]][[w][i][jw
][]][[][i][j
tb
i
t
tt
iXjiw
iXji
iXjiww
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 26
ART-1 Model (7)
bull Given an input vector X=[1 1 1 1 1 0 0 0 0 0]
bull Assume 5 output nodes 3 cases for comparisons
bull Case 1
]0000000001[ ]0000000001[51
1
]0000000011[ ]0000000011[52
1
]0000000111[ ]0000000111[53
1
]0000001111[ ]0000001111[54
1
]0000011111[ ]0000011111[55
1
55
44
33
22
11
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 27
ART-1 Model (8)bull Node 1 matching value=555=0909 similarity
value=55=10bull Node 2 matching value=445=0888 similarity
value=45=08 bull Node 3 matching value=335=0857 similarity
value=35=06bull Node 4 matching value=225=08 similarity
value=25=04bull Node 5 matching value=115=0667 similarity
value=15=02bull The matching value is proportional to similarity value
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 28
ART-1 Model (9)
bull Case 2
bull Assume 6 output nodes
]1111100111[ ]1111100111[58
1
]0111100111[ ]0111100111[57
1
]0011100111[ ]0011100111[56
1
]0001100111[ ]0001100111[55
1
]0000100111[ ]0000100111[54
1
]0000000111[ ]0000000111[53
1
66
55
44
33
22
11
tb
tb
tb
tb
tb
tb
ww
ww
ww
ww
ww
ww
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 29
ART-1 Model (10)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=345=0666 similarity
value=35=06 bull Node 3 matching value=355=0545 similarity
value=35=06bull Node 4 matching value=365=0462 similarity
value=35=06bull Node 5 matching value=375=04 similarity
value=35=06bull Node 6 matching value=385=0353 similarity
value=35=06
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 30
ART-1 Model (11)
bull The same similarity value but different matching value
bull If the number of corresponding bits of output vectors to input vector are the same the one with less ones in output vector will be selected for vigilance test
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 31
ART-1 Model (12)
bull Case 3
bull Assume 3 output nodes
]0000111000[ ]0000111000[53
1
]0000011100[ ]0000011100[53
1
]0000001110[ ]0000001110[53
1
]0000000111[ ]0000000111[53
1
44
33
22
11
tb
tb
tb
tb
ww
ww
ww
ww
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 32
ART-1 Model (13)bull Node 1 matching value=335=0857 similarity
value=35=06bull Node 2 matching value=235=0571 similarity
value=25=04 bull Node 3 matching value=135=0286 similarity
value=15=02bull Node 4 matching value=035=00 similarity
value=05=00bull Although the number of 1rsquos in the output vector are the
same the matching value and similarity values are all different But the matching value is proportional to similarity value
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 33
Continuous-Valued ART (ART-2)
Proceduresbull Given a new training pattern a MINNET (min
net) is adopted to select the winner which yields the min distance
bull Vigilance test A neuron j passes the vigilance test if
bull where the vigilance value determines the radius of a cluster
bull If the winner fails the vigilance test a new neuron unit k is created with weight
|||| jwx
|||| jwx
xwk
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 34
Continuous-Valued ART (ART-2) (2)
bull If the winner passes the vigilance test adjust the weight of the winner j by
where ||clusteri|| denotes the number of members in cluster i
||||1
||||)(
)()(
old
j
old
j
old
jnew
j cluster
clusterwxw
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 35
Continuous-Valued ART (ART-2) (3)
bull Effect of different order of pattern presentationndash The ART is sensitive to the presenting order of
the input patternsbull Effect of vigilance thresholds
ndash In general the smaller the vigilance threshold the more clusters are generated
bull Effect of re-clusteringndash Use the current centroids as the initial reference
for clusteringndash Re-cluster one by one each of the training
patternsndash Repeat the entire process until there is no
change of clustering during one entire sweep
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 36
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
cluster 3
centroid
1 (1001) - - new cluster
(1001)
2 (1308) 1 10 pass test (115045)
3 (1418) 1 16 fail new cluster
(1418)
4 (1505) 1 04 pass test (127047)
5 (0014) 2 18 fail new cluster
(0014)
6 (0612) 3 08 pass test (0313)
7 (1519) 2 02 pass test (145185)
8 (0704) 1 063 pass test (113045)
9 (1914) 2 09 pass test (1617)
10 (1513) 2 05 pass test (15816)
(a) Original order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
2112005 37
order pattern
winner test value
decision cluster 1
centroid
cluster 2
centroid
1 (1513) - - new cluster
(1513)
2 (1914) 1 05 pass test (17135)
3 (0704) 1 195 fail new cluster
(0704)
4 (1519) 1 075 pass test (163153)
5 (0612) 2 09 pass test (06508)
6 (0014) 2 125 pass test (04310)
7 (1505) 1 117 pass test (16128)
8 (1418) 1 072 pass test (156138)
9 (1308) 1 084 pass test (152128)
10 (1001) 2 147 pass test (058078)
The execution sequence of the ART-2 with the vigilance threshold 15
(b) Reverse order
