IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING …vision.ai.illinois.edu/publications/ergulnn2016.pdf · Emrah Ergul, Naﬁz Arica, Member, IEEE, Narendra Ahuja, Fellow, IEEE,

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 1

Clustering Through Hybrid Network ArchitectureWith Support Vectors

Emrah Ergul, Nafiz Arica, Member, IEEE, Narendra Ahuja, Fellow, IEEE, and Sarp Erturk, Senior Member, IEEE

Abstract— In this paper, we propose a clustering algorithmbased on a two-phased neural network architecture. We combinethe strength of an autoencoderlike network for unsupervisedrepresentation learning with the discriminative power of asupport vector machine (SVM) network for fine-tuning the initialclusters. The first network is referred as prototype encodingnetwork, where the data reconstruction error is minimized inan unsupervised manner. The second phase, i.e., SVM network,endeavors to maximize the margin between cluster boundariesin a supervised way making use of the first output. Both thenetworks update the cluster centroids successively by establishinga topology preserving scheme like self-organizing map on thelatent space of each network. Cluster fine-tuning is accomplishedin a network structure by the alternate usage of the encoding partof both the networks. In the experiments, challenging data setsfrom two popular repositories with different patterns, dimen-sionality, and the number of clusters are used. The proposedhybrid architecture achieves comparatively better results bothvisually and analytically than the previous neural network-basedapproaches available in the literature.

Index Terms— Autoencoder (AE) network, clustering neuralnetworks, greedy layerwise learning, prototype encoding (PE)network, support vector machine (SVM).

I. INTRODUCTION

DATA clustering deals with the problem of grouping aset of unlabeled data into subsets, such that the samples

within the same subset are more similar to each other thanthe ones that belong to the other groups [1], [2]. The mainobjective is to find latent patterns reflected by the underlyingparameters, which maximize the likelihood of the input datato their assigned clusters. Clustering is widely used beforethe ultimate goal of classification or regression and can beimplemented for various purposes such as feature extrac-tion, image segmentation, dimension reduction, and functionapproximation.

Although clustering techniques in the literature are diverse,we can group them into hierarchical and partitional methods

Manuscript received December 29, 2014; revised December 30, 2015 andFebruary 21, 2016; accepted March 10, 2016. This work was supportedby the Scientific and Technological Research Council of Turkey underGrant 2214-B.14.2.TBT.0.06.01-214-83.

E. Ergul and S. Erturk are with the Electronics and CommunicationEngineering Department, Kocaeli University, Izmit 41380, Turkey (e-mail:[email protected]; [email protected]).

N. Arica is with the Computer Engineering Department, Bahçesehir Uni-versity, Istanbul 34353, Turkey (e-mail: [email protected]).

N. Ahuja is with the Computer Vision and Robotics Laboratory, BeckmanInstitute, University of Illinois at Urbana–Champaign, Champaign, IL 61801USA (e-mail: [email protected]).

Color versions of one or more of the figures in this paper are availableonline at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TNNLS.2016.2542059

based on the task-specific applications. Hierarchical clusteringmethods aim to build an agglomerating or divisive hierarchy ofpatterns. Partitional algorithms, on the other hand, determineindependent clusters at once. However, both the methods aresubject to a common goal of optimizing the parameters of thedata model. The parameters represent the hidden patterns ofeach cluster, and are basically the data statistics calculated bysome metric. Based on the type of statistics, distance measurescan be used for (dis)similarity computation in a feature space,or the probability is computed for density estimation of poste-rior distributions in the same manner. Thereafter, associationcriteria, which are called objective functions, are employed tofind the optimum solution to the clustering problem.

Among graph- and tree-based structures, neural networkshave justifiable popularity because they are flexible to sim-ulate any learning algorithm for hierarchical or partitionalclustering. Neural networks can be explained as the multilayerstructures that are formed by the composition of successivenonlinear transformations of the input data. They aim toimplicitly achieve intermediate feature representations of theoriginal data in deeper layers. Although neural networksare initially designed for supervised learning, recent researchreveals their strength in the unsupervised learning. Autoen-coder (AE) and restricted Boltzmann machines can be referredas the building blocks of unsupervised representation learningin neural networks. The details about such architectures, theirvariants, and implementations are available in [3]–[7].

In the literature, neural network-based clustering meth-ods mainly build a two-layer network with lateral inhibitiveconnections in the output layer. They typically use theprototype/centroid vectors directly as the weight parametersbetween the input and output layers, since each neuron inthe output layer is assumed to be a cluster. Inhibition in theoutput layer simply determines the characteristics of the clusterupdating procedure, where for each input, only the closestcentroid or all the centroids in a receptive field are updated ina weighting scheme. In general, gradient-based algorithms areused to minimize the cost of assigning data samples to theirpredicted prototypes iteratively.

In this paper, we propose a hybrid network architecturefor partitional clustering that combines two complementarynetworks with different objective functions to optimize thesolution. The proposed architecture embodies both the unsu-pervised representation learning for the initial clustering andthe discrimination power of the support vector machine (SVM)for cluster fine-tuning. In the first network, we introduce thenovel concept of a prototype encoding (PE) network that

2162-237X © 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.


2 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS

attempts to map each input sample to its closest prototypevector in an unsupervised manner. The encoding weightsof the PE network are initialized as orthogonal vectors toVoronoi-like hyperplanes that separate the clusters in a pair-wise manner. To update the clusters, we utilize a topologypreserving grid structure with a neighboring function definedin a latent feature space. The second network with two layersis based on the SVM that basically tries to map the inputsamples to their currently assigned cluster labels. Each neuronin the output layer represents a cluster, and the networkparameters are optimized using a one-vs-all scheme so as tomaximize the margins between clusters. The SVM networkuses the current prototype vectors as landmarks in the kernelfunction.

In the last part of our work, a hybrid network structureis established in a greedy layerwise clustering. Basically,we have two by-products of the first clustering network,i.e., PE. The hidden layer activations represent the data ina new and probably more discriminative feature space, andthey are used as input to the SVM network. Furthermore,the cluster assignments of the PE network are utilized inthe SVM network as a priori knowledge. To conclude, theSVM network fine-tunes the previously established clustersin a supervised manner, and this greedy layerwise schemeenhances the clustering performance remarkably.

Contributions: The contributions of this paper aretwofolded. First, we introduce a PE network scheme for dataclustering. The PE maps the input data to the prototypevectors in the output layer and updates the clusters in a latentspace, established at the hidden layer. PE is clearly differentfrom AE [3]–[5], where the input is associated with itself inthe output layer. AE approaches may increase the variance inthe output layer, and identity functions can be learned when thenumber of neurons in the hidden layer is more than the inputdimensionality. In addition, we initialize the encoding weightsof the PE network in a Voronoi-like manner, which separatesthe clusters in a pairwise manner. This initialization not onlyachieves a better starting point for the encoding weights thanrandom selection for PE network but also it dynamicallycomputes the number of hidden neurons with respect to therandomly initialized prototypes. Second, a greedy layerwiseclustering scheme is proposed where an SVM network is usedfor fine-tuning. After optimizing the PE network, the inputdata in a new feature space are passed through the SVMnetwork where the cluster assignments of PE are used as theinitial supervision. Finally, only the current centroid vectorsare utilized as the landmarks in the kernel function to reducethe complexity of traditional SVM learning.

The rest of this paper is outlined as follows. In Section II,we summarize the related studies based on clustering networkarchitectures. Section III gives the details of the PE and SVMnetworks, and explains the method they are combined forthe greedy layerwise clustering in a hybrid topology. Theexperimental results performed on the public data sets aregiven both visually and numerically in Section IV. Finally,we conclude this paper with the clear findings of the experi-ments and possible future work about the clustering networkalgorithms.

II. RELATED WORK

The initial work on clustering neural networks starts witha simple competitive learning (SCL) network [2], [8], [9].SCL is based on a two-layer fully connected network thatimplements the nearest neighbor paradigm. Basically, the

input data, X = {x(i)|i = 1, 2, 3, . . . , N} where x(i) ε Rd

are approximated by a finite number of prototypes and

C = {c(j)| j = 1, 2, 3, . . . , K } where K � N and c(j) ε Rd ,in the output layer. The SCL objective is to minimize the meansquared error (MSE)

E = 1

N

N∑

i=1

K∑

j=1

1

2μi j ‖x (i) − c( j )‖2; μi j ∈ {0, 1} (1)

where μ j i is the Kronecker delta that takes the value of 1

when c(j) is the closest (i.e., winning) prototype to thedata sample x(i) in the L2-norm metric and 0 otherwise.This nearest prototype condition is called the winner-takes-all (WTA) process, where Kronecker delta prevents updat-ing the other prototypes and only the winning prototypec(w) is updated by each input x(i). The weights betweeninput–output layers represent the prototype vectors C , andthe objective function is optimized by the gradient descentmethod. Thus, the SCL network updates the prototypesiteratively as

∀x; c(w) = c(argmin j=1,2,3,...,K ‖x (i)−c( j)‖) (2)

∂ E

∂c(w)= −[x (i) − c(w)(t)] (3)

c(w)(t + 1) = c(w)(t) + η(t)[x (i) − c(w)(t)] (4)

c( j )(t + 1) = c( j )(t); j �= w (5)

where η(t) is a positively small learning rate, usually chosento be decreasing gradually in time. The SCL network maybe run either in the batch mode where the whole trainingdata set is available or in the incremental mode for theonline data. The number of clusters is predefined and clusterprototypes are randomly initialized as the network weightvectors. One of the drawbacks of the SCL network is thatit is highly dependent on the randomly initialized clusterprototypes, since it updates only the winning one each time.The data sequence also impacts the updates in incrementallearning.

Another popular clustering network is the Kohonen net-work, also called self-organizing map (SOM). The SOM hasmainly the same network structure and objective as in theSCL [10]–[12]. However, unlike the SCL network where onlythe winning prototype is updated by each data sample, SOMintroduces an excitation or neighborhood function that makesall the prototype vectors in a field centered at the winning pro-totype to be updated. First, an imaginary surface of a specificgrid configuration is randomly initialized as the weight vectorsC between input–output layers [24]. The grid configurationreflects static neighboring bindings between neurons at theoutput layer by a predefined metric such as Euclidean distance.Once the winning prototype of a sample c(w) is selected for


ERGUL et al.: CLUSTERING THROUGH HYBRID NETWORK ARCHITECTURE WITH SUPPORT VECTORS 3

the input sample, the update rule for all the prototype vectorsin the neighborhood is

c( j )(t + 1) = c( j )(t) + η(t)h jw(t)[x (i) − c( j )(t)] (6)

h jw(t) = h0e−‖c( j) − c(w)‖2

σ2(t) ∀x (7)

σ(t) = σ0e−t

T; σ0 ∈ (0, 1] (8)

where h jw(t) is the neighborhood Gaussian function, whichdecreases as the distance from the winning prototype c(w)

increases. σ(t) is the scale factor of the Gaussian that deter-mines the excitation magnitude while η(t) is the learning rate,decreasing in time, and h0 is the initial neighboring constantin the range (0, 1]. The cluster update procedure in the SOMis not based on the minimization of an objective function.Therefore, it suffers from unguaranteed convergence, and it isoften dependent on the data sequence.

Given the aforementioned disadvantages of SOM and SCL,a learning vector quantization (LVQ) network is proposedin [12] to fine-tune the unsupervised clustering networks withsupervision. It is based on a supervised learning in that thepreviously found prototype vectors and their data assignmentsare assumed to reflect the priori density function of the inputdata for the LVQ network. It minimizes (1) as

c(w)(t + 1) = c(w)(t) − (−1)y(i)w η(t)[x (i) − c(w)(t)]

y(i)w ∈ {0, 1} (9)

c( j )(t + 1) = c( j )(t); j �= w (10)

where the input data are now given by the pattern pairs ofP = {(x(i), y(i))}, x(i) ε Rd , and y(i) ε {0, 1}K . The binarylabels y(i) indicate the initial cluster assignments and theyare fixed throughout the clustering procedure. Here, η(t) isagain the scalar learning rate (i.e., 0 < η(t) < 1), which isdecreasing in time. When we look in (9), LVQ simply resem-bles the SCL network in which only the winning prototypec(w) is updated by each sample. But the minus sign in (9)the first equation is set to correct the winning prototype inthe opposite direction if the sample x(i) is clustered differentlyfrom the prior assumption. This helps to decrease the densityaround the prototypes on the Bayesian decision surfaces butit also triggers vector divergence [2].

For comparison, LVQ and our proposed SVM networkare both supervised and used for fine-tuning. On the otherhand, they mainly differ from each other in which the SVMnetwork continues to update the cluster assignments in a latentspace iteratively and it uses a max-margin objective function.Thus, the SVM network fine-tuning improves the clusteringperformance notably.

So far, the SCL and LVQ networks are introduced asWTA clustering algorithms where only the winning prototypesare updated in the unsupervised and supervised manners,respectively. SOM, on the other hand, excites all the prototypevectors for each sample with a neighboring function thatupdates the vicinity of the winning prototype. Although such aneighborhood updating scheme holds inspiration, SOM prede-fines static neighborhood relations between cluster prototypes.For this perspective, neural gas network (NGN) is proposed

in [13]. The soft-max rule is employed as an extension to thestandard k-means like clustering used in SOM. The earlierworks also use the maximum entropy algorithm [14] in theupdating step. NGN takes a dynamic neighborhood rankinginto account among the prototype vectors with respect to eachsample.

For the data sample x(i), the neighboring order is establishedby using the Euclidean distance to all the prototype vectors C .Then, each prototype c(j) is assigned to an integer rank value,r j i = 0, 1, 2, . . . , K − 1, where K is the predefined numberof prototypes and 0 indicates the winning prototype c(j). Thegradient descent-based clustering update rule is conducted as

c( j )(t + 1) = c( j )(t) + η(t)h(r j i , t)[x (i) − c( j )(t)] (11)

h(r j i , t) = e−r j iρ(t) (12)

η(t) = η0

(η f

η0

) tT

, ρ(t) = ρ0

(ρ f

ρ0

) tT

(13)

where h(r, t) is the ordering factor that takes rank r j i andthe characteristic decay constant ρ(t) is used to update everyprototype c(j) with respect to a specific sample x(i). Like η(t),the decay constant ρ(t) decreases gradually in time, in whichη0 and ρ0 are the initial decay parameters, and η f and ρ f arethe final decay parameters, respectively. NGN resembles theSOM network except the neighboring function where NGNuses integer ranks for the prototypes with respect to eachsample instead of utilizing the distances between prototypevectors. By using this kind of dynamic ordering r j i and theclustering shape modulator ρ(t), NGN is assumed to reach alower distortion error and be more robust to local minima thanthe others, without the problem of underutilization [2].

Other popular clustering networks in the literature are adap-tive resonance theory (ART) [15] and fuzzy clustering (FC)networks [16]. The idea behind the ART network is to create atwo-layer recurrent structure where the input layer F1, and theoutput layer F2 are fully connected in both directions with dif-ferent weights. The feedforward weights w(i) represent short-term memory that is used to find the winning prototype c(w)

in the output layer. The feedback weights c(j) introduce long-term memory that actually indicates the prototype vector C .Although the ART network tends to get robust clusters inrapidly changing input sequences, the input signal and theweights are binary, it is sensitive to the feeding order of theinput patterns, and its complexity is high due to differentialequations. The FC network, on the other hand, uses the samestructure of the SCL network. But it partitions the data setin a fuzzy manner, meaning that each data sample belongs tomultiple clusters with a membership function, μ(x(i) and c(j)).Because the memberships of a sample for all the clusterscorrespond to the probabilities in the fuzzy set theory, theyare restricted to the sum of unity. An alternating optimizationprocedure is established by first using the soft-max rule forupdating memberships, thereafter taking fuzzy means of thedata samples for updating the prototype vectors [2]. Thememberships and cluster prototypes are initialized randomly.FC is not suitable for large-scale data sets because of highcomputational and storage requirements.



Fig. 1. Proposed hybrid neural network architecture for clustering.

With regard to the SVM-based approaches in clusteringproblems, there are two main concepts in the literature. Theapproaches in the first category basically focus on adoptingtheir max-margin objective function to the one that helpsfinding spherical regions in the original feature space asclusters [25], [26]. The Lagrangian method is introduced intothe objective function, which computes support vectors, rep-resenting hypothetical boundaries. Thereafter, the connectedcomponents are extracted using graph based algorithms forcluster labeling in an unsupervised manner. The approachesin the second category use SVM as a complementary toolto the main clustering algorithm. For example, in [27],k-means is implemented to discover the topical clusters oftextual documents in the upper level. Meanwhile, an SVMclassifier is constructed with the help of k-means supervisionfor each cluster by using other interrelated data sets. As theassignments change, SVM classifiers are updated in addition tothe clusters. Nevertheless, our proposed SVM clustering net-work is completely different from the aforementioned methodsin that it has a network structure that updates the clusters ina latent space, and it is used as a cluster fine-tuning schemewith its simple max-margin objective function.

III. HYBRID NETWORK ARCHITECTURE FOR CLUSTERING

In this paper, we propose a hybrid network for clusteringthat embodies unsupervised representation learning in a newfeature space and uses the power of SVM discriminantsin an additional layer for cluster fine-tuning. The proposedarchitecture is shown in Fig. 1 and is composed of two parts.In the first part, we construct a three-layer network that mapsthe input data to their closest prototype vector. Encodingweights of the network are initialized by Voronoi-like hyper-planes that separate the randomly selected prototypes in apairwise manner. This process also determines the number ofneurons in the hidden layer implicitly. In addition, the hiddenlayer activations of each sample are restricted to its closestprototype activations in the training stage. The samples areassigned to their closest cluster in the latent feature space andnot in the original domain. We call this PE network, and thedetails are given in Section III-A.

The second part is designed for fine-tuning the clustersfound in a previous clustering method, such as a PE network.It gets the final assignments of the PE network as the initial

cluster labels and the hidden layer activations of the data tobe clustered as the new input signals. This network employsthe concept of SVM. It takes the current prototype vectors aslandmarks in its kernel function, and maximizes the marginsbetween the cluster boundaries. This second part, calledthe SVM network in Section III-B, improves the separationbetween clusters. In both the networks, we use a SOM-likeapproach to iteratively update the clusters in the latent spaceswhere the hidden layer and the output layer are used for the PEnetwork and the SVM network, respectively. Finally, a greedylayerwise algorithm is applied for optimizing the parametersof both the networks as a whole in Section III-C.

A. Prototype Encoding Network

PE is based on an idea similar to that of theAE network [3], [5], [17], where the input signal is simplymapped to itself in the output layer by the latent patterns in ahidden layer. However, unlike AE, the PE network is forced toiteratively couple each input sample to its closest prototype,instead of itself. Each cluster is represented by a prototypevector that is initialized randomly. The Euclidean metric isused for cluster assignments in a nearest neighbor fashion asin k-means [18].

Given the data sample-cluster pairs X = {(x(i), c(j))};x(i), c(j) ε Rd , i = 1, 2, 3, . . . , N ; j = 1, 2, 3, . . . , K , whereK is a user-defined parameter defining the number of clustersand N is the number of samples in the data set; the objectivefunction is

∀x; c( j ) = c(argmink=1,2,3,...,K ‖x (i)−c(k)‖) (14)

JX (W, b) = 1

N

N∑

i=1

1

2‖hW,b(x (i)) − c( j )‖2

+ β

V∑

v=1

(hv

c( j) loghv

c( j)

hvx (i)

+ (1− hv

c( j)

)log

1− hvc( j)

1− hvx (i)

)

+ λ

2

∑

W1,W2

‖W‖2. (15)

The objective function Jx (W, b) involves three terms: MSE,divergence, and weight regularization. MSE penalizes thedifferences between the predictions of the network hW,b(x)

and the desired output signals (i.e., the closest prototypes) c(j).The divergence term forces both the input data and theirclosest prototypes to be similar in the latent space, and thelatent space is now represented by the hidden-layer activations.This is achieved by restricting the hidden-layer activations hv

with an entropy-based formula, i.e., Kullback–Leibler (KL)divergence, KL(hc||hx). Because we try to assign each sampleto a cluster in this latent space instead of the original domain,it is a good intuition to add an approximation term to theobjective function for adjusting the basis vectors of the targetfeature space. Weight regularization term, on the other side,tries to avoid over-fitting. β and λ are the tradeoff constantsin KL divergence and the weight regularization, respectively.

Another important point in the PE network is the initializa-tion of encoding weights (i.e., W1 in Fig. 2). We initialize theencoding weights using the Voronoi approach, which mainly



Fig. 2. PE network.

partition the feature space into a finite number of regionsby the hyperplanes [2], [13]. Voronoi hyperplanes are estab-lished by taking the cluster centers as seed points. Afterselecting the prototypes randomly, each hyperplane is foundby a vector orthogonal to the connecting plane between apair of prototypes, and it intersects at the center point onthis plane. However, such combinational hyperplanes lead toincrease the computational complexity exponentially, sincethey represent the neurons in the hidden layer, many of whichmay be redundant. Therefore, we first sort the remainingprototype points in the increasing order of their Euclidiandistances for each selected prototype. A hyperplane is thenproduced between the selected prototype and its closest onein a couplewise fashion. Since we already accept the com-puted hyperplane for the selected prototype with a minimummargin, those prototypes that are separated from the selectedprototype vector by this hyperplane with even larger margincan be intuitively eliminated from the pairwise hyperplaneconstruction. This procedure continues iteratively until there isno coupled prototype. This greedy method reduces the numberof hyperplanes dramatically. Note that we not only initializethe encoding weights rationally but also determine the numberof neurons in the hidden layer dynamically with respect to therelative positions of the randomly selected prototypes. Thedecoding weights W2 are randomly selected like the prototypevectors. The sigmoid function is used both in the hidden andoutput layers as a nonlinear activation function.

The optimization of network parameters is performed by thestochastic gradient descent algorithm. Partial error derivationsare calculated as in the traditional feedforward–back propa-gation algorithm. Hence, the weight and bias parameters areupdated in mini batches while keeping the prototype vectors,input data-cluster assignments, and the hidden layer restric-tions fixed. After each passes through the whole data set to beclustered, the prototypes are updated. We use (six to eight) ofthe SOM network to update the prototype vectors where thewinning prototype vector for each sample c(w) and the factorparameter of the neighboring Gaussian function h jw(t) arecomputed in the latent space. The prototype update procedure,on the other hand, is still conducted in the original domain,since we use the same data in the input layer iteratively.Also note that updating the prototype vectors will also affect

Fig. 3. SVM network for fine-tuning.

the data-cluster assignments and the hidden-layer restrictionsfor the next epoch. The intuition behind this procedure is thatencoding weights represent hyperplanes, which are estimatedas the cluster boundaries and the hidden layer activations foreach sample are the distances from those hyperplanes to eitherpositive or negative side. Because the samples are representedin a higher dimensional space by hidden-layer activations, theycan be discriminated better. The update procedure continuesiteratively until no further cluster changes are achieved or wereach an iteration bound.

B. SVM Network for Cluster Fine-Tuning

SVM is a powerful tool for supervised learning, whichseparates the feature space linearly into two categories: pos-itive and negative. It tries to maximize the margin betweenthe positive and negative sides [18], [20]. The margin in theSVM represents the gap between support vectors of both thesides, which are data samples acceptably close in limits tothe opposite ones. The main idea is to find the optimumhyperplane that achieves the total minimum distance betweenthe support vectors and the hyperplane. After training theSVM, a new sample is classified simply as either positive ornegative by the result of the dot product with the optimizedweight vector.

In fact, SVM is a linear discriminant function and it obvi-ously does not handle nonlinearly separable data sets with asatisfactory accuracy. Kernel functions (Gaussian, polynomial,chi-square, histogram intersection, and so on) are introducedin the literature to establish the nonlinear SVM classifiers andthey achieve a justifiable popularity with SVM. Actually, SVMstill preserves its linearity in this perspective, but the input dataare transferred into a new feature space by nonlinear kernelfunctions beforehand. Hence, we get more complex featurespaces with higher dimensionality instead of complicating thediscriminant function itself.

In the second part of the introduced hybrid architecture,we propose a cluster fine-tuning concept that is based on theone-vs-all SVM topology for multiple clusters. The two-layersupervised clustering network is shown in Fig. 3. Given theinput data X = {x(i)} and the cluster prototypes C = {c(j)},input signals for the SVM network are first achieved by theGaussian kernel function (GKF) K (x(i), C)

K (x (i), c) = e−‖x (i) − c‖2

2σ 2 ; c ∈ C (16)



where σ is the scale parameter that factors the neighborhood.Therefore, each data sample is now represented by its GKFresponses to the current prototypes (i.e., landmarks). Also notethat the input dimensionality now equals to the number ofclusters K . Because the SVM is a supervised learning algo-rithm, the sample cluster assignments obtained in a previousapplication are used as the initial data labels in the SVM layerfor supervision. The unconstrained objective function of theSVM network is

JG X ,y(W, b) = P1

N

N∑

i=1

max(1 − wT Gx (i) y(i), 0)2

+ 1

2

∑‖W‖2 y(i) ∈ {−1, 1}K (17)

where P is the tradeoff constant, penalizing data points thatviolate the margin requirements. Gx represents the GKF outputvector of (16) for each sample x(i), which is the new inputsignal to the SVM network. W is the matrix, which embedsthe parameter vectors, including biases. They are assumedto be orthogonal to the hyperplanes that separate K clustersand randomly initialized like the decoding weights of thePE network. As aforementioned, SVM simply separates thespace into two parts. To extend the SVM network for amulticluster problem, we use the one-vs-all approach. Forthe K -cluster problem, K linear SVMs are independentlytrained, where the data samples from the other clusters formthe negative cases [20]. Therefore, the desired output signalfor each input y(i) is achieved by assigning 1 to the assignedcluster and −1 for the rest. The stochastic gradient descentalgorithm is then employed as

hW (Gx) = W T Gx (18)

δw = δ J (W, b)

δhW= −2Pymax(1 − hW (Gx )y, 0) (19)

w = α

(1

N

∑(GT

x δw) + w

)(20)

wnew = wold − w (21)

where δw is the back-propagated derivation of the error signalper data sample and w is the average weight correction thatincludes L2 regularization without bias terms. Also note thatα is the learning rate and hw is the hypothesis function of thenetwork. The prediction procedure used by the SVM networkis similar to that of the soft-max classifier. But unlike soft-max, where the derivative error is handled in a combined formby matrix multiplication, SVM neurons propagate the errorback independently. This leads to computing the derivativeerrors by weight vectors w(j) instead of using the wholematrix W.

Once the SVM network is set up, we optimize the weightparameters iteratively in mini batches. The hyperplanes areupdated with the max-margin objective function to separatethe samples of each cluster based on the current sample-cluster assignments. After each epoch, we follow the clusterupdating procedure similar to that of the PE network. Hence,the output layer activations of the SVM network are nowused as the new latent space. Then the SOM-like updating

Fig. 4. Pseudocode of the greedy layerwise clustering network.

scheme of (six to eight) for both the prototype vectors andthe assignments are achieved in this latent space. Once wechange the data sample-cluster assignments, the desired outputsignals y(i) are redefined for the next epoch accordingly.Also note that the kernel outputs are recomputed, since wehave updated the prototype vectors as the new landmarks.Therefore, the input signal to the SVM network changes ineach epoch although we still input the same data sequence.The termination conditions are the same as the PE network.

C. Combining PE and SVM Networksin Hybrid Architecture

So far, two networks are independently presented for par-titional (i.e., competitive) clustering and they have differentsubobjectives. The lateral purpose of the PE network is to learna latent space in an unsupervised manner, which introducesmore discriminative data representations for clustering. TheSVM network, on the other hand, aims to fine-tune the priorclusters by maximizing the margins between these clusters.It also uses output layer activations as the new latent spacefor the cluster update procedure. Note that the number ofneurons in each layer of both the networks is determinedimplicitly. We combine these two networks in a greedylayerwise clustering fashion, and the algorithm is pseudocodedin Fig. 4.

We first extract Voronoi-like initialized patterns in theencoding part of the PE network that leads to unsupervisedfeature learning in the hidden layer while we update the clus-ters iteratively. This process can be viewed as the expectation-maximization algorithm, where the network parameters arefirst updated when the prototype vectors are kept fixed duringthe expectation step. Thereafter, the clusters are updated inthe latent space during the maximization step with the fixedencoding weights, representing the basis vectors of this newfeature space.

In the SVM network part, instead of using the input datain its original domain, the hidden layer activations of thePE network with respect to both the input data x(i) and the



TABLE I

SUMMARY OF DATA SETS USED IN THE EXPERIMENTS

current prototype vectors c(j) are computed as

H (ϕ) = Sigm(W T

1 ϕ) = 1

1 + e−W T1 ϕ

(22)

Hx = {h(x (i))= hx (i) |i = 1, 2, 3, . . . , N; hx ∈ Rv} (23)

Hc = {h(c( j ))= hc( j) | j = 1, 2, 3, . . . , K ; hc ∈ Rv }.(24)

The final data-cluster assignments of the PE network arealso used in the SVM network for the initial supervision.In addition, the active input data for the SVM network arecomputed by the GKF that handles current prototype vectorsas the landmarks. Afterward, the cluster fine-tuning procedureis implemented with the max-margin objective. Once prototypevectors and data-cluster assignments are updated with theone-vs-all SVM network in its latent space for each epoch,the new assignments are then fed to the output layer of theSVM network. Hc is recomputed in the encoder part of thePE network, since we update the prototype vectors. Note thatwe do not change the parameters of the PE network becausewe have finalized learning it already. This procedure continuesuntil equilibrium is achieved with a given error threshold orthe iteration bound is reached.

IV. EXPERIMENTAL RESULTS

We compare the clustering results of the proposed networkarchitectures with the widely used clustering networks: SCL,LVQ, SOM, and NG over two types of performance: visualand analytical. To do so, we use nine different data sets thatare summarized at Table I. Note that they have varying numberof data samples, clusters, and dimensions in wide ranges, anddifferent repositories (i.e., UCI machine learning, and speechand image processing) are also explored. Therefore, we canachieve an idea of how the algorithms behave comparativelyin such diverse inputs throughout the experiments. Basically,we use the synthetic data sets to enable the visualization of theclustering results, while the others are handled for analyticalperformance comparisons, which are among the popular datasets in the UCI machine learning repository.

It is a common practice to perform several preprocessingsteps on the raw data before clustering. In this paper, weapply the whitening method of [3] to get uncorrelated featureswith the same variance, and the input is normalized intothe range of (0, 1] at the preprocessing stage. The cluster-ing network algorithms to be compared with our proposedworks are implemented in accordance with their originalpapers [2], [11]–[13] and the formulas in Section III. Since all

Fig. 5. Clustering results of the WTA networks on S1-set.

the methods have different number of parameters to be set,we apply grid search in varying ranges to find the optimumconfigurations on a held-out validation set, and they are keptfixed during experiments. Except LVQ and SVM networks,which are supervised methods, the clustering algorithms startwith the same cluster prototypes that are randomly selected.Meanwhile, we use SCL results for the LVQ network initial-ization, while the updated clusters of the PE network are fedinto the SVM fine-tuning network as proposed.

We repeat the experiments 50 times, and the mean and thestandard deviations in the numerical results are also noted foreach data set to justify the visual results in consistency. Theywill be covered in Section IV-B. With regards to visual results,we select the common and significant clustering performancesamong these trials to display the differences between cluster-ing algorithms. For Tables II–VII, only the best results aredisplayed in bold font for a better comparison.

A. Visual Clustering Analysis

The aggregation and S (S1 − S4) data sets, which we usein the visual experiments, have individual challenges, sincethe cluster characteristics (i.e., shape, size, relative closeness,and scatter) vary in 2-D spatial distribution. Aggregation set ismore compact than S sets, such that the clusters in the visualdomain can be easily selected.

The same prototype vectors that we randomly initializeare used in all the algorithms at the start, and the finalresults are achieved after 100 iterations. We mainly groupthe clustering networks into the WTA and the neighborhoodfunction-based (NFB) methods of which the visual results areshown in Figs. 5 and 6, respectively.

In Fig. 5, we first give Voronoi tessellation of the randomlyselected prototypes to have a clear idea about initialization.



Fig. 6. Clustering results of the NFB networks on S1 set.

K -means results are also displayed, as it is a popular WTAclustering algorithm. Overall, the WTA networks performmuch worse than the NFB clustering methods, because theyonly update the winning prototypes each time. Besides, theWTA networks are highly dependent on the initial prototypevectors, and relative positions (i.e., spatial configuration) arepreserved through the iterations. They generally converge tothe final clusters faster than the NFB algorithms with a greaterror rate, since the sample-cluster assignments do not changeafter a low iteration level. SCL and LVQ networks, on theother hand, achieve better performances than the classicalk-means. The incremental cluster updates with a learningrate and enhances the results, instead of simply taking theaverage among cluster members. Because the LVQ networkis a supervised algorithm, we use the clustering results ofthe SCL network as a prior for supervision with the purposeof fine-tuning. However, no obvious performance increase isobserved when compared with the SCL results. The possiblereason is that the SCL assignments y(i) are kept fixed duringthe LVQ network iterations and it is simply forced to simulatethe SCL behaviors in the cluster update procedure.

In the NFB networks, we visually achieve more meaningfulresults due to the neighboring functions when we comparethem with the WTA-based methods, as shown in Fig. 6.In general, each sample updates all the prototype vectorsproportional to their neighborhood degree. This helps to usethe feature space more reasonably and to avoid the depen-denc on the initial prototypes. In addition, we implement thegradient descent method to update the clusters, which maylead to spatial configuration changes. SOM, NG, and our firstproposed clustering method, PE, have similar visual results.Nevertheless, the SVM fine-tuning network that is supervisedby PE (i.e., PE + SVM network in Fig. 6) boosts the clustering

Fig. 7. Clustering results with noisy initialization on S1 set.

Fig. 8. Clustering results with noisy initialization on aggregation-set.

performance remarkably. Unlike the LVQ, the SVM algorithmupdates the clusters in a latent space iteratively, and theinitial assignments are dynamically changed. Note that boththe PE and SVM networks try to optimize their own weightparameters in their latent spaces during the cluster updateprocedure.

Next, for the visual analysis, we assume that the clusteringresults of the PE network, which is sequentially fed into theSVM network, are not satisfactory. Therefore, we try to findexperimentally to what extent the SVM network may compen-sate for this initially unsatisfactory clustering. To address thisissue, we first add Gaussian noise to the data samples in theinput layer of the PE network. Second, we initialize clustercentroids for the PE network, such that they are deployed inthe outlier regions in the features space. The clustering resultsare shown in Figs. 7 and 8.



In Fig. 7, we initialize the centroids in two distinct regionson the S1 set. As expected, the PE network outperformsthe k-means algorithm and it captures almost all of theclusters. On the other hand, although the PE + SVM net-work mainly follows the same routine of the PE network,it performs better in that cluster #10 and 12 (i.e., in thebottom-right frame) are successfully discriminated through thePE + SVM network.

We repeat the experiments on aggregation-set with a differ-ent configuration where the centroids are initialized in a smalloutlier region in Fig. 8 (bottom right). Surprisingly, in thePE + SVM network, we see that two clusters (#3 and 4 ofthe PE network) are combined in cluster #3 in the PE + SVMnetwork as it is the ground-truth condition, and one out ofseven centroids is implicitly eliminated. We conclude that thePE + SVM network is not strictly sensitive to the PE networkclustering results where it mostly changes the configurationof the PE network. Furthermore, it may even sacrifice somecentroids for better clustering results in the iterative clusterupdating procedure.

Finally, it would be informative to explore the nature oftransformations that have been learned through the proposedapproach by visualizing the hidden-layer patterns. In addition,the improved clustering results can be further investigatedas we adopt different number of hidden layers and neuralnodes in each layer. To make things more clear, we adopt twoPE networks successively such that once the first PE networkis learned with the data samples in the original feature space,its hidden layer activations will be, then, the new input to thenext one. Note that the number of nodes in the hidden layersis also changed implicitly, because the dimensionality of theinput data differs.

In detail, S1 set and randomly initialized prototypes areused for comparison with the previous results. Once the firstPE network parameters are optimized, we project both theinput data and the random prototypes into a new (i.e., higherdimensional) feature space by using the encoder part of thenetwork. Thereafter, we implement the second PE network inthe new feature space with its own setup. We visualize thehidden layer outputs of both the PE networks in Fig. 9, whichrepresent the transformed features in terms of their first twoprincipal components derived from the principal componentanalysis.

As usual, the randomly selected prototypes and the inputdata are plotted with their initial assignments in the originalfeature space. We select this initialization specifically, suchthat cluster #3 has no member at the beginning, and the initialprototypes are gathered on some locations spatially. After thefirst PE implementation, we observe that the original featurespace is transformed in a way where the between-clusterseparations are increased. This also holds for the secondPE network that achieves a better result (i.e., cluster #2 isexploited additionally, while clusters #4 and 14 of the first PEnetwork are unified in a single cluster #3 in the second PEnetwork) by projecting the data samples into another latentspace. It is seen that the proposed approach obtains morediscriminative feature spaces in its hidden layers, which helpto get improved clustering performance.

Fig. 9. Clustering in the latent spaces of successive PE networks on S1 set.

Fig. 10. Iterative error trend plotting on white-wine preference data set.

B. Analytical Performance Analysis

We first analyze the clustering results graphically basedon the MSE trends of clustering networks on the white-wine preference data set in Fig. 10. Thereafter, MSE andtime analysis results of all the data sets are given atTables II and III for compactness. Also note that the sameprototypes with random initializations are used for all thealgorithms, and the iteration bound is set to 100 in theexperiments. Basically, in Fig. 10, the k-means algorithmis the fastest converging method with the worst error rate.Although the LVQ that is supervised by the SCL network hasmuch lower errors at the first iterations, the error rate slightlydecreases in the final iterations. Apart from the WTA networks,the SOM, NG, and PE networks achieve similar perfor-mances because of their neighborhood functions. The proposedSVM fine-tuning network, on the other hand, gets the best



TABLE II

MSE AND TIME ANALYSIS ON SYNTHETIC DATA SETS

TABLE III

MSE AND TIME ANALYSIS ON UCI DATA SETS

result. This is regarded to be due to the PE supervision andthe cluster updates in the latent space during iterations.

Next, we display the MSE (in Euclidean metric) and the tim-ing results (in seconds) for all the data sets at Tables II and III.Because we repeat the experiments 50 times, standard devi-ations and mean values are computed. Overall, we observethat the PE + SVM network achieves the best performancein all the situations. On the other hand, the hybrid algorithmseems to take more time, almost doubling the nearest onefor clustering the data samples. It is because two networkarchitectures are implemented sequentially and they updatethe network parameters iteratively in mini batches. One mayassume that these networks are individually comparable withthe others in timing, and they can also be used together ifpriority is given to the clustering performance.

Compactness within a cluster and separation between clus-ters are two important measurements for cluster validation inthe literature [2]. The validity of clustering results is defined as

Validity= 1

K

K∑

j=1

max j �=m

{dWCS(c( j ))+ dWCS(c(m))

dBCS(c( j ), c(m))

}(25)

TABLE IV

COMPACTNESS AND SEPARATION-BASED CLUSTER VALIDITY

dWCS(c( j )) =

P∑i=1

‖x (i) − c( j )‖P

∀x (i) ∈ C j (26)

dBCS(c( j ), c(m)) = ‖c( j ) − c(m)‖L2 (27)

where within-cluster scatter dWCS indicates the average dis-tance between data points x(i) and their assigned prototypevector c(j). Between-cluster separation for clusters j , m, anddBCS is simply the distance between cluster prototypes. Thebest clustering method minimizes the validity measure inthis manner. Table IV summarizes the validity results of theclustering algorithms on all the data sets for comparison.Although the PE network follows the similar validity results tothose of the SOM and NG networks, the proposed PE + SVMnetwork achieves again the lowest validity in all the situations,and this also supports the previous visual results. Also notethat the NFB networks are well ahead of the WTA ones as thedimensionality and the number of samples increase.

Silhouette metric refers to another method of interpretationand validation for the clustering problems [28]. Given thealready clustered data, it simply gives a matching rate for eachsample with respect to its assigned and the nearest neighborcluster. The Silhouette S(x(i)) is computed as

a(x (i)) =

N∑n=1

‖x (i) − x (n)‖N

∀x (i) ∈ C j (28)

b(x (i)) =

M∑m=1

‖x (i) − x (m)‖M

∀x (i) ∈ C j ∀x (m) ∈ Cz

(29)

Cz = c(argmin j=1,2,3,...,K ‖x (i)−c( j)‖) (30)

S(x (i)) =

⎧⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎩

1 − a(x (i))

b(x (i)); a(x (i)) < b(x (i))

0; a(x (i)) = b(x (i))

b(x (i))

a(x (i))− 1; a(x (i)) > b(x (i))

(31)

where a(x(i)) can be interpreted as how well the sample x(i) ismatched to the cluster it is assigned (i.e., the smaller value and



Fig. 11. Silhouette analysis on red-wine preference data set.

TABLE V

CLUSTERING COMPARISONS ON SILHOUETTE VALUES

the better matching). b(x(i)), on the other hand, represents theaverage dissimilarity of the sample to the neighboring clusterin which, aside from the previously assigned cluster, it fitsbest. From the above definitions, it is obvious that S(x(i)) fallsin a range of (−1, 1), and the best clustering algorithm triesto maximize it. We evaluate the clustering methods on theSilhouette values in Fig. 11 and in Table V. Because eachsample has its own Silhouette value, we first produce a his-togram that quantifies the amount of data points in bins, settingthe bin width to 0.2. Thereafter, the histogram is transformedinto cumulative densities for each clustering method to getbetter intuitions overall. Like in other comparisons so far, thePE + SVM network has the best performance in terms of theSilhouette metric, since its values start increasing last on red-wine preference data set in Fig. 11, and it achieves the highestmean Silhouette value in Table V.

Finally, we evaluate the clustering performances of thenetworks on all the Data sets by means of accuracy and puritymetrics, given in percentages in Tables VI and VII. Given thatwe have the real cluster labels of the data samples, we cancalculate the accuracy by first finding the best combinationsbetween real labels and the cluster assignments. Accuracy is

TABLE VI

ACCURACY AND PURITY ANALYSIS ON SYNTHETIC DATA SETS

TABLE VII

ACCURACY AND PURITY ANALYSIS ON UCI DATA SETS

then the ratio of true positives to the number of samples.With regard to purity, each cluster is initially assigned toa class label, which is established by the most frequent(i.e., majority voting) label in that cluster. Thereafter, puritycan be addressed to the accuracy formula in this setup. Notethat real label-cluster pairs may have one-to-many relation-ships. Although the PE network usually performs worse thanthe SOM and/or NG algorithms in general, the PE + SVMnetwork achieves better results by fine-tuning the PE clustersin its own latent space. The accuracy and purity resultsclearly support the previous statistical and visual clusteringassessments.

V. CONCLUSION

In this paper, we propose a hybrid neural network thatcombines two complementary networks for the clusteringproblem. The first network is referred as PE, since it mapsthe input data samples to their closest prototypes successivelyin the output layer. While optimizing the network, we getunsupervised features in a latent space, which holds for thehidden layer of the PE. The clustering update procedureis carried out in this domain unlike previous works wherethe original feature space is used instead. Encoding weights(i.e., W1) of the PE network are initialized with respectto Voronoi-like hyperplanes in a pair-wisely greedy mannerinstead of random selection. Please note that the number



of neurons in the hidden layer is also found implicitly, andwe eliminate this free parameter in the three-layer networkstructure. In addition, the Gaussian neighboring function isimplemented in the latent space to update not only the winningbut also the other prototypes for each sample. This alsocontributes to the performance increase as confirmed by theexperimental results.

In the second part of the hybrid network, SVM is used tofine-tune the previously established clusters of the PE network.The margins between cluster boundaries are maximized usinga quadratic hinge loss function as the objective. Like PE, theSVM network also updates the clusters iteratively in a latentspace, which is represented by its output layer activations.Another important implementation detail is the utilization ofthe kernel function. This idea fits in our proposed networkquite well, because the prototype vectors are now activatedas landmarks for the GKF. Hence, we avoid the commonapplication of treating all data samples as landmarks in whichcase all the samples are used to compute the kernel. Therefore,the computational complexity is remarkably reduced in thismanner. Furthermore, instead of the original data, we feedthe hidden layer activations of the optimized PE networksuccessively to the GKF, which makes our proposed workmore cooperative.

Our purposed work is compared with similar clusteringnetworks in the literature using different evaluation metrics.We use a wide range of data sets from popular clusteringrepositories that are divided into two subsets to analyzethe experimental results in each performance spectrum. Thesynthetic 2-D data sets are mainly utilized for visual analysiswhile the real-world UCI data sets are explored for the analyt-ical experiments. Both the visual and analytical results clearlyreveal that the PE + SVM clustering network outperforms theother approaches with the same experimental setup.

In addition, the parameter complexity of the proposedmethod is almost equal to its contemporaries, SOM andNG networks. Because we implicitly compute the number ofneurons in all the layers, the weight parameters (except W1 inthe PE network) are initialized randomly in the proposed work.Although the hybrid algorithm seems to take more time, twonetwork architectures are sequentially implemented and theyupdate the network parameters iteratively in mini batches. Onemay assume that these networks are individually comparablewith the others in timing. Also note that we update the clustersin the latent spaces while optimizing the proposed networks,PE and SVM. Especially for the PE network, the latent space isproduced in an unsupervised manner, and this directly impactsthe clustering quality.

We also try to find experimentally to what extent the SVMnetwork may compensate for initially unsatisfactory clusteringof the PE network. To address this issue, we add noise to theinput data samples, and we initialize cluster centroids, so thatthey are deployed in the outlier regions of the features space.We conclude that the SVM network is not strictly sensitive tothe PE network clustering results where it mostly changes theconfiguration of the PE network. Furthermore, it may evensacrifice some centroids for better clustering results in theiterative cluster updating procedure.

Finally, we explore the nature of transformations that havebeen learned through the proposed approach by visualizingthe hidden layer patterns. In addition, the improved clusteringresults are further investigated as we adopt different numberof hidden layers and neural nodes in each layer. To do so, weadopt two PE networks successively, such that once the firstPE network is learned with the data samples in the originalfeature space, its hidden layer activations are then fed into thenext one. The visual analysis shows that the proposed approachobtains more discriminative feature spaces in its hidden layers,which help to get improved clusters.

For the future work, we plan to find the optimum numberof clusters and initial prototype vectors for a given unlabeleddata set in the network architectures. Obviously, network archi-tectures have very elastic configurations, which can help tosimulate any objective function inside. In addition, new latentspaces are achieved in the hidden layers, and each layer can beused for a specific task in a greedy layerwise manner. There-fore, it is envisioned that neural networks can solve complexclustering problems dynamically, and traditional supervisionlimitations may be eliminated in this manner.

REFERENCES

[1] Y. Stekh, F. M. E. Sardieh, and M. Lobur, “Neural networkbased clustering algorithm,” in Proc. 5th MEMSTECH, Apr. 2009,pp. 168–169.

[2] K.-L. Du, “Clustering: A neural network approach,” Neural Netw.,vol. 23, no. 1, pp. 89–107, 2010.

[3] A. Coates, H. Lee, and A. Y. Ng, “An analysis of single-layer networksin unsupervised feature learning,” in Proc. Int. Conf. Artif. Intell.Statist. (AISTATS), 2011, pp. 215–223.

[4] Q. V. Le, “Building high-level features using large scale unsupervisedlearning,” in Proc. Int. Conf. Acoust., Speech Signal Process. (ICASSP),2013, pp. 8595–8598.

[5] R. B. Palm, “Prediction as a candidate for learning deep hierarchicalmodels of data,” M.S. thesis, DTU Inform., Tech. Univ. Denmark,Kongens Lyngby, Denmark, 2012.

[6] G. Hinton, “A practical guide to training restricted Boltzmannmachines,” Dept. Comput. Sci., Univ. Toronto, Toronto, ON, Canada,Tech. Rep. UTML TR 2010-003, Aug. 2010. [Online]. Available:https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf

[7] N. Srivastava, “Improving neural networks with dropout,”M.S. thesis, Dept. Comput. Sci., Univ. Toronto, Toronto, ON, Canada,2013.

[8] J. P. F. Sum, C.-S. Leung, P. K. S. Tam, G. H. Young, W. K. Kan,and L.-W. Chan, “Analysis for a class of winner-take-all model,” IEEETrans. Neural Netw., vol. 10, no. 1, pp. 64–71, Jan. 1999.

[9] E. Yair, K. Zeger, and A. Gersho, “Competitive learning and softcompetition for vector quantizer design,” IEEE Trans. Signal Process.,vol. 40, no. 2, pp. 294–309, Feb. 1992.

[10] M. Mishra and H. S. Behera, “Kohonen self organizing map withmodified K-means clustering for high dimensional data set,” Int. J. Appl.Inf. Syst., vol. 2, no. 3, pp. 34–39, 2012.

[11] T. Kohonen, Self-Organizing Maps (Springer Series in InformationSciences), vol. 30, 3rd ed. Berlin, Germany: Springer, 2001.

[12] T. Kohonen, “The self-organizing map,” Proc. IEEE, vol. 78, no. 9,pp. 1464–1480, Sep. 1990.

[13] T. M. Martinetz, S. G. Berkovich, and K. J. Schulten, “‘Neural-gas’ network for vector quantization and its application to time-seriesprediction,” IEEE Trans. Neural Netw., vol. 4, no. 4, pp. 558–569,Jul. 1993.

[14] K. Rose, F. Gurewitz, and G. C. Fox, “Statistical mechanics and phasetransitions in clustering,” Phys. Rev. Lett., vol. 65, no. 8, pp. 945–948,1990.

[15] G. A. Carpenter and S. Grossberg, “A massively parallel architecturefor a self-organizing neural pattern recognition machine,” Comput. Vis.Graph. Image Process., vol. 37, no. 1, pp. 54–115, 1987.



[16] J. C. Bezdek, “Modified objective function algorithms,” in PatternRecognition With Fuzzy Objective Function Algorithms. Norwell MA,USA: Kluwer, 1981, pp. 155–201.

[17] A. Y. Ng. Autoencoders and Sparsity, accessed on Mar. 8, 2016.[Online]. Available: http://ufldl.stanford.edu/wiki/index.php/Autoencoders_and_Sparsity

[18] E. Alpaydin, “Support vector machines,” in Introductionto Machine Learning. London, U.K.: MIT Press, 2004,pp. 218–225.

[19] F. Aurenhammer, “Voronoi diagrams—A survey of a fundamentalgeometric data structure,” Int. J. ACM Comput. Surv., vol. 23, no. 3,pp. 345–405, Sep. 1991.

[20] C. Cortes and V. Vapnik, “Support-vector networks,” Mach. Learn.,vol. 20, no. 3, pp. 273–297, 1995.

[21] Y. Tang, “Deep learning with linear support vector machines,” in Proc.Workshop Represent. Learn., Int. Conf. Mach. Learn. (ICML), 2013.

[22] Speech and Image Processing Unit. Clustering datasets. School ofComputing, University of Eastern Finland, accessed on Mar. 8, 2016.[Online]. Available: http://cs.joensuu.fi/sipu/datasets/

[23] P. Fränti and O. Virmajoki, “Iterative shrinking method for clusteringproblems,” Pattern Recognit., vol. 39, no. 5, pp. 761–765, 2006.

[24] P. Dostál and P. Pokorný, “Cluster analysis and neural network,”in Proc. Mezinárodní Konference Tech. Comput., Prague, CzechRepublic, 2008, accessed on Mar. 8, 2016. [Online]. Available:http://dsp.vscht.cz/konference_matlab/MATLAB08/prispevky/025_dostal.pdf

[25] A. Ben-Hur, D. Horn, H. T. Siegelmann, and V. Vapnik, “Support vectorclustering,” J. Mach. Learn. Res., vol. 2, pp. 125–137, Mar. 2002.

[26] J. Lee and D. Lee, “Dynamic characterization of cluster structures forrobust and inductive support vector clustering,” IEEE Trans. PatternAnal. Mach. Intell., vol. 28, no. 11, pp. 1869–1874, Nov. 2006.

[27] L. Bolelli, S. Ertekin, D. Zhou, and L. C. Giles, “K-SVMeans:A hybrid clustering algorithm for multi-type interrelated datasets,” inProc. IEEE/WIC/ACM Int. Conf. Web Intell., Nov. 2007, pp. 198–204.

[28] M. Scarpiniti, “Neural networks and cluster analysis,” Lecture Noteson Neural Networks, Dept. INFOCOM, Sapienza Univ. Rome, Rome,Italy, Tech. Rep., 2009, accessed on Mar. 8, 2016. [Online]. Available:http://ispac.diet.uniroma1.it/scarpiniti/files/NNs/Less5.pdf

[29] A. Gionis, H. Mannila, and P. Tsaparas, “Clustering aggregation,” ACMTrans. Knowl. Discovery Data, vol. 1, no. 1, pp. 1–27, 2007.

[30] P. Cortez, A. Cerdeira, F. Almeida, T. Matos, and J. Reis, “Modelingwine preferences by data mining from physicochemical properties,”Decision Support Syst., vol. 47, no. 4, pp. 547–553, 2009.

[31] UCI Machine Learning Repository. Wine Qual-ity Data Sets, accessed on Mar. 8, 2016. [Online].Available: http://archive.ics.uci.edu/ml/datasets/Wine+Quality

[32] UCI Machine Learning Repository. Firm-Teacher_Clave-Direction_Classification Data Set, accessed on Mar. 8, 2016.[Online]. Available: http://archive.ics.uci.edu/ml/datasets/Firm-Teacher_Clave-Direction_Classification

[33] UCI Machine Learning Repository. Letter RecognitionData Set, accessed on Mar. 8, 2016. [Online]. Available:http://archive.ics.uci.edu/ml/datasets/Letter+Recognition

Emrah Ergul received the B.S. degree from theElectrical and Electronics Engineering Department,Turkish Naval Academy, Istanbul, Turkey, in 2001,the M.S. degree in computer engineering from theNaval Science and Engineering Institute, Istanbul,in 2009, and the Ph.D. degree from the Electron-ics and Communication Engineering Department,Kocaeli University, Izmit, Turkey, in 2016. Hestudied with the Computer Vision and RoboticsLaboratory, Beckman Institute, The University ofIllinois at Urbana–Champaign, Champaign, IL,

USA, from 2013 to 2014, with a scholarship by The Scientific and Tech-nological Research Council of Turkey.

Nafiz Arica (M’98) received the B.S. degreefrom the Turkish Naval Academy, Istanbul, Turkey,in 1991, and the M.S. and Ph.D. degrees incomputer engineering from Middle East TechnicalUniversity (METU), Ankara, Turkey.

He spent four years as a Communications andCombat Officer in the Turkish Naval Forces,Kocaeli, Turkey. In 1995, he joined METU. Heconducted research as a Post-Doctoral Associatewith the Beckman Institute, University of Illinois atUrbana–Champaign, Champaign, IL, USA, and the

Department of Information Science, Naval Postgraduate School, Monterey,CA, USA. From 2004 to 2013, he was a Faculty Member with the Com-puter Engineering Department, Turkish Naval Academy. He is currently anAssociate Professor with the Faculty of Engineering and Natural Sciences,Bahçesehir University, Istanbul. His current research interests include objectdetection, recognition and tracking, path planning for unmanned vehicles, andfacial expression analysis.

Associate Prof. Arica’s thesis received the Thesis of the Year Award atMETU in 1998.

Narendra Ahuja (F’92) received the Ph.D. degreefrom the University of Maryland at College Park,College Park, MD, USA, in 1979.

He was the Founding Director of theInternational Institute of Information Technology,Hyderabad, India, where he continues to serve as theInternational Director. He is currently the DonaldBiggar Willet Professor with the Department ofElectrical and Computer Engineering, Universityof Illinois at Urbana–Champaign, Champaign,IL, USA. His current research interests include

next-generation cameras, 3-D computer vision, video analysis, imageanalysis, pattern recognition, human–computer interaction, image processing,image synthesis, and robotics.

Prof. Ahuja is a fellow of the American Association for ArtificialIntelligence, the International Association for Pattern Recognition, theAssociation for Computing Machinery, the American Association forthe Advancement of Science, and the International Society for OpticalEngineering. He is on the Editorial Boards of several journals.

Sarp Erturk (M’99–SM’15) received theB.S. degree in electrical and electronics engineeringfrom Middle East Technical University, Ankara,Turkey, in 1995, and the M.S. degree intelecommunication and information systems andthe Ph.D. degree in electronic systems engineeringfrom the University of Essex, Colchester, U.K.,in 1996 and 1999, respectively.

He carried out his compulsory service withthe Army Academy, Ankara, Turkey, from 1999to 2001. He is currently appointed as a Full

Professor with Kocaeli University, Izmit, Turkey, where he was an AssistantProfessor from 2001 to 2002, and an Associate Professor from 2002 to2007. He currently leads the Kocaeli University Laboratory of Image andSignal Processing that has a full-time researcher capacity of about 20researchers. His current research interests include digital signal and imageprocessing, video coding, embedded systems, remote sensing, and digitalcommunications.

Documents

IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING …vision.ai.illinois.edu/publications/ergulnn2016.pdf · Emrah Ergul, Naﬁz Arica, Member, IEEE, Narendra Ahuja, Fellow, IEEE,