Upload
sudhakar-chavan
View
66
Download
5
Embed Size (px)
Citation preview
Clustering Agenda
• Definition of Clustering• Existing clustering methods• Clustering examples• Clustering demonstration• Clustering validity
Definition• Clustering can be considered the most important unsupervised learning
technique; so, as every other problem of this kind, it deals with finding a structure in a collection of unlabeled data.
• Unsupervised: no information is provided to the algorithm on which data points belong to which clusters.
• Clustering is “the process of organizing objects into groups whose members are similar in some way”.
• A cluster is therefore a collection of objects which are “similar” between them and are “dissimilar” to the objects belonging to other clusters.
What Cluster Analysis is not• Supervised classification• Have class label information
• Simple segmentation• Dividing students into different registration groups
alphabetically, by last name
• Results of a query• Groupings are a result of an external specification
Why and Where to use Clustering?
Why?• Simplifications• Pattern detection• Useful in data concept construction• Unsupervised learning process
Where?• Data mining• Information retrieval• text mining• Web analysis• marketing• medical diagnostic
Applications• Retail – Group similar customers• Biology – Group similar plants/animals to study their common
behavior• Financial services – Groups similar types of accounts or
customers• Air line – Group similar types of customers to offer different
discounts• Insurance – Group similar nature of consumers as well as claims
to take policy decisions• Government – Group similar areas to announce various subsidies
or other benefits
Which method to use?It depends on following:
• Type of attributes in data• Dictates type of similarity
• Scalability to larger dataset• Ability to work with irregular data• Time cost• Complexity• Data order dependency• Result presentation
Major existing clustering algorithms
• K-means and its variants
• Hierarchical clustering
• Density-based clustering
K-means Clustering• Partition clustering approach • Each cluster is associated with a centroid (center point) • Each point is assigned to the cluster with the closest centroid• Number of clusters, K, must be specified• The basic algorithm is very simple
K-means Clustering – Details• Initial centroids are often chosen randomly.• Clusters produced vary from one run to another.
• The centroid is (typically) the mean of the points in the cluster.• ‘Closeness’ is measured by Euclidean distance, correlation, etc.• K-means will converge for common similarity measures
mentioned above.• Most of the convergence happens in the first few iterations.• Often the stopping condition is changed to ‘Until relatively
few points change clusters’• Complexity is O( n * K * I * d )• n = number of points, K = number of clusters,
I = number of iterations, d = number of attributes
Evaluating K-means Clusters• Most common measure is Sum of Squared Error (SSE)• For each point, the error is the distance to the nearest cluster• To get SSE, we square these errors and sum them.
• x is a data point in cluster Ci and mi is the representative point for cluster Ci • can show that mi corresponds to the center (mean) of the
cluster• Given two clusters, we can choose the one with the smallest
error• One easy way to reduce SSE is to increase K, the number of
clusters• A good clustering with smaller K can have a lower SSE
than a poor clustering with higher K
K
i Cxi
i
xmdistSSE1
2 ),(
Limitations of K-means
• K-means has problems when clusters are of differing • Sizes• Densities• Non-globular shapes
• K-means has problems when the data contains outliers.
K-means Clustering Algorithm1. The K-Means algorithm calculates the Arithmetic Mean of each cluster formed in
the dataset. a) The Arithmetic Mean of a cluster is the mean of all the individual records in
the cluster. In each of the first K initial clusters, their is only one record. b) The Arithmetic Mean of a cluster with one record is the set of values that
make up that record. c) For Example if the dataset we are discussing is a set of Avg Txn Amount,
Merchant Categories Transacted and Age with Citrus measurements for USER, where a record P in the dataset S is represented by a P = {Avg TxnAmt, Mer_Cat_Cnt, Age_Citrus).
d) Then a record containing the measurements of a User (9898084242), would be represented as 9898084242 = {2000, 3, 6} where 9898084242’s Txn Amount = 2000 rs, Mer Categories = 3 and Age with Citrus = 6 Months.
e) Since there is only one record in each initial cluster then the Arithmetic Mean of a cluster with only the record for 9898084242 as a member = {2000, 3, 6}.
2. Next, K-Means assigns each record in the dataset to only one of the initial clusters. Each record is assigned to the nearest cluster (the cluster which it is most similar to) using a measure of distance or similarity like the Euclidean Distance Measure or Manhattan/City-Block Distance Measure.
13
K-means Clustering Algorithm3. K-Means re-assigns each record in the dataset to the most similar cluster
and re-calculates the arithmetic mean of all the clusters in the dataset. The arithmetic mean of a cluster is the arithmetic mean of all the records in that cluster.
4. For Example, if a cluster contains two records where the record of the set of measurements for 9898084242 = {2000, 3, 6} and 8652084242 = {1000, 2, 2} then the arithmetic mean Pmean is represented as Pmean= {Avgmean, Mer Catmean, Agemean). Avgmean= (2000 + 1000)/2, Mer Catmean= (3 + 2)/2 and Agemean= (6 + 2)/2. The arithmetic mean of this cluster = {1500, 2.5, 4}. This new arithmetic mean becomes the center of this new cluster. Following the same procedure, new cluster centers are formed for all the existing clusters.
5. K-Means re-assigns each record in the dataset to only one of the new clusters formed. A record or data point is assigned to the nearest cluster (the cluster which it is most similar to) using a measure of distance or similarity
6. The preceding steps are repeated until stable clusters are formed and the K-Means clustering procedure is completed. Stable clusters are formed when new iterations or repetitions of the K-Means clustering algorithm does not create new clusters as the cluster center or Arithmetic Mean of each cluster formed is the same as the old cluster center. There are different techniques for determining when a stable cluster is formed or when the k-means clustering algorithm procedure is completed.
K - means clustering - demonstration
1) k initial "means" (in this case k=3) are randomly selected from the data set
2)k clusters are created by associating every observation with the nearest mean.
3) The centroid of each of the k clusters becomes the new means.
4) Steps 2 and 3 are repeated until convergence has been reached
15
Steps: K-means Clustering analysis• It is important to define the problem to be solved beforehand
so that clustering method, variables, data range can be selected.• Variable identification
• Variable categorization (e.g. Numeric, Categorical, Discrete, Continuous etc.)
• Conversion of non-numeric variables to numeric form• Running Descriptive Analysis
• Importing data • Selecting the variables• Scaling the variables to common metric• Deciding on the number of clusters to be created• Running the analysis• Interpreting the results
Case : K-Means• Step 1: Data preparation and Selecting Variables• Step 2: Scaling data – ruspini.scaled <- scale(ruspini)
Case : K-Means• Step 3: Identify Number of Clusters
Case : K-Means
• Step 4: K-Means Cluster km <- kmeans(ruspini.scaled, centers=6, nstart=10) km
Case : K-Means• Step 4: Plot Cluster plot(ruspini.scaled, col=km$cluster) points(km$centers, pch=3, cex=2) # this adds the centroids text(km$centers, labels=1:4, pos=2) # this adds the cluster ID
Hierarchical Clustering
p4p1
p3
p2
p4 p1
p3
p2
p4p1 p2 p3
p4p1 p2 p3
Traditional Hierarchical Clustering
Non-traditional Hierarchical Clustering Non-traditional Dendrogram
Traditional Dendrogram
Hierarchical clustering
Hierarchical clustering
Agglomerative (bottom up)
1. Start with 1 point (singleton)
2. Recursively add two or more appropriate clusters
3. Stop when k number of clusters is achieved.
Divisive (top down)
1. Start with a big cluster
2. Recursively divide into smaller clusters
3. Stop when k number of clusters is achieved.
Case : Hierarchical Clustering• Step 1: Get distance between data points• dist.ruspini <- dist(ruspini.scaled)
Case : Hierarchical Clustering• Step 1: Create and plot cluster• hc.ruspini <- hclust(dist.ruspini, method="complete")• plot(hc.ruspini)• rect.hclust(hc.ruspini, k=4)
Density Based Clustering : DBSCAN• DBSCAN is a density-based algorithm.• Density = number of points within a specified radius
(Eps)
• A point is a core point if it has more than a specified number of points (MinPts) within Eps • These are points that are at the interior of a cluster
• A border point has fewer than MinPts within Eps, but is in the neighborhood of a core point
• A noise point is any point that is not a core point or a border point.
DBSCAN Algorithm• Eliminate noise points• Perform clustering on the remaining points
Case: DBSCAN Density based clustering• Step 1: Get KKN plot for Epsilon value. library(dbscan) kNNdistplot(ruspini.scaled, k = 3) abline(h=.25, col="red")
Case: DBSCAN Density based clustering• Step 2: Run DBSCAN. db <- dbscan(ruspini.scaled, eps=.25, minPts=3) db
Case: DBSCAN Density based clustering• Step 3: Plot Cluster. plot(ruspini.scaled, col=db$cluster+1L)
Cluster Validity • For supervised classification we have a variety of measures to
evaluate how good our model is• Accuracy, precision, recall
• For cluster analysis, the question is how to evaluate the “goodness” of the resulting clusters?• Then why do we want to evaluate them?• To avoid finding patterns in noise• To compare clustering algorithms• To compare two sets of clusters• To compare two clusters
1. Determining the clustering tendency of a set of data, i.e., distinguishing whether non-random structure actually exists in the data.
2. Comparing the results of a cluster analysis to externally known results, e.g., to externally given class labels.
3. Evaluating how well the results of a cluster analysis fit the data without reference to external information.
- Use only the data4. Comparing the results of two different sets of cluster
analyses to determine which is better.5. Determining the ‘correct’ number of clusters.
For 2, 3, and 4, we can further distinguish whether we want to evaluate the entire clustering or just individual clusters.
Cluster Validation - Different Aspects
• Below are the different types of numerical measures that are applied to for cluster validity• External Index: Measures the extent to which cluster labels
match externally supplied class labels.• Entropy
• Internal Index: Used to measure the goodness of a clustering structure without respect to external information. • Sum of Squared Error (SSE)
• Relative Index: Compare two different clusters. • Often an external or internal index is used for this
function, e.g., SSE or entropy• Sometimes these are referred to as criteria instead of indices• However, sometimes criterion is the general strategy and
index is the numerical measure that implements the criterion.
Cluster Validity: Measures
• Need a framework to interpret any measure. • For example, if our measure of evaluation has the value, 10, is
that good, fair, or poor?• Statistics provide a framework for cluster validity
• The more “atypical” a clustering result is, the more likely it represents valid structure in the data
• Can compare the values of an index that result from random data or clusterings to those of a clustering result.• If the value of the index is unlikely, then the cluster results
are valid• These approaches are more complicated and harder to
understand.• For comparing the results of two different sets of cluster analyses,
a framework is less necessary.• However, there is the question of whether the difference
between two index values is significant
Framework for Cluster Validity
• Cluster Cohesion: Measures how closely related are objects in a cluster• Example: SSE
• Cluster Separation: Measure how distinct or well-separated a cluster is from other clusters
• Example: Squared Error• Cohesion is measured by the within cluster sum of squares
(SSE)
• Separation is measured by the between cluster sum of squares
• Where |Ci| is the size of cluster i
Internal Measures: Cohesion and Separation
i Cx
ii
mxWSS 2)(
i
ii mmCBSS 2)(