Clustering analysis workshopCITM, Lab 318, Oct 2014

Facilitator: Hosam Al-Samarraie, PhD.

Outline

• – The basic concepts of cluster analysis.• – The different types of clustering procedures.• – How to execute and generate clustering

results.• – The SPSS clustering outputs.• – The learning machine outputs.

What Does Data Mining Do?

• Data mining extract patterns from data– Pattern? A mathematical (numeric

and/or symbolic) relationship among data items

• Types of patterns– Association– Prediction– Cluster (segmentation)

Knowledge Discovery

Steps in a Knowledge Discovery process

Supervised vs. Unsupervised Learning

• Supervised learning (classification)– Supervision: I know the output and I want to

examine the effect between the Independent variable on Dependent one.

• Unsupervised learning (clustering)– The class or the nature of the variables is

unknown

– Given a set of measurements, observations, etc. with the aim of establishing the existence of classes or clusters in the data

The concept of cluster analysis

Cluster analysis is unsupervised learning for identifying homogenous groups of objects called clusters.

Cluster share many characteristics, but are very dissimilar to objects not belonging to that cluster.

Cont…

• Measuring distances (differences or dissimilarities between subjects)

• Measuring proximities (similarity between subjects)

Types of Data!!

Gender…. Age groupLength

Numeric Not numeric

Count

Typical research questions the Cluster Analysis answers are as follows:

• Medicine – What are the diagnostic clusters? • To answer this question the researcher would devise a diagnostic questionnaire that entails the

symptoms (for example in psychology standardized scales for anxiety, depression etc.). The cluster analysis can then identify groups of patients that present with similar symptoms and simultaneously maximize the difference between the groups.

• Marketing – What are the customer segments? • To answer this question a market researcher conducts a survey most commonly covering needs,

attitudes, demographics, and behavior of customers. The researcher then uses the cluster analysis to identify homogenous groups of customers that have similar needs and attitudes but are distinctively different from other customer segments.

• Education – What are student groups that need special attention? • The researcher measures a couple of psychological, aptitude, and achievement characteristics.

A cluster analysis then identifies what homogeneous groups exist among students (for example, high achievers in all subjects, or students that excel in certain subjects but fail in others, etc.).

• A discriminant analysis then profiles these performance clusters and tells us what psychological, environmental, aptitudinal, affective, and attitudinal factors characterize these student groups.

Types of clustering

Hierarchical Clustering1. use agglomerative ("bottom-up”) algorithms begin with each element as a

separate cluster and merge them into successively larger clusters.

2. Handles continuous data.

Cont…

• Can be visualized as a dendrogram– A tree-like diagram that records the sequences of

merges or splits

1 3 2 5 4 60

0.05

0.1

0.15

0.2

Non hierarchicalK-means clustering

1. Begin with two starting center points and allocate each item to nearest cluster center.

2. Allocate items to nearest cluster center.

Mix Two-Steps Clustering

1. designed to handle very large data sets. 2. can handle both continuous and categorical variables or

attributes. 3. automatically select the number of clusters.

Generate clustering 1

1. Decide on cluster variables

• At the beginning of the clustering process, we have to select appropriate variables for clustering.

Note!!!

• It is important to avoid using an abundance of clustering variables, as this increases the odds that the variables are no longer dissimilar.

• Meaning? If highly correlated variables are used for cluster analysis, specific aspects covered by these variables will be overrepresented in the clustering solution.

• In this regard, absolute correlations above 0.90 are always problematic.

• For example, measuring happiness and joy of a person.

Insight!!• When we usually use factor analysis, we usually get factor solution that does

not explain a certain amount of variance;

• As such, discarding of information will be performed before identifying the segments.

• However, removing variables with low loadings on all the extracted factors means that some potential information for the identification of segments are discarded.

• This in turn reduce the possibility of identifying different groups.

• Finally, the resulted factors based on the original variables become questionable.

2

2.Decide on the Clustering Procedure

• Refers to the process of forming the cluster.

Dataset

• Lets say I have different people with different measures of height and weight (variables).

• Now, if I want to group those people by weight and height into different groups, then I need to use Cluster analysis.

The SPSS clustering

Variables

People to be clustered. It can be performance, achievement,

etc…

Cont…Hierarchical Methods: If there is a limited number of observation, usually <200.

▸ Analyze Classify Hierarchical Cluster▸ ▸

K-Means: If there are many observations, usually > 500.

▸ Analyze Classify K-Means Cluster▸ ▸

Two-step cluster: If there are many observations and the clusters are measured on different scale levels (5 likert scale, nominal, ordinal, etc..)

▸ Analyze Classify Two-Step Cluster▸ ▸

In Hierarchical Select a Clustering Algorithm

• Ward’s method • (only hierarchical clustering)• ▸ Analyze Classify Hierarchical Cluster ▸ ▸ ▸

Method Cluster Method▸

Select measure of SimilarityIn hierarchal

• Only apply for Hierarchal and two-steps methods

Euclidean is the most commonly used type when it comes to analyzing ratio or interval-scaled data.

Select measure of Similarity

In Two-step

Two-step clustering:• ▸ Analyze Classify Two-Step Cluster ▸ ▸ ▸

Distance Measure

Standardize in Hierarchal only.

In both methods, convert variables with multiple categories (on a range of 0 to 1 or 1 to 1, or use Z score).

3

Identifying the number of clusters?

• For hierarchical clustering by examining the dendrogram:

• ▸ Analyze Classify Hierarchical Cluster▸ ▸• ▸ Plots Dendrogram▸

Not always recommended

Alternative solution

• Draw a scree plot (e.g., using Microsoft Excel) based on the coefficients in the agglomeration schedule. (Elbow method)..2 clusters are possible to use..

0 5 10 15 20 250

1000

2000

3000

4000

5000

6000

7000

8000

9000

Cofficent

Cofficent

For two-step and k-means

• Note: two-step clustering identify the number of clusters automatically.

• However, K-means use default of 2. The most recommended one is 3-4 clusters.

• So you need to try both and see which one provides useful output.

Save membership

• After identifying the number of clusters, we save the memberships between the cases.

Click save

Add 2

Membership to be used

Here is the membership

4

Assess the solution’s stability

• By using other methods and compare between each other.....

Assess the solution’s validity

• Criterion validity: Evaluate whether there are significant differences between the segments resulted from the membership step.

• P<0.05 We are doing well…

Interpret the cluster solution

• Examine cluster centroids and assess whether these differ significantly from each other (e.g., by means of t-tests or ANOVA). As we did earlier.

• Identify names or labels for each cluster and characterize each cluster by means of observable variables, if necessary (cluster profiling).

SPSS

• That’s all…..now lets try it in spss.

Another example

• Lets say I want to explore children that needs special learning.

• So I collected some data about children's reading and cognitive performance gain.

Now I ask the question, • What are children groups that need extra

learning?

• For the data place this url• www.hosamspace.com/data• Download the cluster children data.• Open the file in spss (or just double click)• Now observe the data.

Thank you

• Any further inquiry:• hosam@usm.my• myclasy@gmail.com