Segmentation and Clustering. Segmentation: Divide image into regions of similar contentsSegmentation: Divide image into regions of similar contents Clustering:

Segmentation and ClusteringSegmentation and Clustering


• Segmentation:Segmentation:Divide imageDivide imageinto regionsinto regionsof similar of similar contentscontents

• Clustering:Clustering:Aggregate pixelsAggregate pixelsinto regionsinto regionsof similar of similar contentscontents

But Wait!But Wait!

• We speak of segmenting foregroundWe speak of segmenting foregroundfrom backgroundfrom background

• Segmenting out skin colorsSegmenting out skin colors

• Segmenting out the moving personSegmenting out the moving person

• How do these relate to “similar How do these relate to “similar regions”?regions”?


• Defining regionsDefining regions– Should they be compact? Smooth Should they be compact? Smooth

boundary?boundary?

• Defining similarityDefining similarity– Color, texture, motion, …Color, texture, motion, …

• Defining similarity of regionsDefining similarity of regions– Minimum distance, mean, maximumMinimum distance, mean, maximum

Segmentation and Clustering Segmentation and Clustering ApplicationsApplications

SemanticsSemantics

““IntelligentIntelligentscissors”scissors”

FindingFindingskin-coloredskin-colored

regionsregions

Foreground /Foreground /backgroundbackground

segmentationsegmentation

Finding theFinding themoving objectsmoving objects

Finding theFinding thecars in acars in a

video sequencevideo sequence

Segmentation and Clustering Segmentation and Clustering ApplicationsApplications

““IntelligentIntelligentscissors”scissors”

FindingFindingskin-coloredskin-colored

regionsregions

Foreground /Foreground /backgroundbackground

segmentationsegmentation

Finding theFinding themoving objectsmoving objects

Finding theFinding thecars in acars in a

video sequencevideo sequence

StatisticsStatistics TemplatesTemplates

Clustering Based on ColorClustering Based on Color

• Let’s make a few concrete choices:Let’s make a few concrete choices:– Arbitrary regionsArbitrary regions

– Similarity based on color onlySimilarity based on color only

– Similarity of regions =Similarity of regions =distance between mean colorsdistance between mean colors

Simple Agglomerative ClusteringSimple Agglomerative Clustering

• Start with each pixel in its own clusterStart with each pixel in its own cluster

• Iterate:Iterate:– Find pair of clusters with smallestFind pair of clusters with smallest

inter-cluster distanceinter-cluster distance

– MergeMerge

• Stopping thresholdStopping threshold

Simple Divisive ClusteringSimple Divisive Clustering

• Start with whole image in one clusterStart with whole image in one cluster

• Iterate:Iterate:– Find cluster with largest intra-cluster Find cluster with largest intra-cluster

variationvariation

– Split into two pieces that yield largest inter-Split into two pieces that yield largest inter-cluster distancecluster distance

• Stopping thresholdStopping threshold

Difficulties with Simple ClusteringDifficulties with Simple Clustering

• Many possibilities at each iterationMany possibilities at each iteration

• Computing distance between clusters or Computing distance between clusters or optimal split expensiveoptimal split expensive

• Heuristics to speed this up:Heuristics to speed this up:– For agglomerative clustering, approximate For agglomerative clustering, approximate

each cluster by average for distance each cluster by average for distance computationscomputations

– For divisive clustering, use summary For divisive clustering, use summary (histogram) of a region to compute split(histogram) of a region to compute split

kk-means Clustering-means Clustering

1.1. Pick number of clusters Pick number of clusters kk

2.2. Randomly scatter Randomly scatter kk “cluster centers” in color “cluster centers” in color spacespace

3.3. Repeat:Repeat:a.a. Assign each data point to its closest cluster Assign each data point to its closest cluster

centercenter

b.b. Move each cluster center to the mean of the Move each cluster center to the mean of the points assigned to itpoints assigned to it

• Instead of merging or splitting, start Instead of merging or splitting, start out with the clusters and move them out with the clusters and move them aroundaround









Results of ClusteringResults of Clustering

Original ImageOriginal Image kk-means, -means, kk=5=5 kk-means, -means, kk=11=11


Sample clusters withSample clusters with k k-means clustering-means clusteringbased on colorbased on color

Other Distance MeasuresOther Distance Measures

• Suppose we want to have compact Suppose we want to have compact regionsregions

• New feature space: 5DNew feature space: 5D(2 spatial coordinates, 3 color (2 spatial coordinates, 3 color components)components)

• Points close in this space are close both Points close in this space are close both in color and in actual proximityin color and in actual proximity


Sample clusters with Sample clusters with kk-means clustering-means clusteringbased on color and distancebased on color and distance

Other Distance MeasuresOther Distance Measures

• Problem with simple Euclidean distance:Problem with simple Euclidean distance:what if coordinates range from 0-1000 what if coordinates range from 0-1000 but colors only range from 0-255?but colors only range from 0-255?– Depending on how things are scaled, gives Depending on how things are scaled, gives

different weight to different kinds of datadifferent weight to different kinds of data

• Weighted Euclidean distance: adjust Weighted Euclidean distance: adjust weights to emphasize different weights to emphasize different dimensionsdimensions

22)( iii yxcyx 22)( iii yxcyx

Mahalanobis DistanceMahalanobis Distance

• Automatically assign weights based on Automatically assign weights based on actual variation in the dataactual variation in the data

where C is covariance matrix of all pointswhere C is covariance matrix of all points

• Gives each dimension “equal” weightGives each dimension “equal” weight

• Also accounts for correlations between Also accounts for correlations between different dimensionsdifferent dimensions

)(1T2yxyxyx

C )(1T2yxyxyx

C

Segmentation Based on Graph Segmentation Based on Graph CutsCuts

• Create graph:Create graph:– Nodes = pixels in imageNodes = pixels in image

– Edge between each pair of nodesEdge between each pair of nodes

– Edge weight = similarity (intensity, color, Edge weight = similarity (intensity, color, texture)texture)

• Segmentation = graph cutSegmentation = graph cut– Want high intra-segment weights, low inter-Want high intra-segment weights, low inter-

segmentsegment

– Reasonably fast approximate algorithmsReasonably fast approximate algorithms

Documents

Segmentation and Clustering. Segmentation: Divide image into regions of similar contentsSegmentation: Divide image into regions of similar contents Clustering: