EEE 508

Image Segmentation

• Objective: to determine (extract) object boundaries. • It is a process of partitioning an image into distinct regions by

grouping together neighboring pixels based on some pre-defined similarity criterion.

• It can be viewed as a classification technique that forms regions of similarities in the image.

• The similarity criterion is determined using specific properties or features of pixels representing objects in the image.

• Much more difficult with non-uniform background as compared to uniform background.

1

EEE 508

Image segmentation

• Broad classification of methods: Edge-based methods

The edge information is used to determine boundaries of objects. The boundaries are then analyzed and modified to form closed regions belonging to the objects in the image.

Pixel-based direct classification methods Heuristics or estimation methods derived from the histogram statistics

of the image or clustering algorithms are used to form closed regions belonging to the objects in the image.

Region-based methods Pixels are analyzed directly for a region growing process based on a

pre-defined similarity criterion to form closed regions belonging to the objects in the image.

2

EEE 508

Edge-based image segmentation

• General procedure: Detect edges based on first-order or second-order gradient

information. Track and link relevant edges based on gradient information.

• Edge detection The Gradient magnitude commonly used to detect edges

3

EEE 508

Edge Detection

• One basic tool: Gradient operator

The Gradient magnitude is commonly used for edge detection

( ) ( )( )

( )( )( ) ;

,,

,

,

,,212

211

2

21

1

21

2121

=

∂∂

∂∂

=∇=nngnng

nnnx

nnnx

nnxnngxx(n1,n2) – input image

n2

n1

gradient: points in direction of greatest change (locally at a pixel)

( ) ( ) ( ) 21

2

2

21

2

1

2121

,,,

∂

∂+

∂

∂=

nnnx

nnnxnng

4

EEE 508

Edge Detection

Question: How do we realize (compute) the gradient in discrete-domain?

Example: Robert’s Gradient

Interpretation: this is like a cross-wise 2D first backward difference.

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )21212

21212

21211

21211

,11,,,

,1,1,,

nnxnnxn

nnxnng

nnxnnxn

nnxnng

+−+=∂

∂=

−++=∂

∂=

( ) ( ) ( ) ( ) ( ) 22121

2212121 ,11,,1,1, nnxnnxnnxnnxnng +−++−++=

⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅

n1

n2

5

EEE 508

Edge Detection

In general:

Robert’s Gradient

( ) ( ) ( )( ) ( ) ( )21221212

21121211

,,,,,,

nnhnnxnngnnhnnxnng

∗∗=∗∗=

( ) ( ) ( )22122

21121 ,,, nngnngnng +=

n1

(1)

(-1)

n2

h1( -n1,-n2)

(1)

(-1)

n2

h2( -n1, -n2)

n1

6

EEE 508

Edge Detection

Sobel’s Gradient

n1

(1)(2)

n2

h1( -n1,-n2)

(1)

(-1)

(-1)

(-2) n1

(-1)

(2)

n2

h2( -n1,-n2)

(1)

(-1)

(1)

(-2)

n1

(-1)(-2)

n2

h1(n1,n2)

(-1)

(1)

(1)

(2) n1

(1)

(-2)

n2

h2(n1,n2)

(-1)

(1)

(-1)

(2)

7

EEE 508

Edge Detection

• Prewitt’s Gradient

n1

(1)

(1)

n2

h1( -n1,-n2)

(1)

(-1)

(-1)

(-1)n1

(-1)

(1)

n2

h2( -n1,-n2)

(1)

(-1)

(1)

(-1)

8

EEE 508

Edge Detection

Note: Edge detection methods based on computing some form of gradient or based on computing a difference are sensitive to noise. Example: Background noise appears as isolated edge points ⇒ noise

smoothing can be applied before edge detection (recommended).

9

EEE 508

Edge Detection • Note: Edge detection methods based on computing some form

of gradient or based on computing a difference are sensitive to noise. Example: Background noise appears as isolated edge points ⇒

noise smoothing can be applied before edge detection (recommended).

• Laplacian operator Edge is detected not only when |g(n1, n2)| is large, but also zero

crossings are considered. Laplacian: 2nd derivative in 1D case. Gradient: 1nd derivative in 1D case. Laplacian:

( ) ( ) ( ) ( )22

212

21

212

212

21,,,,

nnnx

nnnxnnxnnL

∂∂

+∂

∂=∇=

10

EEE 508

Edge Detection How do we compute Lapalcian? Several approximations possible:

approximate and with a forward difference

Similarly:

( )1

21,n

nnx∂

∂

( ) ( ) ( ) ( )

( ) ( ) ( ) ( ) ( ) ( ) ( )2121212112111

21

121

212

21212111

21

,1,2,1,1,,,

,,1,,

nnxnnxnnxnngnngn

nnxnn

nnx

nnxnnxnngn

nnx

−+−+=−−=

∂

∂∂∂

=∂

∂

−+==∂

∂

( ) ( ) ( ) ( )

( ) ( ) ( ) ( ) ( ) ( ) ( )1,,21,1,,,,

,1,,,

2121212122122

21

222

212

21212122

21

−+−+=−−=

∂

∂∂∂

=∂

∂

−+==∂

∂

nnxnnxnnxnngnngn

nnxnn

nnx

nnxnnxnngn

nnx

( )21

212 ,

nnnx

∂∂

11

EEE 508

Edge Detection ⇒

Edges detected at zero-crossing of Lx(n1,n2) But, if there is no checking on magnitude of gradient, a lot of edge

points and false edge contours may be generated.

( ) ( ) ( ) ( ) ( ) ( ) ( )( ) ( )2121

2121212121212

21

,,,41,1,,1,1,,

nnhnnxnnxnnxnnxnnxnnxnnxnnL

∗∗=−−+++−++=∇=

n1(1)

n2

h( n1, n2)

(-4)

(1)

(1)

(1)

12

EEE 508

Edge Detection

Solution: If large variance ⇒ edge.

At zero-crossing, check local variance:No edge otherwise.

Note: Edge detection methods are used for detecting object/region boundaries for segmentation and image analysis.

13

EEE 508

Edge Detection First- order gradient operators:

• Sobel• Robert• Prewitt• Canny

First-derivative: GradientEdge is detected at the highest

point

Second-derivative: LaplacianEdge is detected at the zero-crossing point

14

EEE 508

Canny Edge Detector

1.Calculating horizontal gradient Gx and vertical gradient Gy at each pixel location by

convolving with partial derivatives of a 2D Gaussian function, given as

2.Computing the gradient magnitude G and direction θG at each pixel location.

3.Applying Non-maximum suppression (NMS) to thin edges.

4.Computing high and low thresholds based on the histogram of the gradients of the entire

image.

5.Performing hysteresis thresholding to determine the edge map.

−=−=

−−+

− 2

2

2

2

2

22

22

2

1),( σσσ

σσ

yxyx

x exeexyxF xx FIG ∗=

−=−=

−−+

− 2

2

2

2

2

22

22

21),( σσσ

σσ

xyyx

y eyeeyyxF yy FIG ∗=

Horizontal Gradient

Vertical Gradient

GradientMagnitude

AndGradient direction

Non-maximal

Suppression

Compute the high and low

thresholds

Hysteresis Thresholding

edge map

inputimage

15

EEE 508

Canny Edge Detector

• Consider a single row or column of the image– Plotting intensity as a function of position gives a signal

Where is the edge?

Effect of Noise

16

EEE 508

Solution 1: Smooth firstCanny Edge Detector

17

EEE 508

Solution 2: First derivative of the Gaussian function

Canny Edge Detector

18

EEE 508

Edge-based image segmentation

• Boundary tracking Objective: assemble meaningful edges to form closed regions. Usually based on pixel-by-pixel search to find connectivity among

edge segments, based on similarity criterion among edge pixels. Geometrical proximity and topographical properties used to

improve edge linking operations for noisy or occluded pixels. Also, probabilistic approaches, graphs and rule-based methods for

model-based segmentation have been used.

19

EEE 508

Edge-based image segmentation Neighbor search method:

Assume the edge-detection operation produces an edge magnitude e(x,y) and an edge orientation φ(x,y) information.

List of edge pixels can be produced from gradient image obtained from edge detection procedure.

Assume the first edge pixel is a boundary pixel bj. A successor boundary pixel bj+1 can be found in the 4-connected or 8-

connected neighborhood if the following conditions are satisfied:

|e(bj)| >T1 and |e(bj+1)| >T1 and |e(bj)-e(bj+1)| <T2 and |φ(bj)-φ(bj+1)| mod 2π < T3

If there is more than one neighboring pixel satisfying these conditions the pixel that minimizes the difference is selected.

Apply the algorithm recursively until all neighbors are searched. If no neighbor is satisfying the conditions, search is stopped and

procedure starts with new edge pixel. Note: this algorithm might leave many edge pixels and partial

boundaries unconnected.

20

EEE 508

Edge-based image segmentation Graph-based search method:

It attempts to find the path between the start and end nodes in a constructed graph by minimizing a cost function based on a distance measure and transitional probabilities.

– An edge map with magnitude and direction information

– A graph derived from the edge map with a minimum cost path (darker arrows) between the start and end nodes.

21

EEE 508

Edge-based image segmentation

An edge map with magnitude and direction information

A graph derived from the edge map with a minimum cost path (darker arrows) between the start and end nodes.

22

EEE 508

Pixel-based direct classification methods

• Use histogram statistics to define single or multiple thresholds to classify an image pixel-by-pixel.

• A simple approach: Bimodal histogram ⇒ set the threshold to gray value

corresponding to the deepest point in the histogram value. Multimodal histogram ⇒ partition the image using some

heuristics, then determine the threshold for each region using histogram of the region.

Classify every pixel by comparing its gray level to the selected threshold.

( )

≤>

=TyxfTyxf

yxg),( if0),( if1

,

23

EEE 508

Pixel-based direct classification methodsOriginal image Image histogram

Thresholded, T=12 Thresholded, T=166 Thresholded, T=225

24

EEE 508

Pixel-based direct classification methods

• Optimal global thresholding To determine the optimal global threshold, apply parametric

distribution-based methods to the histogram. Assume histogram consists of two Gaussian distributions

belonging to two respective classes such as background and object.

Classification is done according to:

( ) ( ) ( ) 1; 212211 =++= PPzpPzpPzp - class probabilities

( )

>≤

=TyxfTyxf

yxg),( if2 Class),( if1 Class

,

25

EEE 508

Pixel-based direct classification methods

Define error probabilities of misclassifying pixels:

For Gaussian distributed Class 1 and Class 2:

where σi and µi are standard deviation and mean for classes

( ) ∫+∞

=T

dzzpTE )(11 ( ) ∫∞−

=T

dzzpTE )(22and

probability to classify a class 1 pixel to class 2

probability to classify a class 2 pixel to class 1

( ) ( ) ( )TEPTEPTE 2211 += - the overall probability of error

( )( ) ( )

22

22

21

21

2

2

22

1

1

22σµ

σµ

σπσπ

−−

−−

+=zz

ePePzp

26

EEE 508

Pixel-based direct classification methods

Determine optimal T by finding a general solution that minimizes E(T). Using the mixture distribution p(z), solution can be found by solving:

where

If the variances of both classes are equal to σ2:

Note: If both classes are with equal likelihood ⇒ T = (µ1 + µ2)/2.

02 =++ CBTAT

( )

+−=−=−=

21

1222

21

22

21

21

22

212

221

22

21 ln2;2;

PPCBA

σσ

σσσµσµσµσµσσ

−

++

=1

2

21

221 ln

2 PPT

µµσµµ

27

EEE 508

Pixel-based direct classification methods

• Pixel classification through clustering Clustering is the process of grouping data points with similar

feature vectors together. Data points that are close to each other in the feature space are clustered together.

The similarity of feature vectors can be represented by an appropriate distance measure such as Euclidean or Mahalanobis distance.

Each cluster is represented by its mean (centroid) and variance (spread) associated with the distribution of the corresponding feature vectors of the data points in the cluster.

The formation of clusters is optimized with respect to an objective function involving pre-specified distance and similarity measures along with additional constraints such as smoothness.

Clustering may produce disjoint regions with holes ⇒ post-processing algorithm might be needed such as region growing and others

28

EEE 508

Pixel-based direct classification methods

• k-Means clustering (similar to LBG for VQ) It partitions d-dimensional data into k clusters such that an

objective function providing the desired properties of the distribution of the feature vectors in terms of similarity and distance measures is optimized.

Procedure:1. Select the number of clusters k with initial cluster centroids vi ; i=1,…,k. 2. Partition input data points into k clusters by assigning each data point

xj to the closest cluster centroid vi using the selected distance measure, e.g. Euclidean distance:

ijij vxd −= where { }nxxxX ,...,, 21= is the input data set

29

EEE 508

Pixel-based direct classification methods

3. Compute a cluster assignment matrix U representing the partition of the data points with the binary membership value of the j-th data point to the i-th cluster such that:

4. Re-compute the centroids using the membership values as

5. If cluster centroids or the assignment matrix does not change from the previous iteration, stop; otherwise go to Step 2.

∑

∑

=

== n

jij

n

jjij

i

u

xuv

1

1 for all i

ijuU = where { }1,0∈iju for all i, j

11

=∑=

k

iiju for all j and for all inu

n

jij <<∑

=10

30

EEE 508

Pixel-based direct classification methods

The k-Means clustering method optimizes the sum-of-squared-error based objective function:

Note: The k-Means clustering method is quite sensitive to the initial cluster assignment and the choice of the distance measure ⇒additional criterion such as within-cluster and between-cluster variances can be included in the objective function as constraints to force the algorithm to adapt the number of clusters k.

( ) ∑∑= =

−=k

i

n

jijw vxvUJ

1 1

2,

31

EEE 508

Region-based image segmentation

• Region-growing based segmentation Examine pixels in the neighborhood based on a pre-defined

similarity criterion. Neighborhood pixels with similar properties are merged to form closed regions for segmentation.

Merging continues until there is an insufficient number of neighborhood pixels to be added in the current region.

Two criteria needed: A similarity criterion for inclusion of pixels in the region. A stopping criterion for stopping the growth, usually based on the

minimum number of neighborhood pixels required to satisfy the similarity criterion.

32

EEE 508

Region-based image segmentation Example:

Assume the pixel in the center is the origin of the region-growing process. Assume stopping criterion: minimum number of similar neighborhood pixels = 30% of the region to be included.Iteration 1: 8 pixels in 3×3 neighborhood (100%) satisfy similarity criterion, include the 8 pixels.

Iteration 2: 9 pixels in 5×5 neighborhood (56%) satisfy similarity criterion, include the 9 pixels.

Iteration 3: 6 pixels in 7×7 neighborhood (25%) satisfy similarity criterion, include none. Stop region-growing.

Segmented region

33

EEE 508

Region-based image segmentation

Example: Segmented regionOriginal image

34

EEE 508

Region-based image segmentation

• Region-splitting Examine the heterogeneity of a predefined property of the entire

region in terms of its distribution and the mean, variance, minimum and maximum values.

If the region is heterogeneous ⇒ split into two or more regions. Continue splitting until all regions satisfy homogeneity criterion

individually. Generated Ri, i=1, …, n regions satisfy the following conditions:

1. Each region Ri, i=1, …, n is connected.2.

3.4. H(Ri) = TRUE for i=1, …, n5. H(Ri∪Rj) = FALSE for i≠jwhere H(⋅) is a logical predicate for the homogeneity criterion.

RRn

ii =

=

1

jijiRR ji ≠=∩ ;, allfor 0

35

EEE 508

Region-based image segmentation

Region-splitting methods can also be implemented by rule-based systems and quadtrees.

In the quadtree-based method, regions are represented by nodes in the quadtree.

Example: Regions R1 and R3 satisfy homogeneity criterion. Regions R2 and R4 failed homogeneity criterion and are further split.

R21 R22

R24R23

R41 R42

R44R43

R1

R3

R

R1 R2 R3 R4

R41 R42 R43 R44R21 R22 R23 R24

36