Selected Topics in Computer Engineering (0907779) · 2017. 12. 4. · Point, Edge, Line Detection The discontinuities we are interested in can be detected using derivatives First

Selected Topics in Computer

Engineering (0907779)Engineering (0907779)

Image Segmentation – Part I

Chapter 10

Dr. Iyad Jafar

Chapter 10

Sections : 10.1 – 10.2

Outline

� Introduction

�Fundamentals�Fundamentals

�Point, Line, and Edge Detection

�Edge Linking and Boundary Detection

�The Hough Transform �The Hough Transform

2

Introduction� What is image segmentation?

� The process of subdividing an image into its individualcomponents/objects

� A central operation in all machine vision applications� A central operation in all machine vision applications

� The level of subdivision depends on the problem in hand

� It is nontrivial for complex images

� The accuracy of the segmentation process determinesthe correct operation of computerized vision systems

3

Segmentation

Fundamentals

� Let R represents the entire spatial region occupied bythe image. The segmentation processes subdivides Rinto n subregions R1, R2, …, Rn such that1 2 n

1

(1)

(2) is a connected set, i = 1,2,3,...,n

(3) , for all i and j, i j

n

ii

i

i j

R R

R

R R

==

∩ = ∅ ≠

∪

4

(4) Q( ) TRUE, for i = 1,2,3,...,n

(5) Q( ) FALSE, for any adjacent regions and

i j

i

i j i j

R

R R R R

=∪ =

Fundamentals� Segmentation can be achieved by

� Detecting discontinuities (edge-based segmentation)� Assumes that the boundaries between the objects themselves

and the background are sufficiently differentand the background are sufficiently different

� Detecting similarities (region-based segmentation)� Assumes that the objects to be segmented are similar according

to some criteria

Boundary Detection

Region Filling

5

Boundary Detection

Standard deviation similarity

Point, Edge, Line Detection

� Edges

� Edge pixels are pixels at which the intensity ofan image function changes abruptly

� Edges are sets of connected edge pixels� Edges are sets of connected edge pixels

� Lines� Are edge segments in which the intensity ofthe background on either side of the line ismuch higher or lower than the intensity of linepixels

Line

Isolated Point

pixels

� Point

� A line whose length and width is roughly onepixel

6

Edge

Point, Edge, Line Detection

� The discontinuities we are interested in can be detected using derivatives

� First Derivative

� Second Derivative

'( ) ( 1) ( )f x f x f x= + −''( ) ( 1) ( 1) 2 ( )f x f x f x f x= + + − −

� Second Derivative ''( ) ( 1) ( 1) 2 ( )f x f x f x f x= + + − −

� Notes� First derivative generally produce thicker edges � Second order derivative has stronger response to fine details � Second order derivative produces double-edge response at ramp and step

transitions � The sign of the second derivative determines the type of transition

7

Point Detection

� We will use second derivative as it has stronger response to discontinuities

2

21 1 2

ff ( x , y ) f ( x , y ) f ( x, y )

x

∂ = + + − −∂2 2

22 2

f ff

x y

∂ ∂∇ = +∂ ∂

2x∂

2

21 1 2

ff ( x, y ) f ( x, y ) f ( x, y )

y

∂ = + + − −∂

2 1 1 1 1 4f f ( x , y ) f ( x , y ) f ( x, y ) f ( x, y ) f ( x, y )∇ = + + − + + + − −

or

8

0 1 0

1 -4 1

0 1 0

1 1 1

1 -8 1

1 1 1

Point Detection

� We saw earlier that isolated points have very highresponse for second derivatives

� We can detect the points by simply thresholding theabsolute value of the second derivative R(x,y) usingabsolute value of the second derivative R(x,y) using

� The main concern is in specifying the threshold value

1 , |R(x,y)| T g( x, y )

0 , otherwise

≥=

9Image Second Derivative Image Thresholded Second Derivative

Image with T = 0.95 max(R)

Line Detection

� Again , we will use second derivative as it hasstronger response to discontinuities and producesthinner edges than first derivativethinner edges than first derivative

� Keep in mind that the second derivative producesdouble-line effect.

� Solutions� Take the absolute value of the Laplacian image (produces� Take the absolute value of the Laplacian image (produces

thicker lines)

� Take the positive values of the Laplacian image only (innoisy images we have to use a threshold to reduce theeffect of noise)

10

Line Detection

� Example 10.1

Double-line Effect1 1 1

Original Scaled Laplacian Image

Double-line Effect1 -8 1

1 1 1

Laplacian Mask

11Absolute Value of Laplacian Image Positive Values of Laplacian Image Only

Thicker Lines

Line Detection

� The mask used in the previous is isotropic, i.e. it isindependent of line direction.

� It is desirable sometimes to detect lines in specificdirections (0o, +45o, -45o, and 90o). We can use masksdirections (0o, +45o, -45o, and 90o). We can use maskswhose coefficients along the desired direction have largerweights

� These masks provide strong response for bright lines inthe specified direction and that are one pixel thick

� In order to detect a line in a specific direction, we use thecorrect mask then we threshold for positive result only12

Line Detection

� Example 10.2. Detection of lines at 45o

2 -1 -1

Weak Response

Original Filtered Image

Strong Response

2 -1 -1

-1 2 -1

-1 -1 2

Mask

Isolated Points that can be removed easily by point

detection

13 Positive Values of Filtered Image Thresholded Image

detection

A line that is 1 pixel thick in the 45o

direction

Edge Detection� Edges can be classified according to intensity profiles into

� Step Edge (ideal edge)

� Involves transition between two intensity levels over a distance of one pixel

� Occur mostly in computer generated images

Ramp Edge� Ramp Edge

� The transition between two intensity levels occur over a distance that is greaterthan one pixel

� Appear in real images as a result of noise and focusing limitations of imagingdevices

� Roof Edge

� Essentially, they represent blurred lines that pass through a region

14

Ideal Edge Ramp Edge Roof Edge

Edge Detection

� It is not unusual to find the three types of edges in one image

Ideal Edge

15

Ramp Edge Roof Edge

Edge Detection� Detection of edges can be through the use of

first-order or second-order derivatives� For the first derivative, the magnitude can be used to detectthe presence of an edgethe presence of an edge

� For the second derivative, the sign of the second derivative isused to detect the presence of the edge

16

Edge Detection� Although it sounds straight forward to detect edges using first-order

or second-order derivatives, the presence of noise may affect thisoperation significantly depending on noise level

In general, detecting edges,

Incr

easi

ng N

oise

Lev

el

In general, detecting edges, involves :

1) Smoothing to reduce the sensitivity of derivatives to noise

2) Detection of all possible edge points

3) Edge localization to select

17 Edge First Derivative Second Derivative

Incr

easi

ng N

oise

Lev

el

3) Edge localization to select from the candidate point those that comprise the edge

Edge Detection Using Gradient

� The gradient is a powerful tool in finding the strength anddirection of edges.The gradient at pixel (x,y) is defined as

∂∂

=

=∇ x

fGxf

� The magnitude of the gradient measures the strength ofthe edge (maximum rate of change)

∂∂∂=

=∇

y

fx

G

G

y

xf

12 22

f fM( x, y )

∂ ∂ = + | |f G G∇ ≈ +

� The direction of the gradient is perpendicular to the edgedirection

18

M( x,y )x y

= + ∂ ∂ | | x yf G G∇ ≈ +

1tan ( )y

x

G

Gα −=


2f

2x

fG x

fG

∂ − ∂ ∇ = = = ∂

1tan ( ) 135y oGα −= =

� Theoretically, all edge pixels have the same gradientmagnitude and direction

19

f 2y fG

y

∇ = = = ∂ ∂

tan ( ) 135xG

α = =


� Gradient Masks to compute gradient at Z5

� Discrete 1st Derivative

( 1, ) ( , )xf

G f x y f x yx

∂= = + −∂

( , 1) ( , )yf

G f x y f x yy

∂= = + −∂

-1 -1 1

� Not efficient in detecting diagonal edges

� Roberts Cross-gradient Operator

8 5

( 1, ) ( , )

= Z - Z

xG f x y f x yx

= = + −∂

6 5 = Z - Z

y y∂

-1 0

0 1

0 -1

1 0

Z1 Z2 Z3

Z4 Z5 Z6

Z7 Z8 Z9

-1

1

-1 1

� 2x2 masks are not as good as symmetric masks which captureinformation from opposite sides around the center point20

9 5

8 6

x

y

G ( x,y ) z z

G ( x, y ) z z

= −= −

0 1 1 0

Horizontal Operator

Z7 Z8 Z9

Pixel z5 and its neighboursVertical Operator

-1 0 1

-1 0 1

Edge Detection Using Gradient� Gradient Masks to compute gradient at Z5

� Prewitt Operators-1 -1 -1

0 0 0

-1 -2 -1

0 0 0

-1 0 1

-2 0 2

-1 0 1

� Sobel Operators

7 8 9 1 2 3(Z + Z + Z ) (Z + Z + Z )xf

Gx

∂= = −∂

Mask to Compute Gx Mask to Compute Gy

1 1 1

3 6 9 1 4 7(Z + Z + Z ) (Z + Z + Z )yf

Gy

∂= = −∂

0 0 0

1 2 1

-2 0 2

-1 0 1

� They have better response than Prewitt masks and have better smoothingwhich is essential to reduce the effect of noise21

7 8 9 1 2 3(Z + 2Z + Z ) (Z + 2Z + Z )xf

Gx

∂= = −∂ 3 6 9 1 4 7(Z + 2Z + Z ) (Z + 2Z + Z )y

fG

y

∂= = −∂

Mask to Compute Gx Mask to Compute Gy

� Gradient Masks to compute gradient at Z5

� Prewitt and Sobel masks shown before give thestrongest response for horizontal and vertical edges. Wecan modify these masks to obtain better response for


can modify these masks to obtain better response fordiagonal edges

-1 -1 0

-1 0 1

0 1 1

0 1 1

-1 0 1

-1 -1 0

Prewitt Diagonal Masks

22

Sobel Diagonal Masks

-2 -1 0

-1 0 1

0 1 2

0 1 2

-1 0 1

-2 -1 0

� Example 10.3


Image Gx computed using Sobel Operator

Gy computed using Sobel Operator

23|Gx| + |Gy| Angle of Gradient

� Example 10.3. - Continued

� Note that in the previous slide, the edges of the wall bricks were successfully detected. However, this might not be desirable if we are interested in the main edges

We can eliminate such small edges (which might considered


� We can eliminate such small edges (which might considered as noise) by � Smoothing the image before computing the gradient

� Thresholding the gradient image

� Smoothing followed by thresholding

24Gradient Image (|Gx| + |Gy|) after the original

image was smoothed by 5x5 maskGradient Image (|Gx| + |Gy|) without

smoothing



Gradient Image (|Gx| + |Gy|) after the original image was smoothed by 5x5 mask

Gradient Image (|Gx| + |Gy|) without smoothing

25 Thresholded Gradient Image Thresholded Gradient Image (better connectivity for edges)


� The Sobel masks used in the previous slides were those thathave stronger response for vertical and horizontal directions.How about diagonal directions?


Gx computed using Sobel Operator (Horizental) Gy computed using Sobel Operator (vertical)

26Gx computed using Diagonal Sobel Operator (+45) Gx computed using Diagonal Sobel Operator (-45)

The Marr-Hildreth Edge Detector

� The method is based on the following arguments

� Intensity changes are not independent of image size, thusdetection of such changes requires the use of operators ofdifferent sizesdifferent sizes

� An intensity change is associated with a peak or a trough inthe first derivative and a zero-crossing in the secondderivative

� A filter that could achieve such results is defined as theLaplacian of Gaussian (LoG)

27

2 2

2

2 2

2

2 2 2 2

2 2 2 2

4

( , )

2 =

x y

x y

G G x y e

x ye

σ

σσσ

+−

+−

∇ = ∇ = ∇

+ −


� The zero crossings occur when x2 + y2 = 2σ2, i.e. a circle with radius 28 2σ


� The Marr-Hildreth algorithm consists of convoluting theLoG filter with an image f(x,y)

and then finding the zero crossings in g(x,y) to determine

2( , ) ( , ) ( , )g x y G x y f x y = ∇ •

and then finding the zero crossings in g(x,y) to determinethe locations of edges in f(x,y)

� The above equation can be rewritten as

� In summary, the algorithm is performed by

[ ]2( , ) ( , ) ( , )g x y G x y f x y= ∇ •

� In summary, the algorithm is performed by

� Filter the image f(x,y) with an nxn Gaussian mask

� Compute the Laplacian of the smoothed image

� Find the zero crossings of the resulting image in the previousstep

29


� How to generate the LoG mask ?

� Sample the LoG expression then scale its coefficients suchthat they sum to zero

To define a nxn LoG mask, sample the Gaussian function then� To define a nxn LoG mask, sample the Gaussian function thenconvolve it with a Laplacian mask

� How to specify the size of the LoG mask ?

� Recall that 99.7% of the volume under the Gaussian lies in [-3σ,3σ]3σ,3σ]

� As a rule of thumb, n should be selected to be an odd integerthat is equal or greater than 6σ

� Using smaller values reults in truncation of the Gaussian whilelarger values make little difference in the result

30


� How find the zero-crossings?

� Use a 3x3 neighborhood at each candidate edge pixel inthe image filtered by the LoG maskthe image filtered by the LoG mask

� A zero crossing at p implies that the signs of at least twoopposing neighbors pixels must differ

� We have four possible combinations: left/right,top/bottom, and the two diagonals

� Pixel p is considered an edge pixel if at least two of theopposing neighbors differ in signPixel p is considered an edge pixel if at least two of theopposing neighbors differ in sign

� Sometimes it is not sufficient to check sign of the neighbors,but we may need to check their absolute difference against athreshold value

31


� Example 10.4.

Original Result of LoG filtering with σ = 4 and n = 25

32Zero crossings using a

threshold of 0Zero crossings using a threshold of 4% of the max valus of LoG image

Edge Linking and Boundary Detection

� Ideally, edge detection should produce the sets of pixels thatform the edges in the image

� Practically, this is not always correct !� Some detected pixels correspond to noise� Some detected pixels correspond to noise

� Some of the edges pixels are not detected due to breaks in the edges

� Nonuniform illumination

� Efficiency of the detection method

� It is a common practice to follow the detection process bylinking process to fill in the breaks and to remove non-edgepixelspixels

� Three common approaches

� Local - Assumes knowledge about edge points in a local region

� Regional – Requires that the points on the boundary be known

� Global – Operates on the entire edge image33

Local Edge Linking

� It is based on analyzing a small neighborhood about everycandidate edge pixel in the edge image

� All pixels that are similar according to some criteria arelinked to form an edgelinked to form an edge

� Two principle properties used to measure similarity of edgepixels are the magnitude and direction of gradient

� A pixel (s,t) in a small neighborhood Sxy around pixel (x,y) issimilar to pixel (x,y) by gradient magnitude if

� A pixel (s,t) in a small neighborhood S around pixel (x,y) is

( , ) ( , )M s t M x y E− ≤

� A pixel (s,t) in a small neighborhood Sxy around pixel (x,y) issimilar to pixel (x,y) by gradient direction if

� The pixel (s,t) in Sxy is linked to pixel (x,y) if the previous twoconditions are satisfied

34

( , ) ( , )s t x y Aα α− ≤

Local Edge Linking

� The previous approach is expensive as it requires theinvestigation of each edge pixel

� Alternatively, we simplify it by the following steps

1) Compute the gradient magnitude and angle arrays, M(x,y) and α(x,y)1) Compute the gradient magnitude and angle arrays, M(x,y) and α(x,y)of the edge image f(x,y)

2) Form the binary image g(x,y) by using

3) Scan the rows of g(x,y) and fill (set to 1) all gaps (sets of 0s) in each

M 1 , if M(x,y) > T and (x,y) = A( , )

0 , otherwiseAT

g x yα ±

=

3) Scan the rows of g(x,y) and fill (set to 1) all gaps (sets of 0s) in eachrow that don’t exceed a specified length K. A gap is bounded by 1sfrom both ends. Processing is done on each row individually

4) To detect gaps in any other direction θ, rotate g(x,y) by θ andrepeat the horizontal scanning in step 3. Once done, rotate theresult by -θ

35

Local Edge Linking

� Example 10.5.

TM = 30% of maximum gradient value

A = 90o TA = 45o

K = 25 pixels K = 25 pixels

Horizontally Connected Edge

Pixels Logical OR

36

Original Gradient Image

Vertically Connected Edge

Pixels

Regional Edge Linking

� Assumes that the locations of regions of interest in theimage are known or can be determined.

� In other words, the regional membership of pixels in the� In other words, the regional membership of pixels in thecorresponding edge image is known

� A popular approach in this category is functionalapproximation by curve-fitting the known points (assumethat the desired edge/boundary is approximated by somemathematical expression)mathematical expression)

� Another approach is polygonal approximations. Theycapture the essential shape features of a region whilekeeping the representation of the boundary simple

37


� Polygonal Approximation� Given a set of points that represent an open curve with end points

A and B which are assumed to be vertices of the polygon

38


� Algorithm: for a set of ordered distinct points P in a binaryimage, finding a polygonal fit can be done by

1) Specify two starting points, A and B

2) Specify a threshold T and two empty stacks, OPEN and CLOSED

3) If the points in P correspond to a closed curve, then put A into OPEN3) If the points in P correspond to a closed curve, then put A into OPENand B into OPEN and CLOSED. If the points in P correspond to opencurve, put A into OPEN and B into CLOSED

4) Compute the parameters of the line passing through the last vertex inClosed and the last vertex in OPEN

5) Compute the distances from the line in Step 4 to points in P whosesequence places them between the vertices in Step 4. Select the pointVmax that has maximum distance Dmax.Vmax that has maximum distance Dmax.

6) If Dmax > T, then place Vmax at the end of the OPEN stack as a newvertex. Go to Step 4

7) If Dmax <= T, then remove the last vertex from OPEN and insert it asthe last vertex in CLOSED

8) If OPEN is not empty, go to Step 4, else exit. The vertices in CLOSEDare the vertices of the polygonal fit to points in P

39


� Example 10.6.

Processing is performed in the clockwise direction and the points

represent a closed curve1 2 3 4

5 6 7 8

40

Global Edge Linking and Hough

Transform

� The input is the edge image with all edge pixels beingcandidates for linking

� Acceptance or elimination of pixels is based on some globalproperties, usually a specified shape such as a line or circle

� Given n edge points, suppose we want to find subsets ofthese points that lie on straight lines.

� A simple approach is find all possible lines between every pairof points then pick those lines that are close to particularof points then pick those lines that are close to particularlines

� However, this is computationally expensive !! It requiresfinding n(n-1)/2 lines and performing n2(n-1)/2 comparisons !

41

Hough Transform

� Consider a point (xi,yi) in the xy-plane and the general equation of astraight line yi = a xi + b

� Infinite number of lines pass through this point with different values of aand b

� If we rewrite the line equation such that b = - a xi + yi and consider theab-plane (the parameter space), then we have the line equation for afixed pair (xi,yi)

� If we consider another point (xi,yi) that lie on the same line as (xi,yi), thenin the ab-space the two lines corresponding to these two point willintersect at (a’,b’) and so do all the lines for the points on the line in thexy-plane

42

� In order to find the principle lines in the image, we can compute all theparameter-space lines and then select those points that have large number ofintersections

� A major difficulty in this approach is that the slope of the line approaches infinityfor vertical and near vertical lines

One way around is to use the normal representation of the line in the xy-

Hough Transform

for vertical and near vertical lines

� One way around is to use the normal representation of the line in the xy-plane cos sini ix yθ θ ρ+ =

Representation of the two

points in the parameter

spaceTwo points that lie on the

same line

Intersection of the curves that

� In this representation, a vertical line is represented in the parameter-space byθ = 90 and p being the y-intercept.

� Points that lie on the same line have their normal representation intersects at(θ’,p’). As the number of points that lie on the same line increases, we willhave more curve crossing in the parameter space at the point (θ’,p’)43

the curves that represent the

two points

� Procedure for finding principle lines in animage

1. Subdivide the pθ parameter space into the so-called

accumulator cells and initialize them to zero. Therange of each of the parameters is specified by

Hough Transform

range of each of the parameters is specified by

a. -90≤ θ ≤ 90

b. -D ≤ p ≤D with D being the maximum diagonallength in the image

2. For every edge/line candidate pixel (xi,yi) in the edgeimage, we substitute every possible value of θ in p =xi cos θ + yisin θ. Round the values of p to the

allowed subdivision values on the p-axis

If results in p , then increment the count in cellSubdividing the parameter

3. If θm results in pn, then increment the count in cell(m,n) by one, i.e.A(m,n) = A(m,n)+1

4. Select accumulator cells with counts greater thancertain threshold and find the corresponding θ and

p for these cells

5. Draw the lines in the xy-plane for each pair of pairof θ and p found in Step 4

44

Subdividing the parameter space to generate accumulator cells

Example 10.7.Hough Transform

Edge Pixels Horizontal

y

Edge Pixels All Possible LinesHorizontal

lines

Vertical lines

45o line

x

45

-45o line

Pick p and θ with crossing count = 3 to select principle

lines in the image

� Example 10.8. We want to segment the two edges of the principlerunway

Hough Transform

Image Edge Image Parameter Space Representation of Edge Points

46 The Two Principle Lines Detected after Thresholding the Parameter Space

The Two Principle Lines Superimposed on Original Image

Related Matlab Functions

� fspecial

Hough� Hough

� edge

47

Documents

Selected Topics in Computer Engineering (0907779) · 2017. 12. 4. · Point, Edge, Line Detection The discontinuities we are interested in can be detected using derivatives First