Image segmentation · Image segmentation • Image segmentation is a partitioning of an image into parts using image attributes such as pixel intensity, spectral values, and/or textural

7/1/2014

1

Image segmentation

Assoc. Prof. Kadim Taşdemir

Antalya International University

[email protected]

Image segmentation• Image segmentation is a partitioning of an

image into parts using image attributes such as pixel intensity, spectral values, and/or textural properties. Image segmentation produces an image representation in terms of edges and regions of various shapes and interrelationships.

• Segmentation algorithms: – region growing/merging,

– watershed segmentation,

– edge-based segmentation,

– probability-based image segmentation,

– fractal net evolution approach (FNEA),

– and many more…

• Multi-scale image segmentation

Jensen, 2005

7/1/2014

2

• Division of an image into regions so that– the whole scene is covered by regions (spatially continuous,

exhaustive segmentation)

– the regions do not overlap

– the regions are homogeneous within themselves

– the homogeneity criteria of neighboring regions differ

• Region (token):– aggregate of pixels grouped together (directly or indirectly)

• Homogeneity as overarching principle– ‘relatively’ homogeneous regions reflect better the

‘Neardecomposability’ of natural systems

– High heterogeneity creates boundary to neighboring patches, low remaining heterogeneity within patches

– Homogeneity criterion: grey value, color, texture, form, altitude, etc.

Image segmentation

From Centre for Geoinformatics | © 2006 Lang/Albrecht/Blaschke

• Region growing

– Seed cells are distributed over image

– Bottom up (randomly)

– Top-down (content expected)

– Neighbors (4- or 8-neighborhood) are

included into region, if

– they do not belong to another region yet

– the homogeneity criterion H applies

– Two neighboring regions are unified, if H

applies

Image segmentation

7/1/2014

3

Image segmentation

• Split and Merge

– Combination of coarse segmentation and merge

– Example: Quadtree

• Initially: image as one object � division into 4 parts, if H does

not apply

• Resulting quadtree structure

• Merge of homogenous quadtree areas

• Bottom up region merging technique– Starting with each pixel being a region

– A pair of regions is merged into one region, each merge having a merging cost (degree of fitting)

– Objects are merged into bigger objects as long as the cost is below a ‘least degree of fitting’(scale parameter)= the merge fulfils the homogeneity criterion

– Starting points for merging distributed with maximum distance

– Pair wise clustering process considering smallest growth of heterogeneity

• Establishing segmentation levels on several scales using different scale parameters (e.g. 2nd level based on 1st level: larger scale parameter results in larger objects consisting of the objects of the 1st level)


Image segmentation

7/1/2014

4

• Finding an adjacent object B for an

arbitrary object A for merging them

1. Fitting

2. Best fitting

3. Local mutually best fitting

4. Global mutually best fitting


Image segmentation

One criteria used to segment a remotely sensed image into

image objects is a pixel neighborhood function, which

compares an image object being grown with adjacent pixels.

The information is used to determine if the adjacent pixel

should be merged with the existing image object or be part

of a new image object. a) If a plane 4 neighborhood

function is selected, then two image objects would be

created because the pixels under investigation are not

connected at their plane borders. b) Pixels and objects are

defined as neighbors in a diagonal 8 neighborhood if they

are connected at a plane border or a corner point.

Diagonal neighborhood mode only be used if the structure

of interest are of a scale to the pixel size. Example of road

extraction from coarse resolution image.

In all other case, plane neighborhood mode is appropriate

choice

Should be decided before the first segmentation

Pixel neighborhood function

7/1/2014

5

Definition of the degree of fitting

• Color and shape homogeneity are weighted against each other

• Compactness and smoothness make up the shape homogeneity

and are weighted against each other

Two objects are similar which are near

to each other in a certain feature space

Compactness: ideal compact form of

objects (objects don’t become lengthy)

Smoothness: boundaries of the edges

don’t become fringed


Image segmentation

Criteria for segmentation

• The scale parameter is an abstract value to determine the maximum possible change of heterogeneity caused by fusing several objects.

– The scale parameter is indirectly related to the size of the created objects.

– The heterogeneity at a given scale parameter is directly linearly dependent on the object size. Homogeneous areas result in larger objects, and heterogeneous areas result in smaller objects.

– Small scale � small objects, large scale � large objects. This refers to Multiresolution image segmentation.

• Color is the spectral feature

• Shape includes compactness and smoothness which are two geometric features that can be used as "evidence."

– Smoothness describes the similarity between the image object borders and a perfect square.

– Compactness describes the "closeness" of pixels clustered in a object by comparing it to a circle

7/1/2014

6

Color and shape

These two criteria are used to create image objects (patches) of relatively

homogeneous pixels in the remote sensing dataset using the general

segmentation function (Sf):

where the user-defined weight for spectral color versus shape is 0 < wcolor < 1.

If the user wants to place greater emphasis on the spectral (color)

characteristics in the creation of homogeneous objects (patches) in the

dataset, then wcolor is weighted more heavily (e.g., wcolor = 0.8).

Conversely, if the spatial characteristics of the dataset are believed to be

more important in the creation of the homogeneous patches, then shape

should be weighted more heavily.

( ) shapecolorcolorcolorf hwhwS ⋅−+⋅= 1

So the color criterion is computed as the weighted mean of all

changes in standard deviation for each band k of the m bands of

remote sensing dataset. The standard deviation sk are weighted by

the object sizes nob (i.e. the number of pixels) (Definiens, 2003):

where mg means merge (total pixels in all objects 1 and 2 here).

( )[ ]2

2

1

1

1

ob

kob

ob

kob

mg

kmg

m

k

kcolor nnnwh σσσ ⋅+⋅−⋅=∑=

Spectral (i.e., color) heterogeneity (h) of an image object is computed

as the sum of the standard deviations of spectral values of each layer

(sk) (i.e., band) multiplied by the weights for each layer (wk):

kk

m

k

wh σ⋅=∑=1

Usually equal weight for all bands except you know certain band is really important

7/1/2014

7

n

lhcompact =

b

lhsmooth =

⋅+⋅−⋅=

2

22

1

11_

ob

obob

ob

obob

mg

mg

mgdifferencecompactn

ln

n

ln

n

lnh

⋅+⋅−⋅=

2

22

1

11_

ob

obob

ob

obob

mg

mg

mgdifferencesmoothb

ln

b

ln

b

lnh

( )differencesmoothcptdifferencecompactcptshape hwhwh __ 1 ⋅−+⋅=

compactness smoothness

n is number of pixel in the object, l is the perimeter,

b is shortest possible border length of a box bounding the object

Compactness weight makes it possible to separate objects that have quite different

shapes but not necessarily a great deal of color contrast, such as clearcuts versus bare

patches within forested areas.


Criteria for segmentation

n

lhcompact =

b

lhsmooth =

7/1/2014

8

Jensen, 2005

Classification

based on

Image

Segmentation

takes into

account spatial

and spectral

characteristics

Object Based Image Analysis

7/1/2014

9

Source: http://usda-ars.nmsu.edu/PDF%20files/laliberteAerialPhotos.pdf

Pixel-based classification– Spectral values may belong

to more than one

information class

– No spatial relationships

used in classification

– ‘Salt-pepper’ effect

– Considers spectral values &

texture

but not form&shape,

neighborhood, context,

levels


“Human interpreters could derive little information using the point-by-

point approach, because humans derive less information from the

brightness of individual pixels than they do from the context and the

patterns of brightness [i.e. texture], of groups of pixels, and from the

sizes shapes and arrangements of parcels of adjacent pixels.”

(Campbell, 2002)

7/1/2014

10

Object-based classification

• Use spatial autocorrelation (to grow homogeneous regions, or regions with specified amounts of heterogeneity)

• Not only pixel values but also spatial measurements characterizing the shape of the region

• Divide image into segments or regions based on spectral and shape similarity or dissimilarity, i.e., pixel � image objects

• Classification is faster because objects are assigned to specific classes (instead of individual pixels).

• Primarily used for (very) high spatial resolution images


• Meaningful objects rather than pixels

• Improved reliability of statistics

– Several measurements (pixels) per object

– Clear boundaries

• Augmented, uncorrelated feature space

– Texture within objects, shape, neighbors, hierarchy


7/1/2014

11


• Results are not necessarily more correct, but more intuitive, more convincing, more practical

• Object generation suitable for textured or low-contrast image

– VHR-satellite imagery

– Airborne optical scanner data

– Airborne laser scanning (ALS) data

– Synthetic aperture radar (SAR) data

• Semi-automated image interpretation

• Supports image understanding by solving complex semantic problems

• Monitoring of known structures

– e.g. existing land use classification can be used as pre-defined boundaries for segmentation


“The idea of segmentation is not new but it is becoming more widespread

within the EO/RS community recently. While the foundations of the basic

principles were laid out in the 80ies (see Haralick & Shapiro, 1985) and

various applications demonstrated the potential in the following years for

environmental applications (e.g. Véhel & Mignot, 1994, Panjwani & Healey,

1995, Lobo et al., 1996), mainly the availability in commercial software

packages catalysed a boost of applications more recently.”

(Blaschke et al., 2004)

In addition, increased spatial resolution (down to 0.5m) enabled better

representation of objects and posed a challenge for pixel-based approaches.


7/1/2014

12

Software

http://www.ecognition.com/products

• Rule-based classification– Define a class by a rule on one feature or by rules on several

features

– Fuzzy or crisp rule definition

– Hierarchical relations of classes

– Rules can address different kinds of features

– Object features

– Class-related features

• Advantages compared to sample-based classification– Incorporation of expert knowledge in the classification

– Formulation of complex class descriptions

– Transparency (especially compared to neural networks)

– Transferability



7/1/2014

13


Some features for image objects

Some features for image objects

7/1/2014

14

Thomas, et al. 2003, PERS

Example 1

S. W. Myint, P. Gober, A. Brazel, S. Grossman-Clarke, Q. Weng,

“Per-pixel vs. object-based classification of urban land cover

extraction using high spatial resolution imagery”,

Remote Sensing of Environment, 115 (5), 1145–1161, 2011

Example 2

7/1/2014

15

OBIA Disadvantages

• Image segmentation:

A long standing challenge only with partial solutions

• Many parameters & corresponding thresholds to be set (often

specific to the image and zone) for optimal solution

“In RS common practice, any first-stage image-segmentation algorithm is

simultaneously affected by both omission and commission segmentation

errors. It is noteworthy that, although the inherent ill posedness of

image segmentation is acknowledged by a reasonable portion of

existing literature, this is often forgotten by a large segment of the RS

community where, literally, dozens of “novel” segmentation algorithms

are published each year.” (Baraldi et al. 2010, IEEE TGARS)

�Hybrid Methods: � combination of pixel-based & object-based

� combination of different features (spectral, textural, contextual)

Recent Methods

(Hybrid Approaches)

7/1/2014

16

A hybrid approach for urban areas

• A. K. Shackelford and C. H. Davis, “A combined fuzzy pixel-based and

object-based approach for classification of high-resolution multispectral

data over urban areas,” IEEE TGRS., 41(10), 2354–2363, 2003

• A. K. Shackelford and C. H. Davis, “A hierarchical fuzzy classification

approach for high-resolution multispectral data over urban areas,” IEEE

TGRS, 41 (9), 1920–1932, 2003.

• A. K. Shackelford and C. H. Davis, “Fully automated road network

extraction from high-resolution satellite multispectral imagery,” in Proc.

IGARSS, 2003, pp. 461–463.

A hybrid approach for urban areas

7/1/2014

17

eCognition commercial software toolbox

• eCognition User Guide 4, Definiens Imag. GmbH, Munich, Germany,2004.

• T. Blaschke, “Object based image analysis for remote sensing”, ISPRS

Journal of Photogrammetry and Remote Sensing, 65, 2-16, 2010.

eCognition commercial software toolbox

7/1/2014

18

Satellite Image Automatic Mapper™

• A. Baraldi, V. Puzzolo, P. Blonda, L. Bruzzone, and C. Tarantino, “Automatic

Spectral Rule-Based Preliminary Mapping of Calibrated Landsat TM and

ETM+ Images”, IEEE TGRS, 44 (9), 2563-2586, 2006.

• A. Baraldi, L. Durieux, D. Simonetti, G. Conchedda, F. Holecz, and Palma

Blonda, “Automatic Spectral-Rule-Based Preliminary Classification of

Radiometrically Calibrated SPOT-4/-5/IRS, AVHRR/MSG, AATSR,

IKONOS/QuickBird/OrbView/GeoEye, and DMC/SPOT-1/-2 Imagery—Part

I: System Design and Implementation”, IEEE TGRS, 48 (3), 1299-1325, 2010

• A. Baraldi, T. Wassenaar, and S. Kay, “Operational Performance of an

Automatic Preliminary Spectral Rule-Based Decision-Tree Classifier of

Spaceborne Very High Resolution Optical Images”, IEEE TGRS, 48 (9), 3482-

3502, 2010

Satellite Image Automatic Mapper™

7/1/2014

19

Hedge detection

S. Aksoy, H. G. Akcay, T. Wassenaar, "Automatic Mapping of Linear Woody Vegetation Features in

Agricultural Landscapes Using Very High-Resolution Imagery," IEEE TGRS, 48(1), 511-522, 2010.

Hedge detection

7/1/2014

20

Orchard detection

S. Aksoy, I. Z. Yalniz, K. Taşdemir, "Automatic

Detection and Segmentation of Orchards Using Very

High-Resolution Imagery," IEEE TGRS, 50(8), 3117-

3131, 2012.

Unsupervised clustering

(CONN linkage)

k: # of clusters is set by the user

Clusters are labeled to 4 classes:

GAC, woodlands, urban/bare, water

OBIA of clustermap(for texture and spatial analysis)

Inland vegetation and urban vegetation are derived

Taşdemir et al.

COMPAG, 2012

GAC (Good agricultural condition) Assessment

K. Taşdemir, P. Milenov and B. Tapsall, “A hybrid method combining SOM-based clustering and object-based analysis for identifying land in good agricultural condition,” Computers and Electronics in Agriculture,83, 92-101, 2012.

K. Taşdemir, Pavel Milenov and Brooke Tapsall, “Topology-based hierarchical clustering of self-organizing maps,” IEEE Transactions on Neural Networks, 22 (3), 474-485, 2011.

7/1/2014

21

Taşdemir et al., COMPAG 2012

KARD

SOM + OBIA

GAC

Forests

Urban/Bare

Water

Inland veg.

Urban veg.

Table is from Taşdemir et al., COMPAG, 2012

GAC (Good agricultural condition) Assessment

7/1/2014

22

Automatic LPIS assessment• K. Taşdemir, C. Wirnhardt, “Neural network based clustering for agriculture

management”, To Appear in EURASIP Journal on Advances in Signal Processing, Special Issue on Neural Networks for Interpretation of Remotely Sensed Data, Invited Paper, 2012.

• K. Taşdemir, “Vector quantization based approximate spectral clustering of large datasets,” Pattern Recognition, 45 (8), 3034-3044, 2012.

44

Zone1 Zone2 Zone3

Test zones &

Accuracies Red: Ineligible/eligible Blue: Eligible/ineligible

Gray: Urban regions

White: Eligible Black: Ineligible

Zone1 Zone3

Zone2

Confusion Matrix, Producer and User Accuracies (PA, UA)

Confusion Matrix, Producer and User Accuracies (PA, UA) Confusion Matrix, Producer and User Accuracies (PA, UA)

Automatic LPIS assessment

Documents

Image segmentation · Image segmentation • Image segmentation is a partitioning of an image into parts using image attributes such as pixel intensity, spectral values, and/or textural