Learning to Detect Natural Image Boundaries Using Local

Brightness, Color and Texture Cues

by David R. Martin, Charless C. Fowlkes, Jitendra Malik

Heather Dunlop16-721: Advanced Perception

January 25, 2006

What is a Boundary?

CannyMartin,

2002

Human

Dataset“You will be presented a photographic image. Divide the image into some number of segments, where the segments represent ‘things’ or ‘parts of things’ in the scene. The number of segments is up to you, as it depends on the image. Something between 2 and 30 is likely to be appropriate. It is important that all of the segments have approximately equal importance.”

DatasetDatabase of over 1000 images and 5-10 segmentations for each

Martin, 2002

Boundaries

Intensity

Texture

Brightness

Color

Non-boundaries Boundaries

Martin, 2002

Method

ImageOptimized Cues

Boundary Strength

Brightness

Color

Texture

Benchmark

Human Segmentations

Cue Combination

Model

Martin, 2002

Goal: learn the probability of a boundary, Pb(x,y,θ)

Image FeaturesCIE L*a*b* color space (luminance, red-green, yellow-blue)Oriented Energy:

fe: Gaussian second derivativefo: Its Hilbert transform

BrightnessL* distribution

Colora* and b* distributions (joint or marginal)

Texture

2,

2

,,oe fIfIOE

TextureConvolve with a filter bank:

Gaussian second derivativeIts Hilbert transformDifference of Gaussians

Filter responses give a measure of texture

Other Filter BanksLeung-Malik filter set: Schmid filter set:

Maximum Response 8 filter set:

TextonsConvolve image with filter bankCluster filter responses to form textons

Adapted from Martin, 2002 and Varma, Zisserman, 2005

Texton DistributionAssign each pixel to nearest textonForm distribution of textons

Adapted from Martin, 2002 and Varma, Zisserman, 2005

Gradient-based FeaturesBrightness (BG), color (CG), texture (TG) gradientsHalf-disc regions described by histogramsCompare distributions with χ2 statistic

r(x,y)

i ii

ii

hg

hghg

22 )(

2

1),(

Texture GradientTexton distribution in two half circles

Martin, 2002

LocalizationTightly localize boundariesReduce noiseCoalesce double detectionsImprove OE and TG features

OE

TG localized

OE localized

TG

Martin, Fowlkes, Malik, 2004

OptimizationTexture parameters:

type of filter bankscale of filtersnumber of textonsuniversal or image-specific textons

Other possible distance/histogram comparison metricsNumber of bins for histogramsScale parameter for all cues

Evaluation MethodologyPosterior probability of boundary: Pb(x,y,θ)

Evaluation measure: precision recall curveF-measure: Martin,

2002

5.0

)1(

PRPRF

Cue CombinationWhich cues should be used?

OE is redundant when other cues are presentBG+CG+TG produces best results

Martin, 2002

ClassifiersUntil now, only logistic regression was usedOther possible classifiers:

Density estimationClassification treesHierarchical mixtures of expertsSupport vector machines Martin,

2002

Result ComparisonAlternative methods:

Matlab’s Canny edge detector with and without hysteresisSpatially-averaged second moment matrix (2MM) Martin,

2002

ResultsCanny 2MM BG+CG+TG HumanImage

Martin, 2002

Results

Martin, 2002

Canny 2MM BG+CG+TG HumanImage

Results

Martin, 2002

Canny 2MM BG+CG+TG HumanImage

ConclusionsLarge data set used for testingTexture gradients are a powerful cueSimple linear model sufficient for cue combinationOutperforms existing methodsAn approach that is useful for higher-level algorithmsCode is available online:http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/