A Performance Characterization Algorithm for Symbol Localization

A Performance Characterization Algorithmfor Symbol Localization

Mathieu Delalandre1,2, Jean-Yves Ramel2, Ernest Valveny1 and Muhammad Muzzamil Luqman1,2

1 CVC, Barcelona city, Spain2 LI Laboratory, Tours city, France

LaBRI - Partnerships MeetingBordeaux, France

Thursday 14th of October 2010

Recognition

Spotting

r1 r2 r3

sofa

skin

tub

door

door

document database

learning database

Query By Example (QBE)

rank

labels

Symbol localization systems (recognition and spotting)

Spotting/RecognitionSystem

GroundtruthMatching localization

results with groundtruth

Region Of Interest

Characterization measures

sofa

skin

tub

door

door

Labels

r1 r2 r3

RanksQBE

truth results

Learning

Performance characterization

Performance characterization

To make the correspondence in term of localization

To compute characterization measures (recall, precision,

recognition rates, etc.)


groundtruth

results

groundtruth

– Global discrepancy methods

• Number of missed segmented pixels

• Position of missed segmented pixels

– Local discrepancy methods

• Number of region in the image

• Features values of regions


Performance evaluation

image segmentation object localization

coverage of results all image part of

precision of localization high importance weak importance

semantic matching weak importance high importance

Performance evaluation of image segmentation [Zhang’1996]

Performance evaluation : image segmentation vs. object localization

truth results

Single : an object in groundtruthmatches only with one detected object.Split : two objects in groundtruth match with one detected object.Merge : an object in groundtruth matches with two detected objects.

Performance evaluation of object localization [Delalandre2009]

False alarm : a detected object doesn't match with any object in groundtruth.Miss : an object in groundtruth doesn't match with any detected object.

groundtruth

results

groundtruth

Layout analysis [Antonacopoulos1999]

Text/graphics separation [Liu1997]

truth results

Single : an object in groundtruthmatches only with one detected object.Split : two objects in groundtruth match with one detected object.Merge : an object in groundtruth matches with two detected objects.

Performance evaluation of object localization [Delalandre2009]

Symbol spotting [Rusinol2009]

char and text boxes

groundtruth

results results

isothetic polygons

groundtruth groundtruth

results

Convex hulls


False alarm : a detected object doesn't match with any object in groundtruth.Miss : an object in groundtruth doesn't match with any detected object.


in a “part of” segmentation problem, how to make the difference between segmentation errors of background with segmentation errors of objects

Ways to solve ...

1. “naive” : To use thresholds to “reject” some segmentation results (bad ...)2. ideal : To define directed knowledge based approaches to model localization/segmentation algorithms (hard ...)3. intermediate (proposed) : To use “fuzzy-based” approach, tocharacterize the characterization results according to confidence rate

i.e. this is a positive matching between groundtruth and system’s results with a confidence rate of .

Open problem with object localization

probability error

dete

ctio

n ra

te

p1

p3

p2

Groundtruth, gravity centers, contours

Result points

Highest probabilities

Lowest probabilities


probability error

dete

ctio

n ra

te

p1

p3

p2


Result points



Localization comparison

Probability scores

Matching algorithm

Groundtruth Results

probability error

dete

ctio

n ra

te

Result point

Intersection lineIntersection point

Groundtruth, gravity center, contours

lgi

lgr

2s 0

gi

gr

l

ls

Lg

ri

c

probability error

dete

ctio

n ra

te

p1

p3

p2


Result points




Probability scores

Matching algorithm

Groundtruth Results

probability error

dete

ctio

n ra

te


321 sss

s2

s1

s3

g1 r

g2

g3

0321 ppp

321 00 sss

g2

s2

s3

r

g1

g3

101 231 ppp

0

2

2

3

s

Groundtruth pointsResult point

r

gi

s2

s1

s3

g1 r

g2

g3

321 sss

10 123 ppp

null probabilities,= equidistant case

highest probabilities,= nearest points

maximum probability,= equality case

probability error

dete

ctio

n ra

te

p1

p3

p2


Result points




Probability scores

Matching algorithm

Groundtruth Results

probability error

dete

ctio

n ra

te


cases

si = 0sj = k

sj +si = k

si = sj sj= 0si= k

si +sj= k

0 1 +

1 0 0probability error

dete

ctio

n ra

te

p1

p3

p2


Result points




Probability scores

Matching algorithm

Groundtruth Results

probability error

dete

ctio

n ra

te


j

is

sx

gi gj

j

ii s

sxfp

r

How to compute the probability between a groundtruth point g i and the result point r, considering the neighboring groundtruth point gj

we define - pi the probability r gi, regarding gj

- si is the scaling factor between gi and r - sj is the scaling factor between gj and r

gi gj

r

gi gj

r

gi gj

r

gi gj

r

si sj

pi

probability error

dete

ctio

n ra

te

p1

p3

p2


Result points




Probability scores

Matching algorithm

Groundtruth Results

probability error

dete

ctio

n ra

te


0 1 2 3 4

21

2

2

1

2

1)(

xex

x

Gaussian function

0 1

2

2

2

xk

y

j

ix

ek

ssxf

1

j

is

sx

Probability score function

0 1 +

1 0 0

j

is

sx

Thus, our probability function must respect the following properties

Several mathematics functions could be used (affine, exponential, trigonometric, etc.) we choose a Gaussian based function as it is good model of random distribution

j

ii s

sxfp

probability error

dete

ctio

n ra

te

p1

p3

p2


Result points




Probability scores

Matching algorithm

Groundtruth Results

probability error

dete

ctio

n ra

te


n

ijj

jii n

ssfp

,1 1

s2

s1

s3

g1 r

g2

g3

321 sss

10 123 ppp

We extend the computation of probability to a neighboring composed of n groundtruth points like this

we define

- is the set of groundtruth points

- si is the scaling factor between gi and r

- are the scaling factors between and r

- is the probability r gi, regarding

1

,1

n

ijjjg

1

,1

n

ijjjs

1

,1

n

ijjjg

1

,1

n

ijjjg

31211 2

1ssfssfp

32122 2

1ssfssfp

23133 2

1ssfssfp

Groundtruth pointsResult points

n

iig

1

q

jjr

1r1 r2 rq

g1 g2 gn…

…

dg1=2 dgn=0

dr1=1

np

kijjik thprga

1

),,(

dete

ctio

n ra

tes

0

1

0 1p1

score error

ε

single (Ts)

alarm (Tf)

multiple (Tm)

probability error

dete

ctio

n ra

te

p1

p3

p2


Result points



)(sin1 sgledd rjgi

)(0 ffalsedrj

)(11 mmultipledd rjgi

n

sTs

q

fT f

q

mTm


Probability scores

Matching algorithm

Groundtruth Results

probability error

dete

ctio

n ra

te


Qureshi’2008

Drawing level Symbol level

Setting backgrounds 5 models 16

Dataset images 100 symbols 2521



floo

rpla

nsdi

agra

ms

false alarm (Tf)

dete

ctio

n ra

tes

score error1

single (Ts)

multiple (Tm)

max

score error1

dete

ctio

n ra

tes

false alarm (Tf)

single (Ts)

multiple (Tm)

max

floorplans

diagrams

Score 0,11

Ts 0,57

Tf 0,31

Tm 0,20

Score 0,20

Ts 0,62

Tf 0,06

Tm 0,37

max

max


Characterization

Groundtruth 1 Results 1

probability error

dete

ctio

n ra

teA Performance Characterization Algorithm

for Symbol Localization

Characterization

Groundtruth 2 Results 2

probability error

dete

ctio

n ra

te

Each result is context dependent, how to compare them ?

Characterization

Groundtruth Results

probability error

dete

ctio

n ra

teA Performance Characterization Algorithm

for Symbol Localization

Characterization

Groundtruth Results

probability error

dete

ctio

n ra

te

Transform functionsi

ngle

det

ecti

ons

Ts

0

1

0 1

p1score error (ε)

ε

g 1

0 )(

)(

dg

sii

si

0)(

)()(

i

i

s

gs0

1

i

i

0

q

0 global score i

i

1

+

num

ber

of r

esul

ts (

q)

n

nq

jj

qn

ii rgnq

11

krrjqjkqj

q

,1,1,1

00 score error (x) 1

1

i(ε

)

0 )(

)()( d

g

sii

1(ε)

2(ε)

p1

1(1)2(1)

probability error

dete

ctio

n ra

te

We compute the difference between a result and self-matching of his groundtruth (g),to make the new results test-independent.

q i

=0 =0

=n =1

+ 0


Qureshi’2008

Drawing level Symbol level





floo

rpla

nsdi

agra

ms

electrical diagrams

floorplans

1,00

electrical diagrams

floorplans

i(1) = 0.496

i(1) = 0.529

score error (ε)p1

score error (ε)p1

i(ε

)

i(ε

)

Conclusion and perspectives

• Conclusion A new fuzzy way to evaluate object localization

distribution of matching cases regarding a “confidence rate” Experimentation with a real system

electrical and architectural drawings, 200 test images, 3821 symbols

• Perspectives Extending experiments

several systems, to add noise, scalability, real datasets

Documents

A Performance Characterization Algorithm for Symbol Localization