From graph matching problem to Assignment problem ...romain.raveaux.free.fr/document/FromGraphMatchingToAssignment… · Graphs in pattern recognition Assignment problem in graph

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

From graph matching problem to Assignmentproblem :

Application to Content-Based Image Retrieval.

Romain Raveaux

LI lab (EA 6300) – Universite de Tours

Rouen, 1rst December, 2013

1 / 38



Problem statementGraph matching problemsState of the art

Graphs are everywhere...

Graphs in Reality

Graphs model objects andtheir relationships.

Also referred to asnetworks.

All common datastructures can be modeledas graphs.

2 / 38





Graphs in Reality




2 / 38





Graphs in Reality




2 / 38




Definition of a graph

Definition

Let LV and LE denote the set of node and edge labels, respectively.A labeled graph G is a 4-tuple G = (V ,E , µ, ξ) , where

V is the set of nodes,

E ⊆ V × V is the set of edges

µ : V → LV is a function assigning labels to the nodes, and

ξ : E → LE is a function assigning labels to the edges.

Let G1 = (V1,E1, µ1, ξ1) be the source graph

And G2 = (V2,E2, µ2, ξ2) the target graph

With V1 = (u1, ..., un) and V2 = (v1, ..., vm) respectively

3 / 38




Definition and notation of a graph

Labels for both nodes and edges can be given by :

Labelling functions

a set of integers L = {1, 2, 3, . . .}a vector space L = Rn

a finite set of symbolic labels L = {x,y, z, . . .}...

Definition 1 allows us to handle arbitrarily structured graphs withunconstrained labelling functions.

4 / 38




Graph comparison

How similar are two graphs?

Graph similarity is the central problem for all learning taskssuch as clustering and classification on graphs.

At least two ways for comparing graphs

Graph Embedding

Graph Matching

5 / 38




Graph comparison




Graph Embedding

Graph Matching

5 / 38




Graph comparison




Graph Embedding

Graph Matching

5 / 38




Graph isomorphism

Graph isomorphism

Find a mapping f : V1 → V2

i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1

f is an isomorphism iff (f (x), f (y)) is an edge of G2.No polynomial-time algorithm is known for graph isomorphismNeither it is known to be NP-complete

Subgraph isomorphism

Means finding a subgraph G3 of G2 such that G1 and G3 areisomorphic.Subgraph isomorphism is NP-complete

6 / 38




Graph isomorphism

Graph isomorphism


i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1




6 / 38




Graph isomorphism

Graph isomorphism


i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1




6 / 38




Graph isomorphism

Graph isomorphism


i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1




6 / 38




Graph isomorphism

Graph isomorphism


i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1




6 / 38




Graph isomorphism

Graph isomorphism


i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1




6 / 38




Graph isomorphism

Graph isomorphism


i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1




6 / 38




Graph isomorphism

Graph isomorphism


i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1




6 / 38




Graph isomorphism

Graph isomorphism


i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1




6 / 38




Graph isomorphism families

1 Exact Matching : Exact mapping between vertex and/oredge labels.

2 Inexact Matching : No exact mapping between vertexand/or edge labels can be found, but when the mapping canbe associated to a cost.

In pattern recognition

Vertices and edges are labeled with measures/features whichmay be affected by noise.

Direct and exact comparisons are useless.

The matching problem turns from a decision one to anoptimization one.

7 / 38




Inexact matching

Three sub-problems :

1 Substitution-Tolerant Subgraph Isomorphism

2 Inexact Subgraph Isomorphism

3 Inexact Graph Isomorphism

8 / 38




Substitution-Tolerant Subgraph Isomorphism

1 This formulation allows differences between labels.2 The mapping does not affect the topology.3 The mapping should be edge-preserving in one direction.

V’ V E E’

f(v) f-‐1(v’)

f

0.5

0.3

0.15 0.13

Figure: Substitution-tolerant subgraph isomorphism problem

9 / 38




Inexact Subgraph Isomorphism

1 This formulation allows differences between labels.2 Topology can be different. The mapping is not

edge-preserving.

V’ V E E’

f(e)

f

0.5

0.3

0.13

Δe’

0.16

Unmatched

Matched

Figure: Inexact Subgraph Isomorphism problem

10 / 38




Inexact Graph Isomorphism

1 Bijective function2 This formulation allows differences between labels.3 Topology can be different.

V’ V E E’

f

0.5

0.3

0.13

Δe’

0.16

Δe Δv

Figure: Inexact graph isomorphism problem

11 / 38




Graph Matching problems

Figure: Graph isomorphism constraints

12 / 38




Comparison between Classical Graph-Matching Methods

Name Graphmatchingproblem

Models For-mulation

Optimality Complexityclass

Deterministic Optimizationmethod

References

NodeAssignment

IGI Approximate Sub-optimal Polynomial Deterministic Hungarian al-gorithm

[Riesen 2009],[Raveaux 2010],[Jouili 2009] ,[Shokoufandeh 2006]

Neural Net-work

ISGI Approximate Sub-Optimal Polynomial Non-Deterministic

Gradient [Kuner 1988],[Suganthan 2000]

Stringbasedmethods

ISGI Approximate Sub-Optimal Polynomial Deterministic Tree Search [J. Llados 2001],[Anjan Dutta 2012]

ApproximatedGED

IGI Exact Sub-Optimal NP-Complete Deterministic Heuristic Treesearch

[Neuhaus M 2006]

Genetic al-gorithm

ISGI Exact Sub-Optimal Polynomial Non-Deterministic

Genetic Algo-rithm

[A.D.J. Cross 1997],[Ford 1991],[Jiang 1998]

ProbabilisticRelaxation

ISGI Exact Sub-Optimal Polynomial Deterministic ProbabilisticRelaxation

[W. Christmas 1995],[Finch 1998],[Gold 1996]

Similaritysearch

MIGI Exact Sub-Optimal Polynomial Non-Deterministic

Greedy search,Tabu Search

[Solnon 2005]

GED IGI Exact Optimal NP-Complete Deterministic Tree search [Bunke 1998],[Bunke 1999]

GED ILP IGI Exact Optimal NP-Complete Deterministic MathematicalSolver

[Justice D 2006]

ILP STSI Exact Optimal NP-Complete Deterministic Mathematicalsolver

[LeBodic 2012]

Indexingmethod

ISGI Exact Optimal NP-Complete Deterministic Indexed search [Messmer 1998]

SpectralTheory

STSI Exact Optimal Polynomial Deterministic Eigen Decom-position

[Ferrer 2005] ,[Umeyama 1988].

Similaritysearch

MIGI Exact Optimal NP-Complete Deterministic Branch AndBound

[Champin ]

13 / 38




Some highlights

Exact graph matching is useless in many computer visionapplications

Concerning graph matching under noise and distortion

The matching incorporates an error model to identify thedistortions which make one graph a distorted version of theother

Problems for real world applications

Error-tolerant

To measure the similarity of two graphs.

Runtime may grow exponentially with number of nodes

This is an enormous problem for large datasets of graphs

Wanted: Polynomial-time similarity measure for graphs

14 / 38




Some highlights





Error-tolerant





14 / 38




Some highlights





Error-tolerant





14 / 38




Some highlights





Error-tolerant





14 / 38




Some highlights





Error-tolerant





14 / 38




Some highlights





Error-tolerant





14 / 38




Some highlights





Error-tolerant





14 / 38




Some highlights





Error-tolerant





14 / 38




Some highlights





Error-tolerant





14 / 38



DefinitionAssignment-based Matching

Assignment problem

Jacobi (1890) exposes a polynomial time algorithm to solve thefollowing problem:

Let ai ,j be a square matrix nxn, find a permutation σ suchthat

∑ai ,σ(i) be maximal.

Such an algorithm was only rediscovered in 1955 par Kuhn,who called it “Hungarian method”.

It is a well-known problem in mathematical economy, underthe name of “assignment problem”.

15 / 38




Assignment problem

Say you have three workers: Jim, Steve and Alan. You need tohave one of them clean the bathroom, another sweep the floorsand the third wash the windows. What’s the best (minimum-cost)way to assign the jobs? First we need a matrix of the costs of theworkers doing the jobs.

Clean bathroom Sweep floors Wash windows

Jim 1 e 2 e 3 eSteve 3 e 3 e 3 eAlan 3 e 3 e 2 e

Table: Cost matrix

Then the Hungarian algorithm, when applied to the above tablewould give us the minimum cost it can be done with: Jim cleansthe bathroom, Steve sweeps the floors and Alan washes thewindows.

16 / 38




Graph comparison through assignment problem

Basic idea:

Methods are based on an optimization procedure mappinglocal substructures

Any node(un) from G1 can be assigned to any node(vm) of G2,

Incurring some cost that depends on the un-vm assignment.

It is required to map all nodes in such a way that the totalcost of the assignment is minimized.

Cost matrix representation (C ):

Cij correspond to the costs of assigning the i th node of G1 tothe j th node of G2.

17 / 38





Basic idea:







17 / 38





Basic idea:







17 / 38





Basic idea:







17 / 38





Basic idea:







17 / 38





Basic idea:

Methods are based on an optimization procedure mappinglocal substructuresAny node(un) from G1 can be assigned to any node(vm) of G2,Incurring some cost that depends on the un-vm assignment.It is required to map all nodes in such a way that the totalcost of the assignment is minimized.



17 / 38





Basic idea:

Methods are based on an optimization procedure mappinglocal substructuresAny node(un) from G1 can be assigned to any node(vm) of G2,Incurring some cost that depends on the un-vm assignment.It is required to map all nodes in such a way that the totalcost of the assignment is minimized.



17 / 38





Univalent matching

Sets of equal size

Find a bijection b : G1 → G2

Given two sets of subgraphs, G1 and G2 of equal size, the problemcan be expressed as a standard linear program with the objectivefunction Eq 1 to be minimized.∑

i∈G1

∑j∈G2

C (i , j)xij (1)

The variable xij represents the assignment of a subgraph of G1 to asubgraph of G2, taking value 1 if the assignment is done and 0otherwise.

18 / 38




Graph comparison through assignment problem: state ofthe art

Node signature Distance

[Gold 1996] Node degree+Label *

[Shokoufandeh 2006] Eigen vector L2

[Riesen 2009] (1)Node+(2)Edge Edit cost

[Jouili 2009] Node degree+Weight L1

[Raveaux 2010] Sub Graph Edit Distance*: Depends on the graph attribute type.

19 / 38




Graph comparison through assignment problem: anexample

A subgraph (sg):

A structure gathering theedges and theircorresponding endingvertices from a rootvertex.

Romain Raveaux et al. Agraph matching methodand a graph matchingdistance based onsubgraph assignments.Pattern RecognitionLetters (2010)

Figure: Decomposition intosubgraph world

20 / 38




Graph comparison through assignment problem: anexample

A subgraph (sg):

A structure gathering theedges and theircorresponding endingvertices from a rootvertex.

Romain Raveaux et al. Agraph matching methodand a graph matchingdistance based onsubgraph assignments.Pattern RecognitionLetters (2010)

Figure: Decomposition intosubgraph world

20 / 38




Graph comparison through assignment problem: anexample II

A distance between graph

Subgraph decomposition

Optimization algorithm

Figure: Subgraph matching: A bipartite graph

21 / 38









21 / 38









21 / 38




Synthesis

Inexact graph matching problem

Univalent matching

Polynomial algorithm : O(n3)

Two main directions

Node signature based methods (vectors).Subgraph based methods. (not a dot in a feature space.)

22 / 38



CBIR problemCBIR methodExperiments

How Graph Matching can be used in Pattern RecognitionProblems

Hypothesis :

a graph database consisting of n graphs. D = g1, g2, ..., gn

a query graph q

Problem definitions :1 Full graph search. Find all graphs gi in D s.t. gi is the same

as q.2 Subgraph search. Find all graphs gi in D containing q or

contained by q.3 Similarity search. Find all graphs gi in D s.t. gi is similar to

q within a user-specified threshold based on some similaritymeasures.

4 Graph mining. Graph mining problem gathers similar graphor subgraph of D in order to find clusters or prototypes.

23 / 38




CBIR problem

Figure: Feature clustering stage

24 / 38




CBIR scheme

Figure: Feature clustering stage

25 / 38




CBIR scheme

Figure: Images to graphs

26 / 38




CBIR scheme

Figure: Similarity search 27 / 38




(a) (b)

Figure: Regions of Interest found by the SIFT algorithm. ProcessingSIFT took 407ms, 60 features were identified and processed

28 / 38




Figure: A segmentation result. Processing SRM took 1625ms and 26features were identified and processed

29 / 38




Figure: Multiple representations 30 / 38




Figure: CBIR approach categorization according to the amount of spatialinformation captured

31 / 38




Figure: CBIR taxonomy

32 / 38




Figure: Image Samples from the Caltech-101 Data set. The 101 objectcategories and the background clutter category.

33 / 38




Figure: Columbia University Image Library

34 / 38




Recognition rate

(a) Coil-100 (b) Caltech-101

Figure: Comparison between CBIR methods. Summary of resultsobtained with the best number of words for each method.

Romain Raveaux et al: Structured representations in a content based image retrieval context. J. Visual

Communication and Image Representation (2013). Elsevier.

35 / 38




Vocabulary size impact

(a) Coil-100 (b) Caltech-101

Figure: Comparison between the number of words in used by themethods.

Romain Raveaux et al: Structured representations in a content based image retrieval context. J. Visual

Communication and Image Representation (2013). Elsevier.

36 / 38




Time experiments

Average response time in seconds SIFTBoW IFFSBoW IFFSGBR

Coil − 100 28 37 1555

Table: Average response time of a query. A speed comparison of the fourmethods at the worst case.

37 / 38



Conclusion

To take the stock of :

SIFTBoW is better on Caltech data

IFFS is better on Coil-100

The Graph Based Approach gave similar recognition rate thanBoW methods.

A structural approach requires a fewer number of words toreach its best performance.

Future work

To avoid visual features clustering.

To enrich the graph representation.

Combination of BoW and GbR : BoW first and then to processa re-ranking stage with the top k responses based on GbR.

38 / 38



R.C. Wilson A.D.J. Cross and E.R. Hancock.Inexact Graph Matching Using Genetic Search.Pattern Recognition, pages 953–970, 1997.

Josep Llados Anjan Dutta and Umapada Pal.A symbol spotting approach in graphical documents byhashing serialized graphs.Pattern Recognition, pages 752–768, 2012.

Horst Bunke and Kim Shearer.A graph distance metric based on the maximal commonsubgraph.Pattern Recognition Letters, vol. 2, page 255–259, 1998.

Horst Bunke.Error Correcting Graph Matching: On the Influence of theUnderlying Cost Function.

38 / 38



IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, no. 9, pages917–922, 1999.

Pierre-antoine Champin and Christine Solnon.Measuring the similarity of labeled graphs.

Miquel Ferrer, Francesc Serratosa and Alberto Sanfeliu.Synthesis of Median Spectral Graph.In IbPRIA (2), pages 139–146, 2005.

et al. Finch Wilson.An energy function and continuous edit process for graphmatching.Neural Computat, vol. 10, 1998.

G.P. Ford and J. Zhang.A Structural Graph Matching Approach to ImageUnderstanding.

38 / 38



Intelligent Robots and Computer Vision X: Algorithms andTechniques, pages 559–569, 1991.

S. Gold and A Rangarajan.A graduated assignment algorithm for graph matching.Workshops SSPR and SPR, pages 377 – 388, 1996.

E. Marti J. Llados and J. J. Villanueva.Symbol recognition by error-tolerant subgraph matchingbetween region adjacency graphs.Patt. Anal. Mach., pages 1137–1143, 2001.

X. Jiang and H. Bunke.Marked subgraph isomorphism of ordered graphs.Workshops SSPR and SPR, pages 122–131, 1998.

Salim Jouili and Salvatore Tabbone.Graph Matching Based on Node Signatures.In GbRPR, pages 154–163, 2009.

38 / 38



Hero A Justice D.A binary linear programming formulation of the graph editdistance.IEEE Trans Pattern Anal Mach Intell., vol. 28, page1200–1214, 2006.

P Kuner and B Ueberreiter.Pattern Recognition by Graph Matching: Combinatorial versusContinuous Optimization.International Journal in Pattern Recognition and ArtificialIntelligence, vol. 2, page 527–542, 1988.

P. LeBodic and H. Heroux.An integer linear program for substitution-tolerant subgraphisomorphism and its use for symbol spotting in technicaldrawings.Pattern Recognition, pages 4214–4224, 2012.

38 / 38



B. Messmer and Bunke.Error-Correcting Graph Isomorphism using Decision Trees.Int. Journal of Pattern Recognition and Ar, vol. 12, page721–742, 1998.

Bunke H Neuhaus M.A convolution edit kernel for error-tolerant graph matching.Proceedings of IEEE international conference on patternrecognition., vol. 30, pages 220–223, 2006.

Burie J. Raveaux R. and Ogier.A graph matching method and a graph matching distancebased on subgraph assignments.Pattern Recognition Letters, page 394–406, 2010.

Kaspar Riesen and Horst Bunke.Approximate graph edit distance computation by means ofbipartite graph matching.

38 / 38



Image Vision Comput., vol. 27, no. 7, pages 950–959, 2009.

Bretzner L. Macrini D. Fatih Demirci M. Jonsson C.Shokoufandeh A. and S. Dickinson.The representation and matching of categorical shape.Computer Vision and Image Understanding, page 139–154,2006.

Christine Solnon.Reactive Tabu Search for Measuring Graph.pages 172–182, 2005.

P. N. Suganthan.Attributed relational graph matching by neural-gas networks.IEEE Signal Processing Society Workshop on Neural Networksfor Signal Processing X, pages 366–374, 2000.

S. Umeyama.

38 / 38



An eigendecomposition approach to weighted graph matchingproblems.IEEE Pattern Anal. Mach. Intel, page 695–703, 1988.

J. Kittler W. Christmas and M. Petrou.Structural matching in computer vision using probabilisticrelaxation.IEEE Trans. PAMI,, vol. 2, page 749–764, 1995.

38 / 38

Documents

From graph matching problem to Assignment problem ...romain.raveaux.free.fr/document/FromGraphMatchingToAssignment… · Graphs in pattern recognition Assignment problem in graph