Upload
laasya
View
41
Download
0
Embed Size (px)
DESCRIPTION
j. i. d. d. l. k. j. j. j. i. i. i. p. =. (. p. ;. p. ). ,. p. =. (. p. ;. p. ). k. k. ;. 1. k. ;. 2. l. l. ;. 1. l. ;. 2. Image Matching and Retrieval by Repetitive Patterns Petr Doubek, Jiri Matas, Michal Perdoch and Ondrej Chum - PowerPoint PPT Presentation
Citation preview
Image Matching and Retrieval by Repetitive PatternsPetr Doubek, Jiri Matas, Michal Perdoch and Ondrej Chum
Center for Machine Perception, Czech Technical University in Prague, Czech Republic
Detection of repetitive patterns in images is a well-established computer vision problem. However, the detected patterns are rarely used in any application. A method for representing a lattice or line pattern by shift-invariant descriptor of the repeating tile is presented. The descriptor respects the inherent shift ambiguity of the tile definition and is robust to viewpoint change. Repetitive structure matching is demonstrated in a retrieval experiment where images of buildings are retrieved solely by repetitive patterns.
Motivation• Multiple occurrences of a local patch pose a problem to one-to-
one matching algorithms (local matches are ambiguous)• The presence of repeated local patches is in most cases non-
accidental and therefore very distinctive• Regular and even non-regular repetitive patterns give a rise to
geometric constraints
Repetitive Pattern and Lattice Detection
Original image Detected lattice Detected tiles Zero-phased tiles
Shift-Invariant Tile Representation• Each repeated pattern is represented by an “average” appearance of a tile – a mean tile(a more complex representation possible)• After discovering lattice of a repeated pattern an intrinsic shift ambiguity remains• We propose two shift-invariant representations:
• the magnitude of Fourier coefficients of a tile• “zero-phase” normalization: tile shifted so that phase of the first harmonic equals zero
Outline of the AlgorithmDetection1. Detect repeated elements, find the lattice of the repeated
pattern, rectify the lattice and calculate the mean tile2. Compute shift invariant tile descriptorsImage Matching and Retrieval3. For each pair score with the most similar patterns
1. Detection of repeated elements. In our implementation affine covariant regions (MSERs and Hessian Affine) described by SIFT
2. Agglomerative clustering of SIFTs. Each cluster hypothesise a repeated pattern
3. For each element in the repeated pattern, find spatial nearest neighbours in each of the spatial sectors
4. Find dominant vanishing points by Hough transform and form a 2D lattice
5. Rectify lattice and divide pattern into tiles
A cluster of repeated elements
Nearest neighbours and neighbourhood sectors
A lattice from two vanishing points (corresponding to red
and green directions)
• For a pair of images i,j with sets of detected repeated patterns Ci and Cj similarity sk,l of two patterns k,l is computed as
where and are shift invariant tile descriptions and are pairs of peaks in RGB colour histograms.
Repeated Patterns Similarity
Image Retrieval by Repetitive Patterns
Dataset Query Top three best matches .
Experiment 2• Detection and matching of repeated patterns tested on image retrieval• Two publicly available datasets PSU-NRT(subset) and Pankrac+Marseille (
http://cmp.felk.cvut.cz/data/repetitive)• Top three matches for some of the queries
Experiment 1• Performance of shift invariant representation• Ground truth for each query Gi = set of images of
building i, was manually marked
• Tested on two datasets with ~230 images in image retrieval of about 50 buildings
Conclusions• Image retrieval can benefit from repeated patterns if
they are detected and handled properly• Proposed approach is able to detect 1D and 2D
lattices under affine transformation• Shift invariant descriptors addresses tile ambiguity• We have shown retrieval based solely on repeated
patterns, however it can be combined with standard bag-of-words retrieval approaches
References• T. Tuytelaars, A. Turina, and L. Van Gool, “Non-combinatorial
detection of regular repetitions under perspective skew”, PAMI, vol.25, no.4, pp. 418-432, April 2003
• P. Doubek, J. Matas, M. Perdoch and O. Chum, “Detection of 2D lattice patterns of repetitive elements and their use for image retrieval”, technical report, CTU-CMP-2009-16, 2009
• T. K. Leung and J. Malik, “Detecting, localizing and grouping repeated scene elements from an image”, ECCV, 1996, pp. 546-555
The authors were supported by Czech Science Foundation Project 102/07/1317 and by EC project FP7-ICT-247022 MASH.
Example: repeated patterns of two images and their similarity
pik = (pik;1;pik;2), pjl = (pjl;1;pjl;2)dik djl
Abstract
0.56 0.37 0.07 0.08 0.06 0.42
0.00 0.01 0.00 0.00 0.03 0.02
10x10 43px 12x10 27px 6x9 39px 2x6 53px 16x1 15px 7x11 63px
12x10 57px
17x4 73px