Upload
constance-browning
View
92
Download
2
Tags:
Embed Size (px)
DESCRIPTION
Content Based Image Retrieval Using MPEG-7 Dominant Color Descriptor. Student: Mr. Ka-Man Wong Supervisor: Dr. Lai-Man Po MPhil Examination Department of Electronic Engineering City University of Hong Kong August 2004. Outlines of this presentation. Objectives MPEG-7 visual descriptors - PowerPoint PPT Presentation
Citation preview
1
Content Based Image Retrieval Using MPEG-7 Dominant Color Descriptor
Student : Mr. Ka-Man WongSupervisor : Dr. Lai-Man Po
MPhil ExaminationDepartment of Electronic Engineering
City University of Hong KongAugust 2004
2
Outlines of this presentation Objectives MPEG-7 visual descriptors A new similarity measure for dominant color descri
ptor Merged Palette Histogram Similarity Measure
A new relevance feedback for dominant color descriptor
Merged Palette Histogram Relevance Feedback MIRROR – A CBIR system using MPEG-7 visual
descriptors Conclusions
3
Objective of this research study To investigate Content Based Image Retrieval
(CBIR) based on color features To develop efficient techniques for MPEG-7
Dominant Color Descriptor (DCD) Merged Palette Histogram Similarity Measure Merged Palette Histogram Relevance Feedback
Apply proposed methods into a real system
4
MPEG-7 visual descriptors Color
Color structure, scalable color, dominant color, color layout
Texture Homogeneous texture, edge histogram, texture
browsing Shape
Contour shape, region shape, 3D shape Motion (for video contents)
Motion activity, camera motion, motion trajectory, parametric motion
They describe image/video contents in different aspects
5
MPEG-7 color descriptors Dominant color descriptor (DCD)
A compact color descriptor generated by color quantization
Color structure descriptor (CSD) Color histogram generated by structure block scanning a
pproach Scalable color descriptor (SCD)
Color histogram in a quantized HSV space with Haar transform.
Color layout descriptor (CLD) A compact color-spatial descriptor generated by dividing
the image by a 8x8 gird with DCT transform.
6
Relevance feedback Color might perform well, but it might not
match user’s expectation
Effectiveness could be further improved by involving users in the searching
7
MPEG-7 color descriptors Two major problems are found in DCD make it
unable to perform well Problems of its original similarity measure method Cannot use relevance feedback easily
We will focus on DCD in this research study New methods are developed to utilize DCD
Similarity measure Relevance feedback
8
Merged palette histogram similarity measure for dominant color descriptor
Dominant Color Descriptor Shortcomings of the existing similarity function Proposed Merged Palette Histogram Similarity Measure
9
Dominant Color Descriptor Feature representation
The dominant colors Percentage of area of the dominant color Maximum of 8 colors
(1) ),...,2,1(},,{ NipcF ii
Dominant Color Descriptor
percentage
color
10
Dominant Color Descriptor Feature extraction
GLA color quantization Each color have at least Td distance away in a perceptuall
y uniform CIELuv
Dominant Color Descriptor
Original Image
percentage
color
CQ
Color Quantized Image
11
Dominant Color Descriptor Similarity measure
A modified Quadratic Histogram Distance Measure (QHDM)
1 2 1 22 2 2
1 2 1 2 1 ,2 1 21 1 1 1
( , ) 2 (2)N N N N
i j i j i ji j i j
D F F p p a p p
Percentage p
color
Percentage q
color
12
Dominant Color Descriptor Since each DCD may have different set of
colors, QHDM is used to account for identical colors and similar colors.
1 2 1 22 2 2
1 2 1 2 1 ,2 1 21 1 1 1
( , ) 2 (2)N N N N
i j i j i ji j i j
D F F p p a p p
Percentage p
color
Percentage q
color
13
Shortcomings of the QHDM similarity function Limitations of QHDM
Distance upper bound is not fixed Completely different image cannot be identified by its upper
bound The similarity coefficient does not well model color simila
rity It does not balance between color distance and area of matc
hing The new Merged Palette Histogram Similarity Meas
ure method Can compare identical colors as well as similar colors Use area of matching for similarity measure
14
Proposed Merged Palette Histogram Similarity Measure MPHSM Process - 1
Find the closest pair of colors using Euclidian distance in CIELuv color space
2 2 21 2 1 2 1 2 1 2( , ) ( ) ( ) ( ) (3)d C C l l u u v v
15
Proposed Merged Palette Histogram Similarity Measure MPHSM process - 2
If the distance smaller than a threshold Td, merge them to form a new common palette color
Step 1 – 2 iterates until the minimum distance larger than Td
1 1 2 2( , )
1 2
(4)i i j jm i j
i j
p c p cc
p p
16
Proposed Merged Palette Histogram Similarity Measure MPHSM process - 3
A new common palette is then generated Form new descriptors based on the common palette
Dominant Color Descriptor
Common Palette
Merged Palette Histogram
17
Proposed Merged Palette Histogram Similarity Measure MPHSM process - 4
Histogram intersection is used to measure the similarity
Count the non-overlapping area as the distance
1 2 1 2 1 21 1
1( , ) 1 min( , ) (5)
2
m mN N
m m mi mi mi mii i
D F F p p p p
18
Experimental results MPHSM improves DCD for both datasets
While using Corel_1k dataset MPHSM outperforms QHDM significantly
*ANMRR (smaller means better)ANMRR (MPEG-7 CCD) ANMRR (Corel_1k)
DCD-MPHSM 0.2604 0.3946
DCD-QHDM 0.2834 0.5648
19
Experimental results Visual results - Query #32 from MPEG-7 CCD
Query image
QHDM results, ANMRR=0.4 MPHSM result, ANMRR=0.0111
20
Experimental results Visual results – Query #15 from Corel_1k
Query image
QHDM result, ANMRR=0.6464 MPHSM result, ANMRR=0.4819
21
Conclusions on Merged Palette Histogram Similarity Measure MPHSM generates a common palette Can match similar colors Uses area of matching as the similarity Boosts DCD in terms of ANMRR Gives better visual results
22
Merged palette histogram for dominant color descriptor relevance feedback
Feature weighting relevance feedback technique and its limitations Proposed Merged Palette Histogram Relevance Feedback Experimental results
23Feature weighting relevance feedback technique and its limitations Feature weighting relevance feedback
technique Assumes a fixed feature space (histograms) Taking liner combinations on matching histogram
bins. Simple approach: Histogram averaging
1' (7)
k
jij
i
pH
k
+( ) / 2 =
24Feature weighting relevance feedback technique and its limitations But DCDs of images might have different set of
colors, similar images might not have any exactly matched colors.
Two problems
H1 2
1
H2 2
1
H’4
1
25
Limitation of feature weighting relevance feedback technique Problems
The number of colors in updated query may greatly exceed the limit of the number of colors defined by MPEG-7 as the number of selected images increase.
Similar colors are separated. By definition of DCD, similar colors should be grouped together.
H1 2
1
H2 2
1
H’4
1
26
Limitation of feature weighting relevance feedback technique The Merged Palette Histogram Relevance
Feedback The updated query contains common colors among
selected images Represent the selected images efficiently
27
Proposed Merged Palette Histogram for Relevance Feedback Merged Palette Histogram Relevance Feedback
(MPH-RF) process - initialize Obtain all DCD from selected images
28
Proposed Merged Palette Histogram for Relevance Feedback Merged Palette Histogram Relevance Feedback
(MPH-RF) process - 1 Link all DCD together
+ + =
6 colors 8 colors 6 colors 20 colors
29
Proposed Merged Palette Histogram for Relevance Feedback Merged Palette Histogram Relevance Feedback
(MPH-RF) process - 2 Palette Merging
Find the closest pair of colors based on Euclidian distance in CIELuv
If minimum distance smaller than Td merge the color pair and sum up the percentages of merged colors
Iterate until minimum distance > Td1 1 2 2
1 1 2 2
1 1 2 2
(8)i i j jm
i j
m
w p c w p cc
w p w p
p w p w p
20 colors 9 colors
30
Proposed Merged Palette Histogram for Relevance Feedback Merged Palette Histogram Relevance Feedback
(MPH-RF) process - 3 Approximation
Cut the least significant colors if number of colors >8
9 colors 8 colors
31
Proposed Merged Palette Histogram for Relevance Feedback Merged Palette Histogram Relevance Feedback
(MPH-RF) process - 4 Re-normalization
Adjust the histogram sum into 1 An updated query is generated
Approximated MPH Updated QueryHistogram Sum =1
32
Experimental results MPH-RF gives improvement on all combinations of
similarity measures and datasets. Combination of MPHSM and MPH-RF gives a significant
improvement Three iterations of relevance feedback give a significant
result
*ANMRR – smaller means betterMPEG-7 CCD Corel_1k
Initial After 3 RFRF
ImprovementInitial After 3 RF
RF Improvement
DCD-MPHSM
0.2604 0.1752 0.0852 0.3946 0.3298 0.0648
DCD-QHDM
0.2834 0.2117 0.0717 0.5468 0.4900 0.0568
33
Experimental results Visual results – Query #50 from MPEG-7 CCD, MPHSM
Query image
Ground truth images
Initial retrieval, 4 of 8 ground truths hit, NMRR=0.5
First RF retrieval, 6 of 8 ground truths hit, NMRR=0.2782
Second RF retrieval, 7 of 8 ground truths hit, NMRR=0.1541
34
Experimental results Visual results – Query #13 from Corel_1k,
MPHSM
Query image
Ground truth images
Initial retrieval, 7 of 11 ground truths hit, NMRR=0.3043
First RF retrieval, 9 of 11 ground truths hit, NMRR=0.1688
Second RF retrieval, 9 of 11 ground truths hit, NMRR=0.1688
35
Conclusions on Merged Palette Histogram Relevance Feedback MPH-RF generates a new DCD query using
palette merging technique Represents the selected relevant images
naturally and effectively MPH-RF boosts all situations of DCD searching
36
MIRROR – A CBIR system using MPEG-7 visual descriptors
MPEG-7 Image Retrieval Refinement based On Relevance feedback Systems structure Demo
37
MIRROR – A CBIR system using MPEG-7 visual descriptors System structure
ImageDB
SimilarityMeasure
RelevanceFeedback
user initial input user feedback
MPEG-7data
Feature Extraction
reference image relevant image(s)
SimilaritySorting
Output Images
user feedback
-7
)
38
MIRROR – A CBIR system using MPEG-7 visual descriptors Demo
Demo 1: Similarity Measure
Demo 2: Relevance Feedback
http://www.ee.cityu.edu.hk/~mirror/
39
Conclusions of this research work By utilizing MPHSM and MPH-RF DCD, DCD
becomes compact as well as accurate Similarity measure
Merged Palette Histogram Similarity Measure Relevance Feedback
Merged Palette Histogram Relevance Feedback
Proposed methods are implemented into a real system.
CBIR functions Evaluation tools
40
Q & A