Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th...

Using Relevance Feedback in Multimedia Databases

Chotirat “Ann” Ratanamahatana

Eamonn Keogh

7th International Conference on VISual Information Systemsat 10th International Conference on Distributed Multimedia Systems

September 9, 2004

Roadmap

• Time series in multimedia databases and their similarity

measures

• Euclidean distance and its limitation

• Dynamic time warping (DTW)

• Global constraints and R-K Band

• Relevance Feedback and Query Refinement

• Experimental Evaluation

• Conclusions and future work

What are Time Series• A collection of observations made sequentially

in time.• People measure things…

and things…change over time…

• Their blood pressure• George Bush's popularity rating• The annual rainfall in San Francisco• The value of their Google stock

• Their blood pressure• George Bush's popularity rating• The annual rainfall in San Francisco• The value of their Google stock

Time Series in Multimedia Databases

Image data may best be thought of as time series…

Image to Time Series

Video to Time Series

0 50 100 150-1.5

-1

-0.5

0

0.5

1

1.5

Hand moving above holster

Hand moving downto grasp gun

Steady pointing

Hand moving toshoulder level

Hand at rest

Time Series in Multimedia Databases

0 10 20 30 40 50 60 70 80 90

Hand at rest

Aiming gun

Hand moving toshoulder level

Hand moving downto grasp gun

Hand movingabove holster

0 10 20 30 40 50 60 70 80 90

Hand at rest

Aiming gun

Hand moving toshoulder level

Hand moving downto grasp gun

Hand movingabove holster

0 10 20 30 40 50 60 70 80 90

Hand at rest

Aiming gun

Hand moving toshoulder level

Hand moving downto grasp gun

Hand movingabove holster

0 10 20 30 40 50 60 70 80 90

Hand at rest

Aiming gun

Hand moving toshoulder level

Hand moving downto grasp gun

Hand movingabove holster

0 10 20 30 40 50 60 70 80 90

Hand at rest

Aiming gun

Hand moving toshoulder level

Hand moving downto grasp gun

Hand movingabove holster

0 10 20 30 40 50 60 70 80 90

Hand at rest

Aiming gun

Hand moving toshoulder level

Hand moving downto grasp gun

Hand movingabove holster

0 10 20 30 40 50 60 70 80 90

Hand at rest

Aiming gun

Hand moving toshoulder level

Hand moving downto grasp gun

Hand movingabove holster

Video

0 50 100 150 200 250 300 350 400 4500

0.5

1

0 50 100 150 200 250 300 350 400 4500

0.5

1

George Washington’sManuscript

Classification in Time Series

Pattern Recognition is a type of supervised classification where an input pattern is classified into one of the classes based on its similarity to these predefined classes.

Class BClass BClass AClass A

Which class does

belong to?

Euclidean Distance MetricGiven 2 time series

Q = q1, …, qn and

C = c1, …, cn

their Euclidean distance is

defined as

n

iii cqCQD

1

2)(),(

0 50 100 150-1.5

-1

-0.5

0

0.5

1

1.5

0 50 100 150-1.5

-1

-0.5

0

0.5

1

1.5

Q

C

Limitations of Euclidean MetricVery sensitive to some distortion in the data

Training data consistsof 10 instances fromeach of the 3 classes

Training data consistsof 10 instances fromeach of the 3 classes

Perform a 1-nearest neighbor algorithm, with “leaving-one-out”

evaluation, averaged over 100 runs.

Euclidean distance Error rate:29.77%

DTW Error rate:3.33 %

Dynamic Time Warping (DTW)

Euclidean DistanceOne-to-one alignments

Time Warping DistanceNon-linear alignments are allowed

How Is DTW Calculated? (I)

QC

K

k kwCQDTW1

min),(

Warping path w

Q

C

Q

C

How Is DTW Calculated? (II)Each warping path w can be found using dynamic programming to evaluatethe following recurrence:

)}1,(),,1(),1,1(min{),(),( jijijicqdji ji

where γ(i, j) is the cumulative distance of the distance d(i, j) and its minimumcumulative distance among the adjacent cells.

(i-1, j)

(i, j-1)

(i, j)

(i-1, j-1)

Global Constraints (I)

C

Q

C

Q

C

Q

C

Q

Sakoe-Chiba Band Itakura Parallelogram

Prevent any unreasonable

warping

Prevent any unreasonable

warping

Global Constraints (II)

Ri

Sakoe-Chiba Band Itakura Parallelogram

A Global Constraint for a sequence of size m is defined by R, whereRi = d 0 d m, 1 i m.

Ri defines a freedom of warping above and to the right of the diagonal at any given point i in the sequence.

Ratanamahatana-Keogh Band (R-K Band)

Solution: we create an arbitrary shape and size of the band that is appropriate for the data we want to classify.

How Do We Create an R-K Band?First Attempt: We could look at the data and manually create the shape of the bands.

(then we need to adjust the width of each band as well until we get a good result)

50 100 150 200 250

50

100

150

200

250

50 100 150 200 250

50

100

150

200

250

0 50 100 150 200 250-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

0 50 100 150 200 250-2.5

-2

-1.5

-1

-0.5

0

0.5

1

100 % Accuracy!

Learning an R-K Band Automatically

50 100 150 200 250

50

100

150

200

250

0 50 100 150 200 250-2.5

-2

-1.5

-1

-0.5

0

0.5

1

0 50 100 150 200 250-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

50 100 150 200 250

50

100

150

200

250

Our heuristic search algorithm automatically learns the bands from the data.(sometimes, we can even get an unintuitive shape that give a good result.)

100 % Accuracy as well!

Calculate h(1)

Calculate h(2)

h(2) > h(1) ? Yes No

Calculate h(1)

Calculate h(2)

h(2) > h(1) ? Yes No

R-K Band Learning With Heuristic Search

R-K Band Learning in Action!

Classification Examples with R-K Bands

Error rate

Euclidean 32.13%

DTW 10% 4.52%

R-K Bands 0.9%

Face Classification

Relevance Feedback

• A well-known and effective method in improving the query performance, especially in text-mining domains.– Refining the query based on user’s reaction

• Only relatively little research has been done on relevance feedback in images or multimedia data.

Query Refinement

Averaging a collection of time series using DTW, according to their weights and warping (DTW) alignments.

Averaged SequenceAveraged Sequence

1. Gun Problem

2. Leaf Dataset

3. Handwritten Word Spotting data

Experiment: Datasets

0 50 100 150-1

-0.5

0

0.5

1

1.5

2

2.5

0 50 100 150-1

-0.5

0

0.5

1

1.5

2

2.5

Experimental Design

Given an initial query, we measure the precision and recall for each round of the relevance feedback retrieval.• Show the 10 best matches (k-nearest neighbors).• User ranks each result.• Accumulatively build the training set.• Learn an R-K band according to the current training data.• Generate a new query (query refinement), and repeat.

Results: Gun

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Recall

Pre

cisi

on

Iteration 1Iteration 2Iteration 3Iteration 4Iteration 5

Gun

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Recall

Pre

cisi

on

Iteration 1Iteration 2Iteration 3Iteration 4Iteration 5

Gun

Results: Leaf

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Recall

Pre

cis

ion

Iteration 1Iteration 2Iteration 3Iteration 4Iteration 5

Leaf

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Recall

Pre

cis

ion

Iteration 1Iteration 2Iteration 3Iteration 4Iteration 5

Leaf

Results: Wordspotting

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Recall

Pre

cisi

on

Iteration 1Iteration 2Iteration 3Iteration 4Iteration 5

WordSpotting 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Recall

Pre

cisi

on

Iteration 1Iteration 2Iteration 3Iteration 4Iteration 5

WordSpotting

Conclusions

• Different shapes and widths of the band contributes to the classification accuracy / precision.

• We have shown that incorporating R-K Band into relevance feedback can reduce the error rate in classification, and improve the precision at all recall levels in video and image retrieval.

Future Work

• Investigate other choices that may make envelope learning more accurate.– Heuristic functions– Search algorithm (refining the search)

• Is there a way to always guarantee an optimal solution?• Examine the best way to deal with multi-variate time

series for more complex data.• Explore other utilities of R-K Band and relevance

feedback, specifically on real-world problems: music, bioinformatics, biomedical data, etc.

UCR Time Series Data Mining Archive: http://www.cs.ucr.edu/~eamonn/TSDMA

Contact: ratana@cs.ucr.edu eamonn@cs.ucr.edu

Homepage: http://www.cs.ucr.edu/~ratana

All datasets are publicly available at: