Turn angle function and elastic Turn angle function and elastic

time series matching time series matching 

Presented by: Wang , XinzhenAdvisor: Dr. Longin Jan Latecki

AgendaAgenda

•IntroductiIntroductionon•Turn angle functionTurn angle function

•Time series matchingTime series matching

•ConclusiConclusionon

•Future workFuture work

IntroductionIntroduction

• Data set: 1400 images (70 classes * 20 objects)Data set: 1400 images (70 classes * 20 objects)

1400 land mark sequences1400 land mark sequences

Data Preprocessing

IntroductionIntroductionAn example:An example:

Original image

Time series Landmark sequence

Introduction Introduction • Time sequence matchingTime sequence matching

Shape Matching

Sequence Matching

Introduction Introduction

• Problem definition: Problem definition:

Given a query q and a distance function d, find m nearest neighbors of q by calculatingthe distance using d. In our case, m is equalto 40.

Distance function d:

a = (x1 ,x2 , ... xn) b = (y1 ,y2 , ... yn)

Tangent space Tangent space representationrepresentation

• Shape description in tangent spaceShape description in tangent space

Simplified contour

Step function presentation

Problem

Turn angle functionTurn angle function• Modification to tangent space rep.Modification to tangent space rep.

Rotation (turning angle)Rotation (turning angle)

Scaling (normalization)Scaling (normalization)

Starting point (double length)Starting point (double length)

Landmark Landmark sequencesequence• From time series to land mark sequencesFrom time series to land mark sequences

Step I : compare each point with its left neighborStep I : compare each point with its left neighbor Step II : compare each point with its left and right neighborsStep II : compare each point with its left and right neighbors

• Disadvantages (ex: Disadvantages (ex: loss of informationloss of information))

Landmark sequence Landmark sequence matchingmatching

• Step I : Align the highest peak of the query with every peak of Step I : Align the highest peak of the query with every peak of the object, and then align other peaks and valleys of the the object, and then align other peaks and valleys of the query accordingly.query accordingly.

• Step II: Calculate the Euclidean distance between the Step II: Calculate the Euclidean distance between the peaks/valleys of query and object. As we move query along peaks/valleys of query and object. As we move query along the object, we have Euclidean distance for each alignment. the object, we have Euclidean distance for each alignment. A smallest Euclidean distance identifies the optimized A smallest Euclidean distance identifies the optimized alignment.alignment.

• Step III: In the optimized alignment, we introduce a penalty Step III: In the optimized alignment, we introduce a penalty distance if either query or object has extra peaks or valleys.distance if either query or object has extra peaks or valleys.

An example: Aligning query peaks and valleys with object (one optimized alignment)

Landmark sequence Landmark sequence matchingmatching

Query’s peaks

and valleys Object sequence(doubled)

Landmark sequence Landmark sequence matchingmatching

Another example : query has extra peaks and valleys

Back

• Penalty distancePenalty distance

Landmark sequence Landmark sequence matchingmatching

if the query has extra peaks/valleys, a penalty distance is added to the Euclidean distance between the query and object. The penalty distance is calculated by the sum of Euclidean distances between the unmatched peaks/valleys to the closest matched peaks/valleys in the query. See exampleIf instead the object has extra peaks/valleys, a penalty distance is calculated by the sum of Euclidean distances between the unmatched peaks/valleys to the closest matched peaks/valleys in the object.

• Query: The first Query: The first objectobject in the 1 in the 1stst class class

Some experimental resultsSome experimental results

• Search for 40 nearest neighbors in Search for 40 nearest neighbors in the whole dataset.the whole dataset.

• The top The top 40 matches 40 matches found.found.

• Retrieval Rate Retrieval Rate : 100%: 100%

Back

DefinitionDefinition

Retrieval Rate:

Since we have the prior knowledge about those objects within the same class as the query object, we can define the retrieval rate of matching as : RetrievalRate = N / 20 ( N: number of objects in the top 40 matches that belong to the same class as the query object)

Part MatchingPart Matching--- In a primitive stage--- In a primitive stage

• We manually select a significant part of We manually select a significant part of an objectan object, for example the leaves of an apple, and , for example the leaves of an apple, and proceed sub-sequence matching and retrieval proceed sub-sequence matching and retrieval

• Since our query part has only three peaks and Since our query part has only three peaks and three valleys, we define them as three valleys, we define them as LeftMostPeak/Valley, MiddlePeak/Valley, LeftMostPeak/Valley, MiddlePeak/Valley, RightMostPeak/Valley. RightMostPeak/Valley. See hereSee here. .

Back

LeftMostPeak/ValleyRightMostPeak/Valley

MiddlePeak/Valley

Part MatchingPart Matching• The Matching Scope in objectThe Matching Scope in object

The closest peak/valleyto the LeftMostPeak/Valley

The closest peak/valleyto the LeftMostPeak/Valley

• Step I : Calculate the Euclidean distance between the peaks/valleys Step I : Calculate the Euclidean distance between the peaks/valleys of the query part and object part. Only peaks/valleys fall between of the query part and object part. Only peaks/valleys fall between the matching scope in the object are considered for matching. the matching scope in the object are considered for matching.

• Step II: As we move query part along the object, we have Step II: As we move query part along the object, we have Euclidean distance for each alignment. A smallest Euclidean Euclidean distance for each alignment. A smallest Euclidean distance identifies the optimized alignment.distance identifies the optimized alignment.

• Step III: In the optimized alignment, we introduce a penalty Step III: In the optimized alignment, we introduce a penalty distance if either query part or object part has extra peaks distance if either query part or object part has extra peaks or valleys. The penalty distance calculation would be the or valleys. The penalty distance calculation would be the same as previous defined.same as previous defined.

Part MatchingPart Matching

• Query part: The first object in the 1Query part: The first object in the 1stst class class

Some experimental resultsSome experimental results

• Search for 40 nearest neighbors in Search for 40 nearest neighbors in the whole dataset.the whole dataset.

• Retrieval Rate : 60%Retrieval Rate : 60%

• False positivesFalse positives

False PositivesFalse PositivesObj626

Come pretty early in the 40

matches!

False PositivesFalse Positives

Looks like the leaf of an apple?

ConclusionConclusion

• It’s feasible to transform image contour data to time sequence.

• Landmark sequence can capture the important features of time series. Matching based on it is applicable and promising.

• Part Matching brings good result by submitting very limited query information.

Future workFuture work

• Order of Matching (Eliminate Order of Matching (Eliminate crossovercrossover))

• Combination of global matching with part Combination of global matching with part matching. matching.

• Apply the technique on the whole dataset.Apply the technique on the whole dataset.