An Approximate Nearest Neighbor Retrieval Scheme for

Computationally Intensive Distance Measures

Pratyush BhattMS by Research(CVIT)

Nearest Neighbor Retrieval

• Representation of an object– Fixed Length– Variable Length

• (Dis)similarity Function (Distance Measure)• Neighborhood

• Nearest neighbor retrieval problem can now be formalized as retrieving objects similar to a given object, where similarity is in accordance to a given similarity function

Need of NN retrieval schemes

• If the search space is small, Sequential search is applied for accurate results.

• With increase in memory, volume of data stored online has increased.– Sequential search is time consuming– Need to index data for fast retrieval– Birth of NN search algorithms

Computationally Expensive Distance Measures

• X1 more similar to X2 than X3 visually• L1 distance between X1 and X2 is more than X1 and X3

• Time complexity is super-linear to length of input.

• Edit Distance: O(d2)

Chamfer Distance

Nearest Neighbor Classification

• Computationally expensive distance functions– Nearest neighbor classifiers often impractical for real applications.– Takes over 20 minutes to classify a single object on a modern PC using an

optimized C++ implementation.– Larger the available training data, better will be the accuracy but at the

cost of high computation time.

Motivation(Approx. NN)

• Why to find similarity with all samples when decision is based on top K ?

• How to find top K without finding similarity with all the samples ?

• Solution: Compute K1 > K Approx. NN and find best K in that list.

• In order to compute K1-NN, use fewer explicit matches.

Problem Statement

• Improve nearest neighbor retrieval and classification performance in spaces with computationally expensive distance measures.

• Generate an expansible approximate nearest neighbor list for a given query.

• Dealing with points in non-metric space.

Metric Space

Metric Space

Tree based(KD-tree, R-Tree)

Hashing Based

KD Tree for Metric Space

Non-Metric Space

Wrist Rotation

Applications

• Classification based on K-NN– K is determined empirically

• Identification– Stop when similarity is above a fixed threshold

• Retrieval Applications– Optimizing network usage in peer-to-peer computer networks.– Content-based retrieval systems

• Concept of similarity is abstract– Need to generate expansible list– Learn from user feedbacks

Challenges

• Accuracy depends on the similarity function used to compute Approx. NN

• List can not be pre-computed, should be generated on-the-fly• Should be incremental to support scalability• Non-metric space

– Prohibit application of triangular inequality

Manifold Theory

},.....,,{ 21 nxxxO di Rx

},.....,,{ 21 nyyyE

)( dpR p

• Run MDS on similarity matrix to get

• Embedding of new sample is given by

mean of column of squared matrix requires computation of similarity of new sample to points

t

k

tk

tt

k

vvvL

.....

2

2

1

1#

)(# akLy

nnD

nnD

an

Related Work

• FastMap , Random Reference Objects, Random Line Projections, VP-Trees– Finds embedding of the query by computing only a few exact

distances – Assumption: Triangular Inequality

• BoostMap: Used AdaBoost to combine many simple 1-D embedding

FastMapLipschitz Embedding

Problem Formulation

• Goal : Compute approx. NN of a query point q, from a set S of N points in accordance to similarity function, F.

• Solution : – Split the data into a multi-level hierarchy– Exploit local similarity property to direct the search from top to

bottom

Hierarchical Local Maps(HLM)

1. FIND SIMILARITY OF QUERY WITH POINTS AT TOPMOST LEVEL

2. IDENTIFY NEAREST NEIGHBORS3. GET CHILDREN OF NEAREST NEIGHBORHOW TO FIND SIMILARITY OF QUERY

WITH THESE SAMPLES WITHOUT EXPLICITLY CALCULATING IT ?

4. PROJECT POINTS ON THE MANIFOLD

5. PROJECT QUERY ON THE MANIFOLD

6. FIND NEAREST NEIGHBOURS OF QUERY IN

THIS METRIC SPACE

7. IDENTIFY THESE POINTS IN THE TREE8. TRAVERSE DOWN IN

SAME FASHION

Results

• Unipen Handwriting Database– 15953 digit examples– Divided into training and testing set with 2:1 ratio– Distance Measure : DTW

Number of DTW Computations for K nearest neighbor retrieval

Number of DTW Computations for K nearest neighbor retrieval

Number of DTW Computations for K nearest neighbor retrieval

Classification Accuracy on UNIPEN Dataset using exact and approximate k-NN

Classification Accuracy

K 1 5 10 15 20

DTW 98.1 97.73 97.41 96.99 96.83

HLM 97.65 97.27 97.08 96.69 96.44

Difference 0.45 0.46 0.33 0.30 0.39

• Average No. of DTW computations done by HLM : 160• Expected No. of DTW computations done in brute-force : 5315

Biometric ( Special Case)

• High Inter-Class Variation

• Low Intra-Class Variation

• Low variation in inter-class distances

• Indexing for identification

• How to apply HLM in such cases ??

– Local similarity structure becomes degenerate destroying any manifold structure

Biometric Data

High dimensional data

Relative Contrast :

Iris

• Feature Vector Length 1000• If same class- < 100 bit differ • Else equal probability of each bit to match or not match

– On average 500 bit differ

• Imposter Scores would be around 0.5

Biometric Data

• Such Distribution is bad for Indexing

Softness/Hardness

• Measure of overlap between genuine and imposter classes• Soft Biometric (Face, Body Silhouette)

– Poor Classification Accuracy– Better indexing– Correlates to multi dimensional point

• Hard Biometric (Iris, Fingerprint)– Good Classification Accuracy– Poor indexing– Correlates to high dimensional point

• Need a balance between two

Dataset

• CASIA Iris Image Database V3.0– 855 images corresponding to 285 users in training and testing set– 3 samples per eye

• Simlarity Function– Hamming Distance

– Euclidean Distance (Softer Metric)• Average gray value of a block resulted in 160D feature vector

• Penetration Rate– Percentage of data on which we ran biometric matcher.

• False Reject Rate– Percentage of identification instances in which false rejection occurs

HLM on CASIA Iris Dataset

Synthetic Dataset

• Class center sampled from 1D gaussian (0,1)• Generate d-dimensional by sampling d times• Points of same class sampled from gaussian with mean as

class centers and varying variance.• Total Number of classes: 500• Points in same class

– Training : 10– Testing : 5

Indexing performance varying number of dimensions

Indexing performance varying within class to between class variance ratio

Contributions

• A representation scheme for objects in a dataset that allows for fast retrieval of approximate nearest neighbors in non-euclidean space.

• Search mechanism combined with filter and refine approach is proposed that minimizes the number of exact distance computations for computationally expensive distance measure.

• Study performance of our scheme on biometric data and study the parameters affecting its performance.

Conclusion and Future Work

• Local Similarity Property is well exploited by HLM• Incremental and Scalable• Softer biometric in filtering step combined with hard

biometric in refine step would drastically reduce computation time

• Optimal construction of HLM• Defining a measure for similarity function that allows

hierarchical representation.• Learn a function to find degree of indexibility

– Extract parameters from data distribution and similarity function

Thank You

Related Publication:• Pratyush Bhatt and Anoop Namboodiri “Hierarchical Local

Maps for Robust Approximate Nearest Neighbour Computation” Proceedings of the 7th International Conference on Advances in Pattern Recognition (ICAPR 2009), Feb. 4-6, 2009, Kolkatta, India.