Similarities, Distances and Manifold Learning

Prof. Richard C. Wilson

Dept. of Computer ScienceUniversity of York

Background

• Typically objects are characterised by features– Face images

– SIFT features

– Object spectra

– ...

• If we measure n features → n-dimensional space

• The arena for our problem is an n-dimensional vector space

Background

• Example: Eigenfaces

• Raw pixel values: n by m gives nm features

• Feature space is space of all n by m images

Background

• The space of all face-like images is smaller than the space of all images

• Assumption is faces lie on a smaller manifold embedded in the global space

All images

Face images

Manifold: A space which locally looks Euclidean

Manifold learning: Finding the manifold representing the objects we are interested in

All objects should be on the manifold, non-objects outside

Part I: Euclidean SpacePosition, Similarity and Distance

Manifold Learning in Euclidean space

Some famous techniques

Part II: Non-Euclidean ManifoldsAssessing Data

Nature and Properties of Manifolds

Data Manifolds

Learning some special types of manifolds

Part III: Advanced TechniquesMethods for intrinsically curved manifolds

Thanks to Edwin Hancock, Eliza Xu, Bob Duin for contributionsAnd support from the EU SIMBAD project

Part I: Euclidean Space

Position

The main arena for pattern recognition and machine learning problems is vector space– A set of n well defined features collected into a vector

ℝn

Also defined are addition of vectors and multiplication by a scalar

Feature vector → position

Similarity

To make meaningful progress, we need a notion of similarity

Inner product

• The inner-product ‹x,y› can be considered to be a similarity between x and y

i

ii yxyx,

Induced norm

• The self-similarity ‹x,x› is the (square of) the ‘size’ of x and gives rise to the induced norm, of the length of x:

• Finally, the length of x allows the definition of a distance in our vector space as the length of the vector joining x and y

• Inner product also gets us distance

xxx ,

yxyxyxyx ,),(d

Euclidean space

• If we have a vector space for features, and the usual inner product, all three are connected:

),( Distance

Similarity

, Position

yx

yx

yx

d

,

non-Euclidean Inner Product

• If the inner-product has the form

• Then the vector space is Euclidean

• Note we recover all the expected stuff for Euclidean space, i.e.

• The inner-product doesn’t have to be like this; for example in Einstein’s special relativity, the inner-product of spacetime is

i

iiT yxyxyx,

2222

211

21

22

21

)()()(),( nn yxyxyxd

xxx

yx

x

44332211, yxyxyxyx yx

The Golden Trio

• In Euclidean space, the concepts of position, similarity and distance are elegantly connected

PositionX

SimilarityK

DistanceD

Point position matrix

• In a normal manifold learning problem, we have a set of samples X={x1,x2,...,xm}

• These can be collected together in a matrix X

Tm

T

T

x

x

x

X2

1

I use this convention, but othersmay write them vertically

Centreing

A common and important operation is centreing – moving the mean to the origin– Centred points behave better

is the mean matrix, so is the centred matrix

– J is the all-ones matrix

This can be done with C

– C is the centreing matrix (and is symmetric C=CT)

CXXJIC / m

m/JX m/JXX

Position-Similarity

• The similarity matrix K is defined as

• From the definition of X, we simply get

• The Gram matrix is the similarity matrix of the centred points (from the definition of X)

– i.e. a centring operation on K

• Kc is really a kernel matrix for the points (linear kernel)

PositionX

SimilarityK

CKCCCXXK TTc

jiijK xx ,

TXXK

Position-Similarity

• To go from K to X, we need to consider the eigendecomposition of K

• As long as we can take the square root of Λ then we can find X as

PositionX

SimilarityK

T

T

XXK

UUK

Λ

1/2ΛUX

Kernel embedding

First manifold learning method – kernel embedding

Finds a Euclidean manifold from object similarities

• Embeds a kernel matrix into a set of points in Euclidean space (the points are automatically centred)

• K must have no negative eigenvalues, i.e. it is a kernel matrix (Mercer condition)

1/2ΛUX TUUK Λ

Similarity-Distance

SimilarityK

DistanceD

ijsijjjii

jijjii

jijiji

DKKK

d

,

2

2

,2,,

,),(

xxxxxx

xxxxxx

• We can easily determine Ds from K

Similarity-Distance

What about finding K from Ds ?

Looking at the top equation, we might imagine that

K=-½ Ds is a suitable choice

• Not centred; the relationship is actually

CCDK s2

1

ijjjiiijs KKKD 2,

Classic MDS

• Classic Multidimensional Scaling embeds a (squared) distance matrix into Euclidean space

• Using what we have so far, the algorithm is simple

• This is MDS

kernel theEmbed Λ

kernel theposeEigendecom Λ

kernel theCompute 2

1

1/2UX

KUU

CCDK

T

s

PositionX

DistanceD

The Golden Trio

PositionX

SimilarityK

DistanceD

Kernel EmbeddingMDS

ijjjiiijs

s

KKKD 22

1

,

CCDK

Kernel methods

• A kernel is function k(i,j) which computes an inner-product

– But without needing to know the actual points (the space is implicit)

• Using a kernel function we can directly compute K without knowing XPosition

X

SimilarityK

DistanceD

jijik xx ,),(

Kernel function

Kernel methods

• The implied space may be very high dimensional, but a true kernel will always produce a positive semidefinite K and the implied space will be Euclidean

• Many (most?) PR algorithms can be kernelized– Made to use K rather than X or D

• The trick is to note that any interesting vector should lie in the space spanned by the examples we are given

• Hence it can be written as a linear combination

• Look for α instead of u

αX

xxxuT

mm

2211

Kernel PCA

• What about PCA? PCA solves the following problem

• Let’s kernelize:

XuXu

Σuuu

u

u

TT

T

n

1minarg

minarg*

αKα

αXXXXα

αXXXαXXuXu

21

1

)()(11

T

TTT

TTTTTT

n

n

nn

Kernel PCA

• K2 has the same eigenvectors as K, so the eigenvectors of PCA are the same as the eigenvectors of K

• The eigenvalues of PCA are related to the eigenvectors of K by

• Kernel PCA is a kernel embedding with an externally provided kernel matrix

2PCA

1Kn

Kernel PCA

• So kernel PCA gives the same solution as kernel embedding– The eigenvalues are modified a bit

• They are essentially the same thing in Euclidean space

• MDS uses the kernel and kernel embedding

• MDS and PCA are essentially the same thing in Euclidean space

• Kernel embedding, MDS and PCA all give the same answer for a set of points in Euclidean space

Some useful observations

• Your similarity matrix is Euclidean iff it has no negative eigenvalues (i.e. it is a kernel matrix and PSD)

• By similar reasoning, your distance matrix is Euclidean iff the similarity matrix derived from it is PSD

• If the feature space is small but the number of samples is large, then the covariance matrix is small and it is better to do normal PCA (on the covariance matrix)

• If the feature space is large and the number of samples is small, then the kernel matrix will be small and it is better to do kernel embedding

Part II: Non-Euclidean Manifolds

Non-linear data

• Much of the data in computer vision lies in a high-dimensional feature space but is constrained in some way– The space of all images of a face is a subspace of the

space of all possible images

– The subspace is highly non-linear but low dimensional (described by a few parameters)

Non-linear data

• This cannot be exploited by the linear subspace methods like PCA– These assume that the subspace is a Euclidean space as well

• A classic example is the

‘swiss roll’ data:

‘Flat’ Manifolds• Fundamentally different types of data, for example:

• The embedding of this data into the high-dimensional space is highly curved– This is called extrinsic curvature, the curvature of the manifold

with respect to the embedding space

• Now imagine that this manifold was a piece of paper; you could unroll the paper into a flat plane without distorting it– No intrinsic curvature, in fact it is homeomorphic to Euclidean

space

• This manifold is different:

• It must be stretched to map it onto a plane– It has non-zero intrinsic curvature

• A flatlander living on this manifold can tell that it is curved, for example by measuring the ratio of the radius to the circumference of a circle

• In the first case, we might still hope to find Euclidean embedding

• We can never find a distortion free Euclidean embedding of the second (in the sense that the distances will always have errors)

Curved manifold

Intrinsically Euclidean Manifolds

• We cannot use the previous methods on the second type of manifold, but there is still hope for the first

• The manifold is embedded in Euclidean space, but Euclidean distance is not the correct way to measure distance

• The Euclidean distance ‘shortcuts’ the manifold• The geodesic distance calculates the shortest path along the

manifold

Geodesics

• The geodesic generalizes the concept of distance to curved manifolds– The shortest path joining two points which lies completely within

the manifold

• If we can correctly compute the geodesic distances, and the manifold is intrinsically flat, we should get Euclidean distances which we can plug into our Euclidean geometry machine Position

X

SimilarityK

DistanceD

GeodesicDistances

ISOMAP

• ISOMAP is exactly such an algorithm

• Approximate geodesic distances are computed for the points from a graph

• Nearest neighbours graph– For neighbours, Euclidean distance≈geodesic distances

– For non-neighbours, geodesic distance approximated by shortest distance in graph

• Once we have distances D, can use MDS to find Euclidean embedding

ISOMAP

• ISOMAP:– Neighbourhood graph

– Shortest path algorithm

– MDS

• ISOMAP is distance-preserving – embedded distances should be close to geodesic distances

Laplacian Eigenmap

• The Laplacian Eigenmap is another graph-based method of embedding non-linear manifolds into Euclidean space

• As with ISOMAP, form a neighbourhood graph for the datapoints

• Find the graph Laplacian as follows

• The adjacency matrix A is

• The ‘degree’ matrix D is the diagonal matrix

• The normalized graph Laplacian is

otherwise 0

connected are and if

2

jieA t

d

ij

ij

j

ijii AD

2/12/1 ADDIL

Laplacian Eigenmap

• We find the Laplacian eigenmap embedding using the eigendecomposition of L

• The embedded positions are

• Similar to ISOMAP– Structure preserving not distance preserving

TUUL

UDX 2/1

Locally-Linear Embedding

• Locally-linear Embedding is another classic method which also begins with a neighbourhood graph

• We make point i (in the original data) from a weighted sum of the neighbouring points

• Wij is 0 for any point j not in the neighbourhood (and for i=j)• We find the weights by minimising the reconstruction error

– Subject to the constrains that the weights are non-negative and sum to 1

• Gives a relatively simple closed-form solution

i j j

jiji W xx̂

2|ˆ|min ii xx

j

ijij WW 1,0

Locally-Linear Embedding

• These weights encode how well a point j represents a point i and can be interpreted as the adjacency between i and j

• A low dimensional embedding is found by then finding points to minimise the error

• In other words, we find a low-dimensional embedding which preserves the adjacency relationships

• The solution to this embedding problem turns out to be simply the eigenvectors of the matrix M

• LLE is scale-free: the final points have the covariance matrix I– Unit scale

)()( WIWIM T

j

jijii

ii W yyyy ˆ |ˆ|min 2

Comparison

• LLE might seem like quite a different process to the previous two, but actually very similar

• We can interpret the process as producing a kernel matrix followed by scale-free kernel embedding

ISOMAP Lap. Eigenmap LLE

Representation Neighbourhood graph

Neighbourhood graph

Neighbourhood graph

Similarity matrix From geodesic distances

Graph Laplacian Reconstruction weights

Embedding

UXUUΛK

WWWWJIK

T

TT

n

kk

)1(

UDX 2/12/1UX UX

Comparison

• ISOMAP is the only method which directly computes and uses the geodesic distances– The other two depend indirectly on the distances through local

structure

• LLE is scale-free, so the original distance scale is lost, but the local structure is preserved

• Computing the necessary local dimensionality to find the correct nearest neighbours is a problem for all such methods

Non-Euclidean data

• Data is Euclidean iff K is psd

• Unless you are using a kernel function, this is often not true

• Why does this happen?

What type of data do I have?

• Starting point: distance matrix

• However we do not know apriori if our measurements are representable on a manifold– We will call them dissimilarities

• Our starting point to answer the question “What type of data do I have?” will be a matrix of dissimilarities D between objects

• Types of dissimilarities– Euclidean (no intrinsic curvature)

– Non-Euclidean, metric (curved manifold)

– Non-metric (no point-like manifold representation)

Causes

• Example: Chicken pieces data

• Distance by alignment

• Global alignment of everything could find Euclidean distances

• Only local alignments are practical

Causes

Dissimilarities may also be non-metric

The data is metric if it obeys the metric conditions1. Dij≥ 0 (nonegativity)

2. Dij= 0 iff i=j (identity of indiscernables)

3. Dij= Dji (symmetry)

4. Dij≤Dik+ Dkj (triangle inequality)

Reasonable dissimilarites should meet 1&2

Causes

• Symmetry Dij= Dji

• May not be symmetric by definition• Alignment: i→j may find a better solution than

j→i

Causes

• Triangle violations Dij≤Dik+ Dkj

• ‘Extended objects’

• Finally, noise in the measure of D can cause all of these effects

k

i j

0

0

0

ij

kj

ik

D

D

D

Tests(1)

• Find the similarity matrix

• The data is Euclidean iff K is positive semidefinite (no negative eigenvalues)– K is a kernel, explicit embedding from kernel embedding

• We can then use K in a kernel algorithm

CCDK s2

1

Tests(2)

• Negative eigenfraction (NEF)

• Between 0 and 0.5

i

i

i

0NEF

Tests(3)

1. Dij≥ 0 (nonegativity)

2. Dij= 0 iff i=j (identity of indiscernables)

3. Dij= Dji (symmetry)

4. Dij≤Dik+ Dkj (triangle inequality)

– Check these for your data (3rd involves checking all triples)

– Metric data is embeddable on a (curved) Reimannian manifold

Corrections

• If the data is non-metric or non-Euclidean, we can ‘correct it’

• Symmetry violations– Average

– For min-cost distances may be more appropriate

• Triangle violations– Constant offset

– This will also remove non-Euclidean behaviour for large enough c

• Euclidean violations– Discard negative eigenvalues

• There are many other approaches*

* “On Euclidean corrections for non-Euclidean dissimilarities”, Duin, Pekalska, Harol,Lee and Bunke, S+SSPR 08

)(2

1jiijjiij DDDD

),min( jiijjiij DDDD

)( jicDD ijij

Part III: Advanced techniques for non-Euclidean Embeddings

Known Manifolds

• Sometimes we have data which lies on a known but non-Euclidean manifold

• Examples in Computer Vision– Surface normals

– Rotation matrices

– Flow tensors (DT-MRI)

• This is not Manifold Learning, as we already know what the manifold is

• What tools do we need to be able to process data like this?– As before, distances are the key

Example: 2D direction

Direction of an edge in an image, encoded as a unit vector

The average of the direction vector isn’t even a direction vector (not unit length), let alone the correct ‘average’ direction

The normal definition of mean is not correct

– Because the manifold is curved

1x

2x

x

i

inxx

1

Tangent space

• The tangent space (TP) is the Euclidean space which is parallel to the manifold(M) at a particular point (P)

• The tangent space is a very useful tool because it is Euclidean

M

TP

P

Exponential Map

• Exponential map:

• ExpP maps a point X on the tangent plane onto a point A on the manifold– P is the centre of the mapping and is at the origin on the tangent

space

– The mapping is one-to-one in a local region of P

– The most important property of the mapping is that the distances to the centre P are preserved

– The geodesic distance on the manifold equals the Euclidean distance on the tangent plane (for distances to the centre only)

XA

MT

P

PP

Exp

:Exp

),(),( PAdPXd MTP

Exponential map

• The log map goes the other way, from manifold to tangent plane

MX

TM

P

pP

Log

:Log

Exponential Map

• Example on the circle: Embed the circle in the complex plane

• The manifold representing the circle is a complex number with magnitude 1 and can be written x+iy=exp(i)

Re

ImPieP

• In this case it turns out that the map is related to the normal exp and log functions

M

TP PieP

AieA

PAi

i

P

P

A

e

ei

P

AiAX

log

logLog

APAP

P

iii

iXPXA

exp)(expexp

expExp

X

Intrinsic mean

• The mean of a set of samples is usually defined as the sum of the samples divided by the number– This is only true in Euclidean space

• A more general formula

• Minimises the distances from the mean to the samples (equivalent in Euclidean space)

i

igd ),(minarg 2 xxxx

Intrinsic mean

• We can compute this intrinsic mean using the exponential map

• If we knew what the mean was, then we can use the mean as the centre of a map

• From the properties of the Exp-map, the distances are the same

• So the mean on the tangent plane is equal to the mean on the manifold

iMi AX Log

),(),( MAdMXd igie

Intrinsic mean

• Start with a guess at the mean and move towards correct answer

• This gives us the following algorithm– Guess at a mean M0

1. Map on to tangent plane using Mi

2. Compute the mean on the tangent plane to get new estimate Mi+1

i

iMMk An

Mkk

Log1

Exp1

Intrinsic Mean

• For many manifolds, this procedure will converge to the intrinsic mean– Convergence not always guaranteed

• Other statistics and probability distributions on manifolds are problematic.– Can hypothesis a normal distribution on tangent plane, but

distortions inevitable

Some useful manifolds and maps

• Some useful manifolds and exponential maps

• Directional vectors (surface normals etc.)

• a, p unit vectors, x lies in an (n-1)D space

map) (Exp sin

cos

map) (Log )cos(sin

1 ,

xpa

pax

aa

Some useful manifolds and maps

• Symmetric positive definite matrices (covariance, flow tensors etc)

• A is symmetric positive definite, X is just symmetric

• log is the matrix log defined as a generalized matrix function

map) (Exp exp

map) (Log log

0 0 ,

21

21

21

21

21

21

21

21

PXPPPA

PAPPPX

uAuuA

T

Some useful manifolds and maps

• Orthogonal matrices (rotation matrices, eigenvector matrices)

• A orthogonal, X antisymmetric (X+XT=0)

• These are the matrix exp and log functions as before

• In fact there are multiple solutions to the matrix log– Only one is the required real antisymmetric matrix; not easy to find

– Rest are complex

map) (Exp exp

map) (Log log

I ,

XPA

APX

AAA

T

T

Embedding on Sn

• On S2 (surface of a sphere in 3D) the following parameterisation is well known

• The distance between two points (the length of the geodesic) is

Trrr )cos ,sinsin ,cossin( x

xyd

x

y

yxxyyxij rd coscossinsincos 1

xyrθ

xyθ

x

y

More Spherical Geometry

• But on a sphere, the distance is the highlighted arc-length– Much neater to use inner-product

– And works in any number of dimensions

21

2

,cos

coscos,

rrrd

rxy

xyxy

xyxy

yx

yx

Spherical Embedding

• Say we had the distances between some objects (dij), measured on the surface of a [hyper]sphere of dimension n

• The sphere (and objects) can be embedded into an n+1 dimensional space– Let X be the matrix of point positions

• Z=XXT is a kernel matrix• But• And

• We can compute Z from D and find the spherical embedding!

jiijZ xx ,

r

drZ

rrd

ijjiij

xy

cos,

,cos

2

21

xx

yx

Spherical Embedding

• But wait, we don’t know what r is!

• The distances D are non-Euclidean, and if we use the wrong radius, Z is not a kernel matrix– Negative eigenvalues

• Use this to find the radius– Choose r to minimise the negative eigenvalues

)(minarg* rZr or

Example: Texture Mapping

• As an alternative to unwrapping object onto a plane and texture-mapping the plane

• Embed onto a sphere and texture-map the sphere

Plane Sphere

Backup slides

Laplacian and related processes

• As well as embedding objects onto manifolds, we can model many interesting processes on manifolds

• Example: the way ‘heat’ flows across a manifold can be very informative

•

• On a sphere it is

equationheat 2udt

du

2

2

2

2

2

2

2

isit spaceEuclidean 3Din andLaplacian theis

zyx

sin

sin

1

sin

122

2

22 rr

Heat flow

• Heat flow allows us to do interesting things on a manifold

• Smoothing: Heat flow is a diffusion process (will smooth the data)

• Characterising the manifold (heat content, heat kernel coefficients...)

• The Laplacian depends on the geometry of the manifold– We may not know this

– It may be hard to calculate explicitly

• Graph Laplacian

Graph Laplacian

• Given a set of datapoints on the manifold, describe them by a graph– Vertices are datapoints, edges are adjacency relation

• Adjacency matrix (for example)

• Then the graph Laplacian is

• The graph Laplacian is a discrete approximation of the manifold Laplacian

2

2 )/exp(

ij

ijij d

dA

j

ijii AV AVL

Heat Kernel

• Using the graph Laplacian, we can easily implement heat-flow methods on the manifold using the heat-kernel

• Can diffuse a function on the manifold by

kernelheat )exp(

equationheat

tdt

d

LH

Luu

Hff '