1 UNC, Stat & OR Nonnegative Matrix Factorization

1

UNC, Stat & OR

Standard Approach:

Lee et al (1999):

Formulate & Solve Optimization

Major Challenge:

Not Nested, ()

Nonnegative Matrix Factorization

2

UNC, Stat & OR

Standard NMF

But Note

Not Nested

No “Multi-scale”

Analysis

Possible (Scores Plot?!?)

Nonnegative Matrix Factorization

3

UNC, Stat & OR

Same Toy

Data Set

Rank 1

Approx.

Properly

Nested

Nonnegative Nested Cone Analysis

4

UNC, Stat & OR

5-d Toy Example Rank 3 NNCA Approx.

Current

Research:

How Many

Nonneg.

Basis El’ts

Needed?

Nonnegative Nested Cone Analysis

5

UNC, Stat & OR

How generally applicable is

Backwards approach to PCA?

Potential Application: Principal Curves

Hastie & Stuetzle, (1989)

(Foundation of Manifold Learning)

An Interesting Question

6

UNC, Stat & OR

How generally applicable is

Backwards approach to PCA?

An Attractive Answer:

James Damon, UNC Mathematics

Key Idea: Express Backwards PCA as

Nested Series of Constraints

An Interesting Question

7

UNC, Stat & OR

New Topic

Curve Registration

8

UNC, Stat & OR

Context

Functional Data AnalysisCurves as Data Objects

Toy Example:

How Can WeUnderstandVariation?

9

UNC, Stat & OR

Context

Functional Data AnalysisCurves as Data Objects

Toy Example:

How Can WeUnderstandVariation?

10

UNC, Stat & OR

Functional Data Analysis

InsightfulDecomposition

Vertical Variation

Horiz’l Var’n

11

UNC, Stat & OR

Challenge

Fairly Large Literature

Many (Diverse) Past Attempts

Limited Success (in General)

Surprisingly Slippery

(even mathematical formulation)

12

UNC, Stat & OR

Challenge (Illustrated)

Thanks to Wei Wu

13

UNC, Stat & OR

Challenge (Illustrated)

Thanks to Wei Wu

14

UNC, Stat & OR

Functional Data Analysis

AppropriateMathematicalFramework? Vertical Variation

Horiz’l Var’n

15

UNC, Stat & OR

Landmark Based Shape Analysis

Approach: Identify objects that are:

• Translations

• Rotations

• Scalings

of each other

Mathematics: Equivalence Relation

Results in: Equivalence Classes

Which become the Data Objects

16

UNC, Stat & OR

Curve Registration

What are the Data Objects?

Consider “Time Warpings”

(smooth)

More Precisely: Diffeomorphisms

17

UNC, Stat & OR

Time Warping Intuition

Elastically Stretch & Compress Axis

18

UNC, Stat & OR

Curve Registration

Say curves and are equivalent,

When so that

19

UNC, Stat & OR

Data Objects I

Equivalence Classes of Curves

(Set of AllWarps ofGiven Curve)

Notation: for a “representor”

20

UNC, Stat & OR

Data Objects I


(Set of AllWarps ofGiven Curve)

Next Task: Find Metric on Equivalence Classes

21

UNC, Stat & OR

Metrics in Curve Space

Find Metric on Equivalence Classes

Start with Warp Invariance on Curves& Extend

22

UNC, Stat & OR


Traditional Approach to Curve

Registration:

• Align curves, say and

• By finding optimal time warp, , so:

• Vertical var’n: PCA after alignment

• Horizontal var’n: PCA on s

23

UNC, Stat & OR


Problem:

Don’t have proper metric

Since:

Because:

24

UNC, Stat & OR


Thanks toXiaosun Lu

25

UNC, Stat & OR


Note:VeryDifferentL2 norms

Thanks toXiaosun Lu

26

UNC, Stat & OR


Solution:

Look for Warp Invariant Metric

Where:

27

UNC, Stat & OR


I.e. Have “Parallel”

Representatives

Of Equivalence

Classes

28

UNC, Stat & OR


Warping Invariant Metric

Developed in context of:

Likelihood Geometry

Fisher – Rao Metric:

29

UNC, Stat & OR



Computation Based on

Square Root Velocity Function (SRVF)

30

UNC, Stat & OR


Square Root Velocity Function (SRVF)

31

UNC, Stat & OR



Computation Based on SRVF:

So work with SRVF

32

UNC, Stat & OR


Why

square

roots?

33

UNC, Stat & OR


Why

square

roots?

Thanks to

Xiaosun Lu

34

UNC, Stat & OR


Why

square

roots?

35

UNC, Stat & OR


Why

square

roots?

36

UNC, Stat & OR


Why

square

roots?

37

UNC, Stat & OR


Why

square

roots?

38

UNC, Stat & OR


Why

square

roots?

39

UNC, Stat & OR


Why

square

roots?

40

UNC, Stat & OR


Why

square

roots?

41

UNC, Stat & OR


Why

square

roots?

42

UNC, Stat & OR


Why

square

roots? Dislikes Pinching

Focusses Well On

Peaks of Unequal Height

43

UNC, Stat & OR


Note on SRVF representation:

Can show: Warp Invariance

Follows from Jacobean calculation

44

UNC, Stat & OR

Metrics in Curve Quotient Space

Above was Invariance for Individual

Curves

Now extend to:


I.e. Orbits as Data Objects

I.e. Quotient Space

45

UNC, Stat & OR

Metrics in Curve Quotient Space

Define Metric on Equivalence Classes:

For & , i.e. &

Independent of Choice of &

By Warping Invariance

46

UNC, Stat & OR

Mean in Curve Quotient Space

Benefit of a Metric:

Allows Definition of a “Mean”

Fréchet Mean

Geodesic Mean

Barycenter

Karcher Mean

47

UNC, Stat & OR


Given Equivalence Class Data Objects:

The Karcher Mean is:

48

UNC, Stat & OR


The Karcher Mean is:

Intuition: Can Show, for Euclidean

Data

Minimizer = Conventional

49

UNC, Stat & OR


Next Define “Most Representative”

Choice of

As Representer of

50

UNC, Stat & OR


“Most Representative” in

Given a candidate

Consider warps to each

Choose to make

Karcher mean of warps = Identity

(under Fisher Rao metric)

51

UNC, Stat & OR



Thanks to Anuj Srivastava

52

UNC, Stat & OR

Toy Example – (Details Later)

EstimatedWarps

(Note:RepresentedWith KarcherMean At Identity)

53

UNC, Stat & OR



Terminology: The “Template Mean”

54

UNC, Stat & OR

More Data Objects

Final Curve Warps:

• Warp Each Data Curve,

• To Template Mean,

• Denote Warp Functions

Gives (Roughly Speaking):

Vertical Components

(Aligned Curves)

Horizontal Components

55

UNC, Stat & OR

More Data Objects

Final Curve Warps:





Vertical Components

(Aligned Curves)


Data Objects II

56

UNC, Stat & OR

More Data Objects

Final Curve Warps:





Vertical Components

(Aligned Curves)


Data Objects II

~ Kendall’s Shapes

57

UNC, Stat & OR

More Data Objects

Final Curve Warps:





Vertical Components

(Aligned Curves)


Data Objects III

58

UNC, Stat & OR

More Data Objects

Final Curve Warps:





Vertical Components

(Aligned Curves)


Data Objects III

~ Chang’s Transfo’s

59

UNC, Stat & OR

Computation

Several Variations of

Dynamic Programming

Done by Eric Klassen, Wei Wu

60

UNC, Stat & OR

Toy Example

Raw Data

61

UNC, Stat & OR

Toy Example

Raw Data

BothHorizontalAndVerticalVariation

62

UNC, Stat & OR

Toy Example

ConventionalPCAProjections

63

UNC, Stat & OR

Toy Example


PowerSpreadAcrossSpectrum

64

UNC, Stat & OR

Toy Example


PowerSpreadAcrossSpectrum

65

UNC, Stat & OR

Toy Example

ConventionalPCAScores

66

UNC, Stat & OR

Toy Example


Views of1-d CurveBendingThrough4 Dim’ns’

67

UNC, Stat & OR

Toy Example


PatternsAre“Harmonics”In Scores

68

UNC, Stat & OR

Toy Example

Scores PlotShows DataAre “1”Dimensional

So NeedImprovedPCA Decomp.

69

UNC, Stat & OR

Visualization

Vertical Variation:

• PCA on Aligned Curves,

• Projected Curves

70

UNC, Stat & OR

Toy Example

AlignedCurves

(Clear1-dVerticalVar’n)

71

UNC, Stat & OR

Toy Example

AlignedCurvePCAProjections

All Var’nIn 1st

Component

72

UNC, Stat & OR

Visualization

Horizontal Variation:

• PCA on Warps,

• Projected Curves

73

UNC, Stat & OR

Toy Example

EstimatedWarps

74

UNC, Stat & OR

Toy Example

Warps,PCProjections

75

UNC, Stat & OR

Toy Example

Warps,PCProjections

Mostly1st PC

76

UNC, Stat & OR

Toy Example

Warps,PCProjections

Mostly1st PC,But 2nd

Helps Some

77

UNC, Stat & OR

Toy Example

Warps,PCProjections

Rest isNotImportant

78

UNC, Stat & OR

Toy Example

Horizontal Var’n Visualization Challenge:

(Complicated) Warps Hard to Interpret

Approach:

Apply Warps to Template Mean

(PCA components)

79

UNC, Stat & OR

Toy Example

WarpCompon’ts(+ Mean)Applied toTemplateMean

80

UNC, Stat & OR

Refined Calculations

Current Development:

Better Exploit Manifold Data Space:

Principal Nested Spheres

81

UNC, Stat & OR

PNS on SRVF Sphere

Toy Example

Tangent Space

PCA

(on Horiz. Var’n)

Thanks to Xiaosun Lu

82

UNC, Stat & OR

PNS on SRVF Sphere

Toy Example

PNS Projections

(Fewer Modes)

83

UNC, Stat & OR

PNS on SRVF Sphere

Toy Example

Tangent Space

PCA

Note: 3 Comp’s

Needed for This

84

UNC, Stat & OR

PNS on SRVF Sphere

Toy Example

PNS Projections

Only 2 for This

85

UNC, Stat & OR

TIC testbed

Serious Data Challenge:

TIC (Total Ion Count) Chromatograms

Modern type of “chemical spectra”

Thanks to

Peter

Hoffmann

86

UNC, Stat & OR

TIC testbed

Raw Data: 15 TIC Curves (5 Colors)

87

UNC, Stat & OR

TIC testbed

Special Feature: Answer Key of Known Peaks

Found by MajorTime &LaborInvestment

88

UNC, Stat & OR

TIC testbed

Special Feature: Answer Key of Known Peaks

Goal:FindWarpsTo AlignThese

89

UNC, Stat & OR

TIC testbed

Fisher – Rao Alignment

90

UNC, Stat & OR

TIC testbed

Fisher – Rao Alignment

Spike-InPeaks

91

UNC, Stat & OR

TIC testbed

Next Zoom in on This Region

92

UNC, Stat & OR

TIC testbed

Zoomed Fisher – Rao Alignment

93

UNC, Stat & OR

TIC testbed

Before Alignment

94

UNC, Stat & OR

TIC testbed


95

UNC, Stat & OR

TIC testbed


96

UNC, Stat & OR

TIC testbed

Before Alignment

97

UNC, Stat & OR

TIC testbed


98

UNC, Stat & OR

TIC testbed


99

UNC, Stat & OR

TIC testbed

Before Fisher-Rao Alignment

100

UNC, Stat & OR

TIC testbed


101

UNC, Stat & OR

TIC testbed


102

UNC, Stat & OR

TIC testbed


Note:VeryChallenging

103

UNC, Stat & OR

TIC testbed

Before Alignment

104

UNC, Stat & OR

TIC testbed


105

UNC, Stat & OR

TIC testbed


106

UNC, Stat & OR

TIC testbed

Before Alignment

107

UNC, Stat & OR

TIC testbed

Warping Functions

108

UNC, Stat & OR

References for Much More

Big Picture Survey:Marron, Ramsay, Sangalli & Srivastava (2014)

TIC Proteomics Example:Koch, Hoffman & Marron (2014)

109

UNC, Stat & OR

Take Away Message

Curve Registration is Slippery Thus, Careful Mathematics is Useful Fisher-Rao Approach:

Gets the Math Right Intuitively Sensible Computable Generalizable Worth the Complication

Documents

1 UNC, Stat & OR Nonnegative Matrix Factorization