GEOGG121: Methods Linear & non-linear models Dr. Mathias (Mat) Disney UCL Geography Office: 113, Pearson Building Tel: 7670 0592 Email: [email protected]

GEOGG121: MethodsLinear & non-linear models

Dr. Mathias (Mat) Disney

UCL Geography

Office: 113, Pearson Building

Tel: 7670 0592

Email: [email protected]

www.geog.ucl.ac.uk/~mdisney

• Linear models and inversion– Least squares revisited, examples– Parameter estimation, uncertainty

• Non-linear models, inversion– The non-linear problem– Parameter estimation, uncertainty– Approaches

Lecture outline

• Linear models and inversion– Linear modelling notes: Lewis, 2010; ch 2 of Press et al. (1992)

Numerical Recipes in C (http://apps.nrbook.com/c/index.html)

• Non-linear models, inversion– Gradient descent: NR in C, 10,7 eg BFGS; XXX– Conjugate gradient etc.: NR in C, ch. 10; XXX– Liang, S. (2004) Quantitative Remote Sensing of the Land

Surface, ch. 8 (section 8.2).

Reading

Linear Models

• For some set of independent variables

x = {x0, x1, x2, … , xn}

have a model of a dependent variable y which can be expressed as a linear combination of the independent variables.

110 xaay

22110 xaxaay

ni

iii xay

0

xay

Linear Models?

ni

iiiii xbxaay

10 cossin

ni

iiii bxaay

10 sin

nn

ni

i

ii xaxaxaaxay 0

202010

00 ...

xaeay 10

xay

Linear Mixture Modelling

• Spectral mixture modelling:– Proportionate mixture of (n) end-member spectra

– First-order model: no interactions between components

11

0

ni

i iF

1

0

ni

i iiFr Fr


• r = {r 0l , r 1l , … rlm, 1.0} – Measured reflectance spectrum (m wavelengths)

• nx(m+1) matrix:

1

2

1

0

112111101

11210101

10201000

1

1

0

0.10.10.10.10.1 n

nmmmm

n

n

m

P

P

P

P

r

r

r

Fr


• n=(m+1) – square matrix

• Eg n=2 (wavebands), m=2 (end-members)

Fr

rF 1

Reflectance

Band 1

Reflectance

Band 2

r1

r2

r3

r


• as described, is not robust to error in measurement or end-member spectra;

• Proportions must be constrained to lie in the interval (0,1) – - effectively a convex hull constraint;

• m+1 end-member spectra can be considered;• needs prior definition of end-member spectra; cannot

directly take into account any variation in component reflectances

– e.g. due to topographic effects

Linear Mixture Modelling in the presence of Noise

• Define residual vector• minimise the sum of the squares of the error e,

i.e.

eFr

ee

eeFrFrFrml

l

21

0

Method of Least Squares (MLS)

Error Minimisation

• Set (partial) derivatives to zero

021

0

21

0

ml

lii

ml

l

F

FFr

P

Fr

eeFrFrFrml

l

21

0

iiFF

1

0

1

0

1

020

ml

l i

ml

l i

ml

l i

Fr

Fr

Error Minimisation

• Can write as:

PMO

1

0

1

0

ml

l i

ml

l i Fr

1

1

0

1

0

111110

111110

010100

1

0

1

1

0

n

ml

l

nlnlnllnll

lnlllll

lnlllll

ml

l

nll

ll

ll

F

F

F

r

r

r

Solve for P by matrix inversion

e.g. Linear Regression

mxcy

PMO

m

c

xx

x

xy

y nl

l ll

lnl

l ll

l1

02

1

0

1

m

c

xx

x

yx

y2

1

x

xyy

xx

xy

xx

xyxx

2

2

2

22

1

1 2

2

1

x

xxM

xx

222 xxxx

RMSE

1

0

22nl

lii mxcye

mnRMSE

2

y

xx x1x2

Weight of Determination (1/w)

• Calculate uncertainty at y(x)

m

c

xPQxy

1

QMQw

T 11

we

1

2

2

11

xx

xx

w

P0

P1RMSE

P0

P1RMSE

Issues

• Parameter transformation and bounding• Weighting of the error function• Using additional information• Scaling

Parameter transformation and bounding

• Issue of variable sensitivity– E.g. saturation of LAI effects– Reduce by transformation

• Approximately linearise parameters• Need to consider ‘average’ effects

Weighting of the error function

• Different wavelengths/angles have different sensitivity to parameters

• Previously, weighted all equally– Equivalent to assuming ‘noise’ equal for all observations

Ni

i

Ni

imeasured ii

RMSE

1

1

2modelled

1

Weighting of the error function

• Can ‘target’ sensitivity– E.g. to chlorophyll concentration– Use derivative weighting (Privette 1994)

Ni

i

Ni

imeasured

P

iiP

RMSE

1

21

2

modelled

Using additional information

• Typically, for Vegetation, use canopy growth model– See Moulin et al. (1998)

• Provides expectation of (e.g.) LAI– Need:

• planting date• Daily mean temperature• Varietal information (?)

• Use in various ways– Reduce parameter search space– Expectations of coupling between parameters

Scaling

• Many parameters scale approximately linearly– E.g. cover, albedo, fAPAR

• Many do not– E.g. LAI

• Need to (at least) understand impact of scaling

Crop Mosaic

LAI 1 LAI 4 LAI 0

Crop Mosaic

• 20% of LAI 0, 40% LAI 4, 40% LAI 1. • ‘real’ total value of LAI:

– 0.2x0+0.4x4+0.4x1=2.0.

LAI 1

LAI 4

LAI 0

)2/exp())2/exp(1( LAILAI s

visible: NIR 1.0;2.0 s

3.0;9.0 s

canopy reflectance

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49

LAI

refl

ect

ance

visible

NIR

canopy reflectance over the pixel is 0.15 and 0.60 for the NIR.

• If assume the model above, this equates to an LAI of 1.4. • ‘real’ answer LAI 2.0

Linear Kernel-driven Modelling of Canopy Reflectance

• Semi-empirical models to deal with BRDF effects– Originally due to Roujean et al (1992)– Also Wanner et al (1995)– Practical use in MODIS products

• BRDF effects from wide FOV sensors– MODIS, AVHRR, VEGETATION, MERIS

Satellite, Day 1 Satellite, Day 2

X

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

13

6

14

3

15

0

15

7

16

4

17

1

17

8

18

5

19

2

19

9

20

6

21

8

22

6

23

3

24

0

24

7

25

4

26

1

26

8

27

5

28

2

Julian Day

ND

VI

original NDVI MVC BRDF normalised NDVI

AVHRR NDVI over Hapex-Sahel, 1992

Linear BRDF Model

• of form:

,,,, geogeovolvoliso kfkff

Model parameters:

Isotropic

Volumetric

Geometric-Optics

Linear BRDF Model

• of form: ,,,, geogeovolvoliso kfkff

Model Kernels:

Volumetric

Geometric-Optics

Volumetric Scattering

• Develop from RT theory– Spherical LAD– Lambertian soil– Leaf reflectance = transmittance– First order scattering

• Multiple scattering assumed isotropic

Xs

Xl ee

12

cossin

3

2,1

2

LX


• If LAI small:

Xe X 1

Xs

Xl ee

1

2cossin

3

2,1

2

LX

2

12

2cossin

3

2,1 LL

sl

sl L

2

2cossin

3

2,1


• Write as:

sl L

2

2cossin

3

2,1

,,, 10 volthin kaa

2

2cossin

,

volk

slL

a

60

31

lLa

RossThin kernel

Similar approach for RossThick

LBL

exp2

exp

Geometric Optics

• Consider shadowing/protrusion from spheroid on stick (Li-Strahler 1985)

h

b

r

A()

Projection (shadowed)

Sunlit crownshadowed crown

shadowed ground

h

b

r

A()

Projection (shadowed)

Sunlit crownshadowed crown

shadowed ground

Geometric Optics

• Assume ground and crown brightness equal• Fix ‘shape’ parameters• Linearised model

– LiSparse– LiDense

Kernels

-3

-2.5

-2

-1.5

-1

-0.5

0

0.5

1

-75 -60 -45 -30 -15 0 15 30 45 60 75

view angle / degrees

ke

rne

l va

lue

RossThick LiSparse

Retro reflection (‘hot spot’)

Volumetric (RossThick) and Geometric (LiSparse) kernels for viewing angle of 45 degrees

Kernel Models

• Consider proportionate (a) mixture of two scattering effects

,,1

1,,

11

00

geogeovolvol

multgeovol

kaka

aa

Using Linear BRDF Models for angular normalisation• Account for BRDF variation• Absolutely vital for compositing samples

over time (w. different view/sun angles)• BUT BRDF is source of info. too!

MODIS NBAR (Nadir-BRDF Adjusted Reflectance MOD43, MCD43)http://www-modis.bu.edu/brdf/userguide/intro.html

MODIS NBAR (Nadir-BRDF Adjusted Reflectance MOD43, MCD43)http://www-modis.bu.edu/brdf/userguide/intro.html

BRDF Normalisation• Fit observations to model• Output predicted reflectance at standardised

angles – E.g. nadir reflectance, nadir illumination

• Typically not stable

– E.g. nadir reflectance, SZA at local mean

KP ,,

geo

vol

iso

f

f

f

P

,

,

1

geo

vol

k

kK QMQw

T 11

And uncertainty via

Linear BRDF Models to track change

• Examine change due to burn (MODIS)

FROM: http://modis-fire.umd.edu/Documents/atbd_mod14.pdf

220 days of Terra (blue) and Aqua (red) sampling over point in Australia, w. sza (T: orange; A: cyan).

Time series of NIR samples from above sampling

http://modis-fire.umd.edu/Burned_Area_Products.html

http://modis-fire.umd.edu/Documents/atbd_mod14.pdf

http://modis-fire.umd.edu/Documents/atbd_mod14.pdf

MODIS Channel 5 Observation

DOY 275


DOY 277

Detect Change

• Need to model BRDF effects• Define measure of dis-association

wee

predictedobservedpredictedobserved

11

22

MODIS Channel 5 Prediction

DOY 277

MODIS Channel 5 Discrepency

DOY 277


DOY 275

MODIS Channel 5 Prediction

DOY 277


DOY 277

So BRDF source of info, not JUST noise!

• Use model expectation of angular reflectance behaviour to identify subtle changes

5454Dr. Lisa Maria Rebelo, IWMI, CGIAR.

Detect Change

• Burns are:– negative change in Channel 5– Of ‘long’ (week’) duration

• Other changes picked up– E.g. clouds, cloud shadow– Shorter duration – or positive change (in all channels)– or negative change in all channels

Day of burn

http://modis-fire.umd.edu/Burned_Area_Products.htmlRoy et al. (2005) Prototyping a global algorithm for systematic fire-affected area mapping using MODIS time series data, RSE 97, 137-162.



Non-linear inversion

• If we cannot phrase our problem in linear form, then we have non-linear inversion problem

• Key tool for many applications– Resource allocation– Systems control– (Non-linear) Model parameter estimation

• AKA curve or function fitting: i.e. obtain parameter values that provide “best fit” (in some sense) to a set of observations

• And estimate of parameter uncertainty, model fit

5757

Options for Numerical Inversion

• Same principle as for linear– i.e. find minimum of some cost func. expressing difference

between model and obs– We want some set of parameters (x*) which minimise cost function

f(x) i.e. for some (small) tolerance δ > 0 so for all x

– Where f(x*) ≤ f(x). So for some region around x*, all values of f(x) are higher (a local minima – may be many)

– If region encompasses full range of x, then we have the global minima

Numerical Inversion

• Iterative numerical techniques– Can we differentiate model (e.g. adjoint)?

• YES: Quasi-Newton (eg BFGS etc)• NO: Powell, Simplex, Simulated annealing etc.

• Knowledge-based systems (KBS)• Artificial Neural Networks (ANNs)• Genetic Algorithms (GAs)• Look-up Tables (LUTs)

Errico (1997) What is an adjoint model?, BAMS, 78, 2577-2591http://paoc2001.mit.edu/cmi/development/adjoint.htm

http://paoc2001.mit.edu/cmi/development/adjoint.htm

http://paoc2001.mit.edu/cmi/development/adjoint.htm

Local and Global Minima (general: optima)

Need starting point

How to go ‘downhill’?

Bracketing of a minimum

How far to go ‘downhill’?

Golden Mean Fraction =w=0.38197

z=(x-b)/(c-a)

w=(b-a)/(c-a)

w 1-w

Choose:

z+w=1-w

For symmetry

Choose:

w=z/(1-w)

To keep proportions the samez=w-w2=1-2w

0= w2 -3w+1

Slow, but sure

Parabolic Interpolation

Inverse parabolic interpolationMore rapid

Brent’s method

• Require ‘fast’ but robust inversion• Golden mean search

– Slow but sure• Use in unfavourable areas

• Use Parabolic method – when get close to minimum

Multi-dimensional minimisation

• Use 1D methods multiple times– In which directions?

• Some methods for N-D problems– Simplex (amoeba)

Downhill Simplex

• Simplex:– Simplest N-D

• N+1 vertices• Simplex operations:

– a reflection away from the high point

– a reflection and expansion away from the high point

– a contraction along one dimension from the high point

– a contraction along all dimensions towards the low point.

• Find way to minimum

Simplex

Direction Set (Powell's) Method

• Multiple 1-D minimsations– Inefficient

along axes

Powell

Direction Set (Powell's) Method

• Use conjugate directions– Update primary

& secondary directions

• Issues– Axis covariance

Powell

Simulated Annealing

• Previous methods:– Define start point– Minimise in some direction(s)– Test & proceed

• Issue:– Can get trapped in local minima

• Solution (?)– Need to restart from different point

Simulated Annealing

Simulated Annealing

• Annealing– Thermodynamic phenomenon– ‘slow cooling’ of metals or crystalisation of liquids– Atoms ‘line up’ & form ‘pure cystal’ / Stronger (metals)– Slow cooling allows time for atoms to redistribute as they lose

energy (cool)– Low energy state

• Quenching– ‘fast cooling’– Polycrystaline state

Simulated Annealing

• Simulate ‘slow cooling’• Based on Boltzmann probability distribution:

• k – constant relating energy to temperature• System in thermal equilibrium at temperature T has distribution of

energy states E• All (E) states possible, but some more likely than others• Even at low T, small probability that system may be in higher energy

state

kT

E

eE

)Pr(

Simulated Annealing

• Use analogy of energy to RMSE• As decrease ‘temperature’, move to generally

lower energy state• Boltzmann gives distribution of E states

– So some probability of higher energy state• i.e. ‘going uphill’

– Probability of ‘uphill’ decreases as T decreases

Implementation

• System changes from E1 to E2 with probability exp[-(E2-E1)/kT]

– If(E2< E1), P>1 (threshold at 1)• System will take this option

– If(E2> E1), P<1• Generate random number• System may take this option• Probability of doing so decreases with T

Simulated Annealing

T

P= exp[-(E2-E1)/kT] – rand() - OK

P= exp[-(E2-E1)/kT] – rand() - X

Simulated Annealing

• Rate of cooling very important• Coupled with effects of k

– exp[-(E2-E1)/kT] – So 2xk equivalent to state of T/2

• Used in a range of optimisation problems• Not much used in Remote Sensing

(Artificial) Neural networks (ANN)

• Another ‘Natural’ analogy– Biological NNs good at solving complex problems– Do so by ‘training’ system with ‘experience’


• ANN architecture


• ‘Neurons’ – have 1 output but many inputs– Output is weighted sum of inputs– Threshold can be set

• Gives non-linear response


• Training– Initialise weights for all neurons– Present input layer with e.g. spectral reflectance– Calculate outputs– Compare outputs with e.g. biophysical parameters– Update weights to attempt a match– Repeat until all examples presented


• Use in this way for canopy model inversion• Train other way around for forward model• Also used for classification and spectral unmixing

– Again – train with examples• ANN has ability to generalise from input examples• Definition of architecture and training phases critical

– Can ‘over-train’ – too specific– Similar to fitting polynomial with too high an order

• Many ‘types’ of ANN – feedback/forward


• In essence, trained ANN is just a (essentially) (highly) non-linear response function

• Training (definition of e.g. inverse model) is performed as separate stage to application of inversion– Can use complex models for training

• Many examples in remote sensing• Issue:

– How to train for arbitrary set of viewing/illumination angles? – not solved problem

Genetic (or evolutionary) algorithms (GAs)

• Another ‘Natural’ analogy• Phrase optimisation as ‘fitness for survival’• Description of state encoded through ‘string’

(equivalent to genetic pattern)• Apply operations to ‘genes’

– Cross-over, mutation, inversion


• E.g. of BRDF model inversion:• Encode N-D vector representing current state of

biophysical parameters as string• Apply operations:

– E.g. mutation/mating with another string– See if mutant is ‘fitter to survive’ (lower RMSE)– If not, can discard (die)


• General operation:– Populate set of chromosomes (strings)– Repeat:

• Determine fitness of each• Choose best set• Evolve chosen set

– Using crossover, mutation or inversion

– Until a chromosome found of suitable fitness


• Differ from other optimisation methods– Work on coding of parameters, not parameters themselves– Search from population set, not single members (points)– Use ‘payoff’ information (some objective function for selection) not

derivatives or other auxilliary information– Use probabilistic transition rules (as with simulated annealing) not

deterministic rules


• Example operation:1. Define genetic representation of state2. Create initial population, set t=03. Compute average fitness of the set

- Assign each individual normalised fitness value- Assign probability based on this

4. Using this distribution, select N parents5. Pair parents at random6. Apply genetic operations to parent sets

- generate offspring- Becomes population at t+1

7. Repeat until termination criterion satisfied


– Flexible and powerful method– Can solve problems with many small, ill-defined

minima– May take huge number of iterations to solve

– Not applied to remote sensing model inversions

Knowledge-based systems (KBS)

• Seek to solve problem by incorporation of information external to the problem

• Only RS inversion e.g. Kimes et al (1990;1991)– VEG model

• Integrates spectral libraries, information from literature, information from human experts etc

• Major problems:– Encoding and using the information– 1980s/90s ‘fad’?

LUT Inversion

• Sample parameter space• Calculate RMSE for each sample point• Define best fit as minimum RMSE parameters

– Or function of set of points fitting to a certain tolerance

• Essentially a sampled ‘exhaustive search’

LUT Inversion

• Issues:– May require large sample set – Not so if function is well-behaved

• for many optical EO inversions• In some cases, may assume function is locally linear over large

(linearised) parameter range• Use linear interpolation

– Being developed UCL/Swansea for CHRIS-PROBA– May limit search space based on some expectation

• E.g. some loose VI relationship or canopy growth model or land cover map

• Approach used for operational MODIS LAI/fAPAR algorithm (Myneni et al)

LUT Inversion

• Issues:– As operating on stored LUT, can pre-calculate model outputs

• Don’t need to calculate model ‘on the fly’ as in e.g. simplex methods• Can use complex models to populate LUT

– E.g. of Lewis, Saich & Disney using 3D scattering models (optical and microwave) of forest and crop

– Error in inversion may be slightly higher if (non-interpolated) sparse LUT

• But may still lie within desirable limits– Method is simple to code and easy to understand

• essentially a sort operation on a table

Summary

• Range of options for non-linear inversion• ‘traditional’ NL methods:

– Powell, AMOEBA• Complex to code

– though library functions available• Can easily converge to local minima

– Need to start at several points• Calculate canopy reflectance ‘on the fly’

– Need fast models, involving simplifications• Not felt to be suitable for operationalisation

Summary

• Simulated Annealing– Can deal with local minima– Slow– Need to define annealing schedule

• ANNs– Train ANN from model (or measurements)– ANN generalises as non-linear model– Issues of variable input conditions (e.g. VZA)– Can train with complex models– Applied to a variety of EO problems

Summary

• GAs– Novel approach, suitable for highly complex inversion problems– Can be very slow– Not suitable for operationalisation

• KBS– Use range of information in inversion– Kimes VEG model– Maximises use of data– Need to decide how to encode and use information

Summary

• LUT– Simple method

• Sort– Used more and more widely for optical model inversion

• Suitable for ‘well-behaved’ non-linear problems– Can operationalise– Can use arbitrarily complex models to populate LUT– Issue of LUT size

• Can use additional information to limit search space• Can use interpolation for sparse LUT for ‘high information content’

inversion