Introduction to Astronomical Image Processing

5. Mathematical tools:overview and examples in astronomy

Master ISTI / PARI / IV

André JalobeanuLSIIT / MIV / PASEO group

Jan. 2006

lsiit-miv.u-strasbg.fr/paseo

PASEO

Mathematical tools:overview and examples in astronomy

Direct tools (basic ops & filtering)

Modeling (instrument/object description)

Probability theory and statisticsIntroduction (pdfs, estimation, detection)Bayesian inference & graphical models

Transforms and representationsProjectionsFrequencyMultiresolutionMulti- shape and resolutionMultidimensional data

Functional optimization

Mathematical morphology

Direct tools:basic operations & filtering

Focus on operations involving single pixel or their neighbors

Understand the simplest tools for image processing

See how filtering can effectively be used in astronomy

Single pixel (elementwise) operations

๏ Image Addition

‣ Frame co-addition in deep field imaging (increase sensitivity)

๏ Image Subtraction

‣ Sky background and detector bias subtraction‣ Registered images subtraction for comparison/detection purposes

๏ Image Division

‣ Flat-field correction

๏ Pixel value transformation

‣ Look-up tables for speed-up (integer values)‣ Thresholding, contrast enhancement, dynamic range transforms‣ Vector-valued pixels: change of basis (multispectral)

thresholding functions

Multiple pixel operations:kernel filtering, interpolation

๏ Neighborhood filters

‣ Linear: Finite Impulse Response (FIR) filters/masks (3x3, 5x5...)Convolution with a kernel (image space)Sharpen, Blur, 1st/2nd Derivatives, discretized operators (Laplacian...)‣ Rank filters (min, max, median)Sort pixel values within the pixel neighborhoodDenoising (impulse noise)‣ More complex: filter and decision (comparison, threshold...)Denoising, Detection...

๏ Interpolation

‣ Bilinear, Key’s bicubic (1 step)‣ Spline (2 steps: prefiltering-prediction) Synthesis function ≠ interpolant function [Unser 95]

0 1

1

x

y

bilinear interpolation

๏ Linear filtering

‣ Multiply by a factor ak depending on the coefficient index ke.g. Fourier transform, factor = function of spatial frequencye.g. Wiener filter for image deblurring

๏ Nonlinear filtering

‣ Apply a nonlinear function f of the coefficiente.g. thresholding

•Stationary: function f does not depend on the coefficient indexe.g. Denoising via simple wavelet coefficient thresholding

•Adaptive: function fk depends on the coefficient indexe.g. Deblurring via adaptive wavelet coefficient thresholding

Filtering: processing representations

Keep the signal, filter the noise

Work in a different representation (transform: Fourier, wavelet...) to separate signal & noise nois

e

signal

Modeling tools

Find out how to describe astronomical objects in 3D or 2D

Know the basics of forward modeling principles: instrument and sensor description

Tools for object description

๏ Geometry

‣ Simple objects (e.g. lines, spheres, ellipsoids)‣ Polygonal objects (e.g. planetary surface)

๏ Parametric functions (1D, 2D, 3D)

‣ Simple functions (e.g. circular or ellipsoidal Gaussian)‣ Complex analytic functions (e.g. radial sigmoid)‣ Non-analytic functions (e.g. 2D integration of a 3D model)

Tools for image acquisition modeling

๏ Sampling theory [Nyquist 28, Shannon 49]

‣ Regular sampling: regular grids (rectangular, hexagonal lattice)‣ Generalized sampling on irregular grids‣ Object-based rendering: model discretization

๏ Physics: optics, turbulence, motion, sensor

‣ Product of MTFs‣ Analytic expressions or numerical models

๏ Probability & statistics (noise)

x

y

Rx

Ry

sampling grid

Image space

u

v

Ru

Rv

image spectrum

Frequency space

Airy pattern

Hexagonal lattice

€

Ip ≈ AΔp

Δ

∑ ΦΔ

Object-basedrendering

Probability theory& statistics

Give a short introduction to probability theory and the related statistical tools

Understand the principles of Bayesian inference and estimation

Get acquainted with complex modeling tools via graphical models

๏ Random variables: stochastic processesprobability density function (pdf)

‣ distribution theory‣ constraints: positive, normalized‣ discrete or continuous variables

๏ Joint pdf of several variables x,y random var. sets (e.g. pixels, parameters Θ)

๏ Marginal pdf:

๏ Conditional pdf:

๏ Bayes theorem:

Introduction to probabilistic toolsrandom variables and pdfs

marginal

conditional

P(data | Θ)likelihood

P(Θ)prior

P(data)evidence

P(Θ | data)posterior

P(y |x) = P(x |y)P(y)/P(x)

P(x |y) = P(x,y)/P(y)

P(y) =Z

xP(x,y)dx

Statistical tools and function fitting

๏ Function fitting principles:

‣ Provide a parametric function (the model) ‣ Provide an error function

•Squared difference in the Gaussian Case

•Explicit -log P(datak|Θ) in general

•Robust functions allowing for impulse noise‣ Minimize the total error (cost function): regression

•General case: iterative method (see functional optimization)

•Special cases: closed-form solution

๏ Some examples:

‣ Smooth background extractionModel: linear or polynomial function of the coordinatesClosed-form solution: low order moments‣ Star extraction and aperture photometryModel: 2D Gaussian (≈PSF); free parameters: location (x,y), intensity L Approx closed-form solution: 1st order moments for (x,y) “centroid”, mean for L(the exact method is iterative) L = ∑

i, jIi j x = ∑

i, ji Ii j/L y = ∑

i, jj Ii j/L

Gaussian noise least squares

statistical estimation:maximum likelihoodarg maxΘ P(data | Θ)

P(data|Θ) =Πk P(datak|Θ)indep. assumption

Graphical models

๏ Independence properties of random variables

‣ Stochastic independence: P(x,y) = P(x) P(y)‣ Conditional independence: P(x,y|z) = P(x|z) P(y|z)

๏ Dependence graphs or graphical models

‣ Each node is a random variable (or a set of variables)‣ Edges represent dependencies(stochastic independence ⇔ no connexion between nodes)‣ Directed graphs or Bayesian networks:set of converging arrows = conditional pdf (causality)

Joint pdf:

‣ Undirected graphs or Markov networks:graph separation ⇔ conditional independence

Joint pdf:

P(X) = ∏t

P(Xt|Xπ(t))∏r

P(Xr)

P(X) =1Z

e−∑cVc(Xc)

cliquepotentials

parents rootsX3X1 X2

X4

X3X1 X2

X4

V12 V23

V34

Bayes

[Jordan, MacKay]

Markov Random Fields (MRF)

๏ Conditional dependence assumption:P(pixel | all others) = P(pixel | neighbors)‣ Define a neighborhood and clique system‣ Define the clique potentials Vi‣ Hammersley-Clifford theorem

X MRF ⇔ P(X) = e-U(X)/Z

(U: Gibbs energy = sum of Vi, Z: normalizing const.)

‣ Sampling from P(X) using Metropolis or Gibbs‣ Use of Markov Chain Monte Carlo (MCMC) methods for inference & estimation

first orderneighborhood

A pixel only depends on its neighbors!

first order cliques

[Besag 86, Geman 84]

undirected graphical models

Hidden & auxiliary variables

๏ Hidden variables

‣ Underlying process explaining the complexity of a model‣ Causality: directed graphical models (e.g. Gaussian mixture)‣ Examples: class labels in segmentation, line process in adaptive regularization (denoising/deblurring), missing data (bad CCD pixels)

๏ Auxiliary variables

‣ Created to facilitate modeling, optimization or sampling‣ Not necessarily causal relation‣ Examples: edge-preserving regularization (denoising/deblurring)

Yλ,δ

Bx

Bx

XXΜ

spatially variablemodel map

Joint P(X,M)1

2

3

M X Edge processes Bx ByHidden variables: model map

(optimal representations)

Markov trees & multiscale dependencies

๏ Encode dependencies between scaleswavelet coefficients propagate through scales...

‣ Markov tree structure: parent/children dependencies‣ Hidden variables: encode the implicit dependence“the variance propagates, but the value can change”

Joint subband histogram

ξtξπ(t)

Hidden Markov Tree (HMT)

P(ξt , ξπ(t)) ?

P(ξt | st)Gaussian pdf

P(st,sπ(t))discrete probabilities

wav

elet

tran

sfor

m

Multiscale,discrete

Gaussian mixturemodel

Bayesian inference

๏ Express the joint posterior pdf

‣ Build the full joint pdf: entire model + observed datae.g. use graphical models‣ P(entire model | data) ∝ full joint pdf

๏ Marginalization: eliminate the nuisance variableselimination algorithm, integration (Laplace approx.)

๏ Result: posterior marginal P(Θ | data)

‣ Usually there is no closed-form solution‣ Monte Carlo Approx. (sample from P)‣ Gaussian Approx.

•Mode = Maximum A Posteriori (MAP) found by optimization of -log P

•Uncertainties = Covariance Matrix [Σ] provided by the second derivatives of -log P @ optimum

YΘ

Hypothesis testing, model assessment, model selection & mixture...

Bayesian vs. classicalapproach: use prior P(Θ)

z

z

Θ Y

Bayesian estimation:when you have to make a decision

๏ Bayes Estimator:

‣ Statistics: function of the data θ=f(Y)‣ Cost function: estimation error C(θ,θ)‣ Bayes risk: estimator error EP(θ,Y)[C(θ,θ)]‣ Bayes estimator: arg minθ EP(θ,Y)[C(θ,θ)]

relatedestimator

MAPMaximum A Posteriori

PMPosterior Mean

cost fuction

In practice: Empirical Bayesian estimation(Laplace or saddlepoint approx.) [MacKay]

Y=data

[Marroquin 85]

Transforms & representations

Understand how changing representations can help model information in astronomical images

Find out how to efficiently separate the signal from the noise

Grasp the principles and properties of several multiresolution transforms

Projections onto various subspaces

๏ Simple projections: enforce constraints

‣ Strong support constraints (e.g. reduced spatial support)‣ Strong range constraints (e.g. positivity)‣ Projections onto convex sets

๏ Change of basis

‣ Orthonormal change of basis (e.g. Fourier, wavelets)‣ Arbitrary change of basis (e.g. frames, biorthogonal wavelets)

๏ Overcomplete representationsArbitrary number of vectors(e.g. mixture of models)

Project images, scalar or vector-valued pixels,geometric models, parameters...

Frequential representation (Fourier)

๏ Modeling

‣ Spatial convolution = single Fourier coefficient multiplicationdiagonalization of convolution-based operators‣ e.g. stationary self-similar processes (“1/f” power spectrum)

๏ Filtering

‣ Stationary & independent noise: same in the freq. space‣ Apply factor according to spatial frequency:Noise filtering, deconvolution, enhancement

๏ Reconstruction

‣ Blind deconvolution methods (unknown image and psf)‣ Interferometry

๏ Detection

‣ Template matching via cross-correlation (convol. matched filter)

Multiresolution analysis

Starck, Murtagh, BijaouiGalaxy details at different resolutions or scales

Multiresolution analysis & wavelets tutorial

Set of closed nested subspaces of

Approximation aj at scale j : projection of f on Vj

[Mallat, Vetterli, Daubechies, etc.]

Detail dj at scale j : proj. of f on Wj such that Vj-1 = Vj ⊕Wj Basis functions: scaling functions & wavelets

Multiscale Vision Model & applications

First 6 scales of “à trous” wavelet transform

Keeping only significant coefficients (3σ) Segmentation in significant regions

Establish interscale connexions:links between regions

Scale-space isosurface visualization

space

scale

[Bijaoui 95]

Wavelets & sparse representations

Asymptote E~N-1/2

Haar

Symmlet-8

coef noise

Sparse representation:Good approximation achieved by keeping only a small number of coefficients;Information concentrated in a few, high magnitude coefficients

approx.

Wavelet pyramid:few significant coefficientsw.r.t. original image

pixels noise

image space wavelet space

Approximation error vs. number of coefficients

connection withimage compression

Multi shape/resolution representations

๏ One wavelet is not enough!

‣ Different wavelets, different shapes & properties‣ Detection theory: correlation with template function to detectUse multiple templates

๏ Optimal representation

‣ Find “pixons” in images: circular or elliptical objects [Pina & Puetter 93]

Various scales, locations and intensitiesDetect various objects in noisy observations...the residual should be the observation noise‣ Find such objects in spectra (IFS) or in transform spaces:optimal representation (information theory)

examples of 2D wavelets

examples of 1D wavelets

Curse of dimensionality

Problem: high dimensionality of pixels (multi or hyperspectral data)

Statistical learning: number of bins = Nd

number of needed samples increases exponentially!

typical data in integral field spectroscopy:24x24 pixels, d=284

16x16 pixels, d=102480x80 pixels, d=2000

Solution: find low dimensional subspaces & project

Representing multidimensional data

๏ Principal Component Analysis (PCA)Image covariance matrix, eigenvectors = principal variation axesSelect largest eigenvalues and the related eigenvectors

๏ Independent Component Analysis (ICA)Search for independent sources (non-Gaussian!)

๏ Nonlinear manifold representation

๏ Deterministic/Probabilistic versions

‣ various modeling levels: with or without noise, single or mixture of principal components

ICA nonlinearmanifold

Functional optimization

Understand why optimization is needed in most processing tasks

Get to know the main optimization methods in multiple dimensions

Know how to choose between various deterministic and stochastic algorithms

Why optimization is needed

๏ Direct, iterative methods

‣ No explicit definition of a forward model‣ However, the solution is sought by an optimization procedureConstrained optimization of an objective or cost function;

•“Simplest” solution (e.g. max. entropy/smoothness) under data-related constraints: set-theoretic (e.g. positivity) or strong (e.g. data prediction)

•“Most likely” solution (e.g. least squares) under model-related constraints: set-theoretic (e.g. smoothness, bounds) or strong (e.g. normalization)

๏ Inverse, iterative methods

‣ Explicit forward modeling; no closed-form solution‣ Solution given by optimizing a functional F:

•Probabilistic methods: F = - log posterior pdf

•Other methods: F = Distance(prediction, observations)

๏ Different kinds of optimization:‣ image processing: image = arg min F‣ parameter fitting: parameters = arg min F

in some cases, functional F = processed image (e.g. CLEAN)

Deterministic methods

๏ Optimization without derivatives

‣ Simplex method (linear programming)‣ Principal axis method (Brent)

๏ Gradient-based methods

‣ Steepest descent (“simple” functions)Use line optimization only‣ Newton’s method (convex functions)Use the Hessian (2nd derivative)‣ Quasi-Newton & modified Newton methodsUse an approximation/correction of the Hessian‣ Gauss-Newton methods (sum of squares)‣ Conjugate gradient: linear/nonlinear (contours ≈ quadratic form)Linear case for quadratic form: conv. finite number of steps

Deterministic optimization: problems, solutions

๏ Low computational efficiency

‣ Choose an appropriate optimization method!‣ Take into account the dependence structure

•Choose the representation where the dependence is minimized (diagonalization)

•Perform single variable optimization whenever possiblee.g. Iterative Conditional Modes (ICM)

๏ Multiple local optima, nonconvexity

‣ Multigrid optimizationUse coarse to fine approximations of the objective function,initialize each level with the result of the previous level‣ Graduated nonconvexity & approximations‣ Auxiliary variables‣ Stochastic methods...

Objective function: high dimensionality & nonlinearity!

multigrid

Example - quasi-Newton optimization

Iterative optimization scheme: object & camera param. estimation

–! Linearize the intensity (rendering):

⇒ objective function approx. by a quadratic form

! → optimization of S, Θ using a conjugate gradient

–! Result: used to initialize the next iteration–! Convergence: small variation of S, Θ

€

I(S,Θ) ≈ I(S0,Θ0) +∂I∂S

S − S0( ) +

∂I∂Θ

Θ−Θ0( )

1

2

3

Each step is simpler than the original problem(linear optimization)

S = geometric object modelΘ = camera parameters

Auxiliary variable methods

๏ Half-quadratic extensions“ φ-functions”: quadratic near 0, linear or log-like at ∞

‣ Additive extension φ(u)=infb (b-u)2+ψ1(b)‣ Multiplicative extension φ(u)=infb bu2+ψ2(b)‣ Alternate optimizations w.r.t. b and u [Charbonnier 94]

๏ Missing data, augmented process: Expectation-Maximization [Baum 72, Dempster 77]

ϑz

complex P(ϑ | Y)augmented process P(z, ϑ | Y)

Nonlinear regularization(e.g. deblurring or denoising)

E

M

Q simpler to optimize than P(ϑ | Y)

Stochastic methods

๏ Exception: no optimization

‣ Bayesian inference: full pdf P or its properties‣ Monte Carlo methods

•Sample from the distribution P ∝ e-U(X) where U=objective function

•Compute the mode and higher order moments: approx. the pdf P

๏ Simulated annealing [Kirkpatrick 83]

‣ Allow for “wrong” directions:escape from local optima‣ Sample by slowly decreasing thetemperature T in P ∝ e-U(X)/T

e.g. Deblurring, blind deconvolution,optimal representations via nonlinear fitting

Genetic algorithms

๏ Principle:‣ Choose initial population‣ Repeat

- Evaluate the individual fitnesses of a certain proportion of the population

- Select pairs of best-ranking individuals to reproduce

- Apply crossover operator

- Apply mutation operator‣ Until terminating condition

Examples:Multispectral image classification[Petremand et al. 05]Fitting galaxy rotation curves,variable star period determination[Charbonneau 95]

crossover mutation

[Holland 75]

Derivatives & partial differential equations

Learn how to compute the derivatives needed in deterministic optimization methods

See how some iterative processing techniques amount to solving partial differential equations

Computing derivatives

A

B

C

A

B1

C

B2

€

∂C∂A

=∂C∂B

×∂B∂A

€

∂C∂A

=∂C∂B1

×

∂B1∂A

+

∂C∂B2

×

∂B2∂A

series parallel

Basic tool: the chain rule€

∂A∂π i

€

∂π i

∂Pj

€

∂W∂A

2D polygon vertex

polygon area

pixel intensity

Example: rendering a polygonal objectderiv. of a pixel intensity W w.r.t. model parameters Pj

... How does a change in the modelaffect the predicted image intensity?

• deterministic optimization algorithms• help compute uncertainties

why computethe derivatives?

Partial Differential Equations in image processing

๏ Isotropic diffusion: “heat equation”

‣ Related to physics: each pixel has a temperature‣ Gaussian smoothing (convolution) of the image

๏ Anisotropic diffusion

‣ Smooth only along object edges‣ Adaptive smoothing, edge-preserving, sharper details‣ Connexions with nonlinear regularization

Image denoising e.g. semi-implicit schemegeneral form

noisy image isotropic anisotropic

∂tI =12

∇2I

Mathematical morphology

Recall of the basic principles of mathematical morphology

Understand how simple morphological tools can be applied to astronomical images

Principles

๏ Set-theoretic approach

‣ Structuring element (neighborhood system) - shape‣ Sets of values, rank operations (e.g. min, max)

๏ Basic Functions

‣ Erosion: min {In}‣ Dilation: max {In}

๏ Morphological tools

‣ Combination opening (erosion-dilation) / closing (dilation-erosion) / top-hat (opening-subtract.)‣ Hit-and-miss, skeleton, reconstruction, thinning...

๏ Morphological filters

‣ Smoothing: open-close‣ Gradient, Laplacian...

[Matheron, Serra 82]

Application examples

top-hat transform:erosion, dilation, difference

top-hat:background &halo removal

open-close:morphological

smoothing

original imageAbell 3698

[Can

déas

et

al, 19

97]

Star extraction and mapping Star/galaxy classification Multispectral image segmentation

...

Further reading

๏ An interactive image processing coursehttp://www.ph.tn.tudelft.nl/Courses/FIP

๏ Mathworld (Wolfram research)http://mathworld.wolfram.com/topics/Optimization.htmlhttp://mathworld.wolfram.com/topics/ProbabilityandStatistics.html

๏ IRIS tutorial on CCD image processing, C. Builhttp://www.astrosurf.org/buil/us/iris/iris.htm

๏ ICCV’03 course on learning & vision (Blake, Freeman, Bishop, Viola)http://people.csail.mit.edu/billf/learningvision/

๏ Various wavelet resourceshttp://www.ee.umanitoba.ca/~ferens/wavelets.html

๏ MacKay Book on information theory, inference & learninghttp://www.inference.phy.cam.ac.uk/mackay/itprnn/book.html

๏ Introduction to graphical models & bayesian networkshttp://www.cs.ubc.ca/~murphyk/Bayes/bnintro.html

๏ Statistical learning, decision, graphical models (M. Jordan)http://www.cs.berkeley.edu/~jordan/courses.html