Robin HoganJulien DelanoëNicola Pounder

University of Reading

Synergistic cloud, aerosol Synergistic cloud, aerosol and precipitation and precipitation

productsproductsProgress so far in RATECProgress so far in RATEC

HDM

Target ID

IWC LWC

PWC andsize parameter

Vertical Motion

HSRL

Feature Mask

TargetClassification

AerosolIce CloudProperties

CloudProperties

Cloud / AerosolMask

AerosolProperties

CPR ATLID MSI

AM-AerosolAM-Ice Cloud

PropertiesAC – TargetClassification

AM – Aerosol

ACM – Ice cloud

Cloud fraction,overlap, water content

on model grid

3D sceneReconstruction

ACM – Mergedobservations

(common grid)

ACM – Liquid cloud

Along-track heatingrates and fluxes

3D scene heatingrates and fluxes

ACM – Cloud, precip.,aerosol best estimate

AC – Ice cloud

AC – Liquid cloud

AC – Cloud, precip.,aerosol best estimate

CM – Ice cloud

CM – Liquid cloud

CM – Cloud, precip.,aerosol best estimate

AM – Ice cloud

AM – Liquid cloud

AM – Cloud and aerosol best estimate

In c

as

e o

f in

str

um

en

t fa

ilu

re

an

d/o

r c

on

sis

ten

cy

ch

ec

kin

g

If a

ll i

ns

tru

me

nts

w

ork

co

rre

ctly

Develop “best estimate” first

Other products follow from this

Synergy products as defined in CASPER

Overview of our role in RATECOverview of our role in RATEC• “Radiative transfer for EarthCARE” (RATEC)

– Lead contractor: Howard Barker– Started August 23, 2009– 15 month project

• Our role is to start the development of synergy algorithms– Also define the required imager and target classification products

• WP120: Review - first 3 months– Products and Algorithms Requirements Document (PARD)– Algorithm Development Plan (ADP)

• WP 220: Design - next 5 months– Product Definition Document (PDD)

• WP 320: Prototype and test - final 7 months– Algorithm Theoretical Basis Document (ATBD)

Overview of talkOverview of talkLaying the foundations for the algorithm...

• Summary of retrieval framework– State variables (describing ice, liquid, rain and aerosol)– Forward models (for radar, HSRL lidar, infrared & solar

radiometers)– Cost function

• Minimization techniques– Gauss-Newton & Levenberg-Marquardt– Gradient Descent (adjoint method)– Ensemble methods

• Coding progress– C++ library– Scattering code

• Calculation and storage of error descriptors– Solution error covariance matrix– Averaging kernel

• Progress for specific atmospheric constituents– Liquid cloud: gradient constraint

Retrieval Retrieval frameworframewor

kkIngredients developed before

Not yet developed

1. New ray of data: define state vector

Use classification to specify variables describing each species at each gateIce: extinction coefficient , N0’, lidar extinction-to-backscatter ratio

Liquid: extinction coefficient and number concentrationRain: rain rate and mean drop diameterAerosol: extinction coefficient, particle size and lidar ratio

3a. Radar model

Including surface return and multiple scattering

3b. Lidar model

Including HSRL channels and multiple scattering

3c. Radiance model

Solar and IR channels

4. Compare to observations

Check for convergence

6. Iteration method

Derive a new state vector

3. Forward model

Not converged

Converged

Proceed to next ray of data

2. Convert state vector to radar-lidar resolution

Often the state vector will contain a low resolution description of the profile

5. Convert Jacobian to state-vector resolution

Jacobian initially will be at the radar-lidar resolution

7. Calculate retrieval error

Error covariances and averaging kernel

Proposed state variablesState variable Representation with height / constraint A-priori

Ice clouds and snow

Visible extinction coefficient One variable per pixel with smoothness constraint None

Number conc. parameter Cubic spline basis functions with vertical correlation Temperature dependent

Lidar extinction-to-backscatter ratio Cubic spline basis functions 20 sr

Riming factor Likely a single value per profile 1

Liquid clouds

Liquid water content One variable per pixel but with gradient constraint None

Droplet number concentration One value per liquid layer Temperature dependent

Rain

Rain rate Cubic spline basis functions with flatness constraint None

Normalized number conc. Nw One value per profile Dependent on whether from melting ice or coallescence

Melting-layer thickness scaling factor One value per profile 1

Aerosols

Extinction coefficient One variable per pixel with smoothness constraint None

Lidar extinction-to-backscatter ratio One value per aerosol layer identified Climatological type depending on region

Ice clouds largely done in CASPER;Snow & riming in convective clouds needs to be added (VARSY?)

Liquid clouds to be tackled in RATEC

Basic rain to be added in RATEC; Full representation in VARSY?

Basic aerosols added in RATEC; Full representation via collaboration (IRMA?)

Forward model components• From state vector x to forward modelled observations H(x)...

Ice & snow Liquid cloud Rain Aerosol

Ice/radar

Liquid/radar

Rain/radar

Ice/lidar

Liquid/lidar

Rain/lidar

Aerosol/lidar

Ice/radiometer

Liquid/radiometer

Rain/radiometer

Aerosol/radiometer

Radar scattering profile

Lidar scattering profile

Radiometer scattering profile

Lookup tables to obtain profiles of extinction, scattering & backscatter coefficients, asymmetry factor

Sum the contributions from each constituent

Gradient of radar measurements with respect to radar inputs

Gradient of lidar measurements with respect to lidar inputs

Gradient of radiometer measurements with respect to radiometer inputs

Jacobian matrix

H=y/x

Lots of expensive matrix multiplications: likely to be the

most expensive part of the entire algorithm

x

Radar forward modelled obs

Lidar forward modelled obs

Radiometer forward modelled obs

H(x)Radiative transfer models

Solution methods (1/3)• We want to minimize this cost function:

• Possibly using the gradient (a vector):

• …and the second derivative (a matrix):

1. Gauss-Newton method (Rodgers p85):

– Advantage: rapid convergence (instant convergence for linear problems)

– Another advantage: get the error covariance of the solution “for free”– Disadvantage: need the Jacobian of every forward model: can be

expensive

112 BHRHTJ

axBaxxyRxy 11

2

1)()(

2

1 TT HHJ

axBxyRH 11 )(HJ T

JJii

12

1 xx

Solution methods (2/3)2. Gradient descent method (e.g. ECMWF data assimilation system):

– Advantage: we don’t need to calculate the Jacobian so forward model is cheaper!

– Quasi-Newton method to get search direction (conjugate gradient, BGFS etc) – Disadvantage: more iterations needed since we don’t know curvature of J(x)– Disadvantage: don’t get the error “for free” at the end

• Why don’t we need the Jacobian H?

– The “adjoint” of a forward model takes as input the vector {.} and outputs the vector Jobs without needing to calculate the matrix H on the way

– Adjoint can be coded to be only ~3 times slower than original forward model– Tricky coding for newcomers, although some automatic code generators

exist

Jii δ1 xx

)(1 xyRH HJ Tobs

Simple steepest descent requires many steps

Conjugate gradient method more intelligent…

Solution methods (3/3)3. Levenberg-Marquardt (hybrid of Gauss-Newton & steepest descent):

– If cost function reduces, keep small: rapid convergence– If cost function increases, increase so that always go downhill– More robust than Gauss-Newton, but still need the Jacobian

4. Ensemble method (e.g. Ensemble Kalman Filter):– Create ensemble of “trial solutions” or “particles” in state space– Spread of results allows Jacobian to be calculated “automatically” to

create better set of particles: repeat until convergence– Applied to GPM retrievals by Mircea Grecu (NASA)– Advantage: no Jacobian required– Estimate of retrieval error from final spread – Disadvantage: may need many ensemble members

so probably slower than adjoint method5. Simulated annealing (e.g. Donovan in ECSIM)

– Suitable for very non-linear problems– Too slow for this application?

JJii

12

1 ηIxx

......

• Computational cost can scale with number of points describing vertical profile N; we can cope with an N2 dependence but not N3

Radiative transfer forward models

Radar/lidar model Applications Speed Jacobian Adjoint

Single scattering: ’= exp(-2) Radar & lidar, no multiple scattering N N2 N

Platt’s approximation ’= exp(-2) Lidar, ice only, crude multiple scattering N N2

Photon Variance-Covariance (PVC) method (Hogan 2006, 2008)

Lidar, ice only, small-angle multiple scattering

N or N2 N2 N

Time-Dependent Two-Stream (TDTS) method (Hogan and Battaglia 2008)

Lidar & radar, wide-angle multiple scattering

N2 N3 N2

Depolarization capability for TDTS Lidar & radar depol with multiple scattering N2 N2

Radiometer model Applications Speed Jacobian Adjoint

RTTOV (used at ECMWF & Met Office) Infrared and microwave radiances N N

Two-stream source function technique (e.g. Delanoe & Hogan 2008)

Infrared radiances N N2

RADIANT (from Graeme Stephens) Solar radiances N N2 N

• Lidar will use PVC+TDTS, radar will use single scattering+TDTS• Jacobian of TDTS is too expensive: need to develop reduced-resolution Jacobian

or alternatively adjoint code• Also need depolarization forward model with multiple scattering

• Infrared will probably use RTTOV, solar radiances will use RADIANT

Coding progressCoding progress• Coding has started on the underpinning libraries

– Use C++ because of key language features• Matrix and Vector Expression Library

– Operator overloading and expression templates allows complex operations on matrices to be written on one line, with similar efficiency as Fortran-90

– Important feature is to allow a matrix object to behave as part of another matrix – important when different parts of the Jacobian refer to different instruments and need to be passed to functions in bits

• Optimal Synergy Library– Object orientation and polymorphism: Radar and Lidar classes inherit

from generic Observation class; IceCloud and Aerosol inherit from generic Constituent class

– Enables code to be written to be completely flexible: observations can be added and removed without needing to keep track of indices to matrices, so same code can be applied to many platforms

• Also written code to provide scattering libraries in NetCDF files

Storage of Storage of error error

descriptorsdescriptors

“M

easu

rem

en

ts”

Retr

ievals

Retr

ievals

“M

easu

rem

en

ts”

• If ice is present at N points and M variables are retrieved:– MN retrieved variables to

report with MN errors• Error covariance and averaging

kernel matrices also needed– A standard output from a

variational algorithm– But ~(MN)2/2 values in each

matrix, much larger than the retrieved variables and their errors

– Matrix is a different size each ray!

• Need a suitable compression strategy...

Top -> Height -> bottom

Top

-> H

eigh

t ->

bott

om

Top -> Height -> bottom

Top

-> H

eigh

t ->

bott

om

S

N’

N’

Retrieval error covariance matrix (Extinction, lidar ratio, N’)

Retrieval error correlation matrix (Extinction, lidar ratio, N’)

Top -> Height -> bottom

Top

-> H

eigh

t ->

bott

om

Top -> Height -> bottom

Top

-> H

eigh

t ->

bott

om

S

N’

N’

For autocorrelation of For autocorrelation of extinction...extinction...

• 3 Gaussians requires 8N or fewer values to approximate the error covariance matrix

• Need to work on correlations between different variables

Hei

ght (

m)

Information content

1 1 1 1( ) T TA H R H B H R H

Averaging kernelAveraging kernel• The averaging kernel matrix

expresses how the retrieval at a point depends on the truth at every other point– The sum of each column

expresses how much the retrieval at that point derives from the measurements (rather than the prior): 1 is good!

– The shape of each column expresses the effective vertical resolution of the retrieval: more peaked is good!

• Easy to parameterize:– Store integral and peak– Or store integral and the

width for each column

Gradient constraintGradient constraint

• We have a good constraint on the gradient of the state variables with height for:– LWC in stratocu (adiabatic profile, particularly near cloud base)– Rain rate (fast falling so little variation with height expected)

• Not suitable for the usual “a priori” constraint• Solution: add an extra term to the cost function to penalize

deviations from gradient c:

A. Slingo, S. Nichols and J. Schmetz, Q. J. R. Met. Soc. 1982

2

i

i

dxJ c

dz

Example in liquid cloudsExample in liquid clouds• Using simulated observations:

– Triangular cloud observed by a 1- or 2-field-of-view lidar– Retrieval uses Levenberg-Marquardt minimization with Hogan and

Battaglia (2008) model for lidar multiple scattering

Optical depth=30Footprint=100mFootprint=600m

Two FOVs: very good performanceOne 10-m footprint: saturates at optical depth=5One 100-m footprint: multiple scattering helps!One 10-m footprint with gradient constraint: can

extrapolate downwards successfully

Ongoing workOngoing work• Implement various instrument forward models

– Not too difficult: interfacing different codes and languages– Write adjoint for slowest models

• Solver– Test gradient descent algorithms (conjugate gradient, BGFS etc)

• Implementation of different atmospheric constituents– Ice cloud: we know how to do this– Liquid cloud: to be done in RATEC– Aerosol and precipitation: basic implementation in RATEC; longer-

term development needs collaboration with DAME, IRMA etc...• Test datasets

– Several airborne comparisons for validation (TC4, POLARCAT etc)– A-Train– ECSIM (after RATEC)

• Two-instrument synergy products– To be tested using same code removing one instrument (after

RATEC)

Aerosol refractive index• Values from Ellie

Highwood• Strong

dependence on aerosol type

• Varies with hydration

• Can we use one continuous variable to represent different aerosol properties?

• Need to make simplifications if information from multiple wavelengths is to be combined

355-nm lidar

Solar MSI channels Thermal MSI channels

Matrix and Vector Expression Matrix and Vector Expression LibraryLibrary

• Intuitive interface and ability to do complex expressionsMatrix M(3,3);

Vector x(3) = 1.0, 2.0, 3.0; // Simple initialization

M = N * R + exp(P); // ‘*’ invokes element multiplication

y = transpose(x) ()* inverse(R); // ‘()*’ invokes matrix multiplication

x = solve(A, b); // Linear algebra• Important aspects not in any other library (to my knowledge)

S.subset(M, 3, 30, 2, 20); // Matrix subsetting

M(find(M < 0.0)) = 0.0; // Matlab-style indexing

BitfieldMatrix b; b.bit(4) = 1; // Handling of classification data• Efficient and robust implementation

– “Expression templates” to avoid temporary objects in long expressions

– Reference counting for allocated objects protects against memory leaks

Key classesStateMethods:• solve()• add_obs(Observation)Data:• List of Observations• List of Constituents• Vector x, y• Matrix Jacobian etc.

ObservationMethods:• calc_scat_props() • virtual forward_model()Data:• Real wavelength• Vector y, y_error• Matrix Jacobian

ConstituentMethods:• calc_scat_props(wavelength) Data:• Vector x, x_prior

ActiveInstrumentMethods:• forward_model()

Lidar

RadiometerMethods:• forward_model() Radar

IceCloud

LiquidCloud

Aerosol

Rain

BackgroundConstituents

Forward model part 1Forward model part 1xice

.

.

.

.

.xliq

.

.

.

State vector x

ln vis

.

.

.ln N0’ .

ln vis

.

.

.ln N0

*

.

.

.

xhighres,ice / xice

Basis functions for variables

represented at reduced

resolution

=

xhighres,ice W1 xice

radar

.

.

.radar

.

.

.

xradar,ice

/ xhighres,ice

xradar,iceW2

Scattering lookup table

To radiative transfer

radar

.

.

.radar

.

.

.

0000radar

.

.

.

radar

.

.

.radar

.

.

.

radar

.

radar

.

xradar

/ xradar,ice

= + +

xradar,ice

W3

xradar,gas

xradar,liq

xradar

Sum the contributions from each atmospheric constituent

Forward model part 2Forward model part 2

Z...

y’radar / xradar

Radiative transfery'radar Hradar

To solver

Z...'...I

y'

y’ / x

H

Forward modelled observations and Jacobian

y’radar /xice

Hradar,ice

y’radar / xradar xradar

/ xradar,ice

xradar,ice

/ xhighres,ice

xhighres,ice

/ xice

W3 W2 W1Hradar

=