Robin Hogan, Nicola Pounder Robin Hogan, Nicola Pounder University of Reading, UKUniversity of Reading, UKJulien Delanoë Julien Delanoë LATMOS, FranceLATMOS, France

Retrieving Retrieving consistent profiles consistent profiles of clouds and rain of clouds and rain from instrument from instrument synergysynergy

OverviewOverview• The synergy of instruments in the A-Train and EarthCARE offer more

accurate retrievals of cloud & light precipitation than ever before– But to exploit integral measurements (radar PIA, radiances) we must

retrieve cloud and precipitation simultaneously– Variational approach offers rigorous treatment of errors

• Behaviour should tend towards existing two-instrument synergy algos– Radar+lidar for ice clouds: Donovan et al. (2001), Delanoe & H (2008)

• DARDAR available at: http://www.icare.univ-lille1.fr/projects/dardar/

– CloudSat+MODIS for liquid clouds: Austin & Stephens (2001)– Calipso+MODIS for aerosol: Kaufman et al. (2003)– CloudSat surface return for rainfall: L’Ecuyer & Stephens (2002)

• This talk presents progress towards “unified algorithm” for EarthCARE– Retrieval framework – Automatic adjoints– Ice, rain and melting-ice retrieval: testing on simulated profiles– Demonstration on A-Train data– Future work

Unified Unified retrievalretrieval

Ingredients developedImplement previous work

Not yet developed

1. New ray of data: define state vector

Use classification to specify variables describing each species at each gateIce: extinction coefficient, N0’, lidar extinction-to-backscatter ratio

Liquid: extinction coefficient and number concentrationRain: rain rate, drop diameter and melting iceAerosol: extinction coefficient, particle size and lidar ratio

3a. Radar model

Including surface return and multiple scattering

3b. Lidar model

Including HSRL channels and multiple scattering

3c. Radiance model

Solar and IR channels

4. Compare to observations

Check for convergence

6. Iteration method

Derive new state vector using quasi-Newton scheme

3. Forward model

Not converged

Converged

Proceed to next ray of data

2. Convert state vector to radar-lidar resolution

Often the state vector will contain a low resolution description of the profile

7. Calculate retrieval error

Error covariances and averaging kernel

Unified retrieval: forward Unified retrieval: forward modelmodel

• From state vector x to forward modelled observations H(x)...

Ice & snow Liquid cloud Rain Aerosol

Ice/radar

Liquid/radar

Rain/radar

Ice/lidar

Liquid/lidar

Rain/lidar

Aerosol/lidar

Ice/radiometer

Liquid/radiometer

Rain/radiometer

Aerosol/radiometer

Radar scattering profile

Lidar scattering profile

Radiometer scattering profile

Lookup tables to obtain profiles of extinction, scattering & backscatter coefficients, asymmetry factor

Sum the contributions from each constituent

x

Radar forward modelled obs

Lidar forward modelled obs

Radiometer fwd modelled obs

H(x)Radiative transfer models

Adjoint of radar model (vector)

Adjoint of lidar model (vector)

Adjoint of radiometer model

Gradient of cost function (vector)

xJ=HTR-1[y–H(x)]

Vector-matrix multiplications: around the same cost as the original forward

operations

Adjoint of radiative transfer models

yJ=R-1[y–H(x)]

Gauss-Newton method

• Requires the curvature 2J/x2

– A matrix– More expensive to calculate

• Fewer iterations to converge– Assume J is quadratic and

jump to the minimum• Limited to smaller retrieval

problems

J

x

x1

J/x2J/x2

Minimization methods - in Minimization methods - in 1D1DQuasi-Newton method (e.g. L-BFGS)

• Rolling a ball down a hill– Smart choice of direction in

many dimensions aids convergence

• Requires the gradient J/x– A vector (efficient to store)– Efficient to calculate using

adjoint method• Used in data assimilation

J

x

x2x3x4x5x6x7x8

x1

J/x

x2x3

x4x5

Coding up adjoints and Jacobian matrices is very time consuming and error prone – is there a better way?

adouble algorithm(const adouble x[2]) {adouble y = 4.0;adouble s = 2.0*x[0] + 3.0*x[1]*x[1];y *= sin(s);return y;

}// Main codeadept::Stack stack;y = algorithm(x);stack.compute_adjoint(); // Do the hard work

• This can be done automatically in C++ using a special active type “adouble” and overloading mathematical operators and functions

• Existing libraries CppAD and ADOL-C provide this capability but they’re quite slow

• New library “Adept” uses expression templates in C++ to do this much faster

• Freely available at http://www.met.rdg.ac.uk/clouds/adept• Hogan (Mathematics & Computers in Simulation, in review)

Automatic adjoints Automatic adjoints • Algorithm y(x) in C++:

double algorithm(const double x[2]) {double y = 4.0;double s = 2.0*x[0] + 3.0*x[1]*x[1];y *= sin(s);return y;

}

double algorithm_AD(const double x[2],double y_AD[1], double x_AD[2]) {

double y = 4.0;double s = 2.0*x[0] + 3.0*x[1]*x[1];y *= sin(s);/* Adjoint part: */double s_AD = 0.0;y_AD[0] += sin(s) * y_AD[0];s_AD += y * cos(s) * y_AD[0];x_AD[0] += 3.0 * s_AD;x_AD[1] += 6.0 * x[0] * s_AD;s_AD = 0.0;y_AD[0] = 0.0;return y;

}

• Quite fiddly and error-prone to code-up dJ/dx given dJ/dy

Benchmark resultsBenchmark results

Adjoint Jacobian (50x350)

Hand-coded 3.0

New C++ library: Adept 3.5 20

ADOL-C 25 83

CppAD 29 352

• Tested PVC and TDTS multiple scattering algorithms (here for PVC)• Time relative to original code for profile with N=50 cloudy points:

• Adjoint calculation is:– Only 5-20% slower than hand-coded adjoint– 5-9 times faster than leading alternative libraries ADOL-C and

CppAD• Jacobian calculation is:

– 4-20 times faster than ADOL-C/CppAD for a matrix of size 50x350• Now used for entire unified algorithm• Sorry, it won’t work for Fortran until Fortran has template capability!

Retrieved (state) variablesRetrieved (state) variables• Ice clouds and snow (extension of Delanoe and Hogan 2008, 2010)

– [1] log extinction coefficient, [2] log lidar ratio– [3] log number conc parameter N0* (good a priori from temperature)– [4] “Riming factor” to allow Doppler information to be assimilated – Lidar molecular signal is used beyond cloud, when detectable

• Liquid clouds (see Nicola Pounder’s talk)– [1] log LWC, [2] total number concentration (one value per liquid layer)– Extinction info from lidar multiple scattering, radar PIA, SW radiances

• Rain and melting ice– [1] log rain rate, [2] number conc parameter Nw* (one value per profile)– “Flatness constraint” on rain rate: extra terms in cost function penalize

deviations from a constant rain rate with height– Different a-priori values for stratocu drizzle and rain from melting ice– Melting layer radar attenuation uses Matrosov’s empirical relationship:

2-way attenuation [dB] = 2.2 x rain rate [mm h-1]– Additional term in cost function penalizes difference in snow flux above

melting layer and rain rate below– Use radar PIA information over the ocean

Idealized nimbostratus Idealized nimbostratus retrievalretrieval

Constraint of constant flux across melting layer and

smooth rain profile, but we have the classic ill-

constrained attenuation retrieval problem

Idealized nimbostratus Idealized nimbostratus retrievalretrieval

Much better behaviour with

flatness constraint on rain rate

A-Train caseA-Train case• Forward modelled radar and

lidar match observations well, indicating good convergence

• Can also simulate the Doppler velocity that would be observed by EarthCARE– Currently omits multiple

scattering and air motion effects on Doppler

RetrievalsRetrievals• Ice cloud properties

retrieved similarly to Delanoe and Hogan (2008, 2010) algorithm

• Water flux is approximately conserved across the melting layer

• Rain rate is relatively constant with range

• Forward models and observations– Implement LIDORT solar radiance model (has adjoint/Jacobian)– Implement Delanoe & Hogan infrared radiance code– Implement multiple scattering model with depolarization (but are

measurements too noisy?)– Incorporate radar path integrated attenuation– Incorporate Doppler velocity

• Retrieved quantities– Add “riming” factor– Refine aerosol retrieval

• Verification– Consistency of different sources of information using A-Train data– Aircraft data with in-situ sampling from NASA ER-2 and French

aircraft– EarthCARE simulator (ECSIM) scenes using EarthCARE

specification

Further workFurther work

and 2nd derivative (the Hessian matrix):

Gradient Descent methods

– Fast adjoint method to calculate xJ means don’t need to calculate Jacobian

– Disadvantage: more iterations needed since we don’t know curvature of J(x)

– Quasi-Newton method to get the search direction (e.g. L-BFGS used by ECMWF): builds up an approximate inverse Hessian A for improved convergence

– Scales well for large x– Poorer estimate of the error at the

end

Minimizing the cost functionMinimizing the cost function

Gradient of cost function (a vector)

Gauss-Newton method

– Rapid convergence (instant for linear problems)

– Get solution error covariance “for free” at the end

– Levenberg-Marquardt is a small modification to ensure convergence

– Need the Jacobian matrix H of every forward model: can be expensive for larger problems as forward model may need to be rerun with each element of the state vector perturbed

112 BHRHxTJ

axBaxxyRxy 11

2

1)()(

2

1 TT HHJ

axBxyRHx 11 )(HJ T

JJii xxxx

12

1 Jii xAxx 1

Ice fall speedsIce fall speeds• Heymsfield & Westbrook

(2010) expression predicts fall speed as a function of particle mass, maximum dimension and “area ratio”

• Currently we assume Brown and Francis (1995) mass-size relationship, so fall speed is a function of size alone

Terminal fall-speed (m s-1)

Brow

n & F

rancis (1995)

• In convective clouds, intend to add a multiplication factor (or similar) to allow denser particles (e.g. rimed aggregates, graupel and hail) to be retrieved using the Doppler measurements

Simple melting-layer modelSimple melting-layer model

• Minimalist approach:– 2-way radar

attenuation in dB is 2.2 times rain rate (Matrosov 2008)

– No effect on other variables

– Add term to cost function penalizing difference between ice flux above and rain flux below melting layer

Matrosov (IEEE Trans. Geosci. Rem. Sens. 2008)

A mixed-phase caseA mixed-phase case• Observations • Retrievals

Model skillModel skill• Use “DARDAR” CloudSat-

CALIPSO cloud mask• How well is mean cloud

fraction modelled?– Tend to underestimate

mid & low cloud fraction• How good are models at

forecasting cloud at right time? (SEDI skill score)– Winter mid to upper

troposphere: excellent– Tropical mid to upper

troposphere: fair– Tropical and sub-tropical

boundary layer: virtually no skill!

• Hogan, Stein, Garcon & Delanoe (in prep)