GlobAlbedo data processing, products and validation - · PDF fileGlobAlbedo data processing, products and validation ... assets to generate Essential Climate ... all lower than tower

www.GlobAlbedo.org

www.QA4ECV.eu

GlobAlbedo UCM-2 17th September’14 1

GlobAlbedo data processing, products and validation

Jan-Peter Muller, Said Kharbouche, Gill Watson

Neville Shane, Gerardo López

Mullard Space Science Laboratory, UCL

P.Lewis UCL Geography

Jürgen Fischer, Luis Guanter, Réné Preusker

Freie Universität Berlin

Peter North, Andreas Heckel, Swansea University

Olaf Danne, Marco Zülhke, Carsten Brockmann, Brockmann Consult

Simon Pinnock, ESA ESRIN

with contributions from Alessandro Cescatti (JRC); Zhuosen Wang & Crystal Schaaf (U. of Mass.,

Boston), Gabriela Schaapman-Strub (U of Zurich)

www.GlobAlbedo.org

www.QA4ECV.eu


Global Land surface broadband albedo products• Produce 14 years (1998-2014) of Land Surface BroadBand

BRDF/Albedo every 8 days at 1km & monthly 0.05º & 0.5º from

European & US space assets to generate Essential Climate Variables.

• Quality Assurance at each and every pixel using optimal estimation.

• Input data level 1b (radiometrically calibrated, satellite projection) with

a priori climatology derived from MODIS C5 BRDF (3/2000-3/2014)

– VEGETATION (3/1998-12/2011)

– MERIS (6/2002-12/2011)

– Prior derived from MODIS MCD43A1,2 (3/2000-3/2010)

• Validation of final albedo products & intermediate products (e.g.

sensor-to-sensor, cloud masks, aerosol AOD, narrow-to-broadband)

• GlobAlbedo products freely available via wget/curl, http and display via

animations and an OGC-compliant webGIS

www.GlobAlbedo.org

www.QA4ECV.eu


GlobAlbedo Algorithm & Validation• MERIS & VGT I/P

• Subset of GlobAlbedoproductsvalidated

• Focus onPixel IDAOTSDRN-to-BBAlbedo

• Internal validation performed byrelevant producer

• Albedo validation performed over FLUXNET, BSRN, SURFRAD sites intercomparison with other EO

Overall GlobAlbedo processing chain GlobAlbedo product flowchartLewis et al., IGARSS12Muller et al., IGARSS12

BC SU

FUB

www.GlobAlbedo.org

www.QA4ECV.eu


Processing chain in Java BEAM and distributed processing

• VEGETATION

level1b-to-

BBDR at

Swansea

• MERIS level-1b

to-BBDR at FUB

• BBDR-to-

BRDF/albedo at

MSSL using

MERIS &

VEGETATION

BBDRs and

MODIS prior

www.GlobAlbedo.org

www.QA4ECV.eu


What did it take to get to this point in time (Level1b-to-BBDR)

• Level-1b to BBDR (71.2Tb L1b to 38Tb BBDR)

– MERIS:

• INPUT - 27.5Tb of MERIS L1b (≈44,760 orbits)

• 5 months of processing burning 3 staff-months

• OUTPUT – 21 Tb of BBDR

– VEGETATION

• INPUT –8.1 Tb of VGT L1b (≈ 67,380 orbits)

• 5 months of processing (on 240 cores) burning 3 staff-months


– (A)ATSR L1b-BBDR (estimated excluding AATSR-MERIS co-registration)

• INPUT - 22Tb of (A)ATSR L1b (≈77,750 orbits)

• ~4 months of processing (on 240 cores) burning 3 staff-months


www.GlobAlbedo.org

www.QA4ECV.eu


What did it take to get to this point in time?

(BBDR-to-BRDF-to-Albedo-to-mosaics-to-browse)

• BBDR to BRDF/albedo

– INPUT: 38 Tb of VGT+MERIS BBDRs + 17 Tb of MODIS priors

– OUTPUT: 56Tb of BRDF; 28 Tb of Albedo; 7Tb of albedo netCDF

– 200 days of processing time (on 224 cores) burning 4 staff-months

– Number of parallel processes per script

• Daily Accumulators - 8

Full Accumulators - 8

Inversion - 8

Merge - 8

Albedo - 4

MonthlyAlbedo - 4

www.GlobAlbedo.org

www.QA4ECV.eu


Imaging Group Processing System at UCL-MSSL

Local LAN

14-blade Linux cluster : Running RHEL ver. 6.2Two Quad-core Intel Xeon CPU @ 2.67GHz with Hyper Threading enabled (16 logical cores) with 12MB cache48/96GB RAM / 1TB internal disk per blade

RAID storageCurrent dedicated storage capacity:150 Tb – GlobAlbedo100Tb - Planetary Data System mirror

Backup

sever

Automated LT05 tape

backup system7 drive backup system (3 generations)

1-100 Gbps

link to UCL

and outside

world

www.GlobAlbedo.org

www.QA4ECV.eu


GlobAlbedo web-page: source for getting data

N.B. Tile at 1km and Global mosaic at 0.05º and 0.5º available in netCDF4 in scriptable “wget”

www.GlobAlbedo.org

www.QA4ECV.eu


How much netCDF data is there which users can download from

website?

*CEMS data only available by contacting project due to its much larger size.

After 9/14, BRDF & MODIS priors will be available from CEMS after registration

www.GlobAlbedo.org

www.QA4ECV.eu


GlobAlbedo web-page: visually assess the data for browsing

N.B. QuickTime animations of all tiles, global mosaics at 0.05º & 0.5º

www.GlobAlbedo.org

www.QA4ECV.eu


Web Statistics

ESA Living Earth

ESA LandVal

www.GlobAlbedo.org

www.QA4ECV.eu


GlobAlbedo 8-daily 2006 SW BHR (using MSA CLUT)

www.GlobAlbedo.org

www.QA4ECV.eu


GlobAlbedo 8-daily NIR BHR (using MSA CLUT)

www.GlobAlbedo.org

www.QA4ECV.eu


GlobAlbedo 8-daily VIS BHR (using MSA CLUT)

www.GlobAlbedo.org

www.QA4ECV.eu


GlobAlbedo Monthly RGB (SW, NIR, VIS) BHR

www.GlobAlbedo.org

www.QA4ECV.eu


GlobAlbedo 8-daily Coefficient of Variation

www.GlobAlbedo.org

www.QA4ECV.eu


GlobAlbedo 8-daily Weighted Number of Samples

www.GlobAlbedo.org

www.QA4ECV.eu


GlobAlbedo 8-daily Relative Entropy (≥20 MODIS prior high)

www.GlobAlbedo.org

www.QA4ECV.eu


Validation of pre-processing stages from producers

Pixel ID: Cloud, Snow/Ice, Water, Land

VGT vs MERIS

Not shown:

MODIS vs VGT

MODIS vs MERIS

CHRIS vs VGT

CHRIS vs MERIS

BC

SU

FUB

SU

www.GlobAlbedo.org

www.QA4ECV.eu


Prior Art - GlobAlbedo 8-daily BHR for FLUXNET site

Internal subsetting allows any point

or 3 x 3 area to be extracted and

compared with other existing

products including FLUXNET tower

www.GlobAlbedo.org

www.QA4ECV.eu


Prior Art: GlobAlbedo 8-daily BHR: impact of no MERIS prior

to 2002

Internal subsetting allows any point

or 3 x 3 area to be extracted as CSV

and plotted

www.GlobAlbedo.org

www.QA4ECV.eu


European FLUXNET/BSRN test sites (19 FLUXNET, 1 BSRN)

www.GlobAlbedo.org

www.QA4ECV.eu


Europe – Validation (1998-2011), Part 1

www.GlobAlbedo.org

www.QA4ECV.eu


Europe – Validation (1998-2011), Part 2

www.GlobAlbedo.org

www.QA4ECV.eu


BSRN Toravere

N.B. Very noisy tower albedometer data, much higher values from tower cf all other EO values

Only “homogeneous” BSRN site at 1km

www.GlobAlbedo.org

www.QA4ECV.eu


North American test sites (12 Ameriflux, 1 ARM, 7 SURFRAD)

www.GlobAlbedo.org

www.QA4ECV.eu


North American test sites (SURFRAD/ARM)

www.GlobAlbedo.org

www.QA4ECV.eu


GlobAlbedo, MODIS & MISR vs Tower

GlobAlbedo

MISR

MCD43A3

MODIS priors

www.GlobAlbedo.org

www.QA4ECV.eu


GlobAlbedo & MISR uncertainties vs Standard Deviations

of differences cf. Tower Blue-Sky albedo measurements

GlobAlbedo MISR

www.GlobAlbedo.org

www.QA4ECV.eu


Hovmöller Plots of CLARA-SAL vs EO-derived DHR Albedos

CLARA/SAL (AVHRR)

Courtesy of Alexander Loew, MPI

GlobAlbedo MODIS MISR

www.GlobAlbedo.org

www.QA4ECV.eu


Atmospheric contamination of MCD43 MODIS Prior products

http://adam.noveltis.com/

Africa (upper panel) and S. America (lower panel)

ADAM (left) vs GA (right)

Note the whitish hue due to uncorrected aerosol/cloud contamination issues which are highlighted in the GA product

ADAM uses pixel interpolation/extrapolation to explicitly remove aerosol/cloud in monthly composites” Fonds des Sols MODIS “priors”

www.GlobAlbedo.org

www.QA4ECV.eu


Sea-ice albedo – a “moving target”

AATSR+MERIS processing chain

MISR+MODIS processor

www.GlobAlbedo.org

www.QA4ECV.eu


Arctic polar sea-ice maps of albedo (SW-DHR)

NASA MISR+MODIS ESA AATSR+MERIS

www.GlobAlbedo.org

www.QA4ECV.eu


Arctic polar sea-ice maps of albedo (SW-DHR)

NASA MISR+MODIS FMI-VTT-DWD AVHRR CLARA-SAL

www.GlobAlbedo.org

www.QA4ECV.eu


DHR albedo intercomparisons

• Appears to be a bias of

MISR cf AVHRR &

GlobAlbedo

(AATSR+MERIS)

• This is not present for

GlobAlbedo cf AVHRR

• So there appears to be

something wrong with the

MISR spectral-to-

shortwave conversion

• Shunlin Liang looking into

his conversion formulae for

sea-ice as NIR produce

only negative results

MISR vs AVHRR (6/20-25/07) MISR vs AVHRR (6/25-30/07)

GA vs AVHRR (6/20-25/07) MISR vs GA (6/18-25/07)

www.GlobAlbedo.org

www.QA4ECV.eu


NASA CAR vs MISR BRF comparison

Roman et al. (2011)

Comparison of gridded BRFs with MISR BRFs

www.GlobAlbedo.org

www.QA4ECV.eu


Lessons learnt from GlobAlbedo

• Data fusion of MERIS & VEGETATION BBDRs with MODIS priors successful

• GlobAlbedo uncertainties positively correlated with Blue Sky Albedo

magnitudes and with standard deviation of differences

• Production of any global product requires at least 3 sets of processing to iron

out all the bugs. This needs to be included in any project plan/funding schedule

• There is no ideal set of “in situ” blue-sky (BHR) albedo data. It is inherently very

noisy and only covers a small fraction of an EO pixel

• EO-derived results are generally very consistent with each other (with the

exception of METEOSAT) but they are often offset from in situ

• EO albedo results all lower than tower for snow conditions (N.B. issue appears

to be resolved when dealing with daily retrievals MODIS C6)

• MISR is the only EO sensor flying in space which has the potential to map

instantaneous sea-ice albedo, a key Essential Climate Variable

Documents

GlobAlbedo data processing, products and validation - · PDF fileGlobAlbedo data processing, products and validation ... assets to generate Essential Climate ... all lower than tower