Data merging benefits Antoine Mangin, Stéphane Maritorena

Data merging benefitsGlobcolour / Medspiration user consultation, Nov. 20-22, 2007, Oslo

Session 4 –GlobColour applications – November 21, 20071

Data merging benefits

Antoine Mangin, Stéphane Maritorena


for the users



Data merging benefits ?

• Improvement of spatial/temporal coverage

• Error bar estimates

• Trends analysis

Background

For the benefit of every user

For assimilation into models (but also to understand reliability of products)

For the benefit of reliable environmental reporting and carbon cycle studies



Data merging



• Trends analysis

It is a truism.

Outlines



Improvement of spatial/temporal coverage






• Trends analysis

It requests a careful analysis of error estimates as inputs of GSM

Outlines



Error bar estimates

L wN() source 1 LwN() source 2 L wN() source n

LwN () uncertainties source 1 LwN () uncertainties source 2 LwN () uncertainties source n

[L wN() source 1, L wN() source 2, ..., LwN() source n]

LwN () uncertainties source 1, LwN () uncertainties source 2, ..., LwN () uncertainties source n[ ]

GSM01 MODEL INVERSION

wNL bb

bb= f(

a + )

wNL

SeaWiFSMODIS

[Chl] (443)acdm (443)bbp

DATA SOURCES

CONCATENATED ARRAYS

[Chl] confidence interval (443) confidence intervala cdm (443) confidence intervalb bp

RETRIEVALS

Estimates of the uncertainties on input LwN

Estimates of the error model

Estimates of the uncertainties on outputs Chla, bbp,cdm

(Co-variance matrix between all ingredients)



Error bar estimates

Estimates of the uncertainties on input LwN

This is a direct result from the characterisation at sensors level (GC phase 1 ++)



Error bar estimates

Estimates of the full uncertainties as input of GSM



Error bar estimates

Discussion

Interest: Relative importance (weight) of each wavelength in the inversion is a key element

Assumption: Main assumption is that there is no bias – input error is considered as a pure deviation defined by its standard deviation

Future: When we will reach the million match up points (or maybe before) – we should go to error estimates by classes of LwN



Error bar estimates - Validation






wNL bb

bb= f(

a + )

wNL

SeaWiFSMODIS


DATA SOURCES

CONCATENATED ARRAYS


RETRIEVALS

Estimates of the uncertainties on outputs Chla, bbp,cdm

(Co-variance matrix between all ingredients)

Use of the estimates

Nomad DB / EO data Extract samples of concomitent

LwN, Chla bbp and cdom

samples

If the Chla error estimates is reliable

should be close to a standard normal distribution



Error bar estimates - Validations

Inputs: In situ observations (Nomad)Results: very close to expectancy – no significant bias

0

10

20

30

40

50

-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6

Normalised std dev

Po

pu

lati

on

Results

Expected

Inputs: GC productsResults: very close to expectancy – a small bias is detected – the error estimates by GSM (with ad hoc inputs) is slightly underestimated.



Example of products uncertainties - daily

GlobColour Chla-merged product – May,15 2006

GlobColour Chla-merged product relative uncertainties – May,15 2006

100

50

0



GlobColour Chla-merged product – May 2006

Example of products uncertainties

GlobColour Chla-merged product relative uncertainties– May 2006

100

50

0






• Trends analysisDifferences between individual sensor time series for each sensors may (will?) lead to disturbances in merged time series.One aspect which is however not yet fully exploited is the correlation between individual sensor products which is rather good.

Outlines



Background

Context for this trend analysis: EEA reporting

In the frame of GSE Marcoast, reliability of EO to help environmental reporting is explored as well as consistency between missions to ensure continuity of the reporting.

Today the reporting is based on in situ observation and the metric for trend identification is, for a given area, the number of stations that have showed a significant increase/decrease of observed Chla (*) during the last 10 years.

(*) Observed Chla is an average seasonal value built on a very strict protocol.



Background – ingredients for reporting

14 eco-regions

About 6800 Chla samples in 2003-2005

… and thus the report

Within Marcoast we are working to replace/complement in situ sampling by EO (and here more specifically by

GlobColour)



Method used

Setting up of a non parametric test for detection of trends at GC pixel level.

The test is based on summation of sign of difference between one status and the previous ones (eg. season 2005 compared to 2004, 2003 etc..)

Statistical variance 2 of a white noise on such times series is analytically known.

So … any departure above (resp. below) 2 (resp. –2) from this law would indicate that a trend exist with a 95% significance level

Trends analysis

Important distinction: We are not trying here to quantify trends but to identify the probable ones.



2.5% 2.5%

Standard normal distribution

White noise at a level of significance of 95%

Trends detection



Discrepancies and Consistencies between instruments

MERISalone

MODISalone

2003-2006

2.5% 2.5%



MODIS SeaWiFS

MERIS GlobColour

Patchiness of MERISresults is probably

due to coverage

Possible trends are very consistent from one single sensor to the other

2003-2006



2.5% 2.5%

Trends detection – weighted average – the full game

Spatial consistency of possible trends are evidences of trends



Trends detection – GSM – the full game



Correlation coefficient for the seasonal figures

.6/.8/.6

.2/.5/.2

.8/.9/.9

.3/.6/.4

.8/.9/.8

.9/.9/.9.7/.8/.7

.8/.9/.8

.8/.8/.8.9/.9/.8

MERSWF/SWFMOD/MERMOD

.9/.9/.9

This gives a reasonible confidence level (or caution level!) in the merging of all sensors in order to identifiy trends



0% 60% 80%40% 100%20%

2003-2005 trends GC - 1998-2006 trends

Final reporting for EEA






• Improvement of spatial/temporal coverageYes, by construction

• Error bar estimatesA reliable error estimates has been derived through GSM – about to be submitted for publication

• Trends analysisAlthough GC has not yet the right quality for Climate change studies, it already provides means to detect evidence of trends for environmental reporting – under iteration with EEA within GSE-Marcoast

Outcomes/conclusions



A special thanks to

Christophe Lerebourg and Julien Demaria for data handling, impossible, hair-splitting and after-hours computations.

Acknowledgments

…. and thank you for your attention



0

10

20

30

40

50

60

70

80

90

100

-6 -4 -2 0 2 4 6 8

Results

Expected

2-sigma

2 - « sigma »

Error bar estimates – Validation – inputs : GC products



Impact of taking into account input Lwn uncertainties on:1. Retrieved Lwn (from GSM forward)

2. Retrieved Chla, CDM, Bbp GSM forward)

0.000

0.020

0.040

0.060

0.080

0.100

0.120

0.140

400 420 440 460 480 500 520 540 560 580

Wavelength (nm)

RM

S (

ab

s)

Noconsideration of error on inputLwn

Error model from characterisation isapplied on input Lwn

N RMS BIAS RATIO

Mean (%)

DIFF

Median(%)

DIFFMIN In

situMAX In

situmean In situ

median In situ

MIN Model

MAX model

mean model

median model r2 INTCPT SLOPE

CHL 259 0.21 -0.03 1.03 30.91 20.03 0.069 26.91 2.12 0.45 0.017 27.61 1.87 0.40 0.90 -0.03 1.01CHL (no error) 252 0.23 -0.09 0.91 30.92 22.62 0.069 26.91 1.87 0.41 0.019 28.48 1.58 0.29 0.89 -0.08 1.02

CDM 259 0.34 -0.05 1.14 49.29 26.93 0.008 1.97 0.15 0.04 0.001 1.38 0.13 0.04 0.74 0.29 1.26CDM (no error) 252 0.34 -0.04 1.18 51.15 27.98 0.008 1.99 0.13 0.03 0.001 1.38 0.12 0.04 0.72 0.34 1.28

BBP 26 0.103 -0.026 0.964 14.13 8.85 0.0011 0.0054 0.0028 0.0026 0.0012 0.0045 0.0025 0.0024 0.729 -0.994 0.627BBP (no error) 26 0.140 0.082 1.245 32.35 31.16 0.0008 0.0051 0.0022 0.0021 0.0012 0.0045 0.0025 0.0024 0.739 -1.278 0.497



Error bar estimates

Estimates of the model uncertainties






wNL bb

bb= f(

a + )

wNL

SeaWiFSMODIS


DATA SOURCES

CONCATENATED ARRAYS


RETRIEVALS

Nomad DBExtract samples of

LwN

Use of direct bio optical model to derive new LwN

Estimates of the error model



IC ES Eco-region

0.0

0.2

0.4

0.6

0.8

1.0

Co

rrel

atio

n c

oef

fici

ent

A B C DE F G H I J K L M N

A: Greenland and Iceland SeasB: Barents SeaC: FaroesD: Norwegian SeaE: Celtic SeaF: North SeaG: South European Atlantic ShelfH: Western Mediterranean SeaI: Adriatic-Ioanan SeasJ: Aegean-Levantine SeasK: Oceanic Northeast AtlanticL: Baltic SeaM: Black SeaN: Azov Sea

MERSWF SWFMOD MERMOD

Correlation coefficient for the seasonal figures

Documents

Data merging benefits Antoine Mangin, Stéphane Maritorena