Marine Core Service MY OCEAN Scientific Assessment Review – QuARG chair F. Hernandez Review of the Product Quality and Validation phase of V1 Stream1 Production

MarineCore

Service

MY OCEAN

Scientific Assessment Review – QuARG chair F. Hernandez

Review of the Product Qualityand Validation phase of

V1 Stream1 Production Centres

• Context• Quality Assurance & Review Group review of

the ScVR• Validation outcomes, remarks

Marine Core Service

MyOcean : PRR Stream1 V1 – 29th April 2011 / Roma – QuARG review

Ded

icat

ed r

esso

urc

es

pla

nn

ed f

or

Cal

/Val

X.5

Coordination of x.5 activity inside Production Units

Provide diagnostics for scientific assessment in coordination with QuARG reviews

Marine Core Service


Context of the validation phase

Every PC has dedicated ressources to perform scientific assessment activities for:– Monitoring the V1 system performance during operations

– Assessing the quality of V1 Product disseminated to users

The V1 Stream1 subsystem cal/val tools succeeded the VARR: the calibration phase proved already that:– Validation activity is in place in PC/PU

– V1 Stream1 system offer an upgraded quality compared to V0

In parallel, following the Stream2 V1 system agenda:– V1 S2 calibration phase has been performed

– Based on similar system to V1, Reanalysis and Reprocessed Analysis have been performed and scientifically validated (strengthening validation synthesis for the operational system)

– This has been reviewed with success at the PC Acceptance Stream2 V1.

Marine Core Service


Objective of the validation phase

Verify that scientific validation procedures are running operationnaly at PC/PU level to asses the product quality

Verify that these procedures are used, and the product quality is as expected

Ask for a Scientific Validation Report, based on the routine validation performed since the 15th of December 2010

Marine Core Service


Cal/Val activities for the second MyO year

Prepare ScCP, ScVP…

….. V1S1 devlpt.

PC Acceptancereview of ScCP, ScVP,

ScCR

June 2010 End September 2010

VAlid. Read. Rev.Start V1 S1 operation

15/12/2010

Test op. validation

V1S1 scientific assessment ….. V1S1 pre-operated ….. V1S1 in operation

Operate routinely PQ validation: reg. reporting

29/4/2011

P. Read. Rev.Review V1 S1 ScVR

Calibration

….. V1S2 devlpt. / REA V1S2 & REA scientific assessment

PC Acceptancereview of ScCP, ScVP, ScCR, ScVR-Rea

March 2011 June 2011 Sept. 2011

VARRStart V1S2 operation

PRRV1S2 ScVR

….. V1S2 in operation

Validation

Validation requirements:– Compliance of routine validation activity with ScVP– Identify and report about success/defaults of V1– Pay attention to V1 performance versus user requirements

(if any)– Consider dependencies (ie, quality/impact of input data for

the period)

Marine Core Service


Status of the Calibration and Validation

PC Acceptance Stream1 V1, Roma, September 2010

VARR Stream1 V1, Brussels, December 2010

PC Acceptance Stream2 V1, Copenhagen, March 2011

User Forum, Stockholm, April 2011

PRR Stream1 V1, Roma, April 2011

VARR Stream2 V1, June 2011

PRR Stream2 V1, September 2011

Prod. Centres

ScCP/ScVP ScCR ScVR-Rea ScVR (PRR)

GLO x Stream 1 5 docs 1 doc + 2 docs

ARC x Stream 1 1 doc 1 doc

BAL x Stream 1

NWS x Stream 2 2 docs

IBI x Stream 2

MED x Stream 1 1 doc

BLS x Stream 2 1 doc

SL x Stream 1 1 doc

OC x Stream 1/2 1 doc 1 doc

SST No (in prep) Stream 1 in 2 OSTIA doc 1 doc

SIW x Stream 1 7 docs for sea ice no Wind

INS x NO (in prep) 1 doc

Marine Core Service


Flavour of V1 Product Quality assessment, from reporting

Every Production Centres fulfulling the scientific validation, some delays in delivery, but all project management procedures not already efficient:

– GLO MFC: 3 PU, detailed and valuable validation

– ARC MFC: detailed and valuable validation

– MED MFC: detailed and valuable validation

– SL TAC: detailed and valuable validation

– OC TAC: detailed and valuable validation

– SST TAC: ScVP expected. Detailed and valuable validation

– SIW TAC: 7 doc. for sea ice. No doc for wind –> No delivery justified by Quickscat failure in 11/2009. Assessment for other products scheduled in V2

– INS TAC: no ScCR. But RTQC performed regularly. Particular attention also paid to reprocessed dataset

Marine Core Service


Flavour of the V1 Product Quality assessment: SST TAC

SST13

Marine Core Service


Expected outcome from calibration:accuracy levels per product (ex. SST TAC summary)

Comparison to match-up database

Marine Core Service


ScCR results: MEDProducts example

Location of the MED in situ-satellite matchups

Sensor N data r2 RMS bias RPD [%]

MODIS 234 0.74 0.74 0.01 36

MERIS 91 0.46 0.72 0.02 33

OC12

Marine Core Service


Flavour of the V1 Product Quality assessment: OC TAC OC12

Marine Core Service


Flavour of Product Quality assessment, from reporting: proposing new approach: OC TAC and MED MFC validating Chl-a

From MED ScVR

MED9

Marine Core Service


Flavour of the V1 Product Quality assessment: ARC MFC

ARC5

Marine Core Service


Flavour of the V1 Product Quality assessment: GLO MFC

We can see in Figure 1 a very good agreement between both Global and HR-Zoom systems and SLA observations. Nevertheless strong discrepancies are visible in the English Channel, the North Sea and the Baltic Sea. This illustrates one of the current limitations of the system which produces a spicy bias in the North Sea and the Baltic. Spicy biases are also observed in the Celtic Sea

We can see in Figure 1 a very good agreement between both Global and HR-Zoom systems and SLA observations. Nevertheless strong discrepancies are visible in the English Channel, the North Sea and the Baltic Sea. This illustrates one of the current limitations of the system which produces a spicy bias in the North Sea and the Baltic. Spicy biases are also observed in the Celtic Sea GLO

4

Marine Core Service


Flavour of the V1 Product Quality assessment: GLO MFC

GLO4

Marine Core Service


Flavour of the V1 Product Quality assessment: GLO MFC Backup

GLO4

Marine Core Service


Flavour of V1 Product Quality assessment, GLO PHYS PU

WOA09

Synth

Comb

GLO4

Marine Core Service


Product Quality Assessment: V1 Stream1 Scientific Validation Review

Reporting has been performed, reporting mechanism had some minor deficiencies

The scientific validation procedures, and the corresponding reporting is:– Scientifically valuable– Reliable, and proved to be performed anytime (secured by many

types of diagnostics)– Fair: good and bad aspects are addressed and reported– Consistent with plans and project management– Offering a continuity with system and products assessment (refer

to calibration phase, to V0, compliant with MERSEA heritage)– Clear, providing useful synthesis for users

The scientific validation of V1 Stream1 is effective, and corresponding to QuARG expectation

Marine Core Service


Remarks, considerations, and ongoing work

This huge effort of validation has never been done elsewhere in the ocean community

Scientific validation of reanalysis products is promising: a strong effort is done to provide common validation strategies (Global reanalysis) that might be positively considered by the international community

The validation reporting is not fully organised for V1 and V2, but interest to maintain regular reporting is obvious, and will prepare MYO-2 operational validation

Marine Core Service


Link with users

Reporting is established, and could support regular information to users

Request from the User Forum in Stockholm, due to effort in reporting, actions can be commited for:

– Estimated accuracy level (september 2011)

– QUID (october 2011)

Documents

Marine Core Service MY OCEAN Scientific Assessment Review – QuARG chair F. Hernandez Review of the Product Quality and Validation phase of V1 Stream1 Production