C o p e r n i c u s C l i m a t e

C h a n g e S e r v i c e

G l o b a l L a n d a n d

M a r i n e O b s e r v a t i o n s

Climate Change

M a r i n e O b s e r v a t i o n s

D a t a b a s e

S i m o n N o o n e

Climate

Change • Provision of global land and marine surface meteorological holdings

• Multiple meteorological parameters

• Across timescales (sub-daily to monthly)

S e r v i c e r e m i t

• Data IPR clearly articulated and flagged

• Provided under a common data model

Climate

Change Complement rather than replace national / regional holdings, through

provision collated in one place and one format, to realise the true global

value of the historically collected meteorological data

• Can more easily quantify indicies and indicators which require multiple

ECVs or undertake analyses that require use of multiple variables

W h y d o t h i s ?

ECVs or undertake analyses that require use of multiple variables

• Basis for Climate Service development and delivery via Climate Data Store

• Sustainable multi-point support for data stewardship of the fundamental

climate data record (a European contribution to a global endeavor)

Climate

Change • Historical efforts have been highly fractured. In almost all cases one or more of the following apply:

– Timescale specific

– ECV specific

– Regionally or nationally specific

• Upshot is that there are grossly duplicative sets of holdings managed by multiple international parties using a range of data formats and

L a n d m e t e o r o l o g i c a l h o l d i n g s

by multiple international parties using a range of data formats and metadata protocols

• Challenge is to create from these disparate sources a set of unified holdings that is coherent and consistent across both ECVs and timescales

http://journals.ametsoc.org/doi/abs/10.1175/BAMS-D-16-0165.1

Climate

Change • ICOADS is the recognised repository for international surface marine observations

– Benefits from >30 years of heritage

– Data from ships, buoys, fixed platforms etc.

– Numerous sources

• Duplication elimination is imperfect

– Issues over Quality Control

M a r i n e d a t a h o l d i n g s

– Issues over Quality Control

• Trimming limits

• Positional quality control

– Prone to single point of failure due to over-reliance upon single institution and funding stream

http://journals.ametsoc.org/doi/abs/10.1175/BAMS-D-15-00251.1

Climate

Change • Maynooth University

• UK Met Office

• National Oceanography Centre

T h e t e a m

• Science and Technology Facilities Council

• NOAA’s National Centers for Environmental Information

Inventory

Climate Change

Inventory

Climate

Change • Need firstly to ascertain what data exists

• For ICOADS we have archived the sources and individual data decks

that underlie

• For land we are sourcing from a range of international, regional and

nationally available sources

– Underlying data sources at NOAA NCEI have been shared

W o r k o u t w h a t d a t a t h e r e i s

– Underlying data sources at NOAA NCEI have been shared

– Several national repositories have been downloaded

– Several institutional repositories under consideration

– ECMWF MARS archive (basis for reanalysis)

• Only using publically facing sources.

Climate

Change

1 7 1 , 2 1 4 a p p a r e n t s u b - d a i l y r e c o r d s

Climate

Change

1 6 2 , 8 9 2 d a i l y ( m a n y d u p l i c a t e s )

Climate

Change

8 4 , 6 6 8 m o n t h l y ( m a n y d u p l i c a t e s )

Climate

Change • Detailed inventory information will go up on service website shortly

(service website will be part of broader climate.copernicus.eu

domain)

• We shall be building the data holdings throughout the contract

W e m a y n o t h a v e a l l s o u r c e s

• If you have data or know of data you think we should consider send

us an email …

incoming_data@surfacetemperatures.org

Climate

Change

• Discussions ongoing within C3S and with WMO on whether we can

use OSCAR surface ids

• Nuanced issue (not time to cover in detail here, suffice to say its

complicated)

W e h a v e m o r e t h a n c u r r e n t l y i n O S C A R

s u r f a c e

complicated)

• Need to enable discoverability via WMO channels if the data are to be

useful and used globally.

Climate

Change

• For land data the IPR issues are more involved

– Some countries open policies

– Some countries Resolution 40

– Some countries restricted

D a t a p o l i c y l a n d s c a p e

– Some countries restricted

– Some countries under bilateral agreements

– Many countries have mixed data policies across their networks

– Many observations taken by non-National Met Services (not even

clear what policy is)

Climate

Change • We shall document the IPR per source and associate with the source

– Source trace

– IPR trace

– Usage conditions

– Acknowledgements conditions

– Sufficient for service to respond with defensible trace should data service

D o c u m e n t i n g I P R p e r s o u r c e

– Sufficient for service to respond with defensible trace should data service

ever be queried

• IPR flags shall be used to ensure that only data with use condition

commensurate with application are served via the CDS

• In the harmonisation priority will be given to sources with more open

data policies …

Climate

Change

E x a m p l e I P R t a b l e

H a r m o n i s a t i o n

Climate Change

H a r m o n i s a t i o n

Climate

Change • Under continual development with three remaining lots under the

same ITT

• Should contain all relevant data and metadata and be flexible and

extendable

• Table driven EAV model being developed

• Consistent with ODB, BUFR, WMDS and ISO 19115 to extent possible

C o m m o n d a t a m o d e l

• Consistent with ODB, BUFR, WMDS and ISO 19115 to extent possible

• We shall convert sources into a format that is compliant with the

CDM and work with these converted sources

Climate

Change • Sequential merge of data sources

– Location metadata similarity

– Data similarity

– Backfill ECVs and periods

– Cannot be completely automated

• Start from daily data

L a n d h a r m o n i s a t i o n

• Start from daily data

– build monthly averages from dailies by preference (assures consistency)

– Ensure sub-daily consistent

• Use of a consistent identifier approach

Climate

Change

E x a m p l e o f m e r g i n g s o u r c e s

Start-Black line only

a decade of records

Merged with red line

1940-2010

Back-filled with blue

line…

Produced station

records 1883-2010

Climate

Change• Apply quality control

checks (flag don’t

delete)

• Assess homogeneity of

records to extent

possible

L a n d d a t a h a r m o n i s a t i o n

D a t a d e l i v e r y

Climate Change

D a t a d e l i v e r y

Climate

Change

D i s s e m i n a t i o n

Data shall be served via the C3S CDS

to users

User support shall be available

Three supporting one-stop shop

documents:documents:

• Technical service guide

• Land user guide

• Marine user guide

Climate

Change • Datasets and products come and go

– New techniques

– New computational capabilities

– New insights

All mean that its all but guaranteed that todays state-of-the-art products

shall be deprecated 30-50 years hence.

F i n a l l y : t h e p o i n t o f a l l t h i s ?

shall be deprecated 30-50 years hence.

• Whereas the fundamental record is forever

– If we muck up stewardship of the fundamental record, it and the

associated knowledge disappears

– Keep a deep audit trail, flag but don’t remove, do no harm!

– Resource for today, tomorrow and many tomorrows to come.