NOAA IOOS ProgramData Integration Framework (DIF)Project OverviewAdapted from a brief to the NOAA Data Management CommitteeAugust 6, 2008 by Jeff de La Beaujardire, PhDNOAA IOOS Program, DIF Sr Systems ArchitectOcean Observing Systems Interoperability Planning WorkshopAugust 26 - 28, 2008Julie BoschNOAA National Coastal Data Development Center

Briefing to NOAA Data Management Committee

IOOS SubsystemsObserving SystemsData Management and Communications (DMAC)Modeling and Analysis= Providers(observing system, data archive)= Customers(decision support tool, analysis tool, generic client)= Providers (products)= Metadata, Data Discovery, Access, Transport, Online Browse, Archive

Briefing to NOAA Data Management Committee

NOAA IOOS Program ComponentsStandardsadopt a gap from DMACDevelop and maintain a process to document, coordinate, evaluate and recommend standards to be applied to IOOSRegionsInvest in Regional Association efforts that contribute to the definition and implementation of IOOSData Integration Framework (DIF) ProjectConduct a limited-scope implementation of IOOS as a proof of concept and to gather lessons learned to inform the larger IOOS effort

Briefing to NOAA Data Management Committee

Data Integration Framework (DIF)First spiral of IOOS developmentFocus on 5 core variablesexpanded to 7 variablesScope limited to some DMAC functions access, transport, data discoveryWeb service approach and standardize on small number of services & encodingscatalog, access, and conversion servicesdata and metadata encodingsImplement at selected data provider & customer sitesEvaluate in FY 2010

Briefing to NOAA Data Management Committee

DIF Data Provider and Customer Focus AreasInitially working with 3 NOAA data providersNDBCCO-OPSCoastWatchExpanding to Regional OS data providers

Working with 4 customer groupsHarmful Algal BloomsCoastal InundationHurricane IntensificationIntegrated Ecosystem Assessments

Briefing to NOAA Data Management Committee

Recommended Web Services and Data EncodingsIn-situ data (buoys, fixed station, towed sensors)Gridded data (remotely sensed, model outputs)OGC Sensor Observation Service (SOS)OpenDAP and/orOGC Web Coverage Service (WCS)XML based on OGC Observations and Measurements (O&M)NetCDF using Climate and Forecast (CF) conventionsImages of dataOGC Web Map Service (WMS)GeoTIFF, PNG etc. -possibly with standardized stylesOGC: Open Geospatial ConsortiumXML: Extensible Markup LanguageOPeNDAP: Open-source Project for a Network Data Access ProtocolNetCDF: Network Common Data FormatGeoTIFF: Geospatial Tagged Image File FormatPNG: Portable Network GraphicsData TypeWeb ServiceEncoding

Briefing to NOAA Data Management Committee

XML Encoding of In-Situ Data(SOS GetObservation Result)XMLExtensible Markup Language Generic method for structuring text dataOGC GML Open Geospatial Consortium Geography Markup Language XML that can represent any geospatial featureOGC O&M Observations and Measurements Model GML that describes the act of measuring real-world phenomena and the result of the measurementspecialized byspecialized byData providers currently beta-testing this specification (http://www.csc.noaa.gov/ioos/schema/IOOS-DIF/)

Briefing to NOAA Data Management Committee

DIF Data Provider Status(expected by end of CY2008)Sensor Observation ServiceNDBC DAC - buoys, C-MAN, VOS, NOS NERRS, etc.IOOS DAC - Regional observationsTAO DAC - Tropical Atmosphere OceanDART DAC - Deep-Ocean Assessment and Reporting of TsunamisNational Water Level Observation NetworkPORTS NDBCCO-OPSUserRequestsSensor Observation ServiceHigh-Frequency Radar (HFR) surface currentsWeb Coverage Service& OPeNDAPData Content & Encoding StandardsData Content & Encoding StandardsWeb Coverage Service& OPeNDAPSatellite Ocean Color (Aqua MODIS, Terra MODIS, SeaWiFS)CoastWatchData Content & Encoding StandardsData Integrated from One ProviderData Integrated from One ProviderData Integrated from One ProviderNDBC Data Assembly Centers (DACs)IOOS Variables: temperature, salinity, currents, water level, winds, waves, ocean color (chlorophyll)

Briefing to NOAA Data Management Committee

DIF Data Provider Status - August 2008CO-OPS Sensor Observation Service (SOS)Experimental IOOS DIF Implementation

Briefing to NOAA Data Management Committee

DIF Data Provider Status - August 2008National Data Buoy CenterPROTOTYPE IOOS Sensor Observation Service

Briefing to NOAA Data Management Committee

Additional Providers & ServicesIOOS Variables+ other variables t.b.d.SingleUserRequestRegistry/Catalog ServiceDataIntegrationServiceNDBCSOSWCS/OPeNDAPCO-OPSSOSData Archives (NODC?)SOSCoastWatchWCS/OPeNDAPRegion 1SOSRegion 2WCS/OPeNDAPAgency 2SOSAgency 1WCS/OPeNDAPData Integrated fromMultiple Providers

Briefing to NOAA Data Management Committee

Next Steps for the DIF (FY2009)GEOSS Architecture Implementation PilotInteroperability testingRegistry & catalog serviceMetadata (discovery, QA/QC, SensorML)Client components & general-purpose viewerData translation serviceData integration serviceAdditional variables, providers, customersincl. Regional Coastal Ocean Observing Systemspotential customers in Climate Change and Marine Transportation

Briefing to NOAA Data Management Committee

DIF Service LayersTarget for FY2010DataProvidersNOAARegionalPartnersOtherOtherServicesIntegrationTranslationRegistry/CatalogSensor Observation ServiceWeb Coverage ServiceOpenDAPWeb Map ServiceAccessServices

Briefing to NOAA Data Management Committee

DIF Points of ContactCharlie Alexander - Program Manager (Charles.Alexander@noaa.gov)

Marcia Weaks - Customer POC (Marcia.Weaks@noaa.gov)

Rebecca Shuford - Customer POC (Rebecca.Shuford@noaa.gov)

Jeff de La Beaujardire - Technical POC(Jeff.deLaBeaujardiere@noaa.gov)

Briefing to NOAA Data Management Committee

NOTE: Ambiguity: We put modeling on the right, yetwe also are considering the DMAC aspects oftransporting model output data from source to customer

[ANIMATED SLIDE CLICK TO ADVANCE EACH STEP.]To promote standardized access to data we have recommended particular web services and data encodings. First of all, we broadly classify data into three types: in-situ data, gridded data, and images of data. Each of these uses a different type of service. For in situ data, we recommend the Open Geospatial Consortium (OGC) Sensor Observation Service. SOS lets you ask for data from one or more sensors, and to ask for detailed information about the characteristics of a particular sensor. For gridded data, we recommend the OGC Web Coverage Service. WCS lets you ask for data values in the geographic region of interest to you. Finally, for serving images of data we recommend the OGC Web Map Service. Each of these services can send data in different encoding formats. For in-situ data, we are recommending an XML (Extensible Markup Language) format based on the OGC Observations and Measurements specification. For gridded data, we use the NetCDF format coupled with the Climate & Forecast conventions. Finally, for images formats such as GeoTIFF and PNG are suitable.

GEOSS: Global Earth Observing System of SystemsQA/QC: Quality Assurance/Quality ControlSensorML: Sensor Model Language