Q2O – QARTOD to OGC http://q2o.whoi.edu
Integrating QA/QC tests, flags and standards into Sensor Web Enablement
Janet FredericksWoods Hole Oceanographic Institution Martha’s Vineyard Coastal Observatory
August 26-28, 2008
NSSTC/UAH - Huntsville AL
This morning …
• Introduction to Q2O project: who, what, why, when
• Present example implementation from MVCO ADCP waves and CDIP data buoy waves
• Discussion of QA/QC implementation
• And perhaps we can add the where!?!
OOSTETHYS (MVCO in Fall 2006)
Why OGC?
Sensor Observation Service
• “Provides an API for managing deployed sensors and retrieving sensor data”
• Key here is access through a service
• Three mandatory “core” operations
GetObservation
Describe Sensor
GetCapabilities
SensorML
What is it? SensorML provides standard models and an XML
encoding for describing any process, including the process of measurement by sensors
and instructions for deriving higher-level information from observations.
Processes described in SensorML are discoverable and executable.
All processes define their inputs, outputs, parameters, and method, as well as provide
relevant metadata. SensorML models detectors and sensors as processes
that convert real phenomena to data.
MBotts
QARTOD is a multi-organizational effort to address QARTOD is a multi-organizational effort to address
the Quality Assurance and Quality Control issues of the Quality Assurance and Quality Control issues of
the Integrated Ocean Observing System (IOOS).the Integrated Ocean Observing System (IOOS).
MotivationMotivation
• Expected increase in the number of observations, Expected increase in the number of observations, observing systems, and usersobserving systems, and users
• Need for simple, accurate and consistent quality Need for simple, accurate and consistent quality content and descriptions for the real-time data to content and descriptions for the real-time data to convey expected level of data quality to users convey expected level of data quality to users
Methodology: Engaging the community to develop Methodology: Engaging the community to develop standards.standards.
CourtesyCourtesy Julie BoschJulie Bosch
Activity initiated by NOAA NDBC and CO-OPS in 2003Activity initiated by NOAA NDBC and CO-OPS in 2003
Series of Workshops• Workshop format with plenary and breakout sessions• Approach to tackling the QA, QC, and metadata issues
evolved• From separate breakout sessions for QA, QC, and
metadata for all observations combined• To breakout sessions by observation focus group
(covering QA, QC, and metadata) • Each observation focus group addresses the same
guidance questions• Participation
• Approximately 80 participants per workshop• Primarily observing system operators and data managersPrimarily observing system operators and data managers• Representation from federal agencies, oceanographic Representation from federal agencies, oceanographic
institutions, universities, and instrument manufacturersinstitutions, universities, and instrument manufacturers
Julie BoschJulie Bosch
QARTOD I: December 3-5, 2003 NDBC, Stennis Space Center, MSQARTOD I: December 3-5, 2003 NDBC, Stennis Space Center, MS• Task to develop minimum standards for QA/QC methods and metadataTask to develop minimum standards for QA/QC methods and metadata• Considering existing observing systems and existing practices Considering existing observing systems and existing practices
(presentations)(presentations)
QARTOD II: February 28-March 2, 2005 CO-OPS, Norfolk, VAQARTOD II: February 28-March 2, 2005 CO-OPS, Norfolk, VA• Focus on wave, in situ currentFocus on wave, in situ current and remote current (HF Radar) and remote current (HF Radar)
measurementsmeasurements• Identify unique calibration, metadata and QA/QC needsIdentify unique calibration, metadata and QA/QC needs
QARTOD III: November 2-4, 2005 SIO, La Jolla, CA QARTOD III: November 2-4, 2005 SIO, La Jolla, CA • Continued waves, in situ currentsContinued waves, in situ currents and remote currents workand remote currents work• Added CTD focus groupAdded CTD focus group• Primarily addressed QC with some metadata emphasisPrimarily addressed QC with some metadata emphasis
QARTOD IV: June 21-23, 2006 WHOI, Woods Hole, MAQARTOD IV: June 21-23, 2006 WHOI, Woods Hole, MA• Continued wave, in situ current and CTD workContinued wave, in situ current and CTD work• Added DO focus groupAdded DO focus group• Primarily addressed QA; increased emphasis on capturing metadata Primarily addressed QA; increased emphasis on capturing metadata
requirementsrequirements• International considerationsInternational considerations
Courtesy Julie BoschCourtesy Julie Bosch
The WorkshopsThe Workshops
WavesWaves• Results being incorporated into National Waves Program Results being incorporated into National Waves Program
data management plansdata management plans• Results compiled and submitted to the IOOS DMAC Results compiled and submitted to the IOOS DMAC
Standards ProcessStandards ProcessIn situ currentsIn situ currents
• Results specific to ADCPs compiled and submitted to the Results specific to ADCPs compiled and submitted to the IOOS DMAC Standards ProcessIOOS DMAC Standards Process
Remote currents Remote currents • HF Radar community continued independent of QARTODHF Radar community continued independent of QARTOD• Developed Developed Data Management Standards for HF RadarData Management Standards for HF Radar • Submitted to the IOOS DMAC Standards ProcessSubmitted to the IOOS DMAC Standards Process
New initiative New initiative • Implementing QA/QC from QARTOD for in situ ocean sensors Implementing QA/QC from QARTOD for in situ ocean sensors
using OGC Standards/Sensor Web Enablementusing OGC Standards/Sensor Web Enablement
Courtesy Julie Courtesy Julie BoschBosch
OutcomesOutcomes
NOAA-CSC RCOOS awarded Jan 2008
Integrate QARTOD QA/QC recommendations into OGC Sensor Web Enablement:
• develop data dictionaries; set them up in SensorML profiles;
• demonstrate products by integration into oostethys_sos and update or complement cookbooks with QA/QC implementation
• Document and integrate results by providing results to community building organizations, such as MMI, ACT and QARTOD
Principal InvestigatorsJanet Fredericks, Woods Hole Oceanographic Institution (WHOI);
Julie Bosch, NOAA National Coastal Data Development Center/DMAC;
Michael Botts, author of SensorML, along with his OGC development team from the University of Alabama at Huntsville (UAH);
Philip Bogden, the Gulf of Maine Ocean Observing System (GoMOOS)/OOSTethys/OGC Oceans IE; and
Sara Haines, University of North Carolina/ Chapel Hill and SEACOOS
Semantic Interoperability
• Domain experts define best practices and required vocabularies – working with IT experts to keep interoperability goal in mind and guide in developing relationships for RDF definitions (http://qartod.org)
• Build buy-in within and across communities to promote interdisciplinary potential (E.g., http://marinemetadata.org)
For each domain (e.g.., waves/currents)
• Meet with domain experts to develop vocabularies, definitions and relationships
(develop data model/SensorML profiles)
• Present data model and profiles to broader community
(develop tools and guides)
• Hold workshop to demonstrate implementation and introduce/test products
Q2O Schedule
• 2008 – complete waves implementation and demonstration
• 2009 – complete in situ currents implementation begin CTD/DO work
• 2010 – complete CTD/DO work and (if funded) demonstrate in situ currents and CTD/DO
Past Q2O Workshops
• February 2008 – Introduction of QARTOD leads for each domain to the Q2O project and the SWE development team to QARTOD
• June 2008 – Met with waves and in situ currents domain experts to discuss SWE and the concepts; discussed data dictionaries, use cases and reviewed QARTOD recommendations
Tasks for This Workshop
• Review the data model for QARTOD recommended tests and the preliminary demonstration of SensorML profiles for waves (MVCO TRDI-ADCP and CDIP Buoy examples)
MVCO NDBC
Manufacturer’s info & processing
NDBC processing
Serial NumberModel Number
Sensor capabilities
Processing history
Test parameters specific to sensor
Data Provider info & processing
Set up and Deployment descriptions
Processing history
Test parameters specific to deployment
Processing to meet the requirements of IOOS/NDBC etc
Testing not available to a local node (nearest neighbor, regional range checks)
Or Space time toolkit, or NWS or science user (Matlab) or NCDDC ….
SWE-SOS
SWE-SOS?
QC-flags QC-flags
SWE-SOS
QC-flags
What do we have (know) to start with?
- A sensor (wave buoy or ADCP) with certain characteristics
- A sensor history- QA info associated with a
sensor- Deployment
characteristics- Methods to process the
data- QC Tests to apply to the
data- - …
What information can we provide to data users or systems (OOSTethys)?
- What sensors we have available as a service
- Description of the sensor- Description of where /
how / when it is deployed- List of the processing
methods used on the data- List of the QC tests applied- The criteria used in the QC
tests
- The results of the QC tests- The data- - …
How and with What do we convey that information?
Get Capabilities- lists available data (properties)- uses SOS, Observation Offering
- Describe Sensor-Provides sensore characteristics, deployment characteristics and processing methods-Uses SensorML
- Get Observation-Provides the data -Provides test results-Points to file with processing/test info-Uses O&M
Shown as snapshots of information on multiple web
Pages, docs, etc
Show the data model (tests, criteria), dictionary, and where all the
pieces and parts are locatedShow info conveyed in the SWE xml files
Julie Bosch
Recommended
