Q2O – QARTOD to OGC q2o.whoi

Embed Size (px)


Q2O – QARTOD to OGC http://q2o.whoi.edu Integrating QA/QC tests, flags and standards into Sensor Web Enablement Janet Fredericks Woods Hole Oceanographic Institution Martha’s Vineyard Coastal Observatory August 26-28, 2008 NSSTC/UAH - Huntsville AL. This morning …. - PowerPoint PPT Presentation

Text of Q2O – QARTOD to OGC q2o.whoi

  • Q2O QARTOD to OGC http://q2o.whoi.edu

    Integrating QA/QC tests, flags and standards into Sensor Web Enablement

    Janet FredericksWoods Hole Oceanographic Institution Marthas Vineyard Coastal Observatory

    August 26-28, 2008 NSSTC/UAH - Huntsville AL

  • This morning Introduction to Q2O project: who, what, why, when Present example implementation from MVCO ADCP waves and CDIP data buoy wavesDiscussion of QA/QC implementationAnd perhaps we can add the where!?!

  • OOSTETHYS (MVCO in Fall 2006)

  • Why OGC?

  • Sensor Observation ServiceProvides an API for managing deployed sensors and retrieving sensor dataKey here is access through a serviceThree mandatory core operationsGetObservationDescribe SensorGetCapabilities

  • SensorMLWhat is it? SensorML provides standard models and an XML encoding for describing any process, including the process of measurement by sensors and instructions for deriving higher-level information from observations.

    Processes described in SensorML are discoverable and executable.

    All processes define their inputs, outputs, parameters, and method, as well as provide relevant metadata. SensorML models detectors and sensors as processes that convert real phenomena to data. MBotts

  • QARTOD is a multi-organizational effort to address the Quality Assurance and Quality Control issues of the Integrated Ocean Observing System (IOOS).MotivationExpected increase in the number of observations, observing systems, and usersNeed for simple, accurate and consistent quality content and descriptions for the real-time data to convey expected level of data quality to users Methodology: Engaging the community to develop standards.Courtesy Julie Bosch

  • Activity initiated by NOAA NDBC and CO-OPS in 2003

    Series of WorkshopsWorkshop format with plenary and breakout sessionsApproach to tackling the QA, QC, and metadata issues evolvedFrom separate breakout sessions for QA, QC, and metadata for all observations combinedTo breakout sessions by observation focus group (covering QA, QC, and metadata) Each observation focus group addresses the same guidance questionsParticipationApproximately 80 participants per workshopPrimarily observing system operators and data managersRepresentation from federal agencies, oceanographic institutions, universities, and instrument manufacturers

    Julie Bosch

  • The WorkshopsQARTOD I: December 3-5, 2003 NDBC, Stennis Space Center, MSTask to develop minimum standards for QA/QC methods and metadataConsidering existing observing systems and existing practices (presentations)

    QARTOD II: February 28-March 2, 2005 CO-OPS, Norfolk, VAFocus on wave, in situ current and remote current (HF Radar) measurementsIdentify unique calibration, metadata and QA/QC needs

    QARTOD III: November 2-4, 2005 SIO, La Jolla, CA Continued waves, in situ currents and remote currents workAdded CTD focus groupPrimarily addressed QC with some metadata emphasis

    QARTOD IV: June 21-23, 2006 WHOI, Woods Hole, MAContinued wave, in situ current and CTD workAdded DO focus groupPrimarily addressed QA; increased emphasis on capturing metadata requirementsInternational considerations

    Courtesy Julie Bosch

  • OutcomesWavesResults being incorporated into National Waves Program data management plansResults compiled and submitted to the IOOS DMAC Standards ProcessIn situ currentsResults specific to ADCPs compiled and submitted to the IOOS DMAC Standards ProcessRemote currents HF Radar community continued independent of QARTODDeveloped Data Management Standards for HF Radar Submitted to the IOOS DMAC Standards Process

    New initiative Implementing QA/QC from QARTOD for in situ ocean sensors using OGC Standards/Sensor Web Enablement

    Courtesy Julie Bosch

  • more information


  • NOAA-CSC RCOOS awarded Jan 2008Integrate QARTOD QA/QC recommendations into OGC Sensor Web Enablement:

    develop data dictionaries; set them up in SensorML profiles;

    demonstrate products by integration into oostethys_sos and update or complement cookbooks with QA/QC implementation

    Document and integrate results by providing results to community building organizations, such as MMI, ACT and QARTOD

  • Principal InvestigatorsJanet Fredericks, Woods Hole Oceanographic Institution (WHOI);

    Julie Bosch, NOAA National Coastal Data Development Center/DMAC;

    Michael Botts, author of SensorML, along with his OGC development team from the University of Alabama at Huntsville (UAH);

    Philip Bogden, the Gulf of Maine Ocean Observing System (GoMOOS)/OOSTethys/OGC Oceans IE; and

    Sara Haines, University of North Carolina/ Chapel Hill and SEACOOS

  • http://q2o.whoi.edu Funded 3 yrs

  • Semantic InteroperabilityDomain experts define best practices and required vocabularies working with IT experts to keep interoperability goal in mind and guide in developing relationships for RDF definitions (http://qartod.org)

    Build buy-in within and across communities to promote interdisciplinary potential (E.g., http://marinemetadata.org)

  • For each domain (e.g.., waves/currents)Meet with domain experts to develop vocabularies, definitions and relationships (develop data model/SensorML profiles)Present data model and profiles to broader community (develop tools and guides)Hold workshop to demonstrate implementation and introduce/test products

  • Q2O Schedule2008 complete waves implementation and demonstration

    2009 complete in situ currents implementation begin CTD/DO work

    2010 complete CTD/DO work and (if funded) demonstrate in situ currents and CTD/DO

  • Past Q2O WorkshopsFebruary 2008 Introduction of QARTOD leads for each domain to the Q2O project and the SWE development team to QARTODJune 2008 Met with waves and in situ currents domain experts to discuss SWE and the concepts; discussed data dictionaries, use cases and reviewed QARTOD recommendations

  • Tasks for This WorkshopReview the data model for QARTOD recommended tests and the preliminary demonstration of SensorML profiles for waves (MVCO TRDI-ADCP and CDIP Buoy examples)

  • MVCONDBCManufacturers info & processingNDBC processingSerial Number Model NumberSensor capabilitiesProcessing historyTest parameters specific to sensor

    Processing to meet the requirements of IOOS/NDBC etc

    Testing not available to a local node (nearest neighbor, regional range checks)Or Space time toolkit, or NWS or science user (Matlab) or NCDDC . SWE-SOSQC-flags

  • What do we have (know) to start with?

    A sensor (wave buoy or ADCP) with certain characteristics A sensor history QA info associated with a sensor Deployment characteristics Methods to process the data QC Tests to apply to the data

    What information can we provide to data users or systems (OOSTethys)?

    What sensors we have available as a service

    Description of the sensor Description of where / how / when it is deployed List of the processing methods used on the data List of the QC tests applied The criteria used in the QC tests

    The results of the QC tests The data

    How and with What do we convey that information?

    Get Capabilities lists available data (properties) uses SOS, Observation OfferingDescribe SensorProvides sensore characteristics, deployment characteristics and processing methodsUses SensorMLGet ObservationProvides the data Provides test resultsPoints to file with processing/test infoUses O&M

    Shown as snapshots of information on multiple webPages, docs, etcShow the data model (tests, criteria), dictionary, and where all thepieces and parts are locatedShow info conveyed in the SWE xml filesJulie Bosch

    NOAA National Data Bouy CenterNOAA Center for Operational Oceanographic Products and Services

    Scott Woodruff NOAA Earth System Research Laboratory - JCOMM data mgmtChristoph Waldmann - Research Center Ocean Margins / University of BremenYui Hashimoto JAMSTEC

    Scott Woodruff NOAA Earth System Research Laboratory - JCOMM data mgmtChristoph Waldmann - Research Center Ocean Margins / University of BremenYui Hashimoto JAMSTEC