3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO

3SAQS Data Warehouse Update

October 10, 2014CIRA - Fort Collins, CO

3SAQS Steering Committee October 10, 2014

2

Outline

• Data Warehouse Team• System Overview• Phase 1 Operational State• Website Overview• Project & Request Workflow• Future Work

3SAQS Steering Committee October 10, 2014

3

Data Warehouse Team

• Shawn McClure• Tom Moore• Dustin Schmidt• Rodger Ames• Cassie Archuleta (ARS)

System Overview

53SAQS Steering Committee October 10, 2014

3SAQS Steering Committee October 10, 2014

6

Data Warehouse – Phase 1

• Project Management System– Project/inventory tracking, modeling platform

request form (Phase 1=NEPA), data manifest• Hardware Upgrades– New 55tb RAID– USB3 transfer to external disks

3SAQS Steering Committee October 10, 2014

7

Data Warehouse – Phase 1 (cont.)

• Modeling Platforms– Base 2008b, Base 2011a CAMx (release pending

approval)• Data deliveries– Hiawatha (ENVIRON)– Converse County (AECOM)

• Website – Pages for studies, projects, documents, procedures

and results, meetings & events, literature, glossary, about (updates ongoing)

Data Warehouse – Phase 1 (cont.)

• Inventory Management– Data is placed within the warehouse– Warehouse file system is represented in database

• Facilitates change control

– File system items are data• Files• Directories

– Categories are groups of File System Items• Files and Directories are grouped into Categories• Meta data is attached to Category• Categories may be hierarchically related• Data may belong in multiple categories

– Data Packages are groups of Categories• Collection of categories for intended use (NEPA)

3SAQS Steering Committee October 10, 2014

9

Website overview

• Studies– Modeling platforms (completed and forward looking)– Monitoring data overview (pending)

• Documents– Modeling platform protocols, MPEs, etc.

• Procedures and Results– Specific how-to pages for DW web based tools (pending)

• About– General information about 3SAQS and the DW (pending)

• Projects– User initiated AQ modeling using 3SAQS platforms

3SAQS Steering Committee October 10, 2014

10

Demo Website

http://views.cira.colostate.edu/TSDW/Demo/

11

Data Warehouse Website – Phase 1

3SAQS Steering Committee October 10, 2014

123SAQS Steering Committee October 10, 2014

3SAQS Steering Committee October 10, 2014

13

Studies

3SAQS foundational modeling platforms• Phase 1 (done)– 2008b available– 2011a (release pending)

• Phase 2 (forward looking)– 2008_2020a– 2011_2020a (CAMx/CMAQ)– Preliminary 2014 (CAMX/CMAQ)– 2014_FutureCase (CAMX/CMAQ)

3SAQS Steering Committee October 10, 2014

14

3SAQS Steering Committee October 10, 2014

15

Documents• Memorialize work that is already done

– Network Assessment• Reference basis for modeling platforms that the DW is delivering

– Modeling protocols– MPE reports– Emissions reconciliations

• Reports and comments– 2008, 2011 Modeling Protocols & response to comments

• Calls for comments on draft documents– Base 2008b MPE– 2011 WRF MPE

• Technical memos– WestJump to 3SAQS – 3SAQS and 2011 NEI O&G Emission Inventories reconciliation– 3SAQS Methane inventory & response to comments

• Modeling Guidance – NEPA/non-NEPA guidance – SMOKE

• Work plans• Annual & Quarterly Reports

3SAQS Steering Committee October 10, 2014

16

3SAQS Steering Committee October 10, 2014

17

Procedures and Results

• “How to” pages for:– Data download and upload procedures– Application of visualization tools– Understanding modeling platforms– Interpreting emissions inventories– Accomplishments and future work of the 3SAQS

and DW

3SAQS Steering Committee October 10, 2014

18

About

• Informational Flyer– Goals of 3SAQS and role of DW– Studies and project management overview– Modeling toolbox highlights

• Geospatial data selection

• 3SAQS and cooperators– BLM– NPS– Forest Service– States of CO, WY, UT

3SAQS Steering Committee October 10, 2014

19

Projects

User initiated AQ modeling projects• NEPA Projects

– Hiawatha (ENVIRON)– Converse County (AECOM)– Others pending

• Non-NEPA (future work)– NPS N deposition– NPS Vertical Profiles of simulated ammonia– EPA QA 2011 Base A– State of CO RAQC O3 SIP– State on NM O3 nonattainment– Others…

Project Workflow - High Level Goals

• Manage Data Access• Provide Project Managers an overview of data delivery

progress• Provide Users a concise view of available data• Provide Data Requestors with a detailed manifest (receipt)• Detailed data accounting (Inventory Management)

– Meta data– Who got what and when– Change control

• Track usage of supplied data• Capture communication threads (Future work)

3SAQS Steering Committee October 10, 2014

21

Project Workflow - Overview

• 3SDW Administrators designate registered users as “Project Managers”– Cooperating Agency point of contact

• Project Manager creates new analysis project– Provides high level meta data– Designates project team members

• Contractors, Subordinates, Etc

• Project team member initiates a Data Request– Data Selection– Provide detailed meta-data

• 3SDW Team fulfills data request• Project team member signs-off request as completed

Project Workflow – User Roles

Project Workflow - User Roles

• Registered User– View projects (Read-only)– View Data Requests (Read-only)

Project Workflow - User Roles

• Project Manager – A Registered user who is also a member of a

3SDW cooperating agency (BLM, EPA, NPS, etc). – Create new analysis projects– Assign project team members– Monitor data transfer progress– Issue data requests

Project Workflow - User Roles

• Project Team Member– Assigned to team by Project Manager– Issue data requests for assigned projects

Project Workflow

Projects Overview

• Purpose– Present an overview of Data Request status– Entry point for Project process– Create a new project

Projects Overview

Projects Overview

Project Configuration

• Purpose– Capture high level information about the project• High level description (Details provided by data

requestors)• Location• Timeframe

– Managing Data Access• Selected pool of Project Managers (3SDW Cooperators)• Project Managers assign 3SDW registered users to

project

Project Configuration

Project Configuration

Data Request

• Allow users to browse available data• Capture optional information about – Intended use of data– Results/Conclusions from data usage

• Supply detailed manifest of requested data

Data Request

Data Request

Data Request

Data Request

Future Work

• Provide finer granularity in data selection• Tune process for supplying supplemental data• Web interface for Data Cataloging & QA• Enhance performance of online data transfer• Capture e-mail communication traffic for

Project / Data Request• Data Submission Process (Return model

results to warehouse)

39

Data Warehouse Website – Phase 2

3SAQS Steering Committee October 10, 2014