1 2008 AMS Summer Community Meeting Boulder, Colorado – 11-13 August 2008 The Emerging Interface among the Private, Federal & Academic Sectors in the Archive

12008 AMS Summer Community Meeting

Boulder, Colorado – 11-13 August 2008

The Emerging Interface among the Private, Federal & Academic Sectors in the Archive

of Climate Data & Information

A NOAA Climate Data Perspective August 12, 2008

(AMS Commission on Weather/Climate Enterprise)

Tom Karl, Director

NOAA’s National Climatic Data Center



Outline• Overview: NOAA Archive Community Interfaces

– Examples of input to NOAA archive activities • National Research Council Panel on NOAA Archive & Access

• NOAA Science Advisory Board’s Data Archive and Access Requirements Working Group (DAARWG)

• NOAA data user workshops

• A Possible Model for Partnerships Among the Community of Users and Providers– The National Integrated Drought Information System (NIDIS)

• Drought Portal

• A NOAA Archive and Access Business Model– A potential way to help define sector roles



Community Interface for input into NOAA Archive and Access Activities – (highlighted in red)

Under Secretary for Oceans and Atmosphere

Reps. fromNOAA Goals

Reps. fromNOAA Line Offices

NOSCNOAA Observing System Council

NOAA’s Data Management CommitteeThe NOSC agent for coordinating the development and

implementation of data management policy across NOAA

NOAA SAB

Examples of input into NOAA Archive & Access Activities

NOAA SAB Data Archive & Access Requirements Working Group (DAARWG)(includes Academia, Private Sector, and other non-NOAA members)

National Academy of Sciences NRC: Committee on Archiving & Accessing Environmental & Geospatial Data at NOAA

NOAA Data Users’ Workshops and User Surveys



National Research Council Panel on Archiving & Accessing Environmental & Geospatial Data at NOAA

• NRC Panel requested by NOAA’s Data Management Committee (Nov 2005)

– Panel Goals: Develop principles & guidelines • What types of data should be permanently archived

• What types of data could be stored for shorter durations under budgetary constraints

• How best to provide access to different variables, data sets and derived products

– November 2007: Final NRC report completed• 9 main principles and guidelines• June 2008 NOAA workshop to address report

NRC Principles & Guidelines Reports to be used by NOAA Science Advisory Board’s Data Archiving & Access Requirements Working Group (DAARWG)

to evaluate NOAA’s process for prioritization of specific NOAA data sets

NRC report (116 pages)Released Nov 2007



NOAA Science Advisory Board’s (SAB) Data Archive and Access Requirements Working Group

NOAA Procedure for Scientific Records Appraisal and Archive Approval

National Oceanic and Atmospheric Administration

May 30, 2008

Outline

1. Purpose 2. Authority 3. Scope 4. Definitions 5. Procedure:

a. Identify Scientific Records for Appraisal b. Appraise Scientific Records to Determine Archival Value c. Decision/Approval Process for Accessioning and Disposing of Scientific

Records d. Implementation of the Decision

6. Assignment of NOAA Roles and Responsibilities 7. Procedure review 8. References

Appendix I NOAA Scientific Records Covered and not Covered by this Procedure

Appendix II NOAA Scientific Records Appraisal Criteria Questionnaire Appendix III Intent of NOAA Procedure for Scientific Records Appraisal

and Archive Approval

NOAA Response to Dec. 2007 recommendation made by the SABComprehensive procedure developed to determine what scientific records

(new and existing) should be preserved in a NOAA archive

Vetting Process• NOAA reviews:

– April 3-21 & May 1- 23• All NOAA Line Offices (NESDIS, NWS,

OAR, NMFS, NOS)

• External NOAA Reviews:– Apr 23-May 16 & May 19-28

• USGS -EROS • NCAR/UCAR• ECMWF• Academic sector (limited)

To be advertised in Fall 2008 for public comment on the NOAA Policy on Partnerships in the Provision of Environmental Information website: http://www.nesdis.noaa.gov/PSPS/psps_partnership.html



User interaction & feedback are critical to NOAA’s success in terms of feedback on NOAA archive and access activities

2003 WorkshopBoulder, CO350 in attendanceKeynote: VADM Lautenbacher188 recommendations

2005 WorkshopAsheville, NC250 in attendanceKeynote: Jack Kelly102 recommendations

2007 WorkshopAsheville, NC215 in attendanceFocus on 3 business sectors – energy, transportation,

insurance

Data Users’ Workshops 2003 Satisfaction Survey(Scale 1-5; 5 extremely satisfied)

NOAA Offices: NCDC, NODC, NGDC, SAA Population Size: 25K Response Rate: 6.3K (25%)

2008 Survey Update in progressSurvey link: https://ols.nndc.noaa.gov/survey_2008.pdf

NOAA Data User Workshops & Satisfaction SurveysExamples of sector input to NOAA activities



A Potential Model for Partnerships Among the Community of Users and Providers

NIDIS



A National Integrated Drought Information System (NIDIS)

U.S. Western Governor’s Association – 1996: Recommendation: U.S. to prepare for, respond to drought – 2000: Creation of National Drought Policy Commission– 2003: NOAA Partnership on drought monitoring & forecasting– 2004: Formal document published recommending NIDIS

Subcommittee on Disaster Reduction, President’s Nat. Sci. & Technology Council

– 2005: Highlighted drought as one of the grand challenges in disaster reduction

• Proposed action: development of an implementation plan for NIDIS

U.S. Integrated Earth Obs. System– 2005: NIDIS identified as 1 of 6 IEOS near term opportunities



NIDIS – A potential model for partnerships among Gov’t, Academic and Private Sectors

• NIDIS Executive Council– Co-chairs

• NOAA Climate Program• Univ. of Nebraska’s National

Drought Mitigation Center

– Members are senior reps. from lead organizations

• USDA, DOI, others • the Regional Governor’s

Assn., Water Councils, River Basin Commissions

• State government • Private sector• Academia

MajorNCDCEffort



A NOAA Archive and Access Business Model

A potential way to help define sector roles



A NOAA Climate Archive Business ModelData Quality versus Latency Component

National & Intl. Scientific Assessments

State of the Climate

Climate Data Products

Model Reanalysis

Climate Information Records

Confidence Intervals

CDRs with Multiple Independent Observing System

CDRs with Multiple Analysis Team

Climate Data Records (CDRs)

Missing Data

Random Errors

0 days 1 day 1 week 1 month Seasonal Annual 2-10 years

Heavy Moderate Light Little or none Not relevant

Color reflects NOAA Climate Data and Monitoring activity level:

Data Improvements TIME

Adheres to all of the GCOS Monitoring Principles



A NOAA Climate Archive Business Model Data Quality versus Latency Component




Model Reanalysis






Missing Data

Random Errors


Color Reflects NOAA Climate Data and Monitoring Activity Level:



Adheres to many, but no all, of the GCOS Monitoring Principles



A NOAA Climate Archive Business Model Data Quality versus Latency Component




Model Reanalysis






Missing Data

Random Errors


Color Reflects the NOAA Climate Data and Monitoring Activity Level:



Adheres to few, if any, of the GCOS Monitoring Principles



Potential Method to Define Roles Federal, Academic & Private Sector Partners

NOAA Business Model for Archive & Access

• Data Quality versus Latency Component

– NOAA heavy focus:• On data improvements for delayed data (~1 month)• On data improvements from observing systems adhering to GCOS

monitoring principles

– NOAA light focus:• On real-time, short-term(~1 week) data improvements• On observation systems not adhering to GCOS monitoring principles



Possible Method to Define Roles Federal, Academic & Private Sector Partners

NOAA Business Model for Archive & Access

Examples of other components that could be expanded to define Federal, Academic, and Private sector roles

• Data Storage Compression Techniques• Metadata• New Products• New Access Methods• Others

Questions

Documents

1 2008 AMS Summer Community Meeting Boulder, Colorado – 11-13 August 2008 The Emerging Interface among the Private, Federal & Academic Sectors in the Archive