1

Comprehensive Large Array-data Comprehensive Large Array-data Stewardship SystemStewardship System

(CLASS)(CLASS)

http://www.class.noaa.govhttp://www.class.noaa.gov

Comprehensive Large Array-data Comprehensive Large Array-data Stewardship SystemStewardship System

(CLASS)(CLASS)

http://www.class.noaa.govhttp://www.class.noaa.gov

GOES-R -- Overview and Plans

Richard G. ReynoldsCLASS Project Manager

May 12, 2004

2

AgendaAgenda

CLASS System Overview

Project Management

Accomplishments to Date

Current Hardware Architecture

FY04 Goals

“Final” Architecture

Budget … Resources and Commitments

Issues and Risks

3

CLASS OverviewCLASS Overview

NOAA's National Data Centers and their world-wide clientele of customers look to CLASS as the sole NOAA IT infrastructure project in which all current and future large array environmental data sets will reside. CLASS provides permanent, secure storage and safe, efficient access between the Data Centers and the customers.

4

CLASS OverviewCLASS Overview

CLASS is a web-based data archive and distribution system for NOAA/NESDIS environmental data

Archive … ingest, storage, metadate management, and data quality assurance

Distribution … access, visualization, and data delivery

CLASS is an extension of an 1995 operational system … SAA (Satellite Active Archive)

– Transition to the CLASS architecture began in 2001– “Dual-Site” Operational CLASS began 02 April 2004

CLASS currently supports POES, DMSP, and GOES data sets, plus RadarSat (Synthetic Aperture Radar) and SeaWiFS (Ocean Color Product)

CLASS will support additional campaigns, broader user base, new functionality currently being defined

– CLASS concurrently support ongoing operations and new requirements implementation

5

System Overview – Functional Flow Diagram

System Overview – Functional Flow Diagram

Ingest and Store Data

VisualizationData

Data SetInventory

DataCaches

Orders

Maintain,Monitor,Control

ProcessOrders

AccessData

VisualizeData

Interfacewith Users

Data ProductsAnd Metadata

DataProviders

USERSUSERS

CLASSOperators

Archive

CLASS Internet/Intranet

6

System Overview – System Design

System Overview – System Design

Data Storage and Distribution Subsystems

– Ingest, Subscription and Bulk Order Generation, Data Recall, Delivery

 Servers:

– Inventory, Visualization, Order

• Socket messages in XML format

• Object oriented design implemented in C++

7

System Overview – System Design

System Overview – System Design

User Interface– Java/XML-based web interface uses COTS components

• Apache – Web Server;

• Cocoon - publishing framework

• Avalon/Excalibur - database connection pooling

• Tomcat - servlet engine

• LogKit - message logging

• Informix Java Database Connectivity (JDBC) Driver

Monitoring and Maintenance Tools– Log Monitor, Cache Cleanup, Work Space Cleanup,

Independent Monitoring, Operator Interface

• C++

• Perl

8

System Overview– Distributed Redundant SitesSystem Overview–

Distributed Redundant Sites

Ingestprocess

Operationalinventory

Archiver

Archiveinterchange

Roboticstorage

Supplier Supplier Supplier Supplier Supplier Supplier

Ingestprocess

Operationalinventory

Archiver

Archiveinterchange

Roboticstorage

Suitland Asheville

Operationaldatastore

Operationaldatastore

USER USER USERUSERUSER

9

System Overview– Servers

System Overview– Servers

10

System Overview– User Interface

System Overview– User Interface

USERS

11

System Overview– Maintenance and MonitoringSystem Overview–

Maintenance and Monitoring

12

CLASS Overview (Continued)CLASS Overview (Continued)

NOAA Observing System-- Notional Architecture --

The “End-to-End System”

Larger System

Observing System

Data Handling System

Human

Environmental Phenomenon

Environmental Parameter

Sensing Element

Sensor

Platform / Station

part of

measures

is a

contains

characterizes

provides data to

Observation Control System

is controlled by

Location

located at

MobileFixed

is

SpaceAirGroundOcean

SpaceAirGroundOcean

Basic Service Requirement

< drives

provides data directly to

Userprovides info to is type of

Stake-holder

has

Operatoroperated by

situated on

Support

supported by

Owner

owned by

Stakeholder Requirement

< drives< drives

Processing Element

RemoteIn Situ

is type

CLASS

13

Project ManagementProject Management

NESDIS Data Archive Board

CLASS Project

Richard G. Reynolds

CLASS Project Management Team (CPMT)

NGDC Development

Teams (Boulder, CO)

NCDC Development

Team(Fairmont, WV)

OSDPD Development

Team (Suitland, MD)

System Integration & Test Team

(Suitland, MD)

OSDPD Operations

(Suitland, MD)

NCDC Operations

(Asheville, NC)

Archive RequirementsCouncil

ITAT

14

Richard G. Reynolds (OSD)

Charles Bryant (OSD), Chung Wu (OSD)

Suitland Team

Alex Kidd (OSD)Robert Rank (OSD)Tino Cremidis (CSC)David Bowman (NCDC)

• Baseline system • Processes/policies System Architecture & Design• System Integration & Testing• Support Suitland operations

West Virginia Team

Carlos Martinez (TMC)

• GOES Lead• CCR Implementation• NPP Implementation• EOS Implementation• Support Asheville operations

Boulder Team

Eric Kihn (NGDC)

• SABR Integration• DMSP • Data Mining

•Ted Habermann (NGDC)

• Geospatial databases• Geotiff images• Metadata

COAST

Chung Wu (OSD)

• E-commerce• Order Management System

Project Management (Continued)(Continued)

15

DataArchive Board

(DAB)(Tom Karl/NCDC)

CLASS PROJECT

OSD (w/ CSC & TMC)

& Data Centers (w/ Contractors)

Archive RequirementsCouncil (ARC)

(John Bates/NCDC)

Information Exchange

16

CLASS Accomplishments to DateCLASS Accomplishments to Date

Completed overall design of CLASS top-level architecture Prepared Key System Documentation

– Requirements

– ICDs

– CONOPS

– Management Plans and Procedures Established operational, integration and test, and

development environments in Suitland Completed migration from SAA to CLASS

– Migrated about 30TB of data from old tape archive system

– Added more than 50 new derived products to the archive

Consolidated three web sites in one web-based user interface

17

CLASS Accomplishments to Date (Continued)CLASS Accomplishments to Date (Continued)

Enhanced ingest system to be independent of file type

Delivered baseline systems to Suitland and Asheville

Suitland CLASS Operational with POES and DMSP data sets

Completed IJPS/Metop – Archive and Access Segment Preliminary Design Review

Coordinated with NPP/NPOESS for defining the IDPS to CLASS Interface Control Document (draft)

Worked with NASA personnel to define initial requirements to archive EOS/MODIS Level-0 data.

18

CLASS Accomplishments to Date (Continued)CLASS Accomplishments to Date (Continued)

Established the CLASS Operations Team (COT)– Includes OSDPD and NCDC personnel

– In preparation for Baseline Dual-Site Operations

Established a project wide risk management program

Completed documentation of business case for CLASS

Suitland CSC Development Team Certified – SEI-CMM Level-3

Worked with IBM-China for possible implementation of CLASS for National Satellite Meteorological Center (NSMC)

19

CLASS Accomplishments to Date (Continued)CLASS Accomplishments to Date (Continued)

Completed … Summary “10-year” CLASS Project Budget

– FY04/05, plus $25.6M Flat-line …. FY06-FY14

Completed … NPP/NPOESS Campaign Plan

Completed … FY04 Spendplan

– Against the Approved $10.5M NESDIS funding allocation

Completed … Initial CMMI Training at TMC

Completed … CLASS Developers Workshop

– Held 03/04 March 2004 in Suitland

– Brought together all CLASS team members

20

CLASS Accomplishments to Date (Continued)CLASS Accomplishments to Date (Continued)

Completed … Software Release 2.0 / 2.1 / 2.2 – Operational Dual-site configuration

– Operational 02 April 2004

Completing GAA Activities– Final McIdas-less software design in final preparation at TMC

Planning for EOS/NPP Implementation at TMC– Task SOW Completed, sent to TMC, and Proposal received

– NASA/IV&V Facility MOU in preparation for floor space

21

CLASS -- Near-term PlansCLASS -- Near-term Plans

CLASS Release 3.0– Scheduled for 24 May 2004

– Provides: Delivery Manifest and Web Enabled Subscription Management

CMMI Activities – Preparing all CLASS development teams for Internal

Evaluation in July 2004

CLASS Release 3.1– Scheduled for August 2004

– Provides: Ingest Enhancements to support IJPS NOAA data

22

CLASS -- Near-term Plans (Continued)CLASS -- Near-term Plans (Continued)

System Storage Capacity Upgrade– Scheduled for September 2004

– Additional disk space at both CLASS operational sites

CLASS Release 3.2– Scheduled for November 2004

– Provides: Ingest enhancements to support IJPS EUMETSAT data, and IJPS Test readiness

23

FY04 CLASS GoalsFY04 CLASS Goals

Prepare a “5-Year” Budget and Program Plan for CLASS (COMPLETED)

Focus CLASS resources on CLASS Activities (COMPLETED)

– CLASS, GAA, EOS, SAA Projects combined

Initial test of CLASS enhancements for IJPS … NOAA-N (October 2003 - COMPLETED)

Configure and Test CLASS at NCDC and transition to Stand-alone Operations (December 2003 - COMPLETED)

Complete testing and transition to operations of CLASS dual site capabilities (April 2004 - COMPLETED)

Complete testing and transition to operations of the GOES Campaign (April 2004 - COMPLETED)

Complete IJPS/Metop Critical Design Review (21 April 2004 - COMPLETED)

24

FY04 CLASS Goals (Continued)FY04 CLASS Goals (Continued)

Configure and Test CLASS for support of IJPS/Metop, and transition to Operations (September 2004)

Establishment of an integrated Design-Implementation team between Suitland and TMC/Fairmont

Design NPP Campaign Data Ingest as initial phase of NPP Campaign, and with partial implementation

Design and Implement EOS Data Ingest as initial phase of EOS Campaign

Design of the JASON Campaign Data Ingest, as initial phase of the JASON and OSTM Campaign(s)

Completion of the GOES (development) Campaign Continue migration of GOES Retrospective Data Sets into CLASS Continue migration of POES Retrospective Data Sets into CLASS

O&M of Dual-site CLASS System at Suitland and Asheville

25

Current CLASS Hardware ArchitectureCurrent CLASS Hardware Architecture

CLASS hardware for Suitland and Asheville sites:

– Operational Servers, 4 at each site

• IBM P660, 4GB RAM 2CPUs

– Integration and Test Servers, 2 at each site

• IBM P660

– Development Servers, 2 at Suitland

• IBM P660

– Storage Area Network (SAN)

• SUN 9980, 5.2TB

– Tape Archive System & Hierarchical Storage Manager (HSM)

• 9522 LTO Tape Slots, 12 Drives, 8PB capacity @ LTO-3

26

Suitland System Hardware

Suitland System Hardware

27

28

The Future CLASSThe Future CLASS

NESDIS e-Commerce System (NeS)

NeS OracleDatabase

NVDS

The Future - CLASS

i-Store (Online Store)

Catalog withPricing

Archive and Distribution System (ADS)

CLASS

Data Centers (Archive & Delivery - FTP and Manual)

CLASS InformixDatabase

Catalog withoutPricing

CLASS Archive & Delivery (FTP Only)

External

Data

Sources

Accounting andReporting

Order Entry andManagement

Order Entry andManagementReporting

Retrieval andDistribution

Metadata Data Ingest

Archive andMigration

Web Access

Search

Web Interface

Visualization Geospatial

DataBase

ArchiveArchive

ArchiveArchive

ArchiveArchive

ArchiveArchive

Archive and Data Base

SatelliteImage Browser

Sy

ste

mM

an

ag

em

en

t

Inventory andCatalog

Internet

29

CLASS BudgetsCLASS Budgets

FY00 and prior N/A

FY01 $1.995M*

FY02 $3.599M*

FY03 $2.881M*

FY04 $10.5M

FY05 $11.1M

* NET

• FY06 $11.9M

• FY07 $9.9M

• FY08 $9.9M

• FY09 $22.1M

• FY10+ $27.9M/yr

30

“GOES-R Campaign”“GOES-R Campaign”

Requirements Definition and ICDs

Data and Products Ingest

Storage, Processing, and Communications Upgrades

Metadata extension

Catalogue extension

Visualization extension

Reprocessing extension

31

CLASS “GOES-R” BudgetCLASS “GOES-R” Budget

FY07 $2.0M

FY08 $3.8M

FY09 $4.1M

FY10 $7.6M

FY11 $8.5M

FY12 $6.9M

FY13 $6.2M

FY14 $2.0M

$41.0M TOTAL

32

THANK YOU!