1

Roadmap for Satellite Data in the Advanced Weather

Interactive Processing System (AWIPS)

GOES Users’ ConferenceMadison, WI

Deirdre JonesNOAA’s National Weather Service

Office of Science and TechnologySystems Engineering Center (SEC)

November 5, 2009

2

Outline

• Challenges meeting new customer weather needs

• Architectural Vision

• AWIPS Plan to Achieve Architecture

• Next Steps

• Summary

3

Customers Need Improved Weather Services

• Speed at which decisions are made • Demand for decision support services is increasing• US industry needs the most accurate, accessible,

timely and reliable weather data to make critical decisions that impact our national economy– Aviation weather impacts were $41B in 2007– U.S. modeling and data assimilation lags other nations. We

must do better to give the U.S. a competitive advantage in the global economy

• Federal deficits and resource constraints– Integrated observations– More efficient R-T-O (projects, modeling)– Every dollar counts!

4

Environmental Data Challenges

• Huge data explosion– Rapid data assimilation requirements (e.g.,

NextGen)—people, models– Demands on data management architecture– Data access on-demand within resource

constraints

• Integrating all observing data sources to achieve desired effect and outcome

5

NESDIS Archive Data Rate Projections

w/NPP

w/NPOESS C1

w/GOES-R

w/NPOESS C2

w/GOES-S

w/MPAR

Data Volume(non-cumulative view)

AWIPS SBN3-18 Mbps

6

WIDB Cube

CustomGraphic

Generators

CustomGraphic

Generators

Governmental Decision MakingGovernmental Decision Making

DecisionSupportSystems

DecisionSupportSystems

CustomAlphanumeric

Generators

CustomAlphanumeric

Generators

ObservationsObservations

NumericalPredictionSystems

NumericalPredictionSystems

PostprocessedProbabilistic

Output

PostprocessedProbabilistic

Output

NWSForecaster

NWSForecaster

ForecastingForecasting

Automated ForecastSystems

Automated ForecastSystems

Forecast IntegrationForecast Integration

RadarsRadars

AircraftAircraft

SurfaceSurface

SoundingsSoundings

Private SectorPrivate Sector

Private SectorPrivate Sector

Weather IndustryWeather Industry Private IndustryPrivate Industry

Importance of Weather Information Database (WIDB)

7

Infrastructure Readiness Approach

• View requirements in larger context of all future programs

• Develop strategic enterprise infrastructure plan and roadmap for building infrastructure capability over next 10-15 yearsNear term tasking to complete analysis of the

GOES-R impact on NWS architecture: AWIPS, NCEP and NWS Telecommunications Gateway (NWSTG) (FY08-FY10)

Assess other aspects of the enterprise, as well as other system changes (FY10)

Select architecture for efficient access to new satellite data and products for forecast and warning operations (1QFY11)

Initiate acquisition (FY 11-12)

8

NOAA CIO's Architectural Vision

Web Service Interfaces- Utilizing Open Standards &

Protocols

Customers• Experts• Informed • NovicePresentation Tier (i.e., Web Presence)

PortalsMobile

DevicesCollaboration

ServiceRegistry

Governance

(e.g., S

LA

s)D

ata

Arc

hite

ctur

e(e

.g.,

GE

O-I

DE

, IO

OS

)

Infrastructure ServicesCommodity

e.g., Authentication,

Discovery

Core e.g., Common,

Storage, Compute, Enterprise Mgmt

Transporte.g., LAN / MAN /

WAN/Internet

Mission Services

Ecosystem Management & Protection

Climate Preparation & Response

Commerce & Transportation Efficiency, Safety, and

Environmental Soundness

Weather & WaterPreparation & Response

Critical Support (Satellites & Mission Support)

Modeling & Observing Infrastructure

Security e.g., Threat Detection, Network Scanning

9

NOAA Infrastructure Framework

• Consider environmental data changes in context of NOAA’s full infrastructure and in concert with each other

• Service oriented enterprise focused on data and information – Infrastructure services: core, commodity, transport, and security

Transport migration: consolidate NOAA’s disparate networks into NOAANet architecture

– Mission services: monitoring and observing, modeling and forecasting, disseminating, ecosystem management, etc.

– Data architecture: data description, context, and sharing – Governance: service level agreements, data management– Dissemination Services

To include emerging technologies and Internet services: portals, mobile devices, and collaboration

10

Notional Architecture

NOAANet

ANCF/BNCF

AWIPS WFO Subsystems (122)

NOAANet(AWIPS VPN)

AWIPS RFC/NC

Subsystems

AWIPS

AMGS/BMGS

AWIPSSBN

GOES-R/NPOESSCentralized Distribution

Server

NWS Telecom Gateway

Product Distribution

(PD)

Product Generation

(PG)

GOES-R Ground Segment (GS)

GOES-R Access

Subsystem (GAS)

GOES-R Data

Ingest

NPOESS Data Exploitation (NDE)

Other Mission Data (Jason-2, MetOp, etc)

NESDISsectorized cloud

and moisture imagery products

ad hoc and subscription access of GOES-R data and products

GOES-R GRB

NCEP Centers

Other Gateway Servers

Comprehensive Large Array-data Stewardship System (CLASS)

Centralized Server Data Distribution Method

(Note 2)

Note 1: Conceptual; design - TBDNote 2: Requires additional bandwidthNote 3: Requires hardware and software upgradesNote 4: NextGen program supplies data/product registry and catalog to enable access to NOAA repositories. This infrastructure will be in place before GOES-R launch and may reside at Gateway.

(Note 2)

(Note 3)

(Note 3)

(Note 3)

(Note 2)

(Note 3)

(Notes 1 and 4) (Note 4)

11

AWIPS in the GOES-R EraFitting the Pieces Together

• Flexible software infrastructure

• Communications bandwidth -- satellite broadcast and terrestrial networks

• Data distribution paradigm -- push and pull

• SEC leading effort to prepare the enterprise as well as planning AWIPS improvements to support future data

• Robust infrastructure with capacity and throughput for larger volume/higher resolution data

12

AWIPS II Software Re-Architecture Approach

• Perform “black-box” conversion– Preserve existing functionality, – Preserve look and feel of today’s system

• Maintain functionality– Deployed system current with deployed

AWIPS capability (i.e., OB9)

• Use open source projects - No proprietary code– JAVA and open source projects enable AWIPS II to be platform

and OS independent– No plans to move from Linux

• AWIPS II enables collaborative development– Operating System and platform independence allows non-Linux

based research to be easily integrated into AWIPS II

13

AWIPS II Software Re-Architecture Outcome

• Improved software architecture and capabilities to accommodate new data and services

– Services oriented architecture – More flexible in production/delivery – Increased access to data for decision making– Improved RTO efficiency

through use of AWIPS Development Environment

– Enables future capabilities (e.g. flexible data delivery, thin client, collaboration)

14

AWIPS SOA Architecture Logical Layered View

Client/Presentation Services

Platform Layer

Mission Services Layer

Data Access Layer

Metadata Index

Data PersistenceStore

En

terp

ris

e S

erv

ice

Bu

s -

C

om

mu

nic

ati

on

Se

cu

rity

Se

rvic

es

/D

em

ilit

ari

zed

Zo

ne

(D

MZ

)SpatialIndex

Hydro Models

LAPS

FORTRAN/C/C++Command Line

Programs

External Programs

JMX

<<Java>>

DataLayer

PostgreSQLHDF5

<<abstract>>

BaseDao

Hibernate

<<Java>>

HDF5DataStore

HDF5 API

IngestSrv

PersistSrv

IndexSrv

ProductSrv

AdapterSrv

NotifySrv

SubscribeSrv

AutoBldSrv

PurgeSrv

Mbean

CAVE

StagingSrv

UtilitySrv

LocalizationStore

15

AWIPS Product Suite On NOAAPort Today*

* NOAAPort disseminates over one million products (30+GBytes) every day

16

GOES Data in AWIPS Today

Product Name WMO Header

Dissem. Path

Format (AWIPS ID) Approx. Frequency

Approx. Daily Volume

TIG?iiTIC?ii

Sounder-based Imagery TIG?ii 1 GINI hourly 216 MbitsDPI TIGNii 1 GINI hourly 280 MbitsSoundings JUTX ii 2 BUFR 30 min 312 MbitsAE SPE Imagery ZETA98Manual SPE Imagery ZEGA98

TextSPENES

Hi Den Winds (BUFR) J?CXii 2 BUFR 1-3 hourly 136 MbitsTWNAii TextTWSAii (none)

TextSCP?Ri

TextWBCSAT5/6

NOUS71 TextNOUS72 ADMNES

CONUS RTMA (ECA) LAMA98 2 GRIB2 Hourly 56 Mbits

Imager-based Imagery 1 GINI

GOES Products Flowing into AWIPS

30/15/7 min 27200 Mbits

2 GRIB1/Xmrg hourly & event driven

216-320 Mbits

0 Mbits

Hi Den Winds (Text) 2 1-3 hourly 80 Mbits

SPE Text TXUS20 2 event driven

1.6 Mbits

Geodetic Subpnt Predictn. TBUSii 2 daily 0 Mbits

ASOS Cloud Products TCUSii 2 hourly

Administrative Messages 2 event driven 0 Mbits

17

• Environmental Satellite Product Additions• POES AVHRR Imagery (2010)• NPP & NPOESS (From NPP, beginning 2011)• GOES-R Series (2015)

• Currently defining process for building, evaluating and implementing new products in the AWIPS II framework

• NCEP Model Product Additions• High Resolution Window Model (2010)• GFS-Based DNG (2010)• Bias-corrected SREF (2011)• Numerous additional enhancements to NCEP (ongoing)

• Radar (WSR-88D) Product Additions• Dual Polarization (2011)• Higher Pixel Resolution (2013)

Future NOAAPort Product Suite*

* Only selected major planned additions are shown; pending bandwidth expansion; dates tentative

18

AWIPS Communications Improvements

• Migrate AWIPS wide area network (WAN) to NOAAnet– Increased reliability – Increased bandwidth– Enables any-to-any network topology

• Increase AWIPS SBN bandwidth to 45 Mbps (full transponder)– 5 times current capacity– Enables flexible delivery over satellite

• New paradigm for data distribution

• User readiness working group to work with users/ customers to define product access and delivery reqts.

19

AWIPS Data DeliveryPush/Pull Concept

• Robust infrastructure to support “intelligent” access (push/pull) to non-local datasets

• Key Benefits – Mitigates impacts on SBN by addressing significant

growth in data volumes, e.g., ensembles, GOES-R, NPOESS

– Allows users to access just the data they need by space, time, parameter

– Enables synergy with NextGen net-enabled data access requirements

20

NWS Infrastructure ArchitectureImplementation Schedule

CY 2009 2010 2011 2012 2013 2014 2015 2016

NWS GOES-R ActivitiesAssess Impact on NWS Infrastructure

Architectural Requirement Development

Infrastructure Design and Development

Stand up New Architecture

Test and Evaluation of New Architecture

Dual Polarization Milestone

GOES-R and NDEMilestones

NextGen Milestones

Oct 2015GOES-RLaunch

April 2013NPOESS

C1 Launch

January 2011NDE “Distribution

Build” Testing,NPP Launch

July 2013GOES-R “Data

Operations Tests”

IOC

Deployment

MOC

21

Summary

• NOAA is working on the enterprise infrastructure to accept, manipulate, and use data from new capabilities

• Ensure all NWS systems on track for user readiness– Complete assessment– Address in design solution– Build toward NOAA target architecture

• Efficiencies to reduce cost needed; seek out new technologies for computing and data integration architectures

22

Back-up

23

What is AWIPS?

NEXRAD

AWIPS Workstations andServers

GOES/POES/NPOESS

ASOS

NCEPModels

Buoys,River Gauges

Forecasts

Warnings

Advisories

Watches

AWIPSCommunications

Service provided to3066 US Counties

24 hrs/day, 365 days/yr.

169 separate AWIPS systems at137 geographical locations

~900 Workstations (total)~1200 Servers (total)

24

NOAAPortNOAA’s Satellite Broadcast Network

GOES POES

NOAA/NESDIS

NOAA/NWS

DC

S

AV

HR

RH

IRS

/AM

SU

GV

AR

NCEP

Hydromet Sensor Products (e.g., radar, ASOS,

Profiler, River Gages)

TOCNWSTG

SatelliteProduct

Generation

AWIPSNCF

Guidance &Model

Products

Satellite imagery

Other EnvironmentalSatellite Products

AMC-2Satellite

NOAAPORTUplink

Facility (NY)

WFO/RFCForecasts, Watches

Warnings

NOAAPORT UsersIncludes Many Users –

Inside and Outside of NOAA

25

Product Stream Assembly: AWIPS Network Control Facility (Silver Spring Md)NOAAPort Satellite: SES Americom (AMC-2), Transponder: 13C NOAAPort Satellite Location: 101◦ WestDownlink Signal: C BandNetwork Capacity: 10 Mbits/sec (approximate)Transmission Protocol: DVB/S - Digital Video Broadcast by SatelliteProduct Identification: Headings (NOAAPort, WMO/AWIPS & product)Major Product Formats: GRIB, BUFR, GINI, Radar Level III, Text, RedbookCompression: zlib for satellite imagery and most radar products;

GRIB2 supports JPEG2000 compression

There are four NOAAPort “data streams” or “channels”– each with a unique DVB PID.

• GOES/NESDIS• NCEP/NWSTG• NCEP/NWSTG2• OCONUS Imagery/Model/DCP

NOAAPortTechnical Specifications

26

Other Satellite Data in AWIPS

Product Name WMO Header Dissem. Path Format (AWIPS ID) Approx. Frequency Approx. Daily Volume

224 Mbits

(Future)

Soundings IUTXii 2 BUFR 100 min 64 Mbits

Text

WBCSAT5/6

NOUS71 Text

NOUS72 ADMNES

QuikSCAT

Marine Winds

DCP S???ii KWAL 2 Text >100K/day 800 Mbits

ISXAKWNO

POES & DMSP DPI TIT?ii 1 GINI 180min 144 MbitsMTSAT HiDen WindsMETOP ASCAT Marine Winds

JSXX 2 BUFR 110min 160 Mbits

GINI 10 per day per satellite

Geodetic Subpnt Predictn. TBUSii 2 daily

POES Imagery TIT?ii 1

0 Mbits

Administrative Messages 2 event driven 0 Mbits

100min 160 Mbits

ISXX 2 BUFR 110min

512 Mbits

Non-GOES Products Flowing into AWIPS

JUTXii 2 BUFR 1-3 hourly

160 Mbits

DMSP SSM/I Winds 2 BUFR

27

Glossary of Terms

AWIPS: Advanced Weather Interactive Processing SystemCLASS: Comprehensive Large Array-data Stewardship System DMSP: Defense Meteorological Satellites Program GOES: Geostationary Operational Environmental Satellite GOES-R: Geostationary Operational Environmental Satellite R-SeriesGOES-R: Geostationary Operational Environmental Satellite S-SeriesIMETS: Incident MeteorologistsIOC: Initial Operating CapabilityMetOp: Meteorological Operational SatelliteMPAR: Multi-function Phased Array Radar Mid Operating CapabilityNDE: NPOESS Data Exploitation NESDIS: National Environmental Satellite, Data, and Information ServiceNextGen: Next Generation Air Transportation SystemNEXRAD: Next-Generation RadarNPOESS: National Polar-orbiting Operational Environmental Satellite System NPP: NPOESS Preparatory ProjectPOES: Polar Operational Environmental SatelliteR-T-O: Research to OperationsSBN: Satellite Broadcast Network