Upload
katrina-terry
View
219
Download
0
Tags:
Embed Size (px)
Citation preview
CERES FM-5 Delta Design Review 1
CERES FM-5
NPP Science Processing
Delta Design ReviewGSFC, MD
August 26, 2008
Erika GeierJim Closs
Denise CooperSunny Sun-Mack
Lee BoddenMichael Little
NASA Langley Research Center
CERES FM-5 Delta Design Review 2
Agenda
CERES Introduction
Operations Concept
Design Assumptions
Science Data Processing
Requirements
Instrument Subsystem Design
Clouds Subsystem Design
Testing Strategy
ASDC Implementation
Schedule
Risk Analysis
Issues/Concerns
CERES FM-5 Delta Design Review 3
CERES Mission Ground Systems
• NASA Langley Research Center (LaRC) provides critical cloud and Earth radiation budget climate data
records to support global climate change research
• Past ERBE and current CERES instruments on TRMM (1), Aqua (2), and Terra (2), have provided 24 years of sustained radiation budget measurements
• LaRC has collected over 30 Instrument years of CERES data from TRMM, Terra, and Aqua and has processed over 90% of that data to date
CERES FM-5 Delta Design Review 4
CERES Current Capabilities
• Extensive LaRC CERES experience accumulated through the support of the five CERES instruments currently in orbit
• Existing scientific expertise for the development, calibration, management and application of CERES data products
• Data Center Operations already in place providing ingest, archival, production, management and distribution of CERES data products
• Existing Science Computing Facility (SCF) for scientific analysis, investigations and development of CERES data products
• Highly-experienced staff already in place supporting:
– Ongoing algorithm investigation, development and enhancements
– Implementation of algorithms into CERES Data Management System
– Enhancements and development to current and new CERES data production systems
– Production and distribution of CERES data products in a configured operations
• The CERES DM task was successfully appraised at CMMI Capability Level 3 /Equivalent Maturity Level 2
CERES FM-5 Delta Design Review 5
Science
CERES Climate Analysis Research System (CARS) Organization
Data Management
Team(DMT)
Atmospheric Sciences
Data Center(ASDC)
• Implements algorithms• Maintains software• Verifies data• Assists in validation• Provides CM and documentation support
• Ingests data• Places operational software in production• Produces data sets• Distributes data sets• Archives data• Provides User Services
• Derives & refines algorithms• Validates algorithms• Validates CERES data sets• Writes Quality Summary
Algorithm Development Algorithm Implementation Data Production
CERES FM-5 Delta Design Review 6
Required CERES CARS Functionality
• Provide Land PEATE sub-sampling software for aggregated
VIIRS SDRs (Science and DMT)
• Acknowledge data from SDS Land PEATE (ASDC)
• Ingest data from SDS Land PEATE & other sources (ASDC)
• Produce, archive, distribute data products (ASDC)
• Report status of collection (ASDC)
• Support user access to collection (ASDC)
CERES FM-5 Delta Design Review 7
CERES Operations Concept
• CERES instrument raw data transmitted to the C3S and IDPS
• IDPS creates CERES Raw Data Records (RDRs) and provides RDRs to the SDS SD3E and to ADS/CLASS
• SD3E provides CERES RDRs and VIIRS xDRs to the Land PEATE
• Land PEATE passes CERES RDRs along to ASDC
• Land PEATE runs VIIRS Sub-sampler; provides output to ASDC
• ASDC ingests and archives inputs coming from Land PEATE
• ASDC generates and archives higher-level CERES products
• ASDC distributes products to science researchers and public
CERES FM-5 Delta Design Review 8
CERES Operational Data Flow
SD3E L-PEATE
ADS
IDPS
SDS
All RDRs
All xDRs
CERES RDRs, VIIRS Sub-samples
& aerosols
C3S
Commands, Loads and requests
RDRsSDRsEDRs
IPs
CERES InstrumentOps Team
Svalbard
Cmd &
Tlm
Mission Data
Existing data, agreements
Data Providers
Mission Notices and Data
RDRs to PST
CERESCARS
Data and
Science Operations
Mission Notices and Data requests
Data Distribution
Data Users
• Reuse existing systems and interfaces• System enhancements for NPP CERES• Reuse existing systems and interfaces• System enhancements for NPP CERES
NPOESS Systems
NPP Systems
NOAA Systems
CERES Systems
CERES FM-5 Delta Design Review 9
CERES Data Processing Design Assumptions
• CERES CARS data processing development and operations leverages the existing personnel, procedures, production systems, and infrastructure already in place supporting CERES instruments on EOS Terra and EOS Aqua Missions
• CERES CARS will use existing interfaces at the LaRC ASDC
• CERES CARS software development is extensively reusing existing Terra/Aqua processing codes
• Primary required NPP related enhancements are:
– Use CERES raw data in new Raw Data Record (RDR) format:
Develop a preprocessor to distribute/sort RDR contents into Aqua-like files that can be used as inputs to existing CERES processing stream
– Use VIIRS data products:
Use VIIRS sub-sampled calibrated radiance and aerosol products in place of instantaneous MODIS data.
CERES FM-5 Delta Design Review 10
NOAA IDPS Assumptions
• Acquire CCSDS Packets
• HDF Wrap CCSDS Packets
• Generate & Stage RDRs for all APIDs for SDS
& CLASS
• Maintain CDFCB
• Provide samples of each CERES RDR
• Provide 28 hours of CERES RDRs for testing
CERES codes at LaRC
CERES FM-5 Delta Design Review 11
SDS/Land PEATE Assumptions
• Subscribe to SD3E for all CERES RDRs (Science, Diagnostic, and Housekeeping/Telemetry)
• Acquire CERES RDRs and forward to ASDC via existing Land PEATE interface
• Support CERES reordering of missing data
• Produce aggregated VIIRS SDRs (MODIS HDF-EOS format) in 5 minute granules (VIAE, VMAE, VDAE). Sub-sample and ship 5 minute granules (VIMD) to ASDC
• Produce VIIRS aerosol in 5 minute granules (VAOT) and ship to ASDC
• Post-launch: When VIIRS calibration and/or algorithm changes, reprocess VIIRS from covers-open forward and ship to ASDC to support CDRs
CERES FM-5 Delta Design Review 12
New Requirement Received, Logged,
andAnalyzed
Requirement Accepted
and Assigned
Subsystem Updates and Unit
Testing
Updates Integrated Into
Baseline
Subsystem Science Testing
Pre-CM Testing
Delivery to CM
CM Testing
Release to SIT
Operational Testing at
ASDC
ValRx Testing
RequirementsManagement
Subsystem Product Integration and
Verification
CM Validation and Product Integration ASDC Validation
Software Development Process
Software Development Process is in the following document currently under CERES Management review:
Clouds and the Earth’s Radiant Energy System (CERES)National Polar-orbiting Operational Environmental Satellite
System (NPOESS) Preparatory Project (NPP)Data Management System (DMS)
Software Management Plan Version 0
CERES FM-5 Delta Design Review 13
Science Data Processing Approach for CERES FM-5
Processing is divided into 2 Streams:
• Instrument and ERBE-like– Autonomous stream, dependent only on CERES data
– In production within 48 hours of power-on
– Aids in verifying Instrument health
– Critical for establishing CERES calibration/validation
– Produces ERBE-like data sets Data sets exist for CERES instruments on TRMM, Terra, and Aqua Similar data sets exist for ERBE instruments on ERBS, NOAA-9 and NOAA-10
• Fused data sets beginning with Cloud/Convolution/Inversion processing and the SSF
– Use inputs from imager and other data sources Higher resolution imager data is Point Spread Function (PSF) weighted
– Produces higher quality climate data Data sets exist for CERES instruments on TRMM, Terra, and Aqua
– Will not be in production immediately after covers open Requirement is to ingest VIIRS radiance, geolocation, and aerosol data when it
becomes available from Land PEATE
CERES FM-5 Delta Design Review 14
Simplified CERES Processing Flow
SS4Compute Clouds,
Convolve withCERES, compare
TOA/surfacefluxes
CERES RDRs
SSI Instrument
SS2 Instantaneous
ERBE-like
SS3 ERBE-like
TISA
ES-4
ES-9
BDS ES-8
IES
Autonomous Subsystems affected by changing NPP inputs
Autonomous CERES Subsystems
VIIRS Radiances
GeolocationAerosols
SS9TISA
Gridding
SS10TISA
Averaging
SRBAVGSSF SFC
SS5Compute fluxes
at levels
SS6TISA
Gridding
SS7 & 8TISA
Averaging
AVG
SYN
CRS FSW ZAVG
Subsystems affected by NPP input
CERES FM-5 Delta Design Review 15
Current CERES Climate Data Record Production Architecture
Grid TOA andSurface Fluxes
9
ERBE-likeAveraging to
Monthly TOAFluxes
3
Grid GEONarrowband
Radiances11
GEO:Geostationary
Narrowband Radiances
TimeInterpolate, Compute
Fluxes7
Grid Radiative
Fluxes andClouds
6
MOA:Meteorological, Ozone,
and Aerosol Data
ES-8: ERBE-likeInstantaneous
TOA Estimates
ERBE-likeInversion to
InstantaneousTOA Fluxes
2
RegridHumidity and
Temperature Fields12
BDS:BiDirectional
Scans
SRBAVG:Monthly
TOA/Surface Averages
SYNI:Intermediate
Synoptic RadiativeFluxes and Clouds
ComputeMonthly & Regional
TOA and SurfaceAverages
10
DetermineCloud Properties, TOA
and Surface Fluxes4
Geolocateand Calibrate
EarthRadiances
1
SSF: SingleScanner Footprint
TOA/Surface Fluxes and Clouds
CRS: Clouds and Radiative
Swath
MODIS CID: VIIRS CID:Cloud Imager
Data
SURFMAP:Surface Map
INSTR:Instrument
Production Data Set
EID6:ERBE-like
Regional Data
AVG:Monthly Regional Radiative
Fluxes & Clouds
ZAVG:Monthly Zonal & Global
Radiative Fluxes & Clouds
ComputeRegional, Zonal and
Global Averages8
GGEO:Gridded GEO
Narrowband RadiancesFSW: Monthly
Gridded Radiative Fluxes and Clouds
IES: Instrument Earth Scans
CRH:Clear Reflectance
History
GAP:Gridded Analysis
Product
OPD:Ozone
Profile Data
MWH:Microwave
Humidity
APD:Aerosol Data
SFC: MonthlyGridded
TOA/SurfaceFluxes and Clouds
ES-9:ERBE-like
Monthly Regional Averages
ES-4:ERBE-like Monthly
Geographical Averages
ComputeSurface &
Atmospheric Radiative
Fluxes5
SYNSynoptic RadiativeFluxes & Clouds
CERES FM-5 Delta Design Review 16
CERES Input Data
Type of Data Parameter
Description
Freq Source Comments
CERES RDRs Instrument level 0 data, attitude, and ephemeris
~131/day Land PEATE Expecting 3 types of RDRs: Science, Telemetry, and Diagnostic
VIIRS L1B Calibrated Radiance Data
Imager radiances & Geolocation
Aerosols
288/day
~144/day
Land PEATE Have identified radiance subsets and will provide associated code
Aerosol data Aerosol (Coln) Optical thickness, type/size
1/day MODAPS For Terra/Aqua using MODIS MOD08 and MATCH. Plan to do same for NPP
Ozone data Ozone Profile 1/day SMOBA
1/day OMI
NCEP SMOBA
OMI Meteorological data 3-D Met Data
2-D atmospheric data
2-D constants
4/day
24/day
1
GMAO
Precipitable Water 2-D constants 2/day Global Hydrology Resource Center (GHRC)
Geostationary data MCIDAS data from 5 geostationary satellites per month
120/day University of Wisconsin Space Science and Engineering Center (SSEC)
Only every 3rd hour is used for production
SURFMAP(Snow/Ice) Snow/Ice Map 4/day NCEP/NESDIS
SURFMAP(Snow/Ice) Snow/Ice Map 1/day NSIDC
Existing Data Source
CERES FM-5 Delta Design Review 17
CERES Publicly Available Output Products
Data Product Data Product Name File Freq
BDS Bidirectional Scan 1/day
ES-8 ERBE-Like Inst TOA Filtered Radiances 1/day
ES-9 ERBE-Like Monthly Regional Averages 1/mo
ES-4 ERBE-like Monthly Geographical Averages 1/mo
SSF Single Scanner Footprint TOA/Surface Fluxes & Clouds 1/hr
SFC Monthly Gridded Radiative Fluxes & Clouds 36/mo
SRBAVG Monthly TOA/Surface Averages 5/mo
CRS Clouds and Radiative Swath 1/hr
FSW Monthly Gridded Radiative Fluxes & Clouds 60/mo
SYN Synoptic Radiative Fluxes and Clouds 1/day
AVG Monthly Regional Radiative Fluxes and Clouds 1/mo
ZAVG Monthly Zonal and Global Radiative Fluxes & Clouds 1/mo
CERES FM-5 Delta Design Review 18
Data Production Scenarios
• CERES strives to receive all possible FM-5 level 0 data and VIIRS data
• Instrument– Ingest manager at ASDC to monitor missing RDR files and request those files from Land PEATE
– Land PEATE has agreed to provide CERES 99% of RDR files produced by IDPS within 1 month
– Data is actually processed three times Baseline1-QC run ~ 6 hours after end of the day. Used for quick instrument assessment. Files not needed after
Edition1-CV processes. Edition1-CV run ~5-20 days after the end of the month. Assume all available data has been ingested.
Calibration/Validation data set used to compute gains and spectral response functions. Edition2 run 6-12 months later using Edition1-CV as input. Applies best available instrument gains and spectral
response functions.
• Cloud/Convolution/Inversion processing of the SSF– Ingest manager at ASDC monitors missing MODIS files and requests missing files from MODAPS.
VIIRS files to be handled in similar fashion.
– Once in routine production, Land PEATE has agreed to provide CERES 99% of available VIIRS files within 2 months of data date.
– Requires CERES and VIIRS radiance/geolocation data to avoid data gaps in SSF files If either one missing, a data gap results VIIRS aerosol data also needed. If unavailable, aerosol parameters set to CERES default fill values and no data gap
results.
– Current Terra and Aqua SSFs typically run 6-12 months after data date. NPP processing will be very similar to Terra and Aqua.
CERES FM-5 Delta Design Review 19
CERES CARS Requirements Management
• CERES Level 3 and Level 4 requirements are being baselined in a CM-controlled document
• CERES Level 3 requirements have been reviewed by the SDS and will be provided to the SDS for inclusion in the SDS Requirements Specification
• Requirements change requires approval by Project Management and the Configuration Control Board
• Requirements are mapped into subsystem(s) for implementation
• Requirements are mapped into tests for verification and validation
CERES FM-5 Delta Design Review 20
Land PEATE CERES CARS
CERES RDRs
VIIRS Geo/Rad
VIIRS Aerosols
VIIRS Subsample code
CERES RDRs
Sub-sampling
VIIRS Geo/Rad
VIIRS Aerosols
Instrumentprocessing
ERBE-likeprocessing
Cloudsprocessing
Archive, Distribution
User Community
CERES Data Sources
SD3E
1- The Land PEATE receives CERES RDRs and VIIRS xDRs from the SD3E. 2- CERES RDRs are passed directly to the CERES CARS, where they are processed by the Instrument subsystem to produce inputs to the ERBE-like and Cloud subsystems. 3- The Land PEATE produces MODIS-like VIIRS radiance/geolocation and aerosol files.4- The Land PEATE sub-samples the MODIS-like VIIRS radiance/geolocation data using software provided by CERES.5- The Land PEATE sends the sub-sampled VIIRS files and the MODIS-like VIIRS aerosol files to the CERES CARS.6- The CERES CARS runs the downstream subsystems, beginning with Clouds, as data becomes available.7- Climate-quality data products are archived and made available to the user community along with Data Quality Summaries and user support.
CERES FM-5 Delta Design Review 21
CERES SDS Requirements
NPP CERES FM-5 Science Data Segment Requirements:
• The ASDC Ingest subsystem shall have the capability of ingesting, verify by Checksum, and archiving CERES RDRs, sub-sampled VIIRS radiance/geolocation files, and VIIRS aerosol data received from the Land PEATE.
• The CERES CARS Preprocessor of the Instrument Subsystem shall have the capability of producing level-0, attitude, and ephemeris files as expected by the processing software.
• The CERES DMT shall have the capability of generating and delivering to the ASDC the following Science Subsystem code:- Instrument (at launch)- ERBE-like (at launch)- Clouds and subsequent subsystem software (12+ months after launch)
• The CERES DMT shall provide software to the Land PEATE to sub-sample VIIRS radiance/geolocation files, and work with the Land PEATE to ensure that the sub-sampling code executes properly.
CERES FM-5 Delta Design Review 22
CERES SDS Requirements
NPP CERES FM-5 Science Data Segment Requirements (cont.):
• The CERES DMT shall have the capability of modifying and redelivering production code for all 12 CERES Subsystems to produce CERES NPP data products.
• The ASDC shall have the capability of generating, archiving, and disseminating climate-quality data products.
• The CERES DMT and Science Team shall have the capability of validating and writing quality summaries for all CERES data products.
• The ASDC shall have the capability of providing quality summaries and user support for data customers.
CERES FM-5 Delta Design Review 23
Requirements Verification
1. Baseline LaRC FM-5 L4 requirements by placing CERES CARS Requirements Specification Document on NX (CERES CARS, 09/2008)
2. Baseline SDS FM-5 L3 requirements by adding CERES to SDS Requirements Specification Document (SDS, 09/2008)
3. Map FM-5 L3 requirements to SDS Testing Scorecard and schedule (Vic Buczkowski, 10/2008)
4. Produce Requirements Traceability Matrix to map LaRC L4 requirements to CERES CARS Subsystems and test cases (CERES CARS, 10/2008)
5. Conduct FM-5 SDS L3 testing as documented in SDS Testing Scorecard (SDS/LaRC Test Team, (02/2009)
6. Conduct FM-5 LaRC L4 testing as documented in LaRC Requirements Traceability Matrix (LaRC Test Team, (mid-2009)
CERES FM-5 Delta Design Review 24
Instrument Subsystem Design Approach
Instrument Design Approachand
CERES DMT RDR Tests
Denise Cooper
Contributing Team Members:Ashley Alford
Dale WalikainenMark Timcoe
CERES FM-5 Delta Design Review 25
Instrument Design Approach
• Extensive reuse of existing software for instrument subsystem
– Proven system used for current CERES processing
– Majority of software written in Ada
• Newly developed code to be in C++
– RDR Preprocessor Creates 24-hr Level-0 and Toolkit compliant Ephemeris and Attitude data files
• Ada code will be updated to allow processing of FM-5 data
• PGE to process incoming NPP Level-0 data will be the first priority, other PGEs will be converted following successful testing of this PGE.
CERES FM-5 Delta Design Review 26
Instrument Design Approach
• PGE providing data to ERBE-like already in C, only changes will be to allow processing of FM-5 data.
• Initial version of all code will use the SDP Toolkit due to schedule constraints. Updates to remove dependency on the SDP Toolkit expected after launch.
• CERESlib updates to create a C-version of existing F90 Metadata creation routines.
– Facilitate integration of existing Metadata with Ada, C and C++ software.
CERES FM-5 Delta Design Review 27
NPP Instrument Subsystem
CERES RDRs
L0 Data
Coeff Data
Geolocate &
Calibrate
BDSs
IESs
Pre-Es8Generator
Pre-ES8
To Convolution
To ERBE-Like
AttitudeData
Preprocessor
~133 Science & Diagnostic RDRswith Diary data
2 files needed to process(time span noon yesterday -
noon today)
14 files needed to process(time span hr 22 day before -
hr 00 next day)
Up to 3 files per day(Science, Calibration &
Diagnostic)
Up to 24 files per day(HDF4 Vdata containing
Geolocation valuesFiltered radiances)
Up to 6 files per dayBDS, BDSS, BDSD,
BDSM, BDSP & BDSG(HDF4 SDS & Vdatas containing
Geolocation valuesRaw data counts
Filtered radiancesInstrument H&S info)
1 file per day(Binary containing
Geolocated, Filtered radiances)
EphemerisData
AttitudeData
AttitudeData
AttitudeData
AttitudeData
CERES FM-5 Delta Design Review 28
Spectral ResponseFunction
ModtranData
UnfilteringCoefficients
Unfiltering
ERBE-like Spectral Corrections
No code changes needed to support NPP
CERES FM-5 Delta Design Review 29
DailyInversion
Pres8EID6
ES4
ES9
ES8
MonthlyAveraging
ERBE-like Inversion
From Instrument Subsystem
No code changes needed to support NPP
Unfiltering Coefficients
1 file per day(Binary containing
Geolocated, Filtered radiances)
1 file per day(HDF-EOS4 containing
GeolocatedFiltered & Unfiltered radiances &
fluxes)
1 file per day(Binary database
Info)
1 file per month(HDF4 containing
Regional, Zonal & GlobalGridded fluxes)
1 file per month(HDF4 containing
Global Average fluxes)
CERES FM-5 Delta Design Review 30
CERES DMT RDR Tests
• Objectives
– Ensure the RDR preprocessor is able to read the CERES Science and Diagnostic RDR HDF5 files and create the properly formatted Level-0, Ephemeris and Attitude data files that are compatible with the updated Ada and the converted C++ code.
• Description
– Assume ingest of RDRs by ASDC; Alternative is to access RDRs stored in specified location if not available from the ASDC
– Test with selected RDRs on the development platform to provide output that will be used by the Level-0 data processor to show that output Level-0, Ephemeris and Attitude data is as expected by this PGE.
– Test on the development platform with minimum of 28-hrs of RDR data through the Level-0 PGE, on to the Pre-ES8 generator to create the data used by ERBE-like.
CERES FM-5 Delta Design Review 31
CERES DMT RDR Tests
• Description (Cont’d)
– Pre-delivery testing on production platform to ensure all systems work on the production platform and generate scientifically equivalent data.
Ensures all software works as expected on the production platform before delivery
Provides expected output used by CM to verify their test results
– CM testing using Instrument Subsystem Test Plan to verify that all necessary updates have been delivered and provide output equivalent to the provided expected output.
– ASDC SIT operational testing: e.g. Dec. 31, Jan. 1, Feb. 28/29, etc.
• Participants
– Ashley Alford, Denise Cooper, Mark Timcoe, Land PEATE, LaRC ASDC SIT
CERES FM-5 Delta Design Review 32
CERES DMT RDR Tests
RDRs
Science & Diagnostic
~133 per data day(Level-0 packets &
Diary data)
RDRPre-processor
ScienceLevel-0 File
CalibrationLevel-0 File
DiagnosticLevel-0 File
Total 10391 packetsPer data day
EphemerisData Files
AttitudeData Files
Data files (Noon previous day -
Noon next day)12 Data files(2 hr data per file)
CERES FM-5 Delta Design Review 33
Instrument Schedule
• RDR Format Documentation Provided (Jun. 2008)
• Unpacking of RDR Wrapper Implemented (Jul. 2008)
• Conversion of RDR Contents to CERES Level 0 Format Implemented (Sep. 2008)
• Selection and Conversion of Attitude/Ephemeris Data Implemented (Sep. 2008)
• Informal Testing with Actual RDR (dependent on IDPS providing early RDR) (Oct. 2008)
• Instrument Subsystem Enhancements Integrated and Tested (Dec. 2008)
• Instrument Subsystem Verification Testing Complete (Jan. 2009)
• Delivery to ASDC through DMT CM for Formal Validation Tests (by Mar. 13, 2009)
CERES FM-5 Delta Design Review 34
Clouds Subsystem Design Approach
CERES Clouds Subsystem Design Approachand
CERES DMT VIIRS Tests
Sunny Sun-Mack
Contributing Team Members:Yan Chen
Walt Miller
CERES FM-5 Delta Design Review 35
CERES SSF Processing(Subsystem 4)
Subsystem 4 - Determine Cloud Properties, TOA and Surface Fluxes, is made up of 3 different sets of code
• Clouds - VIIRS inputs will necessitate changes to support NPP
– Processes the imager data– Resulting pixel-level output written to temporary file
• Convolution - minor changes needed to support NPP– Merges the pixel-level data with the CERES footprint data– Resulting footprint-level output written to a temporary file
• Inversion - no change needed to support NPP– Reads the footprint level output file containing CERES radiance data and cloud properties– Computes unfiltered radiances, TOA fluxes, and surface fluxes– Generates the SSF data product which is archived and publicly released
CERES FM-5 Delta Design Review 36
NPP Impact on CERES Clouds and Convolution Design Approach
• VIIRS will replace MODIS as imager source– Ability to generate certain cloud properties changes
– Some radiances, CO2 slicing, may be missing
• CERES IES (input from Instrument Subsystem) interface will remain the same
• New Point Spread Function (PSF) ancillary file will be generated to account for VIIRS imager resolution
• SSF interface change possible as CERES migrates to Edition3
CERES FM-5 Delta Design Review 37
Clouds Subsystem Design Approach
• Newly developed VIIRS sub-sampling code at LaRC– VIIRS Sub-sampling Preprocessor
Written in C language Input:
NPP_VIAE ( Radiances from Imaging “I” Channels )
NPP_VMAE ( Radiances and geolocations from
Moderate-Resolution “M” Channels)
NPP_VDNE ( Radiances and geolocations from Imaging
Day-Night Band (channel) “DNB” ) Output:
NPP_VIMD_SS ( Radiances and geolocations from all above
three Imaging, Moderate and DNB, but
sub-sampled and sub-channeled with
certain channels being averaged )
CERES FM-5 Delta Design Review 38
Clouds Subsystem Design Approach
• VIIRS sub-sampling code will run at the Land PEATE
– The package to be delivered to Land PEATE by
CERES CM at LaRC VIIRS sub-sampling code Process Control Generator to produce process control file (PCF) Input data and Expected sub-sampled output Test Plan (PDF)
– VIIRS sub-setting code will be run at Land PEATE The output from VIIRS sub-sampling code, NPP_VIMD_SS, will be
produced at Goddard Land PEATE Land PEATE will then deliver NPP_VIMD_SS to LaRC ASDC
• LaRC ASDC will ingest and archive NPP_VIMD_SS
• NPP_VIMD_SS will be the Cloud Imagery Data (CID) as input to
CERES Clouds Subsystem.
CERES FM-5 Delta Design Review 39
VIIRS Sub-setting Data Formats
1. Sub-setting NPP_VIAE
Resolution: 375 meter
Radiances from VIIRS Imaging Channels
Sub-setting Channels
I1 = 0.64 m
I2 = 0.865 m
I3 = 1.61 m
I4 = 3.74 m
I1 = 11.45 m
I1 = 0.64 m
I3 = 1.61 m
I4 = 3.74 m
I1 = 11.45 m
NPP_VIAE NPP_VIMD_SS
CERES FM-5 Delta Design Review 40
VIIRS Sub-setting Data Formats1. Sub-setting NPP_VIAE ( Cont’d)
M Pixels (NPP_VIMD_SS)I Pixels (NPP_VIAE)
Averaging 4 Imaging Pixels to 1 Moderate Resolution Pixel
CERES FM-5 Delta Design Review 41
VIIRS Sub-setting Data Formats
1. Sub-setting NPP_VIAE ( Cont’d)
Sub-setting Samples
NPP_VIAE
NPP_VIMD_SS
Scan-lines x Pixels = 4608 x 6400
Scan-lines x Pixels = 2304 x 3200
Every other pair of scan-lines
Every other pair of pixels
CERES FM-5 Delta Design Review 42
VIIRS Sub-setting Data Formats
SDS Name (NPP_VIAE) Description Dimensions SDS Name (NPP_VIMD_SS) Dimensions
Radiance_Img_I1 Radiances for I1 in moderate resolution (6144,6400) Radiance_Img_I1_Avg (1536,1600)
Radiance_Img_I3 Radiances for I3 in moderate resolution (6144,6400) Radiance_Img_I3_Avg (1536,1600)
Radiance_Img_I4 Radiances for I4 in moderate resolution (6144,6400) Radiance_Img_I4_Avg (1536,1600)
Radiance_Img_I5 Radiances for I5 in moderate resolution (6144,6400) Radiance_Img_I5_Avg (1536,1600)
Reflectance_Img_I1 Reflectance for I1 in imagery resolution (6144,6400) Reflectance_Img_I1_SubReflectance_Img_I1_Avg
(3072,3200) (1536,1600)
Reflectance_Img_I3 Reflectance for I3 in imagery resolution (6144,6400) Reflectance_Img_I3_SubReflectance_Img_I3_Avg
(3072,3200) (1536,1600)
BrightTemp_Img_I4 Brightness temperature for I4 at imagery resolution
(6144,6400) BrightTemp_Img_I4_Sub BrightTemp_Img_I4_Avg
(3072,3200) (1536,1600)
BrightTemp_Img_I5 Brightness temperature for I5 at imagery resolution
(6144,6400) BrightTemp_Img_I5_SubBrightTemp_Img_I5_Avg
(3072,3200) (1536,1600)
QF_VIIRS_I1_SDR_1 Quality control flag for I1 (6144,6400) QF_VIIRS_I1_SDR_1_Sub (3072,3200)
QF_VIIRS_I1_SDR_3 Scan quality control flag for I1 (192,4) QF_VIIRS_I1_SDR_3 (192,4)
QF_VIIRS_I3_SDR_1 Quality control flag for I3 (6144,6400) QF_VIIRS_I3_SDR_1_Sub (3072,3200)
QF_VIIRS_I3_SDR_3 Scan quality control flag for I3 (192,4) QF_VIIRS_I3_SDR_3 (192,4)
QF_VIIRS_I4_SDR_1 Quality control flag for I4 (6144,6400) QF_VIIRS_I4_SDR_1_Sub (3072,3200)
QF_VIIRS_I4_SDR_3 Scan quality control flag for I4 (192,4) QF_VIIRS_I4_SDR_3 (192,4)
QF_VIIRS_I5_SDR_1 Quality control flag for I5 6144,6400) QF_VIIRS_I5_SDR_1_Sub (3072,3200)
QF_VIIRS_I5_SDR_3 Scan quality control flag for I5 (192,4) QF_VIIRS_I5_SDR_3 (192,4)
1. Sub-setting NPP_VIAE ( Cont’d): SDSs Kept
CERES FM-5 Delta Design Review 43
VIIRS Sub-setting Data Formats
SDS Name (NPP_VIAE)
Description Dimensions SDS Name (NPP_VIMD_SS)
Dimensions
Radiance_Img_I2 Radiances for I2 in moderate resolution
(6144,6400)
Reflectance_Img_I2 Reflectance for I2 in imagery resolution
(6144,6400)
QF_VIIRS_I2_SDR_1 Quality control flag for I2 (6144,6400)
QF_VIIRS_I2_SDR_3 Scan Quality control flag for I2 (192,4)
1.Sub-setting NPP_VIAE ( Cont’d): The SDSs not being subset
CERES FM-5 Delta Design Review 44
VIIRS Sub-setting Data Formats
NPP_VMAE
M1 = 0.412 m M3 = 0.488 m
M2 = 0.445 m M4 = 0.555 m
M5 = 0.672 m M7 = 0.865 m
M6 = 0.746 m M8 = 1.24 m
M10 = 1.61 m M9 = 1.378 m
M12 = 3.7 m M11 = 2.25 m
M13 = 4.05 m M14 = 8.55 m
M15 = 10.763 m
M16 = 12.013 m
2. Sub-setting NPP_VMAE
Resolution: 750 meter
Radiances and Geolocation from VIIRS Moderate Channels
Sub-setting Channels
NPP_VIMD_SS
M3 = 0.488 m
M4 = 0.555 m
M7 = 0.865 m
M8 = 1.24 m
M9 = 1.378 m
M11 = 2.25 m
M14 = 8.55 m
M15 = 10 763 m
M16 = 12.013 m
CERES FM-5 Delta Design Review 45
VIIRS Sub-setting Data Formats
1. Sub-setting NPP_VMAE ( Cont’d)
Sub-setting Samples
NPP_VMAE
NPP_VIMD_SS
Scan-lines x Pixels = 3072 x 3200
Scan-lines x Pixels = 1536 x 1600
Every other scan-line
Every other pixel
CERES FM-5 Delta Design Review 46
VIIRS Sub-setting Data Formats
Latitude
Longitude
Solar Zenith Angle
Solar Azimuth Angle
Sensor Zenith Angle
Sensor Azimuth Angle
Satellite Range
Terrain Height
All Geolocation Info in NPP_VMAE is included in the Subset NPP_VIMD_SS
Scan Start Time
Scan Mid Time
NPP_VIMD_SS: SDS (64 bits float)
NPP_VMAE: Global Attributes
2. Sub-setting NPP_VMAE (Cont’d)
NPP_VMAE and NPP_VIMD_SS
CERES FM-5 Delta Design Review 47
VIIRS Sub-setting Data Formats
• 2. Sub-setting NPP_VMAE (Cont’d): SDSs Kept
SDS Name Description Dimensions SDS Name Dimensions
Radiance_Mod_M11 Radiances for M11 in moderate resolution (3072,3200) Radiance_Mod_M11_Sub (1536,1600)
Radiance_Mod_M14 Radiances for M14 in moderate resolution (3072,3200) Radiance_Mod_M14_Sub (1536,1600)
Radiance_Mod_M15 Radiances for M15 in moderate resolution (3072,3200) Radiance_Mod_M15_Sub (1536,1600)
Radiance_Mod_M16 Radiances for M16 in moderate resolution (3072,3200) Radiance_Mod_M16_Sub (1536,1600)
Radiance_Mod_M3 Radiances for M3 in moderate resolution (3072,3200) Radiance_Mod_M3_Sub (1536,1600)
Radiance_Mod_M4 Radiances for M4 in moderate resolution (3072,3200) Radiance_Mod_M4_Sub (1536,1600)
Radiance_Mod_M7 Radiances for M7 in moderate resolution (3072,3200) Radiance_Mod_M7_Sub (1536,1600)
Reflectance_Mod_M11 Reflectance for M11 in moderate resolution (3072,3200) Reflectance_Mod_M11_Sub (1536,1600)
Reflectance_Mod_M3 Reflectance for M3 in moderate resolution (3072,3200) Reflectance_Mod_M3_Sub (1536,1600)
Reflectance_Mod_M4 Reflectance for M4 in moderate resolution (3072,3200) Reflectance_Mod_M4_Sub (1536,1600)
Reflectance_Mod_M7 Reflectance for M7 in moderate resolution (3072,3200) Reflectance_Mod_M7_Sub (1536,1600)
Reflectance_Mod_M8 Reflectance for M8 in moderate resolution (3072,3200) Reflectance_Mod_M8_Sub (1536,1600)
Reflectance_Mod_M9 Reflectance for M9 in moderate resolution (3072,3200) Reflectance_Mod_M9_Sub (1536,1600)
BrightTemp_Mod_M14 Brightness temp. for M14 at mod resolution (3072,3200) BrightTemp_Mod_M14_Sub (1536,1600)
BrightTemp_Mod_M15 Brightness temp for M15 at mod resolution (3072,3200) BrightTemp_Mod_M15_Sub (1536,1600)
BrightTemp_Mod_M16 Brightness temp for M16 at mod resolution (3072,3200) BrightTemp_Mod_M16_Sub (1536,1600)
CERES FM-5 Delta Design Review 48
VIIRS Sub-setting Data Formats
SDS Name Description Dimensions SDS Name Dimensions
QF_VIIRS_GEO_MOD_2 Quality control flag for geo location (3072,3200) QF_VIIRS_GEO_MOD_2_Sub (1536,1600)
QF_VIIRS_M11_SDR_1 Quality control flag for M11 (3072,3200) QF_VIIRS_M11_SDR_1_Sub (1536,1600)
QF_VIIRS_M11_SDR_3 Scan quality control flag for M11 (192,4) QF_VIIRS_M11_SDR_3 (192,4)
QF_VIIRS_M14_SDR_1 Quality control flag for M14 (3072,3200) QF_VIIRS_M14_SDR_1_Sub (1536,1600)
QF_VIIRS_M14_SDR_3 Scan quality control flag for M14 (192,4) QF_VIIRS_M14_SDR_3 (192,4)
QF_VIIRS_M15_SDR_1 Quality control flag for M15 (3072,3200) QF_VIIRS_M15_SDR_1_Sub (1536,1600)
QF_VIIRS_M15_SDR_3 Scan quality control flag for M15 (192,4) QF_VIIRS_M15_SDR_3 (192,4)
QF_VIIRS_M16_SDR_1 Quality control flag for M16 (3072,3200) QF_VIIRS_M16_SDR_1_Sub (1536,1600)
QF_VIIRS_M16_SDR_3 Scan quality control flag for M16 (192,4) QF_VIIRS_M16_SDR_3 (192,4)
QF_VIIRS_M3_SDR_1 Quality control flag for M3 (3072,3200) QF_VIIRS_M3_SDR_1_Sub (1536,1600)
QF_VIIRS_M3_SDR_3 Scan quality control flag for M3 (192,4) QF_VIIRS_M3_SDR_3 (192,4)
QF_VIIRS_M4_SDR_1 Quality control flag for M4 (3072,3200) QF_VIIRS_M4_SDR_1_Sub (1536,1600)
QF_VIIRS_M4_SDR_3 Scan quality control flag for M4 (192,4) QF_VIIRS_M4_SDR_3 (192,4)
QF_VIIRS_M7_SDR_1 Quality control flag for M7 (3072,3200) QF_VIIRS_M7_SDR_1_Sub (1536,1600)
QF_VIIRS_M7_SDR_3 Scan quality control flag for M7 (192,4) QF_VIIRS_M7_SDR_3 (192,4)
QF_VIIRS_M8_SDR_1 Quality control flag for M8 (3072,3200) QF_VIIRS_M8_SDR_1_Sub (1536,1600)
QF_VIIRS_M8_SDR_3 Scan quality control flag for M8 (192,4) QF_VIIRS_M8_SDR_3 (192,4)
QF_VIIRS_M9_SDR_1 Quality control flag for M9 (3072,3200) QF_VIIRS_M9_SDR_1_Sub (1536,1600)
QF_VIIRS_M9_SDR_3 Scan quality control flag for M9 (192,4) QF_VIIRS_M9_SDR_3 (192,4)
2. Sub-setting NPP_VMAE (Cont’d): SDSs Kept
CERES FM-5 Delta Design Review 49
VIIRS Sub-setting Data Formats
2. Sub-setting NPP_VMAE ( Cont’d): The SDSs not being subset
SDS Name Description Dimensions SDS Name Dimensions
Radiance_Mod_M1 Radiances for M1 in moderate resolution (3072,3200)
Radiance_Mod_M10 Radiances for M10 in moderate resolution (3072,3200)
Radiance_Mod_M12 Radiances for M12 in moderate resolution (3072,3200)
Radiance_Mod_M13 Radiances for M13 in moderate resolution (3072,3200)
Radiance_Mod_M2 Radiances for M2 in moderate resolution (3072,3200)
Radiance_Mod_M5 Radiances for M5 in moderate resolution (3072,3200)
Radiance_Mod_M6 Radiances for M6 in moderate resolution (3072,3200)
Radiance_Mod_M8 Radiances for M11 in moderate resolution (3072,3200)
Radiance_Mod_M9 Radiances for M11 in moderate resolution (3072,3200)
Reflectance_Mod_M1 Reflectance for M1 in moderate resolution (3072,3200)
Reflectance_Mod_M10 Reflectance for M10 in moderate resolution (3072,3200)
Reflectance_Mod_M2 Reflectance for M2 in moderate resolution (3072,3200)
Reflectance_Mod_M5 Reflectance for M5 in moderate resolution (3072,3200)
Reflectance_Mod_M6 Reflectance for M6 in moderate resolution (3072,3200)
BrightTemp_Mod_M12 Brightness temperature for M12 at moderate resolution (3072,3200)
BrightTemp_Mod_M13 Brightness temperature for M12 at moderate resolution (3072,3200)
CERES FM-5 Delta Design Review 50
VIIRS Sub-setting Data Formats
2. Sub-setting NPP_VMAE ( Cont’d): The SDSs not being subset
SDS Name Description Dimensions SDS Name Dimensions
QF_VIIRS_M10_SDR_1 Quality control flag for M10 (3072,3200)
QF_VIIRS_M10_SDR_3 Scan quality control flag for M10 (192,4)
QF_VIIRS_M12_SDR_1 Quality control flag for M12 (3072,3200)
QF_VIIRS_M12_SDR_3 Scan quality control flag for M12 (192,4)
QF_VIIRS_M13_SDR_1 Quality control flag for M13 (3072,3200)
QF_VIIRS_M13_SDR_3 Scan quality control flag for M13 (192,4)
QF_VIIRS_M1_SDR_1 Quality control flag for M1 (3072,3200)
QF_VIIRS_M1_SDR_3 Scan quality control flag for M1 (192,4)
QF_VIIRS_M2_SDR_1 Quality control flag for M2 (3072,3200)
QF_VIIRS_M2_SDR_3 Scan quality control flag for M2 (192,4)
QF_VIIRS_M5_SDR_1 Quality control flag for M5 (3072,3200)
QF_VIIRS_M5_SDR_3 Scan quality control flag for M5 (192,4)
QF_VIIRS_M6_SDR_1 Quality control flag for M6 (3072,3200)
QF_VIIRS_M6_SDR_3 Scan quality control flag for M6 (192,4)
CERES FM-5 Delta Design Review 51
VIIRS Sub-setting Data Formats
DNB = 0.7 m
Latitude
Longitude
Solar Zenith Angle
Solar Azimuth Angle
Sensor Zenith Angle
Sensor Azimuth Angle
Satellite Range
Terrain Height
Subset every other scan-line and every other pixel
3. Sub-setting NPP_VDNE
Resolution: 750 meter
Radiances and Geolocation from VIIRS Day-Night Channel
Only One channel-->No Sub-setting Channel
Keep All Geolocation SDSs
Sub-setting Sample
CERES FM-5 Delta Design Review 52
VIIRS Sub-setting Data Formats
4. File Size Summary
INPUT:
NPP_VIAE ---> 460 MB / granule
NPP_VMAE ---> 560 MB / granule
NPP_VDNE ----> 330 MB / granule
OUTPUT
NPP_VIMD ---> 400 MB / granule
~ 70% Data Reduction
CERES FM-5 Delta Design Review 53
Emissivity Maps x 4
IGBP
Water %
Elevation
Surface Maps
NPP CERES Cloud Subsystem
GMAOMOA
Clear Reflectance
History
CloudsProcessor
TRMM VIRS CID
MODIS Terra CID
MODIS Aqua CID
NPP VIIRS CID
Cloud Imager Data
(CID)
Algorithm Ancillaries
Snow/ice Model
BiDir Model
Directional Model
AngularModels
EIPD (Cookie Dough)
EQCG
EQCB
ECVS
ECV
Output
NPP VIIRS Subset: NPP_VIMD_SS To CERES Convolution
CERES FM-5 Delta Design Review 54
Convolution Subsystem Description
Cloud Pixel
Convolution
Inversion
X 24 hoursInstrument
IES
FQCI
SSFAI
SSFIEIPDCookie dough
FQCPSF
Convolve imager radiances, cloud properties, and aerosol into CERES footprint
CERES FM-5 Delta Design Review 55
CERES DMT VIIRS Tests
• Objectives
Ensure the VIIRS Sub-sampler processor is able to:
(1) Read
NPP_VIAE ( Imager resolution radiances),
NPP_VMAE ( Moderate resolution radiances and geolocations)
NPP_VDNE (Day / Night band radiances and geolocations)
(2) Subset
sub-channel
sub-sample
averaging of imagery resolution data
(3) Produce VIIRS sub-sampled output
NPP_VIMD_SS
CERES FM-5 Delta Design Review 56
CERES DMT VIIRS Tests
• Description– If less than a couple of hours of NPP VIIRS data is available, then Land
PEATE pushes VIIRS NPP_VIAE, NPP_VMAE, and NPP_VDNE to a CERES/VIIRS designated computer
– For larger volumes of data, the Land PEATE sends all test NPP VIIRS data types to the ASDC for ingest, and then subsequently staged for access by the VIIRS Sub-sampler on a CERES/VIIRS computer
– Execute VIIRS Sub-sampler on local CERES/VIIRS computer to create and verify sub-sampled output using input VIIRS data from Land PEATE
– Load VIIRS Sub-sampler and selected VIIRS test data in the file space and computer designated for CERES processing at the Land PEATE, and then conduct testing at the Land PEATE and verify sub-sampled output
– CERES DMT delivers VIIRS Sub-sampler (tar files) and test plan to the CERES CM for delivery to the Land PEATE
– Land PEATE conducts VIIRS Sub-sampler testing creating sub-sample
– Land PEATE verifies sub-sample and also send sub-sampled output to either the ASDC or to a designated CERES/VIIRS computer for additional verification
CERES FM-5 Delta Design Review 57
CERES DMT VIIRS Tests
NPP VIIRS
Sub-setting
Processor
NPP_VIAE
NPP_VDNE
NPP_VMAE NPP_VIMD_SS
• Required Input • Expected Output
• Participants
– Sunny Sun-Mack, Yan Chen, Tammy Ayers, Walt Miller, CERES Science Team members: Patrick Minnis, Norman Loeb, Tom Charlock, Dave Kratz and Dave Doelling, Land PEATE, and LaRC ASDC
CERES FM-5 Delta Design Review 58
CERES CARS Delivery Processto the Land PEATE
• Following unit, integration and verification testing by the CERES DMT, the VIIRS Sub-sampler must be delivered to the Land PEATE for final testing
• CERES DMT creates tar files containing source code, process control generator, selected NPP VIIRS input data and expected sub-sampled output
• CERES DMT updates Test Plan for testing at the Land PEATE
• CERES DMT delivers the above tar files along with the Test Plan to CERES CM
• CERES CM delivers the tar files and Test Plan to the Land PEATE’s designated machine. A “Delivery Notification” email is sent to the Land PEATE and all concerned
CERES FM-5 Delta Design Review 59
CERES CARS Delivery Processto the Land PEATE (cont.)
• Land PEATE conducts VIIRS Sub-sampler testing
• When the Land PEATE testing is successfully completed, the Land PEATE:– Sends an email to all concerned regarding the status
– Promotes the delivery to their production environment
– Makes the delivery (as tar files) available in a designated area for CERES CM to retrieve
– Sends a notification email to CERES CM that this action has taken place.
• Upon receiving this notification, CERES CM: – Retrieves the tarred delivery
– Places it in the CERES CM repository
• Subsequent changes to CM controlled software are documented and tracked using a CERES Software Configuration Change Request (SCCR) and redelivered through CERES CM
CERES FM-5 Delta Design Review 60
NPP VIIRS Sub-sampler Schedule
• Data Format of VIIRS Subset for CERES Defined (June 1, 2008) Completed on Time.
• Data Product Catalog Pages Documenting VIIRS Subset for CERES
Defined (August 1, 2008) Completed on Time
• Sub-sampler Code Complete and Running Locally at LaRC on Linux
Machine (October 1, 2008)
• Goddard Land PEATE Accepts the Sub-sampler Code Delivery and
Produces Sub-sampled Output from Input Data (December 1, 2008)
• Verify a Few Days of Sub-sampled Data Produced at GSFC (January
15, 2009)
• ASDC Ingests Sub-sampled VIIRS Data; Verified by CERES DMT
(February 26, 2009)
CERES FM-5 Delta Design Review 61
CERES DMT Testing Approach
• The CERES DM task was successfully appraised at CMMI Capability Level 3 /Equivalent Maturity Level 2
• SSAI is appraised at CMMI Capability Level 3/Equivalent Maturity Level 2
• CERES DMT uses a structured CMMI-based development process that includes unit and integration testing
• Functional (e.g., verification) tests are executed to verify requirements for all enhancements
• Regression tests are executed to verify existing functionality
• Delivery packages for CERES and VIIRS Sub-sampler to include:
– Test Plan
– Expected output
– All inputs
– Code and scripts
• Every CERES code delivery (to ASDC or Land PEATE) is under configuration management (CM) control
• Documented in CERES NPP DMS Software Management Plan Version 0
CERES FM-5 Delta Design Review 62
CERES CARS Testing Strategy
• Conversion of RDRs to Level 0 data will be tested by the DMT using defined formats and available test RDRs prior to delivery of code to the ASDC
– Testing conducted in development environment
• Transmission of RDRs by the Land PEATE to the ASDC will be tested when:
– CERES RDRs are available to the Land PEATE from the SD3E
– Interface between ASDC and the Land PEATE has been established
• Final testing of conversion of RDRs to Level 0 data will involve:
– Testing in the CERES operations test environment
– Land PEATE sending all three RDR data types to the ASDC
– ASDC ingesting data and acknowledging receipt to the Land PEATE
– Instrument subsystem reading RDRs staged from ASDC archive and correctly converting to Level 0 format
CERES FM-5 Delta Design Review 63
CERES CARS Testing Strategy
• VIIRS Sub-sampler will be tested by the DMT using test data provided by the Land PEATE prior to code delivery to the Land PEATE
– Testing conducted in development environment
– Testing also conducted on designated Land PEATE computer
• Transmission of sub-samples by the Land PEATE will be tested when:
– VIIRS Sub-sampler has been tested and delivered to the Land PEATE
– VIIRS data is available to the Land PEATE from the SD3E or other source
– Interface between ASDC and the Land PEATE has been established
• Final testing of Sub-sampler will involve:
– Execution of Sub-sampler by the Land PEATE to produce sub-sampled data product
– Transmission of the sub-sampled data to the ASDC for ingest and archive
– CERES DMT stages data from ASDC archive and verifies sub-sample
CERES FM-5 Delta Design Review 64
DMT NPP Schedule
10/1/08: VIIRS sub-sampling software complete and tested at LaRC
12/1/08: VIIRS sub-sampling software running at SDS Land PEATE, expected sub-sampled output has been verified by CERES DMT
1/15/09: Large set of VIIRS sub-sampled output produced at SDS Land PEATE and verified at LaRC by CERES DMT
3/13/09: Instrument subsystem code needed to support NCT3 complete and tested by DMT
5/15/09: Instrument subsystem code delivered to ASDC, tested, and promoted to operations
08/13/09: Instrument subsystem code needed to support NCT4 complete and tested by DMT Interim delivery to correct NCT3 problems and prepare for NCT4
10/15/09: Instrument subsystem code delivered to ASDC, tested, and promoted to operations
11/19/09: Instrument subsystem code needed to support launch complete and tested by DMT
1/15/10: Instrument subsystem code delivered to ASDC, tested, and promoted to operations
CERES FM-5 Delta Design Review 65
CERES CARS Build Schedule
CERES Data Management System
Projected Build promotion dates to ASDC Operations:
Build 1: NCT3 Functionality - May 2009
Build 2: NCT4 Functionality - October 2009
Build 3: Launch-Ready System - January 2010
CERES FM-5 Delta Design Review 66
ASDC Implementation Approach
ASDC Implementation Approach
Michael Little
CERES FM-5 Delta Design Review 67
ASDC Supports CERES PI in Ingest, Archive, Distribution, Production
• Ingesting and Archiving inputs for all Data Products– Use by Production in Producing Data Products
– Use by Science Team in analysis and algorithm refinement
– Long-term stewardship of data including migration across media changes Moving from Tape archive with Disk Cache to Disk archive with Tape Backup
• Distribute CERES Data Products– External Customers order data products via ASDC Order Tool, ESDIS ECHO
ASDC Order Tool provides interim support ESDIS ECHO provides access to all EOS data products Subscription Services Plan to supply CLASS with subscription service as an offsite backup
– Internal users via file sharing Moving from Request/deliver model to online Fiberchannel/NFS
• Produce Data Products for CERES PI– Accept and test Production codes from DMT
– Accept Production Requests from DMT
– Run codes consistently to produce expected data products Now by procedure; expect automation within one year
CERES FM-5 Delta Design Review 68
ASDC Evolution Approach
• Provide CERES Production Services at ASDC
– Meet constraints of CERES PGEs Avoid increasing workload on DM/Science workforce
Support FORTRAN, Ada, and IDL languages
Meet increasing memory and disk access requirements
– Use lessons learned to improve CERES production Improve I/O bandwidth limitations to allow multiple instances of some subsystems
Reduce unplanned outages by increasing storage reliability
Reduce dependence on active tape archive to improve labor intensive staging processes
– Increase production capacity for FM-5 and reprocessing Terra/Aqua
– Replace current hardware at end-of-life
• Provide Access to CERES Data Products for Internal Users
– Production
– LaRC Science Community
– CERES Data Management Team
• Improve Order Throughput for External Customers
CERES FM-5 Delta Design Review 69
64 x86 SMP256 GB
ASDC & SCF Integrated ArchitectureFiberchannel Connectivity
Fiberchannel Switch
64 x86 SMP256 GB
56 P6256GB
56 P6256GB
56 P6256GB
56 P6256GB
220TBDS4800
220TBDS4800
220TBDS4800
220TBDS4800
220TBDS4800
64 x86 SMP128 GB
16 x86 HS1264 GB
CommonSANhead
(NFS)(AFP)
(Samba)
ASDC JBOSSServer & TS-1& Ingest
ASDC SDP Production
64 x86 SMP256 GB
56 P6256GB
SCF Cluster(Access through
Sun Grid Engine)
220TBDS4800
220TBDS4800
SCF Disk Array
Inst
rum
ent
Inst
rum
ent
ER
BE
-lik
e
SRB
/Pow
erSR
B/P
ower
SAR
B
14 4P6@8GB
SCF Interactive Processors
ASDC Disk Archive(R/W Production)
(R/O SCF)
Con
sodi
ne
TIS
A
SOFA
Clo
uds
Clo
uds
TIS
A
220TBDS4800
WW
WW
WW
64 x86 SMP128 GB
Cloud ModelingCluster (Xu)
64 x86 SMP128 GB
ECS DUEProcessing
DbServer
DbServer
Tesla
Tesla
3755
3755
GPU Experiment (YongHu)
SimServer
FCMonitor
LB1 LB1
CERES FM-5 Delta Design Review 70
220TBDS4800
ASDC & SCF TCP/IP Architecture
LaRCNet Switch
220TBDS4800
220TBDS4800
220TBDS4800
220TBDS4800
4x16 x86 SMP128 GB
SH1x3650
Common SANhead(NFS, AFP, Samba)
ASDC JBOSS Server & TS-1
ASD
C S
DP
Prod
uctio
n (S
GE
)
SCF Cluster (SGE)
220TBDS4800
220TBDS4800
SCF Disk Array (R/W)In
stru
men
tIn
stru
men
tE
RB
E-l
ike
SRB
/Pow
erSR
B/P
ower
SAR
B
14 4P6@8GBSCF Interactive Processors
ASDC Disk Archive (R/W Production)(R/O SCF)
SAR
B
TIS
A
SOFA
Clo
uds
Clo
uds
TIS
A
220TBDS4800
WW
WW
WW
64 x86 SMP128 GB
CloudModeling
Cluster
64 x86 SMP128 GB
ECS DUE Processing
ANGeDb
Servers
ANGeDb
Servers
Tesla
Tesla
3755
3755
GPU Experiment
SimServer
Load2Magneto
56 P6 256GB
56 P6 256GB
56 P6 256GB
56 P6 256GB
SH2x3650
192.168.16.x
FCMonitor
ASDC SGE Head
SCF SGE HeadLoad
1
LaRCNet 100Mbps EthernetPrivate Net (DS Mgt)Private Net (GPFS)Cluster & Mgt Net (SGE)Green Indicates SubSys Function
Legend
WWW ProxyServers
Cluster Mon(HW TBD)
64 x86 SMP256 GB
64 x86 SMP256 GB
64 x86 SMP256 GB
56 P6256GB
Zamboanga
Center Firewall
NISN WAN
IBM Directorx3650
CERES FM-5 Delta Design Review 71
I/F
To beResolved
Metadata
For OrderableData Products
ASDC Ingest Processes for NPP
ANGeDrop-Box
ASDCArchive
ANGeCore db
IMSdb
AS
DC
In
gest
Pro
cess
or
Metadata
Location
& GUID
Subscription
Files
PDR
Files
ASDCDeliveryServer
Files
LocalProduction
PDRFiles
LandPEATE
PDRFiles
ECHOMetadataAdapter
NOAACLASS
CERES FM-5 Delta Design Review 72
NPP Impacts to the ASDC
Impact ResolutionIncreased storage capacity and performance needed for new CERES data set
Additional disks purchased with fiberchannel connectivity to production processors.
Higher density tapes expand capacity of tape archives.
Increased processing capability needed for new CERES data set
Additional processors purchased and added to Linux cluster increasing processing capacity.
Faster data access from new disk archive vs. tape archive.
Additional CERES data production requirements effect on staffing
Improvements in staging/ingest/archive reliability will offset staff workload for production.
Staff experienced in processing CERES data products.
Operating procedures required for new data set and experience of staff
Job production using on existing Subsystem DMT Ops Manuals requiring minor updates for NPP.
Staff experienced in processing CERES data products.
Ability of facility to accommodate additional hardware with respect to space, HVAC and power
ASDC has space/HVAC for expansion.
Power upgrades 09/08/08
Increased data load impact on network throughput
New architecture will improve internal data production network throughput and access by SCF users
May need additional bandwidth to Land PEATE for VIIRS
User access to current and future CERES data products
New architecture will speed ordering CERES data products
New data products will be available through current ordering tools when released to public
CERES FM-5 Delta Design Review 73
Ingest/Archive Sizing Estimates
ASDC can accommodate expected CERES data volumes from the Land PEATE
CERES RDRs: Science, Diagnostic, and Telemetry
File sizes are 1370.8 KB, 685.5 KB and 27.5 KB; excluding HDF overhead, about 700 KB/file
Estimate: (1370.8 + 700) KB/file * 131 files/day + 685.5 KB/file * 1 file/day+ 27.5 KB/file * 12 files/day
= 272.2903 MB/day
Sub-sampled VIIRS radiance and geolocation data:
Output File Size: Approximately 400 MB per 5-min file
Estimate: 400 MB/file * 288 files/day = 115,200.0 MB/day
VIIRS Aerosol data:
File Size: ~ 632 MB per 5-min file, as of June 1, 2008
57 MB per 5-min file if changing to 6 x 6 aggregation for both Land and Ocean 32 MB per 5-min file if changing to 8 x 8 aggregation for both Land and Ocean.
Estimate: ~12 files per hour or ~288 files/day * 632 MB/file = 182,016 MB/day
~12 files per hour or ~288 files/day * 57 MB/file = 16,416 MB/day
Note: The Land PEATE expects to switch to 6x6, or possibly 8x8, aggregation prior to launch
Total Volume - Large Aerosol Files = 297,488.29 GB/day 108,583.226 GB/yr or 108.58 TB/yr
Total Volume - Small Aerosol Files = 131.888.29 GB/day 48,139.226 GB/yr or 48.14 TB/yr
CERES FM-5 Delta Design Review 74
Security Compliance
• ASDC Approval to Operate Based on August, 2007 Review
– System Security Plan: SC-010-M-LRC-1000 Approving Official: Steve Jurczyk, Deputy Center Director
System Owner: John Kusterer, ASDC Manager
System Security Manager: Michael Little, ASDC Systems Engineer
– Re-certification in progress based on CIO mandated schedule re-alignment
– Re-certification based on ANGe/IBM installation to occur next spring Include consolidation of SCF and ASDC Systems into single System
• IT Security Issues
– Reconciliation of OCIO Mandated ODIN conversion with ASDC ITS needs ODIN to take over support of workstations, desktops, etc
ASDC to retain control/responsibility for servers
– Implementation of unannounced ITS measures by OCIO sometimes interferes with ingest and delivery over network
Agency CIO mandated transfer of all Center Firewalls to NISN is not fully understood
CERES FM-5 Delta Design Review 75
CERES Production Code Test & Evaluation
• Proven Processes Based on 10 Years of Experience in CERES Code Delivery,Acceptance, Testing, Evaluation and Production
• CERES DMT Develops Code for Production of Data Products– PGE(s) for each Sub-system (SS) tested and internally verified
– DMT and ASDC develop test cases
– Delivery to ASDC includes sample data, Operators Manual, code
– Delivery Evaluation and Code Compilation by DMT Configuration Management comparing to sample data products and using test cases
– Delivery to ASDC for SS Integration and Test to evaluate producibility and completeness and comparison to sample data products
– ASDC and DMT work together to identify and resolve any problems through Software Trouble Tickets
– Promotion to Production status and ASDC CM when approved by Operations Readiness Review (ORR)
• Data Product Production Validation Testing– Once code is in production, sample data products (ValRx) are produced with appropriate key
months
– ValRx data products are evaluated by Science Team and approved prior to release into an orderable status
Unsuitable results may result in re-delivery
– Release also requires a Quality Summary by the SS Science Lead
CERES FM-5 Delta Design Review 76
ASDC Validation Testing Approach
Systems delivered to the ASDC undergo Validation testing in a controlled environment
DMT Subsystems
under CERES CM delivered to the ASDC
for Validation Testing
Delivered Subsystems
inspected for completeness
by ASDC CM, unique testing
scenarios identified
Subsystems validated in operations
testing environment for
all new requirements
and enhancements
Delivered subsystems regression tested for existing
functionality
Operations Readiness
Review (ORR) following successful Validation
Testing and/or mitigation of
existing issues
Problem reports generated and tracked for all identified
problems
Problem report resolutions incorporated into
configured software for redelivery (if needed) to the
ASDC for validation
DataProduction
andOperations
CERES CM ASDC CM ASDCValidation
Testing
ASDC CCBAnd ProjectManagement
ASDCRegression
Testing
ASDCOperations
CERES FM-5 Testing to Operations Life-Cycle
Problems Resolutions
CERES FM-5 Delta Design Review 77
NPP-Level Testing and Status
• Testing of the RDRs requires the SD3E to access data from the IDPS, the RDRs are then accessed by the Land PEATE and sent to ASDC
– ASDC will ingest the RDR data and acknowledge receipt
– ASDC will test data production when available after 05/09
• Testing of VIIRS data requires Land PEATE to run sub-sampler and send sub-sampled radiance/geolocation and aerosol data to the ASDC
– Land PEATE must be able to generate or access valid VIIRS data
– ASDC will ingest data and acknowledge receipt
• Ingest capability will be ready to support Early EEO
• Ingest and data production capability will be ready to support NCT3
• Interim delivery to address NCT3 problems to support NCT4 success
• All NPP-related functionality available at L-90 days for Launch
CERES FM-5 Delta Design Review 78
CERES ASDC RDR Tests
• Objective– Verify that the Land PEATE can access CERES RDRs from the SD3E and
successfully transfer the data to the ASDC for ingest and archive
• Description– SD3E reads CERES RDR data from IDPS or through alternative source– Land PEATE accesses RDRs from the SD3E and transfers data to the ASDC– ASDC ingests RDRs and acknowledges receipt back to Land PEATE– ASDC archives data for staging and processing by Instrument Subsystem– ASDC will test data production when systems and RDRs available after 05/09– Additional reorder and data management functions will also be tested
• Input– CERES RDRs generated by IDPS or from Proxy data provided to Land
PEATE
• Output– CERES RDRs ingested and archived at ASDC– CERES Level 0 data for instrument subsystem processing available after 05/09
• Participants– ASDC, Land PEATE, SD3E, CERES DMT– IDPS when CERES RDR capability is available
CERES FM-5 Delta Design Review 79
CERES ASDC VIIRS Tests
• Objective– Verify execution of the VIIRS Sub-sampler by the Land PEATE and the
transfer of the sub-sampled radiance/geolocation and aerosol data to the ASDC• Description
– SD3E reads VIIRS data from IDPS or through alternative source– Land PEATE must be able to generate or access valid VIIRS data– Land PEATE executes VIIRS Sub-sampler and sends sub-sample to ASDC– ASDC ingests sub-sampled data and acknowledges receipt to the Land PEATE– ASDC archives data for staging and processing by Clouds subsystem – Sub-sampled and aerosol data examined by science and DMT staff– ASDC will test data production when systems are available after 05/09– Aerosol data ingest and archive tested when available from the Land PEATE
• Input– VIIRS data generated by IDPS or from Proxy data provided by Land PEATE
• Output– VIIRS sub-samples generated by the Land PEATE – VIIRS aerosol data product generated by the Land PEATE
• Participants– ASDC, Land PEATE, SD3E, CERES DMT, IDPS
CERES FM-5 Delta Design Review 80
Operational Support
• System Upgrades will Enhance Current Capabilities– CERES production system already runs at the ASDC
– ANGe storage management upgrade should be operational this week
– Hardware upgrades to be phased in with existing CERES Ingest, Archive, Distribution and Production capabilities in Dec09, reducing risk to NPP needs
• Procedures and Documentation are available and tested– Operations staff with CERES processing experience already in place with defined,
proven procedures
– ASDC staff already work closely with Land PEATE staff through interactions supporting MODIS data transfers provided from the same data center (MODAPS)
• System availability will remain at 24x7– Current production model has minimized need for 24x7 staffing
– System hardware and architecture evolution will increase system reliability and reduce workforce requirements
• Improved system performance and additional CERES data products will lead to increased customer satisfaction
CERES FM-5 Delta Design Review 81
Documentation and Agreements
• Most documentation and agreements already exist as part of CERES on TRMM, Terra, and Aqua
– http://science.larc.nasa.gov/ceres/docs.html
– Documents may be updated if needed
– New documents may be written explicitly for NPP, to meet special requirements
• Overall roles and responsibilities are documented in CERES Science and Data Products Working Agreement
– Working agreement between CERES IT and NPP Project Office
– May need to be revisited when NPP transfers to NOAA leadership
– Working Agreement in hands of NPP SEWG
• Operations Agreement (OA) being developed between ASDC and Land PEATE
– Based on existing MODAPS OA for MODIS data sent to ASDC
– On-going relationship between MODAPS and ASDC for MODIS
• No OA planned for ADS/CLASS interface– Expect to use standard customer subscription agreement
CERES FM-5 Delta Design Review 82
ASDC Supports NPP Schedule
08/31/08 Initial Operational Capability (IOC): CERES in ANGe09/15/08 Complete Installation/Configuration of IBM hardwareSep-Nov08 Test/Characterize IBM Installation11/13/08 Initial Delivery of CERES RDR to ASDC IngestNov08 Complete LaTIS conversion into ANGeNov/Dec08 Design/Develop/Test NPP Inst, VIIRS into ANGe11/26-12/23/08 Complete CERES RDR and VIIRS ICD12/01/08 IOC IBM ASDC+SCF Archive and Processing SystemJan/Feb09 Debug Aerosols/VIIRS Sub-sample into ANGe with Land
PEATE05/08/09 SDS Test2: Instrument RDRs into ANGe from Land PEATE05/15/09 Build 1 ORR: NCT3 Functionality06/09-15/09 NCT307/01/09-02/02/10 Design/Develop/Test Other NPP into ANGe10/15/09 Build 2 ORR: NCT4 Functionality/NCT3 Problem
Corrections12/02-09/09 SDS Test3/NCT401/15/10 Build 3 ORR: Launch-Ready System02/02/10 Instrument/VIIRS IOC03/02/10 MRR06/02/10 NPP Launch
CERES FM-5 Delta Design Review 83
Overall Schedule
• CERES Schedule is baselined
• CERES Schedule incorporated into NPP Schedule
• Schedule available as separate handout
CERES FM-5 Delta Design Review 84
Resource Assessment
• Required resources have already been assessed and hired
• Current experienced CERES staff leveraged for CERES FM-5
• CERES FM-5 staff augmented with key hires to address technical challenges
– Major SSAI job fair resulted in hundreds of resumes
• Backfill positions created in current CERES support filled with new talent
• Resource estimate based on approximate lines of code to be generated
• Using industry productivity standards, CERES CARS has the resources in place needed to do the job
• Additional resources needed for NPP testing, processes, and working groups not included in industry standards
• Additional issues and requirements expected
• Ongoing CERES projects supported by SSAI can provide short-term support if critical needs arise
CERES FM-5 Delta Design Review 85
Risk Analysis
Raw Data Record (RDR)
Formats NotFully
Defined
Given that CERES Raw Data Record (RDR) formats are not fully
defined or understood in time, there is a possibility the code needed to support testing may
not be complete.
2 1 TechnicalSchedule
Erika Geier
M 05/29/08
The CERES teamis developing agiver/receiver listand this item is on
the list.
07/23/08
IPO/Raytheon andJanet Smith allKnow about this
risk.
Unavailability of CERES Raw Data
Record (RDR) Test
Data
Given that CERES Raw Data Record (RDR) test data from IDPS is not available prior to testing, there is a possibility the code needed to support testing may
not be complete.
3 1 TechnicalSchedule
ErikaGeier
W 07/23/08
We have requestedtest data; although,it doesn't soundlike we'll getmeaningful test
data.
Title Description L C Impact Owner Action Comments
Legend: L – Likelihood C – Consequence M – Mitigate W - Watch
CERES FM-5 Delta Design Review 86
CERES Issues/Concerns
• Need single sample of the Science, Diagnostic, Housekeeping/Telemetry RDRs to verify that we understand format no later than 10/15/08
– CERES data content may be proxy, sample, or fill data
– Prefer attitude/ephemeris not be fill data
• Desire 3 days (~ 28 consecutive hours) of CERES RDRs to run through preprocessor and Instrument Subsystem to aid in code development no later than 11/14/08