Aquarius Data Processing System (ADPS) and Data …Aquarius/SAC-D Science Team Meeting July 19, 2010 Seattle, wA Aquarius Data Processing System (ADPS) and Data Distribution Fred Patt

Aquarius/SAC-D Science Team Meeting

July 19, 2010 Seattle, wA

Aquarius Data Processing System (ADPS)

and Data Distribution

Fred Patt

2 of 29 July 19-21, 2010

ADPS and Data Distribution Aquarius/SAC-D Science Team Meeting

•  Overview •  Science Data Processing •  Level-2 Science Software Implementation •  Level-2 Data Product Format •  Data Archive and Distribution •  Aquarius Web Site •  What We Need

Agenda

3 of 29 July 19-21, 2010


Aquarius Data Processing System Overview

•  The Aquarius Ground System at NASA/GSFC consists of the Aquarius Data Processing System (ADPS) and the Aquarius Command and Control System.

•  The ADPS will support: –  Acquisition and ingest of Aquarius/SAC-D data from CONAE. –  Acquisition of dynamic ancillary required to support the science processing. –  Processing of Aquarius data to standard product levels using algorithms provided

by the Science Team. –  Quality control of data products using methods specified by the Science Team. –  Match-up of Aquarius data with in situ salinity measurements. –  Archive and distribution of all data products during the active life of the mission to

the science team and general science community. –  Delivery of source data (Level-1a) and “validated” Level-2 and Level-3 derived

data products to the PO.DAAC.

•  The ADPS will be supported within the existing infrastructure of the Ocean Biology Processing Group (OBPG).

4 of 29 July 19-21, 2010


•  Overview and Status •  Science Data Processing •  Level-2 Science Software Implementation •  Level-2 Data Product Format •  Data Archive and Distribution •  Aquarius Web Site •  What We Need

Agenda

5 of 29 July 19-21, 2010


Aquarius Data Processing Overview

Level-0 Preprocessor

Level-0 to 1A

Level-1A to 2

Level-1A Merge

Raw Level-0 Downlink

File

Level-1A orbit

Level-2

Level-3 Map

SAC-D HKT Data

Level-2 to 3 Binning

Level-2 to 3 Smoothing

Level-3 Binning

Level-3 Mapping

Dynamic Ancillary

Data

Static Ancillary

Data

Level-1A temp

Level-0

Level-3 Bin

Level-0 Report

Level-1a Report

Automated Telemetry Analysis Tool

Standard Data Processing Web-Based Telemetry Monitoring Tools

Telemetry Report

Predicted Downlink

Times

In-Situ Match-up Process

Level-2 Match-up AVDS Bias Adjustment

Web-Based Browse, Download and Order Tools

CONAE-provided Interactive Telemetry

Analysis Tool

SAC-D HKT Data

6 of 29 July 19-21, 2010


Science Data Processing The Aquarius science data processing consists of the following steps: •  Level-0 Preprocessor – process each HRD downlink file to produce a single time-ordered set

of Aquarius science blocks and removes duplicates– developed by Joel Gales - ADPS. •  Level-0-to-1A – separate Level-0 files into orbits, unpack science data, incorporate overlapping

ephemeris and SAC-D HKT and convert Aquarius HKT – developed by Joel Gales - ADPS. •  Telemetry Analysis – HKT data from Level-1A files analyzed for web display on the Aquarius

web site – developed by Susie Kennison, Norman Kuring and Liang Hong - ADPS/ACCS. •  Interactive Telemetry Analysis - HKT files analyze using CONAE-provided telemetry viewer. •  Level-1A Merge – consolidate Level-1A files from overlapping downlinks into a single, best-

quality full orbit product – Joel Gales - ADPS. •  Level-1A-to-2 – perform calibration, atmospheric correction and salinity retrieval for Aquarius

science data – radiometer and scatterometer code delivered by RSS and JPL, respectively; integration and input/output code developed by Joel Gales and Liang Hong - ADPS.

•  Level-2-to-3 Binning – Level-2 salinity retrievals for one day geographically projected and collected into equal-area bins – existing OBPG software modified by Joel Gales- ADPS.

•  Level-2-to-3 Smoothing – Level-2 salinity retrievals for one day optimally interpolated into equal-area bins – algorithm by Jonathan Lilly/ESR implemented by Joel Gales - ADPS.

•  Level-3 Binning and Mapping – Level-3 binned files aggregated to longer time periods (weekly, monthly, etc.) and projected onto map grid – existing OBPG software.

All processing steps are run automatically when all input data sets are available. All software exists in at least preliminary prelaunch form at this time.

7 of 29 July 19-21, 2010


•  Radiometer and Scatterometer 2009 code deliveries have been integrated and installed in the data processing system.

•  Latest versions of Level-1A and Level-3 software have been installed. •  Ancillary data to support processing have been acquired and ingested. •  The one-year (2007) simulated data set has been processed to Level-1A,

Level-2 and Level-3 products: –  http://oceancolor.gsfc.nasa.gov/cgi/aquarius (Level1A/2 browser) –  http://oceancolor.gsfc.nasa.gov/cgi/l3 (Level-3 browser) –  http://oceandata.sci.gsfc.nasa.gov/Aquarius (FTP-like interface)

•  The PO.DAAC is downloading the simulated data products and making them available (see poster by Gregg Foti).

•  AVDS processing is in progress with the 2007 data set.

Science Code Status

8 of 29 July 19-21, 2010


•  From now until launch we need to:

–  Integrate Radiometer and Scatterometer code updates into data processing system.

–  Incorporate additional quality flags into software and products. –  Develop ingest procedures for additional ancillary data required by

new code. –  Incorporate SAC-D HKT format into Level-1A processing. –  Verify output products with Science Team. –  Begin real-time mission simulation (August 2010 --> Launch). –  Incorporate additional prelaunch updates into the science software

(as needed). –  Participate in mission tests as scheduled.

Science Code TBD

9 of 29 July 19-21, 2010


•  Overview and Status •  Science Data Processing •  Level-2 Science Software Implementation •  Level-2 Data Product Format (separate presentation) •  Data Archive and Distribution •  Aquarius Web Site •  What We Need

Agenda

10 of 29 July 19-21, 2010


Level-2 Software Implementation

Implemented using software delivered by RSS and JPL with I/O by Joel Gales

Capabilities: •  Processes a Level-1A input product. •  Performs radiometer Level-1B calibration and Level-2 processing using

latest code from RSS. •  Performs scatterometer Level-1B calibration and Level-2 processing using

latest code from JPL. •  Reads SST, wind, atmospheric, sea ice and solar flux data from dynamic

ancillary files. •  Computes geolocation. •  Generates HDF5 output product.

11 of 29 July 19-21, 2010


Level 2 Processing Flow

Look-up Tables

Level-1A Product

L2gen_aquarius

Level-2 Product

QMET Ancillary

Files

atm_tb_aquarius

L2gen_scatter

Radiometer Ancillary

Files

Scatterometer ASCII Files

mk_aquarius_ ancillary_data

Solar Flux Ancillary

Data

SST Ancillary

Files

Static Ancillary

Files

ATM Ancillary

Files

Sea Ice Ancillary

Files

Look-up Tables

12 of 29 July 19-21, 2010


Dynamic Ancillary Data

Data Type Source Interface Geopotential height, surface Temperature at surface Atmospheric pressure at surface Relative humidity, entire atmosphere Precipitable water Cloud water U/V wind 10 m Soil moisture Geopotential height (profile) Temperature (profile) Relative humidity (profile) Cloud water mixing ratio (profile)

NCEP Primary: tads1u.ecs.nasa.gov Secondary: dss.ucar.edu

Sea Surface Salinity (model) FSU/JPL Primary: ourocean.jpl.nasa.gov/SSS/

Sea Surface Temperature Reynolds (NCEP) Primary: eclipse.ncdc.noaa.gov

Significant Wave Height NCEP Primary: polar.ncep.noaa.gov/pub/waves

Solar Flux and Flares at 1.4 GHz USAF (NGDC) Primary: www.swpc.noaa.gov/ ftpdir/latest/dayind.txt -- production

Sea Ice Concentration/ Extent/Age NSIDC

Primary: polar.ncep.noaa.gov/pub/cdas/archive -- for 2007 simulation polar.ncep.noaa.gov/pub/cdas/ -- for production

13 of 29 July 19-21, 2010



Agenda

14 of 29 July 19-21, 2010


Aquarius Level-2 Product Content

•  This will actually be a combined Level 1B / Level 2, radiometer/scatterometer product.

•  The current product specification incorporates updated parameter lists based on the recent code deliveries from the radiometer and scatterometer teams.

•  Specific parameters to be output are configurable at runtime. •  Parameters are output per beam and block (1.44 sec) •  All output products include standard metadata, flags and

geolocation fields.

15 of 29 July 19-21, 2010


Radiometer Parameters (1 of 2)

Name Long Name Implementation rad_TaH Radiometer Ta H polarization Current rad_TaV Radiometer Ta V polarization Current rad_Ta3 Radiometer Ta 3rd Stokes Current

rad_toi_H Radiometer Top of the Ionosphere Tb H polarization Current

rad_toi_V Radiometer Top of the Ionosphere Tb V polarization Current

rad_toi_3 Radiometer Top of the Ionosphere Tb +45 polarization Current

rad_toa_H Radiometer Top of the Atmosphere Tb H polarization Current

rad_toa_V Radiometer Top of the Atmosphere Tb V polarization Current

rad_far_TaH Radiometer Faraday Angle Current rad_TbH Earth surface Tb H polarization Current rad_TbV Earth surface Tb V polarization Current SSS Sea Surface Salinity Current

16 of 29 July 19-21, 2010


Radiometer Parameters (2 of 2)

Name Long Name Implementation rad_galact_Ta_dir_V Radiometer galactic direct correction V polarization Current rad_galact_Ta_dir_H Radiometer galactic direct correction H polarization Current rad_galact_Ta_dir_3 Radiometer galactic direct correction 3rd Stokes Current rad_galact_Ta_ref_V Radiometer galactic reflected correction V polarization Current rad_galact_Ta_ref_H Radiometer galactic reflected correction H polarization Current rad_galact_Ta_ref_3 Radiometer galactic reflected correction 3rd Stokes Current rad_solar_Ta_dir_V Radiometer solar direct correction V polarization Current rad_solar_Ta_dir_H Radiometer solar direct correction H polarization Current rad_solar_Ta_dir_3 Radiometer solar direct correction 3rd Stokes Current rad_solar_Ta_ref_V Radiometer solar reflected correction V polarization Current rad_solar_Ta_ref_H Radiometer solar reflected correction H polarization Current rad_solar_Ta_ref_3 Radiometer solar reflected correction 3rd Stokes Current rad_rain_Tb Rain effect on brightness temperature Planned rad_moon_Ta Lunar effect on antenna temperature Planned rad_rough_corr Derived wind and/or roughness correction Planned scat_rough_corr Scatterometer-based TB roughness correction Planned rad_o2_wv_abs O2/water vapor absorption Planned rad_atm_emiss Atmospheric emission Planned rad_cosmic_Ta Radiometer Cosmic Ta correction Planned

17 of 29 July 19-21, 2010


Scatterometer Parameters (1 of 2)

Name Long Name Implementation

scat_VV_ant TOI Scatterometer normalized radar cross-section (NRCS) for VV polarization Current

scat_HH_ant TOI Scatterometer NRCS for HH polarization Current scat_HV_ant TOI Scatterometer NRCS for HV polarization Current scat_VH_ant TOI Scatterometer NRCS for VH polarization Current

scat_VV_TOA Scatterometer NRCS for VV polarization after Faraday rotation correction Current

scat_HH_TOA Scatterometer NRCS for HH polarization after Faraday rotation correction Current

scat_HV_TOA Scatterometer NRCS for HV polarization after Faraday rotation correction Current

scat_VH_TOA Scatterometer NRCS for VH polarization after Faraday rotation correction Current

scat_wind_speed Scatterometer wind speed Current

18 of 29 July 19-21, 2010


Scatterometer Parameters (2 of 2)

Name Long Name Implementation

dTB_V Delta Tb derived from scatterometer data for V polarization and beam Planned

dTB_H Delta Tb derived from scatterometer data for H polarization and beam Planned

Kpc_VV Kpc statistical uncertainty for VV NRCS Planned Kpc_HH Kpc statistical uncertainty for HH NRCS Planned Kpc_HV Kpc statistical uncertainty for HV NRCS Planned Kpc_VH Kpc statistical uncertainty for VH NRCS Planned wind_uncertainty Estimated wind speed error Planned dTB_V_uncertainty Uncertainty in delta-Tb-V Planned dTB_H_uncertainty Uncertainty in delta-Tb-H Planned scat_total NRCS for total power Planned Kpc_total Statistical uncertainty for total power NRCS Planned

19 of 29 July 19-21, 2010



Agenda

20 of 29 July 19-21, 2010


ADPS Data Archive and Distribution

•  Overview of the currently operational web-based browse, order and distribution tool that is available to support the distribution of Aquarius data within the Aquarius Data Processing System that also serves as a data quality monitoring tool for both internal and external use.

21 of 29 July 19-21, 2010


Aquarius Level-1-2 Data Browser http://oceancolor.gsfc.nasa.gov/cgi/aquarius

22 of 29 July 19-21, 2010


Aquarius Level-1-2 Data Browser http://oceancolor.gsfc.nasa.gov/cgi/aquarius

23 of 29 July 19-21, 2010


All Beams - daily

Aquarius Level-3 Data Browser http://oceancolor.gsfc.nasa.gov/cgi/l3

24 of 29 July 19-21, 2010



Agenda

25 of 29 July 19-21, 2010


Aquarius Website

Consolidated Data access, Information, Services and Community Feedback

26 of 29 July 19-21, 2010


Aquarius Wiki

Online Documentation and Software Descriptions available online at: http://aquarius.nasa.gov/ Under Data and Documents Link labeled “Aquarius Ground System and Mission Operations Wiki Documentation”

27 of 29 July 19-21, 2010



Agenda

28 of 29 July 19-21, 2010


What We Need

•  Review and analysis of the 2007 simulation products •  Agreement on the ancillary data sets and sources. •  Agreement on the Level-2 parameters. •  Agreement on the at-launch processing baseline (e.g., NCEP

winds). •  Schedule and responsibility for algorithm and software updates

between now and launch.

29 of 29 July 19-21, 2010


Backup Slides

30 of 29 July 19-21, 2010


Ground System Interfaces

ADPS ACCS Aquarius Aquarius Data Command & Processing Control System System

CONAE Data Mission RF Archive Operations Services

Command Center

Aquarius Command Schedules And Reports

S-Band

X-Band

Wallops Svalbard Alaska McMurdo

NASA Ground Network

Science Team and Algorithm Providers

S-Band

Algorithms & Science Code Evaluation Products

Ancillary Data Sources

Commands Real-Time Telemetry & Coordination

Schedule Confirmation And Reports

AVDS Aquarius Validation Data System PO.DAAC

Level 1, 2, 3 Aquarius Data

Data Users and Science Team

Aquarius L1, 2, 3 Data Ancillary Data Feedback

SeaDAS So)ware

X-Band

ASI Matera

Cordoba

Principal Investigator and Science Planning Team

Calibration & Instrument Configuration Change Requests

Instrument Engineering Team Aquarius Instrument Testbed

Flight So)ware Updates

ASI Malindi

1- SAC-D Ground System to Aquarius Ground System ICD 2- SAC-D / NASA PSLA, Network Requirements Document & Scheduling Interface 3- SD - Observatory Ground IF Specification Document 4- Aquarius Science Team to Aquarius Ground System ICD 5- AQ - Aquarius Ground System – Aquarius Instrument Team Anomaly ICD and Aquarius Test bed User Manual 6- Aquarius Data Product User Guide (PO.DAAC) & Format Specification (ADPS) 7- Ocean Data Processing System Data and Software Management Plan, ATBD’s and Software User’s Guides 8- A V D S to A D P S ICD 9- Aquarius Ground System to PO.DAAC ICD

Insitu Data Bias Adjustments Aquarius Match-up Data

Aquarius Data, SAC-D Telemetry and Orbit Products

CUSS CODS

31 of 29 July 19-21, 2010


Aquarius Software Status

For each software module, we will show: •  High-level flow (inputs and outputs). •  Current status and capabilities. •  Remaining work to be done.

Input File

Intermediate Data

Output File

Processing Module

32 of 29 July 19-21, 2010


Level-0 Preprocessor

Level-0 Downlink

File

l0gen

Level-0 File

Implemented by Joel Gales Current Capabilities: 1.  Processes a Level-0 downlink file

at CONAE to remove duplicate blocks and sort by time.

To be done: 1.  Final data quality checks.

33 of 29 July 19-21, 2010


Level-0 to 1A

SAC-D Telemetry

SAC-D Ephemeris

Data

Level-0 File

l1agen

Level-1A Files

Implemented by Joel Gales Current Capabilities: 1.  Processes a Level-0 multi-orbit

file to multiple Level-1A orbit files with overlap.

2.  Inputs SAC-D ephemeris and telemetry from separate files.

3.  Unpacks Aquarius science data. 4.  Converts Aquarius telemetry

to engineering units. 5.  Generates HDF5 L1A files. To be done: 1.  Implement actual SAC-D

telemetry input file formats when these are known.

34 of 29 July 19-21, 2010


Level-1A Merge

l1amerge

Level-1A Product

Implemented by Joel Gales Current Capabilities: 1.  Merges multiple Level-1A files

for the same orbit from successive downlinks.

To be done: 1.  Data quality checks to keep only “best” data.

Level-1A Files

35 of 29 July 19-21, 2010


Level-2 to 3 Binning

Level-2 Products

l2bin

Level-3 Binned Beam

Product

Implemented by Joel Gales based on current OBPG binning software

Current Capabilities: 1.  Processes multiple Level-2

products (e.g., for 1 week). 2.  Average samples for each beam

in 1 degree bins. 3.  Outputs a Level-3 binned

product per beam. To be done: 1.  Add flag exclusion rules.

36 of 29 July 19-21, 2010


Level-2 to 3 Smoothing

Level-2 Products

l2smooth

Implemented by Joel Gales based on algorithm from Jonathan Lilly


products (e.g., for 1 week). 2.  Generates smoothed values in 1

degree bins. 3.  Outputs a Level-3 smoothed

product. To be done: 1.  Additional smoothing

algorithms or options. 2.  Add flag exclusion rules.

Level-3 Smoothed Product

37 of 29 July 19-21, 2010


Level-3 Binning

Level-3 Binned

Products

l3bin

Level-3 Binned Product



binned products. 2.  Averages samples in common

bins. 3.  Outputs a single Level-3 binned

product.

38 of 29 July 19-21, 2010


Level-3 Mapping

Level-3 Binned Product

l3bin

Level-3 Mapped Product


Current Capabilities: 1.  Reads a Level-3 binned product. 2.  Outputs a Level-3 mapped (lat-

lon gridded) product.

39 of 29 July 19-21, 2010


OBPG Computing Facility

NASA Goddard Space Flight Center Greenbelt, Maryland Server location: Building 28 Room W220 - 900 square feet dedicated to the project - Raised computer floor with under-floor A/C - Primary and backup A/C units - 80 KVA Powerware UPS unit with dual 50 KVA PDUs, additional 5 KVA rack-mounted UPS units - Connections to GSFC high speed networks: SEN

40 of 29 July 19-21, 2010


OBPG System Hardware

Processing Cluster 46 processing nodes 15 TB

4 storage nodes 10.9 TB

Ingest Servers 2 SeaSpace workstations 78

GB 1 storage node 5.1 TB

Distribution Servers (ftp)

8 storage nodes 49.3 TB 1 server 250GB

Distribution and Storage Servers (web)

Production: 3 processing nodes 75 storage nodes 1.03 PB

Testing Cluster

Network Support Servers

Backup Servers 2 storage nodes 19.2 TB

Extreme Networks Black Diamond 8810

Gigabit Ethernet switch 480 ports

Cal/Val & QC Systems

Mission Operations Systems

Development Servers 2 superstore nodes 10 TB

User Desktops

Aquarius Telemetry Viewer

2 Windows PC

Aquarius Command Planning Tool Servers

2 processing nodes 586 GB

Subversion CM Servers 2 superstore nodes 586 GB

Forum Servers (web) Production: 2 processing nodes

Database Servers 2 superstore nodes 16.4 TB

AQOPS Servers (web) Production: 2 processing nodes

41 of 29 July 19-21, 2010


Aquarius Backup Strategy

•  Data •  On-site

•  Downlinks, L0, L1a data files •  primary copy on RAID6 storage node •  mirror copy on separate RAID6 storage node, at file creation

•  Science Products •  primary copy on RAID6 storage node •  mirror copy if desired, or regenerated from L1a

•  Off-site •  Downlinks: stored at CONAE •  L1A archived at PO.DAAC

•  Source Code •  On-site

•  primary copy on subversion server, RAID6 •  mirror copy on backup Subversion server, RAID6, nightly mirror

•  Off-site •  Subversion data copied to external disk, taken off-site every 1 month •  Multiple copies, given to Project Manager, SE, Processing Manager

•  Other •  On-site

•  AQOPS, Forum: database mirrored to backup server nightly •  Telemetry Viewer, other: image backups periodically, manual

•  Off-site •  AQOPS, Forum: databases included in source code off-line backups •  Telemetry Viewer, other: image backups stored off-line like source code

42 of 29 July 19-21, 2010


IT Security Plan

•  ADPS and ACCS: covered under the same plan as other Ocean Biology Processing Group (OBPG) systems.

•  NASA IT Security Plan Identification –  System name: Sciences and Exploration Directorate (SED) Multi-

Program/Project (MPP) IT Science Systems –  System Abbreviation: SED MPP –  System Unique Identifier: CD-014-L-GSF-6004 –  Information System Type: General Support System –  Information System Security Categorization: Low (transitioning to

medium within the year) –  Information System Operational Status: Operational

•  Effective 5/31/2007. Passed Certification & Accreditation in Summer 2007 - in process of being upgraded

•  Plan designated Sensitive But Unclassified by NASA

•  Implementation questions: addressed off-line

43 of 29 July 19-21, 2010


Ground System Staffing

NASA Management - Gene Feldman: Oversight of the overall effort, communication/coordination with other NASA organizations and centers and CONAE; oversight of science support, algorithm integration, QC, validation and in situ data support. Contractor Management - Fred Patt: Oversight of contract support staff, including hiring, tasking, review of deliverables and schedules; ADPS interface definition and product format specification. Mission Operations - Susie Kennison: Technical lead for instrument operations and commanding support; interface to instrument engineers and CONAE mission operations. Data Systems Engineering - John Wilding: Technical lead for the data system integration effort; overall system design, implementation and testing. So)ware Engineering – Joel Gales: Technical lead for the level conversion / product generation so)ware, including development of the Level 0-to-1 so)ware and integration/packaging of the science (Level 2 and 3) processing so)ware to be provided by the algorithm developers. Data Systems Support - Dan Knowles: Integrate the product generation so)ware into the existing automated processing system, develop ingest modules, develop database tables for archive products, develop schedules and recipes for data acquisition, production and migration. Systems Administration - Paul Smith: Specify and acquire new processing and data storage hardware, integrate into existing data processing facility, verify/upgrade network capability for external data transfers. Quality Control and Validation - Liang Hong: Working with the Science Team, develop methodologies, tools, procedures and reports for operational QC; specify data and develop algorithms and tools for data and science algorithm validation. In Situ Data Support - Jeremy Werdell: Work closely with the Aquarius Validation Data System (AVDS) to develop methods and tools for data analysis and quality control, building upon the existing SeaWiFS Bio-optical Archive and Storage System (SeaBASS). Archive and Distribution - Norman Kuring: Incorporate Aquarius product browse, search, order and distribution into existing web-based Ocean Color capability and work with JPL PO.DAAC regarding “institutional” archive support requirements.