AMSR-E SIPS Processing Status

Presented by Kathryn RegnerInformation Technology and Systems Center

at the University of Alabama in Huntsville

JAXA / AMSR-E Working Group MeetingJune 2-3, 2010Huntsville, AL

kregner@itsc.uah.edu www.itsc.uah.edu http://ghrc.msfc.nasa.gov 256.961.7791

2

Outline

• Introduction• Hardware Update• Software Status• Accomplishments• Future Activities

JAXA / AMSR-E Working Group Meeting Huntsville June 2-3,

2010

3

AMSR-E SIPS Personnel

JAXA / AMSR-E Working Group Meeting Huntsville June 2-3, 2010

• NASA MSFC: H. Michael Goodman, PI• UAHuntsville ITSC (Staff shared with GHRC DAAC): Management: Dr. Sara Graves, Director, ITSC

Development: Helen Conover, Lead Bruce Beaumont Lamar Hawkins Mary Nair Matt SmithEngineering: Kathryn Regner Sandra Harper Data Management: Steve Jones Sherry Harrison Systems Administration: Will Ellett, Lead Michele Garrett

TLSCF ScienceSoftware I&T SIPS-GHCC

PO.DAAC

JAXA-EOCL1A Product Generation

SIPS-RSSL2A Product Generation

L2B and L3 Product Generation(science, metadata, qa, ph, subsets, browse)

L2 and L3algorithms

L2A algorithm

NSIDC DAAC Data Archive and Distribution

Level-1

Level-2 and Level-3

Level-1

Level-1

NASAEDOS

GBAD (Pre Ephem) & PDS

GroundStations

L2B and L3 Algorithms

Level-0

Product Delivery Record Server (PDRS)

Algorithm Teams

Ocean Sea Ice

Rain

Land

Snow Data

NASA Flt Dynamics

GBAD(Def Ephem)

AMSR-E Data Flow

JAXA / AMSR-E Working Group Meeting Huntsville June 2-3,

2010

5

SIPS-GHCC HardwareOperational Configuration

8 TB Dell EMC CX300

Storage Area Network Array

2 G

B/s

ec

Fibre

Channel S

wit

ch

Dell PowerEdge 28502 x 3.4 GHz Processors

4 GB RAM, RHEL 5AMSR 1-4

Near line storage of all Level-2A and most recent Level-2B & Level-3

Restricted FTP Server

Multiple processing environments across four machines results in much greater flexibility for processing and

reprocessing.

Sun Storage 7310With J4400 Disk

Array8TB NAS

Pri

vate

Netw

ork

(1 G

B/s

ec)

Forward Processing

Reprocessing

Reprocessing

Dev, Integration & Test

Storage BuildoutCompleted July 2009

JAXA / AMSR-E Working Group Meeting Huntsville June 2-3,

2010

6 JAXA / AMSR-E Working Group Meeting Huntsville June 2-3, 2010

Planned SIPS-GHCC Hardware

Configuration – Summer 2010

Near line storage of all Level-2A and most recent Level-2B and Level-3

Restricted FTP Server

Multiple processing environments across four machines results in much greater flexibility for processing and

reprocessing.

Sun Storage 7310With J4400 Disk

Array10TB NAS

Pri

vate

Netw

ork

(1 G

B/s

ec)

Forward Processing

Reprocessing

Dev, Integration & Test

Sun Fire x4270 x642 x 2.93 GHz Sun Intel Xeon Processors

4 x 2GB DIMM , RHEL 5AMSR 1-3

7 JAXA / AMSR-E Working Group Meeting Huntsville June 2-3, 2010

Software Architecture(Routine Processing )

Level-2A Brightness Temperatures(with Metadata and QA files)

L2B Ocean(1/2-orbit)

L2B Land(1/2-orbit)

L2B Rain(1/2-orbit)

L3 Ocean(Daily)

L3 Land(Daily)

L3 Snow(Daily)

L3 Sea Ice(Daily, 3 Res)

L3 Snow(Pentad)

L3 Ocean(Weekly)

L3 Ocean(Monthly)

L3 Rain(Monthly)

L3 Snow(Monthly)

Once per day

Once every

five days

Once per week

Once permonth

ProductGenerationScheduling

-------------~ 29 times

per day

Level-2A Brightness Temperatures

Near Real Time

Ingest from RSS

L3 Sea IceDrift

8

Algorithm Version Numbers Summary

JAXA / AMSR-E Working Group Meeting Huntsville June 2-3, 2010

AlgorithmShort Name

L2A TbL2B LandL2B OceanL2B RainL3 LandL3 OceanL3 RainL3 Sea IceL3 Snow

Current Versions (as of May 31, 2010)

V10V09V06V10V06V04V06V12V09

9

AccomplishmentsSystems:

Integration of Network Attached Storage (NAS) device (July 2009):supports SIPS’ role as official offsite archive

for the AMSR-E L2A brightness temperature data set.

facilitates reprocessing of higher level products

Worked with the vendor to upgrade firmware on the Sun Storage 7310 Storage Area Network (NAS) device. Operations team reported significant increases in speed when transferring files. (January 2010)

Selected hardware vendor for the FY10 technology refresh procurement (May 2010).

JAXA / AMSR-E Working Group Meeting Huntsville June 2-3, 2010

10

Accomplishments (2)Processing:

Procedures Update (November 2009): per NSIDC request and in response to increased user community interest in preliminary (near real time) products, SIPS implemented a new production rule to regenerate the preliminary daily files whenever regenerated or delayed ½-orbit files are received. 

Integrated Version 12 Sea Ice Algorithm into SIPS processing system and began reprocessing (March 2010). defines the sea ice season to begin on October 1st of

each year and improves upon management of the sea ice mask ancillary files. JAXA / AMSR-E Working Group

Meeting Huntsville June 2-3, 2010

11

Accomplishments (3)Processing (continued):

Integrated Version 4 of the Level-3 Ocean DAP into the SIPS processing system (April 2010) and began reprocessing.utilizes the RFI filter for DirecTV that was implemented

in the L2A algorithm last year. reprocessing the L3 Ocean requires reprocessing the L2

Ocean as well, in order to make use of the new features 

JAXA / AMSR-E Working Group Meeting Huntsville June 2-3,

2010

12

Accomplishments (4)Reprocessing:

Second Mission Dataset Reprocessing Complete (July 2009): using Version 10 of the Level-2A products,

which was enhanced to include RFI quality flags

each of the 13 higher level products are now at the same version number from the beginning of the mission until the present date.

Completed reprocessing Version 12 Sea Ice mission dataset (May 2010).

JAXA / AMSR-E Working Group Meeting Huntsville June 2-3, 2010

13

Accomplishments (5)LANCE:

Attended the AMSR-E Mission Operations Working Group meeting, hosted by Angie Kelly, at GSFC. Learned about the Land Atmosphere Near Real Time Capability for EOS (LANCE). (October 2009)

Attended the LANCE Workshop held in College Park on December 8-9. Began preliminary analysis with ESDIS and the AMSR-E near-real time data providers (MODIS) to determine the optimal way to split up the near real time processing load and integrate algorithms into the LANCE system.

In February 2010, UAHuntsville submitted a proposal and budget to ESDIS requesting funds to implement the Land Atmosphere Near-real-time Capability for EOS processing at the AMSR-E SIPS. 

Contracted awarded in March; official start date April 1st.JAXA / AMSR-E Working Group Meeting Huntsville June 2-3, 2010

14

Near Term Activities

• Integration of new (FY10) hardware into SIPS processing systems; software migration.

• Integration of new and updated algorithmsSea Ice Drift product and Level-3 RainReprocessing those datasets

• Evaluate reprocessing Level-2A based upon new Level-1A.

JAXA / AMSR-E Working Group Meeting Huntsville June 2-3, 2010

15

Summary

• Processing and reprocessing activities are running smoothly.– Integrating new algorithms as received

from TLSCF– Preparing to install, integrate and test

the third generation of hardware

JAXA / AMSR-E Working Group Meeting Huntsville June 2-3,

2010