Landsat Collection 2 Data Products and Characteristics

Landsat Collection 2Data Products and

Characteristics

Christopher Barnes

Landsat Science Office

Landsat International Ground Station Technical Lead

[email protected]

KBR, contractor to the U.S. Geological Survey (USGS)

Earth Resources Observation and Science (EROS) Center, Sioux Falls, SD 57198,

USA. Work performed under USGS Contract G15PC00012.

mailto:[email protected]

2

USGS Briefing Outline

➢Landsat Collection 2 Data Products

➢Landsat Collection 2 U.S. Analysis Ready Data

➢Landsat Collection 2 Processing Architecture, Implementation, and Readiness

➢Landsat Collection 2 Data Distribution, Access and Schedule

3

Landsat Collection 2 Data Products Outline

➢What are Landsat Collections?

➢Collection 1 Current Processing Status

➢Collection 2

❑ High-Level Summary

❑ Level-1 Improvements/Enhancements

❑ Level-2 Characteristics/Changes

❑ Product Generation Timeline

❑ Schedule

❑ Samples Feedback

4

What are Landsat Collections?

➢In 2016, the USGS reorganized the Landsat archive into a tiered collection management structure:❑ Need for consistent Landsat 1–8 sensor data and in anticipation of future periodic reprocessing of the

archive to reflect new sensor calibration and geolocation knowledge

➢Landsat collection-based management structure represented a significant change in the inventory of the USGS Landsat Archive❑ Ensuring consistent quality through time and across instruments

❑ While maintaining the heritage specifications;» ~185 km × ~180 km Worldwide Reference System (WRS-2) path/row scene coordinates

» Universal Transverse Mercator (UTM)/Polar Stereographic projections

» World Geodetic System 1984 (WGS84) datum

5

Collection 1

Current Processing Status

6

➢Collection 1 Level-1 processing is proceeding nominally for all new Landsat 7-8 acquisitions (~1,200 scenes/day)

❑ All Landsat 1-8 archive data that is capable of being processed to Level-1 has been processed into the Collection 1 inventory structure

➢U.S. Analysis Ready Data (Level-2) tiles continue to be generated over the continental United States, Alaska, and Hawaii (1982 – present);

❑ Top of atmosphere reflectance

❑ Top of atmosphere brightness temperature radiance

❑ Surface reflectance

❑ Surface temperature

❑ Quality Assessment

Collection 1 Current Processing Status

7

➢ Level-3 Science Products continue to be generated from U.S. ARD (Level-2) inputs;

❑ Burned Area (1984 – present; CONUS)

❑ Dynamic Surface Water Extent (1982 – present; CONUS, Alaska & Hawaii)

❑ Fractional Snow Covered Area (1984 – present; Western U.S. and Alaska)

➢ Collection 1 Landsat 7-8 processing will continue for at least one year after public release of Collection 2, and all historical Collection 1 data products will remain available for download during this period

Collection 1 Current Processing Status

8

Collection 2

High-Level Summary

9

➢In 2018, the USGS started planning and defining specifications for Landsat Collection 2

➢Landsat Collection 2 will maintain continuity with Collection 1, but there will be several enhancements that

harness more recent advancements in processing, algorithm developments, and access and distribution

services

Including:

1. Improved per-pixel geodetic accuracy by incorporating updated Landsat 8 ground control points

harmonized with the European Space Agency/Copernicus Sentinel-2 Ground Reference Image (GRI)

2. A global inventory of scene-based surface reflectance and surface temperature data products that meet

the solar elevation angle criteria (i.e., <76 deg)

3. Updated digital elevation model sources for various global regions to support Level-1 processing

Sample products, metadata and documentation now available on the USGS Landsat Missions website!

Collection 2 High-Level Summary

https://www.usgs.gov/land-resources/nli/landsat/landsat-sample-products

10


1. Improved per-pixel geodetic accuracy by incorporating updated Landsat 8 ground control points

harmonized with the European Space Agency/Copernicus Sentinel-2 Ground Reference Image (GRI)

❑ Comparisons of GRI and Collection 1 Landsat 8 OLI-based GCPs showed ~26m CE90 registration accuracy

❑ Updated Collection 2 Landsat 8 OLI-based GCPs were extracted and triangulated to Sentinel-2 GRI

❑ Collection 2 Landsat 8 OLI-based GCPs results yield Landsat/Sentinel-2 registration accuracy <10m CE90

Collection-1 OLI-only GRI

Block Type # Tiles CE90 (m) CE90 (m) CE90 (m)

North America All 778 20.80 11.47 7.61

South America All 228 31.75 12.87 6.06

Australia All 77 7.15 7.15 5.44

Islands All 159 28.21 15.01 9.22

Asia All 819 27.83 10.74 6.95

EurAfrica All 1134 25.84 8.33 6.44

Total All 3195 25.56 10.51 6.98

11

Collection 2 Level-2 Characteristics

2. A global inventory of scene-based surface reflectance and surface temperature data products that

meet the solar elevation angle criteria (i.e., <76 deg)

» Landsat 4-8 only (Landsat 9 in 2021)

❑ Surface Reflectance

» Landsat 4-7 (LEDAPS) & Landsat 8 (LaSRC)

❑ Surface Temperature (Single-channel)

➢ Landsat 4-8

12

Collection 2 Updated DEM Sources

➢ Global Land Survey Digital Elevation Model (GLS DEM)

➢ NASA Shuttle Radar Topography Mission (NASADEM) (Reprocessed)

➢ Canadian Digital Elevation Model (CDEM) (Updated)

➢ Sweden, Norway, and Finland (SNF) National Elevation Data

➢ National Elevation Dataset for Alaska (AK_NED) (New)

➢ Greenland Mapping Project (GIMP) Digital Elevation Model

➢ Norwegian Polar Institute (NPI) Elevation Data

➢ WorldView-derived Arctic DEM

3. Updated digital elevation model sources for various global regions to support Level-1 processing

13


➢What’s the same?

❑ Projection, resampling, pixel size and datum

❑ CFMask (v3.3.1)

❑ Global Level-1 path/row scenes for all sensors (MSS to OLI/TIRS)

❑ U.S ARD and Level-3 products (Albers, tiled, TM to OLI/TIRS)

❑ Data delivery services – EarthExplorer and LandsatLook (post Collection 2 release)

➢What’s new?❑ Global scene-based (UTM) surface reflectance & surface temperature (Landsat 4-8)

❑ Consistent Level-1 & Level-2 Quality Assessment bands

❑ Angle bands (solar/sensor viewing zenith & azimuth) are provided for users to generate bespoke

TOA-R bands

❑ Processed in USGS Amazon Web Services (AWS) Cloud Hosting Solution (CHS) environment» Output file format: Cloud Optimized GeoTIFF (COG) format

» Download only the bands you need

❑ Level-2 atmospheric auxiliary data available for download

❑ Forward processing done at EROS and synced to AWS CHS (90 day rolling cache)

14

Collection 2 Level-1

Improvements/Enhancements

15

Collection 2 Level-1 Improvements/Enhancements

➢Collection 2 Level-1 product and band identifiers follow the same nomenclature as Collection 1

❑ Digital Numbers (DN) in an unsigned 16-bit integer format and can be converted to Top of atmosphere (TOA) reflectance or radiance using scaling factors provided in the product metadata

❑ The following bands are available for download individually or as a bundle from EarthExplorer and/or LandsatLook

➢ Spectral Bands in COG format❑ Bands 4-7 (Landsat 1-3 MSS)❑ Bands 1-4 (Landsat 4-5 MSS)❑ Bands 1-7 (Landsat 4-5 TM) ❑ Bands 1-8 (Landsat 7 ETM+)❑ Bands 1-11 (Landsat 8 OLI/TIRS)

➢ Angle Files❑ Solar Azimuth Angle (SAA)❑ Solar Zenith Angle (SZA)❑ View Azimuth Angle (VAA)❑ View Zenith Angle (VZA)❑ Angle Coefficient File

➢ Quality Assessment (QA)❑ Pixel Quality Assessment (QA) Band❑ Radiometric Saturation and Terrain Occlusion QA Band

➢Metadata❑ Material Library File (MTL) (Object Description

Language-based) format ❑ Extensible Markup Language (XML)

16


Geometric processing improvements/enhancements:

1. Remove Extreme Latitude DEM Constraint

Collection 1 scenes with a DEM value beyond the geographical extent (-57.0 degrees latitude) of the elevation source would only process to a Level-1 Systematic Terrain (L1GT) product. Removing this constraint allows for more extreme southern latitude acquisitions to process to a Level-1 Terrain Precision (L1TP) product. For example, this was most pronounced at the southern tip of South America (Path 227 Row 99).

2. Landsat 7 ETM+ and Landsat 4-5 TM Precision Correction LogicThe number of Landsat 4-5 TM and Landsat 7 ETM+ Level-1 scenes achieving the Collection Tier 1 inventory criteria is expected to increase due to improvements in terrain precision modelling.

17


3. Landsat 8 Instrument Alignment

Corrects for minor instrument seasonal dependencies and improves the Landsat 8 Thermal Infrared Sensor (TIRS) to Operational Land Imager (OLI) sensor alignment, which occurred as a result of the December 2014 TIRS Scene Select Mirror (SSM) anomaly.

4. Landsat 7 ETM+ Thermal Band Detector Alignment

Landsat 7 ETM+ Band 6 (Thermal) includes a minor odd-to-even detector misalignment due to pre-launch detector-based focal plane values. After extensive historical on-orbit characterization and calibration efforts of the thermal band detector placement, revised focal plane values have been determined. The revised focal plane values will provide improved detector-to-detector alignment of Band 6.

18


5. Remove GCPs Constraint in Level-1 ProcessingCollection 1 Landsat 1-7 Level-1 processing used a 250 GCP constraint. This GCP constraint may have reduced the number of acquisitions that could have processed to Level-1 Terrain Precision (L1TP). Collection 2 uses all available GCPs and together with the GRI harmonization there is an increase in the number of Landsat 1-7 processing to a Level-1 L1TP product.

6. Increase DEM Scene Framing Extent

Present in some Collection 1 scenes is a duplicated ‘wedge’ of data in the extreme southwest corner of the scene. This primarily occurs in Band 1 as a result of insufficient scene coverage by a DEM tile. An increase in the DEM framing extent ensures scenes prone to this issue are captured by all elevation tiles so they are precisely terrain corrected.

19


7. Landsat 7 ETM+ Instrument to Attitude Control System (ACS) Alignment Update

Landsat 7 ETM+ instrument to ACS alignment was updated between January 2013 - December 2018 and impacted Level-1 Systematic Terrain (L1GT) acquisitions only. The alignment error source was a function of the L1GT scene registration to the DEM and World Geodetic System (WGS84) model without the application of GCPs. The implementation of this update results in an increase accuracy of all Landsat 7 ETM+ L1GT acquisitions.

8. Day Transition Error Fix

Previously, some Landsat 1-7 acquisitions captured during a day transition (primarily over Australia) failed to produce an available Level-1 product, or produced a misregistered Level-1 Terrain Precision (L1TP) or Systematic Terrain (L1GT) product. These affected acquisitions will be processed to a L1GT or L1TP product.

20


9. Landsat 7 ETM+ and Landsat 4-5 TM Shutter Intrusion Indicated in New Sensor Anomaly Metadata FieldDue to improved Internal Calibrator shutter intrusion detection for Landsat 4-5 TM and Landsat 7 ETM+, a new metadata field (<SENSOR_ANOMALIES = “SHUTTER_INTRUSION”>) is included in the metadata file if intrusion is detected.

10. Adjust Landsat 7 ETM+ Focal Plane for Improved Band AlignmentThe Landsat 7 ETM+ cold-to-warm focal plane alignment was adjusted to account for small misalignments that occurred during changes in operations, such as the Scan Line Corrector failure in 2003. This update produces improved co-alignment between the cold and warm focal planes of the Landsat 7 ETM+ instrument.

21


Radiometric processing improvements/enhancements:

➢Landsat 8 TIRS relative gains update

➢ Reduce striping in TIRS data

Collection 1 Collection 2

22


➢ Landsat-5 TM Band 6 Calibration Update

➢ Landsat 7 ETM+ Band 6 Low & High Combination❑ Global UTM surface temperature and U.S. ARD brightness temperature and top of atmosphere

reflectance will be derived from Band 6H if unsaturated (better radiometric resolution) or 6L otherwise

➢ Landsat 8 OLI Absolute Gain Model Change

➢ Landsat 8 OLI Bias Calculation Update

➢ Landsat 8 TIRS Absolute Calibration Update

More information on the USGS Landsat Missions web site

usgs.gov/land-resources/nli/landsat/landsat-collection-2

Landsat 8 satellite background

23


➢Updated and Consistent Metadata Files

❑ To facilitate consistency, machine-to-machine scripting and rapid querying of the Landsat archive:

» Material Library File (MTL) (Object Description Language-based) text format (currently offered)

» Extensible Markup Language (XML) format

» SpatioTemporal Asset Catalog (STAC) Version 0.7

➢Changes include

Group or Parameter From

Collection 1

To

Collection 2

Group renamed or combined GROUP = L1_METADATA_FILE LANDSAT_METADATA_FILE

Group added n/a PRODUCT_CONTENTS

Parameter name duplicated n/a COLLECTION_NUMBER = 02

Parameter renamed DATA_TYPE =“L1TP” PROCESSING_LEVEL = "L1TP"

Parameter added n/a DIGITAL_OBJECT_IDENTIFIER =

"https://doi.org/10.5066/F7WH2P8G"

24


Available on the USGS Landsat Missions web site

25


Characteristics/Changes

26

Collection 2 Level-2 Characteristics

➢Global Level-2 products are processed from Collection 2 Level-1 inputs (Tier 1 or Tier 2) that meet the Solar Zenith Angle (SZA) constraint and that have required auxiliary data

» Landsat 4-8 only (Landsat 9 in 2021)

» Atmospheric correction capabilities for MSS are in the preliminary stages


» Landsat 4-7 (LEDAPS) & Landsat 8 (LaSRC)❑ Surface Temperature (Single-channel)

➢ Landsat 4-8

27

Collection 2 Level-2 Changes

❑ Surface Reflectance» Landsat 4-7 (LEDAPS) & Landsat 8 (LaSRC)

Level-2 atmospheric auxiliary data available for download via EarthExplorer and LMWS

Collection 2 Surface Reflectance Collection 1 Surface Reflectance

Fill Value 0 -9999

Scaling Factor 0.0000275 + -0.2 0.0001

Data Type Unsigned 16 bit integer Signed 16 bit integer

Valid Range 1-65455 0-10000

Quality Band L1 QA_Pixel

L1 QA_RADSAT

L2 SR_QA_AEROSOL

Level-2 PIXELQA

Level-2 RADSATQA

Level-2 SRAEROSOLQA

https://earthexplorer.usgs.gov/

https://www.usgs.gov/land-resources/nli/landsat/

28


Level-2 atmospheric auxiliary data available for download via EarthExplorer and LMWS

❑ Surface Temperature❑ Single-channel

❑ Landsat 4-8

Collection 2 Surface

Temperature

Collection 1 Surface Temperature

Fill Value 0 -9999

Scaling

Factor

0.00341802 + 149.0 0.1

Data Type Unsigned 16-bit integer Signed 16-bit integer

Valid Range 1-65535 1500-3730

Atmospheric

Reanalysis

Source

GEOS-5 FP-IT (2000-

present)

MERRA-2 (1982-1999)

NARR

https://earthexplorer.usgs.gov/


29


➢Brightness Temperature and Top-of-atmosphere Reflectance not available globally❑ Processed only as U.S. ARD tiles❑ Level-1 angle bands (solar/sensor viewing zenith/azimuth) provided for users to generate their own

➢Surface Reflectance and Surface Temperature are packaged separately❑ Downloads by individual band or by bundle

➢Product name changes❑ FROM: LC08_L1TP_140041_20130503_20190828_01_T1 (Currently delivered from ESPA)

❑ TO: LC08_L2SP_140041_20130503_20190828_02_T1 (SR & ST are generated)❑ TO: LC08_L2SR_140041_20130503_20190828_02_T1 (SR only generated)

➢Band name changes❑ FROM: LC08_L1TP_ 222005_20140922_20140923_01_T1_sr_band1.TIF (Currently delivered from ESPA)

❑ TO: LC08_L2SP_222005_20140922_20140923_02_T1_SR_B1.TIF❑ TO: LC08_L2SR_222005_20140922_20140923_02_T1_SR_B1.TIF

30

Collection 1 vs. Collection 2 Quick Reference PDF Guide

Available on the USGS Landsat Missions web site


31

Collection 2 Product Generation Timeline

32

Collection 2 Product Generation Timeline

Downlink & Level-1 processing ≈ 4-6 hours

Auxiliary data≈ 2 days

Auxiliary data

Level-2 SR & ST≈ 3 days

Level-2 processing ≈ 24 hours

33

Collection 2 Samples

➢Collection 2 samples available!❑ Contact USGS User Services ([email protected]) with any questions or comments

❑ Consider feedback as early science input to Collection 3 development


Collection 2U.S. Landsat Analysis

Ready Data Characteristics

35

Collection 2 Landsat U.S. ARD Outline

➢Collection 2 Landsat U.S. ARD concept

➢Collection 2 Landsat U.S. ARD product package

➢Collection 1 to Collection 2 Landsat U.S. ARD differences and statistics

36

U.S. Landsat ARD

U.S. Landsat

Analysis Ready Data

-

37

U.S. Landsat ARD

➢Consistently processed Landsat Level-1 data for direct use in science applications

➢Reduce burden of processing large amounts of Landsat data for time series analysis/”stacking”

➢Landsat Analysis Ready Data (ARD) available for the U.S. only

➢Landsat 4-8 Collection 2 Level-1 Tier 1 products serve as the input

-

38

U.S. Landsat ARD Image Extent

➢Level-1 path/row scene vs ARD tile grid

➢ARD includes all valid image data acquired on a given day, within the extent of the given tile

➢ARD tile image data volume not always the same

39

U.S. Landsat ARD Changes for Collection 2

➢Extend tile grid extent to include all Aleutian Islands

➢Add 15km buffer around CONUS, AK and HI to include coastal waters and international borders (+33 scenes)

➢Fix incorrect precision of WGS84 axis values

➢Fix incomplete file names in the U.S. Landsat ARD Tile XML Metadata

➢Correct Unit of Atmospheric Transmittance in XML Metadata

40


U.S. Landsat ARD

➢U.S. Landsat ARD consists of:❑ Top of Atmosphere Reflectance

-

41

❑ Pixel Quality Assessment (QA)

U.S. Landsat ARD

❑ Top of Atmosphere

Brightness Temperature

-

42

U.S. Landsat ARD Packaging and File Sizes

➢ Packaged into bundles

❑ Top of Atmosphere Reflectance


❑ Top of Atmosphere Brightness Temperature

❑ Quality Assessment

➢ Packages delivered as .tar files

❑ Including an .XML metadata file

➢ File sizes vary greatly

❑ Product(s)

❑ Tile location

❑ Input data volume (scenes)

43

U.S. Landsat ARD Numbers

➢38 years of Landsat data (1982-present):❑ Landsat 4-8 (Landsat 9 in 2021)

❑ 45,900 passes in 5.4 million acquisitions

❑ 11.1 million available data products

❑ ~277 Trillion pixels in each available data product

❑ ~4,300 observations for some areas

➢ Total Data Volume ~286TB CONUS

HawaiiAlaska

Landsat Collection 2 Processing Architecture,

Implementation, and Readiness

Chris Engebretson

Landsat Data Production Systems Manager

[email protected]


45

Collection 2 Implementation

➢The Collection 2 implementation provides the capability to generate Landsat Level-1 and Level-2 products and metadata for the user community❑ Collection Processing in the Cloud

» Collection Processing in the cloud provides the capability to produce Collection 2, Level-1 and Level-2 data, at scale

» Processing of all Landsat 1-8 data acquired prior to June 1, 2020 is being performed in the USGS Cloud Hosting Solutions (CHS) environment

❑ Forward Processing On-Premises» Collection 1 processing will continue on-premises at EROS for at least one year following public availability of

Collection 2

» Forward processing new Landsat 7-9 downlinks will continue on-premise at EROS for all scenes acquired on or after June 1, 2020

» A 90-day rolling archive at EROS will be available for high volume users to download data

» Collection 1 and Collection 2 product data produced on-premise will be pushed to the cloud for access and distribution in a COG format

46

Collection 2 Operational View

47

Cloud Image Processing Design

48

Collection 2 Processing Strategy

➢Goal: Process C2 data in the cloud as quickly as possible, but ensure the data is validated and verified

➢Basic process:1. Start with 1,000 validation scenes for QA

2. Pause for analysis to perform QA activities scenes

3. Start to speed up processing, assigning a priority that compliments the recovery process

» Analysis will continue for a while in parallel

4. Evaluate processing speed and cost regularly, and make changes as necessary

5. Release all data together to the public

49

Collection 2 Nominal Processing Timeline

Landsat Collection 2 Data Distribution,

Access and Schedule

Tom Sohre

EROS Data Center and Systems (DCS) Branch [email protected]

Kristi Kline

Landsat Archive and Access Project [email protected]



51

Schedule

➢August 2020: Collection 2 Scene-based Processing (L1 & L2)❑ July 20, 2020: Collection 2 Processing Operational Readiness Review (successful)

❑ Concurrent forward-processing with Collection 1 (on-premise at EROS)

❑ Full archive C2 processing scheduled to complete in 3-4 weeks (Cloud)

➢September 2020: Collection 2 Public Availability❑ September 11, 2020: Collection 2 Access Operational Readiness Review (scheduled)

❑ Standard Access for Collection 2 via Earth Explorer and Bulk Download (consistent with current C1)

❑ Full Product and Band access for Collection 2 data

❑ Collection 2 distribution via Cloud

❑ Glovis and LandsatLook 1.0 (current) will only include Collection 1 data

➢October 2020: Collection 2 U.S. ARD Processing

➢November 2020: Landsat in the Cloud Enhanced Access❑ Direct Access to AWS S3 (in US-West Region); enables Jupyter Notebook Access

❑ LandsatLook 2.0 (transition from current LandsatLook tool)

❑ Collection 2 U.S. ARD access via Cloud

❑ Collection 1 access via Cloud

52

User Access Updates

➢Updated Earth Explorer Capabilities❑Allows users to order either a full Landsat scene, or user selected bands

➢What will you see in the new LandsatLook?❑Dynamically mosaicked imagery based on a user-specified area of interest (AOI) and

temporal extent from the scene-based Landsat record

❑ Leverages USGS data via AWS including Cloud Optimized GeoTIFFs (COGs) and Spatio Temporal Asset Catalog (STAC) metadata

❑ Filter and replace cloud contaminated pixels flagged in the Landsat Pixel_QA Band

❑Export an image mosaic in a Portable Network Graphics (PNG) format

❑Download specific bands of the user selected source mosaics

❑Generate and download a video animation mosaic timelapse showing how the dynamic mosaic changes over time

❑Allows users to choose selected dynamic displays of band combinations with math (i.e., data displayed as NDVI)

53

Communications

➢Landsat in the Cloud User Videos❑ Introduction to Landsat in the Cloud (released)

❑ How current options of data discovery are still available (complete; August release)

❑ More details on User options to run algorithms in the Cloud (complete; September release)

❑ LandsatLook (in development; October release)

➢Landsat in the Cloud FAQs (to be released 2 weeks prior to public data release)❑ Describes how users will interact with the data in the cloud

❑ 10 FAQs have been written for the Landsat Mission Website

➢Landsat in the Cloud User Guide (to be released after enhanced access release)

➢Jupyter Notebooks (to be released after enhanced access release)❑ Simple/Starter notebooks available via the Landsat Mission Website

➢Social Media Activities❑ Twitter, Facebook

https://www.youtube.com/watch?v=SmG_2t-J_1o

Connect

@USGSLandsat

@NASA_Landsat

@NASA.Landsat

eros.usgs.gov/video-library

Landsat Updates

Landsat Headlines

Contact

[email protected]

605-594-6151

@usgslandsat

Documents

Landsat Collection 2 Data Products and Characteristics