Combined Precipitation Algorithms – IMERG

Combined Precipitation Algorithms – IMERGGeorge J. Huffman

NASA/GSFC Laboratory for AtmospheresContact: [email protected]



Animation located in Hurricane Matthew 10-2-16.mp4

Thanks to:Bob AdlerDavid BolvinEric Nelkin

èSensors, algorithms, and archivesè IMERG algorithm and accessèHow good?èFuture prospectsèFinal pointsèAdditional material



Data Sources (1/3)

The international constellation of “precipitation relevant”• some sensor on the satellite is useful for estimating precipitation

“satellites of opportunity”• the satellites are flown by an agency for its own purposes, and they contribute their

data to the constellation archive

Huffman 10/16

Data Sources (2/3)

Passive microwave p.m. overpass times

• DMSP F08 SSMI was the first “modern” PMW

• we’re now in the “golden age”

• what’s the future hold?

• some satellites drifta lot

• shading indicates precessing TRMM, GPM, Megha-Tropiques

• persistent gap at 00/12 LT

Huffman 10/16http://precip.gsfc.nasa.gov/times_allsat.jpg

Data Sources (3/3)

Sensor types• radar – very

expensive• PMW imagers –

expensive• PMW sounders –

less expensive• geosynchronous

IR (and multi-spectral) – plentiful data

• precipitation gauges – gold standard, but gaps and only over land

“warm”ocean

“warm”land

complexterrain

“cold”ocean

snowy/frozen

“easy” Surface Type “hard”

“w

orst”

Qua

lity

“be

st”

DPR,PRGMI,TMI

AMSR

SSMIMHS

AMSU

TOVS,AIRS

IR

SSMIS

Huffman 10/16

Algorithms

Sensors have specific strengths and weaknessesAlgorithms might or might not do the best job at extracting the information in a sensor’s readingsYou’ve already seen discussions of the algorithms for individual sensors

What is the best way to combine individual sensors?• it’s almost always done using precipitation datasets

– except IR is usually converted from Tb to precip as part of the combination• averages in time/space that use unevenly spaced or sparse data run the risk

of biasesMulti-satellite precip can be

• Climate Data Record (CDR)– homogeneous, generally by neglecting some data– CMAP, GPCP

• High-Resolution Precipitation Product (HRPP)– use “all” the data, homogenous or not– CMORPH, GSMaP, IMERG, TMPA

• although each strives for the other goal as well

Huffman 10/16

Example family tree for Goddard products

The Adjusted GPI (early ‘90’s) led to GPCPGPCP concepts were first used in TRMM, then blended with new multi-satellite conceptsIMERG adds morphing, Kalman smoother, and neural-network IR estimates

TRMM V4,5 3B42

TRMM 3B42RT

Time

thin arrowsdenote heritage TRMM 3B42RTTRMM 3B42RT

CMORPH

IMERG lateIMERG final

IMERG early

GPCP V1SGMAGPI

TRMM V4,5 3B43

GPCP V2,2.1 SGGPCP V1,1.1 1DDGPCP V1,1.1 Pentad

V6,7 TRMM 3B42V6,7 TRMM 3B43

TRMM 3B42RT

GPCP V3

PERSIANN PERSIANN-CCS

KF-CMORPH

Huffman 10/16

Notes on combination algorithms

We consider elimination of bias in the individual input estimates to be critical

• PMM-calibrate all microwave to a single standard • use merged microwave to calibrate IR• calibrate multi-satellite to monthly gauge using large-area averagesCalibrations must be done with both data sets averaged to the same spatial scalefor consistencyData are merged after calibrationCombination schemes are now at the maturity that microwave schemes had about 10 years agoOperational estimates are just starting to be useful over frozen/icy surfacesPassive microwave data too sparse to completely populate a fine time/space grid• need Lagrangian time interpolation • “morphing” is a linear fade algorithm• IR beats interpolated microwave if the nearest orbit is > 90 min. away• actually need “cloud development” algorithm

Huffman 10/16

Archive sites

Once computed, the data sets are held at archive sites• primary – responsible for providing the original data• mirror – archives at other locations to satisfy local requirementsBoth the primary and mirror sites may create value-added products

• reformatted, aggregated or subsetted, etc.Data provenance is always important, particularly for mirror sites and value-added productsIPWG posts 4 tables of primary archive sites for • quasi-global• freely available• long-termprecipitation data sets• organized according to input data types• maintained by George Huffman

see “additional material”

Huffman 10/16

IMERG processing

IMERG is a unified U.S. algorithm that takes advantage of

• Kalman Filter CMORPH (lagrangian time interpolation) – NOAA• PERSIANN with Cloud Classification System (IR) – U.C. Irvine• TMPA (inter-satellite calibration, gauge combination) – NASA• PPS (input data assembly, processing environment) – NASA

The Japanese counterpart is GSMaPInstitutions are shown for module origins, but• package will be an

integrated system• goal is single code system

appropriate for near-realand post-real time

• “the devil is in the details”

Huffman 10/16

GSFC CPCUC Irvine

prototype6

Receive/storeeven-odd IR

files

Import PMW data;grid; calibrate;

combine

Compute even-odd IR files(at CPC)

Compute IRdisplacement vectors

Build IR-PMW precip calibration

IR Image segmentationfeature extraction

patch classificationprecip estimation

Apply Kalman

filter

Build Kalman

filter weights

Forward/backward

propagation

Import mon. gauge; mon. sat.-gauge

combo.;rescale short-intervaldatasets to monthly

Apply climo. cal.RT

Post

-RT

Recalibrateprecip rate

IMERG Data Sets

Multiple runs accommodate different user requirements for latency and accuracy

• “Early” – 5(4) hours (flash flooding)• “Late” – 15(12) hours (crop

forecasting)• “Final” – 3.5(2.5) months (research)Time intervals are half-hourly and monthly (Final only)

0.1° global CED grid • PPS will provide subsetting by

parameter and location• 60°N-S for now

User-oriented services• interactive analysis (Giovanni)• alternate formats (KMZ, KML, TIFF

World files, …)• area averages

Huffman 10/16

Half-hourly data file (Early, Late, Final)

1 [multi-sat.] precipitationCal2 [multi-sat.] precipitationUncal3 [multi-sat. precip] randomError4 [PMW] HQprecipitation5 [PMW] HQprecipSource [identifier]6 [PMW] HQobservationTime7 IRprecipitation8 IRkalmanFilterWeight9 probabilityLiquidPrecipitation [phase]

Monthly data file (Final)

1 [sat.-gauge] precipitation2 [sat.-gauge precip] randomError3 GaugeRelativeWeighting4 probabilityLiquidPrecipitation [phase]

Data Fields from IMERG Test Data (1/4)1430-1500Z 3 April 2014

Huffman 10/16

PMW data collected in the half hour

PMW sensor contributing the data,selected as imager first, then sounder, then closest to center time [PMW] HQprecipSource [identifier]

[PMW] HQprecipSource [identifier] (mm/hr)

GMI

TMIAMSR2

MHSSSMIS


Huffman 10/16

[PMW] HQobservationTime (min)

IRprecipitation (mm/hr)

PMW sensor observation time after start of half hour

precip from merged geo-IR data


Huffman 10/16

[multi-sat.] precipitationCal (mm/hr)

“Final” IMERG field: forward and backwardmorphedmicrowave, Kalman filter with IR data; monthly gauge

[multi-sat. precip] randomError (mm/hr)

estimated random error for the multi-satellite precip


Huffman 10/16

probabilityLiquidPrecipitation [phase] (%)

probabilitythat precipitation phase is liquid; diagnosticcomputed from ancillary data

IRkalmanFilterWeight (%)

weighting of IR in the Kalman filter step

Core Observatory – constellation – “best estimate”

Huffman 10/16

Animation located in GPM_Fleet_IMERG_globe.mp4

PMM home page

New features to improve “discovery” of data and documentation

Huffman 10/16

“get data” hot links all lead to the data access page:

http://pmm.nasa.gov/data-access

GPM data access page

The actual download page lists sources available at GSFC

http://pmm.nasa.gov/data-access/downloads/GPM

Level 0: instrument countsLevel 1: instrument units (Tb) in original swaths

Level 2: geophysical units (precip) in original swaths

Level 3: griddedLevel 4: assimilated data

Huffman 10/16

other Level 3 products (click on arrowhead)

IMERG description, link to documentation

access is by Levels

headers have mouse-over descriptions of column headers

currently available data have the orange arrow

derived datasets are listed separately from re-formatted datasets

To summarize so far,

The inventory of precipitation-relevant satellites varies significantly with time

There are a number of estimates available from both IR and microwave dataCombined-sensor schemes are generally assumed to yield the best overall performance and come in two approaches:• CDR• HRPP

IMERG is a U.S. GPM algorithmThe PMM data access pages demonstrate that data producers are increasingly working to put out data sets in formats that users want

Huffman 10/16

Examples

GPCP V.2.2 SG climatology for 1979-2010 (CDR)

Note ITCZ, dry subtropical highs, mid-latitude storm tracks

Precipitation is concentrated around maritime continent

GPCP V2.2 Precipitation 1979-2010 (mm/d)

Huffman 10/16

Regionally coherent trends do exist• >0.7 mm/d/decade linear trend over 29 years, locally• the pattern appears to be driven by increases in ENSO frequency • data set inhomogeneities require careful examination

Examples (cont.)

Local linear trend in GPCP V.2.1 SG, 1979-2007 (CDR)

Huffman 10/16

Adapted from Kidd; Huffman 11/14

Louisiana, USA, 8-15 August 2016

Examples (cont.)

Near-real-time monitoring for extreme events (1/2)

Adapted from Kidd; Huffman 11/14

3-day heavy rains > 250 mm related to Hurricane Noel produces

• flooding (deduced by hydrologic model running globally in real time)

• landslides (estimated from real-time landslide potential algorithm)

Hispaniola, 1 November 2007, analyzed in real time by Global Hazard System (GHS), Adler and others, NASA funded research

Examples (cont.)

Near-real-time monitoring for extreme events (2/2)

How good?

Precipitation is easy to “measure,” but hard to analyze

• precip is intermittent and non-negative• precip is generated on the microscale• the decorrelation distances and times are short• point values only represent a small area; snapshots only represent a short time• a finite number of samples is a problem

Huffman 10/16

How good? (cont.)

Rainfall for Washington, DC area, July 1994 (in inches)

Convective rain has very short correlation distances - even for a month

Huffman 10/16

3-hoursnapshots

day

5-day

Month

IR vs. PMW (mm/hr) Feb. 2002 30°N-S

How good? (cont.)

At full resolution the correlation of estimated rain is low; averaging over time and spaceimproves the pictureWe provide the fine-scale data sousers get to decide on averaging strategy

Huffman 10/16

How good? (cont.)

Nearly coincident views by 5 sensors southeast of Sri LankaThe offset times from 00Z are given below the algorithm nameThe estimates are related, but differ due to

• time of observation• resolution• sensor/algorithm limitations

Huffman 10/16

How good? (cont.)

The problem of estimating errors for individual grid boxes is challenging

• the theory is not well-developed• necessary data are not always available from the input data sets• a first approximation is available for GPCP, TMPA, and IMERG monthly estimates• a legitimate daily and shorter error scheme is still under development• a full treatment must address the algorithm and sampling sources of error• goals:

- relate random errors across ranges of time/space scales- estimate bias

Huffman 10/16

Huffman 10/16

How good? IMERG V03 Final Run vs. 3B43 for June 2014

Same input satellites, different algorithms, different calibrator

Similar features, but not identical

• features (SPCZ)

• bias (ITCZ)

IMERG Final (mm/d) June 2014

TMPA 3B43 (mm/d) June 2014

MRMS = NOAA Multi-Radar Multi-SensorIMERG better• Wisconsin to

Nebraska • Idaho, Nevada

IMERG worse

• Northern MinnesotaRadar stops just off-shore; satellite doesn’t

How good? Daily 0.25° IMERG V03, 3B42 V7, MRMS for 15 June 2014

Huffman 10/16

[Courtesy J. Wang(SSAI; NASA/GSFC 612)]

Daily IMERG and Pocamoke Fine-Scale Grid, April-August 2014

23 surface gauges in a 6x5 km region near Wallops Island, Virginia

Excellent correlation for most events (warm season)Both over- and under-estimates for largest events

After Kidd; Huffman 10/16[Courtesy J. Tan (UMBC; WFF)] Days after 1 April 2014

Half-hourly IMERG V03 by source vs Pocamoke grid, April 2014 – March 2015

“Violin diagram” for individual sources of the half-hourly IMERG estimates

• width shows relative contribution for each difference bin

GMI is best; AMSR and SSMIS less so

The extra scatter for no-PMW (interpolated) is partly driven by the large number of cases

No-PMW (interpolated) data are competitive with the skill for most of the sensorsThis is pre-launch calibration! the shift to Version 4 should give more consistency

Number of cases

[Courtesy J. Tan (UMBC; GSFC)]

Huffman 10/16

This diagram focusessolely on heavy rain

All sensors are positively biased

• MHS is particularlybiased due to an IMERGerror

• “no PMW” (morphed andIR) is better

• again, low number ofsamples

This is pre-launchcalibration! the shift to Version 4 should givemore consistency

Half-Hourly IMERG V03 vs MRMS Radar/Gauge Product, 2-4 October 2015, South Carolina Floods

Actual accumulations of rain were up to 24”, but IMERG was high by a factor of 2

Huffman 10/16

Number of cases

[Courtesy J. Tan (UMBC; GSFC)]

Future prospects

Combination approaches undergoing vigorous development• to what degree will these be appropriate for or applied to historical data?• gauge data are critical as tie points for best accuracy

Varying amounts of microwave data will be available• despite GPM, LEO constellation will be sparser in the next 5 years• development will continue on GEO-IR schemes to fill gaps

- “cloud development” schemes try to capture pattern changes in GEO- GEO-multi-spectral is getting renewed interest

GPM is focusing attention on the difficult retrievals over icy/frozen surfaces• current proxies will be refined• “high-frequency” algorithms are being pushed

Error estimates will become increasingly important• we have a possible general framework for estimating random errors at arbitrary

time/space averaging volumes• estimates of bias error might fall out of the general framework

The long-term record must be improved to Climate Data Record standards• discontinuities and algorithm uncertainty hamper global change research

Huffman 10/16

Transitioning from Version 3 to Version 4

Version 3 IMERG is available• Final Run from mid-March 2014 to February 2016• Late Run from 7 March 2015• Early Run from 1 April 2015

Early November 2016: Version 4, first-generation GPM-based IMERG archive, March 2014–present

Mid-2017: Version 5 IMERG, March 2014–present

Late 2017: TRMM V.8/GPM V.5 TRMM/GPM-based IMERG archive, 1998–present

Winter 2017-18: Legacy TMPA products retired

Huffman 10/16

Future prospects (cont.)

Bigger data sets keep showing up• subsetting by parameter, location is becoming more standard• major archive sites are driving toward accessibility from anywhere

More non-expert users keep showing up• archives and developers need to provide more support

Publications need to reference data sets (in addition to algorithms)• authors have to be clear on the provenance of the data• Digital Object Identifier (DOI) is definitive• lacking that, identify

- developer / algorithm / version- producer / original dataset / version- archive / any reformatting- any “value-added” originators

Huffman 10/16

Final points

A variety of global precipitation estimates is available

Combination schemes are favored, but no single data set answers every need• trade-off between latency and accuracy

The next five years will see intense development, mostly of combinations• combination schemes have unfinished business• snowfall is still a work in progress

Archives are working to make data more accessible• some datasets are being put out in multiple formats• subsetting is starting to be offeredVersion 4 IMERG addresses a number of issues uncovered in Version 3

Versions will move quickly over the next 18 months

• Currently Version 3• GPM era reprocessed soon in Version 4, then a year from now in Version 5• TRMM-GPM eras reprocessed in Version 5 in mid-2017• TMPA to be run until Winter 2017-18

The future holds some “interesting” challenges, technical and institutionalHuffman 10/16

NASA/GSFC PMM Web Site: http://pmm.nasa.gov/

Contact: [email protected]

Slides: ftp://meso.gsfc.nasa.gov/agnes/huffman/IPWG8_Huffman_training-IMERG_slides.pptx

Movie: http://svs.gsfc.nasa.gov/cgi-bin/details.cgi?aid=4285Huffman 10/16

Animation located in IMERG-LE_1610020230.mp4

IPWG Dataset Tables

Organized according to input data types

• Table 1 – combinations of satellite data, with gauge data• Table 2 – combinations of satellite data• Table 3 – individual satellite data• Table 4 – precipitation gauge analyses

Columns give basic information about the datasets and a pointer to the archive

• algorithm name (with version, where possible)• a high-level list of the input data• the space/time grid on which the data are carried• the areal coverage and start date (and end date where updates are not routine)• how often updates occur• how close to observation time the data set represents• institution and person responsible and a (footnoted) link to the data

Huffman 10/16

APPENDIX: Acronyms and JargonAGPI Adjusted GPIAIRS Advanced IR SounderAMSR Advanced Microwave Scanning Radiometer (Japan)AMSR2 AMSR – 2AMSU Advanced Microwave Sounding UnitCDR Climate Data RecordCMAP CPC Merged Analysis of PrecipitationCMORPH Climate Prediction Center (CPC) Morphing AlgorithmDOI Digital Object IdentifierDMSP U.S. Defense Meteorological Satellite ProgramDPR Dual-frequency Precipitation RadarENSO El Niño/Southern OscillationGEO Geosynchronous Earth Orbit (also, a satellite in GEO)GHS Global Hazard SystemGMI GPM Microwave ImagerGPI Geosynchronous Operational Environmental Satellite (GOES) Precipitation Index GPM Global Precipitation Measurement missionGPCP Global Precipitation Climatology ProjectGSMaP Global Satellite Mapping of Precipitation (Japan)HRPP High-Resolution Precipitation ProductIMERG Integrated Multi-satellitE Retrievals for GPM (used to compute GPM datasets 3IMERGHH and

3IMERGM; runs include Early, Late, and Final)IPWG WMO/CGMS International Precipitation Working GroupIR InfraredITCZ Intertropical Convergence ZoneKF-CMORPH Kalman Filter CMORPHKML, KMZ Keyhole Markup Language, KML with Zip (compression)LEO Low Earth Orbit (also, a satellite in LEO)Level 0,1,2,3,4 0: sensor units, 1: instrument units, 2, 3: geophysical units, 4: assimilated data; 0-2: sensor

footprint locations, 3-4: griddedLT Local Time

Huffman 10/16

APPENDIX: Acronyms and Jargon (cont.)MERRA NASA Modern-Era Retrospective Analysis for Research and ApplicationsMHS Microwave Humidity SounderMRMS Multi-Radar Multi-Sensor precipitation product (NOAA)NASA National Aeronautics and Space AdministrationNOAA National Oceanic and Atmospheric AdministrationPERSIANN Precipitation Estimation from Remotely Sensed Information using Artificial Neural NetworksPERSIANN-CCS PERSIANN-Cloud Classification SystemPMM U.S. NASA Precipitation Measurement MissionsPMW Passive MicrowavePPS U.S. PMM Precipitation Processing SystemPR TRMM Precipitation RadarSG Satellite-Gauge combined precipitation estimate (usually refers to GPCP)SPCZ South Pacific Convergence ZoneSSMI Special Sensor Microwave/ImagerSSMIS Special Sensor Microwave Imager/SounderTb Brightness temperature (usually measured in Kelvin)TIFF Tagged Image File FormatTMI TRMM Microwave ImagerTMPA TRMM Multi-satellite Precipitation Analysis (used to compute TRMM datasets 3B42, 3B43,

3B42RT)TOVS Television-Infrared Operational Satellite (TIROS) Operational Vertical SounderTRMM Tropical Rainfall Measuring Mission V.1, V.2, … Version 1, Version 2, and so on1DD GPCP One-Degree Daily precipitation product3B42 TRMM Plus Other Satellite precipitation product3B42RT TRMM Real-Time VARHQ3B43 TRMM Plus Other Data precipitation product

Huffman 10/16

Documents

Combined Precipitation Algorithms – IMERG