Ming Hu 

*Developmental Testbed Center, NCAR-NOAA/GSD

**Cooperative Institute for Research in Environmental Sciences, Colorado University at

Boulder 

在科研中更好的使用美国业务资料分析系统（ GSI ）

Outline

General introduction to the Gridpoint Statistical Interpolation (GSI) system

Development Testbed Center and Community GSI History, repository, test, and review

committeeAvailable GSI resources for

communityO2R: website, document, tutorial, and

help deskR2O: community contributions

Current and future developmentCommunity GSI and operational GSI

2

General introduction to the Gridpoint Statistical Interpolation system

This section is mainly based on John Derber’s talk in 2013 summer GSI tutorial

3

Overview of GSI

John C. DerberNOAA/NWS/NCEP/EMC

4

History The Spectral Statistical Interpolation (SSI) analysis system was developed

at NCEP in the late 1980’s and early 1990’s. Main advantages of this system over OI systems were:

All observations are used at once (much of the noise generated in OI analyses was generated by data selection)

Ability to use forward models to transform from analysis variable to observations Analysis variables can be defined to simplify covariance matrix and are not tied

to model variables (except need to be able to transform to model variable) The SSI system was the first operational

variational analysis system system to directly use radiances

5

History

While the SSI system was a great improvement over the prior OI system – it still had some basic short-comingsSince background error was defined in spectral space – not

simple to use for regional systemsDiagonal spectral background error did not allow much

spatial variation in the background errorNot particularly well written since developed as a prototype

code and then implemented operationally

6

History

The Global Statistical Interpolation (GSI) analysis system was developed as the next generation global/regional analysis systemWan-Shu Wu, R. James Purser, David Parrish

Three-Dimensional Variational Analysis with spatially Inhomogeneous Covariances. Mon. Wea. Rev., 130, 2905-2916.

Based on SSI analysis systemReplace spectral definition for background errors with grid

point version based on recursive filters

7

HistoryUsed in NCEP operations for

RegionalGlobalHurricaneReal-Time Mesoscale AnalysisRapid Refresh (ESRL/GSD)

Operational at AFWAGMAO collaborationModification to fit into WRF and NCEP

infrastructureEvolution to ESMF

8

General Comments GSI analysis code is an evolving system.

Scientific advances situation dependent background errors -- hybrid new satellite data new analysis variables

Improved coding Bug fixes Removal of unnecessary computations, arrays, etc. More efficient algorithms (MPI, OpenMP) Bundle structure Generalizations of code

Different compute platforms Different analysis variables Different models

Improved documentation Fast evolution creates difficulties for slower evolving research projects

9

General Comments

Code is intended to be used OperationallyMust satisfy coding requirementsMust fit into infrastructureMust be kept as simple as possible

External usage intended to:Improve external testingReduce transition to operations work/timeReduce duplication of effort

10

Simplification to operational 3-D for presentationFor today’s introduction, I will be talking about using the GSI

for standard operational 3-D var. analysis. Many other options available or under development4d-varhybrid assimilationobservation sensitivityFOTOAdditional observation typesSST retrievalDetailed options

Options make code more complex – difficult balance between options and simplicity

11

Basic analysis problemJ = Jb + Jo + Jc

J = (x-xb)TB-1(x-xb) + (H(x)-y0)T(E+F)-1(H(x)-y0) + JC

J = Fit to background + Fit to observations + constraints

x = Analysisxb = BackgroundB = Background error covarianceH = Forward modely0 = ObservationsE+F = R = Instrument error + Representativeness errorJC = Constraint terms

12

Analysis variables

Background errors must be defined in terms of analysis variableStreamfunction (Ψ)Unbalanced Velocity Potential (χunbalanced)Unbalanced Temperature (Tunbalanced)Unbalanced Surface Pressure (Psunbalanced)Ozone – Clouds – etc.Satellite bias correction coefficients

13

Analysis variables

χ = χunbalanced + A Ψ

T = Tunbalanced + B Ψ

Ps = Psunbalanced + C ΨStreamfunction is a key variable defining a

large percentage T and Ps (especially away from equator). Contribution to χ is small except near the surface and tropopause.

14

Background fieldsCurrent works for following systems

NCEP GFSNCEP NMM – binary and netcdfNCEP RTMANCEP Hurricane (not using subversion version yet)GMAO globalARW – binary and netcdf – (not operationally used yet RR -

GSD)FGAT (First Guess at Appropriate Time) enabled up

to 100 time levels

15

Background ErrorsThree paths – more in talk by D. Kleist

Isotropic/homogeneousMost common usage. Function of latitude/heightVertical and horizontal scales separableVariances can be location dependent

Anisotropic/inhomogeneousFunction of location /stateCan be full 3-D covariancesStill relatively immature

Hybrid

16

ObservationsObservational data is expected to be in BUFR format

(this is the international standard)Each observation type (e.g., u,v,radiance from

NOAA-15 AMSU-A) is read in on a particular processor or group of processors (parallel read)

Data thinning can occur in the reading step.Checks to see if data is in specified data time window

and within analysis domain

17

Input data – Satellite currently used

RegionalAMSU-A

NOAA-15 Channels 1-10, 12-13, 15NOAA-18 Channels 1-8, 10-13, 15METOP-A Channels1-6, 8-13, 15AQUA Channels 5, 8-13Thinned to 60km

AMSU-B/MHSNOAA-15 Channels 1-3, 5NOAA-18 Channels 1-5METOP-A Channels 1-5Thinned to 60km

HIRSNOAA-17 Channels 2-15METOP-A Channels 2-15Thinned to 120km

AIRSAQUA 148 ChannelsThinned to 120km

 IASIMETOP-A 165 Channels

Globalall thinned to 145km

 

GOES-15 Sounders

Channels 1-15

Individual fields of view

4 Detectors treated separately

Over ocean only

AMSU-A

NOAA-15 Channels 1-10, 12-13, 15

NOAA-18 Channels 1-8, 10-13, 15

NOAA-19 Channels 1-7, 9-13, 15

METOP-A Channels 1-6, 8-13, 15

AQUA Channels 6, 8-13

ATMS

NPPChannels 1-14,16-22

MHS

NOAA-18 Channels 1-5

NOAA-19Channels 1-5

METOP-AChannels 1-5

HIRS

METOP-AChannels 2-15

AIRS

AQUA 148 Channels

IASI

METOP-A 165 Channels

18

Input data – Conventional currently used Radiosondes Pibal winds Synthetic tropical cyclone winds wind profilers conventional aircraft reports ASDAR aircraft reports MDCARS aircraft reports dropsondes MODIS IR and water vapor winds GMS, JMA, METEOSAT and GOES cloud drift IR and visible winds GOES water vapor cloud top winds

Surface land observations Surface ship and buoy observation SSM/I wind speeds QuikScat and ASCATwind speed and direction SSM/I and TRMM TMI precipitation estimates Doppler radial velocities VAD (NEXRAD) winds GPS precipitable water estimates GPS Radio occultation refractivity and bending angle profiles SBUV ozone profiles and OMI total ozone

19

Simulation of observations

To use observation, must be able to simulate observationCan be simple interpolation to ob location/timeCan be more complex (e.g., radiative transfer)

For radiances we use CRTM Vertical resolution and model top important

20

Quality controlExternal platform specific QCSome gross checking in PREPBUFR file creationAnalysis QC

Gross checks – specified in input data filesVariational quality control Data usage specification (info files)Outer iteration structure allows data rejected (or

downweighted) initially to come back in Ob error can be modified due to external QC marksRadiance QC much more complicated. Tomorrow!

21

Useful References Wan-Shu Wu, R. James Purser and David F. Parrish, 2002: Three-Dimensional Variational Analysis with Spatially

Inhomogeneous Covariances. Monthly Weather Review, Vol. 130, No. 12, pp. 2905–2916.  R. James Purser, Wan-Shu Wu, David F. Parrish and Nigel M. Roberts, 2003: Numerical Aspects of the Application of

Recursive Filters to Variational Statistical Analysis. Part I: Spatially Homogeneous and Isotropic Gaussian Covariances. Monthly Weather Review, Vol. 131, No. 8, pp. 1524–1535. 

R. James Purser, Wan-Shu Wu, David F. Parrish and Nigel M. Roberts, 2003: Numerical Aspects of the Application of Recursive Filters to Variational Statistical Analysis. Part II: Spatially Inhomogeneous and Anisotropic General Covariances. Monthly Weather Review, Vol. 131, No. 8, pp. 1536–1548. 

McNally, A.P., J.C. Derber, W.-S. Wu and B.B. Katz, 2000: The use of TOVS level-1B radiances in the NCEP SSI analysis system.  Q.J.R.M.S., 126, 689-724.

Parrish, D. F. and J. C. Derber, 1992: The National Meteorological Center's spectral statistical interpolation analysis system. Mon. Wea. Rev., 120, 1747 - 1763.

Derber, J. C. and W.-S. Wu, 1998: The use of TOVS cloud-cleared radiances in the NCEP SSI analysis system. Mon. Wea. Rev., 126, 2287 - 2299.

Kleist, Daryl T; Parrish, David F; Derber, John C; Treadon, Russ; Wu, Wan-Shu; Lord, Stephen , Introduction of the GSI into the NCEP Global Data Assimilation System, Weather and Forecasting. Vol. 24, no. 6, pp. 1691-1705. Dec 2009

Kleist, Daryl T; Parrish, David F; Derber, John C; Treadon, Russ; Errico, Ronald M; Yang, Runhua, Improving Incremental Balance in the GSI 3DVAR Analysis System, Monthly Weather Review [Mon. Weather Rev.]. Vol. 137, no. 3, pp. 1046-1060. Mar 2009.

Kazumori, M; Liu, Q; Treadon, R; Derber, JC, Impact Study of AMSR-E Radiances in the NCEP Global Data Assimilation System Monthly Weather Review,Vol. 136, no. 2, pp. 541-559. Feb 2008.

Zhu, Y; Gelaro, R, Observation Sensitivity Calculations Using the Adjoint of the Gridpoint Statistical Interpolation (GSI) Analysis System, Monthly Weather Review. Vol. 136, no. 1, pp. 335-351. Jan 2008.

DTC GSI documentation (http://www.dtcenter.org/com-GSI/users/index.php)

22

DTC comments

GSI is an operation systemWell-tested for operation applicationsVery strict data QCAdvanced operator for radiance

Bias correction is very important radiance data analysis

Model top and channel selectionData coverage for regional application

PrepBUFR and NCEP QC procedureThe operation system supported by

DTC for community users 23

DTC and Community GSI• Who is DTC• Community GSI: Goal and Building• Community GSI repository• Test and evaluation • GSI review committee

24

Who is DTC ?

Developmental Testbed Center (DTC)http://www.dtcenter.org/

Bridge between research and operational community

NWP systems supported by DTCWRF, HWRF, GSI, MET, … 25

Community GSI: GoalsProvide current operation GSI capability to research community (O2R)

Provide a pathway for research community to contribute operation GSI (R2O)

Provide a framework to enhance the collaboration from distributed GSI developers

Provide rational basis to operational centers and research community for enhancement of data assimilation systems 26

Community GSI: Building O2R:

Code portability: well-tested GSI code package for release

Documentation: GSI User’s GuideGSI User’s WebpageAnnual releaseOn-site Tutorial (lectures and Practical

cases)Help desk

R2O:Repository: Build, maintain, sync, testCommit community contributions

GSI Review Committee

The important role of GSI

repository and review

committee27

GSI Code Repository

GSI Trunk

Branch

BranchBranc

h

Tag

Branch

GDAS NDASCommunity release

HWRF

• Use tags or branches for: Release, new development, bug fix …

• Applications may use different revisions in the trunk (“snapshot”).

Which GSI should I use ? There is no “DTC GSI”, “EMC GSI” or “global GSI”. There is only one GSI! For a researcher, community release should be sufficient to use. If you are interested in getting new development back to the GSI trunk, contact GSI helpdesk (gsi_help@ucar.edu) get access to the developmental version of GSI.

28

GSI Review Committee

GSI Trunk

Branch

BranchBranc

h

Tag

Branch

NCEP/EMC

NASA/GMAO

NOAA/ESRL

NESDIS

DTC AFWANCAR/MMM

• DTC is working as the pathway between the research community and operational community.

• Provide visitor program for potential contributions to the operational code.

• Provide equivalent-operational tests of community contributions and provide rational basis for operational implementations.

• Represent research and operational community • Coordinate distributed GSI development • Conduct GSI Code Review

GSI Review Committee

29

History June, 2006:North American Mesoscale (NAM) System, NCEP May, 2007: Global Forecast System (GFS), NCEP 2007: DTC started to document GSI. 2009:

Created GSI code repository over NCEP and DTCFirst GSI release V1.0 Started user support

2010:First Community GSI tutorialCreated GSI Review Committee

2011First Community GSI workshop

30

31

Available GSI resources for community: O2R

•User’s Webpage•Documentation•Annual release package•Annual residential tutorial•Help desk

32

Community GSI – User’s Page

Support mainly through User’s Page and help desk:

http://www.dtcenter.org/com-GSI/users/index.php

33

Community GSI Release and Tutorial

The GSI User’s Guide and the on-line tutorial match with official release code

Summer 2013 GSI Tutorial:

NCEP, Aug. 5-7, 2013 34

Community GSI Release

35

Source code and fixed files were based on:

the GSI EMC trunk r19180 (May 10, 2012)

the community GSI trunk r826

The GSI User’s Guide and the on-line tutorial cases are updated with each official release.

Community GSI - Documents

User’s GuideMatching with each official release

Tutorial lectures

Code browserCalling tree

Publications36

Community GSI- User’s Guide V3.2

37

Community GSI- User’s Guide V3.2

38

Community GSI - Practice

On-line tutorial for each releaseResidential tutorial practice cases

39

Community GSI – tutorial and workshop

40

Community GSI - Help desk

gsi_help@ucar.edu

Any GSI related questions

Most of questions were answered by DTC staff

Forward complex questions to GSI developers

41

Available GSI resources for community: R2O

•Commit the community contributions to trunk•Help desk

42

GSI R2O Transition Procedure

Community research

Code development

candidate1

Code commitment

candidate(Branches)

2

3Code in

repository trunk

1. GSI Review Committee Scientific Review

2. Development, testing and merging

3. GSI Review Committee code and commitment review

(2011 Implementation)

43

GSI R2O applications

DTC contributes to R2O transition, coordinate & assist code commitment:

1. Mariusz Pagowski: Assimilation of surface PM2.5 observations in GSI for CMAQ regional air quality model

2. DTC: Portability issues from trunk head testing

3. Mariusz Pagowski: Add WRF-Chem model file as background for PM2.5 assimilation

4. Tom Augline: Add control and state variables for Cloudy analysis (triggered more work on the bundles of GMAO)

5. Zhiquan Liu: aerosol work has been merged to the trunk

6. GSD: Recent GSI developments for RR

44

CollaborationHFIP program: BUFR processing trainingAMB branch: GSI application for Rapid RefreshMariusz Pagowski (AMB): Assimilation of surface

PM2.5 observations in GSI for CMAQ regional air quality model

Don Birkenheuer (FAB): Thorpex work using GSIEMC GSI group (feedback from community GSI users)NCAR/MMM DA group (hybrid and cloud analysis,,

chemistry analysis ) Space Weather Prediction Center /NOAA (Houjun

Wang ) (whole atmosphere data assimilation using GSI)

Other GSI users (OU regional ENKF, PSD (Jeff Whitaker) global ENKF …)

45

R2O Summary

DTC also works with researchers to bring research community contributions back to the GSI operation repository (R2O)Create Trac to keep community

researchers in developing loopHelp researchers to review and merge

the code on top of the trunkTest the code with regression test suite

and initial the code review for committingSend your questions to

gsi_help@ucar.edu46

Current and future development

•Community GSI•Operational GSI

August 9th, 2013Committee Meeting Agenda8:45 GMAO 9:30 NESDIS 10:15 break 10:25 ESRL 11:10 NCAR 11:55 lunch12:40 DTC 1:10 NCEP 3:00 Adjourn

47

Community GSIMaintaining the current support

GSI review committeeCommunity GSI trunk: sync, testAnnual release with code and updated

documentImproving GSI user’s WebsiteHolding on-site tutorialMaintaining HelpdeskHelp committing community

contributionsApplication tools and new instructions

to GSI (bundle, hybrid, …)Build operation-similar test system

(ongoing)

48

Ongoing and future EMC GSI development work

EMC GSI development teamUpdate Aug. 8, 2013

49

Use of observations

Prepare for METOP-B and CrIS implementation - Andrew Collard Implementation Aug. 20 to WCOSS Includes SEVIRI from Meteosat-10

Use of AURA MLS ozone profiles – Haixia Liu Testing of new version of real time data

underway Cloudy Radiance Assimilation – Emily

Liu, Yanqiu Zhu, Z. Zhang, Will McCarty (GMAO) Updates to trunk – Significant progress –

Testing without convection Improved structure of radiance

diagnostic files – Andrew Collard - TBD

Satellite wind usage upgrade –XiuJuan Su GOES hourly wind project

Ground based Doppler radar data – Shun Liu Conversion to dual-pol

Aircraft based Doppler radar data – Mingjing Tong In operations on WCOSS

SSMI/S data – Andrew Collard, Emily Liu, and Ellen Liang (NESDIS) To be included in Spring 2014

implementation Mesoscale surface observation –

Manuel Pondeca, Steve Levine GLERL surface reduction for lakes under

testing GOES-R/SEVERI radiances – Haixia Liu

Additional quality control under development

Channels used above low clouds CRTM – Paul VanDelst, David Groff,

Quanhua Liu(NESDIS), Yong Chen(NESDIS) Improvements continue

Improve use of significant levels in RAOB profiles – Wan-shu Wu Bug fixed

50

Assimilation techniquesHybrid for NAM – Wan-shu Wu

Work continues – results using global ensemble encouragingHybrid for HWRF – Mingjing Tong

HWRF implementation using global ensemble on WCOSSRegional Strong Constraint – David ParrishCloudy Radiance Assimilation – MinJeong Kim, Emily Liu,

Yanqiu Zhu, X. Zhang, Will McCarty(GMAO)See previous

Add 4d capability to hybrid – Daryl KleistIncluded in trunk – Experimentation now showing significant

impactAdd precip and cloud physics to strong constraint – Min-

Jeong Kim, Daryl KleistHurricane location ensemble in hybrid – Yoichiro Ota –

Russ Treadon

51

Maintenance and general improvements techniques

Maintenance of EMC GSI trunk - Mike Lueken Continues many upgrades completed

Improvement of RTMA application – Manuel Pondeca Continues

Improvement of NAM analysis – Wan-shu Wu Continues

Improvement of GFS analysis – Russ Treadon/Andrew Collard Continues – new resolutions and new observations

Radiance monitoring – Ed Safford Included in trunk and operations

Unification of data monitoring – Ed Safford Underway

FOV updates for new satellite instruments – George Gayno, Ninghai Sun (NESDIS)

Code Structure – Mike Lueken, Ricardo Todling (GMAO) Transfer to different machines – Daryl Kleist, Russ Treadon, Others

WCOSS transition completed – code much more general

52

Additional projects beginning for Sandy Supplemental

DA Focuses Clouds and precipitationAircraft dataModel parameterizationsIncreased resolution4d HybridImproved Ensembles

www.imsg.com

53

Final comments

54

Advantages of community GSI

GSI is a well-supported communalized operation data assimilation systemMany groups (EMC, GMAO, GSD, MMM,…)

are actively working on various new developments for several operation applications (GFS, NMMB, RR, RTMA, …)

DTC has built good expertise to support GSI

Has a well maintained repository to make the latest code available to friendly users

DTC welcome collaboration and support R2O

Users play an important role in improving the support

55

Challenges

GSI is a fast evolving systemHard to follow for some researchersNeed to build new expertise for DTC staff

Limited resources to support GSI adds many new functions each yearUsers application areas are extending Reduced funding for support All support staff are shared with other

projectsCollaboration are important but cost

56