GSI Data Assimilation System

GSI Data Assimilation System

WRF-DA Tutorial, 20-22 July, 2009, Boulder, CO

Acknowledgement: NOAA, AFWA, NCAR

Hui Shao, Ming Hu, Laurie Carson, Louisa Nance, Xiang-Yu Huang, Bill Kuo

Developmental Testbed Center

John Derber, Russ TreadonNOAA/NCEP/Environmental Modeling Center

• NOAA/NCEP/EMC– John Derber, Jim Purser, Russ Treadon, Wan-Shu

Wu, Dave Parrish, Lidia Cucurull, Dave Parrish, Manuel Pondeca, Paul van Delst, Daryl Kleist, Xiujuan Su, Yanqiu Zhu, and others

• NASA/GMAO– Ricardo Todling, Ron Errico, Runhua Yang, Ron

Gelaro, Wei Gu, and others

• And NOAA/GSD, NCAR/MMM,…

Primary Developers

2

• GSI System and Community Support (30 minutes, presented by Hui Shao)

• Installation, Running, and Diagnostics (30 minutes, presented by Ming Hu)

Session Outline

3

GSI System

4

•3D variational assimilation (same as WRF-VAR)

J =1/2 (x-xb)TB-1(x-xb) + 1/2(H(x)-y0)T(E+F)-1(H(x)-y0) + JC

Fit to background + Fit to observations + constraints

x = Analysis (Output)xb = Background (Input)B = Background error covariance (Input)H = Forward model (within GSI)y0 = Observations (Input)E+F = R = Instrument error + Representativeness error (Input)JC = Constraint terms (within GSI)

Gridpoint Statistical Interpolation (GSI)

5

System Implementation date Primary Developers

Physical SST retrieval 9/27/2005 NCEP/EMC*

NAM (regional) 6/20/2006 NCEP/EMC*

RTMA 8/22/2006 NCEP/EMC*

GFS (Global) 5/1/2007 NCEP/EMC*

HWRF 6/19/2007 NCEP/EMC*

Rapid Refresh (RR) 2010 NOAA/GSD

AFWA operational 2010 or 2011 NCAR/MMM

* Collaborated with NASA/GMAO and others

Operational GSI Applications

6

NAMHWRF

7

Rapid Refresh Domain Air Force Weather Agency Domains

User input &initializations

Read in & distributeobservations, background

& background errors

Additionalinitializations

Outer loopa) Set up right hand side of analysis equation

(Compute observation innovations)b) Call inner loop

i. Compute gradient informationii. Apply background erroriii. Compute search directioniv. Compute step sizev. Update analysis increment

Write analysis & related output

GSI Code Flow

8

• Currently works for the following systems‒Global

• GFS• GMAO global

‒Regional • WRF (ARW & NMM) - binary and netcdf• NCEP RTMA

Background Fields (xb)

9

• Radiosondes• Pibal winds• Synthetic tropical cyclone winds• Wind profilers• Conventional aircraft reports• ASDAR aircraft reports• MDCARS aircraft reports**• Dropsondes• MODIS IR and water vapor

winds• GMS, METEOSAT and GOES

cloud drift IR and visible winds• GOES water vapor cloud top

winds

• Surface land observations*• Surface ship and buoy observation• SSM/I wind speeds• QuikScat wind speed and direction• SSM/I precipitable water• SSM/I and TRMM TMI

precipitation estimates• Doppler radial velocities• VAD (NEXRAD) winds• GPS precipitable water estimates• GPS Radio occultation refractivity

profiles• SBUV ozone profiles (other ozone

data under test)

*Some of the data are restricted

Observations(yo)

10

• Regional– GOES-11 and 12 Sounders – thinned

to 120km Channels 1-15

Individual fields of view 4 Detectors treated separately Over ocean only

– AMSU-A – thinned to 60km NOAA-15 Channels 1-10, 12, 15

NOAA-18 Channels 1-8, 10-11, 15

– AMSU-B/MHS – thinned to 60km NOAA-15 Channels 1-3, 5 NOAA-16 Channels 1-5 NOAA-17 Channels 1-5 NOAA-18 Channels 1-5

– HIRS – thinned to 120km NOAA-17 Channels 2-15

• Global– GOES-11 and 12 Sounders –thinned to

180km Channels 1-15 Individual fields of view 4 Detectors treated separately Over ocean only

– AMSU-A – thinned to 145km NOAA-15 Channels 1-10, 12-13, 15

NOAA-18 Channels 1-8, 10-13, 15 METOP Channels 1-13, 15 AQUA Channels 1-6, 8-13, 15

– AMSU-B/MHS – thinned to 240km NOAA-15 Channels 1-3, 5 NOAA-16 Channels 1-5 NOAA-17 Channels 1-5 NOAA-18 Channels 1-5 METOP Channels 1-5

– HIRS - thinned to 180km NOAA-17 Channels 2-15 METOP Channels 2-15

– AIRS – thinned to 180km AQUA 148 Channels

Observations(yo) (cont.)

11

• To use observation, GSI simulates observation using analysis variables – observation operator (H) Can be simple interpolation to ob location/time. Can be more complex (e.g., radiation transfer).

For radiances, GSI uses CRTM.

Observation Operator (H)

12

• Space correlation computed using recursive filters in horizontal and vertical

• Multivariate relation• Flow dependent variability in background error

– Background error variances modified based on 9 and 3 hour forecast differences:

• Variance increased in regions of rapid change• Variance decreased in “calm” regions • Global mean variance ~ preserved

– Being used for regional (US) surface analysis operationally.

Background Errors (B)

13

Surface pressure backgrounderror standard deviation fields

Valid at 00Z November 06, 2007

HPC Surface Analysis

a) without re-scaling

b) with flow dependent re-scaling

rescaled

“as is”

L

14

Option 1: univariate

• Temperature (blue) increment forces large increment in RH (shaded).

Option 2: multivariate

• Temperature (blue) increment

forces increment in q (red).

• much smaller RH (shaded)

increment.

Moisture analysis

• Improved specification of observational errors– Adaptive Tuning

Observation Errors

16

After tuning

Before tuning

After tuning and smoothing

• External platform specific QC• Some gross checking in PREPBUFR file creation• Optimal interpolation quality control (OIQC) – on its way out• Analysis QC

– Gross checks – specified in input data files– Variational quality control (VarQC) – implemented

operationally in Feb 2009

Observation Quality Control

17

Number of data rejected by OIQC VarQC weight (W)

• NCEP uses a two step process for Tb

– Scan angle correction – based on position– Air mass correction – based on predictors

• Predictors– mean– path length (local zenith angle determined)– integrated lapse rate– integrated lapse rate ** 2– cloud liquid water

Bias Correction of Radiance Data

18

DMSP15 July2004 : 1month before bias correction after bias correction

19V

22V

19H

37V

37H

85H

85V

B-O Histogram

19

Community Support

20

System Implementation date

Mode

Physical SST retrieval

9/27/2005 CRTM + analytical solution

NAM (regional) 6/20/2006 3D-VAR

RTMA 8/22/2006 2D-VAR

Global 5/1/2007 3D-VAR

HWRF 6/19/2007 3D-VAR

RR Early 2010

AFWA operational

2010 or early 2011

Research Community

Goals:•Provide current operational GSI capability to the research community (O2R)•Provide a framework for distributed development of new capabilities & advances in data assimilation (R2O)

DTC

Community GSI Code

21

• Free and shared resource• Ongoing distributed development by both

research and operational communities• Maintained under version control• Periodic releases made available to the

community• Includes latest developments of new

capabilities and techniques• Centralized support• Provided in collaboration with developers

Definition of Community Codes

22

Tasks Timeline Note

Beta release (Q1FY09)

Jun, 2009 Friendly user only (through the GSI website)

First release (Q1FY09)

Sep, 2009

Tutorial Jun, 2010

With user support

Residential tutorial and hand-on practical session

GSI Community Release Timetable

23

Success of this structure will depend on communication and collaboration among all GSI developers, users, and DTC.

Community RepositoryCommunity Repository

DTCDTC

releaserelease CommunityCommunity

DevelopersDevelopers

Code Management

Plan

NCEP EMC

Repository

NCEP EMC

Repository

Community GSI Code Repository

Proposed Structure

24

• Release announcement

• System component descriptions

• Documentation– User’s Guide– Presentations

• Registration

• Software downloads

• Bug fix reports

• User support information– [email protected]

• Tutorial information

User’s Websitehttp://www.dtcenter.org/com-GSI/users/

25

mailto:[email protected]

User’s Guide will be updated to be consistent with each new release to the community.

GSI User’s Guide

26

On-line Tutorial (Beta Release)

27

On-line Documents (Beta Release)

28

GSI Web Brower

29

30

O2R2O2R2O2R2O2R2O …

3131

GSI: Compile

3232

• System required libraries – FORTRAN 90/95 compiler

– C compiler

– Perl

– netCDF

• GSI system– Download GSI system tar files (GSIbeta.tar.gz ) from

http://www.dtcenter.org/com-GSI/users/index.php

– gunzip and untar

tar –zxvf GSIbeta.tar.gz (Should see GSI/. directory )

– cd to GSI directory

cd GSI

Requirements

33

makefile Top-level makefile README General information about GSI code arch/ Directory where compile options are gathered clean script to clean created files and executables compile script for compiling WRF code configure script to configure the configure.wrf file for compile sorc Directory for GSI and library source code fix/ Directory containing static/fixed data files lib/ Directory created for libraries include/ Directory created for include/module files run/ Directory for executable, and example script

Compile scripts

Compile rules

Source code directory

Run directory

sorc

makefile mains libs

bacio crtm_gfsgsi bufr sfcio gfsio sigio w3 mpeu makefile sp

GSI Directory

3434

• WRF_DIRWRF needs to be compiled prior to compiling GSI

GSI uses WRF I/O API libraries to do file input and output

WRF directory specified:

setenv WRF_DIR /home/user/WRFV3

• netCDFIf netCDF libraries are not located in the standard

/usr/local , thensetenv NETCDF “path for netCDF”

For LINUX systems, make sure the netCDF libraries are installed using the same compiler (PGI, Intel, g95) that will be used to compile GSI.

Set environment

3535

• To create a GSI configuration file for your computer:

./configure

• This script checks the system hardware and software (mostly netCDF), and then offers the user choices for configuring GSI:

Choices for 32-bit LINUX operated machines are: 1. Linux i486 i586 i686, PGI compiler 2. Linux i486 i586 i686, Intel compiler 3. Linux i486 i586 i686, gfortran compiler

Choices for IBM machines are: 1. AIX xlf compiler with xlc

Configuring GSI

3636

• configure.gsi – Created by the ./configure command

– contains compilation options, rules, etc.

– specific to your computer

– can be edited to change compile options, if desired.

• At this time, the IBM option is well tested. Working on Linux option test.

• The arch/configure.defaults file can be edited to add a new option if needed.

Configuring GSI, cont.

3737

• To compile:./compile >& compile_gsi.log

• To get compile help message:./compile -h

• If the compilation is successful, it will create one executable under bin/:

gsi.exe

Compiling GSI

3838

• To remove all object files and executables./clean

• To remove all built files, including configure.gsi./clean –a

Clean is recommended if compilation failed want to change configuration file

Clean Compilation

39

Running GSIUser’s Guide: Chapter 3

40

• GSI Executable

• Background (first guess) file

• Observations– Not needed for single observation experiment

• Fixed files (within GSI package)

• Run script (namelist included)

To run GSI, you need:

41

• GSI can use– WRF NMM input file in binary format– WRF NMM input file in netcdf format– WRF ARW input file in binary format– WRF ARW input file in netcdf format– GFS input file in binary format– GMAO global model input file in binary format

• DTC has only tested regional analysis with WRF input– On IBM: both binary and nedcdf format– On Linux: netcdf format only

Background

42

• All observations have to be in BUFR format

• prepbufr: NCEP flavor BUFR – Need NCEP BUFR library

Observation

43

• Collection of statistic and control files under fix directory– Background and observation errors

• berror_stats, errtable

– Observation data control file (info files)• convinfo , satinfo

– Bias correction used by radiance analysis • satbias_angle, satbias_in

– Radiance coefficient used by CRTM • EmisCoeff.bin, CloudCoeff.bin

Fixed files

44

– Ask for computer resources to run GSI– Set environment variables for the machine

– Set experiments variables (experiment name, analysis time, background and observation)

– Check the definition of required variables

– Generate a run directory for GSI (working or temp directory)

– Copy GSI executable to run directory– Copy background file to run directory– Copy or link observations to run directory– Copy fixed files to run directory– Generate namelist for GSI

– Run the GSI executable

– Save the GSI analysis results

Run script - structure

45

# analysis time (YYYYMMDDHH)

ANAL_TIME=2008051112

# working direcotry, where GSI runs

WORK_ROOT=./gsi/case

# path and name of background file BK_FILE=./2008051112/bkARW/wrfout_d01_2008-05-11_12:00:00

# path of observations OBS_ROOT=./gsi/case

PREPBUFR=./tutorialcases/data/newgblav.gdas1.t12z.prepbufr.nr

# path of fix files FIX_ROOT=./GSI/fix

# path and name of the gsi executable GSI_EXE=./GSI/bin/gsi.exe

# which background error covariance and parameter will be used (GLOBAL or NAM) bkcv_option=NAM

Run Script - Experiment variables

46

• Run directory– Create a new directory for each run

• Copy or link data– Executable– Background

• Must be copied, will be over-written by analysis result

– Observations• Link or touch if not exist

– Fixed files• Link

Run Script - run directory

47

• Generated by running scripts– Change namelist by editing run script

• Works for both global and regional analysis• Detailed explanation in section 3.4

Run Script : namelist

&SETUP miter=2,niter(1)=5,niter(2)=5, write_diag(1)=.true.,write_diag(2)=.false.,write_diag(3)=.true., qoption=1, gencode=78,factqmin=0.005,factqmax=0.005,deltim=$DELTIM, ndat=59,npred=5,iguess=-1, oneobtest=.false.,retrieval=.false.,l_foto=.false., use_pbl=.false., $SETUP / &GRIDOPTS JCAP=$JCAP,NLAT=$NLAT,NLON=$LONA,nsig=$LEVS,hybrid=.true., wrf_nmm_regional=.false.,wrf_mass_regional=.true., diagnostic_reg=.false., filled_grid=.false.,half_grid=.true.,netcdf=.true., regional=.true.,nlayers(63)=3,nlayers(64)=6, $GRIDOPTS

48

Tuning and DiagnosticsUser’s Guide: Chapter 4

49

• Includes lots of important information

• First place to look after any GSI run– If successful

• Data distribution• Optimal iteration• Maximum and minimum of analysis fields

– If fails, • which part of GSI has problem• what is the possible reason for failure

Standard out file - stdout

50

• read in all data and prepare analysis:– read in configuration (namelist)– read in background– read in constant file (fixed file)– read in observation– partition background and observation data for

parallel analysis

• optimal iteration (analysis)

• save analysis result

stdout - structure

51

0: rmse_var=T

0: ordering=XYZ

0: WrfType,WRF_REAL= 104 104

0: ndim1= 3

0: staggering= N/A

0: start_index= 1 1 1 785191608

0: end_index= 69 64 45 -1603623772

0: k,max,min,mid T= 1 309.9411316 264.5114136 289.7205811

0: k,max,min,mid T= 2 310.6200562 269.5698547 295.0413208

……….

0: k,max,min,mid T= 45 510.0127563 472.5627441 494.7407227

stdout – example(1)

0: Read_SpcCoeff_Binary(INFORMATION) : FILE: amsua_n15.SpcCoeff.bin;

0: SpcCoeff RELEASE.VERSION: 7.01 N_CHANNELS=15

0: Read_SpcCoeff_Binary(INFORMATION) : FILE: amsub_n15.SpcCoeff.bin;


0: Read_SpcCoeff_Binary(INFORMATION) : FILE: hirs3_n16.SpcCoeff.bin;


0: Read_SpcCoeff_Binary(INFORMATION) : FILE: amsua_n16.SpcCoeff.bin;


Top of stdout:read in background

End of stdout:write out analysis results

Read CRTM

coefficient

52

6: READ_LIDAR: bufr file date is 2007 12 20 0

11: READ_AIRS: bufr file date is 2007 12 20 0

5: READ_RADAR: vad wind bufr file date is 2007 12 20 0

1: READ_PREPBUFR: bufr file date is 2007 12 20 0

9: READ_BUFRTOVS: bufr file date is 2007 12 20 0 mhsbufr

5: n,lat,lon,qm= 1 47.04 246.01 -9 -9 -9 -9 -9 -9 -9 -9 2 3 -9 3 -9 2 -9

4: READ_PREPBUFR: messages/reports = 872 / 97301 ntread = 1

stdout – example (2)

0: 0: GLBSOI: START pcgsoi

0: penalty,grad ,a,b= 1 0 0.160014047732364506E+05 0.724764165295960149E+06 0.132279098881E-02 0.00000000000E+00

0: pnorm,gnorm, step? 1 0 0.100000000000E+01 0.10000000000E+01 good

0: penalty,grad ,a,b= 1 1 0.150427003330115986E+05 0.608359222451359266E+06 0.124401271687E-02 0.83938151711E+00

0: pnorm,gnorm, step? 1 1 0.940086232815E+00 0.83938982252E+00 good

Read observations

0: PROGRAM GSI_ANL HAS ENDED. IBM RS/6000 SP

Iteration

53

• A good way to check – ratio of background and observation variance– pattern of background error covariance

• Setup in namelist– SETUP

oneobtest=.true.,

– SINGLEOB_TESTmaginnov=1.0,magoberr=1.0,oneob_type='t',oblat=20.,oblon=285.,obpres=850.,obdattim= 2007081500,obhourset=0.,

Single Observation Test

54

• From namelist and the links of observation files dfile(01)='prepbufr‘, dtype(01)='ps‘, dplat(01)=' ', dsis(01)='ps' dfile(02)='prepbufr‘ dtype(02)='t', dplat(02)=' ', dsis(02)='t', dfile(27)='amsuabufr‘,dtype(27)='amsua‘,dplat(27)='n15‘,dsis(27)='amsua_n15', dfile(28)='amsuabufr‘,dtype(28)='amsua‘,dplat(28)='n16‘,dsis(28)='amsua_n16',

• From info file (convinfo …)otype type sub iuse twindow numgrp ngroup nmiter gross ermax ermin var_b var_pg ps 111 0 -1 1.5 0 0 0 5.0 3.0 1.0 10.0 0.000 ps 132 0 -1 1.5 0 0 0 5.0 3.0 1.0 10.0 0.000 ps 180 0 1 1.5 0 0 0 5.0 3.0 1.0 10.0 0.000 ps 181 0 1 0.25 0 0 0 5.0 3.0 1.0 10.0 0.000 ps 182 0 1 1.5 0 0 0 5.0 3.0 1.0 10.0 0.000 ps 183 0 -1 0.25 0 0 0 5.0 3.0 1.0 10.0 0.000 ps 187 0 1 0.25 0 0 0 5.0 3.0 1.0 10.0 0.000 ps 188 0 -1 0.25 0 0 0 5.0 3.0 1.0 10.0 0.000 t 120 0 1 1.5 0 0 0 7.0 5.6 1.3 10.0 0.000 t 126 0 -1 1.5 0 0 0 7.0 5.6 1.3 10.0 0.000 t 130 0 1 1.5 0 0 0 7.0 5.6 1.3 10.0 0.000 t 131 0 1 1.5 0 0 0 7.0 5.6 1.3 10.0 0.000 t 132 0 1 1.5 0 0 0 7.0 5.6 1.3 10.0 0.000

Control Data Usage

55

Observation fit - statistic files

56

• stdout0: 0: GLBSOI: START pcgsoi0: penalty,grad ,a,b= 1 0 0.160014047732364506E+05

0.724764165295960149E+06 0.132279098881E-02 0.00000000000E+000: pnorm,gnorm, step? 1 0 0.100000000000E+01 0.10000000000E+01 good 0: penalty,grad ,a,b= 1 1 0.150427003330115986E+05

0.608359222451359266E+06 0.124401271687E-02 0.83938151711E+000: pnorm,gnorm, step? 1 1 0.940086232815E+00 0.83938982252E+00 good

• More detail in fort.220

J= 0.000000000000000000E+00 0.000000000000000000E+00 0.000000000000000000E+00 0.000000000000000000E+00 0.111428969204742752E+05 0.262756287766784135E+04 0.256346518803503204E+04 0.921301367380113305E+02 0.244005730835839540E+04 0.000000000000000000E+00 0.000000000000000000E+00 0.000000000000000000E+00 0.000000000000000000E+00 0.000000000000000000E+00 0.000000000000000000E+00 0.000000000000000000E+00 0.000000000000000000E+00 0.000000000000000000E+00 0.253237990246203207E-01 0.225585966101596478E+04

20 items in cost function:5 wind observation 6 satellite radiance observation 7 temperature observation8 precipitable water obs. 9 specific humidity obs. 20 surface pressure observation

Convergence Information

57

Questions, Comments, Suggestions