GEOS–CHEM: Code Management and Benchmarking

GEOS–CHEM:Code Management and Benchmarking

Bob YantoscaSoftware EngineerAtmospheric Chemistry Modeling GroupHarvard University

GEOS-CHEM Users’ Meeting04 April 2005

Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking

2

Topics1. Recent history of GEOS–CHEM versions2. Procedure for updating GEOS–CHEM3. Rules and Responsibilities4. GEOS–CHEM 1–Year Benchmarks5. Future directions

6. Appendix: Downloading GEOS–CHEM


3

Part 1

GEOS–CHEM Recent History


4

Recent HistoryGEOS–CHEM versions are named vX–YY–ZZ

X is the major version number This is changed when a new GEOS–CHEM version contains

an update that is so revolutionary that it is incompatible with all previous versions.

YY is the the minor version number This is changed each time the model is released publicly

on the GEOS–CHEM website. ZZ is the release number

This is changed for each “internal release”. Very often when we have many updates to add to GEOS–CHEM; these will be added in several stages so that each internal release can be tested separately. Then after all internal releases have been tested, a public release will be made.


5

Recent HistoryGEOS–CHEM version numbers throughout history:

v1–YY, v2–YY, and v3–YY (all prior to 2000) Ancient history

v4–YY (2000 – 2003) Denotes the ability to run with GEOS–3 met fields

v5–YY–ZZ (2003) Denotes the installation of SMVGEAR II

v6–YY–ZZ (2004) Denotes the ability to run with GEOS–4 met fields

v7–YY–ZZ (Oct 2004 – present) Denotes new and much improved user interface,

updated diagnostics, new emissions, etc…


6

Recent HistoryIn the last year, GEOS–CHEM development has been particularly rapid. This was mostly driven by:

The need to make G–C compatible with GEOS–4 met fields The need to add science into G–C for the Near-Real-Time

simulations that we would produce for ICARTT

GEOS–4 met fields: Are produced w/ a different GCM than prior GEOS versions Are on a hybrid grid and not a pure-sigma grid Have a different time-stamping scheme than prior GEOS v’s Required substantial recoding of GEOS–CHEM


7

Recent HistoryUpdates required for GEOS–4 meteorology

Installed new fvDAS TPCORE transport code (B. Field) Mass-flux diagnostics had to be totally rewritten

Installed LLNL pressure-fixer for transport (B. Field) P-fixer “modifies” U & V so that the surface pressure

computed from the winds matches the met field pressure read from disk

Installed new convection code from NCAR (S. Wu) GEOS–4 convection fields are different than GEOS–3 New scheme (from MATCH) includes updrafts & downdrafts

GEOS–4 also required coding changes in several places…


8

Recent HistoryImportant Scientific Updates to GEOS–CHEM

Addition of aerosol tracers for full-chemistry & offline runs Hydrophilic & hydrophobic BC & OC (R. Park) Secondary organic aerosols (H. Liao & S. Chung, Caltech) Sea salt aerosols (B. Alexander) Dust aerosols (D. Fairlie)

New types of emissions EPA/NEI99 inventory for North America (R. Park & R. Hudman)

NOx, CO, ALK4, ACET, MEK, PRPE, C2H6, C3H8, CH2O, SO2, and SO4

SO2 emissions from ship exhaust (B. Alexander) AVHRR LAI can be turned on or off (M. Fu)


9

Recent HistoryCoding updates and improvements to GEOS–CHEM

Totally new user interface in v7-yy-zz All GEOS–CHEM switches are now set in one input file

Timeseries diagnostics updated for ICARTT Rewritten for consistency Parallelized with OpenMP for optimal execution

GEOS–CHEM was ported to the SGI/Altix platform Wetdep and cloud convection now parallelize on Altix

Other improvements Production & loss diagnostics were cleaned up Use more allocatable arrays to save memory Removing LOTS of “historical baggage”


10

Recent HistoryOngoing GEOS–CHEM development …

Mercury simulation (w/ U. Wash) Recoding for MPI parallelization (w/ JPL) Creation of a GEOS–CHEM adjoint (w/ U. Toronto, Caltech) Aerosol microphysics modules (w/ CMU) Interface with CMAQ model (w/ U. Houston, U. Tenn) Online data assimilation (w/ GMAO) GCAP: Interface with GISS–GCM winds (w/ GISS) And many other projects!!

This requires that we have a clearly-defined procedure for updating GEOS–CHEM!!


11

Part 2

Updating GEOS–CHEM


12

Updating GEOS–CHEMGEOS–CHEM “standard code” is kept at Harvard

Bob Y. is the “Model Programmer” and maintains the “standard” GEOS–CHEM code

Version control is done with a CVS source code repository Changes from the last released version are documented

All changes to the “standard code” must go through Bob Y. This prevents divergent versions from floating around This also ensures the code will be computationally optimized

Each “standard code” release is benchmarked with a 1-month full-chemistry simulation (more on this later)

For some versions we also run a 1-year benchmark simulation


13

Updating GEOS–CHEMGEOS–CHEM modification process:

1. Users start by downloading a “standard code” version Available via GEOS–CHEM web site (see Appendix)

2. Users add new science into GEOS–CHEM (and test it!) Please test for 2x2.5, 4x5 (and maybe 1x1) grids Please test for different met fields (e.g. GEOS–3, GEOS–4)

3. When revisions are mature, users submit updates to Bob Y. for inclusion in the next “standard code” release Do not submit unless code is 100% ready to go Please also include any relevant documentation and

required data or other input files


14

Updating GEOS–CHEMGEOS–CHEM modification process:

4. Bob Y. will add users’ revisions into the “standard code”

5. Bob Y. will test the “standard code” with a 1–month full-chemistry benchmark simulation Benchmark month is 4 x 5 GEOS–3 for July 2001

6. Daniel Jacob must approve the 1-month benchmark results

7. Bob Y. will release the “standard code” and update the documentation on the web site Notification will be made via GEOS–CHEM mailing list


15

Updating GEOS–CHEM Full chemistry benchmark (July 2001) GEOS-3 meteorology w/ 30 layers @ 4x5 The following types of plots are created:

Tracer ratios w/ previous version (surface, 500hPa) Frequency distribution of tracer and AOD ratios J-Value ratios w/ previous version (surface, 500hPa) Tracer concentrations (surface, 500hPa) and zonal means Budgets for Ox and CO; CH3CCl3 lifetime Emissions table (comparison to previous version) Aerosol optical depths at surface Aerosol optical depth ratios (comparison to previous

version) Results are compared to the previously released

version


16

Part 3

Rules and Responsibilities


17


GEOS–CHEM user responsibilities Maintain communication with other users Update to new “standard code” versions when

available Do not distribute non-standard code versions Give credit to developers in publications Consult documentation pages on the G–C website Become familiar with debugging techniques Be resourceful! Try to isolate locations in the code

where bugs or errors are encountered before reporting to Bob Y.


18


GEOS–CHEM developer responsibilities Start from the latest available “standard code” release Provide new code for future “standard code” releases

Code should adhere to GEOS–CHEM style guide Provide related data and/or input files:

For 1x1, 2x25 and 4x5 horizontal grids For GEOS–1, GEOS–STRAT, GEOS–3, and GEOS–4 vert. grids Also provide any IDL routines etc. used to create these data

Provide PROFUSE documentation for new code & data files!

Your humble “Model Programmer” isn’t a chemist Think of future users when writing documentation Journal references are very helpful to include in source

code!


19


GEOS–CHEM users and developers should familiarize themselves with the following: Fortran 90

GEOS–CHEM is written in this language

IDL Visualization tools (e.g. GAMAP) are written in IDL netCDF, HDF, HDF–EOS files can be read with IDL

Some shell scripting (e.g. tcsh)

Advanced users: Learn Perl!


20


Why is GEOS–CHEM written in Fortran? Fortran is the de-facto standard for scientific computing

Most existing scientific code is written in Fortran Fortran is much easier to learn than C, C++, Java, etc.

C can be cryptic; Fortran is more English-like Fortran was designed for scientific computation

Fortran has more math functions & data types than C Fortran code optimizes better than C code Most Fortran compilers are OpenMP compatible Fortran executes faster than IDL

Fortran 90 brings many improvements over Fortran 77 You can write structured code with modules


21

Part 4

1–Year Benchmarks


22

1–Year Benchmarks 1–month GEOS–CHEM benchmarks are useful for: Tracking changes from one revision to the next Discovering coding errors Ensuring code is ported correctly to new platforms

Shortcomings of 1–month benchmarks 1 month is often not long enough for certain species It is a model-to-model comparison; not model-to-

data


23

1–Year Benchmarks At the last GEOS–CHEM meeting it was decided to also incorporate a 1–year GEOS–CHEM benchmark GEOS–CHEM versions are compared against

observed data

1–year benchmarks are performed when: A new version of GEOS–CHEM contains a significant

scientific change from a previous version A new set of GMAO met fields is released and we

need to compare how GEOS–CHEM performs w/r/t the previous met fields (e.g. GEOS–4 vs. GEOS–3)


24

1–Year Benchmarks Description of 1–year benchmark simulations: Coupled NOx–Ox–HC–aerosol simulation Simulation year: 2001 (with 2000 for spin-up) Horizontal Resolution: 4° latitude x 5° longitude Vertical Resolution:

GEOS–3: 30 level pure-sigma grid (lumped from 48 levels)

GEOS–4: 30 level hybrid grid (lumped from 55 levels) Gas-phase chemical tracers:

NOx, Ox, PAN, CO, Lumped >=C4 Alkanes, ISOP, HNO3, H2O2, MEK, Acetaldehyde, Lumped Aldehyde >=C3, MVK, Methacrolein, PMN, PPN, Lumped Alkyl Nitrate, >=C3 Alkenes, C3H8, HCHO, C2H6, N2O5, HNO4, Methyl Hydro Peroxide


25

1–Year Benchmarks Description of 1–year benchmark simulations: Aerosol tracers:

DMS, SO2, SO4, MSA, NH3, NH4, NIT, Hydrophilic BC, Hydrophilic OC, Hydrophobic BC, Hydrophobic OC, Dust Aerosol (4 size bins), Sea Salt Aerosol (2 size bins)

Secondary Organic Aerosols: ALPH, LIMO, ALCO, SOA1, SOA2, SOA3, SOG1, SOG2, SOG3

Chemical Mechanism: SMVGEAR II (Jacobson et al, 1997), using ~90 chemical

species, ~300 kinetic reactions (including rapid recycling of peroxides as well as H2SO4—HNO3—NH3 chemistry), and 51 photolysis reactions

Photolysis Mechanism: FAST–J algorithm (Wild et al, 2000)


26

1–Year Benchmarks Description of 1–year benchmark simulations: Operations:

Transport (Lin & Rood, 1996, 1997) Cloud Convection

GEOS–3: (Lin, 1996) GEOS–4: (Rasch, 2001)

Boundary Layer Mixing (Allen, 1996) Emissions Dry Deposition (Jacob et al, 1995) Chemistry

SMVGEAR II + aerosol chemistry schemes Wet Deposition (Jacob et al, 2000)


27

1–Year Benchmarks Description of 1–year benchmark simulations: Diagnostic output (saved as monthly means):

Optical Depths and Noontime J(O3), J(NO2) Anthropogenic, biomass, and biogenic emission fluxes Dry deposition fluxes and velocities Sources of CO, NOx, and Acetone Concentrations of chemically produced OH, NO, NO2,

HO2 Tracer concentrations Chemical production & loss of Ox and CO Lifetime of Methylchloroform (CH3CCl3) Selected meteorological quantities (U, V, P, T, etc.) Air masses, grid box heights


28

1–Year Benchmarks Description of 1–year benchmark simulations: The following plots are created:

Model vs. aircraft data (several missions) for: C2H6, C3H8, CO, H2O2, HNO3, NO, O3, PAN

Model CO vs. MOZAIC CO observations Model CO vs. ship track data Model column CO from vs. observations Model CO vs. CMDL CO Model O3 vs. MOZAIC O3 observations Model O3 vs. ozonesonde data Model O3 vs. surface O3 observations


29

1–Year Benchmarks Results from different 1–year benchmark simulations are usually plotted together against observations

Jennifer Logan is in charge of examining 1–year benchmark results

Jennifer will speak about our most recent 1–year benchmark simulations later today.


30

Part 5

Future Directions


31

Future Directions v7–02–04 will be released shortly after this mtg Option to use EPA / NEI 1999 emissions over USA Mercury simulation with online ocean fluxes PBL mixing has been centralized into a single module Planeflight diag. can now read daily files (a la ICARTT) Various bug fixes / improvements

GAMAP v2–03 will also be released shortly Can now read/write to COARDS-compliant netCDF files Other various improvements and minor fixes


32

Future Directions Earth System Model Framework (ESMF) Is a package of libraries which define “couplings”

between models and/or model components If two models are ESMF compliant then they can

“talk” to each other using a common interface Individual components in the same model may also

connect to each other via ESMF interfaces

GEOS–CHEM is amenable to ESMF compliance It is not ESMF compliant at this moment but… We will move towards ESMF when the need arises GEOS–CHEM coding style does not preclude ESMF


33

Part 6

Appendix


34

Downloading GEOS–CHEM

We distribute GEOS–CHEM as a complete software package, with the following components:

Source code and Makefiles Run directories containing user-modifiable input files Data directories containing emissions etc. files Directories containing meteorological field files Web-based documentation Visualization tools (GAMAP)

These components are available via the Internet GEOS–CHEM website FTP server (geos.as.harvard.edu)


35


Source Code Is available from the GEOS-CHEM Website

http://www-as.harvard.edu/chemistry/trop/geos/ Click on Model Versions / Get Code and Data in menu

You must supply the proper password Click on “1-month full chemistry benchmark run

package” This will download GEOS-CHEM.v7-xx-yy.stdrun.tar.gz

Uncompress the file with: gunzip GEOS-CHEM.v7-xx-yy.stdrun.tar.gz

Extract the contents with: tar –xvf GEOS-CHEM.v7-xx-yy.stdrun.tar


36


GEOS-CHEM.v7-xx-yy.stdrun.tar contains Code.v7-xx-yy: Dir w/ GEOS–CHEM source code IDL: Dir w/ IDL routines used to print benchmark results run.fullchem.geos3: Run directory with input files and

1-month benchmark results (more on this later!)

GEOS—CHEM supported compilers SGI MipsPro Linux: Intel (both PC and Altix versions) and PGI HP/Compaq IBM/AIX Sun/SPARC


37


Run directories Are also available from the GEOS-CHEM Website

http://www-as.harvard.edu/chemistry/trop/geos/ Click on Model Versions / Get Code and Data in menu

Supply the user password Click on “4x5 run directories” or “2x25 run

directories” GEOS-CHEM.v7-xx-yy.rundir4x5.tar.gz GEOS-CHEM.v7-xx-yy.rundir2x25.tar.gz

Uncompress the files with: gunzip GEOS-CHEM.v7-xx-yy.rundir4x5.tar.gz, etc…

Extract the contents with: tar –xvf GEOS-CHEM.v7-xx-yy.rundir4x5.tar, etc…


38


GEOS-CHEM.v7-xx-yy.rundir4x5.tar contains: run.geos1.4x5: Dir w/ input files for 4x5 GEOS–1 run.geoss.4x5: Dir w/ input files for 4x5 GEOS–

STRAT run.geos3.4x5: Dir w/ input files for 4x5 GEOS–3 run.geos4.4x5: Dir w/ input files for 4x5 GEOS–4

GEOS-CHEM.v7-xx-yy.rundir2x25.tar contains run.geos1.2x25: Dir w/ input files for 2x2.5 GEOS–1 run.geoss.2x25: Dir w/ input files for 2x2.5 GEOS–S run.geos3.2x25: Dir w/ input files for 2x2.5 GEOS–3 run.geos4.2x25: Dir w/ input files for 2x2.5 GEOS–4


39


Important input files contained in the GEOS–CHEM run directories:

File for setting GEOS–CHEM simulation options: input.geos

For SMVGEAR chemistry mechanism: globchem.dat, mglob.dat, chemga.dat

For FAST-J photolysis: ratj.d, jv_spec.dat, jv_atms.dat

Files containing initial concentrations restart.YYYYMMDDHH (e.g. restart.2001010100)


40


GEOS–CHEM data directories Available via FTP download from geos.as.harvard.edu Contact Jack Yatteau ([email protected]) for access These directories contain “shared” data files (e.g.

emissions, etc.) which are too large to fit in individual users’ space

Data directory structure on geos.as.harvard.edu: /data/GEOS_4x5/: Root-level dir for 4x5 simulation /data/GEOS_2x2.5/: Root-level dir for 2x2.5 simulation /data/GEOS_1x1_CH/: Root-level dir for 1x1 China sim. /data/GEOS_1x1_NA/: Root-level dir for 1x1 N. Am. sim. Data files are grouped into subdirectories by category


41


GEOS–CHEM met field dirs on geos.as.harvard.edu

4x5 met field directories: /data/GEOS_4x5/GEOS_1/YYYY/MM/ /data/GEOS_4x5/GEOS_S/YYYY/MM/ /data/GEOS_4x5/GEOS_3/YYYY/MM/ /data/GEOS_4x5/GEOS_4_v4/YYYY/MM/

2 x 2.5 met field directories: /data/GEOS_4x5/GEOS_1/YYYY/MM/ /data/GEOS_4x5/GEOS_S/YYYY/MM/ /data/GEOS_4x5/GEOS_3/YYYY/MM/ /data/GEOS_4x5/GEOS_4_v4/YYYY/MM/


42


GEOS–CHEM met field dirs on geos.as.harvard.edu

1x1 Nested Grid met field directories: /data/GEOS_1x1_CH/GEOS_3/YYYY/MM/ /data/GEOS_1x1_NA/GEOS_3/YYYY/MM/

NOTES: GEOS–1, GEOS–STRAT, GEOS–3 data files are

gzipped Each month of GEOS–4 data files are stored as a

gzipped TAR file in order to save disk space


43


Visualization tools GAMAP: Global Atmospheric Model Analysis Package

Suite of routines written in IDL Requires IDL version 5.3 or higher

GAMAP routines can be downloaded via the web http://www-as.harvard.edu/chemistry/trop/gamap/ Online documentation is also available at this website

Advantages of using GAMAP Tested on both PC’s and Unix machines; is very stable You can use GAMAP routines with other IDL code IDL has very powerful mapping/imaging capabilities Routines are provided for reading HDF, HDF–EOS,

netCDF files


44


Advantages of using GAMAP GAMAP contains a user-friendly menu-driven interface GAMAP has been tested on PC’s and Unix machines You can combine GAMAP routines with other IDL code

Advantages of using IDL IDL has very powerful mapping/imaging capabilities IDL has routines to read HDF, HDF–EOS, netCDF files You can develop an interactive web interface for

GAMAP with IDL–ION software (sold separately!) See our near-real-time web page (L. Jaegle)


45


Web-based documentation Available through the GEOS–CHEM website

http://www-as.harvard.edu/chemistry/trop/geos/ Click on the Documentation section of the main menu

Types of documentation GEOS–CHEM Frequently-Asked Questions GEOS–CHEM Online Manual GEOS–CHEM Style Guide


46


GEOS–CHEM Frequently-Asked Questions What’s new in GEOS–CHEM? What kinds of simulations can G–C perform? What is the horizontal and vertical resolution of G–C? What kinds of met fields can be used with G–C? What diagnostic quantities can be archived w/ G–C? How is the chemistry mechanism defined in G–C?


47


GEOS–CHEM Online Manual Introduction Installation and compilation Coding: Practice and style

F90 modules Parallelization

Files in the shared data directories Short description of files and their creators References

Files in the run directories Debugging


48


GEOS–CHEM Style Guide Guidelines for writing readable, well-documented code Headers, declarations, indentations, use of white

space New language features of Fortran 90 (including

modules) Fixed-format vs. free-format Numeric and character data types used in GEOS–

CHEM Converting from number to character and vice versa Math optimizations DO Loops

Documents

GEOS–CHEM: Code Management and Benchmarking