Mesoscale & Microscale Meteorological Division / NCAR WRF Modeling System “Short” Tutorial John Michalakes, Jimy Dudhia, Wei Wang, Cindy Bruyere, Michael

Mesoscale & Microscale Meteorological Division / NCAR

WRF Modeling System“Short” Tutorial

John Michalakes, Jimy Dudhia, Wei Wang, Cindy Bruyere, Michael Duda, Dave Gill, William Skamarock

NCAR/MMM


Outline

• Part I– Overview of WRF Modeling System– Work through an example from on-line tutorial on

NYBlue

• Part II– WRF Software Overview and Usage


What is WRF?

• WRF: Weather Research and Forecasting Model– Used for both research and operational forecasting

• It is a supported “community model”, i.e. a free and shared resource with distributed development and centralized support

• Its development is led by NCAR, NOAA/GSD and NOAA/NCEP/EMC with partnerships at AFWA, FAA, NRL, and collaborations with universities and other government agencies in the US and overseas


WRF Dynamical Cores

• The Advanced Research WRF (ARW) and Nonhydrostatic Mesoscale Model (NMM) are dynamical cores– Dynamical core includes mostly advection, pressure-gradients,

Coriolis, buoyancy, filters, diffusion, and time-stepping

• Both are Eulerian mass dynamical cores with terrain-following vertical coordinates

• ARW support and development are centered at NCAR/MMM• NMM development is centered at NCEP/EMC and support is

provided by NCAR/DTC• Both are downloadable in the same WRF tar file• Physics, the software framework, and parts of data pre- and

post-processing are shared between the dynamical cores


Modeling System Components

• WRF Pre-processing System (WPS)– Real-data interpolation for NWP runs– Replaces old Standard Initialization (SI) - still maintained

• WRF-Var (3d-Var)• WRF Model• Graphics tools


WRF 3DVAR

• Supported data types– Conventional surface and upper air, wind profiler– Remote sensing data: Cloud-track winds, ATOVS

thickness, ground-based GPS TPW, SSM/I, SSM/T1, SSM/T2, SSM/I brightness temp, Quikscat ocean surface winds, radar radial velocity

• Two background error covariance models– NCEP model– UK / NCAR


Key features:• Fully compressible, non-hydrostatic (with hydrostatic

option)• Mass-based terrain following coordinate,

where is hydrostatic pressure, is column mass

• Arakawa C-grid staggering v u T u v

ARW Details

ts

t

,


Key features:• 3rd-order Runge-Kutta time-split integration• High-order positive-definite advection• Two-way interacting telescoping moving nests• Grid-nudging and obs-nudging (FDDA)

ARW Details


WRF as a Community Model

• Version 1.0 WRF was released December 2000• Version 2.0 May 2004 (NMM added, EM nesting released)

– Version 2.0.1 Jun 2004– Version 2.0.2 Jun 2004– Version 2.0.3.1 Dec 2004

• Version 2.1 August 2005 (EM becomes ARW)– Version 2.1.1 Nov 2005 (NMM released)– Version 2.1.2 Jan 2006

• Version 2.2 December 2006 (WPS released)– NMM nesting released in 2007– 2.2.1 released in Nov 2007

• Version 3.0 released in April 2008


Version 3

• Nesting & moving nests• Global WRF• Variable time step• New physics, CAM3 lw/sw radiation• Chemistry• Grid and obs. nudging (FDDA)• WPS (SI replacement)• WRF Domain Wizard (right)• New/enhanced support

– IBM (+Blue Gene), SGI, Cray, NEC, Apple, Linux, WindowsCCS, Sun, NVIDIA

– PGI, Intel, Pathscale, g95, gfortran, vendor

www.wrfportal.org



WRF-Var in the WRF Modeling System


User Support

• User Web pages:– Model

http://www.mmm.ucar.edu/wrf/users– Var system

http://www.mmm.ucar.edu/wrf/users/docs/user_guide_V3– Web-based WRF Users’ Forum

http://forum.wrfforum.com

• Documentation– Users’ Guide– Technical Note– Software documentation: http://www.mmm.ucar.edu/wrf/WG2/software_v2– Online tutorial

• Email: [email protected]


Practical Session

• Installation of WRF system on NYBlue• Work through a real-data end-to-end case

Note: the tutorial notes haven’t been updated to WRFV3 yet, but will note differences as we encounter them


Blue Gene Specifics• BlueGene/L has two types of node

– Front-end nodes for interactive use– Compute nodes under batch system control– Programs must be compiled and run differently depending on where they run

• Front end– WPS and WRF utilities– NetCDF and NCAR graphics utilities

• Compute nodes– Components of modeling system itself

• All software installations in this tutorial will be compute node installations. These may be run on the compute nodes if you wish; they won’t work on the front-end

• I have installed a complete set of front-end apps. Add: /gpfs/home2/WRF/BG-Install/fe_tools/gnu32/bin to your path


• NYBlue Shell environment– bash

– csh or tsh

Blue Gene Specifics

export NETCDF=/gpfs/home2/WRF/BG-Install/netcdfexport PNETCDF=/gpfs/home2/WRF/BG-Install/pnetcdfexport NETCDF2=/apps/nco/netcdf-final/netcdf-3.6.2/bin/cdexport JASPERLIB=/gpfs/home2/WRF/BG-Install/JASPER/libexport JASPERINC=/gpfs/home2/WRF/BG-Install/JASPER/include

export PATH=/opt/ibmcmp/xlf/bg/10.1/bin:/gpfs/home2/WRF/BG-Install/fe_tools/gnu32/bin:$PATH

setenv NETCDF=/gpfs/home2/WRF/BG-Install/netcdfsetenv PNETCDF=/gpfs/home2/WRF/BG-Install/pnetcdfsetenv NETCDF2=/apps/nco/netcdf-final/netcdf-3.6.2/bin/cdsetenv JASPERLIB=/gpfs/home2/WRF/BG-Install/JASPER/libsetenv JASPERINC=/gpfs/home2/WRF/BG-Install/JASPER/include

set path=( /opt/ibmcmp/xlf/bg/10.1/bin /gpfs/home2/WRF/BG-Install/fe_tools/gnu32/bin $path )


Blue Gene Specifics• The WRF model and it’s preprocessors can be run as non-MPI

jobs; however, on Blue Gene we always run using MPI– Standard error and standard output from each MPI task appear as files:

rsl.error.<taskid> and rsl.out.<taskid> . “Error” in the file name is a misnomer; these will always appear and have useful data in them.

• Since node memory is constrained, single-reader/writer I/O is not recommended. Use parallel I/O instead (pNetCDF is format number 11 in the namelist.input – more later)

• 32-bit addressing is a limitation for very large problems, where size of an individual field is greater than 2GB.

• WRF (like most atmospheric models) is memory bandwidth hungry. So typically run in “CO” (not “VN”) mode on Blue Gene


NYBlue specific notes to Online TutorialDownload, Configure, and Build

1. Before starting, review previous slides on “Blue Gene Specifics” in this presentation (this will take care of the environment question under Configure WRF in the tutorial).

2. location of tutorial: http://www.mmm.ucar.edu/wrf/OnLineTutorial/index.htm3. GET Source Code: Source code is already downloaded in

~michalak/Tutorial_downloads (see README in that directory). If you use these files you may skip some of the steps below, since they already contain these NYBlue specific mods.

4. Configure WRF: Note there will only be one option: “Linux ppc64 BG blxlf compiler with blxlc (dmpar)”. If you are not using the tar files from step 3 above, modify the Blue Gene section of the arch/configure_new.defaults file so that it reads (change shown in red):

CPP = /opt/ibmcmp/xlf/bg/10.1/exe/cpp -C –P Also, to run on 10,000 nodes or more, the following changes must be made:

In external/RSL_LITE/rsl_lite.h, change the value of RSL_MAXPROC to 36864 (i.e., all the NY Blue nodes). In external/RSL_LITE/c_code.c, change all instances of rsl.out.%04 and rsl.error.%04 to rsl.out.%08 and rsl.error.%08

5. Configure WPS : make the same change as #4 above in the Blue Gene section of arch/configure.defaults for WPS if you are not using the tar files from #3. Also, if the environment is set up correctly (previous slides) you should be able to select configure option 1 (with GRIB2 support).


NYBlue specific notes to Online TutorialRun defaultcase end-to-end

1. Skip to January 2000 Case (http://www.mmm.ucar.edu/wrf/OnLineTutorial/CASES/JAN00/index.html) location of the on-line tutorial. The geog_10m.tar.gz, geog_5m.tar.gz, and geog_general.tar.gz terrestrial data files plus other files, including JAN00.TAR.gz,are in ~michalak/Downloads (see README in that directory)

2. The installed versions of all this data, both non-case-specific and case-specific for the Jan. 2000 case, are in ~michalak/DATA

3. Front-end executable versions of commands like plotgrids.exe, g1print.exe, and ncdump should work if the environment is set correctly (see previous slides). Ignore relative paths to these commands in the on-line tutorial, e.g. ./util/plotgrids.exe should be entered as just plotgrids.exe

4. Setup the Model domain (geogrid.exe). A sample namelist.wps and load leveler batch scripts, run_geogrid, run_ungrib, and run_metgrid are in ~michalak/Build/WRF/WPS

5. Interpolate the input data onto our model domain (metgrid.exe). Add “nocolons = .true.” (without quotes) to &share block of namelist.input, and then run metgrid.

6. Run the model (real.exe & wrf.exe). In addition to the settings shown for the namelist.input file, add “nocolons = .true.” (without quotes) to the &time_control part of namelist.input. Change every value of io_form_* from 2 to 11 (to specify parallel NetCDF). And add “io_form_auxinput1 = 11” (no quotes) to the other io_form settings. There is a sample namelist.input in ~michalak/Build/WRF/WRFV3/test/em_real as well as sample load leveler scripts run_real and run_wrf .

Documents

Mesoscale & Microscale Meteorological Division / NCAR WRF Modeling System “Short” Tutorial John Michalakes, Jimy Dudhia, Wei Wang, Cindy Bruyere, Michael