Zavisa Janjic 1 Nonhydrostatic Multiscale Model on the B grid (NMMB): Global Runs Zavisa Janjic Tom...

Zavisa Janjic 1

Nonhydrostatic Multiscale Model on the B grid (NMMB): Global Runs

Zavisa JanjicTom Black

Ratko VasicDusan Jovic

+MMB

…

Zavisa Janjic 2

Nonhydrostatic Multiscale Modelon the B grid (NMMB)

Intended for wide range of spatial and temporal scales (from meso to global, and from weather to climate)Further evolution of WRF Nonhydrostatic Mesoscale Model (NMM)Built on NWP and regional climate experience (Janjic et al., 2001, MWR; Janjic, 2003, MAP; Janjic & Gall, 2012, NCAR)Pressure based vertical coordinate, nondivergent flow remains on coordinate surfaces

Zavisa Janjic 3

Nonhydrostatic Dynamics

Inviscid adiabatic equationsNo over-specification of w!

TS Difference between hydrostatic pressures at surface and top

Hydrostatic pressure

p Nonhydrostatic pressure

pRT Gas law

Hypsometric (not “hydrostatic”) Eq.

0

ss

ssst ss

v Hydrostatic continuity Eq.

Continued …

Zavisa Janjic 4

1p

Third Eq. of motion

ws

swtw

gdtdwg s

sv

11

p

ssp

tp

cT

ssT

tT

sp

s vv

ss

tgdtdz

w ss

v1 Vertical velocity definition,

nonhydrostatic continuity Eq.

vkv fpdtd

ss )1( Momentum Eq.

Thermodynamic Eq.

Zavisa Janjic 5

Conservation of important properties of the continuous system aka “mimetic” approach in Comp. Math. (Arakawa 1966, 1972, …; Jacobson 2001; Janjic 1977, …; Sadourny, 1968, … ; Tripoli, 1992 …)

Nonlinear energy cascade controlled through energy and enstrophy conservation (finite-volume)A number of properties of differential operators preservedQuadratic conservative finite differencingA number of first order (including momentum) and quadratic quantities conservedOmega-alpha term, transformations between KE and PEErrors associated with representation of orography minimizedMass conserving positive definite monotone Eulerian tracer advection

Discretization Principles

Zavisa Janjic 6

Coordinates and GridsGlobal lat-lonRegional rotated lat-lon, more uniform grid sizeArakawa B grid

h h h v vh h h

Pressure-sigma hybrid (Simmons & Burridge 1981, Eckermann 2008)Flow remains on hydrostatic pressure coordinate surfaces in case of nondivergent flowLorenz vertical grid

Zavisa Janjic 7

No splitting (for computational efficiency)Additive splitting inadequate for large RoSplitting with multi-step iterative schemes too expensive

“Stabilized” Adams-Bashforth for horizontal advection of u, v, T and Coriolis forceCrank-Nicholson for vertical advection of u, v, T (implicit) because of CFL problems with thin layersForward-Backward (Ames, 1968; Gadd, 1978; Janjic and Wiin-Nielsen, 1977, JAS) fast wavesImplicit for vertically propagating sound waves (Janjic et al., 2001, MWR; Janjic, 2003, MAP)

Time Stepping

Zavisa Janjic 8

Lateral and Polar Boundaries, Polar Filter

Regional domain lateral boundariesNarrow zone with upstream advection, no computational outflow BC for advectionBlending zone

Conservative global polar boundary conditionsPolar filter

“Decelerator,” tendencies of T, u, v, Eulerian tracers, divergence, dw/dtPhysics not filtered

Zavisa Janjic 9

Physics

Upgraded NCEP WRF NMM “Unified” physicsRRTM, GFDL radiationNOAH, LISS land surface modelMellor-Yamada-Janjic turbulence

Elevated subgrid surface dragCloud topped marine BL

GFS Gravity Wave DragFerrier, Zhao microphysicsBetts-Miller-Janjic convection

High resolution enabled

Full NEMS GFS physics

Zavisa Janjic 10

27 km Global NMMB27 km Global NMMB

12 km NAM NMMB

12 km NAM NMMB

4 km NAM-nest NMMB

9 km Igor NMMB

9 km Julia NMMB

Hypothetical NMMB Simultaneous Run Global [with Igor & Julia] and NAM [with CONUS nest]

Courtesy of DiMego et al.

Zavisa Janjic 11

2D very high resolution tests

0 0

0

0

0

0

0

0

0

Warm bubble, 100 m resolution

10

0

0

Cold bubble, 100 m resolution

-2.5

-2.5-2.5

-2.5

00000

00

0 0000

00

00 000

0

0 0 0 0

00

00

000

000 0 0 00000 000

00

000

0

0000

00

00

000 000

00

00

00

0

000

00

00

00

00

00

00

000 0

00

0

0

0

0

000 00

0

00

0

00

0

00

00

00000000

0

0

0

0

00

0

0

0

0

00

00

00

00000

00

000

0

00

0000 000

00

2.52.5

2.5

2.5

2.5

2.5

2.5

Full compressible NMM Analytical (Boussinesque) ARPS (Boussinesque)

Nonlinear mountain wave 400 m resolution

0.00

1.00

1000

.00

1295

.77

1591

.55

1887

.32

2183

.10

2478

.87

2774

.65

3070

.42

3366

.20

3661

.97

3957

.75

4253

.52

4549

.30

4845

.07

5140

.85

5436

.62

5732

.39

6028

.17

6323

.94

9000

17000

Normalized vertical momentum flux, 400 m resolution

Zavisa Janjic 12

Mountain waves, 8 km resolution

Convection, 1 km resolution

Zavisa Janjic 13

Atmospheric Spectrum

Numerical models generally generate excessive small scale noise

False nonlinear energy cascade (Phillips, 1954; Arakawa, 1966 … ; Sadourny 1975; …)Other computational errors

Historically, problem controlled by:Removing spurious small scale energy by numerical filtering, dissipationPreventing excessive noise generation by enstrophy and energy conservation (Arakawa, 1966 …)

Zavisa Janjic 14

Atmospheric Spectrum

Classical paper by Sadourny, 1975, JAS:

One does not even need an atmospheric model for that!

Zavisa Janjic 15

Atmospheric Spectrum

Instead (Sadourny, 1975, JAS):

Philosophy built into the design of the compact nonlinear advection schemes for semi-staggered grids (Janjic, 1984, MWR)

Zavisa Janjic 16

Formal accuracy alone does not solve the problemDifferent nonlinear noise levels (green scheme) with identical formal accuracy and truncation error (Janjic et al. 2011, MWR) Green scheme, still energy & (alternative) enstrophy conserving

Second order schemes

Fourth order schemes

116 days

Zavisa Janjic 17

Atmospheric SpectrumNastrom-Gage (1985, JAS) 1D spectra in upper troposphere and lower stratosphere from commercial aircraft dataNo spectral gapTransition at few hundred kilometers from –3 to –5/3 slope in the 102-103 km (mesoscale) rangeInertial range, 0.01-0.3 m s-1 in the –5/3 range, not to be confused with severe mesoscale phenomena!

104

108

103

105

102

102 101103

-3

-5/3

Zavisa Janjic 18

Atmospheric Spectrum

What would the real atmosphere do in a short range run in a limited area domain starting from analysis with truncated spectrum?

Short time for statistical equilibriumLarge scales cut-off by domain size

NMMB well qualified for investigating numerical spectra by design

Zavisa Janjic 19

Atmospheric Spectrum

Are model simulated spectra forced by sigma coordinate errors?

Are model simulated spectra just projections of topography spectrum?

Topography squared

4

5

6

7

8

9

10

11

12

-6.0 -5.5 -5.0 -4.5 -4.0 -3.5

log10(wavenumber)

log

10

(to

po

gra

ph

y s

qu

are

d d

en

sit

y)

mnts k -̂3 k -̂5/3

-3

-5/3

630 km 63 km

Spectrum of topography squared over North America

Zavisa Janjic 20

No Physics With Physics

-5/3 -5/3

-3 -3

● Spectrum in agreement with observed spun-up given physical (or spurious, e.g. sigma) sources on small scales.

● Small scale energy in short regional runs with controlled nonlinear cascade and suppressed noise sources, where from?

Flat bottom (Atlantic), NMMB, 15km, 32 levels, 48h run (Loops)

Zavisa Janjic 21

No Physics With Physics

Where is the small scale energy in the observed spectrum coming from?

Atlantic case, NMM-B, 15 km, 32 Levels, 36-48 hour average

0

1

2

3

4

5

6

7

8

-6.0 -5.5 -5.0 -4.5 -4.0 -3.5 -3.0

oceanphy3648

k^-3

k^-5/3

0

1

2

3

4

5

6

7

8

-6.0 -5.5 -5.0 -4.5 -4.0 -3.5 -3.0

oceannop3648

k^-3

k^-5/3

-5/3 -5/3

-3 -3

No Physics With Physics

Zavisa Janjic 2222

Loop,decaying 3D turbulence, Fort Sill storm, 05/20/77.

NMM-B, Ferrier microphysics, 1km resolution, 32 levels,112km by 112km by 16.4km, double periodic, Smagorinsky constant 0.32.

Note the hump with active convection

Spectrum of w2 at 700 hPa.

Zavisa Janjic 23

-5/3

Hours 3-4 average decaying 3D turbulence, Fort Sill storm, 05/20/77.

NMM-B, Ferrier microphysics, 1km resolution, 32 levels,112km by 112km by 16.4km, double periodic. Smagorinsky constant 0.32.

Spectrum of w2 at 700 hPa.

Zavisa Janjic 24

Parallel runs for testing and tuningInitialized from spectral GFS analyses

Compatibility issues between grid-point and spectral data (Gibbs phenomenon)

Verified against GFS analyses and climatology500 hPa Height Anomaly Correlation CoefficientsAlthough starting from “same” initial conditions, skill of NMMB and GFS forecasts often disparate

Major GFS update on July 28, 2010One year of parallel global forecasts before and after July 28, 2010

Global Scales

Zavisa Janjic 25

Global Resolution

“Resolution comparable to that of the GFS”

Txxx is the GFS triangular truncationΔλ≈360˚/(2 Txxx)Δφ≈Δλ cos(45˚)

Δx≈Δy at 45˚

Zavisa Janjic 26

Courtesy Dusan Jovic

Zavisa Janjic 27

Global NMMB 769 x 541 x 64 pts. vs. GFS T384 x 64

1 year 500 hPa Height Anomaly Correlation Coefficient vs. forecast time

NMMB initialized and verified using GFS analyses and climatology

NMMB comparable or lower resolution

Zavisa Janjic 28

Global NMMB 769 x 541 x 64 pts. vs. GFS T384 x 64

1 year 500 hPa Height Anomaly Correlation Coefficient vs. forecast time

NMMB initialized and verified using GFS analyses and climatology

NMMB comparable or lower resolution

Zavisa Janjic 29

Average of 32 GFS T574 dropout cases from 2011 (Alpert) ran with NMMB with 769x541x64 points from GFS and ECMWF analyses

Above dropout threshold!

From July 28 2010 GFS has 3.8 times more points & updated physics

500 hPa ACC Verification using GFS analyses and ERA Interim climatology

Courtesy of Dr. Vladimir Djurdjevic

Zavisa Janjic 30

NMMB with GFS analysis closer to GFS (14)NMMB with GFS analysis closer to NMMB with ECMWF analysis (14)Ties (2)Anomalous (2)

Courtesy of Dr. Vladimir Djurdjevic

Zavisa Janjic 31

Courtesy of Yuejian Zhu

Zavisa Janjic 32

Courtesy of Yuejian Zhu

Zavisa Janjic 33

Aerosol Climatology

Zavisa Janjic 34

Global NMMB 769 x 541 x 64 pts. vs. GFS T574 x 64

1 year 500 hPa Height Anomaly Correlation Coefficient vs. forecast time

NMMB initialized and verified using GFS analyses and climatology

From July 28 2010 GFS has 3.8 times more points & updated

Zavisa Janjic 35

Global NMMB 769 x 541 x 64 pts. vs. GFS T574 x 64

1 year 500 hPa Height Anomaly Correlation Coefficient vs. forecast time

NMMB initialized and verified using GFS analyses and climatology

GFS has 3.8 times more points

Global NMMB 769 x 541 x 64 pts. vs. GFS T574 x 64

1 year 500 hPa Height Anomaly Correlation Coefficient vs. forecast time

NMMB initialized and verified using GFS analyses and climatology

From July 28 2010 GFS has 3.8 times more points & updated physics

Zavisa Janjic 36

New testing strategyInsufficient computational resources for sustainable parallel NMMB tests with enhanced resolutionLong testing periods required in order to obtain robust representative average scores (a change bringing a noticeable improvement in one season may be detrimental in the other)

Increased Resolution

Zavisa Janjic 37

A sample of 105 cases over one year at three and a half day intervals created (~ 1/7 of the population)Both 00Z and 12Z initial data equally represented Since the cases are chosen randomly, it is expected that test results over this set of cases would be reasonably close to test results over a yearA number of initial data conversion algorithms, resolution settings and physical parameterization sensitivity studies carried out

Increased Resolution

Zavisa Janjic 38

PlanIncrease resolution to become comparable to that of GFSRetune the physics

1. Start from radiation (special kind of parameterization)2. Convection3. …

Increased Resolution

Zavisa Janjic 39

NMMB - RSWINNMMB - RLWIN

ECMWF - RSWINECMWF - RLWIN

NMMB - (RSWIN-RSWOUT)ECMWF - (RSWIN-RSWOUT)

NMMB - RLWTOAECMWF - RLWTOA

Zonal averages of radiation fluxes

forecast start: 2011122048h forecastNMMB vs ECMWF

Difference in RLWIN on north hemisphere50N-90N of about 50W/m^2

Courtesy of Dr. Vladimir Djurdjevic

Zavisa Janjic 40

GFS

NMMB

ECMWF

Mean downward long-wave radiation flux 6-12h forecast hour.

RRTM w. hydrometeor clouds

By far largest fluxes in NMMB!

Courtesy of Dr. Vladimir Djurdjevic

Zavisa Janjic 41

GFS

NMMB NMMB

NMMB

RH clouds, no ice Hydrometeor clouds, no ice

Courtesy of Dr. Vladimir Djurdjevic

Zavisa Janjic 42

ECMWF; Vertical section

Downward LW flux

NMMB; Vertical section

NMMB (F_ICEC=0); Vertical section

Downward LW flux

Downward LW flux

Problem with radiation-ice interaction, fallback to no ice Zhao microphysics

Zavisa Janjic 43

Global NMMB 1149 x 811 x 64 pts. vs. GFS T574 x 64

105 randomly chosen cases from 1 year

500 hPa Height Anomaly Correlation Coefficient vs. forecast time

NMMB initialized and verified using GFS analyses and climatology

New Linux cluster test

Global NMMB 1149 x 811 x 64 pts. vs. GFS T574 x 64

105 randomly chosen 00Z and 12Z cases over 1 year

500 hPa Height Anomaly Correlation Coefficient vs. forecast time

NMMB initialized and verified using GFS analyses and climatology

Zavisa Janjic 44

New Linux cluster test

Global NMMB 1149 x 811 x 64 pts. vs. GFS T574 x 64

105 randomly chosen 00Z and 12Z cases over 1 year

500 hPa Height Anomaly Correlation Coefficient difference vs. forecast time

NMMB initialized and verified using GFS analyses and climatology

Zavisa Janjic 45

New Linux cluster test

Global NMMB 1149 x 811 x 64 pts. vs. GFS T574 x 64

105 randomly chosen 00Z and 12Z cases over 1 year

500 hPa RMSE difference vs. forecast time

NMMB initialized and verified using GFS analyses and climatology

Zavisa Janjic 46

Tropics

What is the truth?Insufficient data, more weight on model in DASWeak connection between mass and windDirect circulationsGravity wavesExtrapolation undergroundDifferent gravity-inertia wave frequency errors and computational dispersion in different models (semi-implicit vs. explicit)Different geostrophic adjustmentEach model should have its own DAS for fair comparison

Zavisa Janjic 47

Janjic, Z., A. Wiin-Nielsen, 1977: On Geostrophic Adjustment and Numerical Procedures in a Rotating Fluid. J. Atmos. Sci., 34, 297–310. doi: http://dx.doi.org/10.1175/1520-0469(1977)034<0297:OGAANP>2.0.CO;2

Zavisa Janjic 48

Semi impliciit, 5 leapfrpg time steps

Correct

Zavisa Janjic 49

Global, nonhydrostatic, full physicsResolution 1149 x 811 x 64, ~ 22 km

500 processors, 7.9 wall clock min/day1000 processors, 4.7 wall clock min/day2000 processors, 3.3 wall clock min/day4000 processors, 2.8 wall clock min/day, (1 year of simulation in < 17 hours, climate studies?)

Scaling on Zeus

Zavisa Janjic 50

Global, nonhydrostatic, full physicsResolution 2305 x 1623 x 64, ~ 11 km (8 x more work than 1149 x 811 x 64, ~ 22 km)

3600 processors, 7.6 wall clock min/dayPerfect scaling!

Technology for ~ 10 km global resolution forecasts already exists!

Scaling on Zeus

Zavisa Janjic 51

Summary and Conclusions

New testing paradigm for high resolution global runsGlobal NMMB shows good results in medium range forecasting ~10 km resolution global forecasts?

Technology existsResolutions less than <10 km km possible soon