The Centre for Australian Weather and Climate ResearchA partnership between CSIRO and the Bureau of Meteorology

On the efficient tracking of grid point storm issues

Ilia Bermous

13 October 2011

2

Grid point storm issuesGrid point storm issues

The right modelling of physical processes and finite difference parameter settings to avoid grid point storm issues

Task of efficient handling/tracking grid point storm issues

Informative diagnostic to be able to track grid point storm issues

Efficient usage of computer resources when numerical analysis becomes unstable and produces rubbish

3

Ineffiency of the UM TRAP_W subroutine Ineffiency of the UM TRAP_W subroutine

Practical aspects

Implementation aspects

4

Ineffiency of the UM TRAP_W subroutine:practical aspects

Ineffiency of the UM TRAP_W subroutine:practical aspects

UM model uses the following capping condition

trap_option flag (0|1|2):

0 – reset, no diagnostic; 1 – reset + diagnostic

2 – diagnostic only

if trap_option=2 => additional meaningless (based on a free choice of Cw) output is produced, so nothing new seen so far

if trap_option=0|1 => the model uses artificial resetting (“cooking” in numerical analysis have not seen before), this kind of actions may only be useful for debugging purposes of the implemented algorithm

NOTE: in tracking down the grid point storm issue we have not been interested on how condition (1) was fullfield, we are interested on a max value of the vertical wind component w.

t

zCw kwijk

|| (1)

5

Ineffiency of the UM TRAP_W subroutine:practical aspects (cont #2)

Ineffiency of the UM TRAP_W subroutine:practical aspects (cont #2)

Cw=1000 – default setting; Cw=2&4 – in Gary’s last meeting report

Let’s consider 2 cases for the “Jan 13” original crash job

Cw=1 (job completes making 900 time steps => 75hour)w_max level proc position run w_max level timestep

0.377E+01 44 113 42.3% East 60.1% North 0.553E+01 46 872

Cw=10 (job completes)w_max level proc position run w_max level timestep

0.479E+01 41 160 40.4% East 81.9% North 0.186E+02 40 174

Cw time step at which W

started to reset

Total points for upward resetting and number of time steps used for

resetting

Total points for downward resetting and number of times

steps used for resetting

1 1 6,020,289 900 1,958,920 900

10 79 17,491 415 none

6

Ineffiency of the UM TRAP_W subroutine:implementation aspects

Ineffiency of the UM TRAP_W subroutine:implementation aspects

Let’s take a piece of a UM code referred by Gary:

IF ((k>Cw_test_lev).OR.(ic_xy(i,j)>0)) THEN ic_xy(i,j) = 1

ic = ic + 1

w_adv(i,j,k) = w_test(i,j)

k>Cw_test_lev condition should be moved outside and merged with the loop

Do k=model_levels-1,1,-1 & k>Cw_test_lev condition =>

Do k=model_levels-1,max(1, Cw_test_lev+1),-1

Condition ic_xy(i,j)>0 is redundant

There is a number of other redundant statements in the subroutine

7

Concepts of a new designConcepts of a new design 2 new terms have been introduced

w_soft_limit – value to trigger a diagnostic to be produced if |w| > w_soft_limit

w_hard_limit – value to stop execution if |w| > w_hard_limit condition is achieved on a PE

New development has been included in the sources by using a pre-processor logic with

#if defined (BOM_TRAP)

statements

Only 4 Fortran files have been modified ./atmosphere/dynamics_diagnostics/trap_w.F90./control/top_level/atm_step.F90./control/top_level/readlsta.F90./include/common/cruntimc.h

8

ResultsResults

New implementation was tested using UM7.5 R12 sources

Chris R12 xbawq job (13 Jan 2011 case) has been used

Here is some output with w_soft_limit=20 and w_hard_limit=100 setting

( “soft_limit” output string was used for an easy grapping)

w_max level proc position N points timestepsoft_limit 1.5E+02 56 149 48.7% East 78.4% North 4 218

On each PE a single output line for each level is produced with a k_total number of points for which the soft limit condition was fulfilled

Execution in this run was stopped after reaching w_hard_limit at time step 218 and 1407 soft_limit messages were produced from all PEs

. . ./dataw1 > grep "soft_limit " *fort6* | wc -l1407

9

Some numerical/algorithm aspects Some numerical/algorithm aspects

R12: Δt=300sec, 70 levels, 400m< Δzk<8000m, k>29 where grid point storm starts

capping

Cw=1 for k=70, Cw=10 for k=30

Semi-implicit finite-difference schema stabilityΔt=600sec => |w| > 100 m/sec at T=3*Δtas |w| may grow during integration this shows that the chosen Δt is not small enough to satisfy the stability requirement of the finite-difference operators used in the modellingran case with Δt=60sec using “tip” from Stuart’s presentation (it takes ~9.5hours for 75h integration on 192 cores), note that for this value of Δt a different range of Cw values should be used for trap_w condition (1)

w_max level proc position run w_max level timestep0.886E+01 42 162 41.1% East 87.4% North 0.218E+02 37 700

wjiwjik

wijk CwCwt

zCw 3

270,,30,, 26||,

3

4||,||

10

ConclusionsConclusions

A new developed version of TRAP_W subroutine produces useful information on tracking relatively large values of w vertical wind component

A moderate size of the output data is produced

Numerical analysis is stopped when “unreal” values of vertical wind component w are produced

One of the possible ways to avoid a grid point storm problem is to use a relatively small time step Δt according to the stability requirement of the finite-difference operators used in the modelling Results comparison between Cw=10, Δt=300sec “cooked” case

and Δt =60sec shows some large instability near the domain boundary

11

Conclusions (cont #2)Conclusions (cont #2)

12

Conclusions (cont #3)Conclusions (cont #3)