Generic parallelization strategies for data assimilation

06-10-11

Nils van Velzen, Martin Verlaan

Outline

● OpenDA ● Coupling of model to DA method● Parallelism of model and EnKF● Parallelism for black box models● Conclusions

06-10-11 2Generic parallelization strategies for data assimilation

OpenDA

• What is OpenDA?

– A generic toolbox for data-assimilation

– Set of interfaces that define interactions between components

– Library of data-assimilation algorithms

– Open source

• Why OpenDA?

– More efficient than development for each application

– Shared knowledge between applications

– Development of algorithms with e.g. universities

– Easier to test

06-10-11 3Generic parallelization strategies for data assimilation

OpenDA

• Object oriented design

– Classes, software building blocks

– Interface (set of functions suitable for all models, observations, etc)

06-10-11 4Generic parallelization strategies for data assimilation

OpenDA

● Formal form of a model

● State of model instance ● Instance state cannot be directly changed

only through the methods like:

GetState, AxpyState, Compute...● Algorithm has no knowledge on model

internals

06-10-11 5Generic parallelization strategies for data assimilation

dxdt

=M (x(t ) , u(t) , p ,w(t ))

x (t ) ,u (t) , p ,w(t )

OpenDA

06-10-11 6Generic parallelization strategies for data assimilation

EnKF semi parallel

06-10-11 7Generic parallelization strategies for data assimilation

ξif (tk )=M (ξi

a(tk−1))+wi (tk )

x f (tk )=1N

ξif (tk )

E f (tk )=[ξ1f (tk )−x f (tk ) ,ξ2f (t k)−x f (t k) , ... ,ξNf (t k)−x f (t k)]⋮

Semi parallelCommunication

volume

Cm+n N

0

ξia(t k)=ξi

f (t k )+K (t k )[ y (t k )−H (t k)ξif (t k )+v i(t k)] n N

● Only parallelize model steps:• Not scalable, often sufficient

EnKF semi parallel

06-10-11 8Generic parallelization strategies for data assimilation

● Lotos-euros air quality model

EnKF semi parallel

06-10-11 9Generic parallelization strategies for data assimilationA

● Generic semi parallel due to OO concepts

EnKF parallel

06-10-11 10Generic parallelization strategies for data assimilation

ξif (t k)=M (ξi

a (t k−1))+wi(t k )

x f (tk )=1N

ξif (tk )

ξia(t k)=ξi

f (t k )+K (t k )[ y (t k )−H (t k)ξif (t k )+v i(t k)]

● Stochastic model time steps

● Mean of Ensemble

● Update (for all ensemble members)

EnKF parallel

06-10-11 11Generic parallelization strategies for data assimilation

Column wise distribution

Separate Combined

n Np

n log2( p ) n log2( p )

n N log2( p−1)

Row wise distribution

Separate Combined

Cm

n Np

+n N log2( p−1)

0 Cm+n Np

00

0n Np

EnKF parallel

06-10-11 12Generic parallelization strategies for data assimilation

Column wise distribution

Separate Combined

n Np

n log2( p ) n log2( p )

n N log2( p−1)

Row wise distribution

Separate Combined

Cm

n Np

+n N log2( p−1)

0 Cm+n Np

00

0n Np

EnKF parallel

06-10-11 13Generic parallelization strategies for data assimilation

● WAQUA shallow water model● Comparison of parallelization strategies for RRSQRT (Roest et al.)

EnKF parallel

• Full parallel using OO concepts:

– All filters run same code

– State vectors are distributed (parallel vector)

– Operations on parallel vectorsby a parallel vector implementation

– NO need to change the model andfilter code

– All complexities hidden in genericsupport layers/implementations

06-10-11 14Generic parallelization strategies for data assimilation

Parallel computing with black box models

• Black box models– No change to model code– Data exchange using files– Note: disk can be slow/more data written than

needed

06-10-11 15Generic parallelization strategies for data assimilation

Parallel computing with black box models

• Swan model for wind generated waves– Operational model + DA for the north sea– Black box model– 1 hour, 8 cpu's 10 min

50% IO– EnKF implementation (?)– Parallelization of filter

06-10-11 16Generic parallelization strategies for data assimilation

Parallel computing with black box models

• Black box model is normal model– Semi parallel+full parallel– Note disk speed,– Use local disks:

• Faster• No sequential

bottleneck

06-10-11 17Generic parallelization strategies for data assimilation

Conclusions

• Generic parallelization stragegies due to object oriented programming concepts.

• Single filter implementation for sequential as parallel computing

• EnKF like algorithms need combination of parallel strategies

• Black box models and IO can be parallelized as well

06-10-11 18eneric parallelization strategies for data assimilation