Parallelizing finite element Parallelizing finite element PDE solvers in an object-PDE solvers in an object-

oriented frameworkoriented framework

Xing CaiXing Cai

Department of InformaticsDepartment of Informatics

University of OsloUniversity of Oslo

Outline of the TalkOutline of the Talk

• Introduction Introduction & background& background

• 33 parallelization approach parallelization approacheses

• Implementational aspectsImplementational aspects

• Numerical experimentsNumerical experiments

The Scientific Software GroupThe Scientific Software Group

Knut Andreas Knut Andreas Lie Lie (SINTEF)(SINTEF)

Kent Andre Kent Andre MardalMardal

Åsmund Åsmund ØdegårdØdegård

Bjørn Fredrik Bjørn Fredrik Nielsen Nielsen (NR)(NR)

Joakim Joakim SundnesSundnes

Wen ChenWen ChenXing CaiXing Cai

Øyvind Øyvind HjelleHjelle(SINTEF)(SINTEF)

Ola Ola SkavhaugSkavhaug

Aicha Aicha BounaimBounaim

Hans Petter Hans Petter LangtangenLangtangen

Are Magnus Are Magnus Bruaset Bruaset (NO)(NO)

Linda Linda IngebrigtsenIngebrigtsen

Glenn Terje Glenn Terje LinesLines

Aslak TveitoAslak Tveito

Part-timePart-timePh.D. Ph.D. Students Students

Post DocsPost DocsFacultyFaculty

Department of Informatics, University of OsloDepartment of Informatics, University of Oslohttp://www.ifi.uio.no/~tpv

TomTomThorvaldsenThorvaldsen

ProjectsProjects

Simulation of electrical activity in human Simulation of electrical activity in human heartheart

Simulation of the diastolic left ventricle Simulation of the diastolic left ventricle

Numerical methods for option pricing Numerical methods for option pricing

Software for numerical solution of PDEsSoftware for numerical solution of PDEs

Scientific computing using a Linux-clusterScientific computing using a Linux-cluster

Finite element modelling of ultrasound wave propagation Finite element modelling of ultrasound wave propagation

Multi-physics models by domain decomposition methods Multi-physics models by domain decomposition methods

Scripting techniques for scientific computingScripting techniques for scientific computing

Numerical modelling of reactive fluid flow in porous mediaNumerical modelling of reactive fluid flow in porous media

http://www.ifi.uio.no/~tpv

DiffpackDiffpack

• O-O software environment for O-O software environment for scientific computation scientific computation (C++)(C++)

• Rich collection of PDE solution Rich collection of PDE solution components - components - portable, flexible, extensibleportable, flexible, extensible

• http://www.nobjects.comhttp://www.nobjects.com

• H.P.Langtangen, H.P.Langtangen, Computational Computational Partial Differential EquationsPartial Differential Equations, , Springer 1999Springer 1999

The Diffpack PhilosophyThe Diffpack Philosophy

Structuralmechanics

Porous mediaflow

Aero-dynamics

Incompressibleflow

Other PDEapplications

Waterwaves

StochasticPDEs

Heattransfer

Field

Grid

MatrixVector

I/O

Ax=b

FEM

FDM

The QuestionThe Question

Starting point: sequential Starting point: sequential PDE PDE solversolverHow to do the parallelization?How to do the parallelization?

Resulting parallel solvers should haveResulting parallel solvers should have good parallel efficiencygood parallel efficiency good overall numerical performancegood overall numerical performance

We needWe need a good parallelization strategya good parallelization strategy a good and simple implementation of the strategya good and simple implementation of the strategy

A generic finite element PDE A generic finite element PDE solversolver

• Time stepping Time stepping tt00, , tt11, , tt22……

• Spatial discretization (computational Spatial discretization (computational grid)grid)

• Solution of nonlinear problemsSolution of nonlinear problems

• Solution of linearized problemsSolution of linearized problems

• Iterative solution of Iterative solution of Ax=bAx=b

An observationAn observation

• The computation-intensive part is the The computation-intensive part is the iterative solution ofiterative solution of Ax=bAx=b

• A parallel finite element PDE solver needs A parallel finite element PDE solver needs to run the linear algebra operations in to run the linear algebra operations in parallelparallel– vector additionvector addition

– inner-product of two vectorsinner-product of two vectors

– matrix-vector productmatrix-vector product

Several parallelization optionsSeveral parallelization options

• Automatic compiler parallelizationAutomatic compiler parallelization

• Loop-level parallelization (special Loop-level parallelization (special compilation directives)compilation directives)

• Domain decompositionDomain decomposition– divide-and-conquerdivide-and-conquer

– fully distributed computingfully distributed computing

– flexibleflexible

– high parallel efficiencyhigh parallel efficiency

A natural parallelization of PDE A natural parallelization of PDE solverssolvers

• The global solution domain is The global solution domain is partitioned into many smaller sub-partitioned into many smaller sub-domainsdomains

• One sub-domain works as a ”unit”, with One sub-domain works as a ”unit”, with its sub-matrices and sub-vectorsits sub-matrices and sub-vectors

• No need to create global matrices and No need to create global matrices and vectors physicallyvectors physically

• The global linear algebra operations The global linear algebra operations can be realized by can be realized by local operations + local operations + inter-processor communicationinter-processor communication

GridGrid ppartitionartition

Linear-algebra level parallelizationLinear-algebra level parallelization

• A SPMD modelA SPMD model

• Reuse of existing code for local linear Reuse of existing code for local linear algebra operationsalgebra operations

• Need new code for the parallelization Need new code for the parallelization specific tasksspecific tasks– grid partition (non-overlapping, grid partition (non-overlapping,

overlapping)overlapping)

– inter-processor communication routinesinter-processor communication routines

Object orientationObject orientation

• An add-on ”toolbox” containing all the An add-on ”toolbox” containing all the parallelization specific codesparallelization specific codes

• The ”toolbox” has many high-level routinesThe ”toolbox” has many high-level routines

• The existing sequential libraries are slightly The existing sequential libraries are slightly modified to include a ”dummy” interface, modified to include a ”dummy” interface, thus incorporating ”fake” inter-processor thus incorporating ”fake” inter-processor communicationscommunications

• A seamless coupling between the huge A seamless coupling between the huge sequential libraries and the add-on toolboxsequential libraries and the add-on toolbox

Straightforward ParallelizationStraightforward Parallelization

• Develop a sequential simulator, without Develop a sequential simulator, without paying attention to parallelismpaying attention to parallelism

• Follow the Diffpack coding standardsFollow the Diffpack coding standards

• Use theUse the add-on add-on toolboxtoolbox for parallel for parallel computingcomputing

• Add a few new statements for Add a few new statements for transformation to a parallel simulatortransformation to a parallel simulator

A Simple Coding ExampleA Simple Coding Example

GridPartAdm* adm; GridPartAdm* adm; // // access to parallelization functionalityaccess to parallelization functionality

LinEqAdm* lineq; LinEqAdm* lineq; // // administrator for linear system & solveradministrator for linear system & solver

// ...// ...

#ifdef PARALLEL_CODE#ifdef PARALLEL_CODE

adm->scan (menu);adm->scan (menu);

adm->prepareSubgrids ();adm->prepareSubgrids ();

adm->prepareCommunication ();adm->prepareCommunication ();

lineq->attachCommAdm (*adm);lineq->attachCommAdm (*adm);

#endif#endif

// ...// ...

lineq->solve ();lineq->solve ();

set subdomain list = DEFAULTset subdomain list = DEFAULT

set global grid = grid1.fileset global grid = grid1.file

set partition-algorithm = METISset partition-algorithm = METIS

set number of overlaps = 0set number of overlaps = 0

SSolving an elliptic PDEolving an elliptic PDE

)())(( xfuxK

• Highly unstructured gridHighly unstructured grid• Discontinuity in the coefficient Discontinuity in the coefficient KK

MeasurementsMeasurements

P # iter Time Speedup

1 480 420.09 N/A

3 660 200.17 2.10

4 691 156.36 2.69

6 522 83.87 5.01

8 541 60.30 6.97

12 586 38.23 10.99

16 564 28.32 14.83

•130,561 degrees of freedom130,561 degrees of freedom•Overlapping subgridsOverlapping subgrids•Global BiCGStab using (block) ILU prec.Global BiCGStab using (block) ILU prec.

Parallel Vortex-Shedding Parallel Vortex-Shedding SimulationSimulation

incompressible Navier-Stokesincompressible Navier-Stokes

solved by a pressure correction methodsolved by a pressure correction method

Simulation SnapshotsSimulation Snapshots

PressurePressure

Some CPU MeasurementsSome CPU Measurements

P CPU Speedup Efficiency

1 1418.67 N/A N/A

2 709.79 2.00 1.00

3 503.50 2.82 0.94

4 373.54 3.80 0.95

6 268.38 5.29 0.88

8 216.73 6.55 0.82

The pressure equation is solved by the CG methodThe pressure equation is solved by the CG methodwith “subdomain-wise” MILU prec.with “subdomain-wise” MILU prec.

Animated Pressure FieldAnimated Pressure Field

Domain DecompositionDomain Decomposition

• Solution of the original large problem Solution of the original large problem through iteratively solving many smaller through iteratively solving many smaller subproblemssubproblems

• Can be used as solution method or Can be used as solution method or preconditionerpreconditioner

• Flexibility -- localized treatment of Flexibility -- localized treatment of irregular geometries, singularities etcirregular geometries, singularities etc

• Very efficient numerical methods -- even Very efficient numerical methods -- even on sequential computerson sequential computers

• Suitable for coarse grained parallelizationSuitable for coarse grained parallelization

Overlapping DDOverlapping DD

Example:Solving the PoissonSolving the Poissonproblem on the unitproblem on the unitsquaresquare

ObservationsObservations

• DD is a good parallelization strategyDD is a good parallelization strategy• The approach is not PDE-specificThe approach is not PDE-specific• A program for the original global problem A program for the original global problem

can be reused (modulo B.C.) for each can be reused (modulo B.C.) for each subdomainsubdomain

• Must communicate overlapping point Must communicate overlapping point values values

• No need for global dataNo need for global data• Data distribution impliedData distribution implied• Explicit temporal schemes are a special Explicit temporal schemes are a special

case where no iteration is needed (“exact case where no iteration is needed (“exact DD”)DD”)

Goals for the ImplementationGoals for the Implementation

• Reuse sequential solver as subdomain Reuse sequential solver as subdomain solversolver

• Add DD management and Add DD management and communication as separate modulescommunication as separate modules

• Collect common operations in generic Collect common operations in generic library moduleslibrary modules

• Flexibility and portabilityFlexibility and portability

• Simplified parallelization process for the Simplified parallelization process for the end-userend-user

Generic Programming Generic Programming FrameworkFramework

Making the Simulator ParallelMaking the Simulator Parallel

class SimulatorP : public SubdomainFEMSolverclass SimulatorP : public SubdomainFEMSolver public Simulatorpublic Simulator{{ // // … just a small amount of code… just a small amount of code virtual void createLocalMatrix ()virtual void createLocalMatrix () { Simulator::makeSystem (); }{ Simulator::makeSystem (); }};};

SubdomainSimulatorSubdomainSimulator

SubdomainFEMSolver

AdministratorAdministrator

SimulatorPSimulatorP

SimulatorSimulator

ApplicationApplication Poisson equation on unit squarePoisson equation on unit square DD as the global solution methodDD as the global solution method Subdomain solvers use CG+FFTSubdomain solvers use CG+FFT Fixed number of subdomains Fixed number of subdomains MM=32 (independent of =32 (independent of

PP)) Straightforward parallelization of an existing Straightforward parallelization of an existing

simulatorsimulator P Sim. Time Speedup Efficiency

1 53.08 N/A N/A

2 27.23 1.95 0.97

4 14.12 3.76 0.94

8 7.01 7.57 0.95

16 3.26 16.28 1.02

32 1.63 32.56 1.02

P: number of processors

A large scale problemA large scale problem

P CPU Speedup MaxNi MinNi

1 283.93 N/A 2,082,625 2,082,625

2 150.56 1.89 1,053,505 1,047,265

4 69.23 4.10 532,417 529,705

8 31.37 9.05 273,722 269,865

16 16.88 16.82 141,418 139,793

32 10.19 27.86 75,617 70,834

Solving an elliptic boundary value problem on an unstructured gridSolving an elliptic boundary value problem on an unstructured grid

Combined ApproachCombined Approach

• Use a CG-like method as basic solverUse a CG-like method as basic solver(i.e. use a parallelized Diffpack linear solver)(i.e. use a parallelized Diffpack linear solver)

• Use DD as preconditionerUse DD as preconditioner(i.e. (i.e. SimulatorPSimulatorP is invoked as a preconditioning solve) is invoked as a preconditioning solve)

• Combine with coarse grid correctionCombine with coarse grid correction

• CG-like method + DD prec. is normally CG-like method + DD prec. is normally faster than DD as a basic solverfaster than DD as a basic solver

ElasticityElasticity Test case: 2D linear elasticity, 241 x 241 global grid.Test case: 2D linear elasticity, 241 x 241 global grid.

Vector equationVector equation ),( 21 uuu

fuu )(

Straightforward parallelization based on Straightforward parallelization based on an existing Diffpack simulatoran existing Diffpack simulator

2D Linear Elasticity2D Linear Elasticity

• BiCGStab + DD prec. as global solverBiCGStab + DD prec. as global solver

• Multigrid V-cycle in subdomain solvesMultigrid V-cycle in subdomain solves

• II:: number of global BiCGStab iterations needed number of global BiCGStab iterations needed

• PP:: number of processors ( number of processors (PP=#subdomains)=#subdomains)

P CPU Speedup I Subgrid

1 66.01 N/A 19 241 x 241

2 24.64 2.68 12 129 x 241

4 14.97 4.41 14 129 x 129

8 5.96 11.08 11 69 x 129

16 3.58 18.44 13 69 x 69

2D Linear Elasticity2D Linear Elasticity

Two-Phase Porous Media FlowTwo-Phase Porous Media Flow

P Total CPU Subgrid CPU PEQ I CPU SEQ

1 4053.33 241x241 3586.98 3.10 440.58

2 2497.43 129 x 241 2241.78 3.48 241.08

4 1244.29 129 x 129 1101.58 2.97 134.28

8 804.47 129 x 69 725.58 3.93 72.76

16 490.47 69 x 69 447.27 4.13 39.64

psv

Tqps

Tsfvst

)(

,0in ))((

0,in 0))((

PEQ:PEQ:

SEQ:SEQ:

BiCGStab + DD prec. for global pressure eq.BiCGStab + DD prec. for global pressure eq.Multigrid V-cycle in subdomain solvesMultigrid V-cycle in subdomain solves

Two-phase Porous Media FlowTwo-phase Porous Media Flow

History of History of water water saturation saturation propagationpropagation

Nonlinear Water WavesNonlinear Water Waves

• Fully nonlinear 3D water waves• Primary unknowns:

wallssolidon 0

surfaceon water 02/)(

surfaceon water 0

olumein water v 0

222

2

n

gzyxt

zyyxxt

,

Nonlinear Water WavesNonlinear Water Waves• CG + DD prec. for global solverCG + DD prec. for global solver• Multigrid V-cycle as subdomain solverMultigrid V-cycle as subdomain solver• Fixed number of subdomains Fixed number of subdomains MM=16 (independent =16 (independent

of of PP))• Subgrids from partition of a global 41x41x41 gridSubgrids from partition of a global 41x41x41 grid

P Execution time Speedup Efficiency

1 1404.44 N/A N/A

2 715.32 1.96 0.98

4 372.79 3.77 0.94

8 183.99 7.63 0.95

16 90.89 15.45 0.97

Parallel Simulation of 3D Acoustic Parallel Simulation of 3D Acoustic FieldField

• A linux-cluster: 48 Pentium-III A linux-cluster: 48 Pentium-III 500Mhz procs, 100 Mbit 500Mhz procs, 100 Mbit interconnectioninterconnection

• SGI Cray Origin 2000: MIPS R10000SGI Cray Origin 2000: MIPS R10000

• LAL parallelization; 2 cases:LAL parallelization; 2 cases:– Linear Model (linear wave equation), Linear Model (linear wave equation),

solved with an explicit methodsolved with an explicit method

– Nonlinear Model, solved with an implicit Nonlinear Model, solved with an implicit methodmethod

Mathematical Nonlinear ModelMathematical Nonlinear Model

0)0,(

)0,(

1

)(),(

)(2

1)(

2

0])(2

/)[(

11

0

0

20

20

22

000

222

222

2

22

xt

xtcn

erJtrp

btct

pp

btc

AB

tctc

tj

x

x

x

x

Rtx

Rtx

cT

T

),(

),(

Results - Linear ModelResults - Linear Model

CPUsCPUs Origin 2000Origin 2000 Linux ClusterLinux Cluster

CPU-timeCPU-time SpeedupSpeedup CPU-timeCPU-time SpeedupSpeedup

11 944.83944.83 N/AN/A 640.7640.7 N/AN/A

22 549.21549.21 1.721.72 327.8327.8 1.951.95

44 282.75282.75 3.343.34 174.0174.0 3.683.68

88 155.01155.01 6.106.10 90.9890.98 7.047.04

1616 80.4180.41 11.811.8 46.3546.35 13.813.8

2424 65.6365.63 14.414.4 34.0534.05 18.818.8

3232 49.9749.97 18.918.9 26.2726.27 24.424.4

4848 35.2335.23 26.826.8 17.7417.74 36.136.1

Results - Nonlinear ModelResults - Nonlinear Model

CPUsCPUs Origin 2000Origin 2000 Linux ClusterLinux Cluster

CPU-timeCPU-time SpeedupSpeedup CPU-timeCPU-time SpeedupSpeedup

22 8670.88670.8 N/AN/A 6681.56681.5 N/AN/A

44 4726.54726.5 3.753.75 3545.93545.9 3.773.77

88 2404.22404.2 7.217.21 1881.11881.1 7.107.10

1616 1325.61325.6 13.013.0 953.89953.89 14.014.0

2424 1043.71043.7 16.616.6 681.77681.77 19.619.6

3232 725.23725.23 23.923.9 563.54563.54 23.723.7

SummarySummary

• Goal: provide software and programming Goal: provide software and programming rules for easy parallelization of sequential rules for easy parallelization of sequential simulatorssimulators

• Applicable to a wide range of PDE problemsApplicable to a wide range of PDE problems• TThreehree parallelization parallelization approachapproaches:es:

– parallelization at the linear algebra level:parallelization at the linear algebra level: ““automatic” parallelizationautomatic” parallelization – domain decomposition:domain decomposition: very flexible, compact visible code/algorithmvery flexible, compact visible code/algorithm– combined approachcombined approach

• Performance: satisfactory speed-upPerformance: satisfactory speed-uphttp://www.ifi.uio.no/~tpv