Scalability analysis of the distributed-memory ...users.monash.edu › ~jdroniou › MWNDEA ›...

Scalability analysis of the distributed-memory implementation of the

Aggregated unfitted Finite Element Method (AgFEM)

Alberto F. Martín∗, Santiago Badia, Francesc Verdugo, Eric Neiva

MWNDEA 2020, Melbourne, Australia, 12/02/2020

Embedded Finite elements

CutFEM, Finite Cell Method, AgFEM, X-FEM, ...

body-fitted mesh unfitted mesh

4 Simplified mesh generation

8 Dirichlet BC? 8 Numerical integration? 8 ill-conditioning? (this talk)

Alberto F. Martín (Monash University) MWNDEA2020 2/35

Embedded Finite elements

CutFEM, Finite Cell Method, AgFEM, X-FEM, ...

body-fitted mesh unfitted mesh

4 Simplified mesh generation

8 Dirichlet BC? 8 Numerical integration? 8 ill-conditioning? (this talk)

Parallel distributed-memory simulation pipeline

1. Unfitted (adaptive) Cartesian grids(p4est)

2. Partition using space filling-curves(p4est)

3. Unfitted FE discretization (AgFEM)

4. AMG linear solver (PETSc)

Unfitted methods at large scales: pros and cons

4 Highly scalable mesh generationbased on octrees (e.g. p4est)

4 Highly scalable mesh partition withspace-filling curves (Parmetis notneeded)

4 Highly scalable adaptive meshrefinement + load balancing

8 Not guaranteed that highly scalablelinear solvers keep their optimalproperties for cut elements.

PETSc CG + AMG preconditioner on unfitted meshes

Poisson equation (weak scaling test with 5 meshes)

AMG+AgFEM AMG + Naive unfitted FEM*

103 104 105 106 107 108

* Nitsche BCs + modified integration in cut cells

Why linear solvers are affected by cut cells?

Condition number estimates (Poisson Eq.)

(a) Body-fitted case

k2(A) ∼ h−2

(b) Naive unfitted FEM

k2(A) ∼ |η|−(2p+1− 2d )

"small cut cell problem"

Possible remedies

Fix the linear solver

Taylor your parallel solver to deal with k2(A) ∼ |η|−(2p+1−2/d)

Example: [S. Badia, F. Verdugo. Robust and scalable domaindecomposition solvers for unfitted finite element methods. Journal ofComputational and Applied Mathematics (2018) ].

Fix the linear system (this talk)

Enhance the unfitted FE method so that k2(A) ∼ h−2

Use a standard scalable solver

Examples: CutFEM, AgFEM

Agenda

1. The AgFEM method (serial case)

2. Parallel implementation

3. Performance of parallel AgFEM + AMG solvers

Agenda

AgFEM method for the Poisson Eq.

−∆u = f in Ω

u = uD on ∂Ω

Weak imposition of Dirichlet BCs

Nitsche’s Method

Find uh ∈ V h such that

ah(uh, vh) = lh(vh) ∀vh ∈ V h (vh does not vanish on ∂Ω!)

ah(uh, vh) :=∑

K∈T acth

K∩Ω

∇uh · ∇vh −∑

F∈T acth ∩∂Ω

(∇uh · n)vh

−∑

F∈(T acth ∩∂Ω)

uh(∇vh · n) +∑

βh−1F

lh(vh) :=∑

K∈T acth

K∩Ω

f vh +∑

βh−1F

uDvh −∑

uD(∇vh · n

The key feature of AgFEM is the definition of the discrete space VhAlberto F. Martín (Monash University) MWNDEA2020 11/35

Starting point: "naive" FE space

V stdh := u ∈ C0(Ωact) : u|K ∈ Qp(K) ∀K ∈ T act

T acth , Ωact V std

Aggregated FE space

Basic idea: improve conditioning by removing problematic DOFs

V aggh :=

u ∈ Vh : u× =

•∈masters(×)

C×•u• ∀× ∈ P

• well-posed dofs× problematic dofs (P)

Definition of constraints via cell aggregates

1. Generate cell aggregates(1 interior cell + several cut cells)

2. Define dof to root cell map root(×)

via the aggregates

3. Define constraints:

u× =∑

•∈dofs(root(×))

φroot(×)• (x×)u•

via the aggregates

u× =∑

via the aggregates

u× =∑

via the aggregates

u× =∑

Results for the unfitted aggregated FEM (Poisson Eq.)1

κ(A) ≤ c1h−2 (Condition number bound)

β ≤ c2h−2 (Nitsche’s penalty coef.)

‖u− uh‖H1(Ω) ≤ c3hp (Optimal convergence order)

‖u− uh‖L2(Ω) ≤ c4hp+1 (Optimal convergence order)

and others (inverse/trace inequalities, bound of aggregate size, boundof the extended solution, ...)

1 [Badia, Verdugo, Martín. The aggregated unfitted finite element method for ellipticproblems. Comput. Methods Appl. Mech. Eng. (2018).]

0.3 0.4 0.5 0.6 0.7

p=1, standard

p=2, standard

p=1, aggr.

p=2, aggr.

0.3 0.4 0.5 0.6 0.7

p=1, standard

p=2, standard

p=1, aggr.

p=2, aggr.

Convergence test

-2.5 -2 -1.5 -1

log10(h)

log10(E

p=1, standard

p=2, standard

p=1, aggr.

p=2, aggr.

slope 1

slope 2

(a) 2D

-2.5 -2 -1.5 -1

log10(h)

log10(E

)p=1, standard

p=2, standard

p=1, aggr.

p=2, aggr.

slope 1

slope 2

(b) 3DAlberto F. Martín (Monash University) MWNDEA2020 18/35

Extension to the Stokes problem2

75.0 |u|0.0

−∆u+∇p = f in Ω

∇ · u = 0 in Ω

u = 0 on ΓD

(∇u− pI) · n = g on ΓN

2[Badia, Martín, Verdugo. Mixed aggregated finite element methods for the unfitteddiscretization of the stokes problem. SIAM J. Sci. Comput., 40(6). 2018.]

0.3 0.4 0.5 0.6 0.7

log10(condest(A

Aggregated

Standard

0.3 0.4 0.5 0.6 0.7

log10(condest(A

Aggregated

Standard

1.2 1.4 1.6 1.8 2.0

log10((DOF)1/d

u‖ H

‖u‖ H

Aggregated

Standard

slope -2

1.2 1.4 1.6 1.8 2.0

log10((DOF)1/d

u‖ L

‖u‖ L

)Aggregated

Standard

slope -3

1.2 1.4 1.6 1.8 2.0

log10((DOF)1/d

p‖ L

‖p‖ L

Aggregated

Standard

slope -2

Agenda

Parallel mesh distribution

(a) D = D1, D2. (b) View from D1. (c) View from D2

Main phases to be parallelized:

• Cell Aggregation

• Imposition of constraints

Cell aggregation (serial)

touched

untouched

aggregated

touched

untouched

aggregated

touched

untouched

aggregated

touched

untouched

aggregated

touched

untouched

aggregated

Aggregates in 3D

Cell aggregation (parallel)

(a) Step 1.

(b) Step 2. (c) Comm. (d) Step 3.

4 Standard nearest neighbor communication to determine root cellsAlberto F. Martín (Monash University) MWNDEA2020 27/35

(a) Step 1. (b) Step 2.

(c) Comm. (d) Step 3.

(a) Step 1. (b) Step 2. (c) Comm.

(d) Step 3.

(a) Step 1. (b) Step 2. (c) Comm. (d) Step 3.

Parallel imposition of constraints

Ds′ Ds′′ Ds

8 Subdomain-local constraints not even possible in some cases

4 At the end, only standard neighbor communication required

Agenda

Weak scaling test setup

• Poisson eq.

• AgFEM vs "naive" unfitted FEM

• Linear solver:PCG from PETSc

• Preconditioner:smooth aggregation AMG fromPETSc (GAMG)

• Up to 16K cores and 1000Mbackground cells

Computed at Mare Nostrum 4

Number of PCG iterations (weak scaling)

103 104 105 106 107 108

(a) Popcorn (b) Spiral (c) Swiss Cheese

agg (load 1)agg (load 2)agg (load 3)

std (load 1)std (load 2)std (load 3)

Computational time (secs) AgFEM stages (weak scaling)

105 106 107 108

]105 106 107 108

Cell aggregation (Alg. 2)Path reconstruction (Algs. 3 and 4)Import data from root cells (Alg. 5)Setup constraints (Sect. 3.8)

Setup of local DOFs ids (Sect. 3.3)Setup of global DOFs ids (Sect. 3.7)FE integration + assembly (Table 2)

Computational time (secs) AMG solver (weak scaling)

105 106 107 108

104 105 106 107

Linear solver setup Linear solver run

• Setup degradation (11.94x CPU time for 4,448x larger problem)

• Similar for standard FEM in a box (2.50x for 355.26x)

Conclusions

4 Embedded FEM enables scalable octree-based meshes

8 ... but can destroy the scalability of linear solvers

4 AgFEM allows to recover the optimal scaling of linear solver

4 ... while keeping the optimal discretization order.

For more details, see papers:

F. Verdugo, A.F. Martín, S. Badia. Distributed-memory parallelization ofthe aggregated unfitted finite element method. CMAME, 357, 2019.

S. Badia, A.F. Martín, F. Verdugo. Mixed aggregated finite elementmethods for the unfitted discretization of the Stokes problem. SIAM J.Sci. Comput., 40(6), 2018.

S. Badia, F. Verdugo, A.F. Martín. The aggregated unfitted finite elementmethod for elliptic problems. CMAME, 336, 2018.

https://github.com/fempar/fempar

Scalability analysis of the distributed-memory ...users.monash.edu › ~jdroniou › MWNDEA ›...

Documents

linux* kernel scalability linux* kernel scalability

L21 scalability

Vaadin scalability-slides

Error Indicators and Adaptive Refinement of Finite Element ...users.monash.edu/~jdroniou/MWNDEA/slides/slides-fang.pdfFinite element thin-plate splines Finite element thin-plate spline

ArcGIS Server Performance and Scalability - Optimization ... · ArcGIS Server Performance and Scalability ... (WebTestWebTest or Unit Test), ... ArcGIS Server Performance and Scalability

Appnexus scalability

Web application scalability

Scala, Scalability

Olap scalability

Parallel Scalability

Inviscid instability of a unidirectional flow sheared in two transverse directionsusers.monash.edu/~jdroniou/MWNDEA/slides/slides-deguchi.pdf · 2020. 2. 13. · Kengo Deguchi, Monash

Scalability In PHP

Stochastic partial differential equations on the sphere.users.monash.edu/~jdroniou/MWNDEA/slides/slides-olenko.pdf · Stochastic partial di erential equations on the sphere. Andriy

Scalability and Heterogeneity · Scalability and Heterogeneity Colin Perkins

DBSight Scalability

B-char: an efficient (and feasible!) approach for mass ...users.monash.edu/~jdroniou/presentations/mwndea-2020.pdf · ˚= jKj (local): IIssue: we only compute Kb, and jM \Kbj ˚ˇjM

Simpler Concurrency, Scalability & Fault-tolerance through …assets.en.oreilly.com/1/event/45/Akka_ Simpler Scalability, Fault... · Akka: Simpler Concurrency, Scalability & Fault-tolerance

B-char: an efficient (and feasible!) approach for mass ...users.monash.edu/~jdroniou/MWNDEA/slides/slides-droniou.pdf · jB K;sj ˚= jKj ˚: Iˆ K jB K;s ... jM \Kbj ˚ˇV b K;M =

Performance, High Availability, and Scalability of Magento ...static.zend.com/topics/Magento-HA-Scalability-using-Zend-Server.pdf · Performance, High Availability, and Scalability

Eth Scalability