Paul Fischer Mathematics and Computer Science Division Argonne National Laboratory

Spectral Element Methods for Transitional Flows in Complex Geometries

Paul Fischer

Mathematics and Computer Science Division

Argonne National Laboratory

fischer@mcs.anl.govwww.mcs.anl.gov/~fischer

Hisham Bassiouny, Henry Tufo University of Chicago

Francis Loth, Seung Lee University of Illinois, ChicagoJerry Kruse Juniata College

Julie Mullen Worcester Polytechnic Inst.

Mathematics and Computer Science Division, Argonne National Laboratory

Accurate simulation of long-time dynamics pattern formation in nonequilibrium flows convection dynamics in deep atmospheres transitional boundary layers heat transfer enhancement vascular flows

Motivation: Spectral Element Simulations

Outline

Temporal/Spatial Discretization Spectral element method Filter-based stabilization

Computational complexity - pressure solve Pressure projection Overlapping Schwarz

Transition in vascular flows

Navier-Stokes Equations

Reynolds number Re ~ 1000 - 2000 small amount of diffusion highly nonlinear (small scale structures result)

Time advancement: Nonlinear term: explicit (3rd-order AB or characteristics) Stokes problem: pressure/viscous decoupling:

3 Helmholtz solves for velocity (consistent) Poisson equation for pressure

Filter velocity field at end of each time step for stability (FM 2001)

Characteristics-Based Convection Treatment(OIFS Scheme - Maday, Patera, Ronquist 90, Characteristics - Pironneau 82)

Unsteady Stokes Problem at Each Step

linear (allows superposition) implicit (large CFL, typ. 2-5) symmetric positive definite operators (conjugate gradient iteration)

Spatial Discretization: Spectral Element Method (Patera 84, Maday & Patera 89) Variational method, GL quadrature.

Domain partitioned into K high-order quadrilateral (or hexahedral) elements (decomposition may be nonconforming - localized refinement)

Trial and test functions represented as Nth-order tensor-product polynomials within each element. (N ~ 4 -- 15, typ.)

KN 3 gridpoints in 3D, KN 2 gridpoints in 2D.

Converges exponentially fast with N for smooth problems.

3D nonconforming mesh for arterio-venous graft simulations:K = 6168 elements, N = 7

Examples of Local Spectral Element Basis Functions in 2D

N th-order Lagrangian interpolants based on the Gauss-Lobatto-Legendre quadrature points in [-1,1]2

N = 10

PN - PN-2 Spectral Element Method for Navier-Stokes (MP 89)

Gauss-Lobatto Legendre points(velocity)

Gauss Legendre points(pressure)

Velocity, u in PN , continuous Pressure, p in PN-2 , discontinuous

Spectral Convergence - Analytic Navier-Stokes Solution (Kovasznay, 1948)

Numerical dispersion as a function of polynomial degree

Convection of non-smooth data on a 32x32 grid (K1 x K1 spectral elements of order N). (cf. Gottlieb & Orszag 77)

Filter-Based Stabilization

Stabilization essential for high Reynolds number flows

At end of each time step, Interpolate u onto GLL points for PN-1

Interpolate back to GLL points for PN

Results are smoother with linear combination: (F. & Mullen 01)

(Gottlieb et al., Don et al., Vandeven, Boyd, ...)

F(u) = (1-) u + IN-1 u ( ~ 0.05 - 0.2)

Preserves interelement continuity and spectral accuracy

Equivalent to multiplying by (1-) the N th coefficient in the expansion

u(x) = uk k (x) (Boyd 98)

k (x) := Lk(x) - Lk-2(x)

F1(u) = IN-1 u

Numerical Stability Test: Shear Layer Roll-Up (Bell et al. JCP 89, Brown & Minion, JCP 95, F. &

Mullen, CRAS 2001)

1282 2562

25621282

Spatial and Temporal Convergence (FM, 2001)

Base velocity profile and perturbation streamlines

Error in Predicted Growth Rate for Orr-Sommerfeld Problem at Re=7500

1D Steady-State Convection-Diffusion Equation

For steady convection-diffusion equation:

ux = uxx + f , x [0,1], f =1

Galerkin formulation is unstable when the number of degrees-of-

freedom is odd and 0 [Canuto 88].

N=16N=15

Solid - unfilteredDashed - filtered

For unsteady convection-diffusion equation:

ut + ux = uxx + f , x [0,1], f =1

can show that steady state discrete solution for the filtered problem satisfies

( A + C + H (F-1- I) ) u = B f

A = SEM Laplacian

C = (skew symmetric) convection operator

H = discrete Helmholtz operator ( A + B/t )

B = SEM mass matrix

The term H (F-1- I) controls the singular mode when 0

and the number of degrees of freedom is odd (N even).

Note that filtering only the (nonlinear) convective term is insufficient to provide stability.

For example, for the unsteady convection-diffusion equation one has:

Hun+1 = (t -1 B - F C ) un +B f

which solves the steady state problem

( A + F C) u = B f

In this case, there is no stability when 0 since F C is singular

when the number of degrees of freedom is odd (N even).

Filtering permits Re99 > 700 for transitional boundary layer calculations

blow up

Re = 700

Re = 1000

Re = 3500

Consistent Splitting for Unsteady Stokes(MPR 90, Perot 93, Couzy 95)

E - consistent Poisson operator for pressure, SPD boundary conditions applied in velocity space preconditioned with overlapping Schwarz method (Dryja & Widlund 87, PF 97, FMT 00)

Computational Complexity

Pressure solve, Epn = gn, most expensive step.

Remedies:

Compute best fit, p*, in space of previous solutions:

|| pn - p*||E || pn - q||E for all q in span{pn-1, . . . , pn-l}

two additional matrix-vector products per step (very cheap)

Solve for perturbation, pn - p*, using overlapping Schwarz PCG

tensor-product based local solves fast parallel coarse-grid solve using sparse-basis projection

Initial guess for Axn = bn via projection ( A=E )

Initial guess for Axn = bn via projection

|| xn - x*||A = O(tl) + O( tol )

two additional mat-vecs per step

storage: 2+lmax vectors

results with/without projection (1.6 million pressure nodes)

Overlapping Schwarz Precondtioning for Pressure

(Dryja & Widlund 87, Pahl 93, PF 97, FMT 00)

z = P-1 r = R0

TA0-1 R0 r + Ro,k

TAo,k-1 Ro,k r

Ao,k - low-order FEM Laplacian stiffness matrix on overlapping domain

for each spectral element k (Orszag, Deville & Mund, Casarin)

Ro,k - Boolean restriction matrix enumerating nodes within

overlapping domain k

A0 - FEM Laplacian stiffness matrix on coarse mesh (~ K x K )

R0T - Interpolation matrix from coarse to fine mesh

Two-Level Overlapping Additive Schwarz Preconditioner

Overlapping Solves: Poisson problemswith homogeneous Dirichlet bcs.

Coarse Grid Solve: Poisson problemusing linear finite elements on spectral

element mesh (GLOBAL).

Overlapping Schwarz - local solve complexity

Exploit local tensor-product structure

Fast diagonalization method (FDM) - local solve cost is ~ 4d K N(d+1) (Lynch et al 64)

2D Test Problem: Startup flow past a cylinder (N=7)

Resistant pressure mode, p166 - p25, (K=1488)Residual history

Impact of High-Aspect Ratio Elements

Nonconforming discretizations eliminate unnecessary elements in the far field and result in better conditioned systems.

Iteration count bounded with refinement - scalable

Fast Parallel Coarse-Grid Solves: x0 = A0-1

Problem: coarse-grid solve important for conditioning

A0 sparse, relatively small (n ~ P)

completely full

x0 , b0 - distributed vectors

all-to-all communication

(F 96, Tufo & F 01)

Any single element in b0 will have a nontrivial influence

throughout the domain, and hence, across all processors.

Fast Parallel Coarse-Grid Solves: x0 = A0-1

Usual approaches: Collect b0 onto each processor and solve A0x0 = b0 redundantly

Communication O(n), no parallelism. OK for P ”64.

Precompute A0-1

and store row-wise on processors. Communication O(n log P). OK for P ”512.

Current approach:

quasi-sparse factorization: A0-1 =XXT

OK for P ”10000.

x0 = XXTb0 - computed as projection onto sparse basis (parallel

matrix-vector product)

choose columns of X to be as sparse as possible

(F 96, Tufo & F 01)

XXT Communication Complexity for 7x7 Example

Nested Dissection Ordering

Processor Map

n1/2 log P

Intel ASCI Red Performance: Poisson problem on q x q mesh

q=63 q=127

latency*2 log P curve is best possible lower bound

Weakly Turbulent Vascular Flows

Vascular flows AV-graft failure

importance of hemodynamic forces (shear, pressure, vibration) in disease genesis and progression

Stenosed carotid arteries turbulence a distinguishing feature of severely stenosed (constricted) arteries high wall-shear (mean and oscillatory) can possibly lead to

embolisms (plaque break-off) thrombis formation (clotting)

Turbulence computations 1-3 orders of magnitude more difficult than laminar (healthy) case

Arterio-Venous Grafts

PTFE plastic tubing surgically attached from artery to vein (short-circuit)

Provides a port for dialysis patients, to avoid repeated vessel injury

Failure often results after 3 months from occlusion (cell proliferation) downstream of attachment to vein, where flow is weakly turbulent

Flow characteristics of the AV graft-vein junction

Over pulsatile cycle, 1000 ReG 2500 (ReV = 1.6 ReG)

Separation region downstream of the toe

Significant velocity and pressure fluctuations

(audible or palpable “bruit”)

LDA Flow Model (F. Loth, N. Arsalan, UIC)

Distal VenalSegment Inlet

GraftInlet

“toe”

Partition bifurcation into 3 branches.

3-way partition avoids‘figure 8’ cross-section

Sweep each branch with standard hex-circle decomposition

Meshing Algorithm for Bifurcating Vessels

Coherent Structures in AV-Graft, Re=1820

Close-up of coherent vortical structures in PVSvisualized with the 2 criterion of Jeong & Hussain

(JFM’95)

Coherent Structures in AV-Graft, Re=1820

Close-up of in-plane velocity distributions at two axial slices

K=6168, N=7, Re=1820

<u’>

<v’>

Experimental/Numerical Comparison at Re=1820

LDA measurements (Arsalan 99) SEM simulation results

Spanwise Vorticity ComparisonSpanwise Vorticity Comparison

Re = 1060

Re = 1820

Hoodmean = 350peak = 1500 - 2500

Floormean = 600peak = 750

InstantaneousAveragerms

WSS along hood and floor (dynes/cm2)

axial distance from toe (Z/D)

position (cms)

time (secs) (715 Hz)

Axial velocity vs. time

Characterization of AV Graft Hemodynamics

Summary

Spectral Element Simulations well suited to minimally dissipative flows exponential convergence in space third-order accurate in time

Capabilities Transitional-Reynolds number applications in complex geometries Weakly turbulent vascular flow simulations ~ 20 days/cardiac

cycle on 32 processors (700 Hz signal) Able to fully characterize hemodynamic environment for AV graft

Current / Future Algorithmic Issues improved linear solvers for complex domains (multigrid?)

improved mesh quality (smoothing) condition number can be strongly influenced by mesh deformation

fluid-structure interaction (linear elasticity)

Example: A=LU, A - tridiagonal

PUPT X X = (PUPT)-1

Lexicographical(Std.) Ordering

O(n2) nonzeros

O(n log n) nonzeros

NestedDissectionOrdering

Parallel Coarse-Grid Solves

Problem: x0 = A0-1

x0 , b0 - distributed vectors, A0-1

completely full, all-

to-all communication

Usual approaches: Collect b0 onto each processor and solve A0x0 = b0 redundantly

Communication O(n), no parallelism. OK for P ”64.

Precompute A0-1

and store row-wise on processors. Communication O(n log P). OK for P ”512.

Current approach: compute sparse factorization of A0

(F 96, Raghavan 98, Tufo & F 01)

Cast x0 = A0-1b0 as x0 = XXTb0 (sparse matrix-vector product)

PUPT X X = (PUPT)-1

StandardOrdering

O(n2) nonzeros

O(n log n) nonzeros

Tridiagonal Matrix Example

NestedDissectionOrdering

Application -Hemisphere/Flat-Plate Mesh

K = 8168, coarse grid dimension n = 10142, P = 1024 XXT time < 1% of Navier-Stokes solution time

Numerical SimulationsMesh Reconstruction from CT data

Axial spacing 1 mm

In plane resolution

~ 0.5 x 0.5 mm

Axial Velocity Component vs. Time, Re=1820

Position (cm)

Time (secs)

715 Hz

Hoodmean = 350peak = 1500 - 2500

Floormean = 600peak = 750

(in dynes/cm2)

Instantaneous

Average

WSSalong hood and floor

(dynes/cm2)

Axial distance from toe (Z/D)

Wall Shear Stress @ Re=1820

Transient Wall Shear, Re=1820, Steady Inflow

Stress Distribution on Venal Surface, Re=1820

Validation: Arterio-Venous Graft Studies(Arsalan, Lee, Loth - UIC, Fischer - ANL)

AV Graft - Re=1820

Paul Fischer Mathematics and Computer Science Division Argonne National Laboratory

Documents

MPI at Exascale Rajeev Thakur Mathematics and Computer Science Division Argonne National Laboratory

Numerical Optimization - University of Chicagoice.uchicago.edu/2012_presentations/Faculty/Munson/...Numerical Optimization Todd Munson Mathematics and Computer Science Division Argonne

COMPUTING A TRUST REGION STEP by Jorge J. More and D. C. Sorensen · 2015. 3. 31. · Jorga J. Mor4 and D. C. Sorensen Applied Mathematics Division Argonne National Laboratory Argonne

Future Scientific Infrastructure Ian Foster Mathematics and Computer Science Division Argonne National Laboratory and Department of Computer Science The

Mathematics and Computer Science Division, Argonne National Laboratory Biofluid Dynamics qMotivation q Objective q Challenges q Proposed timeline qTechnical

Argonne Accelerator Institute Activites Rod Gerig Argonne May 18, 2007 Presentation to the Fermilab-Argonne Directors Collaboration Meeting

conferences.iaea.org · Web view2IAEA-CN-245-141. The submitted manuscript has been created by UChicago Argonne, LLC, Operator of Argonne National Laboratory (“Argonne”). Argonne,

Paul Fischer Mathematics and Computer Science Division ...fischer/me528/sem_course_09.pdf · Semi-implicit: explicit treatment of nonlinear term. Leads to Stokes saddle problem, which

PETSc Users Manual - anl.gov · 2017. 11. 17. · PETSc Users Manual Revision 3.8 Mathematics and Computer Science Division. About Argonne National Laboratory Argonne is a U.S. Department

FISCHER Motorabrichtspindeln FISCHER ... - Fischer Spindle

Grid Technologies Enabling Collaborative Science Ian Foster Mathematics and Computer Science Division Argonne National Laboratory and Department of Computer

High-Performance Numerical Components and Common Interfaces Lois Curfman McInnes Mathematics and Computer Science Division Argonne National Laboratory

PVFS: Design and Implementation Sam Lang Mathematics and Computer Science Division Argonne National Laboratory slang@mcs.anl.gov

PETSc Tutorial - Argonne National Laboratory Tutorial PETSc Team Presented by Matthew Knepley Mathematics and Computer Science Division Argonne National Laboratory ACTS Workshop Lawrence

ANL-5990 Rev. Physics and Mathematics AEC Research .../67531/metadc1020493/...ANL-5990 Rev. Physics and Mathematics (TID-4500, 14th Ed.) AEC Research and Development Report ARGONNE

Argonne National Laboratory, Argonne, IL

Scalable Stochastic Programming Cosmin Petra and Mihai Anitescu Mathematics and Computer Science Division Argonne National Laboratory Informs Computing

Chiba City Mathematics & Computer Science Division Argonne National Laboratory An Open Source Computer Science Testbed Update

MPICH2 User’s Guide Version 1.0.3 Mathematics and ... · MPICH2 User’s Guide∗ Version 1.0.3 Mathematics and Computer Science Division Argonne National Laboratory William Gropp

DSDP4 User Guide - Optimization Online · DSDP4 User Guide 1 Steven J. Benson Mathematics and Computer Science Division Argonne National Laboratory Argonne, IL U.S.A. Yinyu Ye Department