Terascale Terascale Simulation Simulation Tools and Tools and TechnologiTechnologi

eses

Spectral Elements for Anisotropic Diffusion and Incompressible MHD

Paul Fischer

Argonne National Laboratory

Terascale Simulation Tools and Technologies

• Goal: Enable high-fidelity calculations based on multiple coupled physical processes and multiple physical scales

– Adaptive methods– Advanced meshing strategies– High-order discretization strategies

• Technical Approach: – Develop interchangeable and interoperable software components

for meshing and discretization– Push state-of-the-art in discretizations

Terascale Simulation Tools and Technologies

• Goal: Enable high-fidelity calculations based on multiple coupled physical processes and multiple physical scales

– Adaptive methods– Advanced meshing strategies– High-order discretization strategies

• Technical Approach: – Develop interchangeable and interoperable software components

for meshing and discretization– Push state-of-the-art in discretizations

Outline

• Driving physics

• SEM overview

– Costs

– MHD formulation

– Convective transport properties of SEM

• Anisotropic Diffusion

• Brief summary of current results on MHD project

• Conclusions

Driving Physics

• Anisotropic Diffusion: ( S. Jardin, PPPL)

– Central to sustaining high plasma temperatures– Want to avoid radial leakage– Need to understand significant instabilities

• Incompressible MHD: ( F. Cattaneo UC, H. Ji PPPL, … )

– Several liquid metal experiments are under development to understand momentum transport in accretion disks

• Magneto rotational instability (MRI) is proposed as a mechanism to initiate turbulence capable of generating transport• The magnetic Prandtl number for liquid metals is ~ 10-5

Re=106 & Rm=10 for experiments• Numerically, we can achieve Re=104 & Rm=103 ( 2005 INCITE award )

– Free-surface MHD ( H. Ji, PPPL )• Proposed as a plasma-facing material for fusion• Desire to understand the effect of B on the free surface

Spectral Element Overview

• Spectral elements can be viewed as a finite element subset• Performance gains realized by using:

– tensor-product bases (quadrilateral or brick elements)– Lagrangian bases collocated with GLL quadrature points

• Operator Costs: Standard FEM* SEM

# memory accesses: O( EN 6 ) O( EN 3 )# operations: O( EN 6 ) O( EN 4 )

N = order, E = number of elements, EN 3 = number of gridpoints

• Dramatic cost reductions for large N ( > 5 )

2D basis function, N=10

Computational Advantages of Spectral Elements

• Exponential convergence with N• minimal numerical dispersion / diffusion

• for anisotropic diffusion, lements of ker(A||) well-resolved – sharp decoupling of isotropic and anisotropic modes

• Matrix-free form• matrix-vector products cast as efficient matrix-matrix products

• number of memory accesses identical to 7-pt. finite difference

• no additional work or storage for anisotropic diffusion tensor

SEM Computational Kernel on Cached-Based Architectures

– matrix-matrix products, C=AB: 2N 3 ops for 2N 2 memory references

– much of the additional work of the SEM is covered by efficient use of cache – e.g., time for A*B vs A+B for N=10 is:

2.0 x on DEC Alpha, 1.5 x on IBM SP

A*B

mxm vs m+m

A*B

A+BA+B

Incompressible MHD• t

— plus appropriate boundary conditions on u and B

• Typically, Re >> Rm >> 1

• Semi-implicit formulation yields independent Stokes problems for u and B

Incompressible MHD in a Nutshell• t

• Convection:– dominates transport – dominates accuracy requirements

• often the challenging part of the discretization

– treated explicitly in time

• Diffusion:– “easy” ( ?? )

• Projection: div u = 0 div B = 0– dominates work– isotropic SPD operator

• multiple right-hand side information• scalable multilevel Schwarz methods ( 1999 GB

award )• SE multigrid ( Lottes & F 05 )

High-Order Methods for Convection-Dominated Flows

Phase Error for h vs. p Refinement: ut + ux = 0

h-refinement p-refinement

High-Order Methods for Convection-Dominated Flows

• Fraction of accurately resolved modes (per space direction) is increased only through increased order – Savings cubed in R3

• Rate of convergence is extremely rapid for high N– Important for multiscale / multiphysics problems

( Q: Why do we want 10 9 gridpoints? )

• Stability issues are now largely understood– stabilization via DG, filtering, etc.– dealiasing

• Still, must resolve structures (no free lunch…)– Computational costs are somewhat higher– Data access costs are equivalent to finite differences

c = (-x,y)

c = (-y,x)

Stabilizing convective problems:

Models of straining and rotating flows:– Rotational case is skew-symmetric.

– Filtering attacks the leading-order unstable mode.

– Dealiasing (high-order quadrature) yields imaginary eigenvalues – vital for MHD

N=19, M=19 N=19, M=20

strainingfield

rotationalfield

Diffusion – easy ??

CEMM Challenge Problems S. Jardin, PPPL

1. Anisotropic diffusion in a toroidal geometry

2. Two-dimensional tilt mode instability

3. Magnetic reconnection in 2D

• Provides a problem suite that – captures essential physics of fusion simulation– stresses traditional numerical approaches– identifies pathways for next generation fusion codes

• Excellent vehicle for initiating SciDAC interatctions.

Anisotropic Diffusion in Toroidal Domains

•b – normalized B-field, helically wrapped on toroidal surfaces

•thermal flux follows b.

High degree of anisotropy creates significant challenges

• For this problem is more like a (difficult)

hyperbolic problem than straightforward diffusion.

• This is reflected in the variational

statement for the steady case with

• Numerical challenges:– radial diffusion,– nearly singular, with large (but finite) null space– avoid grid imprinting– unsteady case constitutes a stiff relaxation problem– preconditioning nearly singular systems

• CEMM challenge:– establish spatial convergence for steady state case– check unsteady energy conservation when – investigate the behavior of the tearing mode instability

High degree of anisotropy creates significant challenges

total number of gridpoints

cent

erpo

int e

rror

Steady State Error – 3D, || = 10 8

It is advantageous to use few elements of high order

•Fewer gridpoints are required•CPU time proportional to number of gridpoints

(N odd, 3-7)

(N even, 2-6)

High Anisotropy Demands High Accuracy

• A = || A|| + AI

• AI controls radial diffusion

– A|| must be accurately represented when || >> 1

– Error must scale as ~ 1 / ||

Steady-State L2 – error over a range of discretizations

E

N

High Anisotropy Demands High Accuracy

•Difficulty stems from high-frequency content in null space of A||

•High-order discretizations are able to accurately represent these functions.

N=16

k

N=2

k

k

18th mode in circular geometry, || = 108

Error vs. tT vs. r,t

N=14Error vs. r, N=12

Evolution of Gaussian Pulse for

– minimal radial diffusion– no grid imprinting

– careful time integration required (e.g., adaptive DIRK4)

-averaged temperature vs. time

SEM is able to identify critical physics –

• Tearing mode instability– radial perturbation:

• b = b0 + cos(m-n) r

– field lines do not close on m-n rational surface

– magnetic island results, with significant increase in radial conductivity.

– island width scales as:• W ~ in accord

with asymptotic theory

Note: this is a subtle effect!

W

Island width vs. || at onset.

W

Outstanding Challenges for Anisotropic Diffusion Simulation

• Preconditioning– need null-space control

– condition number scales as ||

• Non-aligned grids predict early island formation

a32 a32

max

dT

/dr

Incompressible MHD Results

Axisymmetric Hydro Simulations of Taylor-Couette w/ Rings

• Re=620 steady• Re=6200 unsteady

• Axisymmetric MHD simulations are being carried out now.

• Starting point for 3D simulations, which are being compared with experiments at PPPL.

Com

puta

tion

by O

babk

o,

Fis

cher

, & C

atta

neo

Nor

mal

ized

Tor

que

V

ortic

ity

inner cylinder

outer cylinder

Re=6200

Computational MRI: preliminary results

Com

pu

tati

on

s F

isch

er,

Ob

ab

ko &

Catt

an

eo

Nonlinear development of Magneto-Rotational Instability• Cylindrical geometry similar to Goodman-Ji experiment

• Hydrodynamically stable rotation profile• Weak vertical field

• Use newly developed spectral element MHD code• Try to understand differences between experiments and simulations

• Simulations ReRm (moderate). Experiments Re>>Rm (Rm smallish)

Summary & Conclusions

• Block-structured SEM provides an efficient path to high-order– accurate treatment of challenging physics– fast cache-friendly operator evaluation ( N 4 vs. N 6 )

• For anisotropic diffusion– effects of grid imprinting are minimized– able to capture physics of high anisotropy

• MRI experiment– MRI has been observed with axial periodicity at Re=Rm=1000.– preliminary axisymmetric results indicate hydrodynamic unsteadiness at

Re=6000 for two-ring boundary configuration, may be mitigated in 3D…

• Free-surface MHD– Free-surface NS is working– Coupling with full MHD is underway