View
218
Download
0
Category
Preview:
Citation preview
Terascale Terascale Simulation Simulation Tools and Tools and TechnologiTechnologi
eses
Spectral Elements for Anisotropic Diffusion and Incompressible MHD
Paul Fischer
Argonne National Laboratory
Terascale Simulation Tools and Technologies
• Goal: Enable high-fidelity calculations based on multiple coupled physical processes and multiple physical scales
– Adaptive methods– Advanced meshing strategies– High-order discretization strategies
• Technical Approach: – Develop interchangeable and interoperable software components
for meshing and discretization– Push state-of-the-art in discretizations
Terascale Simulation Tools and Technologies
• Goal: Enable high-fidelity calculations based on multiple coupled physical processes and multiple physical scales
– Adaptive methods– Advanced meshing strategies– High-order discretization strategies
• Technical Approach: – Develop interchangeable and interoperable software components
for meshing and discretization– Push state-of-the-art in discretizations
Outline
• Driving physics
• SEM overview
– Costs
– MHD formulation
– Convective transport properties of SEM
• Anisotropic Diffusion
• Brief summary of current results on MHD project
• Conclusions
Driving Physics
• Anisotropic Diffusion: ( S. Jardin, PPPL)
– Central to sustaining high plasma temperatures– Want to avoid radial leakage– Need to understand significant instabilities
• Incompressible MHD: ( F. Cattaneo UC, H. Ji PPPL, … )
– Several liquid metal experiments are under development to understand momentum transport in accretion disks
• Magneto rotational instability (MRI) is proposed as a mechanism to initiate turbulence capable of generating transport• The magnetic Prandtl number for liquid metals is ~ 10-5
Re=106 & Rm=10 for experiments• Numerically, we can achieve Re=104 & Rm=103 ( 2005 INCITE award )
– Free-surface MHD ( H. Ji, PPPL )• Proposed as a plasma-facing material for fusion• Desire to understand the effect of B on the free surface
Spectral Element Overview
• Spectral elements can be viewed as a finite element subset• Performance gains realized by using:
– tensor-product bases (quadrilateral or brick elements)– Lagrangian bases collocated with GLL quadrature points
• Operator Costs: Standard FEM* SEM
# memory accesses: O( EN 6 ) O( EN 3 )# operations: O( EN 6 ) O( EN 4 )
N = order, E = number of elements, EN 3 = number of gridpoints
• Dramatic cost reductions for large N ( > 5 )
2D basis function, N=10
Computational Advantages of Spectral Elements
• Exponential convergence with N• minimal numerical dispersion / diffusion
• for anisotropic diffusion, lements of ker(A||) well-resolved – sharp decoupling of isotropic and anisotropic modes
• Matrix-free form• matrix-vector products cast as efficient matrix-matrix products
• number of memory accesses identical to 7-pt. finite difference
• no additional work or storage for anisotropic diffusion tensor
SEM Computational Kernel on Cached-Based Architectures
– matrix-matrix products, C=AB: 2N 3 ops for 2N 2 memory references
– much of the additional work of the SEM is covered by efficient use of cache – e.g., time for A*B vs A+B for N=10 is:
2.0 x on DEC Alpha, 1.5 x on IBM SP
A*B
mxm vs m+m
A*B
A+BA+B
Incompressible MHD• t
— plus appropriate boundary conditions on u and B
• Typically, Re >> Rm >> 1
• Semi-implicit formulation yields independent Stokes problems for u and B
Incompressible MHD in a Nutshell• t
• Convection:– dominates transport – dominates accuracy requirements
• often the challenging part of the discretization
– treated explicitly in time
• Diffusion:– “easy” ( ?? )
• Projection: div u = 0 div B = 0– dominates work– isotropic SPD operator
• multiple right-hand side information• scalable multilevel Schwarz methods ( 1999 GB
award )• SE multigrid ( Lottes & F 05 )
High-Order Methods for Convection-Dominated Flows
Phase Error for h vs. p Refinement: ut + ux = 0
h-refinement p-refinement
High-Order Methods for Convection-Dominated Flows
• Fraction of accurately resolved modes (per space direction) is increased only through increased order – Savings cubed in R3
• Rate of convergence is extremely rapid for high N– Important for multiscale / multiphysics problems
( Q: Why do we want 10 9 gridpoints? )
• Stability issues are now largely understood– stabilization via DG, filtering, etc.– dealiasing
• Still, must resolve structures (no free lunch…)– Computational costs are somewhat higher– Data access costs are equivalent to finite differences
c = (-x,y)
c = (-y,x)
Stabilizing convective problems:
Models of straining and rotating flows:– Rotational case is skew-symmetric.
– Filtering attacks the leading-order unstable mode.
– Dealiasing (high-order quadrature) yields imaginary eigenvalues – vital for MHD
N=19, M=19 N=19, M=20
strainingfield
rotationalfield
CEMM Challenge Problems S. Jardin, PPPL
1. Anisotropic diffusion in a toroidal geometry
2. Two-dimensional tilt mode instability
3. Magnetic reconnection in 2D
• Provides a problem suite that – captures essential physics of fusion simulation– stresses traditional numerical approaches– identifies pathways for next generation fusion codes
• Excellent vehicle for initiating SciDAC interatctions.
Anisotropic Diffusion in Toroidal Domains
•b – normalized B-field, helically wrapped on toroidal surfaces
•thermal flux follows b.
High degree of anisotropy creates significant challenges
• For this problem is more like a (difficult)
hyperbolic problem than straightforward diffusion.
• This is reflected in the variational
statement for the steady case with
• Numerical challenges:– radial diffusion,– nearly singular, with large (but finite) null space– avoid grid imprinting– unsteady case constitutes a stiff relaxation problem– preconditioning nearly singular systems
• CEMM challenge:– establish spatial convergence for steady state case– check unsteady energy conservation when – investigate the behavior of the tearing mode instability
High degree of anisotropy creates significant challenges
total number of gridpoints
cent
erpo
int e
rror
Steady State Error – 3D, || = 10 8
It is advantageous to use few elements of high order
•Fewer gridpoints are required•CPU time proportional to number of gridpoints
(N odd, 3-7)
(N even, 2-6)
High Anisotropy Demands High Accuracy
• A = || A|| + AI
• AI controls radial diffusion
– A|| must be accurately represented when || >> 1
– Error must scale as ~ 1 / ||
Steady-State L2 – error over a range of discretizations
E
N
High Anisotropy Demands High Accuracy
•Difficulty stems from high-frequency content in null space of A||
•High-order discretizations are able to accurately represent these functions.
N=16
k
N=2
k
k
18th mode in circular geometry, || = 108
Error vs. tT vs. r,t
N=14Error vs. r, N=12
Evolution of Gaussian Pulse for
– minimal radial diffusion– no grid imprinting
– careful time integration required (e.g., adaptive DIRK4)
-averaged temperature vs. time
SEM is able to identify critical physics –
• Tearing mode instability– radial perturbation:
• b = b0 + cos(m-n) r
– field lines do not close on m-n rational surface
– magnetic island results, with significant increase in radial conductivity.
– island width scales as:• W ~ in accord
with asymptotic theory
Note: this is a subtle effect!
W
Island width vs. || at onset.
W
Outstanding Challenges for Anisotropic Diffusion Simulation
• Preconditioning– need null-space control
– condition number scales as ||
• Non-aligned grids predict early island formation
a32 a32
max
dT
/dr
Axisymmetric Hydro Simulations of Taylor-Couette w/ Rings
• Re=620 steady• Re=6200 unsteady
• Axisymmetric MHD simulations are being carried out now.
• Starting point for 3D simulations, which are being compared with experiments at PPPL.
Com
puta
tion
by O
babk
o,
Fis
cher
, & C
atta
neo
Nor
mal
ized
Tor
que
V
ortic
ity
inner cylinder
outer cylinder
Re=6200
Computational MRI: preliminary results
Com
pu
tati
on
s F
isch
er,
Ob
ab
ko &
Catt
an
eo
Nonlinear development of Magneto-Rotational Instability• Cylindrical geometry similar to Goodman-Ji experiment
• Hydrodynamically stable rotation profile• Weak vertical field
• Use newly developed spectral element MHD code• Try to understand differences between experiments and simulations
• Simulations ReRm (moderate). Experiments Re>>Rm (Rm smallish)
Summary & Conclusions
• Block-structured SEM provides an efficient path to high-order– accurate treatment of challenging physics– fast cache-friendly operator evaluation ( N 4 vs. N 6 )
• For anisotropic diffusion– effects of grid imprinting are minimized– able to capture physics of high anisotropy
• MRI experiment– MRI has been observed with axial periodicity at Re=Rm=1000.– preliminary axisymmetric results indicate hydrodynamic unsteadiness at
Re=6000 for two-ring boundary configuration, may be mitigated in 3D…
• Free-surface MHD– Free-surface NS is working– Coupling with full MHD is underway
Recommended