SCD User Forum 2005

Dynamically adaptive geophysical fluid dynamics simulation using GASpAR:

Geophysics/Astrophysics Spectral-element Adaptive Refinement

Duane Rosenberg, Aimé Fournier, Paul Fischer, Annick Pouquet

NCAR Institute for Mathematics Applied to Geosciences

Turbulence Numerics Team

SCD User Forum 2005

Motivation: Methodology Requirements

•Variable (e.g., high) order method (Patera 1984)

•Must be non-dissipative for DNS

•Must also model small scales, high Re, complex boundaries

•Must support variety of equations, physical models, geometries code extensibility

•Leverage existing NCAR expertise in science applications, methods and performance

SCD User Forum 2005

GASpAR Features

• Designed for turbulence studies

• Object oriented (C++)– extensive use of inheritance, polymorphism,

templates

• Hierarchical: Elements Fields Eq. solvers SEM operators Bases GBLAS

• Adaptive Mesh (currently 2D unstructured)• Parallelized

SCD User Forum 2005

SCD User Forum 2005

Spectral element numerics

• Based on spectral-element operators

• Typically, use operator-vector products

• basic matrix-matrix products (m_x_m)

SCD User Forum 2005

Handling Nonconformity

SCD User Forum 2005

SCD User Forum 2005

Connectivity uses “mortars” (color matched)

SCD User Forum 2005

Locating neighbors and constructing mortars: VDB (Henderson 1993)

SCD User Forum 2005

‘Mortar’ Data Exchange

• Mortar nodes given unique Morton keys (Warren & Salmon 1993)

• Bin-sort keys and create node/processor map

• Do DSS

SCD User Forum 2005

Adaptation Rules

• Isotropic (for now)• Refinement:

– At most 2 child edges (4 child faces in 3D)– Edge bisection– h-refinement only (currently)– Max. number of refinement levels

• Coarsening:– Cannot coarsen root– All siblings must be tagged– Must not violate Refinement rules

SCD User Forum 2005

Refinement & Coarsening Criteria

• a-posteriori spectral estimator of Mavriplis (1990)

• gradient2 magnitude

• open question for turbulent flows (esp. in various formulations of the equations)

SCD User Forum 2005

SCD User Forum 2005

Development/Porting Strategy

• Initial development under Linux using gcc• Test MPI using MPICH with 1 or 2 procs• Locate bottlenecks with port to Lightning (use

PG compilers with -Minfo=loop,inline) --Find C m_x_m will not vectorize,

but Fortran will!--Factor of 10 speedup in m_x_m!

• Port to IBM SPs (VAC++, f77)--Use gprof, HPM--Debug with totalview and printf’s

SCD User Forum 2005

Advecting Gaussian ‘Lump’

SCD User Forum 2005

Advecting Gaussian ‘Lump’ Mesh

SCD User Forum 2005

Radial N-wave Solution for Nonlinear Burgers Eqn.

SCD User Forum 2005

Rotated Burgers Front

SCD User Forum 2005

Dynamical Systems: Large Scale Passive Advection of Small Scale Field

SCD User Forum 2005

Conclusion

• 2D adaptive algorithm is working• Must include dynamic load

balancing/processor distribution.

• Complete 3D AMR, enhance performance based on 2D; debugging difficult

• Better performance monitoring/reporting on the SPs?

SCD User Forum 2005

Nonconforming Connectivity: non-Boolean gather operator Q maps global (j=0:18) to local (i=0:26) d.o.f.

(after Fischer & Kruse 1997)

Q12,2

Qi,j

SCD User Forum 2005