Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energy’s National Nuclear

Security Administration under contract DE-AC04-94AL85000.

First-Order Stokes Model

With Glen’s Law Viscosity

Available BCs:No-Slip Basal SlidingStress-Free Open Ocean

Objectives

A.G. Salinger, I. Kalashnikova, M. Perego, R.S. Tuminaro, M.S. Eldred and J.D. Jakeman, Sandia National LaboratoriesS. Price, M. Hoffman, Los Alamos National Laboratories

Ongoing Work

Development of the Albany/FELIX Land Ice Dycore using Software Components

SAND 2014-xxxxP

Component-Based StrategyComponent-based approach enables rapid development of new production codes embedded with transformational capabilities

Element Level FillMaterial Models

Sensitivities

Field ManagerDiscretization Library

Remeshing

UQ Solver

Nonlinear SolverTime Integration

Optimization

Objective Function

Local Fill

Mesh Database

Mesh ToolsI/O Management

Input File ParserUtilities

UQ (sampling)Parameter Studies

Mesh I/O

Optimization

Geometry Database

Discretizations

Derivative Tools

AdjointsUQ / PCE

Propagation

ConstraintsError Estimates

Continuation

Constrained Solves

Sensitivity AnalysisStability Analysis

V&V, CalibrationParameter List

VerificationVisualization

PostProcessing

AdaptivityModel Reduction

Memory ManagementSystem Models

MultiPhysics Coupling

OUU, Reliability

Communicators

PartitioningLoad Balancing

Analysis Tools (black-box)

Physics Fill

Composite Physics

Data Structures

Direct Solvers

Linear Algebra

Architecture-Dependent Kernels

Preconditioners

Iterative Solvers

Eigen Solver

System UQ

Analysis Tools (embedded)

Matrix Partitioning

Inline Meshing

MMS Source Terms

Grid TransfersQuality Improvement

Mesh Database

Solution Database

Derivatives

Regression Testing

Bug Tracking

Version Control

Software Quality

Porting

Performance TestingCode Coverage

Mailing Lists

Release Process

Unit Testing

Web Pages

Build SystemBackups

Verification Tests

DOF map

Multi-CoreAccelerators

Linear Programming

Graph AlgorithmsData-Centric Algs

SVDsMap-Reduce

Network Models

Dycore Interfaces and Meshes

UQ: Bayesian Calibration

Convergence & Scalability

Sandia’s components effort includes ~100 interoperable libraries

Solution Verificationusing manufactured solutions

Defining a UQ workflow for stochastic inversion of Basal sliding coefficients: 1. Model Reduction (KLE) 2. PCE Emulator 3. MCMC Calibration using Emulator

Albany/FELIX Ice Sheet Dycore

Develop: robust and scalable unstructured-grid finite element ice sheet code: Stand-alone steady-state model for initialization and calibration Dynamic model when linked to MPAS-LI or CISM for advection Future land ice component of DOE-ACME earth system model

Support: DOE climate missions, such as providing Sea Level Rise predictionsLeverage: software and expertise from SciDAC Institutes (FASTMath, QUEST, SUPER) and hardware from DOE Leadership Class Facilities

Funding: “PISCEES” SciDAC Application Partnership (DOE’s BER + ASCR divisions)PIs: S. Price and E. Ng; collaboration with ORNL, LANL, LBNL, UT, FSU, SC, MIT, and NCAR

Mature dynamic evolution capability under MPAS Perform deterministic and stochastic initialization runs Improve coupling to full earth system model Finish conversion to performance-portable kernelsWe acknowledge the contributions of our PISCEES collaborators, including B. Lipscomb, K. Evans, P. Worley, M. Norman, M. Gunzberger, and C. Jackson, and our many Trilinos/Dakota collaborators, including E. Phipps and L. Swiler

• Finite Element Discretization (Hex, Tet)• Parallel, Unstructured Grid with Partitioning• Automatic Differentiation for Jacobians• Globalized Newton’s Method Nonlinear Solves• Preconditioned Krylov Iterative Solvers• Performance-Portable Kernels (in progress)• Software tools: git / cmake / ctest / jenkins

g=10-1.0

g=10-2.5

g=10-6.0

g=10-10

g=10-10

4 cores334K DOFs

(8km GIS, 5 layers)

16384 cores1.12B DOFs

(0.5km GIS, 80 layers)

Robust Nonlinear Solvesusing Homotopy Continuation

3D Mesh convergence study for GIS model gives theoretical 2nd-Order rate

How many vertical layers do you need? Convergence study for GIS 1km mesh:

Scalability results over 4 mesh bisections:

84

The Albany/FELIX solver can be driven by a CISM or MPAS-LI interface:

CISM MPAS

CISM MPAS-LI

T=70 yrT= 0 yr

We are beginning to do dynamic runs:

CISM (Fortran)Thickness evolution, temperature solve,

coupling to ESM

simple_glide Albany/FELIX (C++)velocity solve

MPAS/Land Ice (Fortran)Thickness evlolution,

temperature solve,coupling to ESM

LandIce_model

C++/Fortran interface,

mesh conversion

C++/Fortran interface,

mesh conversion

• Structured rectangles• Extruded to Hexs

• Unstructured polygons• Dual mesh of triangles• Extruded to Tets

Regional Refinement:

# vertical layers/# cores

# dofs Total Time -

Setup (sec)

Solution Average

Error

5/128 21.0M 519.4 2.827 3.17e-2

10/256 38.5M 525.4 2.896 8.04e-3

20/512 73.5M 499.8 2.924 2.01e-3

40/1024 143M 1282 2.937 4.96e-4

80/2048 283M 1294 2.943 1.20e-4

160/4096 563M 1727 2.945 2.76e-5