Numerical simulation of geodynamic processes with the ...foalab.earth.ox.ac.uk/publications/Katz_etal_PEPI07.pdf · Numerical simulation of geodynamic processes with the Portable

Physics of the Earth and Planetary Interiors 163 (2007) 52–68

Numerical simulation of geodynamic processes with thePortable Extensible Toolkit for Scientific Computation

R.F. Katz a,∗, M.G. Knepley b, B. Smith b, M. Spiegelman c,d, E.T. Coon d

a University of Cambridge, Department of Applied Mathematics and Theoretical Physics, Wilberforce Road, Cambridge CB3 0WA, UKb Mathematics and Computer Science Division, Argonne National Laboratory, USA

c Lamont-Doherty Earth Observatory, Columbia University, USAd Department of Applied Physics and Applied Mathematics, Columbia University, USA

Received 14 January 2007; received in revised form 2 April 2007; accepted 26 April 2007

Abstract

Geodynamics simulations are characterized by rheological nonlinearity, localization, three-dimensional effects, and separatebut interacting length scales. These features represent a challenge for computational science. We discuss how a leading softwareframework for advanced scientific computing (the Portable Extensible Toolkit for Scientific Computation, PETSc) can facilitatethe development of geodynamics simulations. To illustrate our use of PETSc, we describe simulations of (i) steady-state, non-Newtonian passive flow and thermal structure beneath a mid-ocean ridge, (ii) magmatic solitary waves in the mantle, and (iii)the formation of localized bands of high porosity in a two-phase medium being deformed under simple shear. We highlight two
supplementary features of PETSc, structured storage of application parameters and self-documenting output, that are especiallyuseful for geodynamics simulations.© 2007 Elsevier B.V. All rights reserved.
PACS: 91.40.Jk; 91.40.St; 91.45. −c; 91.45.Fj; 47.11.Df; 02.60.Cb

cs; Mid
Keywords: Scientific computation; Parallel computation; Geodynami
1. Introduction

Two major challenges in solid-Earth geodynamicsare the derivation of theoretical descriptions of geo-dynamic phenomena in terms of partial differentialequations (PDEs) and the determination of solutions
to these equations for appropriate boundary and initialconditions. Even minimally complex geodynamic mod-els often contain important three-dimensional effects,
∗ Corresponding author.E-mail address: [email protected] (R.F. Katz).URL: www.damtp.cam.ac.uk/user/rfk22 (R.F. Katz).

0031-9201/$ – see front matter © 2007 Elsevier B.V. All rights reserved.doi:10.1016/j.pepi.2007.04.016

-ocean ridge; PETSc

sharp gradients in model variables, strong nonlinearitiesin material properties such as viscosity, and localizationprocesses that lead to separated but interacting lengthscales. These features usually preclude analytical solu-tion of the governing equations and enforce a relianceon numerical simulations. However, the same featuresalso contribute to the difficulty of efficiently generatingnumerical solutions. The development of effectivesimulations can be facilitated by advanced numericalsoftware libraries, of which the Portable Extensible
Toolkit for Scientific Computation (PETSc) (Balay etal., 2001, 2004) is a leading example. The purpose of thispaper is to demonstrate how PETSc has facilitated thedevelopment of large-scale computational simulations
mailto:[email protected]

dx.doi.org/10.1016/j.pepi.2007.04.016

and Pl

os

ptSlsdlSis

1

aMitmpamppicmihzsIsm

Mctvpmf

φ

R.F. Katz et al. / Physics of the Earth

f a classic geodynamic theory for the coupled flow ofolid and molten mantle rock (McKenzie, 1984).

Section 2 gives a general description of the PETScackage and greater detail on the parts of the packagehat we have employed for geodynamics simulations.ection 3 considers two example geodynamic simu-

ations, the first of single-phase Stokes flow and theecond of two-phase Darcy–Stokes flow. Section 4escribes special features of PETSc that are particu-arly useful for developing geodynamics simulations.ection 5 discusses both the future developments

n PETSc and the challenges of future geoscienceimulations.

.1. Governing equations for mantle dynamics

Many of the expressions of mantle dynamics observ-ble on the surface of the Earth are related to volcanoes.ajor elements, trace elements and the isotopic chem-

stry of lavas, for example, are partially controlled byhe spatial distribution of melting and the paths of

elt transport. The position of volcanoes relative tolate boundaries as well as to other volcanoes is alsoconsequence of the coupled dynamics of magma andantle rock. Theoretical models are required to inter-

ret these observations in terms of fluid mechanicalrocesses occurring at depth. A theory for the dynam-cs of the mantle should therefore describe both a solid,rystalline phase, which makes up the vast bulk of theantle, and a liquid phase (magma or fluid), which

s present in the mantle beneath volcanically activeotspots and tectonic plate-boundaries. In the limit ofero fluid fraction, this theory should reduce to thetandard equations used to describe mantle convection.n the limit of a rigid crystalline mantle, the theoryhould reduce to Darcy’s law for flow in a permeableedium.One derivation of such a theory is provided by

cKenzie (1984), which considers the macroscopiconservation of mass, momentum, and energy forwo interpenetrating continua consisting of a low-iscosity fluid in a high-viscosity deformable andermeable matrix. The equations for conservation ofass and momentum for both phases can be written as

ollows:∂ρfφ

∂t+ ∇ · [ρfφv] = Γ, (1)

∂ρs(1 − φ)

∂t+ ∇ · [ρs(1 − φ)V] = −Γ, (2)

(v − V ) = −K

μ[∇P − ρfg], (3)

anetary Interiors 163 (2007) 52–68 53

∇P = ∇ · (η[∇V + ∇VT])

+ ∇[(

ζ − 2

3η

)∇ · V

]+ ρg. (4)

Here φ is porosity, ρf and ρs are the fluid and solid den-sities, v and V are the fluid and solid velocity fields, Γ

the rate of mass transfer from solid to liquid (i.e. melt-ing/crystallization rate), K the permeability, μ is the meltviscosity, P the fluid pressure, ρ = φρf + (1 − φ)ρs thephase-averaged density, g the acceleration due to grav-ity, and η, ζ are the shear and bulk viscosities of the solid(see Spiegelman, 1993a, b, for further discussion).

Eqs. (1) and (2) conserve mass for the fluid andsolid individually and allow mass-transfer between thephases through Γ . Eq. (3) is an extended form of Darcy’slaw governing the separation of melt from solid. Thisseparation flux is proportional to the permeability andfluid-pressure gradients in excess of hydrostatic. Eq.(4) governs momentum conservation of the solid phasewhich is modeled as a compressible, inertia-free viscousfluid.

An important feature of Eqs. (1)–(4) is that they con-sistently couple solid stresses and fluid pressure. Thefluid pressure responds to solid deformation and grav-ity which drives fluid flow and changes the porosity.Variations in porosity and stress can then feed-backthrough the constitutive relations for the permeabilityand viscosity. Such feedbacks lead to a wide range ofbehavior including non-linear porosity waves (e.g., Scottand Stevenson, 1984, 1986; Barcilon and Richter, 1986;Barcilon and Lovera, 1989) and spontaneous flow local-ization (e.g., Stevenson, 1989; Katz et al., 2006). In mostcases, solution of these equations requires numericalmethods because nonlinearities in the governing equa-tions preclude analytical treatment. To find numericalsolutions, we need software that is capable of solvinglarge systems of coupled, non-linear algebraic equations;PETSc is a leading example of such software.

1.2. Methods for solving nonlinear PDEs

Discretization of nonlinear partial differential equa-tions onto a mesh leads to a system of nonlinear algebraicequations. Such a system can be represented as

F (u) = 0, (5)

where u ∈Rn is a vector containing the exact solution ofthe problem and F is a nonlinear function of u that maps
R
N → RN . In practice, for large systems of equations,

it is difficult to find a vector u such that F (u) is exactlyequal to the zero vector. More typically, one is satisfiedwith an approximation to the exact solution that satisfies

and Pl
54 R.F. Katz et al. / Physics of the Earth
the inequality:

||F (u)|| < tol, (6)

where tol is some specified tolerance and the vector norm|| · || is chosen according to the context. The key to find-ing a good approximation to Eq. (5) is to employ aniterative method that reduces ||F (u)|| on each iteration.In describing iterative methods, un is used to representthe approximate solution after n iterations of the solutionmethod.

The most straightforward iteration solution strategyis a fixed-point iteration for the equation u = u + F (u).An initial guess for the solution u0 is used to determine aresidual F (u0), which is then applied as an update to theguess, u1 = u0 + F (u0). If successive iterates converge,they will converge to the solution of F (u) = 0.

A more robust method, referred to as Picard itera-tion in the groundwater literature (e.g., Paniconi andPutti, 1994; Mehl, 2006), can be used to linearize anonlinear system of equations, making it amenable tosolution by a readily available linear solver. This con-version is accomplished by substitution of best currentguesses for field variables into nonlinear terms of thegoverning equations. We employ this method belowto split a set of coupled PDEs into two parts, whichare solved separately (see Section 3.9). Paniconi andPutti (1994) and Mehl (2006) have quantitatively com-pared this successive substitution method with Newton’smethod.

Newton’s method, which is described in more detailbelow, can converge faster than Picard for nonlinear sys-tems derived from PDEs if the initial guess u0 is close tothe solution. To be practical, however, Newton’s methodrequires ready evaluation of the Jacobian as well as somecontrol feature such as line-search or trust-region tech-niques (Steihaug, 1983) to make it more robust. PETScprovides these features. In the next section we give a gen-eral description of PETSc and highlight its capabilitiesrelevant for solving problems resulting from systems ofnonlinear PDEs.

2. PETSc basics

The Portable Extensible Toolkit for Scientific Com-putation is designed to assist the development of “fullphysics” simulations. Its goal is to eliminate imple-mentation as the bottleneck in developing and runningcomplex, multiphysics simulations at a scale that allows
the rapid advancement in science through simulation.PETSc’s focus is on the numerical solution of thealgebraic systems arising when using implicit methodsbased on conventional finite-difference, finite-element,
anetary Interiors 163 (2007) 52–68

and finite-volume techniques for PDEs. Moreover, themodular design fosters reuse of scientific components,such as a Navier–Stokes solver, among different sim-ulations. PETSc also supports a limited but growingset of tools for managing the data distribution and dis-cretization techniques needed to construct the algebraicsystems.

Several excellent general-purpose software environ-ments for numerical computation exist; perhaps thebest known of these is Matlab. In addition, easy-to-use“scripting” languages such as Python allow the rapidprototyping of numerical simulations. However, manyscientific simulations in geodynamics require (and willrequire even more in the future) resolution at a scalein both time and space leading to a system size that isunavailable in these environments. Only parallel (mul-tiprocessor) computing systems can tackle these largeproblems. In fact, many realistic simulations will requirecomputer systems with thousands of processors. Thus,simulations developed in these general-purpose environ-ments must often be completely redeveloped by usinghand-coded parallel computational numerics, using theforegoing system only for visualization and post pro-cessing for statistics.

2.1. Background and philosophy

A wide variety of “parallelization” techniques havebeen proposed over the years to achieve both efficientuse of parallel machines and ease of programmingfor scientists. They have all been, essentially, failures.The result is that the message-passing-model is thestandard parallelization approach for engineering andscientific simulation. One fortunate result of researchand development in parallel computing is the devel-opment the Message Passing Interface (MPI), whichprovides a powerful common interface on all hard-ware systems (Message Passing Interface Forum, 1994,1998).

Message-passing parallel programming has some-times been called the assembly language of parallelcomputing. The programmer must manage every detailof the parallelism and data movement between pro-cesses. With this comes great power and flexibility—infact, an overwhelming amount of flexibility. One goalof PETSc is to eliminate the direct use of MPI pro-gramming from numerical simulations involving thesolution of PDEs. Specifically, the PETSc libraries are
used to manage the details of the communication andthe user is left to orchestrate the overall flow of the com-munication and computation, plus the detailed physicsmodules.

R.F. Katz et al. / Physics of the Earth and Pl

2

cacmttpdp

batPiarda(at

Fig. 1. Partition of mesh across processors.

.2. Distributed arrays and parallelism

PETSc is predicated on mesh decomposition, alsoalled domain decomposition, in order to partition datand computational work among the processes. Each pro-ess is typically assigned a contiguous portion of theesh, as in Fig. 1. The physical data for this portion of

he mesh is stored in local process memory, and compu-ations on that portion of the mesh are performed by thisrocess. Of course, these local calculations will requireata from neighboring partitions, which are termed ghostoints, shown in Fig. 2.

Managing the data movement and coordinationetween processes is the bulk of the difficulty with par-llel computing. This effort is vastly simplified by usinghe local physics/global algebraic solver paradigm inETSc. The fields are restricted to a local representation,

ncluding the ghost points; local physics computationsre performed; the result is used to update the globalepresentation. The conversion between local and globalata layouts, as well as communication, is handled
utomatically by PETSc. Thus the algebraic solversNewton’s method, linear iterative solvers, etc.) see onlyglobal algebraic representation of the problem, while
he physics modules see only the data on a local portion

Fig. 2. Examples of G


of the mesh. The beauty is that at this abstract level, thedata management is independent of the particular phys-ical model, be it fluid dynamics, structural mechanics,MHD, or the like.

The DA construct in PETSc is the special case of thisparadigm for computation on structured tensor productgrids. It combines specification of the topology, geome-try, and process interconnection. The user supplies onlya routine to evaluate the nonlinear residual and option-ally the Jacobian over a given local piece of the mesh.The local fields are presented to the user code, not asabstract objects, but as the more familiar multidimen-sional array in both C and Fortran, giving expressionsthat conform to stencil indexing (i.e., u(i, j, k)) insteadof vector (i.e., u(I)) indexing. If the Jacobian isalso user provided, PETSc provides additional supportfor index translation with the MatSetValuesStencilmethod. It automatically translates (i , j , k ) meshcoordinates to global matrix indices needed for thealgebraic solver. This approach works equally wellfor tensor product finite-element, finite-volume, orfinite-difference formulations. The key point is thatthe physics modules need only know about the localmesh representation, never the global algebraic solverrepresentation.

In addition, there is support for geometric multigridwith automatic or custom interpolation and coarse-gridoperators. Since the user has provided an evaluation rou-tine for a general grid patch, coarse representations ofthe operator can be obtained directly. They may alsobe obtained automatically by PETSc via the algebraicGalerkin process.

2.3. Solvers

The PETSc algebraic solvers always work with the
global algebraic representation of the fields. This allowsthe solver software to be used in virtually any application,independent of the particular physics, discretization, oreven local representation of the fields.
host points.

and Pl
PETSc uses (truncated, approximate) Newton’smethod to solve the nonlinear algebraic equations. Thatis,

un+1 = un + δun, (7)

where δun is obtained by approximately solving:

J(un)δun = −F (un), (8)

where F (un) is the global residual at iteration n andJ is the (approximate) Jacobian. The computation ofF (un) is done as described above in the local meshrepresentation. The DA PETSc infrastructure automat-ically manages the translation of the results of thelocal physics modules into the global representationused by algebraic solvers. For non-matrix-free Newton’smethod, PETSc computes the Jacobian matrix usingfinite differences via J∗j(un) = ∇ujF (un) ≈ (F (un +hej) − F (un))/h, where h is computed dynamically toprovide the best approximation (unless the Jacobian isprovided by a subroutine in the application code). HereJ∗j represents the jth column of the Jacobian, ej is thejth column of the identity matrix, ∇ujF (un) is the vectorof derivatives of all of the Fi with respect to ujwhichby definition is the jth column of the Jacobian, and his a suitably chosen differencing quantity, roughly 10−7

times the norm of u. Naively, one would need N com-putations of the residual to compute all the columns ofJ; again N is the total number of unknowns. Fortunately,because of the sparsity of J, all the columns of the J thatdo not share a common row can be computed by usingthe same discrete residual evaluations. This computationreduces the number of discrete residual calculations tothe number of colors of a particular graph of the matrixJ that is bounded and independent of N, the size of theproblem. In addition, since the perturbations needed inthe differencing are local, one can perform all of thesefunction evaluations without parallel communications,dramatically decreasing the cost.

To specify the problem to PETSc, the user defines asolution tolerance and provides call-back functions thatgenerate an initial guess of the solution and calculateeach component of the residual vector rn = F (un) givena vector of field variables un. All of the physics of theproblem resides in the latter of these two call-back func-tions. We have found that incomplete LU preconditionedGMRES (Demmel, 1997) gives robust and scalable per-formance on these particular problems, and we have usedit in generating the simulation results described below.

PETSc provides a range of linear solvers (see www.mcs.anl.gov/petsc/petsc-as/documentation/linearsolvertable.html for the current comprehensive list). Thesolvers are generally a combination of a fixed-point


solver (called a preconditioner) such as Gauss–Seidelor multigrid, plus a Krylov method that acceleratesthe convergence of the preconditioner, such as theconjugate gradient method or GMRES. Since the use ofthese solvers is independent of the particular physics,they may all be selected at runtime. This approachallows the optimal solver for a particular applicationto be determined rapidly via a series of runs, withoutrequiring recompiles between the changes in the solvers.

3. Simulations of one and two-phase mantledynamics

3.1. Single phase limit

In the limit that porosity, φ, and melting rate, Γ ,are both zero, Eqs. (1)–(4) reduce to Stokes flow foran incompressible fluid. Coupled with an equation forthe conservation of energy, this system can be solvedfor the thermal and flow structure of a convecting fluid.Many authors have used these equations to study thekinematically driven flow of mantle rock in mid-oceanridge and subduction zone settings (e.g., van Keken et al.,2002; Kelemen et al., 2002; Gerya and Yuen, 2003). Thechallenge in performing such calculations arises fromnonlinearities of the constitutive equation that describesthe viscosity of mantle rock. Over geologic timescales,the mantle behaves as a fluid with a non-Newtonian,temperature-dependent viscosity (Karato and Wu, 1993;Kelemen et al., 1997a):

η = A0 exp

(E∗ + PV ∗

RT− αφ

)ε

(1−n)/nII , (9)

where A0 is a constant of proportionality, E∗ and V ∗ arethe activation energy and activation volume, R the gasconstant, α ≈ 27 is an empirically determined constant(Hirth and Kohlstedt, 1995a, b; Mei et al., 2002), φ theporosity in volume fraction, εII the second invariant ofthe strain rate tensor, and n is a constant describing thestrain-rate dependence of viscosity. For dislocation creepn ≈ 3.5, while for diffusion creep n = 1.

Neglecting the temperature and strain-rate dependen-cies (i.e., assuming a constant viscosity) with certainkinematically prescribed boundary conditions allows foranalytic solution of incompressible Stokes for the “cor-ner flow” solution (Batchelor, 1967). This solution hasbeen used extensively to model the mantle at tectonicplate boundaries (e.g. McKenzie, 1969; Spiegelman
and McKenzie, 1987). Kelemen et al. (2002), however,showed that isoviscous models cannot meet constraintsderived from petrologic and heat-flow data at subduc-tion zones. Variable viscosity models of mantle flow
http://www.mcs.anl.gov/petsc/petsc-as/documentation/linearsolvertable.html




Fa

cpdev(

tioae

3

es1uoeg

wa�

titma

ig. 3. Control volume Ωijk , labeled to illustrate the position of vari-bles on the staggered mesh. Adapted from Albers (2000).

an meet these constraints, but such models generallyreclude analytic solution because the rheology intro-uces nonlinearity into the governing equations. We havemployed PETSc to generate numerical solutions forariable-viscosity flow and thermal structure in ridgeKatz et al., 2004) and arc (Knepley et al., 2006) settings.

Next we detail the discretization and solution strategyhat we have employed to model the three-dimensional,ncompressible, single-phase flow and thermal structuref the mantle beneath mid-ocean ridges. We discuss par-llel performance of this code and briefly explore anxample of the solutions that we obtain.

.2. Discretization

We use a finite volume discretization of the governingquations on a staggered mesh, shown in Fig. 3, to avoidpurious grid-scale oscillations of pressure (Patankar,980). Calculating fluid velocities on the control vol-me boundaries leads to simple discrete representationsf the governing equations. For example, the continuityquation ∇ · V = 0 is integrated over a control volumeiving:

uijk − ui−1jk

�x+ vijk − vij−1k

�y+ wijk − wijk−1

�z= P r

ijk,

(10)

here u, v, and w are the velocity in the x, y,nd z-direction and the cell dimensions are given byx, �y, �z. P r

ijk is the residual of the continuity equa-ions in cell Ωijk and corresponds to the pressure variable
n our approach, even though Eq. (10) does not containhe pressure. Pressure in cell Ωijk is constrained by the
omentum equations for u, v, w in Ωijk and its immedi-te neighbors. The residuals for the discrete momentum


equations are assigned to urijk, vr

ijk, and wrijk. These equa-

tions are presented in detail in Albers (2000).Temperature is governed by the conservation of

enthalpy equation:

ρcPV · ∇T = ∇ · k∇T, (11)

which requires that, in steady state, advection of heatbe balanced by diffusion of heat. Here T is the mantlepotential temperature, and k, ρ, and cP are the ther-mal conductivity, density and specific heat of the solidmantle. This equation is discretized (see Albers (2000))to give an expression for the residual T r

ijk. FollowingTrompert and Hansen (1996) we use a Fromm upwindadvection scheme.

To completely specify the discrete problem, we mustspecify boundary conditions. The grid of control vol-umes is constructed so that the domain boundaries fallon the edges of cells and there are a sufficient numberof buffer cells outside the domain boundary to accom-modate the boundary stencil. The vertical velocity, w,has a mesh position that coincides with the domainboundary when a cell is adjacent to a horizontal bound-ary. Thus, to impose w = 0 on the top boundary ofthe domain, we must specify that wr

ij0 = wij0. We uselinear interpolation to enforce Dirichlet boundary con-ditions on variables that, because of staggering of themesh, do not fall on the domain boundaries. For exam-ple, the residual for temperature on the top boundary isgiven by T r

ij0 = Tij0 + Tij1. When T rij0 = 0, we have suc-

cessfully imposed T (x, y, 0) = 0 (T represents the non-dimensional temperature in this case). The complete setof boundary conditions for the domain is given in Table 1.

Each control volume Ωijk has five degrees of free-dom {u, v, w, P, T } and their corresponding residuals.The total set of degrees of freedom and residuals can beassembled into two vectors, u and r, of length N = 5Nc,where Nc is the total number of grid cells. The approxi-mate solution to the discrete nonlinear system F (u) = 0is then given by an unknown vector u such that ||F (u)|| =||r|| ≤ tol, where tol is a tolerance specified by the userand is typically chosen to be 1 × 10−5 or less.

3.3. Solution strategy: continuation

In general, the convergence of Newton’s methodrequires a good initial guess. We have certainly foundthis situation to be true in the case of variable viscosityflow beneath a mid-ocean ridge. While the 2D analytic
corner flow solution for constant viscosity is a good start-ing guess for 2D simulations, it is clearly not good for 3Dflow beneath a ridge with transform offsets. We there-fore adopt a continuation method, forcing the variation

58 R.F. Katz et al. / Physics of the Earth and Pl

Table 1Boundary conditions for the 3D ridge simulation

Boundary Variable Boundary condition

z = 0 u u(x, y, 0) = U(x, y), where U is theimposed plate motion.

v v = 0 on top boundaryw w = 0 on top boundaryP This BC is irrelevant for the interior of

domainT dimensionless T = 0 on top boundary

z = D u u = 0 on bottom boundary.v v = 0 on bottom boundaryw dw/dz = 0 on bottom boundaryP P = 0 on bottom boundaryT Dimensionless T equals the

(dimensionless) mantle potentialtemperature

x = 0, L u u satisfies Eq. (10).v σxy = 0w σxz = 0P This BC is irrelevant for the interior of

the domainT dT/dx = 0

y = 0, W all von Neumann (reflection) condition

Many authors have developed 3D numerical modelsof mantle flow beneath mid-ocean ridges with trans-form faults. Early examples used constant viscosity

Fig. 4. Normalized run times as a function of the number of proces-sors for solution of the 3D, single-phase, steady-state flow and thermalstructure problem of a mid-ocean ridge. The problem size is held con-stant as the number of processors increases. (The grid spacing is 2.5 kmin each direction for a total of ∼7 × 105 degrees of freedom.) Ideally,doubling the number of processors would reduce the run time by a fac-

The domain is size x ∈ [0, L], y ∈ [0, W], z ∈ [0, D]. The spreadingridge is parallel to the y-axis, transform faults, and the spreading ratevector parallel to the x-axis.

in viscosity to go from zero to the full predicted varianceover a set of iterations of the nonlinear solve (Knepley etal., 2006). To smoothly control the variation of viscosity,we set an upper limit on viscosity and use this limit tonormalize it to a range between zero and one. The vis-cosity field η∗

m used in iteration m of the continuationloop is then given by

η∗m = ηαm, αm ∈ [0, 1], (12)

where m = 1, 2, . . . , M. In the first iteration α1 = 0,giving the solution to the isoviscous case. This solutionis then used as a guess for the next solve with α2 > α1.The iteration loop ends with the solution at αM = 1,which has full variation in viscosity. For diffusion creepviscosity (Newtonian, temperature-dependent), between5 and 10 continuation steps are required, where dislo-cation creep (non-Newtonian, temperature-dependent)typically requires between 10 and 30 iterations. For itera-tions with m < M, a relaxed nonlinear tolerance may beused (e.g., tol = 10−3) to reduce the number of Newtonsteps that are necessary and hence to speed the continu-ation method.

3.4. Parallel scaling

Although most 2D simulations can be run in serialon a single processor, 3D simulations with reason-


able spatial resolution typically cannot. The simulationsdiscussed above, which require a grid resolution of1–5 km in a domain of order 300 km in each direc-tion, must be run in parallel. Using PETSc facilitatesthe transition from a serial platform to a parallel super-computer in terms of code development: in most cases,application code contains no explicit interprocess com-munications. A properly written application code thatuses PETSc-provided data structures and methods isinherently parallel. This parallelization, however, doesnot guarantee perfect scaling. How the performance ofthe application code scales to large numbers of proces-sors depends on the chosen solvers and on the structureof the application code itself.

Fig. 4 shows scaling results for the 3D ridge simula-tion code compared with ideal scaling for a fixed problemsize. The application uses an additive Schwartz methodfor domain decomposition with a ILU preconditioner oneach block (Cai et al., 1997).

3.5. 3D ridge results

tor of one-half; the data show that this application achieves about 80%of ideal scaling. The calculations were performed on an IBM BlueGene/L system with 1024 dual PowerPC compute nodes. Code wascompiled with IBM’s BG/L-optimized C compiler. More informationon this system is available at www.bgl.mcs.anl.gov.

http://www.bgl.mcs.anl.gov


Fig. 5. Output from a mid-ocean ridge simulation with a full spread-ing rate of 10.8 cm/year. The pattern of ridge segments and transformfaults, shown by the magenta lines, approximates the geometry ofthe Eastern Pacific Rise around the Siqueiros and Clipperton trans-form faults at about 9◦N. (a) Three-dimensional representation ofupwelling. The transparent, colored isosurfaces connect points of con-stant upwelling rate. Yellow, green, and red surfaces denote the 3.0,3.5, and 4.05 cm/year isosurfaces, respectively. A 2D slice throughthe potential temperature field of the model is shown on the back-leftpanel of the graph. On this panel, blue is 0 ◦C and red is 1300 ◦C, theassigned potential temperature of the deep mantle. The base of thethermal boundary layer is the depth at which the potential tempera-ture reaches 1300 ◦C. This depth is shown, over the interior of the 3Ddomain, by the mesh surface. (b) The vertically integrated melting ratein color. Regions over which melt is focused to each ridge are boundedby white lines. The ridge trace is shown in magenta. Calculations per-formed to determine the generation and focusing of melt are describedifi

((sStijwdt

3rctfls

The 3D ridge models provide the large-scale frame-

n the main text. (For interpretation of the references to colour in thisgure legend, the reader is referred to the web version of the article.)

Phipps Morgan and Forsyth, 1988), layered viscosityRabinowicz et al., 1993), and temperature and pres-ure dependent viscosity (Shen and Forsyth, 1992).parks and Parmentier (1993) considered 3D convec-

ion beneath a mid-ocean ridge. Georgen and Lin (2002)nvestigated 3D isoviscous flow beneath a ridge tripleunction. The model described here differs from earlierork in that it incorporates the effects of non-Newtonianislocation creep viscosity on the flow and thermal struc-ure beneath a mid-ocean ridge.

While a thorough investigation of the behavior of theD ridge model is beyond the scope of this paper, a rep-esentative result is shown in Fig. 5. In this case, we havehosen the ridge geometry and spreading rate to mimiche region around the Clipperton and Siqueiros trans-
orm faults of the Eastern Pacific Rise at about 9◦northatitude. This is a fast-spreading ridge with a robust meltupply (Crawford and Webb, 2002).

Grid resolution for this simulation is 4 km in eachdirection. The computation, performed on 512 nodes ofthe Blue Gene/L at Argonne National Laboratory, solvedfor 2.8 million degrees of freedom (however this simula-tion was not used for the scaling test, shown above). Thedeformation rheology used in the simulation combinesdiffusion and dislocation creep and predicts mantle vis-cosities as low as 5 × 1018 Pa s in the region beneath thespreading center.

Fig. 5a shows isosurfaces of upwelling velocity.These surfaces suggest that upwelling beneath a fast-spreading ridge is uniform beneath most of a ridgesegment. Near the ends of segments bounded by largetransform offsets, upwelling rates are diminished, andmantle is drawn into the spreading zone laterally fromacross the transform fault. Vertically integrated meltproduction, shown in Fig. 5b, reflects this variation inupwelling, with lower melt production rates beneathsegment ends than beneath segments centers. Note, how-ever, that a small offset in the ridge has little effect onupwelling or melt production. The low predicted viscos-ity could permit buoyant convection beneath the ridge,driven by horizontal gradients in porosity (Buck and Su,1989). This would lead to a time-dependent solution andshould be explored in the context of two-phase models.

For this 3D model, melting is calculated by usingthe lherzolite solidus of Katz et al. (2003) and an adia-batic productivity of 0.4% per kilometer of upwelling,chosen to give a reasonable crustal thickness (Katz etal., 2004). Melt focusing to the ridge is parameterizedas a process of melt percolation up a sloping solidus(e.g., Sparks and Parmentier, 1991; Magde and Sparks,1997). The maximum melt focusing distance is arbitrar-ily set to 80 km in Fig. 5. Changing this distance affectsthe predicted crustal production rate. For the simulationshown in Fig. 5, changing the maximum melt focusingdistance from 80 to 30 km reduces the predicted crustalthickness from about 7.5 to 4.9 km. Changes in meltfocusing may also affect crustal thickness asymmetryacross transform faults on migrating ridges (Carbotteet al., 2004; Katz et al., 2004). Simple parameteriza-tions of melt transport are useful for gaining insight intomid-ocean ridge processes, but more quantitative mod-els will depend on solutions to the equations governingtwo-phase mantle/magmatic flow.

3.6. Two phase flow

work for understanding the solid flow field and thermalstructure beneath mid-ocean ridges. A more completedescription of these regions, however, includes the

and Pl
production and migration of partially molten rock. For-tunately, Eqs. (1)–(4) provide a consistent and tractableextension of solid-state mantle convection to includemagma dynamics. These equations can be rewritten in aform that is amenable to numerical solution as a couplingof compressible Stokes with Darcy’s law. The surprisingfeature of this coupled system is that it has consider-ably richer behavior than either of the two subproblemsalone. In particular, these equations are unstable to thespontaneous formation of time-dependent, small-scalecoherent structures such as solitary waves or “meltbands” (see below).

The key to the new formulation is to partition the totalpressure P into three components:

P = Pl + P + P∗, (13)

where Pl = ρ0s gz is the reference background “litho-

static” pressure, P = (ζ − 2η/3)∇ · V is the “com-paction” pressure due to expansion or compaction of thesolid, and P∗ includes all remaining contributions to thefluid pressure (particularly the dynamic pressure due toviscous shear of the matrix).

With these definitions and a bit of algebra, we caneliminate the melt velocity v from the equations using thesame basic manipulations as in Spiegelman (1993a) (seealso Spiegelman et al., 2001; Spiegelman and Kelemen,2003) If we approximate ρf, ρs to be constant (but notequal), we can rewrite the equations as

Dφ

Dt= (1 − φ)

Pξ

+ Γ

ρs, (14)

−∇ · K

μ∇P + P

ξ= ∇ · K

μ[∇P∗ + �ρg] + Γ

�ρ

ρfρs,

(15)

∇ · V = Pξ

, (16)

∇P∗ = ∇ · η(∇V + ∇VT) − φ�ρg, (17)

where Dφ/Dt = ∂φ/∂t + V · ∇φ is the material deriva-tive of porosity in the frame of the solid, ξ

= (ζ− = 2η/3) and �ρ = ρs − ρf.Eq. (14) is an evolution equation for porosity in a

frame following the solid flow. In this frame, porositychanges are driven by the balance of physical volumechanges (P/ξ ≡ ∇ · V) and melting. Eq. (15) is a modi-fied Helmholtz equation for the compaction pressure P,which reduces to Darcy flow in rigid porous media in
the limit ξ → ∞. This equation is responsible for muchof the novel behavior in this system and has been dis-cussed in detail in Spiegelman (1993a, b). Eq. (16) relatesthe divergence of the solid flow field to the compaction

pressure, and Eq. (17) is Stoke’s equation for the solidvelocity and P∗ with porosity-driven buoyancy. Givenφ,P, P∗ and V, the melt flux is reconstructed as

φv = φV − K(φ)

μ[∇(P∗ + P) + �ρg]. (18)

All of these equations are in forms readily amenableto analytic and numerical techniques. Eqs. (14)–(17)form a coupled hyperbolic-elliptic set of equations forporosity, pressure and solid flow. To solve these prob-lems requires initial conditions and inflow conditionsfor the porosity and boundary conditions on pressure,solid velocity or stress. Most natural boundary condi-tions for the compaction pressure can be written in termsof the melt flux and tend to be Neumann conditionson P.

3.7. Magmatic solitary waves

The novel features of these equations arise from Eqs.(14)–(15). In the limit of small porosity φ � 1, constantviscosities and neglecting melting or large-scale solidshear, Eqs. (14)–(15) can be written in dimensionlessform as

Dφ

Dt= P, (19)

−∇ · φn∇P + P = ∇ · φng. (20)

These equations have been shown to produce nonlin-ear solitary waves of porosity in 1, 2, and 3 dimensionsthat propagate over a uniform porosity backgroundwith fixed form and constant speed (e.g., Scott andStevenson, 1984; Scott et al., 1986; Scott and Stevenson,1986; Richter and McKenzie, 1984; Barcilon andRichter, 1986; Barcilon and Lovera, 1989; Wiggins andSpiegelman, 1995).

These waves are a natural consequence of the abil-ity of the matrix to dilate or compact in response tovariations in melt flux. Perhaps more importantly, theyprovide an excellent benchmark test for computationalmethods. Given a single solitary wave of the appropriatedimension, it should propagate with unchanging formand constant phase velocity. Any other behavior is anartifact of the numerical method. We have developedseveral PETSc-based examples for 2D solitary wavesthat incorporate a highly accurate spectral solution forindividual solitary waves in all dimensions (G. Simpson,
personal communication, 2006). The PETSc codes aresolved on a regular mesh that takes advantage of the DAabstraction as well as geometric multi-grid precondi-tioners and a semi-Lagrangian method of characteristics

and Pl

sb

3

fipeppdeiaetevcp

tebttaa(bepta

ttvpcdfil(

Nrtb


olver extension to PETSc to accurately solve the hyper-olic advection components.

.8. Localization and the formation of melt bands

The solitary wave solutions represent excellent veri-cation tests for simulations. However, a more difficultroblem arises in solving the full system of two-phasequations where shear deformation of the solid cou-les with volumetric deformation and the evolution oforosity. Such problems are motivated by experimentsescribed by Zimmerman et al. (1999) and Holtzmant al. (2003a) that demonstrate spontaneous melt local-zation in a deforming, partially molten two-phaseggregate. The pattern of melt bands observed in thesexperiments may have important implications for meltransport and seismic anisotropy in the mantle (Holtzmant al., 2003b). These experiments provide both directalidation of the equations of magma dynamics and aonsiderable computational challenge for solving cou-led non-linear PDEs.

Katz et al. (2006) describes PETSc-based computa-ional models of the experiments that are solutions of thequations of magma dynamics assuming no melting oruoyancy. These simulations require a matrix viscosityhat depends on both porosity and strain-rate (Eq. (9))o reproduce the observed pattern of melt bands at lowngles (∼20◦) to the shear plane. Linearized stabilitynalysis that assumes only porosity dependent viscositye.g. Stevenson, 1989; Spiegelman, 2003) predicts meltands oriented at 45◦to the shear plane. Katz et al. (2006)xtended this analysis to non-Newtonian viscosity androvided full non-linear solutions that demonstrated howhe strain-rate dependence of viscosity controls the anglet which melt bands emerge.

The simulations were developed by using a discretiza-ion of Eqs. (14)–(17) on a 2D staggered mesh similaro that shown in Fig. 3. In this mesh, horizontal andertical velocities reside on cell boundaries, while theorosity and pressures (P and P∗) reside on the cellenters. The mesh is periodic in the x-direction. Theiscrete equations are derived with finite-difference ornite-volume approximations that conform to the mesh

ayout. For example, the discrete form of continuity, Eq.10), becomes:

Uij − Ui−1j

�x+ Vij − Vij−1

�y− Pij

ξij

= P∗rij . (21)

ote that as in Eq. (10), the residual in Eq. (21) cor-esponds to a pressure variable that does not appear inhe equation. P∗ is in fact constrained by the momentumalance Eq. (17) and the compaction rate equation (15).


The time-dependence in the melt band simulationis derived from the advection–compaction equation forporosity, (14), which is hyperbolic, while the other gov-erning equations are elliptic. We employ a semi-implicittime discretization of (14) and, as above for the ridgesimulation, a Fromm scheme for advection. Becauseof the time-dependence, our strategy for solving thediscrete equations derived from (14)–(17) differs fromour approach in the 3D ridge simulation, where webundled all the discrete equations into one system ofalgebraic equations and solved them simultaneously.Here we split the discrete advection–compaction equa-tion from the others and solve it separately, iteratingat each timestep between solution of the hyperbolicequation and the elliptic equations. The advantagesand disadvantages of this approach are discussed inSection 3.9.

While a complete description of the physics of thismodel of shear band formation is beyond the scope ofthe paper, a short summary follows. As described byKatz et al. (2006), the angle of melt bands in a deform-ing two-phase aggregate results from a balance betweentwo modes of rheological weakening. When n = 1 inEq. (9), the viscosity does not depend on strain rate. Inthis case, simulations show that melt bands weakenedby the porosity dependence of viscosity emerge at 45◦tothe plane of shear. These bands grow fastest because theyare perpendicular to the direction of maximum extensionin simple shear. For a strain-rate-dependent viscosity(n > 1), however, concentrated shear deformation fur-ther weakens the bands, allowing them to more easilydecompact under extension. Enhanced shear strain islargest for porosity bands oriented at zero and 90◦andgoes to zero for 45◦bands (see Katz et al. (2006)). Thustwo competing processes affect the preferred angle ofmelt bands. Linear analysis by Katz et al. (2006) sug-gests that the balance between favorable orientation forextension (45◦) and favorable orientation for concen-trating shear (0◦and 90◦) is controlled by the factor(1 − n)/n.

In simulations, the contrast in porosity between meltbands and the compacted regions between them growswith time. Concurrent with this localization of porosity isa localization of shear strain. For n > 1, strain becomeslocalized in bands that are roughly coincident with poros-ity bands in space. The combination of high porosityand enhanced shear over narrow regions produces sharpgradients in viscosity. Fig. 6 shows the evolution of the
maximum viscosity gradient for simulations with differ-ent values of n and different initial porosity conditions.The increase of viscosity gradients within the domainis associated with a breakdown in convergence of the

62 R.F. Katz et al. / Physics of the Earth and Pl

Fig. 6. The maximum gradient in viscosity as a function of strain. Ateach timestep the magnitude of the viscosity gradient is calculated at

each point in the domain as ||∇η|| =√

(∂η/∂x)2 + (∂η/∂z)2. Valueson the y-axis of this plot are determined by taking the maximum valueof ||∇η|| over the domain at each step in strain. Line colors representthe value of n used in each simulation, as given in the plot legend.Solid lines represent simulations in which the initial porosity pertur-bation is white noise on the grid with amplitude 0.1%. Dashed linesrepresent simulations initiated with a porosity field perturbed by noisewith an amplitude spectrum that decreases with |k|−0.9, where k is atwo-dimensional wavevector. For simplicity, in these simulations we
have used a constant bulk viscosity, ζ0 = 10η0. (For interpretation ofthe references to colour in this figure legend, the reader is referred tothe web version of the article.)
Newton solver and is sometimes accompanied by anincrease in the condition number of the Jacobian matrixof the discretized elliptic equations. As shown by theterminal values along lines in Fig. 6, when the maxi-mum dimensionless viscosity gradient reaches O(103),the Newton solver fails to converge for the set of ellip-tic Eqs. (15)–(17), although the linear solve for thecorrection δu typically continues to converge. At thispoint the simulation cannot be integrated further. Noclear relationship exists between the maximum viscos-ity gradient and the condition number of the Jacobianmatrix.

3.9. Elliptic-hyperbolic solver iteration

The set of coupled equations that describemagma/mantle interaction contains one hyperbolic,time-dependent Eq. (14). The other three Eqs. (15)–(17),represent instantaneous balances. One approach to han-
dling the discrete versions of these equations is tocombine them into a single system of nonlinear equa-tions that can (ideally) be solved with a single call toPETSc’s nonlinear solver. A more flexible and efficient

method is to split the equations into smaller systems.In the time-dependent simulations described above,we have separated the hyperbolic equation from theothers.

In practice, two issues arise with the single-solveapproach. First, PETSc treats all variables as uncon-strained. This practice can lead to problems whensolving for porosity, which is physically constrained tolie between zero and one. Even for constitutive equa-tions that, in theory, prevent porosity from becomingnegative, practice shows that a discrete version of theequations over a finite time-step can produce porosi-ties slightly below zero. In this case, floating pointexceptions caused by negative porosity can cause asimulation to fail. A second problem concerns simula-tion efficiency. The simplest magma-dynamics systemhas one evolution equation for porosity, but morecomplex systems may contain several (e.g., tempera-ture, concentrations of various chemical species). Thesevariables are governed by advection–diffusion–reactionequations that, when discretized, result in diagonallydominant Jacobian matrices. Such matrices are associ-ated with linear systems that can be solved with aboutthree iterations of GMRES, as opposed to the Jaco-bian of elliptic equations that requires many more.Combining parabolic and elliptic variables in a sin-gle system of equations leads to a Jacobian matrixin which some rows are diagonally dominant. Fur-thermore, because the computational work of GMRESscales as O(N2) where N is the number of unknowns,dividing the problem into parts can produce significantspeedup.

Splitting the system of equations has costs, how-ever. Of these, the most significant is the need to iteratebetween the solves at each timestep. Typically we iteratetwice, although a more rigorous approach would be tocheck the residual of the combined system of equationsafter each iteration. There is also a cost in code com-plexity and in the number of communications betweenprocessors in parallel computations. The latter has aninsignificant effect on overall code performance for fastcluster interconnection networks.

PETSc provides a set of tools for splitting the fieldsof unknowns with the Fieldsplit preconditioner. Thisallows one to apply different linear solvers to differentsets of fields. For example, consider a three componentproblem where the first two components are coupledelliptic equations. One can specify, on the command line
when invoking the executable, −pc typefieldsplit −pc fieldsplit 0 fields 0 − 1 − pc fieldsplit1 fields 2 − fieldsplit 0 pc typeboomeramg −fieldsplit 1 pc typejacobi . The first two fields are

and Pl

pBwfispsp

4

4

mmmpnomu

ptPecderataeptwua

fidPiem

4

n


re-conditioned as a coupled system using the HypreoomAMG algebraic multigrid pre-conditioner (seeww.llnl.gov/CASC/linear solvers), while the thirdeld is pre-conditioned with a simple Jacobi step. Alightly more powerful tool is the PCCompositereconditioner; this allows one to easily string togethereveral preconditioners, where each is selected to handlearticular parts of the solution space.

. PETSc special features

.1. Parameter handling using PetscBag

One characteristic common to many geodynamicsodels is the large number of physical parameters thatust be included. These parameters are associated withaterial properties such as rheology, buoyancy, and

hase change; boundary conditions; and control of theumerical solution. PETSc provides a convenient setf data structures and methods that facilitate manage-ent of these large sets of program parameters for code

sability.Specifically the PetscBag object manages user

arameters such that the actual data structure is opaqueo the user. The user must declare a variable of typeetscBag and may then use functions to put param-ters of any standard type (integer, real, Boolean,haracter strings, etc.) “into the bag”. Example codeemonstrating this interface is shown in Section 4.3. Forach parameter the user calls PetscBagRegisterXXX ,eplacing XXX with a suffix corresponding to the vari-ble type. The purpose of registering a parameter iso simultaneously (i) set a default value, (ii) providen identifier string to precede and identify the param-ter when it is set upon program execution, and (iii)rovide a descriptive “help” string that is printed withhe variable’s command-line identifier and default valuehen requested. This help string is useful to remind theser of the meaning of a parameter, including its unitsnd type.

The PetscBag object may be written to a binaryle with other PETSc objects for both output and theocumentation of simulation input. When writing theetscBag to a file, the application code need not spec-

fy the size or contents of the bag; as an application codevolves and new parameters are added, the input/outputodules are unchanged.

.2. Self-documenting output

As the code of a simulation evolves to incorporateew physical models, new constitutive equations, or new


solution strategies, the output files generated by that codemay also evolve in their structure and content. With eachchange, new scripts to load, post process, and visualizethe output files would be needed, making it difficult toload old output files from earlier versions of a code.Self-documenting output is a way to avoid this diffi-culty. A common solution to this problem is provided byXML, which has a self-documenting structure. However,it is ill-suited to the large, structured data sets producedby PETSc simulations and is difficult to integrate intoexisting analysis or visualization environments such asMatlab.

PETSc includes an output mode that combines thecreation of a binary data file, which contains the Vec ,Mat , and PetscBag data output by the user, withthe generation of an ASCII descriptor file, which con-tains a script for reading the binary file into Matlab. Thisscript, when executed in Matlab, creates a struct withfields containing the data from the associated binaryfile. The field names of this structure are specified inthe simulation source code. These field names allow theuser who loads the simulation output to understand thenature of the data, even if this user has no knowledgeof the simulation source code. Code that demonstratesthe use of self-documenting PETSc output is shown inSection 4.3.

The pair of files comprising of the binary output fileand its plain text descriptor file is independent of anychanges to the simulation code; hence there is no ambi-guity about the content or structure of the binary fileand no issue of backward compatibility. This integra-tion of binary output with a descriptor file provides aself-contained method for quickly and easily readingsimulation data.

4.3. Example code

The following program, written in C, demon-strates the use of self-documenting output and thePetscBag object, as discussed in the preceding sub-sections. A struct, Parameter , is defined to containthe needed data. Then the bag is allocated and dataregistered, including default values, identifiers, andhelp strings. Once the simulation is completed, aviewer of type PetscViewerBinaryMatlab is usedto generate both the binary data file and descrip-tor file. These files can later be read into Matlab,
using a script provided with the PETSc distribution,PetscReadBinaryMatlab .m . For function documen-tation and more specific usage information, see Balay etal. (2001).
http://www.llnl.gov/CASC/linear_solvers

and Planetary Interiors 163 (2007) 52–68

and Planetary Interiors 163 (2007) 52–68 65

5

5

itgtbKemv

Multigrid also obeys this model, with restriction andupdating operating between grid resolutions. Structur-


. Conclusions

.1. Future of PETSc

The key concepts that organize PETSc’s DAnterface may be generalized to a treatment of unstruc-ured meshes in arbitrary dimension. In this moreeneral setting, however, one must clearly separatehe various concerns. The PETSc Mesh object isased on the Sieve topology interface presented by
nepley and Karpeev (2005). Under Sieve, geom-
try is described as merely another field over theesh. Local discretization information can be pro-

ided by the FIAT system (Kirby, 2004), which is

used to generate quadrature information for arbitraryfinite elements. The local physics computations arestill segregated from the global communication andnumbering.

Moreover, under this model the local physics/globalsolver paradigm can be greatly extended. Finite elementsthemselves fit this model, as local approximations forrestricted fields are then used to update a global field.

ing the algorithms and interface in this way allows globaldata management and its attendant hardships (number-ing, indexing, communication, etc.) to be divorced fromthe specificities of local computation (dimension, ele-

and Pl
ment shape, finite element order, etc.). Future versionsof PETSc will incorporate such developments and hencefacilitate the development of finite element simulationson domains with complex geometry.

5.2. Future of geodynamics simulations

Geochemical and petrological analyses of volcani-cally derived rocks provide a powerful constraint on thedynamics of magmatic systems. A wealth of such anal-yses exist and have been used to develop hypothesesabout the distribution and style of melting beneath plateboundaries (e.g., Kelemen et al., 1997b; Langmuir et al.,2004; Kelley et al., 2006). Tectonic-scale simulationsof magma/mantle interaction can provide quantitativetests of these hypotheses and can generate insight intomelt transport processes. In order to produce predictionstestable by comparison to geochemical and petrologicdata, simulations must generate self-consistent solutionsfor mantle flow, melting, and magmatic transport ofmass, energy, and chemistry. The development of suchmodels represents a major computational challenge.

Past simulations used to generate predictions of lavachemistry have typically simplified either the melt trans-port or the mantle flow field. van Keken et al. (2001)used parameterized melt transport to calculate heliumdegassing of the mantle in a global model of man-tle convection with spherical geometry. Spiegelmanand Kelemen (2003) calculated trace element signa-tures of reactive melt channels using detailed modelsof magma/mantle interaction that neglect large-scaledeformation of the host rock. Spiegelman and Reynolds(1999) made predictions of the spatial distribution of lavachemistry at mid-ocean ridges by solving for both solidmantle deformation and magmatic flow; however, thiswork neglected reactive melting, melt-lithosphere inter-action, and the variable-viscosity of mantle rock. Chobletand Parmentier (2001) simulated 3D, variable viscositymantle flow, melting, and melt transport beneath a mid-ocean ridge but did not calculate geochemical signaturesof the flow. A simulation that self-consistently combinesa current understanding of magma dynamics and large-scale mantle flow to make geochemical predictions isstill lacking.

Some of the challenges of developing such a modelstem from the difficulties associated with simulationssuch as those described in Section 1.1. These simulationsseek to isolate one part of the system, for example, solid
mantle flow or two-phase flow but in a simple geometrywith no melting. In both cases, non-Newtonian viscosityresults in strong nonlinearity, sharp viscosity contrasts,and, ultimately, near-singular systems of linear equations

that are not easily solved. In models of magma dynamicsthat include shear (Katz et al., 2006) and reactive flow(Aharonov et al., 1997; Spiegelman et al., 2001), local-ization of porosity into small-scale features contributesto the difficulty of the problem.

Additional complexities arise when mantle flow sim-ulation is combined with magma dynamics. Localizationleads to a hierarchy of length scales from the tectonicscale of ∼100 km to smaller than the compaction scale(the relevant length scale for magma localization) of∼1 km. Resolving the compaction scale in a domainthat is hundreds of kilometers on each side requiressignificant computational power. Rheological and ther-mal interaction between the liquid and solid phases insuch simulations means that the two flow fields cannotbe decoupled and solved separately. A self-consistentset of fluid dynamical equations such as those givenin Section 1.1, plus a set of equations governing theevolution of chemistry (Aharonov et al., 1997) and tem-perature in the two-phase region, is required. Theseequations should handle both partially melted (φ > 0)and melt-free (φ = 0) zones. Together these require-ments represent a significant challenge to computationalscientists and geodynamicists.

These challenges are being addressed by theComputational Infrastructure for Geodynamics (www.geodynamics.org). CIG is an NSF-sponsored partnershipbetween computational scientists and solid-earth scien-tists to develop the next generation of modeling softwarefor the geodynamics community for investigating awide range of geodynamics problems, including magmamigration, mantle convection, short and long-term litho-spheric deformation, computational seismology, and thegeodynamo. The long-term goal for CIG is to developan interoperable software suite for investigating a rangeof multiphysics problems in geodynamics. An impor-tant design criterion for all new software, however is toleverage as much as possible from existing high-qualitycomputational software projects. In particular PETScand the affiliation with Argonne is a fundamental compo-nent for solver technologies in both the new earthquakephysics codes PyLith and a Lithospheric Deformationcode GALE and will continue to be the principal devel-opment platform for magma-dynamics problems.

We have demonstrated how PETSc, a leading exam-ple of an advanced scientific computation library, canbe used to facilitate simulation of complex nonlinearsystems with localization instabilities and large memory
requirements. Advanced scientific computation librarieswill become increasingly important to the success ofmagma-dynamics simulation as researchers tackle prob-lems that connect fluid-dynamical processes occurring
http://www.geodynamics.org

http://www.geodynamics.org

and Pl

aovd

A

tDGBMDFtaGbKIpRu

R

A

A

B

B

B

B

B

B

C

C


t depth in the mantle with geochemical and petrologicbservations made at the surface. Such models will pro-ide powerful tools for interpreting chemical data and foreveloping an understanding of volcanic source regions.

cknowledgments

We thank Argonne National Laboratory and the Sys-ems Group in the Mathematics and Computer Scienceivision for access to and management of the Blueene/L and Jazz systems. We are also grateful to Satishalay for PETSc support and to Suzanne Carbotte forOR consultations. This work was supported by a U.S.epartment of Energy Computational Science Graduateellowship and NSF International Research Postdoc-

oral Fellowship grant 0602101 for Katz, as well asU.S. Department of Energy Computational Scienceraduate Fellowship for Coon. Spiegelman is supportedy NSF grants 0525973 and 0610138. Contributions ofnepley and Smith are supported by the Mathematical,

nformation, and Computational Sciences Division sub-rogram of the Office of Advanced Scientific Computingesearch, Office of Science, U.S. Department of Energy,nder Contract DE-AC02-06CH11357.

eferences

haronov, E., Spiegelman, M., Kelemen, P., 1997. Three-dimensionalflow and reaction in porous media: implications for the Earth’smantle and sedimentary basins. J. Geophys. Res. 102, 14821–14833.

lbers, M., 2000. A local mesh refinement multigrid method for 3-Dconvection problems with strongly variable viscosity. J. Comput.Phys. 160, 126–150.

alay, S., Buschelman, K., Gropp, W.D., Kaushik, D., Knepley, M.,McInnes, L.C., Smith, B.F., Zhang, H., 2001. PETSc home page.http://www.mcs.anl.gov/petsc.

alay, S., Buschelman, K., Eijkhout, V., Gropp, W.D., Kaushik, D.,Knepley, M.G., McInnes, L.C., Smith, B.F., Zhang, H., 2004.PETSc Users Manual. Technical Report ANL-95/11-Revision2.1.5. Argonne National Laboratory.

arcilon, V., Lovera, O., 1989. Solitary waves in magma dynamics. J.Fluid Mech. 204, 121–133.

arcilon, V., Richter, F.M., 1986. Non-linear waves in compactingmedia. J. Fluid Mech. 164, 429–448.

atchelor, G.K., 1967. An Introduction to Fluid Dynamics. CambridgeUniversity Press, Cambridge.

uck, W., Su, W., 1989. Focused mantle upwelling below mid-oceanridges due to feedback between viscosity and melting. Geophys.Res. Letts. 16 (7), 641–644.

ai, X., Keyes, D., Venkatakrishnan, V., 1997. Newton–Krylov–Schwarz: an implicit solver for CFD. In: Proceedings of the Eighth
International Conference on Domain Decomposition Methods, pp.387–400.
arbotte, S., Small, C., Donnelly, K., 2004. The influence of ridgemigration on the magmatic segmentation of mid-ocean ridges.Nature 429, 743–746.


Choblet, G., Parmentier, E., 2001. Mantle upwelling and meltingbeneath slow spreading centers: effects of realistic rheology andmelt productivity. Earth Planet. Sci. Lett. 184, 589–604.

Crawford, W., Webb, S., 2002. Variations in the distribution of magmain the lower crust and at the Moho beneath the East Pacific Rise at9 degrees–10 degrees N. Earth Planet. Sci. Lett. 203 (1), 117–130.

Demmel, J.W., 1997. Applied Numerical Linear Algebra. SIAM.Georgen, J., Lin, J., 2002. Three-dimensional passive flow and temper-

ature structure beneath oceanic ridge–ridge–ridge triple junctions.Earth Planet. Sci. Lett. 204 (1/2), 115–132.

Gerya, T., Yuen, D., 2003. Rayleigh-taylor instabilities from hydrationand melting propel ‘cold plumes’ at subduction zones. Earth Plan.Sci. Lett. 212 (1/2), 47–62.

Hirth, G., Kohlstedt, D.L., 1995a. Experimental constraints on thedynamics of the partially molten upper mantle: deformation in thediffusion creep regime. J. Geophys. Res. 100 (2), 1981–2002.

Hirth, G., Kohlstedt, D.L., 1995b. Experimental constraints on thedynamics of the partially molten upper mantle 2. Deformation inthe dislocation creep regime. J. Geophys. Res. 100 (8), 15052–15441.

Holtzman, B.K., Groebner, N.J., Zimmerman, M.E., Ginsberg, S.B.,Kohlstedt, D.L., 2003a Stress-driven melt segregation in partiallymolten rocks. Geochem. Geophys. Geosyst. (Art. No. 8607).

Holtzman, B.K., Kohlstedt, D.L., Zimmerman, M., Heidelbach, F.,Hiraga, T., Hustoft, J., 2003b. Melt segregation and strain partition-ing: implications for seismic anisotropy and mantle flow. Science301 (5637), 1227–1230.

Karato, S., Wu, P., 1993. Rheology of the upper mantle: a synthesis.Science 260 (5109), 771–778.

Katz, R., Spiegelman, M., Langmuir, C., 2003. A new parameterizationof hydrous mantle melting. Geochem. Geophys. Geosys. 4 (9), 4.

Katz, R., Spiegelman, M., Carbotte, S., 2004. Ridge migration,asthenospheric flow and the origin of magmatic segmentationin the global mid-ocean ridge system. Geophys. Res. Letts. 31,15605.

Katz, R., Spiegelman, M., Holtzman, B., 2006. The dynamics of meltand shear localization in partially molten aggregates. Nature 442,676–679 (doi:10.1038/nature05039).

Kelemen, P., Hirth, G., Shimizu, N., Spiegelman, M., Dick, H., 1997a.A review of melt migration processes in the adiabatically upwellingmantle beneath oceanic spreading ridges. Phil. Trans. R. Soc. Lond.A 355 (1723), 283–318.

Kelemen, P., Hirth, G., Shimizu, N., Spiegelman, M., Dick, H., 1997b.A review of melt migration processes in the adiabatically upwellingmantle beneath oceanic spreading ridges. Phil. Trans. R. Soc. Lond.A 355 (1723), 283–318.

Kelemen, P., Parmentier, M., Rilling, J., Mehl, L., Hacker, B., 2002.Thermal convection of the mantle wedge. Geochim. Cosmochim.Acta 66 (15A), A389.

Kelley, K., Plank, T., Grove, T., Stolper, E., Newman, S., Hauri, E.,2006. Mantle melting as a function of water content beneath back-arc basins. J. Geophys. Res. 111 (B9).

Kirby, R.C., 2004. Algorithm 839: Fiat a new paradigm for computingfinite element basis functions. ACM Trans. Math. Software 30 (4),502–516.

Knepley, M., Katz, R., Smith, B., 2006. Developing a geodynamicssimulator with PETSc. In: Bruaset, A., Tveito, A. (Eds.), Numerical
Solution of Partial Differential Equations on Parallel Computers,vol. 51 of Lecture Notes in Computational Science and Engineer-ing. Springer-Verlag, pp. 413–438.
Knepley, M.G., Karpeev, D.A., 2005. Flexible representation ofcomputational meshes. Preprint ANL/MCS-P1295-1005, Argonne

http://www.mcs.anl.gov/petsc

http://dx.doi.org/10.1038/nature05039

and Pl

matic solitary waves in 3-D. Geophys. Res. Lett. 22 (10), 1289–


National Laboratory, October 2005. URL: ftp://info.mcs.anl.gov/pub/tech reports/reports/P1295.pdf.

Langmuir, C., Michael, P., Lehnert, K., Goldstein, S., Soffer, G., Gier,E., Snow, J., 2004. Constraints from the Gakkel ridge on the originof MORB. Geochim. Cosmochim. Acta 68 (11), A690.

Rabinowicz, J.S.M., Rouzo, S., Rosemberg, C., 1993. Three-dimensional mantle flow beneath mid-ocean ridges. J. Geophys.Res. 98, 7851–7869.

Magde, L., Sparks, D.W., 1997. Three-dimensional mantle upwelling,melt generation and melt migration beneath segmented slow-spreading ridges. J. Geophys. Res. 102, 20571–20583.

McKenzie, D., 1969. Speculations on consequences and causes of platemotions. Geophys. J. 18, 1–32.

McKenzie, D., 1984. The generation and compaction of partiallymolten rock. J. Petrol. 25, 713–765.

Mehl, S., 2006. Use of Picard and Newton iteration for solving nonlin-ear ground water flow equations. Ground Water 44 (4), 583–594.

Mei, S., Bai, W., Hiraga, T., Kohlstedt, D., 2002. Influence of melton the creep behavior of olivine-basalt aggregates under hydrousconditions. Earth Planet. Sci. Lett. 201, 491–507.

Message Passing Interface Forum, 1994. MPI: a Message-PassingInterface standard. Int. J. Supercomput. Appl. 8 (3/4), 165–414.

Message Passing Interface Forum, 1998. MPI2: a Message PassingInterface standard. Int. J. High Performance Comput. Appl. 12(1/2), 1–299.

Paniconi, C., Putti, M., 1994. A comparison of Picard and Newton itera-tion in the numerical solution of multidimenional variably saturatedflow problems. Water Resources Res. 30 (12), 3357–3374.

Patankar, S., 1980. Numerical Heat Transfer and Fluid Flow. Hemi-sphere Publishing Corp.

Phipps Morgan, J., Forsyth, D., 1988. 3-D flow and temperature per-turbations due to transform offset: effects on oceanic crustal andupper mantle structure. J. Geophys. Res. 93, 2955–2966.

Richter, F.M., McKenzie, D., 1984. Dynamical models for melt segre-gation from a deformable matrix. J. Geol. 92, 729–740.

Scott, D.R., Stevenson, D., 1984. Magma solitons. Geophys. Res. Lett.11, 1161–1164.

Scott, D.R., Stevenson, D., 1986. Magma ascent by porous flow. J.Geophys. Res. 91, 9283–9296.

Scott, D.R., Stevenson, D., Whitehead, J., 1986. Observations of soli-tary waves in a deformable pipe. Nature 319, 759–761.

Shen, Y., Forsyth, D., 1992. The effects of temperature- and pressure-dependent viscosity on three-dimensional passive flow of the
mantle beneath a ridgetransform system. Geophys. J. Int. 97,19717–19728.
Sparks, D., Parmentier, E., 1991. Melt extraction from the man-tle beneath spreading centers. Earth Planet. Sci. Lett. 105 (4),368–377.


Sparks, D., Parmentier, E., 1993. The structure of three-dimensionalconvection beneath oceanic spreading centers. Geophys. J. Int. 112,81–91.

Spiegelman, M., 2003. Linear analysis of melt band formation bysimple shear. Geochem. Geophys. Geosyst. 4 (9), Article 8615(doi:10.1029/2002GC000499).

Spiegelman, M., 1993a. Flow in deformable porous media. Part 1.Simple analysis. J. Fluid Mech. 247, 17–38.

Spiegelman, M., 1993b. Flow in deformable porous media. Part 2.Numerical analysis—the relationship between shock waves andsolitary waves. J. Fluid Mech. 247, 39–63.

Spiegelman, M., Kelemen, P.B., 2003. Extreme chemical variability asa consequence of channelized melt transport. Geochem. Geophys.Geosyst. 4 (7), 1055.

Spiegelman, M., McKenzie, D., 1987. Simple 2-D models for meltextraction at mid-ocean ridges and island arcs. Earth Planet. Sci.Lett. 83, 137–152.

Spiegelman, M., Reynolds, J., 1999. Combined theoretical and obser-vational evidence for convergent melt flow beneath the EPR. Nature402, 282–285.

Spiegelman, M., Kelemen, P.B., Aharonov, E., 2001. Causes and con-sequences of flow organization during melt transport: The reactioninfiltration instability in compactible media. J. Geophys. Res. 106(B2), 2061–2077, www.ldeo.columbia.edu/ mspieg/SolFlow/.

Steihaug, T., 1983. The conjugate gradient method and trust regionsin large scale optimization. SIAM J. Numer. Anal. 20, 626–637.

Stevenson, D.J., 1989. Spontaneous small-scale melt segregation inpartial melts undergoing deformation. Geophys. Res. Lett. 16 (9),1067–1070.

Trompert, R., Hansen, U., 1996. The application of a finite vol-ume multigrid method to 3D flow problems in a highly viscousfluid with variable viscosity. Geophys. Astrophys. Fluid Dyn. 83,261.

van Keken, P., Ballentine, C., Porcelli, D., 2001. A dynamical inves-tigation of the heat and helium imbalance. Earth Planet. Sci. Lett.188, 421–434.

van Keken, P., Kiefer, B., Peacock, S., 2002. High-resolution models ofsubduction zones: implications for mineral dehydration reactionsand the transport of water into the deep mantle. Geochem. Geophys.Geosys., 3.

Wiggins, C., Spiegelman, M., 1995. Magma migration and mag-

1292.Zimmerman, M.E., Zhang, S.Q., Kohlstedt, D.L., Karato, S., 1999.

Melt distribution in mantle rocks deformed in shear. Geophys. Res.Lett. 26 (10), 1505–1508.

ftp://info.mcs.anl.gov/pub/tech_reports/reports/P1295.pdf

ftp://info.mcs.anl.gov/pub/tech_reports/reports/P1295.pdf

http://dx.doi.org/10.1029/2002GC000499

http://www.ldeo.columbia.edu/_mspieg/SolFlow/

Documents

Numerical simulation of geodynamic processes with the ...foalab.earth.ox.ac.uk/publications/Katz_etal_PEPI07.pdf · Numerical simulation of geodynamic processes with the Portable