SOFTWARE  PRODUCTIVITY  CHALLENGES  IN  ENVIRONMENTAL  APPLICATIONS    

SIAM  CSE  15,  Salt  Lake  City,  UT.  

David  Moulton,  Ethan  Coon  Los  Alamos  Na:onal  Laboratory  Carl  Steefel,  Sergi  Molins  Lawrence  Berkeley  Na:onal  Laboratory  Sco8  Painter  Oak  Ridge  Na:onal  Laboratory    March  15,  2015.  

LA-­‐UR-­‐15-­‐21857  

Collaborators  (ASCEM,  LDRD,  IDEAS,  …)  

2  

Ben  Andre  (NCAR)  Erin  Barker    (PNNL)  Markus  Berndt    (LANL)  David  Bernholdt  (ORNL)  Ethan  Coon  (LANL)  Marc  Day  (LBNL)  Vicky  Freedman  (PNNL)  Carl  Gable  (LANL)  Rao  Garimella  (LANL)  Dylan  Harp  (LANL)  Glenn  Hammond  (SNL)  Mike  Heroux  (SNL)    

Hans  Johansen    (LBNL)  Jeff  Johnson  (LBNL)  Konstan:n  Lipnikov  (LANL)  Lois  Curfman  McInnis  (ANL)  Terry  Miller  (LANL)  Sergi  Molins  (LBNL)  Sco[  Painter  (ORNL)  Tim  Scheibe  (PNNL)  Carl  Steefel  (LBNL)  Daniil  Svyatskiy  (LANL)  Haruko  Wainwright  (LBNL)  Cathy  Wilson  (LANL)  …    

Outline  

¨  Background/MoKvaKon  ¤  Science  Ques:ons  ¤  Science  and  So_ware  Challenges  

¨  Environmental  ApplicaKons  ¤ Use  a  wide  range  model  complexity    ¤  Process-­‐rich  models  go  beyond  tradi:onal  mul:physics  

¨  ProducKvity  Challenges  ¤  Community-­‐based  interdisciplinary  opportuni:es  (IDEAS)  ¤ Driving  collabora:on  and  development  through  Use  Cases  

¨  Conclusions  

Environmental  ApplicaKons  

4  

¨  Climate  impacts  and  feedbacks  (carbon  and  nitrogen  cycling)  

¨  Contaminant  transport  and  reac:ons.  ¨  Complex  interac:on  between  

subsurface  and  land  surface  processes.  

 

Science  and  SoQware  Challenges  

¨  Terrestrial  systems  span  science  challenges:  ¤  Inherently  multsicale  with  coupling  that  must  be  

explored  through  simula:on  ¤  Development  and  leveraging  of  mechanis:c  

models  at  all  scales  is  required  ¤  Mul:scale  data  must  integrate  seamlessly  with  

itera:ve  development  of  predic:ve  models    

¨  Established  codes  have  significant  capabili:es  but  are  not  ready/suitable  for:  ¤  refactoring  for  emerging  architectures    ¤  interdisciplinary  development  teams  ¤  are  not  sufficiently  flexible  or  interoperable  

So_ware  produc:vity  is  a  cri:cal  factor  in  realizing  exascale  simula:ons  of  terrestrial  systems.  

5  

Amanzi:  High-­‐Level  ObjecKves  

6  

Wide  Range  of  Complexity  

Wide  Range  of  Plaiorms  

¨  Flexible  and  Extensible:  Modular  simula:on  capability  for  waste  form  degrada:on,  variably  saturated  flow,  and  reac:ve  transport.  

¨  Performance  Portable:  ¨  Efficient,  robust  

simula:ons  from  laptops  to  supercomputers.  

¨  Design  and  build  for  refactoring  required  by  emerging  mul:-­‐core  and  accelerator-­‐based  systems.  

¨  Community  Par@cipa@on:  Open-­‐source  project  with  strong  mee:ng  the  needs  of  a  strong  interdisciplinary  community.  

Leverage  advances  from  across  DOE  (e.g.,  ASCR,  ASC)  and  Academia.  

Amanzi:  Simulator  CapabiliKes  

¨  Process  Kernels  ¤  Transient  unsaturated  flow  with  Richards  

equa:on,  including  op:ons  to  steady-­‐state  ini:aliza:on.  

¤  Transient  single-­‐phase  flow  with  specific  storage/yield  

¤  Volume  based  sinks/sources  ¤  Reac:ve-­‐transport,  with  operator  splikng  

for  reac:ons.  ¤  Support  for  a  wide  range  of  chemical  

reac:ons.  

¨  Framework  and  Infrastructure:  ¤  Unstructured  meshes  with  polyhedral  cells,    

block-­‐structured  AMR,  and  internal  genera:on  of  hexahedral  meshes  in  rectangular  domains.  

¤  Designed  to  integrate  with  Akuna/Agni  model  setup  and  toolsets.  

¤  Flexible  and  extensible  MPC/PK  APIs.  ¤  Parallel  I/O:  visualiza:on  &  restart  

Amanzi/ATS Design

MPC

(Base)

PK:: Flow

RichardsPK:: Transport

Advective

PK:: Reactions

Geochemistry

HPC Toolsets

Data management Mesh Infrastructure Discretizations Solvers

HPC Core Framework (services) and Third Party Libraries

8

ASCE

M-HPC

-F-Are

a-Sava

nnah R

iver S

ite

ASCEM-HPC. F-Area. Geochemical System. ASCEM-F-Area. Meshes for Amanzi.

!  4 meshes at different horizontal resolution: !  5 m. !  2.5 m. !  1.25 m.

!  0.625 m.

!  ExodusII format. !  Available in the

HPC wiki.

Alquimia:  A  geochemistry  interface  library  

¨  Alquimia  currently  assumes  reac:ve  transport  uses  operator-­‐splikng.    

¨  Fully-­‐implicit  reac:ve  transport  support  is  planned.    

¨  Assists  in  enforcing  geochemical  condi:ons  (specia:on)  for  transport  boundary  condi:ons.  

¨  Alquimia  can  facilitate  benchmarking  of  geochemical  capabili:es  in  exis:ng  codes.    

¨  Geochemistry  libraries,  such  as  PFLOTRAN  and  CrunchFlow,  are  implemen:ng  interfaces  to  Alquimia.    

Several  geochemistry  libraries  are  established  in  the  community  making  geochemistry  ideal  to  explore  componen:za:on  and  interface  design.    Alquimia  is  an  interface,  and  does  not  perform  any  reacKon  calculaKons.    

Alquimia  is  open  source,  h[ps://bitbucket.org/berkeleylab/alquimia    

SRS  F-­‐  Area:  Phase  II  Background  and  Modeling  

Ø  F-­‐Area  at  the  Savannah  River  Site  (SRS):    §  Variably  saturated  flow,  with  uncertain  vadose  zone  source  term  §  Unstructured  grids  capturing  hydrostra:graphy/topography.  §  Mineral  precipita:on  and  dissolu:on  plus  sorp:on  in  groundwater.  §  UQ  studies  in  2D,  and  representa:ve  3D  simula:on  of  F-­‐Area.  

ASC

EM-H

PC-F

-Are

a-Sa

vann

ah R

iver

Site

ASCEM-HPC. F-Area. Geochemical System. ASCEM-F-Area. Domain.

ASCE

M-HPC

-F-Are

a-Sava

nnah R

iver S

ite

ASCEM-HPC. F-Area. Geochemical System. ASCEM-F-Area. Meshes for Amanzi.

!  4 meshes at different horizontal resolution: !  5 m. !  2.5 m. !  1.25 m.

!  0.625 m.

!  ExodusII format. !  Available in the

HPC wiki.

SRS  F-­‐Area:  SimulaKon  of  Uranium  ReacKve  Transport  

Simula'on  shows  the  retarda'on  of  the  Uranium  plume  due  to  sorp'on,  rela've  to  pH  (non-­‐reac've  tracer),  highligh'ng  the  importance  of  biogeochemistry.  

Thaw-­‐induced  topographic  reorganizaKon  

Progressive  Degrada:on     Jorgenson  

Evolu'on  of  polygonal  paAerned  ground  is  expected  to  control  hydrology  and  thus  the  carbon  cycle  

Hand-­‐coded  model  couplings  become  unmanageable  with  more  than  a  few  ecohydrochemical  processes    

Managing  complexity  in  process-­‐rich  simulaKons  

The  Arc:c  Terrestrial  Simulator  (ATS)  leverages  Amanzi’s  infrastructure,  adds  an  innova:ve  process-­‐kernel  management  system,  new  process  kernels,  and  new  meshing  workflow.  

A  New  MulK-­‐Physics  Framework  

¨  Use  object  oriented  concepts  for  flexibility  and  extensibility,  ¤  Dynamic  data  management  allows  Process  Kernels  (PKs)  to  register  their  

data  needs  at  run'me  ¤  This  registra'on  provides  the  necessary  informa'on  to  represent  the  

rela'onships  between  variables  (i.e.,  dependent  vs.  independent)  and  to  enable    a  dynamically  configured  model  and  simula:on.  

¨  Enables  integrated  hierarchical  approach  to  tes:ng  and  QA  ¤  build  confidence  in  models  and  implementa'ons  ¤  study  structural  model  uncertainty,  and  understand  process  coupling  ¤  support  hypothesis  tes'ng  (“What  if?”  scenarios)  

¨  Significantly  enhances  interdisciplinary  collabora:ons  by  allowing  scien'sts  to  engage  effec'vely  in  their  area  of  exper'se,  and  immediately  seeing  the  impact  of  their  contribu:ons.  

Benefits  of  the  new  MulK-­‐Physics  Framework  Developing a Multi-Physics Model

MPC

SubsurfaceWater

PK::Flow

Permafrost

PK::Energy

?????

Flow:@

@t

[�(⇢l

s + ⇢g

(1 � s)] = �r · (⇢V)

V = �k(s)K(x)

µrp

Energy: ??

Constitutive Relations: ??

4

•  Flow:  

 •  Energy:  ??    •  Cons@tu@ve  Rela@ons:    ??  

Developing a Multi-Physics Model

MPC

SubsurfaceWater

PK::Flow

Permafrost

PK::Energy

?????

Flow:@

@t

[�(⇢l

s + ⇢g

(1 � s)] = �r · (⇢V)

V = �k(s)K(x)

µrp

Energy: ??

Constitutive Relations: ??

4

Automated  Dynamic  ConstrucKon  of  the  Dependency  Graph  (DAGs)  

Consider  three  alterna:ve  models  for  the  thermal  energy  

Dynamic Construction of Dependency Graph

Consider three alternative models for the thermal energy

p

s

µ ⇢

k

r

V

res

flow

energy not useds(p), µ, ⇢(p)

p

T

s

µ ⇢

k

r

V

res

flow

prescribed temperatures(p), µ(T), ⇢(p, T)

p

T

s

µ ⇢u

h

k

r

V

res

flow

res

energy

conservation of energys(p), µ(T), ⇢(p, T)u(p, T), h(p, T)

5

Dynamic Construction of Dependency Graph

Consider three alternative models for the thermal energy

p

s

µ ⇢

k

r

V

res

flow

energy not useds(p), µ, ⇢(p)

p

T

s

µ ⇢

k

r

V

res

flow

prescribed temperatures(p), µ(T), ⇢(p, T)

p

T

s

µ ⇢u

h

k

r

V

res

flow

res

energy

conservation of energys(p), µ(T), ⇢(p, T)u(p, T), h(p, T)

5

Dynamic Construction of Dependency Graph

Consider three alternative models for the thermal energy

p

s

µ ⇢

k

r

V

res

flow

energy not useds(p), µ, ⇢(p)

p

T

s

µ ⇢

k

r

V

res

flow

prescribed temperatures(p), µ(T), ⇢(p, T)

p

T

s

µ ⇢u

h

k

r

V

res

flow

res

energy

conservation of energys(p), µ(T), ⇢(p, T)u(p, T), h(p, T)

5

Energy  not  used    s(p),  μ,  ρ(p)  

Prescribed  temperature    s(p),  μ(T),  ρ(p,T)  

Conserva:on  of  Energy      s(p),  μ(T),  ρ(p,T),  u(p,T),  h(p,T)  

Notz,  Pawlowski,  Sutherland,  ACM  TOMS  (2012);  Coon,  Moulton,  Painter  submi[ed  to  Environmental  Modelling  &  So_ware  (2014).  

ATS  Process  Modeling  CapabiliKes  

16  

¨  Surface  flow:  ¤  Diffusive  wave  approxima:on  ¤  Non-­‐isothermal  

¨  Subsurface  flow:  ¤   3-­‐phase,  Richards-­‐like  equa:on(nonlinear  parabolic)  

¤   Non-­‐isothermal    

Coupling  Surface/Subsurface  Flow  

17  

Visualiza'on  enhances  understanding  and  analysis  of  strongly  coupled  surface  and  subsurface  flow  models.  Here  flow  to  and  from  the  lower  troughs  depends  on  external  forcing  and  current  satura'on  of  the  subsurface.  

ProducKvity  Crisis  (and  Opportunity)  

¨  Further  strengthen  the  role  modeling  and  simula:on  play  in  science.  

¨  Enable  researchers  to  study  more  complex  mechanis:c  models  of  single  and  coupled  processes.    

¨  Add  more  processes  to  exis:ng  systems  models.  

¨  Enable  simula:ons  that  bridge  the  vast  range  of  temporal  and  spa:al  scales  found  in  natural  systems.  

Con:nuous  growth  in  computa:onal  power  should  …  

Companion  BER  workshop  report:  h8p://www.doesbr.org/VirtualEcosystems/  

a

DRAFT

Interoperable  Design  of  Extreme-­‐scale  ApplicaKon  SoQware  (IDEAS)  

19  

Motivation Enable  increased  scien@fic  produc@vity,  realizing  the  poten:al  of  extreme-­‐  scale  compu:ng,  through  a  new  interdisciplinary  and  agile  approach  to  the  scien@fic  soNware  ecosystem.  

Objectives Address  confluence  of  trends  in  hardware  and  

increasing  demands  for  predic:ve  mul:scale,  mul:physics  simula:ons.  

Respond  to  trend  of  con:nuous  refactoring  with  efficient  agile  so_ware  engineering  methodologies  and  improved  so_ware  design.  

Approach  ASCR/BER  partnership  ensures  delivery  of  both  crosscukng  methodologies  and  metrics  with  impact  on  real  applica:on  and  programs.  

Interdisciplinary  mulK-­‐lab  team  (ANL,  LANL,  LBNL,  LLNL,  ORNL,  PNNL,  SNL)  ASCR  Co-­‐Leads:  Mike  Heroux  (SNL)  and  Lois  Curfman  McInnes  (ANL)  BER  Lead:    David  Moulton  (LANL)  Topic  Leads:  David  Bernholdt  (ORNL)  and  Hans  Johansen  (LBNL)  

Integra@on  and  synergis@c  advances  in  three  communi@es  deliver  scien:fic  produc:vity;  outreach  establishes  a  new  holis:c  perspec:ve  for  the  broader  scien:fic  community.  

Impact on Applications & Programs Terrestrial  ecosystem  use  cases  @e  IDEAS  to  modeling  and  simula@on  goals  in  two  Science  Focus  Area  (SFA)  programs  and  both  Next  Genera:on  Ecosystem  Experiment  (NGEE)  programs    in  DOE  Biologic  and  Environmental  Research  (BER).  

SoQware  ProducKvity  for  Extreme-­‐Scale  

Science  Methodologies  for  SoQware  ProducKvity  

Use  Cases:  Terrestrial  Modeling  

Extreme-­‐Scale  ScienKfic  SoQware  Development  Kit  

(xSDK)  

Use  Cases:  SelecKon  Criteria  &  Goals  

20  

¨  Use  exisKng  modeling  capabiliKes  and  align  with  development  plans  in  currently  funded  projects:  ¤  Science  Focus  Area  Projects  (SFAs)  at  LBNL  and  PNNL,  ¤  Next  Genera:on  Ecosystem  Experiment  (NGEE)  projects  in  Arc:c  and  

Tropics  (mul:-­‐ins:tu:on  led  by  ORNL  and  LBNL  respec:vely).  ¨  Leverage  soQware  producKvity  tools  and  methodologies  to  enable  

efficient  refactoring,  or  new  development,  of  components  to  address  important  scien:fic  ques:ons,  

¨  Demonstrate  higher  fidelity  soluKons  to  scien:fically  interes:ng  problems  by  taking  advantage  of  advanced  architectures,    

¨  Use/couple  mechanis:c  models  across  a  range  of  scales,  ¨  Use  exisKng  observaKons  to  demonstrate  higher  produc:vity  workflows  

for  model/data  integra:on,    ¨  Support  a  longer-­‐term  vision  of  a  "virtual  ecosystem"  with  mechanis:c  

representa:ons  of  vegeta:on  processes  embedded  in  large-­‐scale  watershed  models.  

Use  Case  1:  Hydrological  and  Biogeochemical  Cycling  in  the  Colorado  River  System.  

Photos  courtesy  of  R.  Kaltschmidt  h[p://photos.lbl.gov/albums.php?albumId=428202  (LBNL)  

Hyporheic  zone  biogeochemistry  involves  a  collec:on  of  coupled  mul:scale  processes  that  play  a  cri:cal  role  in  carbon  cycling  and  watershed  performance.  

Use  Case  1:  Highlights  

22  

Lower  East  River  Catchment  ¨  High  resolu:on  models  of  individual  stream  

meander  flow  and  biogeochemistry  using  Amanzi/ATS  and  ParFlow  

¨  Upscale  flow  and  biogeochemistry  within  lower  East  River  and  in  hyporheic  zone  while  maintaining  high  spa:al  resolu:on  locally  (<  0.1  m).  

Science  Outcomes:  Use  Case  1  

Science  Objec@ve:  A  be[er  understanding  of  aquifer  redox  status  and  climate  impacts  on  watershed  carbon  and  nitrogen  cycling  through  higher  fidelity,  mul:-­‐scale  models  simulated  at  high  spa:al  resolu:on.  23 ¨  Year  1:  Perform  a  high-­‐resolu:on  subsurface/

surface  flow  simula:on  for  a  single  meander  of  the  East  River  with  new  EcoTrait  biogeochemical  reac:on  network,  with  horizontal  spa:al  resolu:on  of  ~0.1m.  

¨  Year  2:  Extend  simula:ons  to  a  small  meandering  sec:on  of  the  East  River  with  lateral  inputs,  and  upscale  this  flow  and  the  EcoTrait  subsurface  reac:ve  transport  model  over  this  domain.    

 ¨  Year  3:  Incorporate  the  effects  of  vegeta:on  on  hydrology  and  biogeochemical  fluxes  within  integrated  surface  and  subsurface  water  flow  system  within  the  larger  East  River  watershed.        

Use  Case  2:  Thermal  hydrology  and  carbon  cycling  in  tundra  at  the  Barrow  Environmental  Observatory.  

5-­‐polygon  cluster  

Lobster  catchment  

Models  of  warming  tundra  are  very  process-­‐rich  and  mul:scale  (microtopography  to  large  basins),  including  thermal-­‐hydrology,    biogeochemistry,  and  deforma:on.  

Use  Case  2:  Highlights  

¨  Mul:scale  approach  combining  1-­‐D  thermal  hydrology  models  with  2-­‐D  overland  flow  using  Amanzi-­‐ATS    

0 5 10 154.04.55.05.56.0

¨  Large-­‐scale  simula:ons  using  Amanzi/ATS  for  thermal  hydrology,  PFLOTRAN  for  biogeochemistry,  and  CLM  for  land  surface  processes  

Soil  organic  ma[er  decomposi:on  rates  in  a  undegraded  polygon    

Science  Outcomes:  Use  Case  2  

Science  Objec@ve:    Determine  how  thawing  permafrost  caused  by  warming  Arc:c  temperatures  alters  the  hydrologic  and  carbon  cycles  of  Arc:c  lowland  tundra.  26 ¨  Year  1:  Demonstrate  that  a  mul:scale  model  comprising  

independent  1D  ver:cal  TH  models  coupled  to  a  2D  rou:ng  of  surface  water  is  feasible  in  the  new  Amanzi/ATS  framework.  

¨  Year  2:  Verify  that  this  intermediate-­‐scale  representa:on,  with  ~10m  resolu:on,  is  an  adequate  upscaling  of  the  fine-­‐scale  representa:on,  with  ~0.1m  resolu:on,  including  subsurface  biogeochemistry  and  land  surface  processes  (through  PFLOTRAN,  and  CLM  respec:vely).  

¨  Year  3:  Perform  simula:ons  on  domains  of  1-­‐10km  using  the  intermediate-­‐scale  model,  and  use  those  simula:ons  to  evaluate  exis:ng  parameteriza:ons  in  global  land  surface  models.  

 

Conclusions  

27  

¨  Environmental  applica:ons  offer  significant  challenges  and  exci@ng  opportuni@es  for  interdisciplinary  collabora@on  

¨  Significant  progess  has  been  made  in  recent  years:  ¤  Modular  open-­‐source  applica:on  codes  provide  a  bridge  between  

disciplines  ¤  Flexible  and  extensible  frameworks  can  manage  the  complexity  of  

process-­‐rich  models  enabling  run:me  experimenta:on  ¨  Significant  challenges  remain:  

¤  Significant  exper:se  resides  in  monolithic  established  codes  ¤  Current  tes:ng  prac:ces  do  not  support  refactoring  required  for  new  

architectures  with  more  abstract  programming  models  ¤  Inherently  mul:scale:  understanding  and  predic:ons  gained  at  finer  

scales  are  needed  to  impact  watershed,  and  even  Earth  System  scales.  ¨  A  new  interdisciplinary  and  agile  approach  to  the  scien@fic  soNware  

ecosystem  is  cri@cal  and  will  be  explored  in  the  IDEAS  project,  driven  by  use  cases  from  terrestrial  ecosystems.