SI2-SSI: Community Software for Extreme-Scale Computing in Earthquake System Science (SEISM2)PI: Thomas H. Jordan (USC)

co-PI’s: Yifeng Cui (SDSC), Kim Bak Olsen (SDSU), Ricardo Taborda (University of Memphis)

S C E Can NSF+USGS center

SCEC researchers are conducting the SI2-SSI: Community Software for Extreme-Scale Computing in Earthquake System Science (SEISM2) Project. This project will push validated simulation capabilities to higher seismic frequencies and into the domain of extreme-scale computing. Project researchers are ad-dressing scientific problems that limit the accuracy and scale in current numerical representations of earth-quake processes by targeting three main computational objectives:(O1). Incorporate into community SSEs the physics necessary to extend deterministic simula-tions to the high seismic frequencies of engineering interest (> 1 Hz).(O2). Improve the accuracy of SEISM simulations, and work with a diverse community of re-searchers and stakeholders to validate simulation products for practical use.(O3). Enhance the performance and robustness of the SEISM computational platforms, and pre-pare them to operate on next-generation supercomputers.

High Performance I/O Library Software: (M5)More at: http://hpgeoc.sdsc.edu/software.html

Earthquake simulations at the spatiotemporal scales required for probabilistic seismic hazard analysis present some of the toughest computational challenges in geoscience, requiring extreme-scale comput-ing. The Southern California Earthquake Center is creating a Software Environment for Integrated Seismic Modeling (SEISM) that provides the extreme-scale simulation capability needed to transform probabilistic seismic hazard analysis into a physics-based science. This research will address fundamental scientific problems that limit the accuracy and scale range in current numerical representations of earthquake pro-cesses, which will benefit earthquake system science worldwide. This project will educate a diverse STEM workforce from the undergraduate to early-career level, and it will cross-train scientists and engineers in a challenging high-performance environment. The SEISM2 workplan defines 14 milestones (M1-M14) over the course of the 3-year project that will show progress towards the Project’s three objectives (O1-O3).

Year 1: M1: AWP-ODC released with near fault plasticity (O1) M2: Hercules released with frequency-dependent attenuation (O1) M3: Simulations validated against historic events using using SWUS procedures (O2) M4: CyberShake 1Hz Los Angeles hazard model (O1)Year 2: M5: Integrate SEISM-IO Library into AWP for checkpointing (O3) M6: Hercules released with nonlinear plastic yielding (O1) M7: Prototype of parallel discontinuous mesh AWP-DM (O3) M8: Simulations validated against historic events using GMSV SDoF Procedures (O2) M9: CyberShake 1.5Hz LA hazard model (O1)Year 3: M10: Parallelization and optimization of discontinuous mesh AWP-DM (O3) M11: CyberShake SGT data used by Broadband platform (O1) M12: High-complexity ShakeOut simulation (O3) M13: AWP-ODC ported onto MIC (O3) M14: CyberShake EKS hazard model (O1)

The SEISM2 Project is supported by the National Science Foundation award No. ACI-1450451. Additional SCEC team members include: S. Azizzadeh-Roodpish (University of Memphis); J.W. Baker, G.C. Beroza (Stanford); J. Bielak (CMU); S. Callaghan (SCEC); S.M. Day (SDSU); D. Gill (SCEC); C. Goulet (SCEC); R.W. Graves (USGS); N. Khoshnevis (University of Memphis); N. Luco (USGS); P.J. Maechling (SCEC); K. Milner (SCEC); S. Nie (SDSU); E. Poyraz (SDSC); S. Rezaeian (USGS); D. Roten (SDSC/SDSU); W. Savran (SDSU); F. Silva (SCEC); K. Withers (SDSU); H. Xu (SDSC); This work used the Ex-treme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation grant number ACI-1053575.This research is part of the Blue Waters sustained-petascale computing project, which is supported by the National Science Foundation (awards OCI-0725070 and ACI-1238993) and the state of Illinois. Blue Waters is a joint effort of the University of Illinois at Urbana-Champaign and its National Center for Supercomputing Applications. This work is also part of the "Extending the Spatiotemporal Scales of Physics-based Seismic Hazard Analysis" PRAC allocation support by the National Science Foundation (award number OCI-1440085). An award of computer time was provided by the Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program. This research used resources of the Argonne Leadership Computing Facility, which is a DOE Office of Science User Facility supported under contract DE-AC02-06CH11357. This research also used resources of the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725.Private and international partners include: Pacific Gas & Electric Co. and NVIDIA, Corp. Scientific publications and additional information about the various software elements and platforms can be found at http://scec.usc.edu/scecpedia and at www.scec.org.

SCEC’s Anelastic Wave Propagation high performance earthquake simulation software (AWP-ODC) simu-lates the dynamic rupture and wave propagation that occurs during an earthquake. AWP-ODC is a well-es-tablished high performance scientific software that can perform ground motion simulations using forward and reciprocal techniques, with both CPU and GPU implementations. Recent physics-based improvements to AWP-ODC include frequency-dependent attenuation via a power law above a reference frequency, and off-fault viscoplastic yielding (Roten et al., 2014).

AWP-ODC Finite Difference Earthquake Simulation Software: (M1)More at: http://hpgeoc.sdsc.edu/AWPODC/

Figure 2: Results of AWP-ODC ground motion simulations for a M7.7 southern San Andreas fault rupture showing the reduction of peak ground velocities from the effects of plastic yielding with different types of rocks, (left) in map view and (right) in (a) and (b) as probability distributions of peak ground velocities pro-duced by simulations using alternative input models of southern California geological properties.

The CyberShake Seismic Hazard Model Platform: (M4) (M11)More at: http://scec.usc.edu/scecpedia/CyberShake

Simulations of dynamic rupture and wave propagation are now performed on petascale supercomputers to resolve high-frequency wavefields during large earthquakes (e.g. Cui et al., 2013). With grid dimensions of several hun-dred-billion mesh points, the performance of input and output (I/O) operations represents a major concern. The SEISM-IO (Software Environment for Integrated Seismic Modeling) library has been previously developed to reduce the duplicate work on handling I/O operations in many HPC applications sharing similar I/O operations used within SCEC. Generalized IO interfaces as part of the SEISM-IO library were developed, which provide highly condensed, easy-to-understand APIs for users to choose. To accommodate the generalized interface, the earlier SEISM-IO li-brary was modified to integrate different initialization/open/write processes in MPI-IO, HDF5, PnetCDF and ADIOS.

Hercules Finite Element Earthquake Simulation Software: (M2) (M8)More at: https://github.com/CMU-Quake/hercules

Figure 5. Left: Example use of Hercules in engineering-oriented ground motion validation study that involved sim-ulations of recent California earthquakes. Center: Adjustment of Hercules’ new viscoelastic model for frequen-cy-dependent Q attenuation. Right: Screenshot of Hercules repository on Github.

Hercules is an octree-based parallel finite element (FE) earthquake simulator developed by the Quake Group at Carnegie Mellon, ASEIS at the University of Memphis, and SCEC. Hercules is written in C using MPI libraries. It integrates an efficient unstructured hexahedral mesh generator and an explicit FE formulation to solve the visco-elastodynamic equation for 3D wave propagation problems in highly heterogeneous media due to earthquake ki-nematic faulting. Alternative distributions of Hercules solve problems in elasto-plastic media, and account for surface topography and the presence of the built environment. A recently developed version of Hercules uses CUDA to execute the code on hybrid CPU/GPU systems.

Erro

r (%

)

10

6

2

–10

–2

–6

10–2 10–1 100

Normalized Frequency10–3 10–2 10–1 100

Normalized Frequency10–3

1.0

0.8

0.6

0

0.4

0.2

Q–1

/Qo–1

Figure 3: Left: CyberShake Study 15.4 ran AWP-ODC-SGT 3D wave propagation software using NSF Blue Waters and DOE Titan supercomputers to produce this seismic hazard map for the Los Angeles area showing peak spectral acceleration at 2 seconds period (PSHA2.0) with a 2% chance of exceedance in 50 years. Right Top: Accelograms shows 1Hz CyberShake seismograms overlayed with 10Hz Broadband CyberShake hybrid ground motion seismogram showing higher frequency ground motions. Right Bottom: CyberShake hazard curves comparing 1Hz CyberShake curves to Broadband CyberShake curves PSHA2.0 (left) and PSHA5.0 (right)

SCEC’s CyberShake software platform utilizes 3D earthquake wave propagation simulations and finite-fault rupture descriptions to compute probabilistic seismic hazard estimates for Southern California. CyberShake computational demands are intense, requiring parallel algorithms and high throughput workflows. Broadband CyberShake results combine 3D ground motion simulation results up to 1Hz with stochastic simulation results to produce peak ground motion estimates for higher frequencies (10Hz+).

Roten, D., Olsen, K.B., Day, S.M., Cui, Y. and Fåh, D. (2014). Expected seismic shaking in Los Angeles reduced by San Andreas fault zone plasticity. Geophysical Research Letters. 41 DOI: 10.1002/2014GL059411

Figure 1: The SCEC SEISM2 Project is developing scientific software that can be used to advance physics-based ground motion simulation to frequencies of interest to civil engineers.

Figure 4: Architecture of I/O interface and new SEISM-IO library. Scientific application programs use the SEISM-IO interface for I/O operations. The I/O interface determines which subroutine in the SEISM-IO library to call accord-ing to parameters provided by the user. SEISM-IO library modules call other high performance I/O libraries for I/O tasks. Adapting to the compute environment and file system technology is taken care of by the underlying I/O libraries. Simulation results using the AWP-ODC wave propagation software with integrated I/O Library are shown in the maps in Figure 2.

SEISM2 Project Goals: More at: http://scec.usc.edu/scecpedia/SEISM2_Project

SEISM2 Project Milestones: More at: http://scec.usc.edu/scecpedia/SEISM2_Project