Webinar on the VERCE Portalverce.eu/Training/UseVERCE/Magnoni_Webinar2014.pdf · ROVR PLMA MAIM CGRP SALO 0 30 60 90 120 0 30 60 90 120 0 30 60 90 120 ASQU CRMI VARE PESA MSSA GORR

Virtual Earthquake and seismology Research Community e-‐science environment in Europe Project 283543 – FP7-‐INFRASTRUCTURES-‐2011-‐2 – www.verce.eu – [email protected]

Webinar on the VERCE Portal

Federica Magnoni

10 &17 July 2014

10 & 17 July 2014Webinar on the VERCE Portal

Forward Simula-ons using SPECFEM3D


Spectral Element MethodsPRO CON

Full complexity (topography, 3D tomography, anisotropy, aYenuaZon…)

!Exponential ACCURACY !Computational EFFICIENCY!!!!

(Diagonal Mass Matrix -‐ Explicit time scheme) !Suitable for implementation on parallel architectures !Mechanical proprieties can vary inside each element !Low number of elements per wavelength

!Flexibility

(unstructured, anisotropic, conforming mesh honoring the geology)

HEX

Meshes of Hexahedral Elements


SPECFEM3D_Cartesian (hAps://github.com/geodynamics/specfem3d)

SimulaZons of acousZc (fluid), elasZc (solid), coupled acousZc/elasZc, poroelasZc or seismic wave propagaZon in any type of conforming mesh of hexahedra (structured or not)

!

Can model seismic waves propagaZng in sedimentary basins or any other regional geological model

!

ImplementaZon of free or absorbing surfaces, aYenuaZon, anisotropy, structural heterogeneity (topography, tomography,…), point or finite kinemaZc/dynamic sources

!

Forward, adjoint, noise cross-‐correlaZon simulaZons !

Finite frequency sensiZvity kernels for elasZc/anelasZc and isotropic/anisotropic media

!

Can be also used for exploraZon seismology, non destrucZve tesZng, ocean acousZcs,…

https://github.com/geodynamics/specfem3d


Mesher

SOLVER

ParZZoner SCOTCH or METIS xdecompose_mesh

-‐1

xspecfem3D

CUBIT kernel

CUBIT interface

PYMPI

“GEO

”cub

it

-‐2 xmeshfem3D

Databases xgenerate_databases

Simula-on Strategy

ParZZoner: SCOTCH 5.1.12b (provided with SPECFEM3D)

Par--oning

• Par_file • 10 mesh files from CUBIT

• nproc proc*_Database (binary files)

xdecompose_mesh (serial)

INPUT OUTPUT

Code: SPECFEM3D_Cartesian

xgenerate_databases (parallel)

• nproc proc*_external_mesh.bin (binary files) !

• if ATTENUATION = true nproc proc*_a,enua/on.bin (binary files)

• if SAVE_MESH_FILES = true nproc proc*_[vp,vs,rho,x,y,z,ibool].bin (binary files) nproc proc*_[vp,vs, a,enua/on, res_minimum_period].vtk (ASCII files)

OUTPUT

• Par_file • nproc proc*_Database • velocity model ASCII file

INPUT

Distributed Databases


xgenerate_databases (parallel)

• Par_file • nproc proc*_Database • velocity model ASCII file

INPUT


• nproc proc*_external_mesh.bin (binary files) !

• if ATTENUATION = true nproc proc*_a,enua/on.bin (binary files)

• if SAVE_MESH_FILES = true nproc proc*_[vp,vs,rho,x,y,z,ibool].bin (binary files) nproc proc*_[vp,vs, a,enua/on, res_minimum_period].vtk (ASCII files)

OUTPUT

MAIN FLAGS in the Par_file: !

SUPPRESS_UTM_PROJECTION = .false. !UTM_PROJECTION_ZONE = UTM projec/on zone in which your model resides !NPROC = The number of MPI processors !MODEL = default !TOMOGRAPHY_PATH = Directory in which the tomography files are stored !LOCAL_PATH = Directory in which the distributed databases will be wri,en !

!

OpZonal flags in the Par_file: !

SAVE_MESH_FILES = .true. to save ParaView (*.vtk) mesh files !STACEY_ABSORBING_CONDITIONS = .true. to turn on Clayton-‐Enquist absorbing boundary condi/ons !ATTENUATION = .true. if a,enua/on should be incorporated …



MAIN FLAGS in the Par_file: !

SUPPRESS_UTM_PROJECTION = .false. !UTM_PROJECTION_ZONE = UTM projec/on zone in which your model resides !NPROC = The number of MPI processors !MODEL = default !TOMOGRAPHY_PATH = Directory in which the tomography files are stored !LOCAL_PATH = Directory in which the distributed databases will be wri,en !

!


SAVE_MESH_FILES = .true. to save ParaView (*.vtk) mesh files !STACEY_ABSORBING_CONDITIONS = .true. to turn on Clayton-‐Enquist absorbing boundary condi/ons !ATTENUATION = .true. if a,enua/on should be incorporated …


xspecfem3D (parallel)

Forward Simula-ons

• Par_file • CMTSOLUTION file for point or finite seismic sources • STATIONS file • nproc proc*_external_mesh.bin (and proc*_a,enua/on.bin if aYenuaZon)

INPUT

• nstat*ncomp seismograms (acc, vel, displ; ASCII files) !• if MOVIE_SURFACE = true

nstep/nstep_between_frames moviedata* (binary files)

OUTPUT


xspecfem3D (parallel)

• Par_file• CMTSOLUTION file for point or finite sources• STATIONS file• nproc proc*_external_mesh.bin (and proc*_attenuation.bin if attenuation)

INPUT

Forward Simula-ons

OUTPUT

• nstat*ncomp seismograms (acc, vel, displ; ASCII files) !• if MOVIE_SURFACE = true

nstep/nstep_between_frames moviedata* (binary files)

OUTPUT

Forward Simula-onsMAIN FLAGS in the Par_file: !

SIMULATION_TYPE = 1 for forward simula/ons !NSTEP = number of /me steps of the simula/on !DT = length of each /me step in seconds !NTSTEP_BETWEEN_OUTPUT_SEISMOS = interval at which synthe/c seismograms are wri,en !


NTSTEP_BETWEEN_OUTPUT_INFO = interval at which basic informa/on about a run is wri,en to the file system (/mestamp* files) !MOVIE_SURFACE = .true. to create a movie of seismic wave propaga/on on the Earth’s surface !MOVIE_TYPE = 1 to show propaga/on on the top surface !NTSTEP_BETWEEN_FRAMES = number of /mesteps between movie frames !HDUR_MOVIE = half dura/on of the source /me func/on for the movie simula/ons !CREATE_SHAKEMAP = .true. to create a ShakeMap® …



MESH

MATERIAL PROPERTIES (velocity model)

EVENT (INGV, CSEM, GRZ)

STATIONS (ORFEUS)

NUMERICAL CODE (SPECFEM3D)

HPC CONNECTION

HPC CENTER POLICIES

HPC CENTER

SEISMOLOGICAL INGREDIENTS HPC INGREDIENTS

VOLUMETRIC DATAMovieShakemap

SYNTHETIC SEISMOGRAMS

SEISMOLOGICAL OUTPUT

MESH


EVENT (INGV, CSEM, GRZ)

STATIONS (ORFEUS)

NUMERICAL CODE (SPECFEM3D)

SEISMOLOGICAL INGREDIENTS


HPC CONNECTION

HPC CENTER POLICIES

HPC CENTER

HPC INGREDIENTS

VOLUMETRIC DATAMovieShakemap

SYNTHETIC SEISMOGRAMS

SEISMOLOGICAL OUTPUT


MESH


SEISMOLOGICAL INGREDIENTSUSER MESH+MODEL VALIDATION !

MESH+MODEL LIBRARY

ITALY (5.8 MHex)

C. ITALY (0.2 MHex)

N. ITALY (1.6 MHex)

S. ITALY (1.8 MHex)

FULL ITALY (Di Stefano 2014)

3s 4s

CENTRAL ITALY (Chiarabba 2010)

3s

NORTH ITALY (Di Stefano 2011)

4s

SOUTH ITALY

(Di Stefano 2011)

3sM

esh

Model


21 June 2013 - M 5.2 LUCCASEISMOLOGICAL OUTPUT

BASIC PARAMETERS Zme step 0.003s staZons 114 cores 500 1 minute seismograms !

PERFORMANCE (SuperMUC) input 2 min decomposiZon 8 min solver+movie 14 min seismograms: 8 min stageout: 30 min

NEXT STEPS: what we can do with the portal output right now?

FIRST NEXT STEPS: 3D visualizaZon (volumetric data)


20 May 2012 -‐ M 5.8 EMILIA -‐ finite fault


Tangential Radial Vertical

TEOL

ROVR

PLMA

MAIM

CGRP

SALO

0 30 60 90 1200 30 60 90 120 0 30 60 90 120

ASQU

CRMI

VARE

PESA

MSSA

GORR

Tangential Radial Vertical

0 30 60 90 1200 30 60 90 120 0 30 60 90 120

3.62e-02 cm/s

2.54e-02 cm/s

2.57e-02 cm/s

2.49e-02 cm/s

2.42e-02 cm/s

1.85e-02 cm/s

4.24e-02 cm/s

3.13e-02 cm/s

1.89e-02 cm/s

1.90e-02 cm/s

3.94e-02 cm/s

1.25e-02 cm/s

1D 3D

20 May 2012 -‐ M 5.8 EMILIA

FIRST NEXT STEPS: MT inversion

quality check of models


Available Meshes

ITALY C. ITALY N. ITALY S. ITALY

Num of Hex (Million) 5.8 0.2 1.6 1.8

UTM zone 33 33 32 33

Suggested DT 1E-‐02 5E-‐03 4E-‐03 5E-‐03

CPU Zme (100 cores -‐

60 s)~3 h 25 min ~2 h ~1.5 h

Given a mesh, compuZng Zme scales linearly with: NUM CORES, DT, SIMULATION DURATION


Mesh FilesDESCRIPTION FORMAT

materials_file material associaZons for each element element_ID material_ID

nummaterial_velocity_file material properZes domain_ID negative_material_ID tomography elastic file_name 1 (domain_ID material_ID rho vp vs Qkappa Qmu anisotropy_flag)

nodes_coords_file point locaZons in Cartesian coordinates of the mesh element corners

num_nodes id_node node_coord_x node_coord_y node_coord_z

mesh_file mesh element connecZvity nspec element_ID id_node1 ... id_node8

free_or_absorbing_surface_file_zmaxfree surface connecZvity or surface

connecZvity of the absorbing boundary surface at the top (Zmax)

nspec2D_top id_(element containing the face) id_node1_face .. id_node4_face

absorbing_surface_file_xmax surface connecZvity of the absorbing boundary surface at Xmax

nspec2D_xmax id_(element containing the face) id_node1_face .. id_node4_face

absorbing_surface_file_xmin surface connecZvity of the absorbing boundary surface at Xmin

nspec2D_xmin id_(element containing the face) id_node1_face .. id_node4_face

absorbing_surface_file_ymax surface connecZvity of the absorbing boundary surface at Ymax

nspec2D_ymax id_(element containing the face) id_node1_face .. id_node4_face

absorbing_surface_file_ymin surface connecZvity of the absorbing boundary surface at Ymin

nspec2D_ymin id_(element containing the face) id_node1_face .. id_node4_face

absorbing_surface_file_boAom surface connecZvity of the absorbing boundary surface at the boYom (Zmin)

nspec2D_bottom id_(element containing the face) id_node1_face .. id_node4_face

hYps://github.com/geodynamics/specfem3d/tree/devel/doc/USER_MANUAL

https://github.com/geodynamics/specfem3d/tree/devel/doc/USER_MANUAL


Submit your own Mesh

START SUBMITTING AFTER AUGUST 25 !

Submit the mesh through the portal providing info about: suggested DT (also related to the wave speed model) suggested NPROC UTM zone mesh corners

Fit the available resources: max ~512 cores and run max 2 hours

Produce the 10 mesh files using CUBIT -‐ GEOCUBIT (hYps://github.com/geodynamics/specfem3d/tree/devel/CUBIT_GEOCUBIT)


Available Wave Speed Models

FULL ITALY (for ITALY mesh)

FULL ITALY (for N. ITALY mesh)

CENTRAL ITALY (for C. ITALY mesh)

NORTH ITALY (for N. ITALY mesh)

SOUTH ITALY (for S. ITALY mesh)

UTM zone 33 32 33 32 33

Min Resolved Period (s) 3 4 3 4 3

vp range (m/s) 768 -‐ 8806 804 -‐ 8774 1782 -‐ 8000 1039 -‐ 8609 1732 -‐ 8245

vs range (m/s) 428 -‐ 4725 434 -‐ 4493 1000 -‐ 4444 600 -‐ 4971 1000 -‐ 4761


Tomography File

Produce an ASCII file

START SUBMITTING AFTER AUGUST 25 !

Submit the tomography through the portal providing info about: UTM zone model corners

Documents

Webinar on the VERCE Portalverce.eu/Training/UseVERCE/Magnoni_Webinar2014.pdf · ROVR PLMA MAIM CGRP SALO 0 30 60 90 120 0 30 60 90 120 0 30 60 90 120 ASQU CRMI VARE PESA MSSA GORR