Embed Size (px)
Citation preview
Grads Meeting - San Diego Feb 2000
The Cactus Code
Gabrielle AllenAlbert Einstein Institute
Max Planck Institute for Gravitational Physics
2
Cactus Code Versions 1,2,3Cactus Code Versions 1,2,3
A code for Numerical Relativity collaborative, portable, parallel, ...
Model of “Flesh” + “Thorns” Flesh: core code, provides parallelisation, IO. Thorns: plug in modules, written in Fortran or C, which provide the
application code.
Successful for numerical relativity, but problems as number of thorns and number of users increased
Redesign, incorporating lessons learnt from previous versions Cactus 4.0
3
Current Version Cactus 4.0Current Version Cactus 4.0
Cactus 4.0 beta 1 released September 1999 Flesh and many thorns distributed under GNU GPL Wednesday: Cactus 4.0 beta 6 Supported Architectures:
SGI Origin SGI 32/64 Cray T3E (142GF on 1024 nodes)
Dec Alpha Intel Linux Windows NT (HP Exemplar) (IBM SP2) (Sun Solaris)
0
20
40
60
80
100
120
0
20
40
60
80
10
0
12
0
Processors
Sc
alin
g
Origin
NT SC
4
Cactus 4 Design GoalsCactus 4 Design Goals
Generalization meta-code that can be applied to any system of PDEs
– mainly 3D cartesian finite differencing codes, but not only...
Abstraction Identify key concepts that can be abstracted
– Evolution skeleton. Reduction operators. I/O. Etc...
Encapsulation Protect the developers of thorns from other thorns ...
Extension Prepare for new concepts in future thorns Overloading, Inheritance, etc...
In some way, make it a little Object Oriented
5
Design issuesDesign issues
Modular and Object Oriented Keep the concept of thorns Encapsulation, Polymorphism, Inheritance, ...
Fortran influences most design issues
Portable Parallelism Support for FMR and AMR as well as Unigrid Powerful Make system Tools such as “Testsuite checking technology”
6
RealizationRealization
Perl The final code is created from Thorn configuration files by perl
scripts that are some sort of seed for a new language:– The Cactus Configuration Language (CCL):
• variables, (functions), • parameters, • scheduling
Perl scripts also take care of testsuite checking
Flesh written in ANSI C Thorns written in C, C++, Fortran77, Fortran90
7
Cactus Flesh: interface between Application Thorns and Computational Infrastructure ThornsCactus Flesh: interface between Application Thorns and Computational Infrastructure Thorns
8
The FleshThe Flesh Abstract API
– evolve the same PDE with unigrid, AMR (MPI or shared memory, etc) without having to change any of the application code.
Interfaces– set of data structures that a thorn exports to the world (global), to its friends
(protected) and to nobody (private) and how these are inherited. Implementations
– Different thorns may implement e.g. the evolution of the same PDE and we select the one we want at runtime.
Scheduling– call in a certain order the routines of every thorn and how to handle their
interdependencies. Parameters
– many types of parameters and all of their essential consistency checked before running
9
Cactus Computational ToolkitCactus Computational Toolkit
Parallel Evolution Drivers PUGH
– MPI domain decomposition based unigrid driver– Distributed using globus
GrACE/PAGH– Adaptive Mesh Refinement driver
Elliptic Solvers PETSc BAM
Interpolators I/O
FlexIO, ASCII, HDF5, Panda, etc...
Visualization, etc...
10
Data StructuresData Structures
Grid Arrays An multidimensional and arbitrarily sized array distributed among
processors
Grid Functions A field distributed on the multidimensional computational grid (a Grid
Array sized to the grid)– Every point in a grid may hold a different value “f(x,y,z)”
Grid Scalars Values common to all the grid points
Parameters Values/Keywords that affect the behavior of the code (initialization,
evolution, output, etc..)– parameter checking, steerable parameters
11
Data TypesData Types
Cactus data types to provide portability across platforms
CCTK_REAL CCTK_REAL4, CCTK_REAL8, CCTK_REAL16
CCTK_INT CCTK_INT2, CCTK_INT4, CCTK_INT8
CCTK_CHAR
CCTK_COMPLEX CCTK_COMPLEX8, CCTK_COMPLEX16, CCTK_COMPLEX32
12
SchedulingScheduling Thorns schedule when their routines should be executed
Basic evolution skeleton idea standard scheduling points INITIAL, EVOL, ANALYSIS fine control: run this routine BEFORE/AFTER that routine
Extend/customise with scheduling groups Add my routine to this group of routines Run the group WHILE some condition is met
Future redesign The scheduler is really a runtime selector of the computation flow. We can add much more power to this concept
13
InterfaceInterface
The concept: contract with the rest of the code Now it is only for the data structures : variables and parameters adding routines and arguments
Private The variables that you want the flesh to allocate/communicate but no
other thorn to see.
Public The variables that you want everybody to see (that means that
everybody can modify them too!) Inheritance
Protected Variables that you want only your friends to see! [Watch out for the change of meaning from C++ names]
14
ImplementationImplementation
Why Two or more thorns that provide the same functionality but different internal
implementation– Interchangeable pieces that allow easy comparison and evolution in the
development process– They are compiled together and only one is activated at runtime
How If all the other thorns need to see the same contract, then thorns
implementing a certain functionality must– Have the same public variables– and their protected ones!!– The same concept applies to parameters and scheduling
Example Wildly different evolution approaches for the same equations, so all the
analysis and initial data thorns remain the same.
15
Parallelism in CactusParallelism in Cactus
Cactus is designed around a distributed memory model. Each thorn is passed a section of the global grid.
The actual parallel driver (implemented in a thorn) can use whatever method it likes to decompose the grid across processors and exchange ghost zone information - each thorn is presented with a standard interface, independent of the driver.
driver:nghostzones = 1
16
PUGHPUGH
The standard parallel driver supplied with Cactus is supplied by thorn PUGH
Driver thorn: Sets up grid variables, handles processor decomposition, deals with processor communications
1,2,3D Grid Arrays and Grid Functions (beta6)
Uses MPI
Custom processor decomposition
Otherwise decomposes in z, then y, then x directions
17
Parallizing an Application ThornParallizing an Application Thorn
CCTK_SyncGroup– synchronise ghostzones for a group of grid variables
– just added Synchronization to Scheduler configuration file as well
CCTK_Reduce– call any registered reduction operator, e.g. maximum value over the grid
CCTK_Interpolate– call any registered interpolation operator
CCTK_MyProc– unique processor number within the computation
CCTK_nProcs– total number of processors
CCTK_Barrier– waits for all processors to reach this point
18
Building an executableBuilding an executable
Compiling Cactus involves two stages
creating a configuration compiling the source files to an executable
Configuration:
Cactus can be compiled with different compilers, different compilation options, with different lists of thorns, with different external libraries (e.g. MPICH or LAM), and on different architectures.
To facilitate this Cactus uses configurations, which store all the distinct information used to build a particular executable (Cactus/configs)
Each configuration is given a unique name.
19
Configuration OptionsConfiguration Options
gmake MyConfig-config <options> (or options file)
Default options decided by autoconf Compiler and tool specification e.g.
F77=/weirdplace/pgf90
Compilation and tool flags e.g. CFLAGS=save-temps DEBUG=ALL
Library and include file specification Precision options e.g.
REAL_PRECISION=16
20
Configuring with External PackagesConfiguring with External Packages
Cactus currently knows about the external packages:
MPI (NATIVE, MPICH, LAM, WMPI,CUSTOM) HDF5 GRACE
and will search standard locations for them
gmake MyConfig MPI=NATIVE
gmake MyConfig MPI=MPICH MPICH_DEVICE=globus GLOBUS_LIB_DIR=/usr/local/globus/lib
gmake MyConfig MPI=CUSTOM MPI_LIBS=mpi MPI_LIB_DIRS=/usr/lib MPI_INC_DIRS=/usr/include
21
Compile OptionsCompile Options
gmake MyConfig <options>
Parallel build FJOBS=<n> TJOBS=<n>
Compilation debugging SILENT=no
[Compiler warnings] WARN=yes
22
Running CactusRunning Cactus
./exe/cactus_MyConfig MyParameterFile.par
Additional command line options-h help
-O[v] details about all parameters
-o <param> details about one parameter
-v version number, compile date
-T list all thorns
-t <thorn> is thorn compiled
-r redirect stdout
-W <num> reset warning level
-E <num> reset error level
23
Parameter FilesParameter Files
Cactus runs from a user’s parameter file chooses the thorns to be used for the run (so that inactive thorns
can’t do any damage) sets parameters which are different from default values
!desc = “Demonstrates my new application”
ActiveThorns = “PUGH WaveToyF77 Boundary CartGrid3D”
driver::global_size = 30 # Change the grid size
wavetoy:: amplitude = 1.56 # Initial wave amplitude
24
Coming up (Cactus 4.2)Coming up (Cactus 4.2) Cactus communication layer
Parallel driver thorn (e.g. PUGH) currently provides both variable management and communication …
abstract send and receives etc Abstract communication from driver thorn
easily implement different parallel paradigms shared memory, threads, Corba, OpenMP, PVM, ...
Compact groups (different layout in memory for improved Cache performance)
Unstructured Meshes/Finite Elements/Spectral Methods Unstructured Multigrid Solver Convergence/Multiple Coordinate Patches Capability browsing mechanism Command line interface … connect directly to Cactus, scheduling GUIs, Documentation, GUIs, Documentation ….
25
www.CactusCode.orgwww.CactusCode.org
Documentation IEEE Computer December 1999 Users Guide Maintainers Guide
Download CVS distribution (stable and development versions)
Development Bugs and feature requests Mailing lists (e.g. [email protected], [email protected])
Showcase Presentations, publications, movies...
News, and Links to related institutions, software
