EU Network MeetingJune 2001

CactusGabrielle Allen, Tom Goodale

Max Planck Institute for Gravitational Physics,

(Albert Einstein Institute)

OutlineOutline

Brief what is Cactus? Review of developments since last meeting

Current release Beta 10 New Viz Tools, thorns, architectures Current work

General Relativity thorns Released thorns in CactusEinstein Other relevant thorns Matter thorns from the WashU Relativity Group Matter in Cactus: current status and plans

Rest of the meeting Talk with me and Tom about what you want to use Cactus for Talk with me if you want to work through some examples, make sure your

laptop has everything installed right, get some advice on how to start, etc Any time left … demos of new viz/control stuff

Cactus in a NutshellCactus in a Nutshell

Cactus acts a the “main” routine of your code, it takes care of e.g. parallelism, IO, checkpointing, parameter file parsing for you (if you want), and provides different computational infrastructure such as reduction operators, interpolators, coordinates, elliptic solvers, …

Everything Cactus “does” is contained in thorns (modules), which you need to compile-in. If you need to use interpolation, you need to add in a thorn which does interpolation.

It is very extensible, you can add you own interpolators, IO methods etc. Not all the computational infrastructure you need will be there, for

example we don’t have a thorn for 2D coordinates yet, you may need to add your own (then hopefully make it available for others to use).

We’re trying to provide a easy-to-use environment for collaborative, high-performance computing, from easy compilation on any machine, to easy visualization of your output data.

Porting Your CodePorting Your Code

If your existing code has this kind of structure

Split into subroutines Clear argument lists

it should be straightforward to put into Cactus.

Cactus will take care of the parameter file, and (hopefully) the coord system and IO, and if you’re lucky you can take someone elses Analysis, Evolution, Initial Data, … modules.

program YourCode

call ReadParameterFilecall SetUpCoordSystemcall SetUpInitialDatacall OutputInitialData

do it=1,niterations call EvolveMyData call AnalyseData call OutputDataend doend

Thorn EvolveMyDataThorn EvolveMyData

Probably you will make a thorn to do the evolution. This thorn will simply contain the source code of your existing evolution routines (with hopefully not to many changes).

You will need to add configuration files which detail how your thorn interacts with the Flesh and other thorns.

Make Information

Source Code

Documentation!

Configuration Files

Parameter Files and Test Suites

????

????Fortran

RoutinesC++

RoutinesC

Routines

Thorn EvolveMyData

Black Hole RunBlack Hole Run

Extract

Exact

PsiKadelia

ADM

AHFinder

ADMConstraints

IDAnalyticBH

PUGHInterp

PUGHReducePUG

H

IOASCII

CartGrid3D

IOBasic

IOUtil

Physics Computational

Einstein

MyThorn FLESH

Parameter FileParameter File

ActiveThorns = "Time CartGrid3D PUGH pughslab pughreduce Einstein IDAnalyticBH ADM IOASCII IOUtil iobasic"

cactus::cctk_itlast = 160grid::type = "byspacing"grid::dx = 0.15grid::dy = 0.15grid::dz = 0.15grid::domain = "octant"driver::global_nx = 20;driver::global_ny = 20;driver::global_nz = 20;einstein::initial_lapse = "one"einstein::evolution_system = "ADM"einstein::initial_data = "schwarzschild"einstein::slicing = "1+log"adm::bound = "radiative"adm::method = "stagleap"IO::outdir = "einstein1"IOASCII::out1D_vars = "einstein::grr einstein::alp"IOASCII::out1D_every = 5 IOBasic::outInfo_vars = "einstein::grr"IOBasic::outInfo_every = 20

How To Find Out About ThornsHow To Find Out About Thorns

Look at web pages http://www.cactuscode.org/Documentation/Thorns/ AEI and WashU thorns ??

Ask a colleague for a sample parameter file New checkout script to make it easy to check out

Parameters? Look in param.ccl in any thorn Use command line options from executable e.g.

./exe/cactus_<config> -O

Thorn documentation: gmake <config>-ThornGuide gmake ThornGuide

No documentation? Complain to authors.

Current Release: Beta 10Current Release: Beta 10

New thorns: CactusUtils/NaNChecker CactusPUGH/PUGHInterp (replaces CactusPUGH/Interp)

Flesh: Changes to time levels, interpolation API, parameters (more error checking

for parameter file) Thorns:

Boundary conditions, initialize memory to NaN/Zero/uninitialized Tools:

Recombiner for HDF5 data New architectures:

MacOS X (no Fortran or MPI) Thorn documentation

gmake <config>-ThornGuide

Planned DevelopmentPlanned Development Easier scripts for checking out and compiling

GetCactus works now from a ThornList Adding GetCactus from a parameter file, and also automatically getting configuration

file for a know machine Einstein:

Improve spacetime – matter interface so that any thorns can be used together for spacetime, matter evolution

Add timelevels (gxx, gxx_p, gxx_p_p), should improve performance, and be ready for AMR

Easier viz, set up networks for OpenDX, Amira Release Cactus 4.0 !!

Mainly documentation and odd and ends Elliptic Solvers:

Get what is there (SOR, PETSc, <BAM>) fully working and better documented using the elliptic interfaces

Coordinates: standard 2D, 1D, spherical Cool stuff: web interface, EMail/SMS thorn, lots more … Ask for what you need, what is difficult: cactusmaint@cactuscode.org

From Other ProjectsFrom Other Projects

AMR Good progress with FMR and AMR now

Portal Web interface from anywhere to all your machines, parameter files,

collaborative and other awesome stuff Grid Computing

For us this really means not needing to know anything at all about the machines you have access to, codes automatically seek out and exploit your resources

Visualization Remote visualization very important for big simulations Also trying to make it easier, scripts to automatically find the right viz clients and

networks for data (VizLauncher), networks specifically for GR Data description and management

Again important for big simulations being run on many different machines Archiving data also really important, understanding and being able to reproduce

old data sets

Users ViewUsers View

Computational ToolkitComputational Toolkit

CactusBase Boundary, IOUtil, IOBasic,

CartGrid3D, IOASCII, Time

CactusBench BenchADM

CactusConnect HTTPD, HTTPDExtra

CactusExample WaveToy1DF77, WaveToy2DF77

CactusElliptic EllBase, EllPETSc, EllSOR, EllTest

CactusPUGH PUGHInterp, PUGH, PUGHSlab,

PUGHReduce

CactusPUGHIO IOFlexIO, IOHDF5, IsoSurfacer

CactusIO IOJpeg

CactusTest TestArrays, TestCoordinates,

TestInclude1, TestInclude2, TestComplex, TestInterp, TestReduce

CactusWave IDScalarWave, IDScalarWaveC,

IDScalarWaveCXX, WaveBinarySource, WaveToyC, WaveToyCXX, WaveToyF77, WaveToyF90, WaveToyFreeF90

CactusExternal FlexIO, jpeg6b

CactusUtils NaNChecker

Einstein ToolkitEinstein Toolkit

Thorn CactusEinstein/Einstein Sets up standard GR variables for 3D problems:

– gxx, gxy, gxz, gyy, gyz, gzz,

– kxx,kxy,kxz,kyy, kyz, kzz,

– alp, betax, betay, betaz Other tools

– Mixed slicing methods (us maximal every 10 timesteps or if ????)

– Courant timestep condition

– Calculate spherical components

– Generic mask function You need to use these variables if you want to use any of the other

Einstein thorns or thorns from AEI/WashU

Einstein ToolkitEinstein Toolkit

Spacetime evolution: ADM: staggered leapfrog, iterative Crank-Nicholson

Analysis: AHFinder: locates apparent horizons using either minimization or

flow algorithm. ADMConstraints: violation of constraint equations. PsiKadelia: calculate gauge invariant quantities Extract: waveforms from nearly axisymmetric, nearly linear regions

Initial Data: IDAnalyticBH, IDBrillData,IDLinearWaves,IDAxiBrillBH

Other ThornsOther Thorns

Links for more information on these will be added to the EU web pages AEI:

Web Page? ADM_BSSN: BS evolution with different shift conditions Lazarus: NP quantities, perturbative evolution Excision: masks out region inside a black hole

AMR: Carpet PAGH

Zeus: Newtonian MHD

Alpha/BetaThorns Maximal, Cartoon2D

Now Tom …Now Tom …

Tom will talk about what’s available at WashU, how Matter is currently treated with the CactusEinstein thorns, and how it is going to be improved so that we can use different spacetime and matter evolvers together.

More Information ...More Information ... Look at news on Cactus home page www.cactuscode.org Also lots more information on Cactus pages:

Worked example of WaveToy showing how to use web server thorn, viz clients

Mailing lists: news, users, developers CVS updates of the thorns you are using Email cactusmaint.cactuscode.org (or me, Tom, Ed, ..)

Go!

Dynamic Grid ComputingDynamic Grid Computing

Clone job with steered parameter

Queue time over, find new machine

Add more resources

Found a horizon,try out excision

Look forhorizon

Calculate/OutputGrav. Waves

Calculate/OutputInvariants

Find bestresources

Free CPUs!!

NCSA

SDSC RZG

SDSC

LRZ Archive data