DRAFT VERSION FEBRUARY - arxiv.org · Draft version February 2, 2008 ABSTRACT ... Having ﬁrst made their appearance in plasma physics , ... (Melott et al. 1997). We

arX

iv:a

stro

-ph/

0411

795v

2 1

0 Ju

n 20

05LA-UR-04-5954DRAFT VERSION MAY 16, 2018

Preprint typeset using LATEX style emulateapj v. 26/01/00

ROBUSTNESS OF COSMOLOGICAL SIMULATIONS I: LARGE SCALE STRUCTURE

KATRIN HEITMANN 1, PAUL M. RICKER2,3, M ICHAEL S. WARREN4, AND SALMAN HABIB 5

1 ISR-1, ISR Division, The University of California, Los Alamos National Laboratory, Los Alamos, NM 875452 Dept. of Astronomy, University of Illinois, Urbana, IL 61801

3 National Center for Supercomputing Applications, Urbana,IL 618014 T-6, Theoretical Division, The University of California, Los Alamos National Laboratory, Los Alamos, NM 875455 T-8, Theoretical Division, The University of California, Los Alamos National Laboratory, Los Alamos, NM 87545

Draft version May 16, 2018

ABSTRACT

The gravitationally-driven evolution of cold dark matter dominates the formation of structure in the Universeover a wide range of length scales. While the longest scales can be treated by perturbation theory, a fully quan-titative understanding of nonlinear effects requires the application of large-scale particle simulation methods.Additionally, precision predictions for next-generationobservations, such as weak gravitational lensing, can onlybe obtained from numerical simulations. In this paper, we compare results from several N-body codes using testproblems and a diverse set of diagnostics, focusing on a medium resolution regime appropriate for studying manyobservationally relevant aspects of structure formation.Our conclusions are that – despite the use of differentalgorithms and error-control methodologies – overall, thecodes yield consistent results. The agreement over awide range of scales for the cosmological tests is test-dependent. In the best cases, it is at the 5% level or better,however, for other cases it can be significantly larger than 10%. These include the halo mass function at lowmasses and the mass power spectrum at small scales. While there exist explanations for most of the discrepancies,our results point to the need for significant improvement in N-body errors and their understanding to match theprecision of near-future observations. The simulation results, including halo catalogs, and initial conditions used,are publicly available.

Subject headings:methods: N-body simulations — cosmology: large-scale structure of the universe

1. INTRODUCTION

1.1. Motivation

There is strong evidence that structure formation in the Uni-verse is largely driven by gravitational instability. In the cos-mological ‘standard model’ the evolution and structure of themass distribution is supposed to be dominated by ‘dark matter,’an as yet undetected type of matter that interacts only gravi-tationally. In order to study spatial and temporal aspects of themass distribution today, it is convenient to consider threelengthscales: very large length scales where a linear treatment oftheinstability suffices, an intermediate length scale where nonlin-earities cannot be neglected yet the dynamics of baryons arenotof significant importance, and a high-resolution regime wheregas dynamics, star formation, etc. need to be taken into ac-count. While an analytic treatment is adequate at the largestscales, because of the nonlinearity, both the intermediateandsmall-scale regimes require a numerical approach.

In cold dark matter-dominated cosmologies, the transitionfrom the linear regime to the nonlinear regime atz= 0 occurs atlength scales of order severalh−1 Mpc (ork∼ 0.1 hMpc−1). De-pending on the application, length scales larger than this can of-ten be satisfactorily treated by linear theory, augmented by per-turbative corrections as needed (Sahni & Coles 1995). Galaxiesexist within dark matter halos, and typical halo size scalesare∼ 0.1− 1 Mpc. Thus, in order to resolve the dynamics of halos,a spatial resolution∼ 10 kpc appears to be necessary. (Here wewill not be interested in questions relating to the cores of halosand halo substructure, where even higher resolution is needed.)

This is the first of two papers where we carry out comparisonstudies of cosmological simulations focusing only on the dy-namics cold dark matter. Given the dominance of this compo-

nent in the matter budget and expected precision observationsof the matter distribution from large scale structure and lens-ing studies, however, we believe these studies to be timely andimportant. Realistic simulations of clusters and galaxy forma-tion require high-resolution treatments of both dark matter andgas dynamics including feedback from star formation, AGNs,gas preheating, etc. There is not yet complete agreement onhow all of these effects should be incorporated; therefore,webelieve that it is too early to attempt a more complete study.

In this paper, we focus on medium resolution simulations,i.e., where the spatial resolution is∼ 10− 100 kpc. These sim-ulations are adequate for matter power spectrum computations,for weak lensing studies, and for gathering cluster statistics. Inthe next paper, we will focus on higher-resolution simulationswhich aim to resolve individual galactic halos.

The main purpose of this work is to characterize the vari-ation of results from different cosmological simulation codes,all codes being given exactly the same initial conditions, andall results being analyzed in identical fashion. The point of thetests carried out in this paper is to identify systematic differ-ences between code results and, to the extent possible, attemptto identify the reasons behind gross discrepancies, shouldsucharise. The aim is not to tune the codes to agree within somepre-defined error but rather to explore the variability in the re-sults when the codes are run using “reasonable” simulation pa-rameters. In addition, it is also not our purpose to establishwhich code is “best” for which application; while we have oftenrun codes with different force resolutions to illustrate generalpoints, we have made no effort to carry out a detailed bench-marking exercise. Finally, the characterization of intrinsicallystatistical issues such as limitations arising from finite samplingand studies of the physical/observational relevance of thediag-

1

http://arxiv.org/abs/astro-ph/0411795v2

2 Robustness of Cosmological Simulations

nostic tools employed (e.g., halo finders) lie beyond the scopeof this paper.

1.2. N-body Simulation

At the scales of interest to structure formation a Newto-nian approximation is sufficient to describe gravitationaldy-namics (Peebles 1980). The task is therefore reduced to solv-ing a (collisionless) Vlasov-Poisson equation. This equation isa six-dimensional partial differential equation, thus, onmem-ory grounds alone, a brute force approach cannot be applied.In addition, because of its essential nonlinearity, the Vlasov-Poisson equation generates ever-smaller spatial structures withtime; it therefore becomes necessary to utilize a robust schemethat avoids instabilities as structure is created on sub-resolutionscales. Having first made their appearance in plasma physics,N-body codes form a now-standard approach for dealing withthis problem (Klypin & Shandarin 1983; Efstathiou et al. 1985;Sellwood 1987; Hockney & Eastwood 1989; Birdsall & Lang-don 1991; Bertschinger 1998; Couchman 1999).

In the N-body approach, the six-dimensional phase spacedistribution is sampled by ‘tracer’ particles and these particlesare evolved by computing the inter-particle gravitationalforces.Since forN particles this is a computationally intensiveO(N2)problem, approximation techniques are used to reduce the forcecomputation to∼ O(N) or∼ O(N logN). These approximationtechniques are typically grid-based or employ multipole expan-sions, or are hybrids of the two. Finally, the addition of gasdynamics is accomplished by coupling the N-body solver to ahydro-code (other effects such as star-formation, etc. canbetreated by employing sub-grid techniques).

As stated above, the purpose of this paper is to investi-gate the consistency of results from purely gravitational N-body codes in the moderate resolution regime. (While perfor-mance issues are also important, they will not be consideredhere.) In this regime, planned observations are now reachingthe point where it is necessary that simulations be performed tolow single-digit (percentage) accuracy (Refregier et al. 2004).To establish an initial baseline, we systematically compare re-sults from the following codes: MC2 (Mesh-basedCosmologyCode) (Habib et al. 2004), a direct parallel particle-mesh(PM) solver; FLASH (Fryxell et al. 20001), an adaptive-mesh-refinement (AMR) grid code; HOT (Hashed-Oct Tree) (War-ren & Salmon 1993), a tree-code; the tree code GADGET(GAlaxies withDark matter andGas intEracTions) (Springel etal. 20012); HYDRA, an adaptive-mesh P3M-SPH code (Couch-man et al. 19953), and TPM, a tree particle mesh code (Xu1995 and Bode et al. 20004). FLASH, GADGET, HYDRA,and TPM are publicly available.

A variety of tests and diagnostic tools have been employed;in all cases, the codes were run with identical initial conditionsand the results analyzed with identical diagnostic tools. We be-gin with the Zel’dovich pancake collapse test to investigate is-sues of convergence and collisionality (Melott et al. 1997). Wethen turn to two relatively realistic situations, the SantaBar-bara cluster comparison (Frenk et al. 1999) and ‘concordance’ΛCDM simulations. Several diagnostics such as velocity statis-tics, halo catalogs, mass functions, correlation functions andpower spectra, etc. were used to compare code results. The ini-

1http://flash.uchicago.edu2http://www.mpa-garching.mpg.de/gadget/right.html3http://coho.mcmaster.ca/hydra/hydra_consort.html4http://www.astro.princeton.edu/ bode/TPM/

tial conditions, outputs from all codes atz= 0, and halo catalogsused in this paper are publicly available5.

The organization of the paper is as follows. In Sec. 2, we givea short general introduction to N-body methods, followed bysummaries of the six different codes employed here. In Sec. 3we discuss results from the Zel’dovich pancake test, focusingon questions regarding symmetry-breaking and collisionality.In Secs. 4 and 5, we describe results from the Santa Barbaracluster test and simulations based on the parameters for thecos-mological Standard Model as measured by WMAP (Spergel etal. 2003) and other observations. In Sec. 6 we discuss our re-sults for the different codes and summarize our conclusions.

2. THE N-BODY PROBLEM FOR DARK MATTER

The Vlasov-Poisson equation in an expanding Universe de-scribes the evolution of the six-dimensional, one-particle distri-bution function,f (x,p) (Vlasov equation):

∂ f (x,p)∂t

= −p

ma2·∇ f (x,p) + m∇Φ(x) · ∂ f (x,p)

∂p, (1)

wherex is the comoving coordinate,p = ma2x, m, the parti-cle mass, andΦ(x), the self-consistent gravitational potential,determined by solving the Poisson equation,

∇2Φ(x) = 4πGa2[ρ(x, t) −ρb(t)], (2)

whereρb is the background mass density andG is the New-tonian constant of gravitation. The Vlasov-Poisson system,Eqns. (1) – (2) constitutes a collisionless, mean-field approx-imation to the evolution of the full N-particle distribution. Asmentioned earlier, N-body codes attempt to solve Eqns. (1) –(2), by representing the one-particle distribution function as

f (x,p) =N∑

i=1

δ(x − xi)δ(p − pi). (3)

Substitution of Eqn. (3) in the Vlasov-Poisson system of equa-tions yields the exact Newton’s equations for a system ofNgravitating particles. It is important to keep in mind, however,that we are not really interested in solving the exact N-bodyproblem for a finite number of particles,N. The actual problemof interest is the exact N-body problem in the fluid limit, i.e.,asN → ∞. For this reason, one important aspect of the nu-merical fidelity of N-body codes lies in controlling errors dueto the discrete nature of the particle representation off (x,p).Once the representation (3) is accepted, errors also arise fromtime-stepping of the Newton’s equations as well as in solvingthe Poisson equation (2). While heuristic prescriptions are rou-tinely followed and tested (see, e.g., Power et al. 2003), a com-prehensive theory of N-body errors does not yet exist.

Determining the forces forN particles exactly requiresO(N2)calculations. This is an unacceptable computational cost at thelarge values ofN typical in modern N-body codes (N ≥ 107),especially as we are not interested in the exact solution in thefirst place. Two popular approximate methods are used to getaround this problem – the first introduces a spatial grid andthe second employs a multipole expansion. Hybrid algorithmsthat meld grid and particle-force calculations are also com-mon. The simplestO(N logN) grid code uses a particle-mesh(PM) methodology wherein the particle positions are sampledto yield a density fieldρ(x) on a regular grid. The Poisson equa-tion is then solved by Fourier or other (e.g., multi-grid) meth-ods, and the force interpolated back on the particles. Improve-ment on the grid resolution can be achieved by adaptive mesh

5http://t8web.lanl.gov/people/heitmann/arxiv/

Heitmann, Ricker, Warren, Habib 3

refinement (AMR) and by direct particle force computation forparticles within a few grid cells – the particle-particle particlemesh (P3M) method. Tree codes are based on the idea of ap-proximating the gravitational potential of a set of particles bya multipole expansion. All the particles are arranged in a hi-erarchical group structure with the topology of a tree. Givena point at which the potential needs to be evaluated and a par-ticular particle group, one decides if the point is sufficientlyfar away (opening angle criterion) in order for a (suitably trun-cated) multipole expansion to be used; if not, the subgroupsofthe parent group are investigated. The process is continuedun-til all groups are fully searched down to the individual particlelevel.

Basic grid methods such as PM are fast, relatively simpleto implement, and memory-efficient. Consequently, they havegood mass resolution, but due to the limited dynamic range ofthe three-dimensional spatial grid, only moderate force resolu-tion. Tree codes are gridless and have much better force resolu-tion, but this typically involves a performance and memory cost.Adaptive mesh and hybrid algorithms such as tree-PM (TPM)constitute further steps in attaining better compromises betweenmass and force resolution. (Due to discreteness/collisionalityerrors, excellent force resolution with poor mass resolution ispotentially just as problematic as the reverse.)

2.1. The Codes

The codes utilized in this study have different algorithms,possess different error modes and employ different error controlstrategies. Thus, good convergence to a single solution is astrong test of the validity of the N-body approach in modelingdissipationless gravitational dynamics at the resolutionscalesprobed by the tests. We emphasize that the codes were runwith no special effort to optimize parameters separately for theindividual test problems; to conform to typical usage, onlytherecommended default parameter values were used. Separateintroductions to the codes are given below.

2.1.1. MC2

The multi-species MC2 code suite (Habib et al. 2004) in-cludes a parallel PM solver for application to large scale struc-ture formation problems in cosmology. In part, the code de-scended from parallel space-charge solvers for studying high-current charged-particle beams developed at Los Alamos Na-tional Laboratory under a DOE Grand Challenge (Ryne et al.1998; Qiang et al. 2000). It exists in two versions: a sim-pler version meant for fast prototyping and algorithm develop-ment written in High Performance Fortran (HPF), and a high-performance version based on Viktor Decyk’s F90/MPI UPICframework (Decyk & Norton6). This code has excellent massresolution and has proven its efficiency on multiple HPC plat-forms, with scaling being verified up to 2000 processors.

MC2 solves the Vlasov-Poisson system of equations for anexpanding universe using standard mass deposition and forceinterpolation methods allowing for periodic or open boundaryconditions with second and fourth-order (global) symplectictime-stepping and a Fast Fourier Transform (FFT)-based Pois-son solver. The results reported in this paper were obtainedusing Cloud-In-Cell (CIC) deposition/interpolation. Theover-all computational scheme has proven to be remarkably accurateand efficient: relatively large time-steps are possible with ex-ceptional energy conservation being achieved.

6http://exodus.physics.ucla.edu/research/UPICFramework.pdf

2.1.2. FLASH

FLASH (Fryxell et al. 2000) originated as an AMR hy-drodynamics code designed to study X-ray bursts, novae, andType Ia supernovae as part of the DOE ASCI Alliances Pro-gram. Block-structured adaptive mesh refinement is providedvia the PARAMESH library (MacNeice et al. 2000). FLASHuses an oct-tree refinement scheme similar to Quirk (1991) andde Zeeuw & Powell (1993). Each mesh block contains thesame number of zones (163 for the runs in this paper), and itsneighbors must be at the same level of refinement or one levelhigher or lower (mesh consistency criterion). Adjacent refine-ment levels are separated by a factor of two in spatial resolution.The refinement criterion used is based upon logarithmic densitythresholds. The mesh is fully refined to the level at which theinitial conditions contain one particle per zone. Blocks withmaximum densities between 30 and 300 times the average arerefined by one additional level. Those with maxima between300 and 3000 times the average are refined by two additionallevels, and each factor of 10 in maximum density above thatcorresponds to an additional level of refinement. FLASH hasbeen extensively verified and validated, is scalable to thousandsof processors, and extensible to new problems (Rosner et al.2000; Calder et al. 2000; Calder et al. 2002).

Numerous extensions to FLASH have been developed, in-cluding solvers for thermal conduction, magnetohydrodynam-ics, radiative cooling, self-gravity, and particle dynamics. Inparticular, FLASH now includes a multigrid solver for self-gravity and an adaptive particle-mesh solver for particle dynam-ics (Ricker et al. 2004). Together with the PPM hydrodynamicsmodule, these provide the core of FLASH’s cosmological sim-ulation capabilities. FLASH uses a variable-timestep leapfrogintegrator. In addition to other timestep limiters, the FLASHparticle module requires that particles travel no more thanafraction of a zone during a timestep.

2.1.3. HOT

This parallel tree code (Warren & Salmon 1993) has beenevolving for over a decade on many platforms. The basic algo-rithm may be divided into several stages (the method of errortolerance is described in Salmon & Warren 1994). First, par-ticles are domain decomposed into spatial groups. Second, adistributed tree data structure is constructed. In the mainstageof the algorithm, this tree is traversed independently in eachprocessor, with requests for non-local data being generated asneeded. AKey is assigned to each particle, which is basedon Morton ordering. This maps the points in 3-dimensionalspace to a 1-dimensional list, maintaining as much spatial lo-cality as possible. The domain decomposition is obtained bysplitting this list intoNp (number of processors) pieces. An ef-ficient mechanism for latency hiding in the tree traversal phaseof the algorithm is critical. To avoid stalls during non-localdata access, effectively explicit ‘context switching’ is done us-ing a software queue to keep track of which computations havebeen put aside waiting for messages to arrive. This code archi-tecture allows HOT to perform efficiently on parallel machineswith fairly high communication latencies (Warren et al. 2003).This code was among the ones used for the original Santa Bar-bara Cluster Comparison Project (Frenk et al. 1999) and alsosupports gas dynamics simulations via a smoothed particle hy-drodynamics (SPH) module (Fryer & Warren 2002).


2.1.4. GADGET

GADGET (Springel et al. 2001) is a freely available N-body/hydro code capable of operating both in serial and par-allel modes. For the tests performed in this paper the parallelversion was used. The gravitational solver is based on a hier-archical tree-algorithm while the gas dynamics (not used here)are evolved by SPH. GADGET allows for open and periodicboundary conditions, the latter implemented via an Ewald sum-mation technique. The code uses individual and adaptive timesteps for all particles.

2.1.5. HYDRA

HYDRA (Couchman et al. 1995) is an adaptive P3M (AP3M)code with additional SPH capability. In this paper we use HY-DRA only in the collisionless mode by switching off the gasdynamics. The P3M method combines mesh force calculationswith direct summation of inter-particle forces on scales oftwoto three grid spacings. In regions of strong clustering, thedi-rect force calculations can become significantly expensive. InAP3M, this problem is tackled by utilizing multiple levels ofsubgrids in these high density regions, with direct force com-putations carried out on two to three spacings of the higher-resolution meshes. Two different boundary conditions are im-plemented in HYDRA, periodic and isolated. The timestep al-gorithm in the dark matter-only mode is equivalent to a leapfrogalgorithm.

2.1.6. TPM

TPM (Xu 1995 and Bode et al. 2000), a tree particle-meshN-body algorithm, is a hybrid code combining a PM and a treealgorithm. The density field is broken down into many iso-lated high-density regions which contain most of the mass inthe simulation but only a small fraction of the volume. In theseregions the gravitational forces are computed with the treealgo-rithm while for the bulk of the volume the forces are calculatedvia a PM algorithm, the PM time steps being large comparedto the time-steps for the tree-algorithm. The time integrator inTPM is a standard leap-frog scheme: the PM timesteps are fixedwhereas the timesteps for the tree-particles are variable.

3. THE ZEL’DOVICH PANCAKE TEST

3.1. Description of Test

The cosmological pancake problem (Zel’dovich 1970; Shan-darin & Zeldovich 1989) provides a good simultaneous test ofparticle dynamics, Poisson solver, and cosmological expansion.Analytic solutions well into the nonlinear regime are availablefor both dark matter and hydrodynamical codes (Anninos &Norman 1994), permitting an assessment of code accuracy. Af-ter caustic formation, where the analytic results fail, theprob-lem continues to provide a stringent test of the ability to trackthin, poorly resolved features (Melott 1983).

As is conventional, we set the initial conditions for the pan-cake problem in the linear regime. In a universe withΩ0 = 1 atredshiftz, a perturbation of wavenumberk which collapses toa caustic at redshiftzc < z has comoving density and velocitygiven by

ρ(xe;z) = ρ

[

1−1+ zc

1+ zcos(kxℓ)

]−1

, (4)

v(xe;z) = −H0(1+ z)1/2(1+ zc)sinkxℓ

kxℓ, (5)

whereρ is the comoving mean density. Herexe is the distanceof a point from the pancake midplane, andxℓ is the correspond-ing Lagrangian coordinate, found by iteratively solving

xe = xℓ −1+ zc

1+ zsinkxℓ

k. (6)

The Lagrangian coordinates for the dark matter particlesxℓ areassigned to lie on a uniform grid. The corresponding perturbedcoordinatesxe are computed using Eqn. (6). Particle velocitiesare assigned using Eqn. (5).

As particles accelerate toward the midplane, their phase pro-file develops a backwards “S” shape. At caustic formation thevelocity becomes multivalued at the midplane. The region con-taining multiple streams grows in size as particles pass throughthe midplane. At the edges of this region (the caustics, or theinflection points of the “S”), the particle density is formally in-finite, although the finite force and mass resolution in the sim-ulations keeps the height of these peaks finite. Some of theparticles that have passed through the midplane fall back andform another pair of caustics, twisting the phase profile again.Because each of these secondary caustics contains five streamsof particles rather than three, the second pair of density peaksare higher than the first pair. In principle, this caustic formationprocess repeats arbitrarily many times. In practice, however,the finite number of particles and the finite force resolutioninsimulations limit the number of caustics that can be tracked.

In Melott et al. (1997) the pancake simulation was used totest different codes for unphysical collisionality and symmetry-breaking. It was found that tree and P3M codes suffered fromartifacts if the value of the softening parameter was chosentoosmall, leading to incorrect results for the pancake test. Wehavecarried out a set of simulations to distinguish between lackofconvergence due to collisional effects and the failure of the testdue to the inability to strictly track the planar symmetry ofthemass distribution. Our basic conclusions are that for the massand force resolutions chosen in the main cosmological simula-tions in this paper, collisional effects are unlikely to be impor-tant. While it is true that the high-resolution codes fail atmain-taining the symmetry of the pancake test, the PM-code MC2

does not. Thus, one may verify whether this failure is an issuein realistic simulations by comparing results for the differentcodes, as we do later in the paper. Our results are consistentwith the finding that, at least in the moderate resolution regime,this failure in the pancake test does not translate to makingsig-nificant errors in more realistic simulations.

3.1.1. Pancake Simulations

Our main aim in performing the pancake simulations is toprobe whether the codes are working correctly in the dynamicalrange of interest. For this purpose, small simulations are suffi-cient; what is important is to match the mass resolution and thespatial resolution to the ones used in the larger cosmologicalsimulations. Most of these larger simulations used 2563 parti-cles, the box sizes varying between 64 Mpc, 90 Mpc, and 360Mpc, with different cosmologies. This leads to particle massesof 1.08·109 M⊙, 1.9·109 M⊙, and 1.22·1011 M⊙. We performthe pancake test in a

√3 ·10 Mpc box, with 643 particles. This

leads to a particle mass of 1.4·109 M⊙, which provides a goodrepresentation for the cosmological test problems. In order toalmost match the force resolution of the PM code with the othercodes we ran it using a 10243 grid in the cosmological simula-tions. For the 64 Mpc and 90 Mpc boxes, this is approximatelyequivalent to running the pancake test with a 2563 PM grid.


Based on these considerations, the following simulationswere run: for the cosmology we chose a SCDM model, i.e.Ωm = 1, H = 50 km/s/Mpc,Ωb = ΩΛ = 0, in a

√3 ·10 Mpc box

with 643 particles in all cases. HOT was run with a (Plummer)smoothing of 20 kpc while the smoothing for the HYDRA-simulation was chosen to be 100 kpc. In AMR mode, theFLASH grid size was refined up to three times, leading up toan effective 5123 grid. We varied the mesh size for the MC2

simulation between 643 and 5123 to check convergence for in-creasing number of grid points. The TPM and GADGET runsdid not complete successfully for this test.

The initial conditions are set up such that the pancake normalis aligned with the box diagonal, i.e., inclined at 54.7 degreeswith respect to the base plane. In order to check our resultswe performed a high-resolution one-dimensional pancake sim-ulation to serve as a comparison template. Due to the lack ofan analytical solution, this numerical comparison is importantfor verifying our results in the nonlinear regime, after severalcaustics have formed.

3.1.2. Results

The initial conditions were chosen so that the redshift atwhich the first collapse occurs iszc = 5. We measured the po-sitions and velocities of the particles atz = 7, i.e., before thecollapse, and also atz = 0 after several caustics have formed.The results are displayed in phase-space plots, where positionsand velocities are projected onto the box diagonal.

At z= 7, before caustic formation occurs, all codes performvery well as can be seen in Fig. 1. In addition, we performed aconvergence test with the PM-code MC2. We ran a 643 particlesimulation on a mesh which was refined four times, from 643 to5123 mesh points (Fig. 1 shows the MC2 result from the 2563

mesh). Atz = 7, even the lowest resolution simulation witha 643 mesh, e.g., the analogous result for FLASH in Fig. 1),traces the exact solution accurately. Expectedly, increasing thenumber of mesh-points improves the result mainly near the re-gion of maximal density variation, i.e., atx = 2.5 Mpc. Nofailure of convergence was noted at the highest-resolution5123

simulation.In contrast, by the time several caustics have formed, as

shown in Fig. 2 forz= 0, the code results display considerablevariation and lack of convergence. To understand this behavior,we first turn to a convergence test with MC2 shown in Fig. 3.Here we have zoomed into the inner part of the phase-space spi-ral by restricting thex-axis between 2.3 and 2.7 Mpc. While the643 mesh simulation can track the first two caustics, the innerpart of the spiral is not resolved. A 1283 mesh provides someimprovement, while a 2563 mesh – corresponding to a resolu-tion of 67 kpc – leads to satisfactory agreement with the essen-tially exact 1-D numerical solution (in Fig. 2, the MC2 resultsare shown for the 2563 mesh simulation). This is the level ofMC2 spatial resolution used for the cosmological simulations.

As the mesh is refined further, collisional effects must enterat some point. In fact, by refining the grid one more time andrunning the 643 particles on a 5123 mesh (corresponding to aresolution of 34 kpc) the failure of convergence can be directlyobserved. While the increase from a 643 mesh to a 2563 meshclearly led to a better refinement of the inner spiral, i.e. thesimulation was converging towards the exact solution, a furtherincrease of the spatial resolution doesnotcontinue the improve-ment. The particles on the inner part of the spiral are mov-ing away from the exact solution instead of converging towards

FIG. 1.— Phase plane for the pancake test from four simulations with 643

particles atz = 7, prior to caustic formation. As an illustration, the FLASHresult is taken from a low-resolution 643 mesh to demonstrate the resultingsmall inaccuracies near the curve minimum and maximum. The solid lines arefrom a one-dimensional high-resolution simulation.


FIG. 2.— Pancake test atz= 0, 643 particles, following Fig. 1. Very close tothe center of the spiral, there is a seven-stream flow. Here, FLASH is run withan effective resolution equivalent to a 5123 mesh (For the equivalent resolutionMC2 results, see Fig. 3. For a discussion of all of the results, see the text.

FIG. 3.— Failure of convergence near the midplane for the pancake test:MC2 results, 643 particles with four grid sizes atz = 0. Convergence fails atthe final resolution reduction step (going from a 2563 mesh to a 5123 mesh).See the text for a discussion of these results.

it. Collisional effects destroy the good convergence propertieswhich were found for the smaller meshes. Therefore, small-scale resolved structures at this high level of spatial resolutioncan turn out to be incorrect. Convergence tests performed withFLASH with uniform resolution give similar results.

It is clear, however, that the results in Fig. 2 show artifactsmuch worse than the mild lack of convergence observed inFig. 3. For example, the result from running FLASH with re-finement turned on, yielding an effective 5123 mesh, shows se-vere artifacts: the particle distribution is smeared out close tothe center of the spiral. Note that the smearing seen here can-not be explained as a consequence of “over-resolution” as thiseffect is not seen in the MC2 results of Fig. 3, which have thesame effective resolution. A similar result was obtained withthe tree-code HOT, where the structure of the inner spiral ismore or less completely lost; the results from the AP3M codeHYDRA are not any better. It is the failure of the AMR, tree,and AP3M codes to maintain the strict planar symmetry of thepancake collapse that is apparently the root cause of the prob-lem. To reiterate, the 2563 and 5123 PM runs roughly span theforce resolutions used for the other codes, and since only the5123 run shows a very mild failure of convergence, force reso-lution alone cannot be the source of the difficulty.

Our results provide a different and more optimistic interpre-tation of the findings of Melott et al. (1997). (See also Bin-ney 2004.) While high-resolution codes when run with smallsmoothing lengths (or several refinement levels in the case ofAMR) are not able to pass the pancake test after the formationof several caustics, the main culprit appears to be an inabilityto maintain the planar symmetry of the problem and not directcollisionality (at least at the force resolutions relevantfor thispaper), which would have been far more serious. Whether thefailure to treat planar collapse is a problem in more realisticsituations can be tested by comparing results from the high-resolution codes against brute-force PM simulations. A batteryof such tests have been carried out in Secs. 4 and 5. At the force


TABLE 1

SOFTENING LENGTHS/MESH SPACING FOR THESANTA BARBARA RUNS

MC2 FLASH HOT GADGET TPM HYDRA1283 particles 62.5 kpc 250 kpc 5 kpc (physical after z=9) 10 kpc 10kpc 10 kpc2563 particles 62.5 kpc 62.5 kpc 5 kpc (physical after z=9) 50 kpc N/A N/A

Note. — Resolutions of the different codes for the differentSanta Barbara runs; the highest resolution runs were carried out with HOT. If not so noted, the resolutionis quoted in comoving coordinates.

resolutions investigated, these tests failed to yield evidence forsignificant deviations.

4. THE SANTA BARBARA CLUSTER

4.1. Description of the Test

Results from the Santa Barbara Cluster Comparison Projectwere reported in 1999 in Frenk et al. (1999). The aim of thisproject was to compare different techniques for simulatingtheformation of a cluster of galaxies in a cold dark matter universeand to decide if the results from different codes were consis-tent and reproducible. For this purpose outputs from 12 differ-ent codes were examined, representing numerical techniquesranging from SPH to grid methods with fixed, deformable, andmultilevel meshes. The starting point for every code was thesame set of initial conditions given either by a set of initial po-sitions or an initial density field. Every simulator was thenal-lowed to evolve these initial conditions in a way best suitedforthe individual code, i.e., implementations of smoothing strate-gies, integration time steps, boundary conditions, etc. were notspecified but their choice left to the individual simulators. Thecomparison was kept very general and practical; the main in-terest was not to quantify the accuracy of individual codes butto investigate the reliability of results from differing astrophys-ical simulations. For this purpose, images of clusters at differ-ent epochs were compared, as well as the mass, temperature,cluster luminosity, and radial profiles of various dynamical andthermodynamic quantities.

In this paper we restrict ourselves to the dark matter compo-nent of the simulation, run with periodic boundary conditions,and show results only atz= 0. In a departure from the originalSanta Barbara project, we enforce strict uniformity by usingthe same particle initial condition for all the codes and usingonly one framework to analyze the final outputs of the simu-lations, e.g., the strategy of finding the center of the cluster isexactly the same for all simulations. This procedure allowsfora uniform comparison of the different codes. In the followingwe briefly describe the simulations performed and then analyzeand compare the individual results.

4.2. Simulations Performed

The simulation of a single cluster in a small box is a timeconsuming task for most cosmology codes. The very highdensity region of the cluster demands very small time stepsto resolve short orbital timescales. In order to avoid long runtimes, we therefore decided to run the original Santa Barbaratest with 2563 particles on only a subset of the tested codes:MC2, HOT, FLASH (our three main codes), and GADGET ata lower resolution than HOT. In addition we reduced the initialconditions from 2563 particles to 1283 particles by averagingover every eight particles and ran all six codes with these initial

conditions. In this second set of simulations, lower-resolutionFLASH results are used to convey some measure of sensitiv-ity to resolution limits. The cosmology employed in the SantaBarbara test is SCDM, i.e.,Ωm = 1.0 and the Hubble constantH = 50 km/s/Mpc. The box size is 64 Mpc. The properties ofthe cluster were all measured atz= 0. Table 1 summarizes theresolutions at which the different simulations were performed.

4.3. Results

As in the original Santa Barbara test paper (“SB paper” inthe following) we study some specific properties of the clusterat z= 0, the velocity and density profiles, and projected imagesof the cluster itself.

We start our discussion with a useful but qualitative compar-ison, using images of the two-dimensional projected density ofthe cluster. All results are shown atz= 0. Figs. 4 and 5 displaythe cluster in an 8 Mpc box from the 1283 and the 2563 simula-tions, respectively. In order to provide the best direct compari-son we have not filtered the images. (A Gaussian smoothing of250 kpc as used in the original paper washes out far too manydetails of the cluster which are real.) The density is projectedonto a 10242 xy-plane grid and displayed in logarithmic units.The pixel size is 62.5 kpc which is substantially larger thantheformal spatial resolution in the higher-resolution runs.

The overall appearance of the cluster – orientation, shape,size, highest density region in the center – agrees very wellfor all simulations. Comparing the 1283 simulation from MC2,HOT, HYDRA, and TPM shows that many small scale featuresare reproduced with high fidelity. All four simulations showthree small high density regions in the NE as well as three highdensity regions in the SW, and a high density region in the up-per left corner of the plot. The FLASH image shows most ofthese features as well, but due to the lower resolution employedsome of the very small features are missing – especially on theedge of the cluster. The GADGET results show some smalldeviations compared to the other simulations: The three highdensity regions in the NE visible in the other results reducetotwo in the GADGET image, and the lower left corner doesn’tshow as much structure. Overall one gets the impression thatthe GADGET image is not atz = 0, but slightly shifted. Thismay be due to a non-optimal setting of the maximum timestepin the GADGET configuration file (0.01). While this value isappropriate at low redshifts, in the linear regime GADGET willtake timesteps equal in size to the MaxSizeTimestep parame-ter. We have not investigated whether making this parametersmaller would resolve this discrepancy. Sub-structure is clearlypresent in the HYDRA, HOT, and TPM images. However, thereis a lack of one-to-one agreement at the level of these individualfeatures.

The situation for the images from the 2563 simulations shownin Fig. 5 is very similar. Due to better mass resolution the clus-


TABLE 2

CLUSTER PROPERTIES MEASURED ATz= 0

MC2 FLASH HOT GADGET HYDRA TPM Average

r200 [Mpc]

1283 2.753 2.753 2.753 2.753 2.753 2.753 2.753Residual – – – – – –2563 2.753 2.753 2.753 2.753 N/A N/A 2.753Rel. Residual – – – – N/A N/A

mtot [1015 M⊙]

1283 1.215 1.196 1.220 1.214 1.218 1.212 1.213Rel. Residual 0.002 -0.01 0.006 0.001 0.005 0.00042563 1.208 1.208 1.215 1.246 N/A N/A 1.220Rel. Residual -0.01 -0.01 -0.004 0.02 N/A N/A

σDM [km/s]

1283 989.72 900.93 988.15 1009.66 1011.77 1007.03 984.54Rel. Residual 0.005 -0.08 0.004 0.026 0.028 0.0232563 998.09 1016.79 1005.62 1012.05 N/A N/A 1008.14Rel. Residual -0.01 0.009 -0.003 0.004 N/A N/A

ter appears to be smoother than in the 1283 simulation. Theagreement of the MC2, FLASH, and HOT results is excellent,as almost all obvious features (except the sub-structure) areidentical. The HOT image shows more substructure than theMC2 and FLASH images which is to be expected due to thehigher force resolution of the tree-code compared to the mesh-codes. As in the 1283-particle simulation, the GADGET imageseems to be taken at a slightly different redshift than the otherones. In summary the result of this qualitative comparison isvery good – much better than in the original paper.

We continue with a more quantitative comparison of clusterproperties. In Table 2 results for the radiusr200 of the clustermeasured in Mpc, the total mass measured in 1015 M⊙, and therms velocity dispersion of the cluster in km/s are listed. Wealso give the average measurement for each quantity and thedeviation from the average for the simulations.

Independent of the number of particles (1283 or 2563) allcodes lead to exactly the same result for the radius of the clus-ter r200 = 2.753 Mpc, which is slightly larger than the resultfrom the original test,r200 = 2.70 Mpc with an rms scatter of0.04 Mpc.

The average mass for the cluster is 1.21· 1015 M⊙ for the1283 particle simulation and 1.22· 1015 M⊙ for the 2563 sim-ulation. This is a little higher than the average mass found inthe original test; the highest mass reported from two runs intheSB paper was 1.21·1015 M⊙. The deviation from the averagefor all runs is of the order of 1%, for some codes even sub-1%.Here we quote the relative residual calculated as (〈x〉− x)/〈x〉,〈x〉 being the average of all results. In the original paper theagreement was reported to be sub-10%, an order of magnitude

worse than our result. For the radius, the results from all theruns agree to three significant figures so no residuals are shown.

The average of the velocity dispersionσDM (following theconvention of the original paper, the one-dimensional velocitydispersionσDM = σ/

√3 with σ being the three-dimensional ve-

locity dispersion) is around 1000 km/s, slightly higher than inthe SB paper. The agreement is remarkably good for both 1283

and 2563 particle simulations with a deviation of only around1%. Due to the resolution limits, the 1283-FLASH simulationproduces a value 8% lower than the average. In the originalpaper the differences in the individual code results were againmuch less satisfactory than in our comparison, with deviationsof up to 10%. We discuss the reasons for the improved resultslater below.

Finally, we compare three different cluster profiles: the darkmatter density, velocity dispersion, and the radial velocity asfunctions of radius. These profiles were obtained by first deter-mining the center of the cluster, and then averaging the particleinformation in 15 spherical shells evenly spaced in log radius.While we have shown all results for all codes down to the small-est radius, the results from MC2, FLASH, and GADGET for the2563 particle runs can be only trusted down tor = 0.06 Mpc be-cause of the resolution employed. For the 1283 particle runsthe force resolution of HOT, GADGET, HYDRA, and TPM issufficient to generate correct results over the entire measuredr-range while for MC2 resolution artifacts are expected to beseen atr < 0.06 Mpc and for FLASH atr < 0.2 Mpc. Follow-ing the original SB paper, the main plots are in the lower panelwith the average over the code results shown with a solid curve(we have removed the FLASH results from the average in the


FIG. 4.— Santa Barbara cluster simulation: Projected dark matter density in logarithmic units atz= 0, 1283 particles projected on a 10242 mesh, no smoothing.The dynamic range of the density variation in this figure is roughly 5 orders of magnitude. The lower force resolution usedhere for the FLASH simulation isapparent in the figure.

FIG. 5.— Projected dark matter density following Fig. 4, but for2563 particles. The FLASH force resolution is better by a factor of 4 than for the 1283 particlerun while the GADGET force resolution is 5 times worse. The force resolution for MC2 and HOT are the same as in the 1283 runs (see also Table 1). Note theincreased definition of substructure independent of the force resolution.


1283 particle simulations). In the upper panel the residuals aredisplayed, calculated as lnx− ln〈x〉 for the dark matter densityand velocity dispersion profiles andx− 〈x〉 for radial velocityprofiles.

The dark matter density profile for the 1283 particle runs isshown in Fig. 6. Down to a radius ofr = 0.25 Mpc the agree-ment of all codes is very good, at the sub-10% level. For smallerradii the FLASH profile falls off faster due to resolution limita-tions. The residuals for FLASH in the regionr < 0.1 Mpc arenot shown. The agreement of the remaining five codes is stillvery good (better than 10%) untilr = 0.06 Mpc is reached, theresolution limit of the MC2 simulation. At the smallest radiir < 0.05 Mpc HOT, HYDRA, and TPM are still in very goodagreement while the GADGET profile is somewhat steeper. Forcomparison we plot a profile suggested by Navarro, Frenk, &White (1995) (“NFW profile” in the following) with the con-centration parameterc = 7.12. We calculated the optimal con-centration parameter based on the average mass andr200 ob-tained from our simulations using a publicly available codefrom Navarro7. The value we found is smaller than in the orig-inal SB paper wherec = 7.5 was chosen. The lower value hereresults from the higher cluster mass as compared to the originalSB paper. The profile is an excellent match to the simulationresults.

In Fig. 7 we show the result for the four 2563 particle runs.Down to r = 0.06 Mpc, the resolution limit for MC2, FLASH,and GADGET in this simulation, the agreement of all runs issub-8%, again excellent. (The agreement in the original paperwas 20% or better.) In this simulation, only HOT was run witha very small smoothing length, thus at small radii it is the onlycode with enough resolution to provide converged results. Theother three codes were run with roughly equivalent force reso-lution and they do provide consistent results at small radii, thefall-off of the halo density forr < 0.1 Mpc resulting from theforce smoothing. The corresponding NFW profile is shown forcomparison.

In Figs. 8 and 9 the dark matter velocity dispersion profilesare shown for the 1283 particle and 2563 particle runs. In bothcases the velocity dispersion peak is higher than in the originalSB paper, 1250 km/s for the 1283 runs and 1200 km/s for the2563 runs, while in the SB comparison the average maximumwas around 1100 km/s. This could be due to the higher forceresolutions in the present runs, a conclusion consistent with thehigherσDM peak values obtained in the original paper by thehigher-resolution codes. This argument can also be checkedby the low-resolution FLASH-1283 run: The maximum hereis around 1000 km/s. The general agreement in both the 1283

and 2563 particle cases is very good for the outer shells, betterthan 5%. In the inner shells the agreement is better than 10%,with only the low-resolution FLASH-1283 run disagreeing upto 20%. Following the results for the density profile, the overallagreement between the different runs is better by a factor oftwo than in the original SB paper. We do not observe a largescatter of the results in the last bin as reported in the SB paper.There it was argued that the scatter is due to noise arising fromsubclustering but we see no evidence to support this.

Finally, we compare the radial velocity profiles. The resultsare shown in Figs. 10 and 11. Net infall is seen at roughlythe same radius as in the original paper, between 600 kpc and700 kpc for both cases. The scatter in the 1283 particle simula-tions is around 150 km/s (note that the residual here is quoted

7http://pinot.phys.uvic.ca/ jfn/mywebpage/home.html

-0.2

0

0.2

0.4

Res

idua

ls0.01 0.1 1 10

r [Mpc]

1e+11

1e+12

1e+13

1e+14

1e+15

1e+16

ρ D

M[M

O/M

pc3 ]

·

MC2

FLASHHOTGADGETHYDRATPMMean ProfileNFW Profile

FIG. 6.— Dark matter density profile for the Santa Barbara cluster, 1283

particles. The mean profile (solid curve) is a simple averageover all the simu-lations, with the exception of the FLASH results which were run at low reso-lution.

-0.2

0

0.2

0.4

0.6

Res

idua

ls

0.01 0.1 1 10r [Mpc]

1e+11

1e+12

1e+13

1e+14

1e+15

1e+16

ρ D

M[M

O/M

pc3 ]

·

MC2

FLASHHOTGADGETMean ProfileNFW Profile

FIG. 7.— Dark matter density profile for the Santa Barbara cluster, 2563

particle simulations. Unlike Fig. 6, here MC2 and FLASH are run with thesame effective force resolution.


-0.2

-0.1

0

0.1

0.2

Res

idua

ls

0.01 0.1 1 10r [Mpc]

600

800

1000

1200

σ D

M[k

m/s

]

MC2

FLASHHOTGADGETHYDRATPMMean Profile

FIG. 8.— Dark matter velocity dispersion profile for the Santa Barbaracluster, 1283 particles. The low-resolution FLASH results are not included inthe mean profile.

-0.2

-0.1

0

0.1

0.2

Res

idua

ls

0.01 0.1 1 10r [Mpc]

600

800

1000

1200

σ D

M[k

m/s

]

MC2

FLASHHOTGADGETMean Profile

FIG. 9.— Dark matter velocity dispersion profile for the Santa Barbaracluster, 2563 particle simulations.

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

Res

idua

l

0.01 0.1 1 10r [Mpc]

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

v r[100

0 km

/s]

MC2, 128

3

FLASH, 1283

HOT, 1283

GADGET, 1283

HYDRA, 1283

TPM, 1283

Mean Radial Profile

FIG. 10.— Dark matter radial velocity profile for the Santa Barbara cluster,1283 particles. The low-resolution FLASH results are not included in the meanprofile.

-0.1

0

0.1

0.2

Res

idua

ls

0.01 0.1 1 10r [Mpc]

-0.8

-0.6

-0.4

-0.2

0

0.2

v r[100

0 km

/s]

MC2

FLASHHOTGADGETMean Radial Velocity

FIG. 11.— Dark matter radial velocity profile for the Santa Barbara cluster,2563 particle simulations.


without the logarithm), while the scatter in the 2563 simulationsis slightly smaller with 100 km/s. The original SB result hasa scatter of 200 km/s. Overall the agreement of the differentcodes is very satisfactory, however the 2563 particle FLASHsimulation shows a relatively high radial velocity in the innerregion.

Our results for the Santa Barbara cluster comparison for thesix tested codes are very positive. The agreement of all mea-sured quantities across the codes is much better than in theoriginal test. This could be due to several reasons: (i) Everycode was run with exactly the same initial particle positionsand velocities, resulting in a more uniform comparison (usingonly the initial density field can lead to slightly differentinitialconditions since every simulator would have to implement theZel’dovich step to obtain positions and velocities). (ii) We usedthe same analysis code for all simulations, thereby eliminatingany uncertainties introduced by variations in diagnosticsrou-tines, e.g., estimating the center of the cluster, etc. (iii) Most ofthe simulations were run with very similar resolution, varyingfrom 5 kpc to 64 kpc for the high resolution runs, while the res-olutions in the original test varied between 5 kpc and 960 kpc.(iv) The present test used dark-matter only simulations.

5. TESTS WITH THE COSMOLOGICAL CONCORDANCE MODEL

5.1. Test Description

In this Section, we study results from two realistic N-bodycosmological simulations of aΛCDM universe, a “small” 90Mpc box and a “large” 360 Mpc box using 2563 particles.These boxes straddle a representative range of force and massresolutions for state-of-the-art large scale structure simulationsdesigned to study power spectra, halo mass functions, weaklensing, etc. The 90 Mpc box is somewhat small as a repre-sentative simulation due to lack of large-scale power. Never-theless, we chose this volume for code comparison because itallows a relatively high resolution for MC2, which we are usingas a collisionless standard.

The initial linear power spectrum was generated using thefitting formula provided by Klypin & Holtzman (1997) for thetransfer function. This formula is a slight variation of thecom-mon BBKS fit (Bardeen et al. 1986). It includes effects frombaryon suppression but no baryonic oscillations. We use thestandard Zel’dovich approximation (Zel’dovich 1970) to pro-vide the initial particle displacement off a uniform grid and toassign initial particle velocities. The starting redshiftfor bothbox-sizes iszin = 50. In each case, we choose the followingcosmology: Ωm = 0.314 (whereΩm includes cold dark mat-ter and baryons),Ωb = 0.044,ΩΛ = 0.686,H0 = 71 km/s/Mpc,σ8 = 0.84, andn = 0.99. These values are in concordancewith the measurements of cosmological parameters by WMAP(Spergel et al. 2003). Both simulations used 2563 simulationparticles, implying particle masses of 1.918·109 M⊙ for thesmall box and 1.227·1011 M⊙ for the large box.

The analysis of the runs is split into two parts. First, the par-ticle positions and velocities and their two-point functions arestudied directly. We measure velocity distributions, the masspower spectrum, and correlation functions, and compare posi-tion slices from the simulation box directly. Second, halo cata-logs are generated and investigated. Here we compare velocitystatistics, correlation functions, mass functions, and for somespecific mass bins, the positions, masses, and velocities ofse-lected individual halos.

The MC2-simulations are carried out on a 10243 mesh for

both physical box sizes. A detailed convergence study withMC2 will be presented elsewhere (Habib et al. 2004). Forthe test simulations, FLASH’s base grid is chosen to be a2563 mesh, with two levels of mesh refinement leading to aneffective resolution equivalent to a 10243 mesh in high densityregions. The softening length for the other four codes are cho-sen in such a way that sufficient resolution is guaranteed andreasonable performance on a Beowulf cluster is achieved. Theindividual softening lengths used ranged between 10 to 40 kpc.These choices of softening length are consistent with the massresolution set by the number of particles and this is borne out bythe results of Sec. 5.2.5. In Table 3 the force resolutions fromthe different codes and runs are summarized. LCDMs refersto the 90 Mpc box simulation, LCDMb to the 360 Mpc boxsimulation.

5.2. Results

We use the following color coding for the different codes inall comparison plots: MC2 results are shown with black linesand circles, FLASH with turquoise lines and squares, HOTwith blue lines and down-triangles, GADGET with red linesand up-triangles, HYDRA with green lines and left-triangles,and TPM with orange lines and right-triangles. As an arbitraryconvention, all residuals shown are calculated with respect tothe results from GADGET, rather than from an average overthe results from all the codes. This is done mainly in orderto avoid contaminations of averages from lower resolution re-sults or from individual simulations with some other systematicsource of error.

5.2.1. Positions and Velocities of the particles

In order to first compare the codes qualitatively and picturethe ‘cosmic web’ it is helpful to look at the particles directly.In Figs. 12 and 13 we show the results for the six codes. Forthe small box (Fig. 12) we cut out a 1 Mpc thick slice in thexy-plane through the center of the box. The velocities of theparticles are divided into 4 equal bins between 0 and 440 km/s.Each particle is individually colored according to the bin it be-longs to. The agreement of the 6 codes is very impressive, witha visual impression of almost particle by particle agreement (ifone were to zoom in, this would of course not hold in the small-scale virialized regions). Even though the grid codes MC2 andFLASH were run at lower resolution than the other four codes,in this qualitative comparison they are almost indistinguishablefrom the higher resolution runs. The results for the big boxare shown in Fig. 13. Here we have cut out a 3 Mpc thickslice in thexy-plane through the center of the box. Again, eventhough the resolutions of the different simulations are different,the plots are very similar.

5.2.2. Velocity Statistics of the Particles

We now turn to a more detailed comparison of the particlevelocities. We measure the distribution of the particle speeds

v =√

v2x + v2

y + v2z in bins of 25 km/s for the small box and of

30 km/s for the large box. In order to compare the distributionsfrom the different codes we only include velocities between1 km/s and 2500 km/s in the small box and between 1 km/sand 3000 km/s for the big box.

Let us first consider the results for the 90 Mpc box. In Fig. 14the comparison of the velocity distribution for all 6 codes is


FIG. 12.— Comparison of the particle positions and velocities in a 1 Mpc thick slice of the 90 Mpc box. The colors show four (uniform) velocity bins between0 and 440 km/s. Dark blue corresponds to a velocity between 0 and 110 km/s, light blue to a velocity between 110 km/s and 220 km/s, dark green to a velocitybetween 220 km/s and 330 km/s, and red corresponds to the highest velocities between 330 km/s and 440 km/s.


FIG. 13.— Comparison of the particle positions and velocities in a 3 Mpc thick slice of the 360 Mpc box. As in the small box, the velocities are divided into 4equal bins of size 175 km/s with a similar color-coding, the maximum speed being 700 km/s.


TABLE 3

SOFTENING LENGTHS/MESH SPACINGS FOR THEΛCDM RUNS

MC2 FLASH HOT GADGET HYDRA TPMLCDMs, 2563 particles 90 kpc 90 kpc 10 kpc (physical after z=9) 20 kpc 40 kpc 10 kpcLCDMb, 2563 particles 350 kpc 350kpc 20 kpc (physical after z=9) 20 kpc 40kpc 20 kpc

shown. HOT, GADGET, HYDRA, and TPM are almost indis-tinguishable. In the upper panel of the plot the relative resid-ual in comparison to GADGET is shown. The relative devia-tion of HOT, HYDRA and TPM from GADGET is below 3%up to 1500 km/s, while MC2 (on a 10243 mesh) and FLASHdeviate in this region by less than 5%. By combining a 5123

mesh simulation and the 10243 mesh simulation from MC2, itis possible to predict the result for the continuum from the PMsimulations using Richardson extrapolation with linear conver-gence (the order of convergence was found numerically usinga range of grid sizes). This reduces the systematic deviationfrom the higher resolution codes very significantly (∼ 5% up to2000 km/s), as evidenced by the top panel of Fig. 14. The mainsignificance of this result lies in the fact that MC2 is not colli-sional up to the resolution employed in the cosmological tests,thus the Richardson extrapolation to a higher effective resolu-tion does not bring in any artificial heating from collisions. Themuch improved agreement with the higher resolution codes isstrong evidence that, over the velocity range studied, collisionaleffects are subdominant in those codes as well. [Details of theextrapolation methodology and how it may be used to improveresolution for mesh codes will be provided in a forthcoming pa-per (Habib et al. 2004).] The deviation of FLASH from GAD-GET is very similar to the deviation of the 10243 simulationfrom MC2, indicating that most relevant regions were refinedin the FLASH simulation to the highest level. Overall, this firstquantitative comparison may be considered very satisfactory.

Next, we analyze the corresponding results for the 360 Mpcbox. The comparison of all six codes is shown in Fig. 15. Over-all, the results are not as good as for the smaller box. Nev-ertheless, HYDRA and TPM agree with GADGET at∼ 10%over the whole range and∼ 5% up to 1500 km/s. Relative tothese codes, the velocities generated by HOT are slightly loweroverall, however, the agreement is still around 6% over mostof the range. The raw MC2 and FLASH results deviate fromthe higher resolution runs up to 40% for very large velocitiesas the grid resolution is not adequate to resolve high density re-gions properly. Once again, Richardson extrapolation using a5123 and a 10243 mesh succeeds in converting the MC2 resultsto those very similar to the higher-resolution simulations(lessthan 5% difference from GADGET out to 1500 km/s). Notethat at large velocities, systematic deviation trends across thecodes are clearly visible, especially in the large box.

Around the peak of the distribution, MC2 performs slightlybetter than FLASH. This is easy to understand: As velocitiesare highest in the high density regions (e.g., the clumped re-gions in Fig. 13), the effective resolution in these regionscon-trols the accuracy with which the tail of the velocity distributionis determined. Fig. 16 shows a log-density slice from the simu-lation with the AMR-grid superimposed onto it. The high den-sity regions are appropriately refined by FLASH (smaller sub-meshes). However, the lower density regions are treated withpoorer resolution, therefore, we expect good agreement with

a 10243 mesh simulation for the higher velocities, and for thelower velocities, good agreement with the corresponding reso-lution PM simulation. In Fig. 17 we show the relative residualswith respect to GADGET of the 5123 mesh and 10243 meshMC2 simulations, and the AMR-FLASH simulation. In thehigh velocity tail the higher resolution simulation from MC2

and FLASH agree very well, while for the lower velocity re-gion the MC2 5123 mesh simulation agrees better with FLASH(at late times, near z=0, the main box is run by FLASH at aresolution of 5123).

Finally, we compare results forv12(r), the (pair-averaged)relative pairwise streaming velocity considered as a functionof the separationr between two mass tracers. If the tracers aredark matter particles then the pair conservation equation [fol-lowing the standard BBGKY approach (Davis & Peebles 1977;Peebles 1980)], connects thev12(r) field to the two-point corre-lation function of the particle distributionξ(r). Measurementsof v12(r) from peculiar velocity surveys provide useful con-straints onΩm andσ8 (Feldman et al. 2003). While theoreticalpredictions for these measurements are best carried out with thecorresponding velocity statistics for halos, computingv12(r) forparticles and comparing this with the same quantity calculated

0 500 1000 1500 2000 2500v[km/s]

0

2e+05

4e+05

6e+05

8e+05

dn/d

v

MC2 (Richardson)

MC2 (1024

3)

FLASHHOTGADGETHYDRATPM

-0.1

-0.05

0

0.05

0.1

Rel

. Res

idua

ls

FIG. 14.— Particle velocity distribution for all six codes atz= 0, 90 Mpc box.The upper panel shows the relative residuals in comparison with GADGET.


for halos is used to determine halo velocity bias. The pairwisepeculiar velocity dispersionσ12(r) = 〈[v(r) − 〈v(r)〉]2/3〉1/2 isalso an important statistical quantity, connected toΩm and thetwo-point correlation function via the Cosmic Virial Theorem(Peebles 1976a and 1976b).

Results for the two quantities are displayed in Figs. 18 and19 for the small and large boxes respectively. Absolute resid-uals with respect to GADGET are provided for thev12(r) re-sults (sincev12 can vanish), while relative residuals are givenfor σ12(r). The v12 results for the small box show a smallbump in the GADGET results with respect to the other codesat r ≃ 0.5 Mpc, otherwise the results are better than 10%, and,in the actual range of physical interest,r > 1 Mpc, (except forthe FLASH results), the agreement is better than 5%. (See Sec.5.2.4 for a discussion of issues with the time-stepping in GAD-GET.) The results forσ12(r) from the high resolution codes arevery good, with agreement at the few % level achieved, ex-cept on small scales where resolution limitations are manifest.The results from the lower-resolution mesh codes, as expected,show a small 5% suppression inσ12(r) in the region of the peak.(Here we have not attempted extrapolation to improve the re-sults.) Results from the large box forv12 again show a sys-tematic difference between the other high-resolution codes andGADGET at roughly 10% in the region around 1 Mpc. Theresults forσ12(r) for the high-resolution codes are very consis-tent, with better than 5% agreement across the range. The meshcodes (without extrapolation) are at a particular disadvantage inthis test since the peak inσ12(r) occurs relatively close to theirresolution limit.

0 500 1000 1500 2000 2500 3000v[km/s]

0

2e+05

4e+05

6e+05

8e+05

dn/d

v

MC2 (Richardson)

MC2 (1024

3)


-0.3

-0.2

-0.1

0

0.1

Rel

. Res

idua

ls

FIG. 15.— Particle velocity distribution following Fig. 14 forthe 360 Mpcbox. Results from the grid codes deviate at large velocitiesas their resolutionis not adequate to resolve high density regions. Note the small systematicdeviations at high velocities for the codes run a high resolution.

FIG. 16.— Refinement levels from a FLASH-AMR simulation superposedon a partial density slice from the center of the 360 Mpc simulation box. Theactual highest resolution grid scale is 16 times smaller than the smallest boxshown in the figure.

0 500 1000 1500 2000 2500 3000v[km/s]

-0.6

-0.4

-0.2

0

0.2

Rel

ativ

e R

esiu

dals

MC2, 512

3 mesh

MC2, 1024

3 mesh

FLASH, AMR up to 10243 mesh

FIG. 17.— Comparison of the relative residuals for particle velocities fromthe MC2 5123 grid and 10243 grid runs with the FLASH results, 360 Mpc box.At higher velocities, the AMR results match better with the 10243 grid results,while at lower velocities, they agree with the 5123 grid results. See the text fora discussion.


0

0.1

0.2

0.3

Rel

. Res

id. f

or σ 12

0.01 0.1 1 10r[Mpc]

-200

0

200

400

600

800

1000

v 12[k

m/s

] and

σ 12[k

m/s

]

MC2


-20

-10

0

10

20

Abs

. Res

id. f

or v 12

FIG. 18.— Comparison of the relative particle pairwise streaming veloc-ity v12(r) (lower set of curves) and the pairwise peculiar velocity dispersionσ12(r) (upper set) for all six codes atz = 0, 90 Mpc box. At small distances,the GADGET results forσ12(r) deviate from those of the other codes. For adiscussion of this problem, see Sec. 5.2.4.

-0.2

-0.1

0

Rel

. Res

id. f

or σ 12

0.1 1 10 100r[Mpc]

-200

0

200

400

600

800

v 12[k

m/s

] and

σ 12[k

m/s

]

MC2


-40

-20

0

20

40

Abs

. Res

id. f

or v 12

FIG. 19.— Comparison of the relative particle pairwise streaming velocityv12(r) (lower set of curves) and the pairwise peculiar velocity dispersionσ12(r)(upper set) for all six codes atz= 0, 360 Mpc box. The lower resolution of thegrid codes is more apparent in the results forσ12(r).


5.2.3. Power Spectrum of the Particle Density Field

The next quantitative comparison performed for the particlesis the measurement of the power spectrumP(k). The powerspectrum is obtained in the following way: First the densityis calculated with a CIC (Cloud-In-Cell) scheme on a 10243

spatial grid. Then a 10243 FFT is employed to obtain the den-sity in k-space and squared to yield the power spectrum. (Inorder to compensate for the CIC softening we deconvolve thek-space density with the CIC window function.) Finally, thethree-dimensional power spectrum is binned to get the one-dimensional power spectrum. We did not attempt to compen-sate for particle shot noise or use a bigger FFT as the aim hereis to look for systematic differences between codes rather thanactually obtain a more accurateP(k). The power spectrum willbe studied in more detail in later work.

We begin again with the small box, the results being shownin Fig. 20. The lower panel shows the results from the six codesand in addition a power spectrum generated by Richardson ex-trapolation from a 5123 and a 10243 mesh MC2 simulation. Theupper panel displays the relative residuals with respect toGAD-GET. The agreement between the HOT, GADGET, HYDRA,and TPM is very good, better than 10% up tok = 10 Mpc−1.For higherk the HOT and HYDRA simulations have slightlyless power than the GADGET and TPM simulations. The re-sults from the two mesh codes, MC2 and FLASH, are in goodagreement with the other runs up to a few Mpc−1. For higherk,the limited resolution of these two runs leads to larger discrep-ancies, as expected. In these runs, the slightly higher resolutionof MC2 compared to FLASH is again visible. The Richard-son extrapolation result from MC2 differs from the GADGETsimulation by roughly 5% up tok=10 Mpc−1. For largerk thedifference becomes much larger. This is as expected since theextrapolation procedure cannot generate power which was notpresent in the underlying simulations; it can only improve theaccuracy of the results on scales where convergence holds. Dueto this property, it functions as an excellent test of code conver-gence.

Next we compare the results for the large box, as shown inFig. 21. The overall result is very similar to the small box. Overroughly two orders of magnitude ink the high resolution runsfrom HOT, GADGET, HYDRA, and TPM agree better than5%. The TPM simulation agrees with GADGET over the en-tire range to better than 2%. The HYDRA run shows again aslight fall-off of the power withk before the other high reso-lution codes do. The results from HOT appear to be system-atically low in the normalization. The MC2 result shows goodagreement (sub-5%) with GADGET up tok=1 Mpc−1, while theFLASH power spectrum falls off faster already atk=0.4 Mpc−1.This is consistent with the fact that in the large box less of thevolume was refined (14.3%) than in the small box (40.8%). Asseen for the small box, Richardson extrapolation again leads toa large improvement: the difference with GADGET becomessub-5% up tok=4 Mpc−1.

As with the results obtained for the particle velocity statis-tics, the results for the power spectrum are also very satisfac-tory. From the point of view of force resolution, all six codesagree over thek-range for which consistent results are to beexpected: The four high resolution codes agree at the sub-5%level in the main regions of interest, while the the results for thetwo mesh codes are also consistent with the expected resolutionand convergence properties of these codes.

-0.3

-0.2

-0.1

0

Rel

. Res

idua

ls

0.1 1 10k[Mpc

-1]

1

100

10000

P(k

)[M

pc3 ]

MC2 (Richardson)

MC2 (1024

3)


FIG. 20.— Comparison of the mass density power spectra for all six codesat z = 0, 90 Mpc box. The Richardson-extrapolated result from MC2 is alsoshown. The early fall-off of the two grid codes is expected and consistent withthe resolution employed. The high result from GADGET is discussed in Sec.5.2.4.

-0.2

-0.1

0

Rel

. Res

idua

ls

0.1 1 10k[Mpc

-1]

10

100

1000

10000

P(k

)[M

pc3 ]

MC2 (Richardson)

MC2 (1024

3)


FIG. 21.— Mass density power spectrum results for the 360 Mpc box, fol-lowing Fig. 20.


5.2.4. Particle Correlation Function

After comparing the power spectrum we now investigate itsreal-space counterpart, the correlation function generated fromthe particle positions. TheO(N2) calculation is straightforward,though tedious. In principle the correlation function containsthe same information as the power spectrum, however, sincethey were obtained in different ways, comparison against thecorrelation function provides a simple cross-check. The corre-lation functions were generated by randomly choosing 65,536particles from each simulation, and calculating the numberofparticles contained within logarithmically spaced radialbins us-ing a directN2 search.

The results for the small box are shown in Fig. 22. As al-ways, the lower panel shows the results from the six differentcodes while the upper panel displays the relative residualswithrespect to the GADGET simulation. The agreement of all codesis very good for larger, between 200 kpc and 40 Mpc, the devi-ation from GADGET for all other codes being less than 5%. Forr smaller than 200 kpc the limited resolution of the MC2 andFLASH simulations becomes apparent, the correlation func-tion falling off much faster than for the other simulations.Atr ≈ 20 kpc the correlation function from GADGET rises muchmore steeply than from the other simulations. This is an artifactdue to time integration errors: The design of GADGET’s mul-tistep integrator is prone to the build-up of secular integrationerrors which can manifest themselves in effective energy lossesof some particles, which in turn boosts the phase-space den-sity in halo centers. While this can be avoided with finer (andmore expensive) time-stepping, this is not an optimum solution.

-0.6

-0.4

-0.2

0

Rel

. Res

idua

ls

0.01 0.1 1 10r[Mpc]

0.01

0.1

1

10

100

1000

10000

ξ(r

)

MC2


FIG. 22.— Comparison of the particle two-point correlation functions fromthe six codes atz = 0 for the 90 Mpc box. See the text for a discussion of therise in the GADGET results at small distances. The results for the grid codesare consistent with the employed resolution.

-0.6

-0.4

-0.2

0

0.2

Rel

. Res

idua

ls

0.01 0.1 1 10 100r[Mpc]

0.001

0.01

0.1

1

10

100

1000

ξ(r

)

MC2


FIG. 23.— Particle two-point correlation function for the 360 Mpc box,following Fig. 22.

GADGET has been updated 2 years ago to avoid this problem,but this version is not public yet.8 This explains the large rel-ative residuals with respect to GADGET at small scales. Up tothis point HOT, HYDRA, and TPM agree better than 5% withGADGET.

The results for the large box are shown in Fig. 23 and aresimilar to those from the small box. For larger the agree-ment is once again excellent, between 0.8 Mpc and 160 Mpcall codes agree to better than 5%. At smallr, FLASH and MC2

suffer from degraded resolution, and the disagreement betweenthe other codes becomes larger. The correlation function fromHOT is systematically lower in the region 0.1 Mpc< r <5 Mpc,while HYDRA’s correlation function, due to the slightly lowerresolution employed than HOT, GADGET, and TPM, falls offa little earlier than GADGET and TPM. For very smallr thediscrepancy between the codes is at the 10% level, but here thecodes are being pushed too close to their resolution limits.

Overall, as expected, the results for the correlation functionsare very similar to those for the power spectrum. The resolutionlimits of the grid codes MC2 and FLASH are clearly exposed atthe correct scales, with the four high resolution codes agreeingwell with each other until scales close to the resolution limitsare reached.

5.2.5. Halo Mass Function

We now turn from comparing particle position and velocitystatistics to comparing results extracted from halo catalogs. Inmany ways, it is the halo information that is of more direct rel-evance to comparison with cosmological observations, so thesetests are of considerable importance. The halo catalogs were

8We thank Volker Springel for providing this information.


10 12 14log10

(M)[MO]

-6

-5

-4

-3

-2

log 10

(dn/

dlog

10M

)[M

pc3 ]

MC2, b=0.2

MC2, b=0.17

GADGET, b=0.2GADGET, b=0.17

0.6

0.8

1

1.2

1.4

Hal

ofb=

0.17

/Hal

ofb=

0.2

MC2

GADGET

..

FIG. 24.— Comparison of results from two FOF halo finders with linkinglengths ofb = 0.2 andb = 0.17 run on the MC2 and GADGET simulations atz = 0 for the 90 Mpc box. The top panel shows the ratio of theb = 0.17 massfunction and theb = 0.2 mass function for the two codes.

generated using a Friends-Of-Friends (FOF) algorithm (Daviset al. 1985). This algorithm defines groups by linking togetherall pairs of particles with separation less thanbn−1/3, wheren isthe mean particle density. This leads to groups bounded approx-imately by a surface of constant local density,ρ/ρ≈ 3/(2πb3).We have chosen a linking length ofb = 0.2 corresponding toa density thresholdρ/ρ ≈ 60. As discussed in Lacey & Cole1994, for a spherical halo with a density profileρ(r) ∝ r−2,this local density threshold corresponds to a mean overdensity〈ρ〉/ρ≈ 180.

In order to confirm the results obtained with this halo finderwe also analyzed two of the small box simulations (GAD-GET and MC2) with a different FOF halo finder9 with a link-ing length of b = 0.17 corresponding to a density thresholdρ/ρ ≈ 97. (For a discussion on the dependence of the massfunction on the chosen linking length for a FOF halo finder,see, e.g., Lacey & Cole 1994.) We checked that the two FOFhalo finders gave the same results forb = 0.17.

The comparison between the halo mass functions for the twochoices of linking length is shown in Fig. 24. In the lowerpanel the mass functions found with the two halo finders forGADGET and MC2 are shown. In the upper panel we showthe ratio of the two mass function results for MC2 and the ra-tio of the two mass functions found for GADGET. The ratio isthe b = 0.17 mass function divided by theb = 0.2 mass func-tion. The smaller linking length should find fewer halos in the

9We thank Stefan Gottlöber for generating the alternate set of halo catalogs.

small mass region region and tend to “break up” heavier halosin the high mass region. Consequently the ratio should show adip at both extremes (since there are far fewer high mass halos,overall the smaller linking length will lead to fewer halos). Ourresults are very consistent with this expectation and also withthe fact that more MC2 halos are “lost” by theb = 0.17 finderat small masses – due to the lower resolution, these halos aremore diffuse as compared to the GADGET halos.

Discounting the high mass end, there is still a systematicvariation of up to 10% in the two halo mass functions. Thisresult should be kept in mind when comparing the mass func-tion to semi-analytic fits: even the same type of halo finder withslightly different parameter choices can easily affect themassfunction at the several % level. Furthermore, changing the massresolution can also lead to systematic effects on halo masses us-ing the same FOF algorithm. These aspects will be consideredin detail elsewhere (Warren et al. 2005).

Next we analyze the halo catalogs generated with the sameFOF halo finder for all six codes. We start by comparing thenumbers of halos found in the different simulations. The resultsare listed in Table 4. The smallest halos considered contain10particles, leading to a minimum halo mass for the small box of2·1010 M⊙. In this box, the most massive halos have masses ofthe order of 7·1014 M⊙. Allowing 10 particles to constitute ahalo is a somewhat aggressive definition, and we are not argu-ing that this is appropriate for cosmological analysis. However,as a way of comparing codes, the very small halos are the onesmost sensitive to force resolution issues and therefore trackingthem provides a useful diagnostic of resolution-related deficien-

10 12 14log10

(M)[MO]

-6

-5

-4

-3

-2

log 10

(dn/

dlog

10M

)[M

pc3 ]

MC2


-0.4

-0.2

0

0.2

0.4

Rel

. Res

idua

l

.

FIG. 25.— Comparison of the halo mass function for all six codes at z= 0,90 Mpc box. See the text for a discussion of code systematics.


TABLE 4

NUMBERS OF HALOS FOUND IN THE DIFFERENT SIMULATIONS

MC2 FLASH HOT GADGET TPM HYDRALCDMs 49087 halos 32494 halos 54417 halos 55854 halos 34367 halos 54840 halosLCDMb 65795 halos 33969 halos 73357 halos 72980 halos 55967 halos 74264 halos

cies as well as providing clues to other possible problems.Largely due to the generous definition of what constitutes a

halo, the variation in the total number of halos is rather large:For the small box, HOT, GADGET and HYDRA found thelargest number of halos, around 55000, while FLASH and TPMfound almost 40% fewer halos. The number of halos from theMC2 simulation is in between these two extrema, a little closerto the higher numbers. For the large box the agreement of thenumber of identified halos from HOT, GADGET, and HYDRA,around 73000, is much better, but again the TPM simulationleads to almost 20000 fewer halos, and from the FLASH simu-lation, only 33000 halos were identified. The result from MC2

is slightly lower than the results from HOT, GADGET, and HY-DRA but much higher than the halos identified from TPM andFLASH.

We have generated mass functions from the different halocatalogs using a FOF algorithm with linking lengthb= 0.2. Thecomparison of the mass functions is shown in Fig. 25. As men-tioned above, far fewer halos were found in the FLASH andTPM simulations, which becomes very apparent in this figure:At the low mass end of the mass function the results from thesetwo codes are discrepant up to 40%. In contrast, MC2, HOT,GADGET, and HYDRA agree very well, to better than 5%.

In case of the grid codes, the fall-off at the low-end of themass function can be easily connected to the resolution limit.To see this, given a particular halo mass, one computes a corre-sponding scale radius, say, e.g.,r180 and compares this with thegrid resolution,∆. The scale radiusr180 is defined to be suchthat the mean matter density within it is 180 times the meanbackground density (numerically, its value is roughly thatof thevirial radius). Clearly, if∆ > r180, halos in the particular massregion will be suppressed, as the finite force resolution will pre-vent the formation of compact objects on this scale. We haveverified that the threshold for the fall-off in the mass functionfor MC2 is set by the approximate condition∆ ∼ r180 in boththe small and large boxes. One may wonder whether the lackof smaller halos has any effect on the mass function at masseshigher than the resolution threshold set by∆∼ r180. As shownby our results from both the mass function, and the comparisonof individual halo masses below, this is not the case. The parti-cles that would have ended up in smaller halos are not lost, andend up streaming into the larger halos so that there is no system-atic bias in the mass function above the resolution threshold.

The result found for FLASH is also easy to understand. Thebase grid in the FLASH simulation was chosen to be 2563

which was then refined up to 10243, as explained above. Atthe beginning of the simulation this base resolution is not suf-ficient to form small halos. As the simulation proceeds for-ward in time, the resolution increases only where the densitycrosses the AMR threshold, elsewhere the small halos cannotbe recovered. Therefore AMR codes will lead to very good re-sults for the large halos and their properties, since for these themesh will be appropriately refined. But without the implemen-

tation of AMR-specific initial conditions which help to solvethe problem of starting with a pre-refined base grid, the massfunction will be suppressed for small masses. This work is nowin progress (Lukic et al. 2005). The condition∆≃ r180 imposedusing the base 2563 grid correctly predicts the fall-off point inthe mass function for the FLASH simulations. [A similar resultfor the mass function was found by O’Shea et al. (2003), inwhich GADGET was compared to ENZO, an AMR-code de-veloped by Bryan & Norman (1997,1999): The mass functionfrom the ENZO simulation was lower for small masses than theone obtained from GADGET.]

We are unable to explain the fall-off in the mass function forthe TPM simulations (seen in both the small and large boxes)in terms of a simple force resolution argument. It is possiblethat this effect is due to the grid-tree hand-off, but at presentthis remains a speculation.

The results for the large box show trends similar to the smallbox results. The overall agreement of the six codes shown inFig. 26 is better than for the small box. The mass function from

12 13 14 15log10

(M)[MO]

-8

-7

-6

-5

-4

log 10

(dn/

dlog

10M

)[M

pc3 ]

MC2


-0.6

-0.4

-0.2

0

0.2

Rel

. Res

idua

l

.

FIG. 26.— Comparison of the halo mass function for all six codes at z= 0,360 Mpc box. For discussion of code systematics, see text.


the TPM simulation is again lower than the others by roughly40% for the small mass halos. The FLASH result for the smallhalos is up to 60% lower than that from GADGET. As in thesmall box, the condition∆ ≃ r180 determines the mass belowwhich halos are lost. The deviation from GADGET for theother codes is less than 2% over a large range of the mass func-tion, a pleasing result.

5.3. Halo Power Spectrum

The distribution of dark matter halos is biased with respecttothe overall distribution of the dark matter (Mo & White 1996).One way of estimating this bias is to compare the halo powerspectrum with the (linear or nonlinear) power spectrum of theparticle density field. Another method is to compare the re-spective correlation functions (halo-halo vs. particle-particle)(Colin et al. 1999; Kravtsov & Klypin 1999). We discuss boththe halo power spectrum and the correlation function below.

In order to minimize problems with selection effects due tothe differing code force resolutions and possible systematic un-certainties with halo finders, for the spatial analysis of halos weselected only those halos with more than a 100 particles. In thesmall box, this implies a halo mass threshold of 2· 1011 M⊙

which, as comparison with Fig. 25 shows, succeeds in avoid-ing the low halo mass region where MC2 and TPM are some-what deficient. For the larger box, the corresponding halo massthreshold is 1.2 · 1013 M⊙ which also avoids the low-mass re-gion (Fig. 26). For both boxes the FLASH results, because ofthe AMR issues discussed above, continue to have fewer halosuntil a higher threshold is reached. In the small box there are

-0.05

0

0.05

0.1

0.15

0.2

Rel

. Res

idua

ls

0.1 1 10k[Mpc

-1]

1000

10000

P(k

)[M

pc3 ]

MC2

FLASHHOTGADGETHYDRATPMLinear P(k)

FIG. 27.— Comparison of the halo power spectrum for all six codesatz= 0,90 Mpc box. Only halos with more than 100 particles were used.The upperset of curves in the lower panel has not been noise-subtracted while the lowerset of curves was corrected for shot-noise. The dashed line is the linear powerspectrum from the corresponding initial condition realization.

-0.1

0

0.1

0.2

0.3

0.4

Rel

. Res

idua

ls

0.1 1k[Mpc

-1]

100

1000

10000

1e+05

P(k

)[M

pc3 ]

MC2

FLASHHOTGADGETHYDRATPMLinear P(k)

FIG. 28.— Comparison of the halo power spectrum for all six codesatz= 0,360 Mpc box, following Fig. 27. The upward bias of the FLASH result isdiscussed in the text.

roughly 6000 halos for each code, except for FLASH which hadless than 5000.

The halo power spectrum was computed in essentially thesame way as the particle power spectrum in Sec. 5.2.3 by firstgenerating a CIC density field and then using a 5123 FFT. Sincethe number of halos is much smaller than the number of parti-cles, here shot noise subtraction is essential to obtain physi-cally sensible results at small scales. We show both the noise-subtracted and raw halo power spectra in Figs. 27 and 28 for thesmall and large boxes respectively. However, for code compar-ison purposes the direct halo power spectra are more relevantas noise subtraction induces large changes in the power spectra.Therefore, the relative residuals are computed for these quanti-ties.

The results for the small box show a roughly 5% deviation tillk = 10 Mpc−1. Note that this agreement is much better than thecase for the particle power spectrum where the largek regimeis dominated by small-scale particle motions. The case for thelarge box is very similar except for the FLASH results. Thereason why the FLASH results are higher is because, (i) as ear-lier noted, there are fewer halos overall and (ii) the small masshalos are disproportionately absent with a resulting increase inbias.

5.3.1. Halo Correlation Function

The halo correlation functions were computed in the sameway as the particle correlation function in Section 5.2.4. Unlikethe statistics of the raw particle distribution, these correlationfunctions provide a more direct measure of the galaxy distribu-tion and are an essential input in the halo model framework forstudying the clustering of galaxies (for a review, see Cooray &


-0.3

-0.2

-0.1

0

0.1

0.2

0.3

Rel

. Res

idua

ls

1 10r[Mpc]

0.1

1

10

ξ(r

)

MC2


FIG. 29.— Halo two-point correlation function for the 90 Mpc box. Onlyhalos with more than 100 particles are considered, corresponding to a thresholdhalo mass of 2·1011 M⊙.

Sheth 2002).The results for the small box are shown in Fig. 29. For

r ≥ 0.6 Mpc the deviations of the codes from the GADGETresult are smaller than 10%; forr ≥ 1 Mpc the agreement isbetter than 5%. The FLASH result here is slightly worse be-cause of the systematic undercounting of halos above the massthreshold of 2·1011 M⊙. The results for the large box, shownin Fig. 30, are very similar. Forr ≥ 3 Mpc the agreement of thecodes with GADGET is again better than 5%, with the excep-tion of FLASH which disagrees over most of the range at about10% (for the same reason as earlier). The dips inξ(r) at thelargest values ofr in both boxes is a finite volume artifact, whilethe suppression at the smallest values ofr is largely due to thelength scale corresponding to the halo mass cutoffs in the twocases, approximately 0.1−0.2 Mpc for the small box and 0.5−1Mpc for the large box. One expects the two-point function re-sults to lose the suppression on length scales at roughly twicethese values, and the simulation results are consistent with thisexpectation.

5.3.2. Halo Velocity Statistics

The single-point halo velocity statistics for the individual ha-los were computed using the more aggressive definition of 10particles per halo, while the calculations ofv12(r) andσ12(r)used halos with at least 100 particles as described above. Tofa-cilitate direct comparison, the individual velocity distributionswere normalized by dividing by the total number of halos foreach simulation.

In Fig. 31 we present the halo velocity distribution for thesmall box. Aside from the skewed distribution from the FLASHresults due to the missing lower-mass halos, all the other codes

are in reasonably good agreement. We have defined the velocityof a halo as the average velocity taken over all the constituentparticles; consequently, there is an inherent statisticalscatter inthe comparison of the results. Light halos will have a large ve-locity scatter, and as light halos contribute over the entire rangeof velocities, this will lead to scatter in the comparative results.(Note that in this analysis we are accepting halos with only 10particles.) The larger the halo mass, the less the scatter. Fur-thermore, heavier halos tend to have lower velocities and thistrend is easy to spot in the better agreement below and aroundthe peak of the distribution in Fig. 31. Up to 750 km/s the agree-ment is at the±10% level. The results from the large box areshown in Fig. 32.

Another perspective, less sensitive to halo finder vagaries(White 2002), will appear in the later analysis of direct com-parisons of individual halos, given below. For halos with morethan 30 particles which are simultaneously identified in there-sults from all the codes, the velocity scatter in the small box isapproximately∼ 10 km/s or less, with a corresponding valueof ∼ 20 km/s for the larger box. As halos get heavier, there areoften individual cases with relatively significant mass variationdue to the FOF halo finder, and, as expected for these halos, thevelocity can then show less agreement. Nevertheless, for thehalos that are simultaneously found by the FOF finder in all thecodes, the velocity agreement is impressive, better than a fewpercent.

The relative pairwise velocity for the halos and the corre-sponding dispersion is shown for the small and large boxes inFigs. 33 and 34. These results should be compared with the cor-responding data from the individual particle analysis (Figs. 18and 19). The points to note are that, as is well-known,v12(r)

-0.2

-0.1

0

0.1

0.2

Rel

. Res

idua

ls

1 10 100r[Mpc]

0.001

0.01

0.1

1

ξ(r

)

MC2


FIG. 30.— Halo two-point correlation function following Fig. 29 for the360 Mpc box. In this case the lower halo mass cutoff is 1.2 ·1013 M⊙.


0 500 1000v[km/s]

0.0001

0.001

0.01

dn/d

v/nh

alo

MC2


-0.8

-0.4

0

0.4

0.8

Rel

. Res

idua

ls

FIG. 31.— Comparison of the halo velocity distribution for all six codes atz= 0, 90 Mpc box. This comparison uses all halos, i.e., halos with a minimumnumber of 10 particles.

0 500 1000v[km/s]

0.0001

0.001

0.01

dn/d

v/nh

alo

MC2


-0.2

0

0.2

0.4

0.6

0.8

Rel

. Res

idua

ls

FIG. 32.— Comparison of the halo velocity distribution for all six codes atz= 0, 360 Mpc box, following Fig. 31.

is relatively insensitive as to whether particles or halos are con-sidered, whereas this is not true ofσ12(r), which is significantlylower for the halos.

Because of the halo sizes corresponding to the lower masscutoffs, the smallest pair-distance scale at which the small boxresults should begin to make sense isr ∼ 0.5 Mpc, with a cor-responding condition,r ∼ 2 Mpc for the larger box. As an in-spection of Figs. 33 and 34 demonstrates, it is only after theseconditions are satisfied that the code results converge, as to beexpected. The results from the large box are substantially betterthan the results from the small box. This is because the haloshere have much larger masses (by a factor of 60) and are takingpart in a smoother regime of the flow. Beyondr ∼ 2 Mpc, thecodes give consistent results at the 5% level forσ12(r), and, ex-cept for FLASH, also forv12(r). For the lower mass halos in thesmaller box, the situation is not as good. At the lower end ofthe acceptabler range, the error for both quantities is∼ 20%,improving to better than 5% at a separation of a few Mpc. Fromthe point of view of comparison to observations this is stillquiteacceptable, as present observational results have far worse er-rors and statistics limitations (Feldman et al. 2003).

-0.4

0

0.4

Rel

. Res

id. f

or σ 12

1 10r[Mpc]

-200

-100

0

100

200

300

400

v 12[k

m/s

] and

σ12

[km

/s]

MC2


-0.8

-0.4

0

0.4

Rel

. Res

id. f

or v 12

FIG. 33.— Relative mean pairwise velocityv12(r) (lower set of curves)and the velocity dispersionσ12(r) (upper set) from halos with more than 100particles (Mhalo > 2 ·1011 M⊙) for the 90 Mpc box.


-0.05

0

0.05

Rel

. Res

id. f

or σ 12

1 10 100r[Mpc]

-400

-200

0

200

400

v 12[k

m/s

] and

σ12

,rm

s[km

/s]

MC2


-0.1

0

0.1

Rel

. Res

id. f

or v 12

FIG. 34.— Results from the 360 Mpc box following Fig. 33. The lower halomass cutoff here is 1.2 ·1013 M⊙.

5.3.3. Direct Comparison of Positions, Masses, and Velocitiesof Individual Halos

In this section we analyze and compare results for positions,masses, and velocities for specific halos directly. In contrastto the individual particles – for which such an exercise wouldmake little sense in higher density regions – halos may be con-sidered, in an informal sense, to be the fundamental stable massunits of cosmological N-body simulations. Thus, to the extentthat this is true, we not only expect codes to agree on halo statis-tics but also on individual halo properties such as the individualhalo masses, positions, and velocities, as well as individual halostatistics describing the internal structure of the halos.This lastpart has already been considered in our study of the Santa Bar-bara cluster where the box size and mass resolution are roughlysimilar to the small boxΛCDM simulation.

We have chosen three different mass bins selecting in turnthe very heaviest halos from the simulations, medium sized ha-los, and small mass halos in such a way that every mass bincontains roughly 40 halos. For the lighter mass bins we alsoemploy a spatial cut, i.e., restrict ourselves to only a fractionof the simulation volume in order to keep the numbers of halosmanageable. Details of the chosen cuts are given below.

First we investigate the heaviest halos in the small box. Weidentify all halos above a mass of 5· 1013 M⊙ which corre-sponds to 25000 particles per halo. By imposing this mass

bound we find between 38 and 39 halos in the six different sim-ulations. Next we compare the position for every single halo.The center of each halo as defined as the particle with the largestnumber of friends (and thus the highest density smoothed overthe scale of the linking length). We allow for a discrepancy of0.5% in the position (which translates to 0.5 Mpc) inx, y, andzdirections for identifying the same halo in all six simulations.In Fig. 35 we show the halos found in this way with a black star.(All halos are projected onto thexy-plane.) Among the roughly39 halos found in every simulation, 36 halos (92%) were effec-tively at the same position in all six simulations. In addition, wealso display the halos which arenot found in every simulation.

Next, for the halos which are identified to be at the sameposition in all simulations, we compute the average mass andvelocity and label every halo with a labeling index. Fig. 36shows the average mass at each index and Fig. 37 shows theaverage velocity at each index. In addition, we have calculatedthe deviation of every halo mass and velocity from the overallaverage:δmi = |mi −〈m〉| andδvi = |vi −〈v〉|, i = 1, ... The averageof these deviations:16

∑

j=1,6δmj and 16

∑

j=1,6δv j describing thescatter from the average are displayed as error bars; if no errorbar is present the error bar is smaller than the symbol itself; wefind that the scatter for most halos is smaller than 2% for massesand velocities. The agreement in masses and in velocities isexcellent; the only error bars distinguishable from the symbolsare still much smaller than the naive expected statistical scatter.

The medium mass halos are analyzed in the same way. Themass slice in this case is chosen to be between 3·1012 M⊙ and2 ·1013 M⊙ which corresponds to roughly 1500 to 10000 par-ticles. In order to obtain roughly 40 halos, a spatial cut is nec-essary. The region analyzed in this case is a 10 Mpc slice inzfrom z= 40 Mpc toz= 50 Mpc. In Fig. 38 the identical halosfrom all six simulations are marked by black stars and all ad-ditionally found halos from all codes are shown. In this mass

0 20 40 60 80x[Mpc]

0

20

40

60

80

y[M

pc]

MC2

FLASHHOTGADGETHYDRATPMCommon

FIG. 35.— Comparison of the positions of the heaviest halos (Mhalo >5·1013 M⊙) found in the six simulations atz= 0 in the 90 Mpc box. Black starsmark halos which were found in all simulations at the same position with lessthan 0.5% deviation. In order to visually separate the different code symbols,the halos from the MC2 simulation were moved by 0.5 Mpc in thex direction,the HYDRA halos by -0.5 Mpc in thex direction, the FLASH halos by 0.5 Mpcin they direction, and the TPM halos by -0.5 Mpc in they direction.


__

0 10 20 30Index

0

20000

40000

60000

80000

Mas

s [1

010 M

O] ·

FIG. 36.— Comparison of the masses of the heaviest halos identified in allsix codes atz= 0, 90 Mpc box. Thex-axis in the figure represents an arbitraryindex label for each halo. The error bar describes the scatter of all codes fromthe average (see text).

bin, 26 out of 39 halos (66%) are at the same positions in allsix codes. As carried out for the heaviest halos, we calculatethe average mass and velocity for each of the 26 identified ha-los and measure the scatter. The results are shown in Figs. 39and 40. If no errorbar is shown, the scatter in mass and veloc-ity is for most halos again below 2%. The masses agree verywell, and only two halos show a small amount of scatter aroundthe average. The result for the velocities is not as impressive;nevertheless, only five of the halos show significant scatter.

Finally, we analyze a subset of the lightest halos in the simu-lations. While our halo catalog contains halos identified atthelevel of 10 particles per halo, for this test we only considerhaloswith more than 30 particles as this is often the lowest numberused in cosmological simulations. The mass range in this caseis 6·1010 M⊙ to 6·1011 M⊙ which translates to 30− 300 parti-cles. In addition we restrict ourselves to the inner cube between40 Mpc and 53 Mpc in thex, y, andz directions. Again, weidentify all halos which have the same positions in the six sim-ulations allowing for 0.5% deviation. We find roughly 40 halosin this volume and mass bin in the different codes. All codesagree on 15 halos; since the FLASH results are known to bedeficient in low-mass halos, a check without FLASH revealsagreement on 25 halos (Fig. 41). Nine of the missing halos areall at the low-mass end (Fig. 42).

The agreement in the low-mass bin is – excluding the FLASHresults – much the same as in the medium mass case. The av-erage masses and their scatter from the 25 identified halos areshown in Fig. 42. Here the scatter, though still small, is no-ticeably bigger than in the higher-mass bins. The halo velocitydistribution shows excellent consistency for 15 halos (outofthe heaviest 16 halos) with significantly more scatter for the re-maining halos (Fig. 43). If the velocity scatter is not showninFig. 43, it is smaller than 2%.

Overall the results of the direct halo comparison for the smallbox are very good. Most halos are found at identical placesand with very close masses and velocities in all the simulations.(Only in the smallest mass bin were some halos lost in the lowerresolution FLASH simulation.)

__

__

0 10 20 30Index

0

200

400

600

800

v[km

/s]

FIG. 37.— Comparison of the velocities of the heaviest halos identified in allsix codes atz= 0, 90 Mpc box. Thex-axis shows the same index as in Fig. 36.The error bar describes the scatter of all codes from the average.

Next, the large box results are analyzed in a similar way. Asbefore, we begin by considering the heaviest halos from the sixsimulations. All halos above a mass of 7.32· 1014 M⊙ corre-sponding to roughly 6000 particles are considered. The heaviesthalo found has a mass of almost 2.8 ·1015 M⊙(≈ 22000 parti-cles). Overall 38 different halos were identified in this mass bin.From these 38 halos, 27 (71%) are at the same position in allsix simulations within 0.5% corresponding to roughly 1.5 Mpc.The positions of the halos are shown in Fig. 44, black starsagain showing the 27 halos found at the same position in all sixcodes. The halos which are not identified in all the codes arestill found in most of the codes or in all the codes with a slightlylarger difference in the position with only two exceptions.Re-

0 20 40 60 80x[Mpc]

0

20

40

60

80

y[M

pc]

MC2


FIG. 38.— Halo positions from all six codes following the conventions ofFig. 35, but for halos containing 1500 to 10000 particles. Again, to improvevisibility the halos from the different codes were shifted by ±0.5 Mpc.


__

__

0 5 10 15 20 25Index

400

600

800

1000

1200

1400

1600

Mas

s [1

010 M

O] ·

FIG. 39.— Comparison of halo masses from all six codes for halos contain-ing 1500 to 10000 particles, 90 Mpc box.

_

_

__

__

_

_

_

_

_

_

0 5 10 15 20 25Index

0

100

200

300

400

500

600

v[km

/s]

FIG. 40.— Comparison of halo velocities for halos containing 1500 to 10000particles, 90 Mpc box.

markably, all the identified 27 halos are the 27 heaviest halos,i.e., if we would have chosen the mass cut slightly higher theagreement would have been 100%. Next we analyze the massesand velocities of the 27 common halos, the results being shownin Figs. 46 and 47. While the scatter in mass is negligible formost halos (below 3%), halo 2, halo 3, and halo 11 show a scat-ter between roughly 5 to 10%, still within expected statisticalerrors. The mass of halo 19 varies significantly in the differentsimulations, with the scatter lying at 20%.

In order to understand this scatter, we have analyzed halo 19in more detail: in Fig. 44 the red marker indicates the positionof this specific halo. Halo 2 is located very close to halo 19.We show halo 19 from the MC2 simulation (which has a rel-atively large mass of 1.36· 1015 M⊙) and from the GADGETsimulation (with a lower mass of 9.42· 1014 M⊙) in Fig. 45.The discrepancy is immediately clear: While in the GADGETsimulation, a more or less spherical halo was found by the FOFhalo finder, in the MC2 simulation some particles close to the

host halo built a bridge to a smaller satellite halo below thebighalo. This is a well-known problem with FOF halo finders (seee.g. Gelb & Bertschinger 1994; Summers et al 1995). Due tothe choice of a specific linking length, two halos which wouldbe identified by eye as separate can be identified as being onehalo if “bridging” particles between the halos are very closeto each other. Spherical density algorithms are an alternative toFOF halo finders (see e.g. Lacey & Cole 1994), but this methodalso does not identify satellite halos separately. In the last sev-eral years, improved halo finding algorithms have appeared,es-pecially targeted to finding subhalos in high resolution simula-tions. These methods are based on hierarchical FOF and bounddensity maxima algorithms (see e.g. Klypin et al. 1999). Sincein this paper we are mainly interested in medium resolution re-sults, a basic FOF halo finder suffices. Nevertheless, the readershould keep in mind that results as found for halo 19 can occurwith a halo finding method which is insensitive to small scalestructure. Indeed, the minor code discrepancies found so farfor halos could easily result from the nondefinitiveness of FOFhalo-finding.

The result for the velocities (Fig. 47) is very similar to theresults for the masses: six halos show significant scatter inthevelocities, three of them also displaying scatter in the mass. Forall six halos the scatter is around 10%, therefore below the naivestatistical error. For all other halos the scatter is smaller than3%. Overall the agreement of the heaviest halos in position,mass, and velocity is rather good.

The investigation of the medium mass halos begins by choos-ing a mass slice between 3.682·1014 M⊙ and 6.137·1014 M⊙

corresponding to 3000− 5000 particles per halo and a spatialcut in z between 135 Mpc and 325 Mpc. Altogether 46 differ-ent halos are identified in the different simulations. MC2 has 32halos in this mass and spatial bin, FLASH has 38, HOT has 32,GADGET has the lowest number of halos in this bin with 30,HYDRA has 31, and in the TPM simulation 34 halos are found.We again ask which of these halos are at the same position if we

40 45 50x[Mpc]

40

45

50

y[M

pc]

MC2

FLASHHOTGADGETHYDRATPMCommonCommon, no FLASH

FIG. 41.— Positions of halos, following Fig. 35, for halos containing 30to 300 particles, 90 Mpc box. The additional points marked with crosses arehalos found by all the codes, excluding FLASH. In this plot, the halos fromMC2, FLASH, HYDRA, and TPM were shifted by only 0.2 Mpc in the fourdifferent directions.


__

__ _

___

__

_

__

_ __ _

___

__

__

_

_

_

_

__

_

_

_

__

__

___ _

_ __

__

__ _

_

0 5 10 15 20 25Index

5

10

15

20

25

30

35

40

45

Mas

s [1

010 M

O] ·

FIG. 42.— Comparison of halo masses for halos containing 30 to 300 parti-cles, 90 Mpc box. From halo 16 onwards, the FLASH results werenot takeninto account.

_

_

_

_

_

__

_

_

_

_

_

_

_

_

__

_ _

_

0 5 10 15 20 25Index

100

150

200

250

v[km

/s]

FIG. 43.— Comparison of halo velocities for halos containing 30to 300particles, 90 Mpc box. From halo 16 onwards, the FLASH results were nottaken into account.

allow for an error of 0.5% and this time find only 20 halos. InFig. 48 these 20 halos are shown with black stars and the rest ofthe halos colored according to the simulation they are foundin.In this mass bin, more halos are identified only in single simu-lations than is the case for the very heavy halos: This could bedue to the fact that on the edges of the mass and spatial cuts,halos from some simulations are getting lost. Next we com-pare the masses and velocities of the 20 halos common to allsix simulations. The results are shown in Figs. 49 and 50. Thelargest scatter in the mass is around 10%, still of the expectedstatistical fluctuation. If no errorbar is displayed, the scatter iswell below 2%. More medium mass halos show a measurablescatter (half of them) than for the heaviest halos (15%). Thescatter of the velocities is very small (below 2% if no errorbaris shown), the agreement of all codes being excellent. Overall,the agreement for the medium mass halos is not as good as for

the heaviest halos but still satisfactory.The last mass range we consider consists of halos in the mass

range between 3.682·1012 M⊙ and 3.68·1013 M⊙, correspond-ing to 30− 300 particles. We cut out an inner cube of the simu-lation withx, y, andzbetween 158 Mpc and 202 Mpc. Overall,43 different halos were found in all six simulations. The MC2

simulation has 25 halos in this bin, FLASH only 15, HOT 27,GADGET 35, HYDRA 33, and TPM 35. From these 43 halos,10 are found in all simulations, again allowing for 0.5% of spa-tial deviation. Fig. 51 shows these 10 halos with a black staraswell as all other halos. This number appears to be very smallbut is mainly due to the fact that the resolution of FLASH isnot sufficient to resolve very small halos (as observed and dis-cussed already for the mass function). The scatter in the massesand velocities for the 10 halos is shown in Figs. 52 and 53. Theresult here is very good, and all halos agree within expectedstatistical fluctuations.

In summary, the results for the direct halo comparison forthe large box are very consistent with our expectations. Thevery heavy halos show excellent agreement across all the codes,while for the very small halos the effects of force resolutions forthe different codes become apparent and the results are not asgood. Nevertheless, given the fact that the resolution of the twomesh codes MC2 and FLASH is an order of magnitude worsethan of the other codes in these tests, the agreement for eventhesmallest halos is surprisingly good.

6. DISCUSSION AND CONCLUSIONS

In this paper we have reported the results from a detailedcode comparison study with 6 codes for different test problems.For all tests the codes were given exactly the same initial condi-tions and the analysis of the simulation results was carriedoutin an identical fashion. This ensured that we only analyzed de-viations arising from differences in the algorithms and their im-

0 100 200 300x[Mpc]

0

100

200

300

y[M

pc]

MC2


FIG. 44.— Comparison of the positions of the heaviest halos (Mhalo >7.32 · 1014 M⊙) found in the six simulations atz = 0 in the 360 Mpc box.Black stars mark halos which were found in all simulations atthe same positionwith less than 0.5% deviation. To improve visibility of the code symbols,the remaining halos from MC2 had their positions shifted by 2 Mpc inx, theHYDRA halos by -2 Mpc inx, the FLASH halos by 2 Mpc iny, and the TPMhalos by -2 Mpc in they direction; the positions of the halos from GADGETand HOT were not changed.


FIG. 45.— Halo 19 from the MC2 (red crosses) and GADGET simulations(black triangles). The FOF halo finder has bridged two particle “lumps” intoone halo for the MC2 simulation, but not for the GADGET simulation.

_

_

__

_

_

_

_

0 5 10 15 20 25Index

1e+05

1.5e+05

2e+05

2.5e+05

Mas

s [1

010 M

O] ·

FIG. 46.— Comparison of the masses of the heaviest halos identified in allsix codes atz= 0, 360 Mpc box. Thex-axis represents an arbitrary index labelfor each halo. The error bar describes the scatter of all codes from the average.

__

__

__

_

_

__

__

0 5 10 15 20 25Index

0

200

400

600

800

v[km

/s]

FIG. 47.— Comparison of the velocities of the heaviest halos identified inall six codes atz = 0, 360 Mpc box. Thex-axis shows the same index as inFig. 46.

0 100 200 300x[Mpc]

0

100

200

300

y[M

pc]

MC2


FIG. 48.— Comparison of halo positions, following Fig. 44, for halos con-taining 3000 to 5000 particles, 360 Mpc box. As for the largest halos, thepositions of the colored halos were shifted by±2 Mpc to improve visibility.


__

_

_

__ _

_ __

_

_

__

__

_

_

__

0 5 10 15 20Index

35000

40000

45000

50000

55000

60000

Mas

s [1

010 M

O] ·

FIG. 49.— Comparison of halo masses for halos containing 3000 to5000particles, 360 Mpc box.

__

__

__

0 5 10 15 20Index

0

200

400

600

800

v[km

/s]

FIG. 50.— Comparison of halo velocities for halos containing 3000 to 5000particles, 360 Mpc box.

160 170 180 190 200x[Mpc]

160

170

180

190

200

y[M

pc]

MC2


FIG. 51.— Comparison of halo positions, following Fig. 44, for halos con-taining 30 to 300 particles, 360 Mpc box. The code symbols were shiftedby 0.3 Mpc for MC2, FLASH, HYDRA, and TPM in the four directions toimprove visibility.

_

_

_

_ _

_

_

_ __ _

_

__

__ __

__

1 2 3 4 5 6 7 8 9 10Index

500

1000

1500

2000

2500

3000

Mas

s [1

010 M

O] ·

FIG. 52.— Halo masses for halos containing 30 to 300 particles.


__

__

__

__

__

__

__

____

__

1 2 3 4 5 6 7 8 9 10Index

100

200

300

400

500

600

v[km

/s]

FIG. 53.— Comparison of halo velocities for halos containing 30to 300particles, 360 Mpc box.

plementations in simulating the evolution of dark matter tracerparticles. We restricted ourselves to medium resolution simu-lations, resolutions between∼ 10-100 kpc being sufficient formany applications. The codes investigated herein representeda variety of algorithms, including a plain PM-code, an AMR-code, tree-codes, a tree-PM code, and an AP3M code. Fourof the codes are publicly available (FLASH, GADGET, HY-DRA, TPM), and three were largely developed by us (MC2,HOT, FLASH). We analyzed particle and halo statistics in con-siderable detail. All results (particle and halo catalogs)and theinitial conditions are publicly available: We encourage otherresearchers to test their codes against our results and to devisenew comparisons.

The tests employed force and mass resolutions typical ofmodern state-of-the-art codes used for analyzing gravitationalclustering in cosmology. Overall, the results obtained arethefollowing: the codes agreed to better than 5% in some tests andat the approximately 10% level for others. Larger systematicdisagreements were usually due to understandable causes suchas insufficient force resolution. While the purpose of this paperis to present a broad suite of results, certain specific observa-tions can and should be investigated in more detail, e.g., the re-sults for the power spectrum, where the “only” 5% agreementmay already provide cause for some concern.

We began our series of investigations with the Zel’dovichpancake collapse test. The PM code MC2 passed this test veryeasily even with relatively high force resolution while theothercodes (including the AMR-code at the same force resolution)could not track the caustics forming in the nonlinear regime.We concluded that the reason for this failure is the inability ofthe high-resolution codes to maintain the planar symmetry ofthe problem and not because of unphysical collisions. Testswith cosmological initial conditions showed that the failure ofthe pancake test from the non-PM codes does not lead to sig-nificant errors in these more realistic situations. The encourag-ing agreement of Richardson extrapolation-improved MC2 andGADGET results for the particle velocity distribution is alsostrong evidence against the presence of unphysical collisional-ity.

Next we investigated the dark matter component of the SantaBarbara cluster simulation running all codes with identical par-ticle initial conditions, obtaining much better agreementbe-tween the different codes than in the original paper. Two otherreasons that contributed to this were that we analyzed all simu-lations in the same way, and the resolution range of the simula-tions did not vary as much as in the original paper.

Finally, we simulated aΛCDM cosmology using two boxes,one being 90 Mpc on a side, the other, 360 Mpc. Particle statis-tics (including velocity statistics, power spectra, and correlationfunctions) agreed well, the lower resolution of the mesh codesbecoming apparent, as expected, in the power spectrum and thecorrelation functions. The agreement for halo statistics was fur-ther improved, with even the plain PM-code MC2 obtaining re-sults very close to those from the high-resolution codes. OnlyFLASH and TPM deviated from the other codes, for the mostpart with known underlying causes. The results for the halostatistics are particularly significant since for many problems(generating mock catalogs, evolution of the halo mass function,etc.), cosmologically relevant information resides more in thespatial and mass distribution of halos and their internal struc-ture, and less in the statistics of the underlying tracer particles.

While our findings are in general satisfying – the codesshowed almost no unexpected or unexplainable behavior and,despite the use of differing algorithms, agreed relativelyclosely– in the coming age of “precision cosmology” one has to askwhether the level of agreement found here is sufficient for ap-plication to certain next-generation observations and, ifnot,where improvements are needed. As one example, weak grav-itational lensing observations promise to deliver measurementsof the mass power spectrum to an accuracy of 1% (Huterer &Takada 2004). At this level one may worry about feedback ef-fects from baryons (White 2004; Zhan & Knox 2004) and neu-trinos (Abazajian et al. 2004) as well as intrinsic code errors inmodeling the distribution of dark matter (including the imple-mentation of initial conditions, not addressed here).

As a starting point, the present agreement over a broad rangeof tests is gratifying, nevertheless, the lack of a rigorousquan-tification of error for gravitational N-body solvers is a seriousbarrier to future progress. As error control requirements be-come more severe, the need for such a theory becomes furthermanifest. In addition, as more (uncontrolled) physics is added,and subgrid modeling incorporated as an essential part of thesimulations, it becomes ever harder to extract error-controlledresults. Finally, in order to be useful, numerical results mustbe connected to observations. Often this can be done only indi-rectly and rather imprecisely, independent of the quality of theobservations themselves.

Thus, to constrain the cumulative error from all links in thesimulation/observation chain to less than 1%, the developmentof a multi-step error-control methodology is necessary. Itispossible that for some applications this will remain a hopelesstask, but for others it may be viable. Although the present paperin no way pretends to address the global problem, we hope thatit contains useful hints for taking some of the initial steps.

S.H., K.H., and M.S.W. acknowledge support from the De-partment of Energy via the LDRD program of Los Alamos Na-tional Laboratory. P.M.R. acknowledges support from the Uni-versity of Illinois at Urbana-Champaign and the National Cen-ter for Supercomputing Applications (NCSA). P.M.R., K.H.,and S.H. acknowledge the hospitality of the Aspen Center for


Physics where part of this work was carried out. The calcula-tions described herein were performed using the computationalresources of NCSA, NERSC, and Los Alamos National Lab-oratory. A special acknowledgement is due to supercomput-ing time awarded to us under the LANL Institutional Com-puting Initiative. Development of FLASH was supported byDOE under grant number B341495 to the Center for Astro-physical Thermonuclear Flashes at the University of Chicago.We wish to thank our many colleagues who have developedcosmological simulation and diagnostic tools and have madethe tools and the results available for public use. We are in-debted to Kev Abazajian, Gus Evrard, Nick Gnedin, StefanGottlöber, Daniel Holz, Gerard Jungman, Anatoly Klypin, An-drey Kravtsov, Adam Lidz, Zarija Lukic, Adrian Melott, Ue-liPen, James Quirk, Robert Ryne, Sergei Shandarin, Ravi Sheth,Volker Springel, Martin White, and Yongzhong Xu for discus-sions and helpful advice.

REFERENCES

Abazajian, K., Switzer, E.R., Dodelson, S., Heitmann, K., &Habib, S. 2004,astro-ph/0411552.

Anninos, W.Y. & Norman, M.L. 1994, ApJ, 429, 434.Bardeen, J.M., Bond, J.R., Kaiser, N., & Szalay, A.S. 1986, ApJ, 304, 15.Bertschinger, E. 1998, ARAA, 36, 599.Binney, J. 2004. MNRAS, 350, 939.Birdsall, C.K. & Langdon, A.B. 1991Plasma Physics via Computer

Simulation(IOP).Bode, P., Ostriker, J.P., & Xu, G. 2000, ApJS, 128, 561.Bryan, G. L. & Norman, M. L. 1997, inComputational Astrophysics; 12th

Kingston Meeting on Theoretical Astrophysics, proceedings of meeting heldin Halifax; Nova Scotia; Canada October 17 – 19; 1996, ed. D. A. Clarke &M. Fall (ASP Conference Series # 123).

Bryan, G. L. & Norman, M. L. 1999, in Workshop onStructured AdaptiveMesh Refinement Grid Methods, ed. N. Chrisochoides (IMA Volumes inMathematics No. 117), 165.

Calder, A.C., et al. 2000, in Proc. Supercomputing 2000.Calder, A.C., et al. 2002, ApJS, 143, 201.Colella, P., & Woodward, P. 1984, J. Comp. Phys., 54, 174.Colin, P., Klypin, A.A., Kravtsov, A.V. & Khokhlov, A.M. 1999, ApJ, 523, 32.Cooray, A., & Sheth, R.K. 2002, Phys. Rep., 372, 1.Couchman, H.M.P. 1999, J. Comp. App. Math., 109, 373.Couchman, H.M.P., Thomas, P.A. & Pearce, F.R. 1995, ApJ, 452, 797.Davis, M., Efstathiou, G., Frenk, C.S. & White, S.D.M. 1985,ApJ, 292, 371.Davis, M. and Peebles, P.J.E. 1977, ApJS, 34, 425.Efstathiou, G., Davis, M., Frenk, C.S., & White, S.D.M. 1985, ApJS, 57, 241.de Zeeuw, D. & Powell, K.G. 1993, JCP, 104, 56.Feldman, H., et al. 2003, ApJ, 596, L131.Frenk, C.S., et al. 1999, ApJ, 525, 554.Fryer, C.L., & Warren, M.S. 2002, ApJ, 574, L65.Fryxell, B., et al. 2000, ApJS, 131, 273.Gelb, J.M. & Bertschinger, E. 1994, ApJ, 436, 467.Habib, S., et al. 2004, in preparation.Hockney, R.W., & Eastwood, J.W. 1989Computer Simulation Using Particles

(Adam Hilger, New York).Huterer, D. & Takada, M. 2004, astro-ph/0412142..Klypin, A.A. & Shandarin, S.F. 1983, MNRAS, 204, 891.Klypin, A.A. & Holtzman, J. 1997, astro-ph/9712217.Klypin, A.A., Gottlöber, S, Kravtsov, A.V., & Khokhlov, A.M. 1999, ApJ, 516,

530.Kravtsov, A.V. & Klypin, A.A. 1999, ApJ, 520, 437.Lacey, C. & Cole, S. 1994, MNRAS, 271, 676.Lukic, Z., et al. 2005, in preparation.MacNeice, P., Olson, K.M., Mobarry, C., de Fainchtein, R., &Packer, C. 2000,

CPC, 126, 330.Melott, A.L. 1983, ApJ, 264, 59.Melott, A.L., Shandarin, S.F., Splinter R.J., & Suto Y. 1997, ApJ, 479, L79.Mo, H.J. & White, S.D.M. 1996, MNRAS, 282, 347.Navarro, J.F., Frenk C.S., & White, S.D.M. 1995, MNRAS, 267,401.O’Shea, B.W., Nagamine, K., Springel, V., Hernquist, L., Norman, M.L., astro-

ph/0312651.Peebles, P.J.E. 1976a, Ap&SpS, 45, 3.Peebles, P.J.E. 1976b, ApJ, 205, L109.Peebles, P.J.E. 1980,The Large-Scale Structure of the Universe(Princeton

University Press, Princeton).Power, C., Navarro, J.F., Jenkins, A., Frenk, C.S., White, S.D.M., Springel, V.,

Stadel, J., & Quinn, T. 2003, MNRAS, 338, 14.

Qiang, J., Ryne, R.D., Habib. S., and Decyk, V. 2000, J. Comp.Phys., 163, 434.Quirk, J.J. 1991, PhD thesis, Cranfield Institute of Technology.Refregier, A., Massey, R., Rhodes, J., Ellis, R., Albert, J., Bacon, D.,

Bernstein, G., McKay, T., & Perlmutter, S. 2004, AJ, 127, 3102.Ricker, P.M., et al. 2004, in preparation.Rosner, R., et al. 2000, CiSE, 3, 22.Ryne, R.D., Habib, S., Qiang, J., Ko, K., Li, Z., McCandless,B., Mi, W., Ng, C.-

K., Saparov, M., Srinivas, V., Sun, Y., Zhan, X., Decyk, V., &Golub, G.1998, The US DOE Grand Challenge in Computational Accelerator Physics,Proceedings LINAC98 (Chicago, IL).

Sahni, V. & Coles, P. 1995, Phys. Repts., 262, 2.Salmon, J.K. & Warren, M.S. 1994, J. Comp. Phys., 111, 136.Sellwood, J.A. 1987, ARAA, 25, 151.Shandarin, S., & Zeldovich, Ya.B. 1989, Rev. Mod. Phys., 61,185.Spergel, D.N., et al. 2003, ApJS, 148, 175.Springel, V., Yoshida, N., & White, S.D.M. 2001, New Astronomy, 6, 79.Summers, F.J., Davis, M. & Evrard, A.E. 1995, ApJ, 454, 1.Warren, M.S. & Salmon, J.K. 1993, inSupercomputing ’93, 12, Los Alamitos

IEEE Comp. Soc.Warren, M.S, Fryer, C.L., & Goda, M.P. 2003, inProceedings of the ACM/IEEE

SC2003 Conference(ACM Press, New York).Warren, M.S., et al. 2005, in preparation.White, M.J., 2002, ApJS, 143, 241.White, M.J., 2004, astro-ph/0405593.Xu, G. 1995, ApJS, 98, 355.Zel’dovich, Y.B. 1970, A&A, 5, 84.Zhan, H. & Knox, L., 2004, astro-ph/0409198.

40 45 50x[Mpc]

40

45

50

y[M

pc]

MC2


Documents

DRAFT VERSION FEBRUARY - arxiv.org · Draft version February 2, 2008 ABSTRACT ... Having ﬁrst made their appearance in plasma physics , ... (Melott et al. 1997). We