Millennium Data Dissemination

databases and other services

2012-12-17 Millennium Workshop 1

Raw data:Particles

FOF groups and Subhalos

Density fields

Subhalo merger trees

Synthetic galaxies (SAM)

Mock catalogues

Mock images

2012-12-17 Millennium Workshop 2

SUBFIND Tree Builder

Halo Merger Trees

SAMSSP

FOF GroupFinder

(Sub-)HalosFOF GroupsSimulation Particles

Gadget

MoMaF’

SAM Galaxies

Light-cone Galaxies

SkyMaker’SExtractor

Synthetic Images

SExtractor Catalogues

PSF, Noise,Background

Scaling

IGM

MRObs

2012-12-17 Millennium Workshop 3

Virgo-Millennium Database

• Online access to Millennium simulations etal– portal: http://www/mpa-garching.mpg.de/millennium

• Databases – public: http://gavo.mpa-garching.mpg.de/Millennium – private + MyDB:

http://gavo.mpa-garching.mpg.de/MyMillennium• Improved database access– MillenniumTAP batch queries – http://galformod.mpa-garching.mpg.de/millenniumtap

2012-12-17 Millennium Workshop 4

Millennium data model (old)

5 5

last progenitor main leaf and root

Trees in Database

2012-12-17 Millennium Workshop 6

Merger trees

particles halos

2012-12-17 Millennium Workshop 7

Joining Garching and Durham

8 8

2012-12-17 Millennium Workshop 9

Most recent data sets(@MPA)

• New simulations: – MR-XXL: Angulo etal 2012– Millennium WMAP7 (Thomas etal, MilliGas?)

• Updated semi-analytics, with (scaling to) WMAP7, MCMC– Guo etal 2011, 2012– Henriques etal 2012/3

• New synthetic light cone catalogues– Henriques etal 2012– also some all-sky (SDSS depth)

• Millennium Run Observatory– Overzier etal 2012, images+Sextractor catalogues

• Most loaded in new databases– soon: MR-XXL trees, Millennium WMAP7 (sim +scaled, halos + galaxies)

• New DB functions:– spatial queries– possibly: scaling, …

2012-12-17 Millennium Workshop 10

Millennium Workshop 11

AllSky(Paritosh Verma)

2012-12-17

“Red”“Blue”

“All” “All”smoothed

Most recent data sets(@MPA)

• New simulations: – MR-XXL: Angulo etal 2012– Millennium WMAP7 (Thomas etal, MilliGas?)

• Updated semi-analytics, with (scaling to) WMAP7, MCMC– Guo etal 2011, 2012– Henriques etal 2012/3

• New synthetic light cone catalogues– Henriques etal 2012– also some all-sky (SDSS depth)

• Millennium Run Observatory– Overzier etal 2012, images+Sextractor catalogues

• Most loaded in new databases– soon: MR-XXL trees, Millennium WMAP7 (sim +scaled, halos + galaxies)

• New DB functions:– spatial queries– possibly: scaling, …

2012-12-17 Millennium Workshop 12

Spatial queries in the databaseSpatial3D library (with Tamas Budavari)

• Searches for objects in spatial subsets of simulation box is hard

• C# library for optimising such spatial queries using PH space filling curve

• Soon in Millennium DBs (needs documentation)

2012-12-17 Millennium Workshop 13

with cover as ( -- some details left outselect * from dbo.fSimulationCoverSphere(

Simulation::Millennium,Sphere3D::newInstance(0,0,0,20),6) )

select f.* from cover c , millennium..fof f where f.snapnum=67 and f.phkey between c.keymin and keymax and c.shape.ContainsPoint(f.x, f.y, f.z)=1

14

Query shape syntax• Primitives :

– box[ 0,0,0,10,10,10 ], sphere[10,10,10,20]– cone[0,0,0,1,1,1,.1,600] vertex,direction,angle,min-

depth,max-depth, – conesegment[0,0,0,1,1,1,.1,500,600]

vertex,direction,angle,min-depth,max-depth• Combinations :

– union/intersection/difference [ shape (, shape)* ]

15

courtesy RaulAngulo

used in MXXL browser

In the future: light cones in SQL(requires galaxy catalogues with dM/dz)

2012-12-17 Millennium Workshop 17

Usage statistics• 589 registered users• > 17.3 million queries (7532/day)• > 287 billion rows (137 million/day)queries,rows,users

2012-12-17 Millennium Workshop 18

Distributions over users

2012-12-17 Millennium Workshop 19

Millennium TAP• New DB access service

– All MyMillennium users have access• Batch queue

– longer timeout– capability to kill queries– history

• Implements IVOA’s Table Access Protocol (TAP)– AJAX frontend: TAPFace– TAP aware tools can query it, TOPCAT

• Some differences– schema vs database (Non SQLServer DBs can be served)– no mydb (yet)

• Demo this afternoon

2012-12-17 Millennium Workshop 20

It’s not always as simple as it seems.One should try to understand the data when using the database

thanks to

Kevin Bundy, Tommaso Treu, Richard Ellis

http://astro.berkeley.edu/~kbundy/millennium/

2012-12-17 Millennium Workshop 21

Other Millennium data dissemination services

• portal: http://galformod.mpa-garching.mpg.de • Simulation service(s)– SeSam online (Neistein, Weinmann , Bourges)– L-Galaxies online (see talk by Matthias Egger)

• Image Browsers: – may help define where to point your virtual

telescope.– MR-XXL browser (public | interactive) (Angulo), – MRObs image browser (public | private) (Overzier

etal)

2012-12-17 Millennium Workshop 22

Service vs other data sets• Databases– HydroClusters database (Dolag)

• Virtual telescopes– Smac (Dolag, Reynecke, Bourges)– PHOX (Biffi etal)

• Image Browsers: – Phoxy browser + PHOX

2012-12-17 Millennium Workshop 23

Thank you.

2012-12-17 Millennium Workshop 242012-12-17 Millennium Workshop 24