A Grid Approach to Geographically Distributed Data Analysis for Virgo
F. Barone, M. de Rosa, R. De Rosa, R. Esposito, P. Mastroserio, L. Milano, F. Taurino, G.Tortone
INFN NapoliUniversità di Napoli “Federico II”
Università di Salerno
L. Brocco, S. Frasca, C. Palomba, F. RicciINFN Roma1
Università di Roma “La Sapienza”
GWADW 2002 – Isola d’Elba (Italy) – May 19-26 2002
Outline scientific goals and requirements basic concepts of GRID what the Grid offers layout of VIRGO Virtual Organisation application to gravitational waves data analysis conclusions
Scientific goals and requirements the coalescing binaries and periodic sources analysis
needs large computing power ~ 300 Gflops for coalescing binaries search ~ 1000 Gflops for periodic sources search
computational grids allows to use computing resources
available in different laboratories/institutions
GRID: a definition
GRID:
an infrastructure to allow the sharing and coordinated use of resources within large, dynamic and multi-institutionals communities;
Basic resources of DataGrid Middleware DataGrid is an European Community project (3 years) to
develop Grid Middleware and testbed infrastructure on European scale;
need to execute a program Computing Element (CE)
need to access data Storage Element (SE)
need to move data network
GRID resource that provides CPU cycles
Examples:• clusters of PCs• supercomputers• ...
Computing Element (CE)
GRID resource that provides disk space to store files
Examples:• simple disks pool• big Mass Storage System• ...
Data is accessible to all processes running on CEs via multiple protocols
Storage Element (SE)
Grid resource
A Grid resource provides a standard interface (protocol and API) that is common to that type of resource:
all CEs talk the same protocol (CE protocol) independently of the underlying batch system;
all SEs talk the same protocol (SE protocol) independently of the underlying Mass Storage System
What the Grid offers independence from execution location
the user doesn’t want to know where a job will run (what CE)
independence from data location the user doesn’t want to know where is data (what SE);
security authentication, authorization;
Independence from execution location
Workload Management System
Resource Broker (RB)a Resource Broker tries to find a good match between the job requirements and preferences and the available resources, in particular CEs
Job Submission Service (JSS)the Job Submission Service then guarantees a reliable job submission and monitoring
Scheduling criteria1. authorization information2. data availability3. job requirements4. job preferences5. accounting
Monitoring/Information System The Resource Broker needs some information:
what are available resources ? what is their status ?
The Resource Broker query the Monitoring Information System to locate producers (CE, SE,...) and then obtain data directly from producers;
status update “pushed” on MIS
data obtained from CE
Logging and bookkeeping
The LB service is a database of events concerning jobs and the other service of Workload Management System (RB and JSS)
provides status info for jobs; designed to be highly reliable and available;
Independence from data location
Replica Catalogue (RC) With Replica Catalogue the same file (master) can
exists in multiple copies (replicas) LFN – Logical File Name: name for a set of replicas
example: lfn://virgo.org/virgofile-1.dat PFN – Physical File Name: location of a replica
example: pfn://virgo-se.na.infn.it/virgo/virgofile-1.dat
it’s up to RB to translate LFN in PFN
to locate the SE “closed” to a CE
GridFtp
GridFtp is an efficient data transfer protocol Features:
GSI security; multiple data channels for parallel transfers; partial file transfers; third-party (direct server-to-server) transfers; interrupted transfer recovery;
Napoli - Bologna file transfer performance for different file size and number of sockets using GSIFTP
0
5
10
15
20
25
30
1 2 4 8 16 32 64
number of sockets used
ba
nd
wit
h (
Mb
it/s
)
1 MB 10 MB 50 MB 100MB 500 MB
saturation of lowest bandwith
INFN Napoli – 34 Mbit/s
CNAF Bologna – 98 Mbit/s
GridFTP tests period
“standard FTP” average bandwith
Grid Approach to Geographically Distributed Data Analysis for Virgo
Layout of VIRGO Virtual Organisation
Computing Element
Worker Node 1
Worker Node 3
Storage Element
Worker Node 2
CNAF-BolognaCNAF-Bologna
Resource Broker
Information Index
Replica Catalogue
Computing Element
Worker Node 1
Worker Node 2
User Interface
INFN Roma1INFN Roma1
Computing Element
Worker Node 1
Worker Node 2
User Interface
INFN NapoliINFN Napoli
GARR
E0 run
Storage Element
Storage Element
Computing Element
Worker Node 1
Worker Node 2
Worker Node 3
Storage Element
Computing Element
Worker Node 1
Worker Node 1
Computing Element
Worker Node 1
Worker Node 1
Job submission mechanismUser Interface
PBS
ResourceResource BrokerBroker I II I
IS
IS
OS
OS
Job submission mechanism The general scheme for distributed computation is the following:
multiple jobs submission from the Rome UI;
the Resource Broker interrogates the Information Index and submit each job to an available WN; the Input Data file is staged from the SE on the WN;
the output is sent back to the UI or published on SE;
the Resource Broker automatically distributes the jobs among the nodes (according to specifications in the JDL file) unless we decide to tie a given job to a particular node;
job scheduling at the node level is done via PBS.
Grid tests for coalescing binaries search 1/2
Algorithm: standard matched filters Templates generated at PN order 2 with Taylor approximants Data
VIRGO E0 run start GPS time: 685112730 data length: 600 s
Conditions raw data resampled at 2 kHz lower frequency: 60 Hz upper frequency: 1 kHz search space: 2 – 10 solar masses minimal match: 0.97
number of templates: ~ 40000
Grid tests for coalescing binaries search 2/2
Step 1The data were extracted from CNAF-Bologna Mass Storage System. The extraction process reads the VIRGO standard frame format, performs a simple resampling and publishes the selected data file on the Storage Element;
Step 2The search was performed dividing the template space in 200 subspace and submitting from Napoli User Interface a job for each template subspace.Each job reads the selected data file from the Storage Element (located at CNAF-Bologna) and runs on the Worker Nodes selected by Resource Broker in the VIRGO VO.Finally, the output data of each job were retrieved from Napoli User Interface.
Grid tests for periodic sources search
The analysis for periodic sources search is based on a hierarchical approach in which coherent steps, based on FFTs and incoherent ones, based on the Hough Transform, alternates. At each iteration a more refined analysis is done on the selected candidates.
This procedure fits very well in a geographically distributed computational scheme.
The whole problem can be divided in a number of independent smaller tasks, each performed by a given computational node. E.g. each node can analyze a frequency band and/or a portion of the sky.
We have performed some preliminary test to evaluate the DataGrid software with respect to our analysis problem.
For the GRID tests we have used the code for the Hough Transform. The source spin-down is not taken into account. The input of the code is given by a “peak map” in the time-frequency plane.
Grid tests for periodic sources search 1/2
The tests consists of two phases:
1. Production of input data on the SE;
2. Distributed computation.
We start from raw data of engineering run E1 (~ 5 hours) and the steps are the following:
channel extraction;
decimation at 1 kHz;
generation of periodograms by computing interlaced and windowed FFT (T_FFT=4194.304 s);
peaks selection (above two times the average noise);
The produced time-frequency peaks map covers 20 Hz in frequency (from 480 to 500 Hz).
Grid tests for periodic sources search 2/2
Each computing node processes a subset of the whole frequency band. Each job runs according to this scheme:
reads its initial reference frequency and the velocity vector direction;
migrates on a worker node;
takes from the SE the input data corresponding to the frequency band associated to that job;
calculates the current frequency band of interest, i.e the Doppler band;
calculates the Hough Transform;
iterates on the reference frequency until the full band has been processed.
The output of each job would be a set of candidates which will be followed in the next coherent phase.
Conclusions we have successfully verified that multiple jobs can be
submitted and the output retrieved with small overhead time;
computational grids seems very suitable to perform data analysis for coalescing binaries and periodic sources searches;
Future plans
testing MPI-job submission for coalescing binaries search (feature provided in next DataGrid release);
testing the whole data analysis chain for periodic sources search;
first tests for network analysis among interferometers;