INFN - FerraraINFN - Ferrara
BaBar MeetingBaBar Meeting
SPGrid: status in FerraraSPGrid: status in Ferrara
Enrica Antonioli - Paolo VeronesiFerrara, 12/02/2003
Ferrara - 12/02/03 2
Topics Topics
The DataGrid project
Ferrara Farm Configuration
First SP submissions through the Grid
Work in Progress
Future Plans
Conclusions
Ferrara - 12/02/03 3
INFN-GRID
2001- 2003
To manage and use computing resources distributed on Garr-b sites
Deployment of Testbed sites, in order to validate EDG software release and to adapt them to High Energy Physics requests
Special project of INFN
European DataGrid and INFN-GRIDEuropean DataGrid and INFN-GRID
2001 - 2003
Funded by European Union
Computing Grids permit:
• High Throughput Computing
• Analysis of large dimension data
• Sharing resources and data
Applications involved:• Biomedical Sciences
• Earth Observation
• High Energy Physics
EDG
CT
BO
PD
TO
CA
ROMA
MI
To USA
To Russia/Japan
Current prototipe of INFN DataGrid testbed connected to EDG testbed – US and ASIA
Cern
FEFE
R.A.L
Manchester
Ferrara - 12/02/03 4
EDG Architecture and ServicesEDG Architecture and Services
OS & Net services
APPLICATION Layer
ALICE ATLAS CMS LHCb BaBarBaBar
High level GRID middleware
Basics ServicesGLOBUS
toolkit
DataGRIDArchitecture
Ferrara - 12/02/03 5
CECE//WNWN
SESE
The DataGrid Testbed consists of different types of machines (Grid Elements).
In Ferrara the farm is composed by one Computing Element (CE), three Worker Nodes (WN), one User Interface (UI) and one Storage Element (SE).
All these machines are managed by a LCFGng (Local ConFiGuration system new generation) server and they are automatically configured.
UIUILCFGngLCFGngServerServer
Grid Elements in FerraraGrid Elements in Ferrara
Ferrara - 12/02/03 6
UI (User Interface): component for accessing to the
workload management system.
Users can submit a job and retrieve the output, they
sholud have an account and a personal certificate
installed in their home directory.
UIUICertificateCertificate AuthoritiesAuthorities
To access the GRID you have to request a certificate to a certification authority. INFN-GRID users can obtain a certificate from the INFN CA (http://security.fi.infn.it/).
User InterfaceUser Interface
To use the BaBar Grid, you must register that certificate with the BaBar Virtual Organisation (BaBar VO).
http://www.slac.stanford.edu/BFROOT/www/Computing/Offline/BaBarGrid/registration.html
Ferrara - 12/02/03 7
Job SubmissionJob Submission
StorageElement
UIJDL
Logging &Book-keeping(LB)
ResourceBroker (RB)
Job SubmissionService (JSS)
ComputingComputingElementElement
Information Service (IS)
ReplicaCatalogue
Job SubmitEvent
Input Sandbox
Job Status
submitted
waiting
ready
scheduled
Input Sandboxrunning
Job Status
done
outputready
Job StatusJob Status Output Sandbox
Output Sandbox
cleared
Ferrara - 12/02/03 8
SPGrid Farm in FerraraSPGrid Farm in Ferrara
LCFGngServer
Management
Lock serverLock server
Data serverData server
SCSISCSI
250 GB250 GB
IR 0A DSESE
CE-WNCE-WN
UIUI
Ferrara - EDG 1.4.3
RBRB
CNAF - BolognaCERN
Ferrara - 12/02/03 9
ConfigurationConfiguration INFN Grid Testbed Status: EDG 1.4.3 (RedHat 6.2).
A BaBar software special release (12.3.2y) has been built and installed to:
Write Kanga files Run Moose on RH 6.2
A special tag of ProdTools has been installed to perform tests.
A pool of BaBar accounts (babar000, babar001,…) has been created in the EDG farm of Ferrara.
Each member of BaBar VO is able to submit jobs to the farm of Ferrara through the RB located at CNAF (grid009g.cnaf.infn.it).
Ferrara - 12/02/03 10
Current StatusCurrent Status
Created a JDL file to run Moose on Grid resources.
Created scripts containing EDG commands to submit jobs, to check their status and retrieve output files.
An user can submit a range of runs.
For each run a job is created and submitted to the Resource Broker, then it is sent to the Ferrara CE (grid0.fe.infn.it).
The output file is then transferred to the closest SE (grid2.fe.infn.it).
Ferrara - 12/02/03 11
Moose.jdlMoose.jdl
grid1> more Moose.jdl Executable ="Moose.csh";InputSandbox ={"Moose.csh",".cshrc","config.csh"};StdOutput ="Moose.txt";StdError ="Moose.log";OutputSandbox ={"Moose.txt","Moose.log"};
General environment configurations
Config file for BaBar.Similar to SP standard
scripts
Similar to SP standard scripts (Job.Xsh)
[…]tar -czvf run${RUNNUM}.tar.gz *.rootglobus-url-copy -vb file://`pwd`/run${RUNNUM}.tar.gz \gsiftp://grid2.fe.infn.it/flatfiles/SE00/paolo/run${RUNNUM}.tar.gz
Globus command:To copy output files from WN to SE
Ferrara - 12/02/03 12
grid1> more launch #!/bin/tcsh -v
@ num_f = $1@ fin = $2
while ( $num_f <= $fin )
####build the run directories[…]####build a config.csh with the appropriate environment variablesecho "#\!/bin/tcsh -v" > config.csh[…]#### now run the jobdg-job-submit -o run$num_f.jobid -r \grid0.fe.infn.it:2119/jobmanager-pbs-long Moose.jdl
cd ..@ num_f++end
The launch scriptThe launch script
A config file is created for each
run
EDG job submission command
Range of runs to submit
For each run a job is created runtime
Ferrara - 12/02/03 13
Job SubmissionJob Submission
grid1> ls1962016 1962017 Moose.csh Moose.jdl config.csh launch monitor retrieve
grid1> ./launch 1962016 1962017[…]dg-job-submit -o run$num_f.jobid -r grid0.fe.infn.it:2119/jobmanager-pbs-long Moose.jdl
Connecting to host grid009g.cnaf.infn.it, port 7771Logging to host grid009g.cnaf.infn.it, port 15830
================== dg-job-submit Success ================ The job has been successfully submitted to the Resource Broker. Use dg-job-status command to check job current status. Your job identifier (dg_jobId) is: https://grid009g.cnaf.infn.it:7846/193.206.188.102/104224188091275?grid009g.cnaf.infn.it:7771 The dg_jobId has been saved in the following file: /home/enrica/stress/1962016/run1962016.jobid[…]
grid1> ls 1962016/Moose.csh Moose.jdl config.csh run1962016.jobid
grid1> ls 1962017/Moose.csh Moose.jdl config.csh run1962017.jobid
Range of runs to submit
Job ID
CNAF RB
Ferrara - 12/02/03 14
grid1> ./monitor 1962016 1962017Run 1962016 is Status = Scheduled Status Reason = initial Run 1962017 is Status = Scheduled Status Reason = initial
grid1> ./monitor 1962016 1962017Run 1962016 is Status = Running StatusRun 1962017 is Status = Running Status
The monitor scriptThe monitor script
grid1> ./monitor 1962016 1962017Run 1962016 is Status = Ready Status Reason = job acceptedRun 1962017 is Status = Ready Status Reason = job accepted
grid1> more monitor#!/bin/tcsh
@ num_f = $1@ fin = $2
while ( $num_f <= $fin )echo Run $num_f is `dg-job-status -i \ $num_f/run$num_f.jobid | grep Status`
@ num_f++end
EDG command
grid1> ./monitor 1962016 1962017Run 1962016 is Status = OutputReady Status Reason = terminated Run 1962017 is Status = OutputReady Status Reason = terminated
Ferrara - 12/02/03 15
The retrieve scriptThe retrieve scriptgrid1> more retrieve #!/bin/tcsh -v@ num_f = $1@ fin = $2while ( $num_f <= $fin )cd $num_f#### get logfilesdg-job-get-output -i run$num_f.jobid --dir $PWD#### get rootfilesglobus-url-copy \ gsiftp://grid2.fe.infn.it/flatfiles/SE00/paolo/run$num_f.tar.gz \ file://`pwd`/run$num_f.tar.gz
tar -xzvf run$num_f.tar.gzrm -f run$num_f.tar.gz#### delete rootfiles form SEglobus-job-run grid2.fe.infn.it /bin/rm \/flatfiles/SE00/paolo/run$num_f.tar.gzcd ..@ num_f++end
EDG command
Globus command:Direct copy of file from SE to UI
Globus command:delete file fromSE
Ferrara - 12/02/03 16
Retrieving OutputRetrieving Output
grid1> ls1962016 1962017 Moose.csh Moose.jdl config.csh launch monitor retrieve
grid1> ls 1962016/150546318633191 Moose.jdl rootdef-tru.root Moose.csh config.csh run1962016.jobidrootdef-tag.root rootdef-aod.root
grid1> ls 1962017/150551318931039 Moose.jdlMoose.csh config.csh run1962017.jobidrootdef-tru.root rootdef-tag.root rootdef-aod.root
grid1> ls 1962016/150546318633191/Moose.log Moose.txt
grid1> ls 1962017/150551318931039/Moose.log Moose.txt
Ferrara - 12/02/03 17
Future PlansFuture Plans
Data serverData server
Lock serverLock server
Objectivity DBObjectivity DBRBRB(UK)(UK)
Ferrara
MOOSEMOOSERPM
RPM
3) Install Objy DB on the SE
2) MOOSE in RPM format
1) Use of IC RB and others
Integration of MooseIntegration of Moose Application withApplication with
EDG software releasesEDG software releases
SPGrid Farm
Management
LCFGngServer
SESE
CE-WNCE-WN
UIUI
Ferrara - 12/02/03 18
DocumentationDocumentation
The DataGrid Project: http://eu-datagrid.web.cern.ch/eu-datagrid/default.htm
EDG tutorials Archive Web Site: http://hep-proj-grid-tutorials.web.cern.ch/hep-proj-grid-tutorials/loginex.html
INFN-Grid Testbed: http://server11.infn.it/testbed-grid/
BaBar-Grid: http://www.slac.stanford.edu/BFROOT/www/Computing/Offline/BaBarGrid/
Status of the Farm in Ferrara: http://print.fe.infn.it/status/