Embed Size (px)
Citation preview
Agent based parallel processing tool for Big Scientific Data
A. Tsaregorodtsev, CPPM-IN2P3-CNRS, Marseille
IHEP, Beijing, 5 May 2015
Plan
2
} The problem of the HEP data processing
} DIRAC Project } Agent based Workload Management System } Accessible computing resources } Data Management } Interfaces
} DIRAC as a service
} Conclusions
Big Data
3
} Data that exceeds the boundaries and sizes of normal processing capabilities, forcing you to take a non-traditional approach for the treatment
} Google trends:
Clouds
Grids Big Data
Big Data
http://www.wired.com/magazine/2013/04/bigdata
4
LHC RAW data per year
§ Already in 2010 the LHC experiments produced 13 PB of data
– That rate outstripped any other scientific effort going on
Big Data
http://www.wired.com/magazine/2013/04/bigdata
5
LHC RAW data per year
WLCG data on the Grid
§ LHC RAW data volumes are inflated by storing derived data products, replication for safety and efficient access, and by the need for storing even more simulated data than the RAW data:
§ ~130 PB overall
6
200-400 MB/sec
Data flow to permanent storage: 4-6 GB/sec
~ 4 GB/sec
1-2 GB/sec
1-2 GB/sec
Massive computations: HEP
7
} LHC experiments pioneered the massive use of
computational grids as a solution to the High Energy Physics Big Data problem } 10s of PBytes of data per year } 100s of thousands CPUs in 100s of centers } 10s of Gbyte/s data transfers } 100s of users from 100s of institutions
} CERN Director General Rolf Heuer about the Higgs discovery: "It was a global effort and it is a global success. The results today are only possible because of the extraordinary performance of the accelerators, including the infrastructure, the experiments, and the Grid computing.”
} Other domains are catching up quickly with the HEP experiments } Life sciences, earth sciences, astrophysics, social sciences, etc
Worldwide LHC Computing Grid Collaboration (WLCG)
• Distributed infrastructure of 150 computing centers in 40 countries • 300+ k CPU cores (~ 2M HEP-SPEC-06) • The biggest site with ~50k CPU cores, 12 T2 with 2-30k CPU cores • Distributed data, services and operation infrastructure 8
Computing grids
9
} Grids are providing common infrastructure and rules to integrate multiple computing centers } Common computing element access protocols } Common user payload scheduling system } Common policies and security rules
} Users are signing up once and have access to multiple computing centers
} Common monitoring of the user jobs } Common accounting of computing resources
Standard Grid Architecture
10
} Centralized Workload Management System } Computing sites publish information on their availability } Users submit jobs to the central repository } The central Resource Broker (RB) pushes user jobs to sites
after matching the job/resource compatibility
Standard grid problems
11
} Heavy reliance on the site status information flow } Any delays result in suboptimal scheduling decisions
} Insufficient RB power for large infrastructures } Necessity to deploy multiple central (!) RB’s
} Heavy infrastructures required on the sites – resources providers } Providing timely status, accounting and monitoring information } Ensuring per user authentication/authorization } Managing community policies via local batch queues
parameterization
Virtualized Computing Resources
12
} Since recently computing centers started to provide their resources using cloud technologies } Virtualized computing infrastructures } Very flexible – provides resources adapted to the user needs
} Operating systems, memory, CPU power, etc.
} However, large scientific collaborations can have access to multiple computational clouds } Dealing with independent clouds separately requires a huge
management effort } Federating multiple clouds into a single coherent system is
necessary to provide a transparent access for users.
DIRAC Grid Solution
} DIRAC is developed originally for the LHCb experiment at with the goals:} Integrate all the heterogeneous computing resources
available to the community} Provide solution for both WMS and DMS tasks} Minimize human intervention at sites providers of resources } Make the grid convenient for the users:
} Simpler intuitive interfaces} Fault tolerance, quicker turnaround of user jobs} Enabling Community policies
} DIRAC has introduced a notion of Pilot Jobs} A concept of distributed agents bringing together various
disjoint computing resources
13
DIRAC Workload Management
14
PhysicistUser
EGIPilotDirector
EGI/WLCGGrid
NDGPilotDirector
NDGGrid
AmazonPilotDirector
Amazon EC2 Cloud
CREAMPilotDirector
CREAMCE
MatcherService
ProductionManager
15
WMS: using heterogeneous resources
} Including resources in different gridsand standalone clusters is simple with Pilot Jobs} Needs a specialized Pilot
Director per resource type} Users just see new sites
appearing in the job monitoring
16
WMS: applying VO policies
17
u In DIRAC both User and Production jobs are treated by the same WMS } Same Task Queue
u This allows to apply efficiently policies for the whole VOª Assigning Job Priorities for different
groups and activitiesª Static group priorities are used currentlyª More powerful scheduler can be plugged in
● demonstrated with MAUI scheduler
● Users perceive the DIRAC WMS as a single large batch system
DIRAC as a resource manager
18
Computing Grids
19
} DIRAC was initially developed with the focus on accessing conventional Grid computing resources } WLCG grid resources for the LHCb Collaboration
} It fully supports gLite middleware based grids } European Grid Infrastructure (EGI), Latin America GISELA, etc
} Using gLite/EMI middleware } Northern American Open Science Grid (OSG)
} Using VDT middleware } Northern European Grid (NDGF)
} Using ARC middleware
} Other types of grids can be supported } As long we have customers needing that
Clouds
20
} VM scheduler developed for Belle MC production system} Dynamic VM spawning taking
Amazon EC2 spot prices and Task Queue state into account
} Discarding VMs automatically when no more needed
} The DIRAC VM scheduler by means of dedicated VM Directors is interfaced to } OCCI compliant clouds:
} OpenStack, OpenNebula } CloudStack} Amazon EC2
Standalone computing clusters
21
} Off-site Pilot Director} Site delegates control to the central
service} Site must only define a dedicated
local user account} The payload submission through the
SSH tunnel
} The site can be a single computer or a cluster with a batch system} LSF, BQS, SGE, PBS/Torque,
Condor, OAR} More to come:
} SLURM, LoadLeveler. etc
} The user payload is executed with the owner credentials} No security compromises with respect
to external services
Standalone computing clusters
22
Examples: } DIRAC.Yandex.ru
} >2000 cores} Torque batch system, no grid
middleware, access by SSH} Second largest LHCb MC production
site
} LRZ Computing Center, Munich} SLURM batch system, GRAM5 CE
service} Gateway access by GSISSH} Considerable resources for
biomed community (work in progress)
} Mesocentre Aix-Marseille University} OAR batch system, no grid
middleware, access by SSH} Open to multiple communities (work
in progress)
BOINC Desktop Grids
23
} On the client PC the third party components are installed: } VirtualBox hypervisor } Standard BOINC client
} A special BOINC application } Starts a requested VM within
the VirtualBox } Passes the Pilot Job to the VM
and starts it
} Once the Pilot Job starts in the VM, the user PC becomes a normal DIRAC Worker Node
Data Storage and Catalogs
24
} Managing data is more complicated than workload management } Already existing data in various storage resources must be
connected } Contents of already existing File Catalogs must be either
connected or copied to the DIRAC File Catalog
} Need for common interfaces to various types of storage and catalog services
Data Management components
} For DIRAC users the use of any Storage Element or File Catalog is transparent} Up to a user community to
choose components to use} Different SE types can be
mixed together} Several File Catalogs can be
used in parallel} Complementary functionality} Redundancy
} Users see depending on the DIRAC Configuration} Logical Storage Elements
} e.g. DIRAC-USER, M3PEC-disk} Logical File Catalog
25
Interware: DMS
26
} Data Management } Multiple types of Storage Elements ( SRM, XRootD,
iRods, DAV, DIP, … ) seen as logical entities for the users
} SE Proxy service for protocol independence } Data replication:
} FTS3 ¨ File Transfer Service
} Third party } Local pipe
Interware: DMS
27
} DIRAC File Catalog } Replica Catalog } User Metadata Catalog } Support for data provenance information } Efficient storage usage reports
} Allow for user quota policies
} Support for user defined datasets
} LCG File Catalog
} Specialized File Catalogs
Resource provisioning service
28
} DIRAC attempts to provide access to all kinds of existing resources and services useful for its users
} At the same time it provides its own solutions, e.g. catalogs, storage and computing element, transfer service, etc.
} By design services from different providers can be used together in parallel not necessarily replacing one another } Example: use of DFC and LFC together, see below
} The final choice of the services to use is left to the user
Interware
29
} DIRAC has all the necessary components to build ad-hoc grid infrastructures interconnecting computing resources of different types. This allows to speak about the DIRAC interware.
Interfaces
30
DIRAC: Secure Web Portal
31
} Focus on the Web Portal as the main user tool for interactions with the grid
} Intuitive desktop application like interface} Ajax, Tornado, ExtJS Javascript library
} Monitoring and control of all activities} User registration, proxy upload} User job monitoring and manipulation, downloading results} Data manipulation and downloads} DIRAC Systems configuration and management
} Secure access } Standard grid certificates} Fine grained authorization rules
Web Portal examples
32
Application Web Portals
} Specific application portals can be built in the DIRAC Web Portal framework } Community Application Servers
} DIRAC RESTful interface} Language neutral} Suitable to use with portals written in Java, PHP, etc
} Other interfaces include} Extensive Python API} A rich set of command line tools ( >200 commands )
} Available on UNIX-like systems
33
DIRAC Users: large communities
34
LHCb Collaboration
35
} Up to 100K concurrent jobs in ~120 distinct sites} Equivalent to running a virtual computing center with a power of
100K CPU cores} Further optimizations to increase the capacity are possible● Hardware, database optimizations, service load balancing, etc
Experiments: Belle II
36
} Combination of the non-grid, grid sites and (commercial) clouds is a requirement
} 2 GB/s, 40 PB of data in 2019} Belle II grid resources
} WLCG, OSG grids} KEK Computing Center} Amazon EC2 cloud
} First production run is done Thomas Kuhr, Belle II
Belle II
37
} DIRAC Scalability tests
Hideki Miyake, KEK
Community installations
38
} ILC/CLIC detector Collaboration, Calice VO } Dedicated installation at CERN, 10 servers, DB-OD MySQL server } MC simulations } DIRAC File Catalog was developed to meet the ILC/CLIC requirements
} BES III, IHEP, China } Using DIRAC DMS: File Replica and Metadata Catalog, Transfer services } Dataset management developed for the needs of BES III
} CTA } CTA started as France-Grilles DIRAC service customer } Now is using a dedicated installation at PIC, Barcelona } Using complex workflows
} Geant4 } Dedicated installation at CERN } Validation of MC simulation software releases
} Alpes Laser } Company using dedicated DIRAC setup for simulation of the laser hardware } Mostly using commercial Amazon resources
} DIRAC evaluations by other experiments } LSST, Auger, TREND, Daya Bay, Juno … } Evaluations can be done with general purpose DIRAC services
DIRAC as a Service
39
DIRAC as a service
40
} DIRAC client is easy to install} Part of a usual tutorial
} DIRAC services are easy to install but} Needs dedicated hardware for hosting} Configuration, maintenance needs expert manpower} Monitoring computing resources is a tedious every-day task
} Small user communities can not afford maintaining dedicated DIRAC services} Still need easy access to computing resources
} Large grid infrastructures can provide DIRAC services for their users.
France-Grid DIRAC service
41
} Several regional and university campus installations in France} Complex maintenance
} Joint effort to provide France-Grid DIRAC service} Hosted by the CC/IN2P3,
Lyon, T1 center} Distributed team of service
administrators} 15 VOs, ~100 registered users
} Mostly biomed applictions
http://dirac.france-grilles.fr
Multi-VO DIRAC service in IHEP
42
} IHEP Computing Center is the main resource provider for the BES experiment
} The BES Collaboration decided to use DIRAC to build its distributed computing project } Developed a good expertize level in developing and managing DIRAC
services
} The BES dedicated DIRAC service is being extended to support other user communities
} CEPC, Juno, Daya Bay, … } Non-HEP communities ?
} Reuse of acquired experience } Single Administration team } Minimization of the user support and maintenance effort
Conclusions
43
Conclusions
44
} The computational grids are no more something exotic, they are used in a daily work for various applications
} Agent based workload management architecture allows to seamlessly integrate different kinds of grids, clouds and other computing resources
} DIRAC is providing a framework for building distributed computing systems and a rich set of ready to use services. This is used now in a number of DIRAC service projects on a regional and national levels
} Services based on DIRAC technologies can help users to get started in the world of distributed computations and reveal its full potential
http://diracgrid.org
