Upload
fritzi
View
31
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Perspectives of JINR Grid-infrastructure. V.V. Ivanov , Gh. Adam, V.V. Korenkov, T.A. Strizh, P.V. Zrelov Laboratory of Information Technologies Joint Institute for Nuclear Research. Participating in LHC, RHIC, TEVATRON… In future: FAIR, ILC …. LHC-2008. FAIR-2015. ILC – after 2020 (?). - PowerPoint PPT Presentation
Citation preview
Perspectives of JINR Perspectives of JINR Grid-infrastructureGrid-infrastructure
V.V. IvanovV.V. Ivanov, Gh. Adam, V.V. Korenkov, T.A. Strizh, P.V. Zrelov, Gh. Adam, V.V. Korenkov, T.A. Strizh, P.V. Zrelov
Laboratory of Information TechnologiesLaboratory of Information TechnologiesJoint Institute for Nuclear ResearchJoint Institute for Nuclear Research
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
Participating in LHC, RHIC, TEVATRON… In future: FAIR, ILC …
LHC-2008 FAIR-2015 ILC – after 2020 (?)
The JINR participation in the LHC experiments and in other large-scale projects asked for a substantial increase of its networking and information resources as well as the deployment of a large volume of work toward the development of the JINR Grid-segment and its integration in the Russian Grid-infrastructure RDIG (Russian Data IntensiveGrid).
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
The Laboratory of Information Technologies (LIT) is responsible for the computing infrastructure development in JINR and hosts the Central Information and Computing Complex (CICC) of the Institute.
Further perspective tasks follow from the decision of the JINR Committee of the Plenipotentiary Representatives, concerning the radical improvement of the JINR computer telecommunication links with major partner laboratories in the JINR Member States during the years 2010-2015 and the development of a unified Grid-environment of the JINR Member States. This is a cornerstone task of the JINR seven-year plan on 2010–2016 within the direction ”Networks. Computing. Computational Physics”.
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
Development of the JINR Development of the JINR Grid-environment – 2010-2016Grid-environment – 2010-2016
Three levels within the JINR Grid-environment can be distinguished: network, resource, and applied.
NETWORK
RESOURCE
APPLIED
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
CooperationCooperationCollaborators from the JINR Member States (protocols of cooperation with
INRNE (Bulgaria), ArmeSFo (Armenia), FZK Karlsruhe GmbH (Germany), IHEPI TSU (Georgia), NC PHEP BSU (Belarus), KFTI NASU (Ukraine), etc.).
Grants of Plenipotentiaries of the JINR Member States: Poland, Bulgaria, Slovakia, Romania. Agreement on Joint Research Programme “Meshcheryakov-Hulubei” LIT-Romania.
BMBF grant “JINR Network Security, Failover Measures, Network monitoring”.
CERN-JINR Cooperation Agreement on several topics, and JINR in the LCG Project (participation of JINR in the LCG project).
JINR-South Africa Cooperation Agreement - project “Development of Grid segment for the LHC experiments”.
Participation in joint projects: Enabling Grids for E-sciencE (EGEE -a project co-funded by the European Commission through the Sixth Framework Programme, CERN-INTAS project and "SKIF-GRID“ project of the Program of the Belarusian-Russian Union State as a part of joint propositions of NASB and Federal Agency of Science and Innovations of Russian Federation “The development and use of hardware and software in GRID-technologies and advanced supercomputer systems SKIF in 2007-2010 (SKIF-GRID)”.
Joint research grants from RFBR and Ministry of Science and Education with MSU and SINP MSU, Saratov University, IM SB RAS, IMM RAS, TU Gomel (Belorussia).
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
June 2009 - two important projects completed
Two large-scale very important projects have been completed: -- JINR - Moscow 20Gbps telecommunication channel -- JINR CentralJINR Central Information and Computing Complex major upgrade.Information and Computing Complex major upgrade.
Comprehensive presentation was done at the meeting of the Programme Advisory Committee for Particle Physics, on June 10.
All organizations involved in the project realizations, together with the representatives of Ministry of Communications, Federal Agency for Science and Innovation, attended the Presentation.
Our community colleagues and prospective users - residents of the special economical zone - also took part in the event.
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
Collaboration with RSCC, RIPN, MSK-IX, JET Infosystems, Nortel
JINR - Moscow 20Gbpstelecommunication channel
In perspective, the used DWDM and 10Gb Ethernet technologies allow the creation of up to 80 channels of 10 Gbps each, resulting in a throughput of up to 800 Gbps.
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
JINR Central Information and Computing ComplexJINR Central Information and Computing Complex
In 2008In 2008, the total CICC , the total CICC performance was 1400 performance was 1400 kSI2KkSI2K, , the disk storage capacity 100 TB the disk storage capacity 100 TB
At presentAt present, the CICC , the CICC performance equals performance equals 2300 kSI2K2300 kSI2K and the disk storage capacity and the disk storage capacity 500 TB500 TB
20082008 20092009 20102010
CPUCPU(kSI2K)(kSI2K)
12501250 17501750 25002500
DiskDisk (TB)(TB)
400400 800800 12001200
TapesTapes(TB)(TB)
-- 100100 200200
PLAN
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
JINR Local Area Network Backbone (LAN)
• Comprises 6579 computers and nodes,Users – 3593,IP – 7633,Remote VPN users (Lanpolis, Contact, TelecomMPK) - 1248;• High-speed transport (1Gbps) (Min. 100 Mbps to each PC);• Controlled-access (Cisco PIX-525 firewall) at network entrance;• Partially isolated local traffic (8 divisions have ownsubnetworks with Cisco Catalyst 3550 as gateways);• General network authorization system involves many services (AFS, batch systems, Grid, JINR LAN remote access, etc.)
• Comprises 6579 computers and nodes,Users – 3593,IP – 7633,Remote VPN users (Lanpolis, Contact, TelecomMPK) - 1248;• High-speed transport (1Gbps) (Min. 100 Mbps to each PC);• Controlled-access (Cisco PIX-525 firewall) at network entrance;• Partially isolated local traffic (8 divisions have ownsubnetworks with Cisco Catalyst 3550 as gateways);• General network authorization system involves many services (AFS, batch systems, Grid, JINR LAN remote access, etc.)
Plans: Step-by-step modernization of the
JINR Backbone – transfer to 10 Gbps Development and modernization of
the control system of the JINR highway network
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
• upgrade of the JINR backbone to a data transfer rate of 10 Gbps and gradual connection to it of various JINR laboratories; • increase of the performance of the LAN central telecommunication node kernel; • increase of the LAN protection at the hardware level;• wireless access to the LAN within the JINR territory; • implementation of new solutions and methods for the traffic control, information security, development of effective tools for control and the quality of service of JINR LAN; • development of the system of protection of the JINR LAN resources from non-authorized access.
The plan for the JINR LAN development for the years 2010-2016 foresees several general purpose tasks as well as specific site-related ones. A selective enumeration points to the:
First stage
JINR Laboratories
JINR Laboratories
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
JINR Central Information and Computing ComplexJINR Central Information and Computing Complex
12 1212121212
64KW
Super Blades5 x 10
APS
CICCServers
Users servers
dCacheServers
JINR nx 1 GbE x 10 GbE - plan)
Backbo e 2 (2
5
2
22
3
6
6
1
1
4 2 2
A1-1282 TB
B1-8144 TB
B9-16144 TB
RAID
Twins20 X 2
HP20
Т- 20
platforms
Batch Cluster
2
3
4
5
HP Procurve 5412zl-966
HP Procurve 3500yl-24G
Hp Procurve 2810-24 G
Hp Procurve 2810-48 G
Cisco Catalyst 2970
Cisco Catalyst 3750
CICCServers
Т- 20
platforms
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
JINR JINR CICC structural schemeCICC structural scheme
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
The protocol between CERN, Russia and JINR on participation in LCG Project was approved in 2003. MoU on Worldwide LHC Computing Grid (WLCG) was signed by JINR in October 2007
The tasks of the JINR in the WLCG:•WLCG-infrastructure support and development at JINR;WLCG-infrastructure support and development at JINR;• participation in WLCG middleware testing/evaluation, participation in WLCG middleware testing/evaluation, • participation in Service/Data Challenges,participation in Service/Data Challenges,• grid monitoring and accounting tools development;grid monitoring and accounting tools development;• FTS-monitoring and testing;FTS-monitoring and testing;• participation in ARDA activities in coordination with experiments; participation in ARDA activities in coordination with experiments; • JINR LCG portal support and development;JINR LCG portal support and development;• HEP applications; HEP applications; • MCDB development; MCDB development; • support of JINR Member States in the WLCG activities;support of JINR Member States in the WLCG activities;• User & Administrator training and education.User & Administrator training and education.
LHC Computing Grid Project
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
support and development WLCG/EGEE infrastructure;
participation in middleware testing/evaluation,
participation in Data and Service Challenges,
grid monitoring and accounting system development;
FTS-monitoring and testing
MCDB development;
Participation in ARDA activities in coordination with experiments;
HEP applications;
User & Administrator Training and Education
support of JINR member states in the GRID activities.
JINRJINR in the LCG/EGEE in the LCG/EGEE
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
CICC comprises:65 servers
4 interactive nodes960 computing nodes, Xeon 5150, 8GB RAM (GEthernet)(160 computing nodes, Xeon X5450, 16GB RAM, InfiniBand).
Site name: JINR-LCG2Internal CICC network – 1Gbit/secOperating system -Scientific Linux CERN 4.6 (transfer to Scientific Linux SL release 5.2.) ; Middleware version GLITE-3.1 File Systems – AFS (the Andrew File System) for user Software and home directories is a world-wide distributed file system. AFS permits to share easily files in an heterogeneous distributed environment (UNIXes, NT) with a unique authentication scheme (Kerberos). dCache- for data.User registration system – Kerberos 5 ( AFS use Kerberos 5 for authentication )
JINR WLCG JINR WLCG infrastructureinfrastructure
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
JINR provides the following services in the WLCG environment:JINR provides the following services in the WLCG environment:
Basic services:Basic services: - Berkley DB Information Index (top level BDII); - Berkley DB Information Index (top level BDII); - site BDII; - site BDII; - 2 x Computing Element (CE); - 2 x Computing Element (CE); - Proxy Server (PX); - Proxy Server (PX); - 2 x Workload Management System (WMS); - 2 x Workload Management System (WMS); - Logging&Bookkeeping Service (LB); - Logging&Bookkeeping Service (LB); - RGMA-based monitoring system collector server (MON-box); - RGMA-based monitoring system collector server (MON-box); - LCG File Catalog (LFC); - LCG File Catalog (LFC); - Storage Element (SE), dCache - Storage Element (SE), dCache 400400 TB, 4 x gridftp door, 14 x pool; TB, 4 x gridftp door, 14 x pool; - 4 x User Interface (UI), installed in AFS. - 4 x User Interface (UI), installed in AFS.
Special Services - Special Services - VO boxes for ALICE and for CMS; ROCMON;VO boxes for ALICE and for CMS; ROCMON;PPS and testing infrastructure - PPS and testing infrastructure - Pre-production gLite version; Pre-production gLite version;
Software for VOs: dCache xrootd door, AliROOT, ROOT, GEANT packages for ALICE; Software for VOs: dCache xrootd door, AliROOT, ROOT, GEANT packages for ALICE; ATLAS packages; CMSSW packages for CMS and DaVinchi, Gauss packages for ATLAS packages; CMSSW packages for CMS and DaVinchi, Gauss packages for LHCb.LHCb.
JINR WLCG JINR WLCG infrastructureinfrastructure
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
dCache in JINRdCache in JINR
Internet / GRID
lxfs07
lxfs07lxfs07
lxfs07lxpub01
lxfs07lxfs07
lxfs07wn…
lxfs07lxfs07
lxfs07lxfs71
RAID
500 TBAdmin
Doors
local
WNs
JINR Backbone
Protocols gFTP, SRM, XROOT
DCAP PNFS
lxfs07lxfs07
lxfs07rda02
Pools
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
Frames for Grid cooperation of JINR in 2009Frames for Grid cooperation of JINR in 2009 Worldwide LHC Computing Grid (WLCG); Enabling Grids for E-sciencE (EGEE); RDIG Development (Project of FASI) CERN-RFBR project “GRID Monitoring from VO perspective” BMBF grant “Development of the GRID-infrastructure and tools
to provide joint investigations performed with participation of JINR and German research centers”
“Development of Grid segment for the LHC experiments” was supported in frames of JINR-South Africa cooperation agreement;
NATO project "DREAMS-ASIA“ (Development of gRid EnAbling technology in Medicine&Science for Central ASIA);
JINR-Romania cooperation Hulubei-Meshcheryakov programme Project "SKIF-GRID" (Program of Belarussian-Russian Union
State). Project GridNNN (National Nanotechnological Net) We work in close cooperation and provide support to our
partners in Armenia, Belarus, Bulgaria, Czech Republic, Georgia, Germany, Poland, Romania, South Africa, Ukraine, Uzbekistan.
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
Development of the JINR Development of the JINR Grid-environment – 2010-2016Grid-environment – 2010-2016
Network level: links between Moscow and Dubna on the basis of state-of-the-art technologies DWDM and 10Gb Ethernet. JINR Local area network :JINR High-speed backbone construction – 10Gbps
Resource level:requirements of the LHC experiments stimulate the development of a global Grid-infrastructure, together with the resource centers of all the cooperating organizations. First of all, this is of primary concern for such large research centers as the JINR. To reach effective processing and analysis of the experimental data, further increase in the JINR CICC performance and disk space is needed.
Now the RDIG infrastructure comprises 15 Resource Centers with > 7000 kSI2K CPU and > 1850 TB of disc storage.
The RDIG infrastructure
• RDIG Resource Centres:– ITEP– JINR-LCG2– Kharkov-KIPT– RRC-KI– RU-Moscow-KIAM– RU-Phys-SPbSU– RU-Protvino-IHEP– RU-SPbSU– Ru-Troitsk-INR– ru-IMPB-LCG2– ru-Moscow-FIAN– ru-Moscow-GCRAS– ru-Moscow-MEPHI– ru-PNPI-LCG2– ru-Moscow-SINP
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
RDIG monitoring&accounting http://rocmon.jinr.ru:8080
Monitored valuesCPUs - total /working / down/ free / busyJobs - running / waiting Storage space - used / availableNetwork - Available bandwidth Accounting valuesNumber of submitted jobsUsed CPU time
Totally sum in secondsNormalized (with WNs productivity)Average time per job
Waiting timeTotally sum in secondsAverage ratio waiting/used CPU time per job
Physical memoryAverage per job
JINR CICC
• Monitoring – allows to keep an eye on parameters of Grid sites' operation in real time
• Accounting - resources utilization on Grid sites by virtual organizations and single users
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
Production Normalised CPU time per EGEE Regions (June-August 2009)
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
RDIG Normalised CPU time per Site and per VO
(June-September 2009)
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR) 24
Production Normalised CPU Production Normalised CPU time per EGEE site for VO time per EGEE site for VO LHC (June – August 2009)LHC (June – August 2009)
GRID-site CPU time Num CPU time/N------------------------------------------------------------------------FZK-LCG 6,039,304 8620 700INFN-T1 3,493,208 2862 1220CERN-PROD 3,309,795 6812 485CYFRONET-LCG 3,198,111 2384 1341PIC 3,037,977 1337 2272JINR 2,938,039 960 3060IN2P3-CC 2,668,914 4544 587RAL-LCG2 2,629,889 2532 1038UKI-GLASGOW 2,610,116 1912 1365UKI-LT2-IC-HEP 2,533,230 960 2638
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
JINR-LCG2 in RDIG
JINR - 40% of total RDIG
CPU time
Normalised CPU time per VO at JINR site
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
Further participation in the WLCG project
Support and development of the JINR WLCG-segment Support and development of the JINR WLCG-segment
in frames of global WLCG infrastructure;in frames of global WLCG infrastructure; Participation in Service/Data Challenges in coordination Participation in Service/Data Challenges in coordination
with LHC experiments;with LHC experiments; WLCG middleware testing/evaluation;WLCG middleware testing/evaluation; Grid-monitoring the LCG-infrastructure at JINR and Grid-monitoring the LCG-infrastructure at JINR and
other sites of the Russian Tier2 cluster; other sites of the Russian Tier2 cluster; Evaluation of new Grid technologies in context of their Evaluation of new Grid technologies in context of their
usage in WLCG; usage in WLCG; MCDB development (data base structure; basic MCDB development (data base structure; basic
modules; modules; Participation in ARDA activities in coordination with Participation in ARDA activities in coordination with
experiments;experiments; JINR WLCG web-portal improvement; JINR WLCG web-portal improvement; User’s support to stimulate their active usage of WLCG User’s support to stimulate their active usage of WLCG
resources (courses, lectures, trainings, publication of resources (courses, lectures, trainings, publication of user guides in Russian); user guides in Russian);
Support of JINR member states in WLCG activities.Support of JINR member states in WLCG activities.
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
Applied level developments of the Applied level developments of the JINR Grid-environmentJINR Grid-environment (I)(I)
1. The applied level covers the user applications working in a virtual organization (VO) environment which comprises both users and owners of computing resources.
2. In the existing Grid-systems, a VO defines a collaboration of specialists in some area, who combine their efforts to achieve a common aim.
3. The virtual organization is a flexible structure that can be formed dynamically and may have a limited life-time. VOs working within the WLCG project are the VOs on the LHC VOs working within the WLCG project are the VOs on the LHC
experiments - ATLAS, CMS, Alice, LHCb, the first three being experiments - ATLAS, CMS, Alice, LHCb, the first three being carried out with the noticeable and direct participation of the carried out with the noticeable and direct participation of the JINR. Nowadays, as a Grid-segment of the EGEE/RDIG, the JINR JINR. Nowadays, as a Grid-segment of the EGEE/RDIG, the JINR CICC supports computations of the virtual organizations CICC supports computations of the virtual organizations registered in RDIG ( LHC experiments, BioMed, PHOTON, eEarth, registered in RDIG ( LHC experiments, BioMed, PHOTON, eEarth, Fusion, HONE, Panda.Fusion, HONE, Panda.
In the future, as the interest arises at a large-scale level, VOs In the future, as the interest arises at a large-scale level, VOs can be organized at JINR in the fields of nuclear physics and can be organized at JINR in the fields of nuclear physics and condensed matter physics and, most probably, in the new condensed matter physics and, most probably, in the new promising direction related to the research of the nanostructure promising direction related to the research of the nanostructure properties.properties.
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
The creation of new VOs gets The creation of new VOs gets possible and necessary under possible and necessary under maturation of the algorithmic maturation of the algorithmic approaches to the problem solution, approaches to the problem solution, the development of corresponding the development of corresponding mathematical methods and tools:mathematical methods and tools:
methods and tools for simulation of methods and tools for simulation of physical processes and analysis of physical processes and analysis of experimental data software and experimental data software and computer complexes for computer complexes for experimental data processing;experimental data processing;
numerical methods, algorithms and numerical methods, algorithms and software for modeling complex software for modeling complex physical systems; physical systems;
methods, algorithms and software of methods, algorithms and software of computer algebra;computer algebra;
new computing paradigmsnew computing paradigms;; adaptation of specialized software for adaptation of specialized software for
solving problems within the Grid-solving problems within the Grid-environment.environment.
Applied level developments of the Applied level developments of the JINR Grid-environmentJINR Grid-environment (II)(II)
V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)
Formation of the Grid environment of the Formation of the Grid environment of the JINR-Member StatesJINR-Member States
Creation of the Grid environment of the JINR-participating countries Creation of the Grid environment of the JINR-participating countries will allow to bring out the research underway at JINR in the field of will allow to bring out the research underway at JINR in the field of high energy physics, nuclear physics, solid state and condensed high energy physics, nuclear physics, solid state and condensed matter physics, biophysics, biosensor (nano)technologies, etc., on a matter physics, biophysics, biosensor (nano)technologies, etc., on a qualitatively new level. qualitatively new level.
Basic directionsBasic directions:: Creation of a working group of the representatives of the JINR-Creation of a working group of the representatives of the JINR-
participating countries for the development of the main directions of participating countries for the development of the main directions of the activity toward forming a joint Grid environment.the activity toward forming a joint Grid environment.
Definition, together with the JINR Member States, of the basic Definition, together with the JINR Member States, of the basic directions of activity toward forming the joint Grid environment.directions of activity toward forming the joint Grid environment.
Preparation of corresponding multi-side agreements.Preparation of corresponding multi-side agreements. Development of the network level of the Grid environment of the Development of the network level of the Grid environment of the
JINR Member States.JINR Member States. Development of the resource level of the Grid environment of the Development of the resource level of the Grid environment of the
JINR Member States.JINR Member States. Formation of the applied level of the Grid environment of the JINR Formation of the applied level of the Grid environment of the JINR
Member States.Member States. Preparation and implementation of joint projects.Preparation and implementation of joint projects.
V.Ivanov (LIT, JINR)V.Ivanov (LIT, JINR)
Thank you for your attention !Thank you for your attention !Thank you for your attention !Thank you for your attention !