Upload
lenguyet
View
218
Download
3
Embed Size (px)
Citation preview
Southeast University and AMS-02 Experiment
Fang Dong
School of Computer Science and Engineering, Southeast University
The 14th International Conference on Cryptology and Network Security
(CANS 2015)
n SEU’s contribution to AMS
n Conclusion and future work
n Background of AMS
Outline
2
~90% of the matter in the
Universe is not visible, called Dark Matter
Collisions of Dark Matter will yield characteristic distributions
of charged cosmic rays which can be detected by AMS.
The Origin of Dark Matter ~ 90% of Matter in the Universe is not visible and is called Dark Matter
A Galaxy as seen by telescope If we could see Dark Matter in the Galaxy
AMS is the only large-scale physics experiment that directly searches for dark matter in the space
Alpha Magnetic Spectrometer
4
Help scientists find the source of the dark matter and measuring cosmic ray sources as well as antimatter universe
USA A&M FLORIDA UNIV. JOHNS HOPKINS UNIV. MIT - CAMBRIDGE NASA GODDARD SPACE FLIGHT CENTER NASA JOHNSON SPACE CENTER UNIV. OF MARYLAND-DEPRT OF PHYSICS UNIV. OF MARYLAND-E.W.S. S.CENTER YALE UNIV. - NEW HAVEN
MEXICO UNAM
DENMARK UNIV. OF AARHUS
FINLAND HELSINKI UNIV. UNIV. OF TURKU
FRANCE GAM MONTPELLIER LAPP ANNECY LPSC GRENOBLE
GERMANY RWTH-I RWTH-III MAX-PLANK INST. UNIV. OF KARLSRUHE
ITALY ASI CARSO TRIESTE IROE FLORENCE INFN & UNIV. OF BOLOGNA INFN & UNIV. OF MILANO INFN & UNIV. OF PERUGIA INFN & UNIV. OF PISA INFN & UNIV. OF ROMA INFN & UNIV. OF SIENA
NETHERLANDS ESA-ESTEC NIKHEF NLR PORTUGAL LAB. OF INSTRUM. LISBON
ROMANIA ISS UNIV. OF BUCHAREST
RUSSIA I.K.I. ITEP KURCHATOV INST. MOSCOW STATE UNIV.
SPAIN CIEMAT - MADRID I.A.C. CANARIAS.
SWITZERLAND ETH-ZURICH UNIV. OF GENEVA
中 国 KOREA EWHA KYUNGPOOK NAT.UNIV.
中科院电⼯工所 中科院⾼高能所 东南⼤大学 上海交⼤大 中⼭山⼤大学 ⼭山东⼤大学 航天部⼀一院 航天部五院 中⼭山科学院 中央研究院 中央⼤大学 成功⼤大学 太空计划室 交通⼤大学
AMS is US Dept of Energy (DOE) led International Collaboration 16 Countries and regions, 60 Institutes and 600 Physicists, 17 years
6 5m x 4m x 3m
7.5 tons Silicon layer
7 Silicon layers
TRD
TOF 1, 2
TOF
RICH
ECAL
Magnet
300,000 electronic channels 650 microprocessors
Radiators 11,000 Photo Sensors
Silicon layer
6
STS-134 launch May 16, 2011 @ 08:56 AM
May 19: AMS installation completed at 5:15 CDT, start taking data 9:35 CDT
Huntsville, AL
Regional SOC Ø Data storage Ø Data reconstruction Ø Monte Carlo production Ø Data analysis
SEU RWTH … CIEMAT Milano
POCC Ø AMS Flight Status Monitoring
POCC Ø AMS Flight Status Monitoring
SOC Ø Data reconstruction Ø Data analysis Ø Data distribution Ø Monte Carlo production
CERN JSC
太空粒⼦子束
Tracking and Data Relay Satellite,TDRS
Marshall Space Flight
Center
KU band
AMS Data Flow
AMS Scientific Operation Center
9
Sim ulation D ata
D ata R econstruction
AMS-02Package
C alibration
D etector Sim ulation
R aw D ataD A Q D ata
R econstruction
GEANTPhysics A nalysis
ROOT
R esults
V isulization
M C Sim ulation
q Data storage、data reconstruction、Monte Carlo simulation、physics analysis、massive data transfer
Missions of AMS-SOC
10
n SEU’s contribution to AMS
n Conclusion and future work
n Background of AMS
Outline
11
Starting its collaboration with AMS from 2002, SEU is the first Chinese university to join in AMS experiment.
AMS 1. AMS-C
2. AMS-AIS
SEU
USA MIT - CAMBRIDGE NASA GODDARD SPACE FLIGHT CENTER NASA JOHNSON SPACE CENTER NASA KENNEDY SPACE CENTER FLORIDA STATE UNIVERSITY (NAT. HIGH MAG. FIELD LAB) TEXAS A&M UNIVERSITY UNIV. OF MARYLAND YALE UNIV.
- NEW HAVEN
MEXICO UNAM
DENMARK UNIV. OF AARHUS
FINLAND HELSINKI UNIV. UNIV. OF TURKU
FRANCE GAM MONTPELLIER LAPP ANNECY LPSC GRENOBLE
GERMANY RWTH - I RWTH - III MAX - PLANK INST. UNIV. OF KARLSRUHE
ITALY ASI CARSO TRIESTE IROE FLORENCE INFN & UNIV. OF BOLOGNA INFN & UNIV. OF MILANO INFN & UNIV. OF PERUGIA INFN & UNIV. OF PISA INFN & UNIV. OF ROMA INFN & UNIV. OF SIENA
NETHERLANDS ESA - ESTEC NIKHEF NLR
ROMANIA ISS, UNIV. OF BUCHAREST
RUSSIA I.K.I. ITEP KURCHATOV INST. MOSCOW STATE UNIV.
SPAIN CIEMAT - MADRID I.A.C. CANARIAS.
SWITZERLAND ETH - ZURICH UNIV. OF GENEVA
CHINA
KOREA EWHA
KYUNGPOOK NAT.UNIV.
PORTUGAL LAB. OF INSTRUM. LISBON
SINICA (Taiwan) AIDC (Taiwan) CSIST(Taiwan) NCU (Chung Li) NCKU (Tainan) NCTU(Hsinchu) NSPO(Hsinchu)
BISEE (Beijing) IEE (Beijing) IHEP (Beijing) SJTU (Shanghai) SEU (Nanjing) SYSU (Guangzhou) SDU (Jinan)
NLAA (Beijing) SEU JSC
CERN
AMS-SOC@SEU
AMS-C@SEU
12
13
• SEU’s Collaboration with AMS and MIT lasts for more than a decade
• ~40 SEU Professors/Assistant Professors/students visited MIT
• MIT faculties including Nobel laureate Professor Samuel Ting、Research Professor Vitaly Choutko visited SEU regularly in the past few years
Prof. Samuel Ting visited SEU
Talents exchange between SEU and AMS
Talents exchange between SEU and AMS
14
Prof. Li Qi at MIT Prof. Luo Junzhou at NASA
Prof. Vitaly Choutko at SEU Discussions at CERN
High-tech breakthroughs at SEU
15
AMS-02 silicon trackers AMS-02 Control System
AMS-C at SEU Prof. Samuel Ting at SEU
Starting its collaboration with AMS from 2002, SEU is the first Chinese university to join in AMS experiment.
AMS 1. AMS-C
2. AMS-AIS
3. AMS-SOC
SEU
USA MIT - CAMBRIDGE NASA GODDARD SPACE FLIGHT CENTER NASA JOHNSON SPACE CENTER NASA KENNEDY SPACE CENTER FLORIDA STATE UNIVERSITY (NAT. HIGH MAG. FIELD LAB) TEXAS A&M UNIVERSITY UNIV. OF MARYLAND YALE UNIV.
- NEW HAVEN
MEXICO UNAM
DENMARK UNIV. OF AARHUS
FINLAND HELSINKI UNIV. UNIV. OF TURKU
FRANCE GAM MONTPELLIER LAPP ANNECY LPSC GRENOBLE
GERMANY RWTH - I RWTH - III MAX - PLANK INST. UNIV. OF KARLSRUHE
ITALY ASI CARSO TRIESTE IROE FLORENCE INFN & UNIV. OF BOLOGNA INFN & UNIV. OF MILANO INFN & UNIV. OF PERUGIA INFN & UNIV. OF PISA INFN & UNIV. OF ROMA INFN & UNIV. OF SIENA
NETHERLANDS ESA - ESTEC NIKHEF NLR
ROMANIA ISS, UNIV. OF BUCHAREST
RUSSIA I.K.I. ITEP KURCHATOV INST. MOSCOW STATE UNIV.
SPAIN CIEMAT - MADRID I.A.C. CANARIAS.
SWITZERLAND ETH - ZURICH UNIV. OF GENEVA
CHINA
KOREA EWHA
KYUNGPOOK NAT.UNIV.
PORTUGAL LAB. OF INSTRUM. LISBON
SINICA (Taiwan) AIDC (Taiwan) CSIST(Taiwan) NCU (Chung Li) NCKU (Tainan) NCTU(Hsinchu) NSPO(Hsinchu)
BISEE (Beijing) IEE (Beijing) IHEP (Beijing) SJTU (Shanghai) SEU (Nanjing) SYSU (Guangzhou) SDU (Jinan)
NLAA (Beijing) SEU JSC
CERN
AMS-SOC@SEU
AMS-C@SEU
16
Docker
q SEU’ data center dedicated to AMS experiment q 5500 CPU cores (70 TFLOPS at peak) and 920TB storage.
q Docker 、OpenStack、Hadoop q 1Gb/s dedicated network between SEU and CERNET Beijing
gateway, 2.5Gb between CERNET and CERN
q Cloud based AMS-SOC for better supporting AMS data storage、computing and data transfer
AMS-02 SOC at SEU
17
…… ……
4 × IBM 42U Rack40Gbps QDR
Infiniband Switch
1Gpbs Ethernet Switch
IBM X3850 X5
IBM X3550 M3
IBM H22 Blade Server
IBM X3650 M3
8 × IBM 42U RackFC SAN Switch
IBM DS5300
To CERNET Beijing Exchange Point
40Gbps Infiniband 1Gbps Ethernet
8Gbps Fiber Channel 1Gbps Dedicated LinkTo SEU Campus Network
18
SEU’s AMS Data Transfer Network
Guangzhou
NanjingSEU
Shanghai
Wuhan
NLAACERN
10GPOS
2.5GPOS
Tier 1 Tier 2
1GPOS Dedicated for AMS
IPv6 Link
BeijingCNGI-6IX
Topology of China Education and Research NETwork(CERNET) Backbone
19
A Glance at SEU’s AMS-SOC (cost: RMB 25M)
Prof. Ting and the MIT delegation visited SEU’s AMS-SOC to discuss the data analysis solutions
20
Sim ulation D ata
D ata R econstruction
AMS-02Package
C alibration
D etector Sim ulation
R aw D ataD A Q D ata
R econstruction
GEANTPhysics A nalysis
ROOT
R esults
V isualization
M C Production
Data processing at SEU’s AMS SOC
21
CERN
BladeCenter
1 2 3 4 5 6 7 8 9 10 11 12 13 14
BladeCenter
1 2 3 4 5 6 7 8 9 10 11 12 13 14
BladeCenter
1 2 3 4 5 6 7 8 9 10 11 12 13 14
…
…
Database
Data Storage
c01 c12 c17
1. MC Job request
LSF Server
2. MC Job/Env. download
4. Env. update
3. Job/Env. registration
5. Job submit
6. production
7. status update
7. status update
8. data transfer
Data Transfer Servers
Ser
OS
Ser
OS
Ser
OS
Ser
OS
BladeCenter
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Cluster(VMs)
c18
HPC Cluster(Physical nodes)
BladeCenter
1 2 3 4 5 6 7 8 9 10 11 12 13 14
c02
BladeCenter
1 2 3 4 5 6 7 8 9 10 11 12 13 14
c16
BladeCenter
1 2 3 4 5 6 7 8 9 10 11 12 13 14
BladeCenter
1 2 3 4 5 6 7 8 9 10 11 12 13 14
…
…
Facility for AMS MC production:HPC Cluster(168 physical servers) and cloud based VM Cluster(more than180 instances)
3 4REC jobs downlad copy REC jobs
21
http://pcamss0.cern.ch/rc.html
IBM Cluster@SEUSugon Cluster@SEU
5
Remotely transfer jobs
v5.00mcscripts.tar.gzv5.00mcdb.tar.gz v5.00mcdb.addon.tar.gz
v5.00mcscripts.tar.gzv5.00mcdb.tar.gz v5.00mcdb.addon.tar.gz
Filemonitor &Database6AMS reconstruction jobs
v5.00mcscripts.tar.gzv5.00mcdb.tar.gz v5.00mcdb.addon.tar.gz
Remotely transfer data
Facility for AMS data reconstruction :HPC Cluster(168 physical servers), 1Gb/s network between SEU and CERN
AMS userLogin node
Login
Physics analysis jobs
Job submission
SEU’s AMS SOC
Computing resources
Data analysis
Job scheduling
Software resources
Analysis results
168 IBM HS22 blade
servers
LSF ROOT/Geant/...
42 IBM Flex240 blade
servers
Data source: http://ams.cern.ch/ProdPlot/index.php
Data processing at SEU’s AMS SOC
SEU
Data Production at SEU CPU hours spent at SEU
q 696TB data are produced at SEU, including 138TB Raw data,368TB Monte Carlo data and 190TB Reconstructed data
q 1.53M CPU hours are spent at SEU: 1st place in 6 regional AMS SOC
22
• Data transfer architecture
– CERNèSEU( Raw Data ) • client:LFTP • server:VSFTP • Request: 200GB/day
23
Data transfer at SEU’s AMS SOC
*.*.23.242
Base server at CERN
CERN SEU
Server(VSFTP)
Upload
Client(LFTP)
*.*.23.242
Base server at CERN
CERN SEU
Server(VSFTP)client
(LFTP)
download
*.*.23.241
*.*.23.240
• Data transfer architecture – SEUèCERN( MC Simulation Data / Reconstructed Data )
• client:LFTP • server:VSFTP • Request: 1~2TB/day
24
Data transfer at SEU’s AMS SOC
• Data transfer improvement by optimizing – two parameters: the size of socket buffer at both sending and
receiving sides; the number of parallel connections – Increasing the socket buffer size of the sending and receiving
side will help to improve the data transfer performance – Increasing the number of parallel connections does not always
increase the transfer performance
25
Data transfer at SEU’s AMS SOC
Transfer bandwidth with different number of parallel connections
Transfer time and packet loss rate with different number of parallel connections
26
Data transfer at SEU’s AMS SOC
Data Transfer Status
AMS data transfer reaches 400Mbit/sec after optimization.
27
Data transfer at SEU’s AMS SOC
SEU is able to transfer more than 1 TB AMS data to CERN per day
Sep 10 2015 Aug 10 2015 Jul 10 2015 Jun 10 2015 May 10 2015 0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Dai
ly tr
ansf
erre
d da
ta s
ize
(TB
)
Date
Cloud supporting platform layer
Hardwares in DC
Virtualization Elastic Resource Management
Big data storage and management layer
Pre-processing
Data processing layerGraph
ProcessingStreaming data
processingData Query
AMS application layerData
ReconstructionData AnalysisMC Simulation
Placement Migration
N etw ork R esources
C om puting R esources
StorageR esources
28
Researches on AMS big data processing
29
Cloud based AMS big data processing platform
…… ……Hardwares
in DC
Raw data
Ser
OS
Ser
OS
Task Scheduler
Map/Reduce
AMS-SOC portalSEU usersAMS users
MC Simulation Data Reconstruction Data Analysis
AMS applications
SCSI/SATA
TAPE
Data Storage
……
Data Placement
Data Migration
SSD
Big data processing
Data Query Data Analysis
Streaming Data Processing
Elastic Resource
ManagementOpenstack Ceph
o
Big data processing platformsHadoop Yarn/Spark/Tez/Storm
n SEU’s contribution to AMS
n Conclusion and future work
n Background of AMS
Outline
30
Conclusion
31
q After these 13 years collaborations, we have obtained some fruitful achievements Ø Successfully construct AMS-C, AMS-AIS and
AMS-SOC at SEU Ø Process and transfer more than 700TB
scientific data Ø Breakthrough several key technologies Ø Exchange more than 40 teachers and students
Future Work
32
q AMS-02 will continuously produce massive data(~4PB in next 10 years of AMS operation), including : Ø raw data (~40TB/year) Ø reconstructed data(~200TB/year) Ø MC data(~200TB/year)
q Big data processing techniques need to be further employed including: Ø Geo-distributed data analysis among multiple DCs Ø High performance data transfer between AMS-SOCs
Thank You !