View
229
Download
0
Category
Preview:
Citation preview
System-Level Heterogeneity
with Intel® Xeon PhiTM Processors
Estela Suarez
Jülich Supercomputing Centre
This project has received funding from the European Union's Seventh Framework Programme for research, technological
development and demonstration under grant agreements 287530 (DEEP) and 610476 (DEEP-ER).
Collaborative R&D in DEEP & DEEP-ER
European Union Exascale projects 20 partners Total budget: 28.3 M€ EU-funding: 14.5 M€ Combined term: 5 years
2 www.deep-project.eu www.deep-er.eu
Visit us @ ISC 2016, Frankfurt (Germany)
June 19 – 23, 2016 Booth #1340
DEEP-ER – ISC 2016, Intel Booth – 21.06.2016
DEEP/-ER Approach
Co-design between applications, system SW and HW
– Application and operational requirements do shape system architecture
– HW provides performance, scalability and energy efficiency
– System SW enables applications to leverage HW potential
Objectives
– Deliver highest scalability and workload performance
– Provide leading energy efficiency (energy per result)
– Offer familiar, easy to use programming environment and APIs based on standards
– Ensure sustainability of system architecture and SW
Design elements
– Exploit benefits of processor heterogeneity
– Leverage technology advances in storage-class memory and interconnects
DEEP-ER – ISC 2016, Intel Booth – 21.06.2016 3
Co-Design Applications
DEEP + DEEP-ER applications
• Brain simulation (EPFL)
• Space weather simulation (KULeuven)
• Climate simulation (CYI)
• Computational fluid engineering (CERFACS)
• High temperature superconductivity (CINECA)
• Seismic imaging (CGG)
• Human exposure to electromagnetic fields (INRIA)
• Geoscience (BADW-LRZ)
• Radio astronomy (Astron)
• Oil exploration (BSC)
• Lattice QCD (UREG)
Goals
• Drive co-design cycle
• Evaluate and validate system prototypes
DEEP-ER – ISC 2016, Intel Booth – 21.06.2016 4
System-Level Heterogeneity
DEEP-ER – ISC 2016, Intel Booth – 21.06.2016 5
Accelerated Cluster Cluster-Booster Architecture
– Fixed, static ratio and assignment of accelerators to CPUs
– Static management of resources
– Accelerators do not act autonomously
– General-purpose Cluster interconnect
– Programming via local offload interfaces (OpenCL, CUDA, CELO, OpenACC, …)
– No fixed ratio or assignment between
resources (Multicore & Manycore nodes)
– Dynamic management and association
of resources
– High-throughput network in the Booster
– Programming via MPI and “global”
tasking interfaces
DEEP System Architecture
DEEP-ER – ISC 2016, Intel Booth – 21.06.2016 6
Cluster part – High single thread & complex code
performance Intel® Xeon® processors
– High any-to-any connection performance & infrastructure integration standard switched HPC fabric (InfiniBandTM)
Booster part – High throughput, autonomous
operation Intel Xeon Phi coprocessor (codenamed “Knights Corner”)
– Need for low latency, spatial application structures 3D Torus direct-connected network (EXTOLL)
– Network bridging and KNC control Booster Interface layer
Both parts – Efficiency needs use of liquid cooling
& dense packaging
DEEP Prototype Systems
Eurotech Aurora Prototype
DEEP-ER – ISC 2016, Intel Booth – 21.06.2016 7
Cluster part 128 dual-socket
Intel Xeon E5-2403
nodes
QDR InfiniBandTM
Eurotech Aurora
liquid cooling &
packaging
Booster part 384 Intel Xeon Phi
7120X nodes
FPGA
implementation of
EXTOLL
interconnect
24 Booster interface
nodes with Intel
Xeon processor
Eurotech Aurora
liquid cooling &
packaging Cluster
Booster
DEEP/DEEP-ER Programming Model
DEEP-ER – ISC 2016, Intel Booth – 21.06.2016 8
ParaStation Global MPI layer
Expert-level programming
Efficient communication across the whole system
Dynamic process spawning an control in both
directions
Task-based OmpSs programming model
Pragma based, emphasizes ease of use
Efficient communication across the whole system
Dynamic spawning of massively parallel tasks in
both parts
Massively Parallel Tasks in OmpSs
DEEP-ER – ISC 2016, Intel Booth – 21.06.2016 9
Published in: Sainz, F, Bellón, J, Beltran, V, Labarta, J, “Collective Offload for Heterogeneous Clusters”, IEEE 22nd International Conference on High Performance
Computing (HiPC), 2015
Rank 0
master
Rank 0-15
slave0
Rank 0
Worker Rank 1
Worker Rank 2
Worker Rank 3
wk63
Rank 16-31
slave1
Rank 0
Worker Rank 1
Worker Rank 2
Worker Rank 3
wk127
Rank 240-255
slave15
Rank 0
Worker Rank 1
Worker Rank 2
Worker Rank 3
wk1023
x16
x16
x16
Figure 7: FWI hierarchical MPI architecture
64
128
256
512
1024
64 128 256 512 1024
Sp
ee
d-u
p
# nodes (16 cores)
IdealOmpSs Offload
OmpSs Offload (no I/O)
Figure 8: Scalability of FWI application on up to 1024 nodes
VI. Concl usions and fut ur e wor k
This paper presents the OmpSs O✏oad model that was
originally developed to ease the porting of complex ap-
plications to the highly heterogeneous cluster architecture
proposed on the DEEP Exascale project. The OmpSs O✏oad
model has completely fulfilled its design goals, combining
the ease of use of Intel O✏oad with the flexibility, per-
formance and scalability of the native MPI Comm spawn
API. Moreover, our approach is fully integrated with the
rest of features provided by OmpSs, such as support for
OpenMP codes and CUDA or OpenCL kernels. Although
it was originally conceived for heterogeneous clusters we
have also successfully used it to develop hierarchical MPI
applications such as FWI. We think that these hierarchical
MPI architectures will play an important role in exploiting
future Exascale systems. Hence, tools such as OmpSs Of-
fload will be essential for designing such architectures and
helping with their implementation for complex and large
applications.
As future work, we plan to integrate our allocation API
with a resource manager/job scheduler to avoid the need
to reserve all the resources that will be required before
the program is launched. We also plan to investigate the
potential of OmpSs O✏oad to improve the malleability of
existing MPI applications, as well as the implications of
using this o✏oad model from the resilience point of view.
Refer ences
[1] D. A. Mallon, N. Eicker, M. E. Innocenti, G. Lapenta, T. Lip-pert, and E. Suarez, “On the scalability of the clusters-boosterconcept: a critical assessment of the DEEP architecture,” inProceedings of the Future HPC Systems: the Challenges ofPower-Constrained Performance. ACM, 2012, p. 3.
[2] A. Duran, E. Ayguade, R. M. Badia, J. Labarta, L. Martinell,X. Martorell, and J. Planas, “OmpSs: a proposal for pro-gramming heterogeneous multi-core architectures,” ParallelProcessing Letters, vol. 21, no. 02, pp. 173–193, 2011.
[3] K. O. W. Group et al., “The OpenCL specification,” A.Munshi, Ed, 2008.
[4] C. Nvidia, “Compute Unified Device Architecture program-ming guide,” 2007.
[5] C. J. Newburn, R. Deodhar, S. Dmitriev, R. Murty,R. Narayanaswamy, J. Wiegert, F. Chinchilla, andR. McGuire, “O✏oad compiler runtime for the Intel R
Xeon Phi R coprocessor,” in Supercomputing. Springer,2013, pp. 239–254.
[6] “OpenMP 4.0 specification,” http://www.openmp.org/mp-documents/OpenMP4.0.0.pdf, 2013, [Online; accessed 20-Dec-2013].
[7] O. W. Group et al., “TheOpenACC application programminginterface,” 2011.
[8] F. Sainz, S. Mateo, V. Beltran, J. L. Bosque, X. Martorell,and E. Ayguade, “Leveraging OmpSs to exploit hardwareaccelerators,” in 26th IEEE International Symposium onComputer Architecture and High Performance Computing,SBAC-PAD 2014, Paris, France, October 22-24, 2014.IEEE, 2014, pp. 112–119. [Online]. Available: http://dx.doi.org/10.1109/SBAC-PAD.2014.26
[9] J. Duato, A. J. Pena, F. Silla, R. Mayo, and E. S. Quintana-Orti, “ rCUDA: Reducing the number of GPU-based accel-erators in high performance clusters,” in High PerformanceComputing and Simulation (HPCS), 2010 International Con-ference on. IEEE, 2010, pp. 224–231.
[10] A. Barak and A. Shiloh, “Themosix Virtual OpenCLl (VCL)cluster platform,” in Proc. Intel European Research andInnovation Conference, 2011.
[11] F. Sainz and V. Beltran. (2015) OmpSs CollectiveO✏oad. User Manual. [Online]. Available: http://pm.bsc.es/ompss-docs/user-guide/run-programs-archs-o✏oad.html
Measurements for BSC FWI (full waveform inversion) code
From DEEP to DEEP-ER
DEEP-ER – ISC 2016, Intel Booth – 21.06.2016 10
Simplified Interconnect
On-Node NVM
Self-Booting Nodes
Network attached memory
DEEP-ER Scalable I/O
Leverage presence of fast local
NVM storage
– Scalable caching of read/write data
close to requesting node
– Prefetching stages read data into
caches
– Write-back scheme saves data to
permanent storage
– Synchronous (done) and
asynchronous (WIP) versions/APIs
DEEP-ER – ISC 2016, Intel Booth – 21.06.2016 11
DEEP-ER Resiliency Scheme
DEEP-ER – ISC 2016, Intel Booth – 21.06.2016 12
BSC Full Waveform Inversion
Results
Using 60 cores per Xeon Phi coprocessor node with 180 threads
0
2
4
6
8
10
12
020406080
100120140160
Spe
ed
up
Gfl
op
s/s
Impact of different optimizations of wave propagator on Xeon Phi
Gflops/s
speedup
DEEP-ER – ISC 2016, Intel Booth – 21.06.2016 14
INRIA MAXW-DGTD Results
0
1
2
3
4
16 64 256 1024Spe
ed
up
ove
r in
itia
l ve
rsio
n
# of cores
Before
After
0.50
0.75
1.00
1.25
1.50
16 64 256 1024
Par
alle
l Eff
icie
ncy
# of cores
Before
After
Improvements applied below:
• Non-blocking communication
• Renumbering scheme
• Vectorisation and locality
Performance improvement up to 3.3x Almost perfect parallel efficiency now
Setup: - Human head
- DEEP Cluster
- Mesh: 1.8 million cells
- 16 processes per node
- Pure MPI.
- P1 approximation.
15 DEEP-ER – ISC 2016, Intel Booth – 21.06.2016 *Leger R., Alvarez Mallon D., Duran A., Lanteri S., “Assessing the DEEP-ER Cluster/Booster Architecture with a fine-element
type solver for bioelectromagnetics”, Submitted to PARCO2015, Contribution ID: 25. www.parco2015.org
INRIA MAXW-DGTD I/O Results
Inria: Assessment of Human exposure to EM fields
24 MPI processes, 1 thread per process
0
10
20
30
40
50
60
P1 P2 P3 P4
Wri
tin
g ti
me
[s]
I/O performance of MAXW-DGTD
sdv-work NVMe
Increasing
model precision
P1<P2<P3<P4
Performance gain by using Intel DC P 3700
SSDs
DEEP-ER – ISC 2016, Intel Booth – 21.06.2016 16
Partners
DEEP-ER – ISC 2016, Intel Booth – 21.06.2016 17
Recommended