Thomas Gerhold Institute of Aerodynamics and Flow Technology German Aerospace Center (DLR)
DLR CFD Solver TAU & Flucs for external Aerodynamic
Industrial Use of EsDs – ETP4HPC Workshop 22 June 2017 Frankfurt
DLR TAU-Code
t = T
Parallel datamanagement
Dataextraction
Adaptation
(Re-)PartitioningDeformation
Primary grid
Solution
MPICPU-j
CPU-k
CPU-i
Transitionprediction
Preprocessor
Dual grid
Solver
direct (in memory) data access between modules
t = T + ∆t
python interface forscript basedwork flow
t = T
Parallel datamanagement
Dataextraction
AdaptationAdaptation
(Re-)PartitioningDeformationDeformation
Primary gridPrimary grid
SolutionSolution
MPICPU-j
CPU-k
CPU-i
TransitionpredictionTransitionprediction
Preprocessor
Dual grid
Solver
Preprocessor
Dual gridDual grid
SolverSolver
direct (in memory) data access between modules
t = T + ∆t
python interface forscript basedwork flow
python interface forscript basedwork flow
Unstructured 2nd Order Finite Volume compressible CFD solver Used in European aircraft industry, research organization and academia Since more than 15 years > 100 daily/frequently users today
Used e.g. for A380 & A350 wing design
Pre-Flight prediction
1% L/Dmax
Mach
Pre-Flight prediction
1% L/Dmax
MachSource: Airbus
Numerical Simulation of Aircraft Aerodynamics Status Cruise
CFD-based scaling
W/T-based scaling A380 Performance
CFD & wind tunnel based performance prediction
are comparable
flight test
Europe’s Vision for Aviation
Maintaining Global Leadership & Serving Society’s Needs
ACARE 2020 / Flightpath 2050
Goals (relative to typical aircraft in 2000)
CO2 emissions reduced by 75% NOx emissions reduced by 90% 65% reduction in perceived
aircraft noise
-ACARE: Advisory Council for Aeronautics Research in Europe
Consequence Heavy demands on future product performance Step changes in aircraft technology required New design principles mandatory
Future aircraft Design may be driven by unconventional layouts Flight characteristics may be dominated by
non-linear effects
Numerical Simulation Key Enabler for Future Aircraft Design
High-fidelity methods indispensible for design & assessment of step changing aircraft
Reliable insight to new aircraft technologies Comprehensive sensitivity analysis with
risk & uncertainty management Best overall aircraft performance through integrated
aerodynamics / structures / systems design Consistent and harmonized aerodynamic and
aero-elastic data across flight envelope
Further improvement of simulation capability necessary
Numerical Simulation for Aircraft Design Current Status
Computational Fluid Dynamics (CFD) has significantly evolved over the last 30 years
Mature tool for configurations at their design point in flight envelope
Complementary to wind tunnel testing and flight tests Key tool for aeronautical research and
aircraft technology development Total potential not yet fully exploited:
full flight envelope, all relevant disciplines, multidisciplinary optimization
Numerical Simulation for Aircraft Design Vision: Digital Aircraft
Challenge: consider algorithmic and parallel efficiency and trade off
Collaboration of experts in application domain and parallel computing necessary to extract – or even generate – data parallelism in algorithms
+ Related research and prototypic Implementations done in the recent years
algorithmic efficiency
parallel efficiency
HPC for Aircraft Design
Next Generation Solver Flucs Activity started mid of 2012, supported by several prototype codes
DLR CFD Code Strategy
Flucs – what is new New code design – started from scratch C++ 11 / Python
Finite Volume & HO Discontinous Galerkin & zonal coupling
Focus on algorithmic efficiency: use of strong implicit solvers Focus on HPC
GASPI / MPI + OpenMP (2 level parallelization)
Sub-partitions per node/socket are assigned to threads, thread- synchronization avoided wherever possible
Implementation supports overlap of communication & computation
Flucs – what is new
Airbus accepted Flucs as a basis for their next generation CFD capability further development needed EIS planned ~2020/21
HPC for Aircraft Design Status TAU
Ivy Bridge 2x12 cores
95%
78%
68%
89%
65% 48%
31%
parallel efficiency (vs 24 cores)
80%
50%
63%
73%
scalability limit
~6.5K points/core
TAU: pure MPI Parallelization (typical grid sizes between 20 -100 Mpoints)
4W multigrid vs. SG gain ~factor 5-10
HPC for Aircraft Design Status TAU TAU: pure MPI Parallelization, effect of interconnect (RK 31 mio points)
Aries Benchmark by M. Pütz, Cray Europe
FDR fat tree interconnect
Cray Aries interconnect
-6/29/2017
Exa2ct Slide 15
0
20
40
60
80
100
0 5000 10000 15000 20000 25000
linear
comm_free
mpi_bsync_se_gs
mpi_early_recv_se_gs
mpi_async_se_gs
gaspi_bsync_gs
gaspi_async_gs
mpi_fence_bsync_se_gs
mpi_fence_async_se_gs
mpi_pscw_bsync_se_gs
mpi_pscw_async_se_gs
F6 Full Model 31 mio mesh points, 3 V Multigrid • 1345 mesh points / thread @level 1 • 180 mesh points / thread @level 2 • 24 mesh points / thread @level 3
Strong Scaling
TAU CFD-Proxy Xeon Phi EXA2CT Christian Simmendinger
Flucs Parallel Efficiency – Strong Scaling Scenario 1.9 Mio Elements @ C²A²S²E Cluster
40
80 200 80%
(each with two 12-core IVB EP)
TAU @ 2 mio. points
20
Note: single grid results: no multi grid technique yet implemented
Flucs
Summary / Outlook Flucs
Flucs will become TAU successor for industrial applications
Still important functionality to be implemented: strong implicit solvers, multigrid method, further turbulence models, etc. – This is ongoing.
Increased scalability demonstrated vs. legacy pure MPI parallelization
Further work for improved HPC performance planned, e.g. SIMD usage EIS planned for 2020/21
We have interest to test Flucs on EsDs
Possibly NDA for Flucs solver necessary
Flucs is a python extension – use of dynamic libraries needed