The Effect of InfiniBand In-Network Computing on CAE Simulations · 2020-02-14 ·...

Simulation in the Automotive Industry: Creating the Next Generation Vehiclenafems.org/americas November 14th, 2019 | Troy, MI

The Effect of InfiniBand In-Network

Computing on CAE Simulations

HPC-AI Advisory Council

October 1st, 2019 | Columbus, OH

The HPC-AI Advisory Council

• World-wide HPC non-profit organization

• More than 400 member companies / universities / organizations

• Bridges the gap between HPC-AI usage and its potential

• Provides best practices and a support/development center

• Explores future technologies and future developments

• Leading edge solutions and technology demonstrations

HPC Advisory Council Members

HPC-AI Advisory Council Cluster Center (Examples)

• Supermicro / Foxconn 32-node cluster

• Dual Socket Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz

• Dell™ PowerEdge™ R730/R630 36-node cluster

• Dual Socket Intel® Xeon® 16-core CPUs E5-2697A V4 @ 2.60 GHz

• AMD Daytona_X

• Dual Socket AMD Rome 128 core 8-node cluster @ 2.25GHz

• Lattice QCD

• LAMMPS

• LS-DYNA

• miniFE

• MILC

• MSC Nastran

• MR Bayes

• MM5

• MPQC

• NAMD

• Nekbone

• NEMO

• NWChem

• Octopus

• OpenAtom

• OpenFOAM

• OpenMX

• OptiStruct

• PARATEC

• PFA

• PFLOTRAN

• Quantum ESPRESSO

• RADIOSS

• SNAP

• SPECFEM3D

• STAR-CCM+

• STAR-CD

• VASP

• VSP

• WRF

Multiple Applications Best Practices Published

• Abaqus

• ABySS

• AcuSolve

• Amber

• AMG

• AMR

• ANSYS CFX

• ANSYS FLUENT

• ANSYS Mechanical

• BQCD

• BSMBench

• CAM-SE

• CCSM

• CESM

• COSMO

• CP2K

• CPMD

• Dacapo

• Desmond

• DL-POLY

• Eclipse

• FLOW-3D

• GADGET-2

• Graph500

• GROMACS

• Himeno

• HIT3D

• HOOMD-blue

• HPCC

• HPCG

• HYCOM

• ICON

HPC-AI Advisory Council Activities

• HPC-AI Advisory Council– More then 400 members, http://www.hpcadvisorycouncil.com/

– Application best practices, case studies– Development and benchmarking center with remote access for users– World-wide conferences

• Conferences– USA (Stanford University) – February– Switzerland (CSCS) – April– Student Cluster Competition (ISC) – July– China (HPC China) - August– Australia - August– UK – September– China – November

• Competitions– APAC HPC-AI Competition - March– China - 6th Annual RDMA Competition - May– ISC Germany - Annual Student Cluster Competition - June

• For more information – www.hpcadvisorycouncil.com– info@hpcadvisorycouncil.com

HPC|Works Community

Computing Evolution – Compute Centric to Data Centric

Von NeumannMachine

NeuralNetworks

Compute-Centric Data-Centric

The Need for Intelligent Data Center

CPU-Centric (Onload) Data-Centric (Offload)

Move Data to the ComputeMust Wait for the Data

Creates Performance Bottlenecks

Onload Network

Move Compute to the DataAnalyze Data Everywhere

Higher Performance and Scale

Scalable Hierarchical Aggregation and Reduction Protocol (SHARP)

• Reliable Scalable General Purpose Primitive

• In-network Tree based aggregation mechanism

• Large number of groups

• Multiple simultaneous outstanding operations

• Applicable to Multiple Use-cases

• HPC Applications using MPI / SHMEM

• Distributed Machine Learning applications

• Scalable High Performance Collective Offload

• Barrier, Reduce, All-Reduce, Broadcast and more

• Sum, Min, Max, Min-loc, max-loc, OR, XOR, AND

• Integer and Floating-Point, 16/32/64 bits

DataAggregated

AggregatedResult

Aggregated Result

Host Host Host Host Host

SwitchSwitch

Switch

SHARP AllReduce Performance Advantages (128 Nodes)

SHARP AllReduce Performance Advantages 1500 Nodes, 60K MPI Ranks, Dragonfly+ Topology

The Niagara Supercomputer – University of Toronto

OpenFOAM

• Toolbox in an open source CFD applications that can simulate– Complex fluid flows involving

– Chemical reactions

– Turbulence

– Heat transfer

– Solid dynamics

– Electromagnetics

– The pricing of financial options

OpenFOAM Profiling – MPI/User Time Ratio

• OpenFOAM simpleFOAM solver uses mainly non-blocking communications

• 23% of overall runtime spent on MPI communication at 16 nodes / 640 MPI cores

• Both Intel MPI and HPC-X spent the same time in overall runtime on MPI communications

• Overall of MPI time spent in MPI non-blocking communications (MPI_Waitall 47%, MPI_Isend,

• Most of the MPI calls made by OpenFOAM are MPI_Waitall

OpenFOAM Profiling – MPI Time

• MPI profiler shows the type of underlying MPI network communications

– Majority of communications occurred are non-blocking communications

• Majority of the MPI time is spent on non-blocking communications at 32 nodes

– MPI_Waitall (11% wall), 8-byte MPI_Recv (1.4% wall), 1-byte MPI_Recv (0.7% wall)

– Only 14% of the overall runtime is spent on MPI communications at 32-nodes (when EDR

is used)

OpenFOAM Profiling – MPI Communication Topology

• Communication topology shows communication patterns among MPI ranks

• MPI processes mainly communicates with neighbors, but also shows some other patterns

32 Nodes16 Nodes8 Nodes4 Nodes

OpenFOAM Performance E5-2697A v4 @ 2.60GHz, HDR100

OpenFOAM Performance Using (HPC-X 2.5 MPI)

LS-DYNA

• LS-DYNA – A general purpose structural and fluid analysis simulation software

package capable of simulating complex real world problems

– Developed by the Livermore Software Technology Corporation (LSTC)

• LS-DYNA used by – Automobile

– Aerospace

– Construction

– Military

– Manufacturing

– Bioengineering

2019, 28 - 29 October 35th INTERNATIONAL CAE CONFERENCE AND EXHIBITION

LS-DYNA PerformanceIntel Xeon Gold 6138 CPU 2.00GHz , HDR100

ANSYS Fluent

• Computational Fluid Dynamics (CFD)

– Enables the study of the dynamics of things that flow

– Enable better understanding of qualitative and quantitative physical phenomena in the flow which is used to improve engineering design.

• CFD brings together a number of different disciplines

– Fluid dynamics, mathematical theory of partial differential systems, computational geometry, numerical analysis, Computer science.

• ANSYS FLUENT is a leading CFD application from ANSYS

– Widely used in almost every industry sector and manufactured product.

ANSYS FluentE5-2697A v4 @ 2.60GHz, HDR100

ANSYS Fluent E5-2697A v4 @ 2.60GHz, HDR100

InfiniBand QoS

• IBTA Standard

• Application SL -> VL mapping

• WWR / Strict Priority setting

QoS LS-DYNA test• Run the test with no background traffic, on 4 nodes

• Add massive background traffic but without enabling

any QoS; both the application and massive

background traffic use the same SL and the same

network resource.

• Add massive background traffic, but with enabling

QoS, and setting a priority to the LS-DYNA application

over the background traffic. 2019, 28 - 29 October 35th INTERNATIONAL CAE CONFERENCE AND EXHIBITION

LS-DYNA QoSE5-2697A v4 @ 2.60GHz, HDR100

Summary• HPC cluster environments impose high demands on connectivity throughput and low latency

with low CPU overhead, network flexibility, and high efficiency

• Fulfilling these demands enables the maintenance of a balanced system that can achieve high application performance and high scaling

• With the increase in number of CPU cores and application threads, there is a need to develop a new HPC cluster architecture - a data-focused architecture

• The Co-Design collaboration enables the development of In-Network Computing technology that breaks the performance and scalability barriers

• The OpenFOAM, LS-DYNA and ANSYS Fluent applications were benchmarked over AMD Rome and Intel CPUs for this study to demonstrate the advantages of In-Network Computing technology

• We have witness 50% performance advantage and linear scalability with InfiniBand In-Network Computing technology

• InfiniBand QoS can be considered in network design. By enabling QoS, we can achieve similar performance for LS-DYNA, with and without the background noise.

All trademarks are property of their respective owners. All information is provided “As-Is” without any kind of warranty. The HPC Advisory Council makes no representation to the accuracy and completeness of the information

contained herein. HPC Advisory Council undertakes no duty and assumes no obligation to update or correct any information presented herein

Thank You

The Effect of InfiniBand In-Network Computing on CAE Simulations · 2020-02-14 ·...

Documents

Ethernetv infiniband

Seminar on Infiniband

Sonderausgabe CAE Stellenmarkt - nafems.org · Altair steht seit mehr als 25 Jahren für Innovation und ... und des Datenmanagements im Umfeld der numerischen Simulation ... - Anforderungsmanagement

Infiniband Oracle

Delivering Application Performance with Oracle's ... · EoIB (Ethernet over InfiniBand) — The Ethernet over InfiniBand protocol is a network interface implementation over InfiniBand

InfiniBand FPGA Project - seniord.ece.iastate.eduseniord.ece.iastate.edu/.../may0904/May0904-final.docx · Web viewT. Figure 1 - An InfiniBand switch panelhe InfiniBand architecture

Presentation Infiniband

VPI / InfiniBand - mellanox.com · Performance Accelerated Mellanox InfiniBand Adapters Provide Advanced Data Center Performance, Efficiency and Scalability VPI / InfiniBand

DrorGoldenberg InfiniBand Technology Overview

QLogic InfiniBand Ethernet and Fibre Channel Bridge ...€¦ · QLogic InfiniBand Ethernet Bridge Module for IBM BladeCenter and the QLogic InfiniBand Fibre Channel Bridge Module

InfiniBand Trade Association

InfiniBand: The Production SDN - Mellanox Technologies · The introduction of OpenFlow, ... Mellanox InfiniBand SDN Components The InfiniBand control plane protocol specification

Security in Mellanox Technologies InfiniBand Fabrics › related-docs › whitepapers › WP...Security in Mellanox Technologies InfiniBand Fabrics Technical Overview InfiniBand is

InfiniBand on Debian

To Infiniband and Beyond

HPE EDR InfiniBand Adapters · QuickSpecs HPE EDR InfiniBand Adapters Overview Page 1 HPE EDR InfiniBand Adapters . HPE EDR InfiniBand 100Gb 1-port 841QSFP28 Adapter is based on Mellanox

InfiniBand Presentation

Infiniband subnet management Discuss the Infiniband subnet management system Discuss fat tree and subnet management in an Infiniband with a fat tree topology

Introduction to High-Speed InfiniBand Interconnect • Industry standard defined by the InfiniBand Trade Association – Originated in 1999 • InfiniBand specification defines an

InfiniBand Architecture Overview Back To · PDF file© 2009 MELLANOX TECHNOLOGIES - CONFIDENTIAL - InfiniBand Technical Overview What is InfiniBand? • InfiniBand is an open standard,