Upload
others
View
7
Download
0
Embed Size (px)
Citation preview
www.mellanox.com Mellanox Technologies
Mellanox Scalable Hierarchical Aggregation and
Reduction Protocol
(SHARP)™
Release Notes
Rev 1.4.3
Mellanox Technologies 2
Overview
Mellanox Technologies
350 Oakmead Parkway Suite 100
Sunnyvale, CA 94085
U.S.A.
www.mellanox.com
Tel: (408) 970-3400
Fax: (408) 970-3403
© Copyright 2017. Mellanox Technologies Ltd. All Rights Reserved.
Mellanox®, Mellanox logo, Accelio®, BridgeX®, CloudX logo, CompustorX®, Connect-IB®, ConnectX®,
CoolBox®, CORE-Direct®, EZchip®, EZchip logo, EZappliance®, EZdesign®, EZdriver®, EZsystem®,
GPUDirect®, InfiniHost®, InfiniBridge®, InfiniScale®, Kotura®, Kotura logo, Mellanox CloudRack®, Mellanox
CloudXMellanox®, Mellanox Federal Systems®, Mellanox HostDirect®, Mellanox Multi-Host®, Mellanox Open
Ethernet®, Mellanox OpenCloud®, Mellanox OpenCloud Logo®, Mellanox PeerDirect®, Mellanox ScalableHPC®,
Mellanox StorageX®, Mellanox TuneX®, Mellanox Connect Accelerate Outperform logo, Mellanox Virtual Modular
Switch®, MetroDX®, MetroX®, MLNX-OS®, NP-1c®, NP-2®, NP-3®, Open Ethernet logo, PhyX®, PlatformX®,
PSIPHY®, SiPhy®, StoreX®, SwitchX®, Tilera®, Tilera logo, TestX®, TuneX®, The Generation of Open Ethernet
logo, UFM®, Unbreakable Link®, Virtual Protocol Interconnect®, Voltaire® and Voltaire logo are registered
trademarks of Mellanox Technologies, Ltd.
All other trademarks are property of their respective owners.
For the most updated list of Mellanox trademarks, visit http://www.mellanox.com/page/trademarks
NOTE:
THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED
DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY
KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT
USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST
ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY
QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES
CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE
HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR
ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE
PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE.
Rev 1.4.3 Mellanox Technologies 3
Overview
Table of Contents
Document Revision History .................................................................................................................. 5
1 Overview .......................................................................................................................................... 6
1.1 Packages ................................................................................................................................ 6
1.2 Prerequisites ........................................................................................................................... 6
1.3 Supported OS and Platforms .................................................................................................. 6
2 Changes and New Features ........................................................................................................... 8
2.1 Parameters Changes .............................................................................................................. 8
3 Limitations ....................................................................................................................................... 9
4 Change Log History ...................................................................................................................... 11
4.1 Parameters Changes ............................................................................................................ 12
4.2 API Updates .......................................................................................................................... 13
4 Mellanox Technologies Rev 1.4.3
Overview
List of Tables
Table 1: Document Revision History ....................................................................................................... 5
Table 2: Supported OS and Platforms ..................................................................................................... 6
Table 3: Limitations ................................................................................................................................. 9
Table 4: Change Log History .................................................................................................................. 11
Rev 1.4.3 Mellanox Technologies 5
Overview
Document Revision History
Table 1: Document Revision History
Revision Date Description
Rev. 1.4.1 Oct 31, 2017 Initial version of this release
6 Mellanox Technologies Rev 1.4.3
Overview
1 Overview
Mellanox Scalable Hierarchical Aggregation and Reduction Protocol (SHARP)™
technology improves upon the performance of MPI operation, by offloading collective
operations from the CPU to the switch network, and eliminating the need to send data
multiple times between endpoints. This innovative approach decreases the amount of data
traversing the network as aggregation nodes are reached, and dramatically reduces the MPI
operations time. Implementing collective communication algorithms in the network also has
additional benefits, such as freeing up valuable CPU resources for computation rather than
using them to process communication.
1.1 Packages
Package Version
MLNX OFED 4.2-x.x.x
HPC-X 2.0.x
UFM (Aggregation Manager v1.4.1 only) 5.9.5
1.2 Prerequisites
Name Version
Externally managed Switch-IB 2 Firmware version: 15.1460.0162 or later
MLNX OS 3.6.4008
Subnet Manager 4.9 (MLNX OFED 4.2-x.x.x ) or later .
4.7-4.8 require additional configuration in Aggregation
Manager.
1.3 Supported OS and Platforms
Table 2: Supported OS and Platforms
Distro Platform Kernel
RHEL 6.1 x86-64 2.6.32-131.0.15
RHEL 6.2 x86-64 2.6.32-220
RHEL 6.3 x86-64 2.6.32-279
RHEL 6.4 x86-64 2.6.32-358
RHEL 6.5 x86-64 2.6.32-431
RHEL 7.0 x86-64 3.10.0-123
RHEL 7.2 x86-64 3.10.0-327
RHEL 7.2 ppcle 3.10.0-327
RHEL 7.3 Arm 4.5.0-15.el7.aarch64
RHEL 7.4 x86-64 3.10.0-693
Fedora14 x86-64 2.6.35.6-45
Rev 1.4.3 Mellanox Technologies 7
Overview
Distro Platform Kernel
Fedora16 x86-64 3.1.0-7
Fedora17 x86-64 3.3.4-5
Fedora18 x86-64 3.6.10-4
Fedora24 x86-64 4.5.5-300
Fedora26 x86-64 4.11.8-300
SLES 11 SP1 x86-64 2.6.32.12-0.7
SLES 11 SP2 x86-64 3.0.13-0.27
SLES 11 SP3 x86-64 3.0.76-0.11
Ubuntu12.04 x86-64 3.2.0-37
Ubuntu13.10 x86-64 3.11.0-12
Ubuntu14.4 x86-64 3.13.0-24
Ubuntu15.10 x86-64 4.2.0-16
CentOS6.3 x86-64 2.6.32-279
CentOS6.0 x86-64 2.6.32-71
8 Mellanox Technologies Rev 1.4.3
Changes and New Features
2 Changes and New Features
Table 2: Changes and New Features
Component Category Description
Aggregation
Manager (sharp_am)
Fabric extension Enabled adding/replacing new non-root aggregation nodes
without restarting Aggregation Manager.
Aggregation
Manager (sharp_am)
Fabric extension Optimized root placement on tree topologies (improved the
location of Mellanox SHARP trees roots on the tree topologies).
Aggregation
Manager (sharp_am)
Resiliency Added the option to notify running jobs about Aggregation
Manager (sharp_am) restart.
Mellanox SHARP
Daemon (sharpd)
Out-of-the-box
improvement
Added Systemd support.
Mellanox SHARP
Daemon (sharpd)
Out-of-the-box
improvement
Added Socket-Based-Activation support for Mellanox SHARP
daemons on systems with Systemd.
Mellanox SHARP
Daemon (sharpd)
Out-of-the-box
improvement
Removed static binding to network IP interface in Mellanox
SHARP daemons.
2.1 Parameters Changes
Parameter Component Description
accumulate_log sharp_am/sharpd New Parameter: Accumulates log file over multiple
sessions. If set to FALSE and log rotation is disabled,
the log file is truncated on startup
Note: No support on update runtime.
syslog_verbosity sharp_am/sharpd New Parameter: Syslog verbosity level: 1 - Errors, 2 -
Warnings. Default value is "1".
Note: Supported on update runtime.
persistent_dir Sharp_am New Parameter: Path to persistent data directory.
Note: No support on update runtime.
Rev 1.4.3 Mellanox Technologies 9
Limitations
3 Limitations
Table 3: Limitations
Internal Ref. Issues
1179747 Description: Changing smx_sock_interface configuration parameter is not supported.
Workaround: N/A
Keywords: Mellanox SHARP Daemon
1074862 Description: Fabric extension: adding new non-root switches or non-root switch
replacement are currently not supported by the Aggregation Manager for tree topologies.
(Fat-Tree, Quasi-Fat-Tree)
Workaround: Restart Aggregation Manager after the Subnet Manager completes fabric
reconfiguration followed by the fabric changes.
Keywords: Aggregation Manager
- Description: Aggregation Manager High Availability is currently not supported in
HPCX/MLNX OFED packages. Therefore, only a single instance of Aggregation
Manager can run in the IB fabric.
Workaround: Use Aggregation Manager in UFM.
Keywords: Aggregation Manager
- Description: Aggregation manager should run on the same Host where the Master Subnet
Manager (SM) is running.
Workaround: N/A
Keywords: Aggregation Manager
- Description: In case of HPCX/MLNX OFED packages, upon Subnet Manager
handover/failover, another instance of Aggregation Manager should be started on the Host
where the new Master SM is running
Workaround: Use Aggregation Manager in UFM.
Keywords: Aggregation Manager
- Description: Aggregation Manager should be started after completion of fabric
configuration by the Subnet Manager.
Workaround: N/A
Keywords: Aggregation Manager
- Description: Fabric extension: adding new switches or switch replacement are currently
not supported by the Aggregation Manager for hypercube topologies
Workaround: Restart Aggregation Manager after the Subnet Manager completes fabric
reconfiguration followed by the fabric changes.
Keywords: Aggregation Manager
- Description: Fat-Tree, Quasi-Fat-Tree and Hypercube topologies are supported by the
Aggregation Manager.
Workaround: N/A
Keywords: Fabric Topology
10 Mellanox Technologies Rev 1.4.3
Limitations
Internal Ref. Issues
- Description: Only IB fabrics where all compute nodes are connected to Mellanox
SHARP capable switches (Switch-IB 2) are supported by the Aggregation Manager.
Workaround: Manually configure mapping between the compute port and the
Aggregation Node.
Keywords: Fabric Topology
- Description: Upon changes in configuration file beyond parameters in 3.3, Aggregation
Manager should be restarted to deploy new configuration.
Workaround: N/A
Keywords: Configuration
1071876 Description: Rebooting the switch the SM port is connected to could affect Aggregation
Trees.
Workaround: Restart Aggregation Manager
Keywords: Aggregation Manager
Rev 1.4.3 Mellanox Technologies 11
Change Log History
4 Change Log History
Table 4: Change Log History
Feature/Change Description
Rev. 1.3
Aggregation Manager (sharp_am)
Out-of-the-box improvement Added support for extended fabric format (SMDB). Note: This
requires Subnet Manager 4.9 or later.
Fabric extension Compute hosts can be added/replaced without Aggregation Manager
restart.
Configuration Added the ability to update some configuration parameters in
runtime without application restart.
Mellanox SHARP Daemon (sharpd)
Out-of-the-box improvement Removed static binding to IB port.
Configuration Added the ability to update some configuration parameters in
runtime without application restart.
Rev. 1.2
Aggregation Manager (sharp_am) Added support for IB fabric events (flapping links, switch/host
reboot)
Resiliency: Mellanox SHARP Tree QP Recovery
Added support for Hyper-cube topology (needs OpenSM 4.8.1 or
later)
HCOLL Added new non-blocking API for Mellanox SHARP collectives
Job Scheduler Added new API for integration with Job Scheduler
UFM Enabled Aggregation Manager integration with UFM
Rev. 1.1
HCOLL Enables UD MCAST result distribution
Enables multiple group leaders per compute nodes
Delivers error to an application
Enables Mellanox SHARP Group trim
Added support for ppcle platform
Rev. 1.0
MPI 2.x Barrier and Allreduce collective operations using Mellanox SHARP
protocol are supported in Open MPI, MPICH, ScalableSHMEM
with HCOLL library.
HCOLL Enables running of Mellanox SHARP collective with the mpirun
utility.
For the complete list of flags that can be used when running
Mellanox SHARP software, please refer to the SHARP Deployment
Guide.
12 Mellanox Technologies Rev 1.4.3
Change Log History
4.1 Parameters Changes
Parameter Component Description
Rev. 1.3
ib_mad_timeout sharp_am Removed
ib_mad_retries sharp_am Removed
hyper_cube_coordinat
es_file
sharp_am Deprecated (with Subnet Manager 4.9
and later).
root_guids_file sharp_am Deprecated (with Subnet Manager 4.9
and later).
ib_dev sharpd Removed
log_verbosity sharp_am /
sharpd
Modified behavior: Added the option to
update on runtime.
lst_file_timeout sharp_am Modified behavior: Added the option to
update on runtime.
lst_file_retries sharp_am Modified behavior: Added the option to
update on runtime.
generate_dump_files sharp_am Modified behavior: Added the option to
update on runtime.
max_quota sharp_am Modified behavior: Added the option to
update on runtime.
default_quota sharp_am Modified behavior: Added the option to
update on runtime.
span_all_agg_nodes sharp_am New Parameter: Generate trees that span
all possible aggregation nodes
Relevant only if "topology_type" is tree.
Rev. 1.2
Environment variable:
SMX_SOCK_PORT
AM / SD Replaced by smx_sock_port
parameter
Environment variable:
SMX_SOCK_INTERFACE
AM / SD Replaced by smx_sock_interface
SHARP_COLL_SHARP_ENA
BLE_MCAST_TARGET
HCOLL Replaced by
SHARP_COLL_ENABLE_MCAST_TARG
ET
smx_sock_interface sharp_am /
sharpd
New Parameter: Network interface to be
used by SMX.
Default: empty string - Use first interface
found in UP state
smx_sock_port sharp_am /
sharpd
New Parameter: The external port to be
used by SMX. Default - 6126
lst_file_timeout sharp_am New Parameter: Length of timeout in
seconds between attempts to load the
LST file. Default - 3 seconds.
lst_file_retries sharp_am New Parameter: Max number of retry
attempts when loading the LST file and
Rev 1.4.3 Mellanox Technologies 13
Change Log History
Parameter Component Description
encountering "No such file" errors.
Default - 0 meaning no retries.
log_max_backup_files sharpd New Parameter: Number of backup log
files. Used for log rotation
log_file_max_size sharpd New Parameter: Maximum size of a log
file, in MBs. If value is 0, log rotation
isn't used
mgmt_mode sharpd New Parameter: When running in
managed mode, SHARPD expects
notifications from the Resource manager
(Job scheduler). The possible values are:
0 - Unmanaged mode; 1 - Managed mode
smx_sock_backlog sharpd New Parameter: Defines the maximum
length to which the queue of pending
connections for the SMX listen socket
may grow
group_allocate_timeo
ut
sharpd New Parameter: Maximum time [in
milliseconds] to wait for group allocation
transaction to complete.
4.2 API Updates
API Category Description
Rev. 1.2
sharp_coll_do_allreduce_nb HCOLL Changed
sharp_coll_do_barrier_nb HCOLL Changed
sharp_coll_do_reduce_nb HCOLL Changed
sharp_coll_req_test HCOLL Changed
sharp_coll_req_wait HCOLL Changed
sharp_coll_req_free HCOLL Changed
sharp_job_quota Job Scheduler Added