Mellanox 1U EDR 100Gb/s InfiniBand Switch Systems and IB ... · PDF file Mellanox Technologies Mellanox 1U EDR 100Gb/s InfiniBand Switch Systems and IB Router Hardware User Manual

Embed Size (px)

Citation preview

  • Mellanox Technologieswww.mellanox.com

    Mellanox 1U EDR 100Gb/s InfiniBand Switch Systems and IB RouterHardware User ManualModels: SB7700/SB7790/SB7800/SB7890/SB7780

    Rev. 1.8

  • Mellanox Technologies350 Oakmead Parkway Suite 100Sunnyvale, CA 94085U.S.A.www.mellanox.comTel: (408) 970-3400Fax: (408) 970-3403

    Copyright 2018. Mellanox Technologies Ltd . All Rights Reserved .

    Mellanox, Mellanox logo, Accelio, BridgeX, CloudX logo, CompustorX, Connect -IB, ConnectX, CoolBox , CORE-Direct , EZchip, EZchip logo, EZappliance, EZdesign, EZdriver, EZsystem, GPUDirect, InfiniHost, InfiniBridge, InfiniScale, Kotura, Kotura logo, Mellanox CloudRack , Mellanox CloudXMellanox , Mellanox Federal Systems, Mellanox HostDirect , Mellanox Multi-Host , Mellanox Open Ethernet, Mellanox OpenCloud , Mellanox OpenCloud Logo , Mellanox PeerDirect , Mellanox ScalableHPC, Mellanox StorageX , Mellanox TuneX , Mellanox Connect Accelerate Outperform logo , Mellanox Virtual Modular Switch, MetroDX, MetroX, MLNX-OS, NP-1c, NP-2, NP-3, NPS, Open Ethernet logo, PhyX, PlatformX, PSIPHY, SiPhy, StoreX, SwitchX, Tilera, Tilera logo, TestX, TuneX, The Generation of Open Ethernet logo, UFM, Unbreakable Link , Virtual Protocol Interconnect , Voltaire and Voltaire logo are registered trademarks of Mellanox Technologies , Ltd.

    All other trademarks are property of their respective owners .

    For the most updated list of Mellanox trademarks, visit http://www.mellanox.com/page/trademarks

    NOTE:THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (PRODUCT (S)) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS-IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT , STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

    Document Number: 2Mellanox Technologies

  • Rev. 1.8 3Mellanox Technologies

    List of Tables

    Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Chapter 1 Introduction to Mellanox SB77X0/SB78X0 Switch Systems and SB7780

    InfinBand Router . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101.2 Speed and Switching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111.3 Management Interfaces and FRUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111.4 Features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121.5 Certifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121.6 Ordering Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

    Chapter 2 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.1 Safety Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.2 Air Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.3 Package Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.4 Mounting Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

    2.4.1 19 Systems Mounting - Static Rail-Kit . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.4.2 19 Systems Mounting- Telescopic Rail-Kit. . . . . . . . . . . . . . . . . . . . . . . . . 23

    2.5 Cable Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292.6 Initial Power On . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302.7 System Bring-Up of Managed Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

    2.7.1 Configuring Network Attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312.7.2 Remote Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

    2.8 FRU Replacements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362.8.1 Power Supply. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362.8.2 Fans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

    Chapter 3 Software Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393.1 InfiniBand Subnet Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393.2 Fabric Inspector (Diagnostics) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393.3 Upgrading Software (on Managed Systems) . . . . . . . . . . . . . . . . . . . . . . . 403.4 Updating Firmware on Externally Managed Systems . . . . . . . . . . . . . . . . 40

    3.4.1 Updating Firmware In-band (Typical) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

    Chapter 4 Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424.1 Supported Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

    4.1.1 Data Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424.1.2 Speed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

  • Rev. 1.84 Mellanox Technologies

    4.1.3 RS232 (Console). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434.1.4 Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434.1.5 USB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434.1.6 Reset Button . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

    4.2 LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444.2.1 LED Notifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

    4.3 Inventory Pull-out Tab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50Chapter 5 Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

    5.1 Troubleshooting Instructions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51Chapter 6 Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

    6.1 SB77X0/SB78X0 Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53Appendix A Accessory and Replacement Parts . . . . . . . . . . . . . . . . . . . . . . . 54Appendix B Thermal Threshold Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . 55Appendix C Interface Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

    C.1 QSFP Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56C.2 RJ-45 CONSOLE and IC Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58C.3 RJ45 to DB9 Harness Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

    Appendix D Disassembly and Disposal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60D.1 Disassembly Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60D.2 Disposal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

    Appendix E Safety Warnings (Multiple Languages) . . . . . . . . . . . . . . . . . . . . 61E.1 Nordic Countries Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61E.2 Installation Safety Warnings (English) . . . . . . . . . . . . . . . . . . . . . . . . . . 61E.3 () . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64E.4 (Chinese) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68E.5 Avertissements de scurit pour l'installation (French). . . . . . . . . . . . 72E.6 Installation Sicherheitshinweise(German). . . . . . . . . . . . . . . . . . . . . . . 76E.7 Advertencias de seguridad de instalacin (Spanish) . . . . . . . . . . . . . . 79E.8 (Russian) 83E.9 Avertismente privind sigurana la instalare (Romanian) .