InfiniBand SNMP Agent

PSL Lab Spring 2005 – Winter 2005 2006

2

People

• Student– Ali Ayoub

• Supervisors– Liran Liss– Eitan Zahavi– Amit Krig

3

Abstract

The rapid growth in InfiniBand usage requires more standard mange tools, most popular monitoring applications for networks use Simple Network Management Protocol (SNMP); the goal of this project is to enable monitoring the InfiniBand network using this protocol.

4

Introduction

InfiniBand networks, as other networks, need to be monitored and managed for administrator and development needs, the usage of this technology become more widespread; it’s involved in bigger clusters and high-performance networking, therefore, the need for such a tool becomes more essential.Besides, a lot of information about InfiniBand traffic is stored in the hardware, this is derived from its architecture which implements most of the layers in hardware, and this information is usually not exposed to the application layer, which makes the task to retrieve this information more difficult without a tool designed for this purpose.

5

Introduction

To achieve this goal, we used in this project a standard network management protocol called SNMP, because this protocol is one of the most popular solutions for network monitoring, there are many monitors which relies on this protocol to offer to the user a friendly interface such as IBM Tivoli and HP OpenView, for the same reason, collecting information from InfiniBand devices and other devices (CPU, memory, NIC…) , can be done using the same tool. So far, there was no such a tool for InfiniBand network management, although it was invented and used since 1996.The tool we present in this project is the only one of its kind in this field.

6

Background

• SNMPSimple Network Management Protocol (SNMP) is an application layer protocol that facilitates the exchange of management information between network devices. It is part of the Transmission Control Protocol/Internet Protocol (TCP/IP) protocol suite. SNMP enables network administrators to manage network performance, find and solve network problems, and plan for network growth.

7

Background

• MIBA Management Information Base (MIB) is a collection of information that is organized hierarchically. MIBs are accessed using a network-management protocol such as SNMP. They are comprised of managed objects and are identified by object identifiers.

8

Background

• OIDThe Object ID (OID), each SNMP element manages specific objects with each object having specific characteristics. Each object / characteristic has a unique object identifier (OID) consisting of numbers separated by decimal points (i.e., 1.3.6.1.4.1.2682.1).

9

10

Overview of the solution

• Components– InfiniBand SNMP Agent– NET-SNMP

Advantages vs. disadvantages– Management Network System.

Examples– InfiniBand Management Information Bases (IB MIBs)

11

Overview of the solution

• Flow descriptionThe MNS have the MIBs definitions; according to them the MNS sends a request to a machine with InfiniBand SNMP Agent asking it for the information, the InfiniBand SNMP Agent includes the MIBs module which are used to retrieve the requested data and send it back to the MNS, then the MNS display the information to the user by its own interface, polling InfiniBand information in high frequency can serve to draw graphs for InfiniBand traffic for example.

12

IB network

InfiniBand SNMP agent and NET-SNMP run on the machines in the network.

SNMP Monitor

IB MIBs

IP network

13

Implementation details

• InfiniBand MIBs– IPoIB workgroup– InfiniBandMIB tree– Implemented IB MIBs

• IB SNMP Agent– High-level layers– Agent subagnet architecture

AgentX– Data caching

• Design– Low-level layers

• Tools• Testing• Issues uncovered

14

InfiniBand MIBs

• The MIBs we use and implement are written by the IP over IB workgroup as published in IETF (Internet Engineering Task Force) internet drafts, they are part of set of MIBs for IB fabric management, these MIBs are located under InfiniBandMIB (OID: 1.3.6.1.3.117) sub-tree, and will have a unique OIDs once they are registered by IANA (Internet Assigned Numbers Authority).

15

O root-node | | iso[1] | | +---+------+ | | | | | |[1] [2] identified-organization[3] |

| +-------+--+--+ | | | | | |[1] ... [5] dod[6] |

| internet[1] |

| +-------------+-----+------------------+----------------+ | | | | | | | |directory[1] management[2] experimental[3] private[4] |

| InfiniBandMIB[117]InfiniBandMIB[117] |

| +----------+-----------+----------+-+---------+-----------+----------+ | | | | | | | | | | | | | |ibTcMIB[1] ibIfMIB[2] ibSmaMIB[3] ibCaMIB[4] ibBmaMIB[5] ibPmaMIB[6] ibSmMIB[7]

16

InfiniBand MIBs

• IPoIB MIBs– ibTcMIB: Textual Conventions MIB

– ibIfMIB: InfiniBand Interfaces MIB

– ibSmaMIB: Subnet Management Agent MIB

– ibCaMIB: Channel Adapters MIB

– ibBmaMIB: Baseboard Management MIB

– ibPmaMIB: Performance Management Agent MIB

– ibSmMIB: Subnet Manager MIB

17

InfiniBand MIBs

1. Channel adapter MIB (ibCaMib)– This MIB describes the objects for managing the InfiniBand Channel

Adapters.– OID: 1.3.6.1.3.117.4– The purpose of this MIB is to define the objects necessary to

manage Channel Adapters (HCAs and TCAs).– This includes general CA information, CA attributes, CA port

attributes, and the associated CA port GID info.– Current I-D: draft-ietf-ipoib-channel-adapter-mib-07.txt– Expiration date: March 2006

18

InfiniBand MIBs

2. InfiniBand Interfaces MIB (ibIfMib) – Describes the managed objects for managing IBA-defined InfiniBand interfaces.– OID: 1.3.6.1.3.117.2– The purpose of this MIB is to augment the standard SNMP Interface MIB (RFC

2863). – Defines standard InfiniBand port ifType as InfiniBand(199).– It contains an InfiniBand Port statistics table, a Virtual Lane Traffic Table for each

port’s VLs, and a table for the IPOIB Link layer addresses (20-octet value that uniquely identifies each IPOIB client end-point, i.e., QPN + GID).

– Current I-D: draft-ietf-ipoib-ibif-mib-08.txt– Expiration date: March 2006

19

InfiniBand MIBs

• MIBs Modules Integration– Extending master agent– Separated shared libraries to be loaded into the master agent– Implementing MIBs modules into a SNMP subagent

• Which way did we use? Why?– We’ll discuss this way later

20

InfiniBand SNMP Agent

• Slave agent

• MIBs– Definitions, where they are used?– MIBs Modules = implementation

• Generic

• AgentX

21

InfiniBand SNMP Agent

• AgentX, – why?

The two most commonly used protocols for subagents are AgentX and SMUX.SNMP Master/Subagent protocols overcome a very real need to dynamically extend the managed objects in a node. The protocols allow you having a single SNMP Agent and several Subagents that can connect and register several managed objects without having to interrupt the management service. We decided to use AgentX protocol, because it is the first IETF standard-track specification for extensible SNMP agents and is expected to gradually replace all the other open and proprietary agent extensibility solutions (such as SMUX, DPI etc.).

22

InfiniBand SNMP Agent

• AgentX, – How?

The subagent (InfiniBand SNMP Agent) registers itself in the master agent, when this process is completed the master agent knows that there is a subagent which implements some OIDs

when the monitor sends a SNMP request to retrieve data from a machine with InfiniBand SNMP Agent, first the master agent receives the request, according to the OIDs requested, it passes the request to the subagent (using AgentX protocol), where the suitable function in InfiniBand SNMP Agent will be called to supply the needed information requested, and update the data structures involved

Then the master agent will return to the requester the requested data, which could be a single attribute or sub-tree of attributes. Besides, this protocol is supported by NET-SNMP, that’s why we used AgentX protocol in implementing our InfiniBand SNMP Agent.

23

24

25

InfiniBand SNMP Agent

• Data caching– Why?

SNMP monitors usually poll data from the SNMP agent in high frequency (for drawing graphs for example), this could create stress on the InfiniBand device to answer all the agent’s queries.

– How? Data structure Refresh rate Configuration

– IB-CA-MIB vs. IB-IF-MIB

26

Design

• InfiniBand SNMP Agent

• IBSNMP Agent driver API

• SNMP agent gen2 layer

• SNMP agent gen1 layer

• Agent IPoIB layer

• MADs structures

27

Design

28

Design

• Advantages– Isolation

SNMP vs. IB MADs and IPoIB libraries

– Reusability For future MIBs For future drivers

– Genericity Easy way for linkage with different stacks

29

How to retrieve InfiniBand MIBs information

• MADs– Management datagrams for IB-IF-MIB

• Driver verbs– Static data, for IB-CA-MIB

• Operating system commands and files– Like /proc/ files and IP interfaces commands.

30

Tools

• Languages– C, ASN.1

• Protocols– SNMP v2/v3

• AgentX

• Net-SNMP Master agent

• Mib2c

31

Testing

• Different Operating systems

• Different IB Stacks

• Direct tests

• Real events

• Monitors

32

Issues revealed

• Bad InfiniBand cables

• Unplugged cables

• Unloaded drivers

• Also, for retiring information for the HCA remotely without logging on the machine– Number of ports– MAC– GUID

33

SNMP Monitors

• MRTG (Multi Router Traffic grapher)– Open source SNMP monitor for displaying traffic on graphs on real time.– Could be installed on web-server for real time monitoring from

everywhere.– Good for dynamic data (IB-IF-MIB)

• IReasoning– Commercial – Good for viewing tables (IB-CA-MIB)

34

IBMRTG Monitor

35

IReasoning

36

Future Work

• Extending the IB SNMP Agent– More MIBs

Some are available in IPoIB workgroup– Support future IB stacks

37

Thank you!