DE5 NetFPGA Reference Router User Guide

Revision History

Date Comment Author O8/30/2014 Initial draft Kekai Hu

Contents

1.   Introduction  ....................................................................................................................................  4  

2.   System Requirements  .....................................................................................................................  4  

3.   Obtaining Altera DE5 Board  ..........................................................................................................  5  

4.   Installing NetFPGA DE5 Board in the Host PC  .............................................................................  5  

5.   Obtaining DE5 NetFPGA Reference Router  ..................................................................................  8  

6.   Directory Structure  .........................................................................................................................  8  

7.   Compiling and configuring the DE5 NetFPGA Reference Router in Windows  .............................  9  

7.1.   Compilation  .............................................................................................................................  9  

7.2.   Configuration  ........................................................................................................................  12  

7.2.1.   JTAG Mode  ...................................................................................................................  12  

7.2.2.   PCIe Mode  .....................................................................................................................  14  

8.   Testing  ..........................................................................................................................................  16  

8.1.   Test  Setup  ..............................................................................................................................  16  

8.2.   Running  the  Test  ...................................................................................................................  17  

9.   References  ....................................................................................................................................  17  

1. Introduction DE5 NetFPGA is an open source port of the NetFPGA [2] platform on Altera DE5 board. DE5 NetFPGA provides network researchers with a powerful open platform to build complex network applications. Its features:

Ø Stratix V FPGA with more logic and memory resources available than on Stratix IV. Ø PCI Express Interface for faster host PC to FPGA data transfers. Ø 10G High speed Ethernet interface.

This tutorial describes the steps to obtain DE5 NetFPGA hardware, download the DE5 NetFPGA reference router design from the project website, compile the design using Altera Quartus II software, and verify the reference router by sending test packets.

2. System Requirements To successfully install and test DE5 NetFPGA, you must satisfy the following system requirements.

Operating System Requirements

Ø Microsoft Windows (Verified with Windows XP) Software Requirements

Ø Altera Quartus II 13.1 or later and Qsys Builder Ø Microsoft Visual Studio 2010 (PCIe mode in Windows only) Ø Jungo Windriver 32bit (PCIe mode in Windows only)

Hardware Requirements

Ø PC with PCIe x8 or x16 slot (PCIe mode in Windows only) Ø Altera DE-5 board Ø SFP+ Transceivers Ø 10GbE Duplex Multimode LC/LC Fiber Patch Cable

IP License Requirements

Ø Altera 10G BASE-R PHY IP Core Ø Altera 10G Ethernet MAC

Note: Altera provides a free time limited Open Core Plus core of 10G Ethernet with a time limit. This core is available as part of the Altera Quartus II software. Table 1 provides additional information on obtaining the hardware and software necessary to compile DE5 NetFPGA reference router.

Table 1: Software and Hardware Requirements

Requirement Source

Altera Quartus II 13.1 and Qsys Builder https://www.altera.com/download/software/quartus-ii-se

Microsoft Visual Studio 2010 http://www.microsoft.com/visualstudio/en-us/try

Jungo Windriver http://www.jungo.com/st/windriver_usb_pci_driver_development_software.html

Altera DE5 board http://www.terasic.com.tw

Altera 10G Ethernet IP Core http://www.altera.com/support/ip/interface-protocols/ips-inp-10gbe.html

3. Obtaining Altera DE5 Board Altera DE5 is available from Terasic Technologies that integrates the state of the art Altera Stratix V FPGA with four 10G Ethernet interfaces, several high speed I/Os, and high density static and dynamic memory interfaces. The DE5 board can be purchased from Terasic’s website [8].

4. Installing NetFPGA DE5 Board in the Host PC

The DE5 NetFPGA can be connected to the host PC in two configurations as follows:

Ø JTAG mode Ø PCI Express mode (Windows only)

JTAG mode:

To operate the Altera DE5 in JTAG mode, set the SW0 switch in ON position. This is shown in Figure 1.

Figure 1: SW0 switch ON position

The Altera DE5 board is powered up using an external stand alone power supply in the JTAG mode. This configuration is shown in Figure 2. The FPGA bit-stream and the router configuration are programmed through the USB/JTAG interface.

Figure 2: JTAG mode

PCI Express mode (Windows only):

To operate Altera DE5 in PCI Express mode, perform the following two steps:

1. Enable PCI mode by setting the SW0 switch in OFF position as shown in Figure 3.

2. Make sure the PCI Express Mode SW7 is switched to x8 mode. Table 2 lists the switch controls and description. Figure 2 shows the default configuration of SW7 on the bottom of the FPGA board.

Table 2: SW7 PCIe Control DIP Switch

Board Reference Signal Name Description Default SW7.1 PCIE_PRSNT2n_x1 On : Enable x1 presence detect

Off: Disable x1 presence detect OFF

SW7.2 PCIE_PRSNT2n_x4 On : Enable x4 presence detect Off: Disable x4 presence detect

OFF

SW7.3 PCIE_PRSNT2n_x8 On : Enable x8 presence detect Off: Disable x8 presence detect

ON

Figure 3: SW0 switch OFF position

Figure 4: PCI Express Control DIP Switch

The PCI Express mode does not require a standalone power supply. Instead, the Altera DE5 board can be powered up through the x8 or x16 PCI Express edge connector attached to a PC as shown in Figure 5.

Figure 5: PCI Express Mode

5. Obtaining DE5 NetFPGA Reference Router The NetFPGA reference router v1.0 is an open line rate (1Gbps) IPv4 router [3] available for education and research purposes from Stanford University. The DE5 NetFPGA reference router provides a port of the NetFPGA reference router on the Altera DE5 platform. The gateware and software for DE5 NetFPGA reference router is bundled as two separate packages as follows:

Ø DE5 Reference Router project Ø DE5 Reference Router source file package, a modified version of Stanford University’s

NetFPGA reference router design.

The DE5 NetFPGAreference router project package can be downloaded directly from the project website [1]. The DE5 Reference Router source file package is not included in the distribution available from the project website. To obtain this package, please send an e-mail to Professor Russell Tessier (email:tessier@ecs.umass.edu).

6. Directory Structure Compiling the DE5 NetFPGA reference router requires two separate packages as described in Section 5:

Ø DE5 Reference Router project directory (DE5_Reference_Router.zip) Ø DE5 reference router source code package (DE5_Reference_Router_v1.0.zip).

After getting both packages, first unzip DE5_Reference_Router.zip will create the project directory, then unzip DE5_Reference_Router_v1.0.zip and copy the source code files in “src” directory to the “src” directory in the DE5_Packet_Generator project package.

The projects folder contains the following subdirectories:

Ø src – The DE5 NetFPGA reference data path verilog source code files are placed here. You need to request these files from Professor Tessier.

Ø sw - It has two subdirectories: pcie_config and jtag_config. ‘pcie_config’ contains the Visual Studio software application for configuring the user defined MAC, IP address, LPM and ARP values to the reference router’s routing table in PCIe mode. ‘jtag_config’ contains tcl scripts that configure the routing tables in JTAG mode.

Ø phy_10gbaser and phy_10gbaser_sim – These two folders include all the Altera 10G Base-R Phy source code files for the 10G Ethernet PHY.

7. Compiling and configuring the DE5 NetFPGA Reference Router in Windows

7.1. Compilation   The compilation steps for Linux are described in Section 错误 ! 未找到引用源。 .The detailed compilation steps for Windows are described below:

1. Open the DE5_Reference_Router project in Quartus by double clicking DE5_Reference_Router/DE5_Reference_Router.qpf

2. Open Qsys builder (Quartus → Tools →qsys)

3. Generate the Qsys System. Load DE5_Reference_Router_Qsys.qsys into Qsys.

4. Generate the design by click Generate → Generate.

5. Once Qsys system generation is successful, compile the design. (Quartus → Processing → Start Compilation)

6. Install the Altera DE5 board and turn it on if using JTAG mode. Make sure that the board is connected to PC via the USB/JTAG cable.

7. Install USB/JTAG drivers for the PC [11]. 8. Download the bit file generated after the compilation to the target DE5 board by double

clicking the "Program Device" in the Task Window.

9. Click Start in the Programmer window to begin download.

If you are using PCI Express mode, restart the host PC after downloading the bit-file.

7.2. Configuration Once the reference router bit-stream has been programmed onto the FPGA, the router tables must be configured before we can test the DE5 NetFPGA reference router (hereafter referred to as set_router_tables). This step (set_router_tables) assigns user defined MAC, IP address, LPM and ARP values to the reference router’s routing table. The configuration file is in {project root}\sw\.

MAC 0RegisterSpace

MAC 1RegisterSpace

MAC 2RegisterSpace

MAC 3RegisterSpace

MAC table

Destip table

ARP table

LPM table

ReferenceRouter

PHY 0RegisterSpace

PHY 1RegisterSpace

PHY 2RegisterSpace

PHY 3RegisterSpace

DE4 board

Stratix IV

Figure 6: Configuration Register Spaces

Next, we describe the procedure to execute these steps in detail for both JTAG mode and PCIe modes of operation.

7.2.1. JTAG Mode Before you start, make sure that the board is installed in JTAG mode according to the steps described in section 4 and the FPGA has been programmed with the reference router bit-stream.

To configure the DE5 reference router in JTAG mode perform the following steps:

1. Open system console (Tools → System Console).

2. Change the current directory (synth/windows/) to jtag_config

> cd ../../sw/jtag_config

3. The script set_router_tables.tcl contains the user defined values for configuring the reference router’s routing table. In this example, we configure the MAC tables as shown in the Table 2.

Table 2: MAC Configuration Table

Interface MAC MAC Address ETHERNET 0 MAC 0 00:4e:46:32:43:00 ETHERNET 1 MAC 1 00:4e:46:32:43:01 ETHERNET 2 MAC 2 00:4e:46:32:43:02 ETHERNET 3 MAC 3 00:4e:46:32:43:03

In this example, we configure the IP tables as shown in the Table 3 .

Table 3: Interface IP Configuration Table

Interface IP IP Address ETHERNET 0 IP 0 192.168.1.1 ETHERNET 1 IP 1 192.168.2.1 ETHERNET 2 IP 2 10.2.3.1 ETHERNET 3 IP 3 10.1.4.1

In this example, we configure the IP tables as shown in the Table 4.

Table 4: IP LPM Configuration Table

LPM Table Entry Destination IP Mask Next Hop IP Port

1 10.1.4.0 255.255.255.0 10.1.4.1 3 2 10.2.3.0 255.255.255.0 10.2.3.1 2

In this example we configure the ARP tables as shown in the Table 5.

Table 5: ARP Configuration Table

Entry Next Hop IP MAC Address 1 10.1.4.1 00:4e:46:32:43:03 2 10.2.3.1 00:4e:46:32:43:01

4. To configure the routing table run the following command. > source set_router_tables.tcl

7.2.2. PCIe Mode Before you start, make sure that the board is installed in PCIe mode according to the steps described in section 4 and the FPGA has been programmed. Restart the host PC after downloading the reference router bit stream.

The software architecture to configure the FPGA via PCI Express bus is shown in Figure 7.

PCI Express Bus

DE5 NetFPGA Stratix V FPGA

Jungo Windriver

PCI Express DE5 Reference Router Configure Utility

OnRCSlaveWrite() OnRCSlaveRead()DMA_Write() DMA_Read()

Figure 7: Software Architecture

We use Jungo’s Win Driver [7] 32-bit to establish communication between the host PC running Windows XP and the Altera DE-5 board. Win Driver exposes two APIs (OnRCSlaveWrite and OnRCSlaveRead) for applications to read and write using the PCI Express bus. The Visual Studio 2010 based application uses these APIs to program routing table entries to the DE-5 board.

The file structure of the pcie_software folder is shown below:

Next, we describe the steps to configure the Interface MAC tables, Interface IP tables, IP LPM tables and ARP tables in PCIe mode below:

1. Double click on de5_router_pcie_diag.sln: The file is present at the location {project root}/sw/pcie_software/x86/msdev_2010

2. The Visual Studio 2010 window pops up with the following file structure.

3. Rebuild the design: In Visual Studio, click Build → Rebuild Solution.

4. Click the green Run button (shown in Figure 8) to start the configuration utility. The

configuration utility configures Interface MAC tables, Interface IP tables, IP LPM tables and ARP tables.

Figure 8: Start Debugging

8. Testing  The DE5 reference router can be tested once the reference router bit file has been downloaded into the FPGA and router tables have been programmed successfully into the FPGA. The reference router can be tested by sending sample packets to it and collecting the forwarded packets back. We have also ported the NetFPGA packet generator reference design to the DE5 platform. The DE5 based packet generator can be used to generate test packets and capture forwarded packets from the reference router. Unlike software-based packet generation utilities, packet generator allows packets to be transmitted and captured at line rate with a high degree of accuracy. The user can load a custom PCAP file into the packet generator and subsequently transmit the packets in the PCAP file at user defined rates through NetFPGA ports. For more information refer to the DE5 Packet Generator Design user guide [14].

8.1. Test  Setup 1. We configure a DE5 board with the packet generator reference design. You can download

the reference design from the website. 2. Setup the packet generator as per the DE5 packet generator user guide [14]. 3. The test topology is shown in Figure 9.

MAC 0

MAC 1

MAC 2

MAC 3

Packetgenerator

MAC 0

MAC 1

MAC 2

MAC 3

Referencerouter

10.2.3.2

10.1.4.2

Figure 9: Reference Router Test Setup

8.2. Running  the  Test   For testing, a pcap file available in the {project root}/sw/pcie_software/x86/msdev_2010 directory. This file contains the packet data to be sent. A different pcap file could be used as well. Run the packet generator software from the Visual Studio. Observe the output from the output.txt file.

9. References [1]. DE5 NetFPGA Project Web, http://keb302.ecs.umass.edu/DE5web/DE5_NetFPGA/ [2]. NetFPGA Project, http://netfpga.org/ [3]. NetFPGA reference router,

http://netfpga.org/foswiki/bin/view/NetFPGA/OneGig/ReferenceRouterWalkthrough [4]. Altera 10G Ethernet MAC, http://www.altera.com/literature/ug/10Gbps_MAC.pdf. [5]. Altera PCI Express, http://www.altera.com/literature/ug/ug_pci_express.pdf [6]. Altera PCI Express Wiki,

http://www.alterawiki.com/wiki/PCI_Express_in_Qsys_Example_Designs [7]. Jungo PCI Driver, http://www.jungo.com/st/windriver_windows.html [8]. Terasic Technologies, http://www.terasic.com.tw/en/ [9]. Altera Quartus II 13.1 and Qsys Builder, https://www.altera.com/download/software/quartus-

ii-se [10]. Microsoft Visual Studio 2010, http://www.microsoft.com/visualstudio/en-us/try [11]. USB Blaster, http://www.altera.com/literature/ug/ug_usb_blstr.pdf [12]. WinPcap for Windows, http://www.winpcap.org/install/default.htm [13]. WinPcap Developer Resources, http://www.winpcap.org/devel.htm [14]. Altera DE5 Packet Generator Userguide