Upload
others
View
28
Download
1
Embed Size (px)
Citation preview
Multi Channel DMA for PCI ExpressIP Design Example User Guide
Updated for Intel® Quartus® Prime Design Suite: 20.2
IP Version: 20.0.0
SubscribeSend Feedback
UG-20303 | 2020.08.05Latest document on the web: PDF | HTML
Contents
1. Terms and Acronyms...................................................................................................... 3
2. Design Example Detailed Description..............................................................................42.1. Design Example Overview.......................................................................................42.2. Hardware and Software Requirements...................................................................... 42.3. Avalon-ST PIO using MCDMA Bypass mode............................................................... 5
2.3.1. Simulation Result.......................................................................................52.3.2. Hardware Test Result................................................................................. 6
2.4. Avalon-ST Packet Generate/Check........................................................................... 62.4.1. Simulation Results..................................................................................... 72.4.2. Hardware Test Results................................................................................ 9
2.5. Avalon-ST Device-side Packet Loopback.................................................................. 122.5.1. Simulation Results................................................................................... 142.5.2. Hardware Test Results.............................................................................. 15
2.6. Avalon-MM PIO using MCDMA Bypass mode.............................................................172.6.1. Simulation Results................................................................................... 172.6.2. Hardware Test Results.............................................................................. 18
2.7. Avalon-MM DMA...................................................................................................182.7.1. Simulation Results................................................................................... 192.7.2. Hardware Test Results.............................................................................. 19
3. Design Example Quick Start Guide................................................................................ 223.1. Design Example Directory Structure....................................................................... 223.2. Generating the Example Design using Intel Quartus Prime.........................................23
3.2.1. Procedure............................................................................................... 233.3. Simulating the Design Example..............................................................................25
3.3.1. Testbench Overview................................................................................. 253.3.2. Example Testbench Flow for DMA Test with Avalon-ST Packet Generate/
Check Design Example..............................................................................263.3.3. Run the Simulation Script..........................................................................273.3.4. View the Results...................................................................................... 28
3.4. Compiling the Example Design in Intel Quartus Prime............................................... 283.5. Running the Design Example Application.................................................................29
3.5.1. Program the FPGA....................................................................................293.5.2. Set Up the Linux Software.........................................................................293.5.3. Run the Test Application Software.............................................................. 31
4. Revision History............................................................................................................32
Contents
Multi Channel DMA for PCI Express IP Design Example User Guide Send Feedback
2
1. Terms and AcronymsTable 1. Acronyms
Term Definition
PCIe* Peripheral Component Interconnect Express (PCI Express*)
DMA Direct Memory Access
MCDMA Multi Channel Direct Memory Access
PIO Programmed Input/Output
H2D Host-to-Device
D2H Device-to-Host
H2DDM Host-to-Device Data Mover
D2HDM Device-to-Host Data Mover
QCSR Queue Control and Status register
GCSR General Control and Status Register
IP Intellectual Property
HIP Hard IP
PD Packet Descriptor
QID Queue Identification
TIDX Queue Tail Index (pointer)
HIDX Queue Head Index (pointer)
TLP Transaction Layer Packet
IMMWR Immediate Write Operation
MRRS Maximum Read Request Size
CvP Configuration via Protocol
PBA Pending Bit Array
Avalon®-MM Avalon Memory-Mapped Interface
Avalon-ST Avalon Streaming Interface
UG-20303 | 2020.08.05
Send Feedback
Intel Corporation. All rights reserved. Agilex, Altera, Arria, Cyclone, Enpirion, Intel, the Intel logo, MAX, Nios,Quartus and Stratix words and logos are trademarks of Intel Corporation or its subsidiaries in the U.S. and/orother countries. Intel warrants performance of its FPGA and semiconductor products to current specifications inaccordance with Intel's standard warranty, but reserves the right to make changes to any products and servicesat any time without notice. Intel assumes no responsibility or liability arising out of the application or use of anyinformation, product, or service described herein except as expressly agreed to in writing by Intel. Intelcustomers are advised to obtain the latest version of device specifications before relying on any publishedinformation and before placing orders for products or services.*Other names and brands may be claimed as the property of others.
ISO9001:2015Registered
2. Design Example Detailed Description
2.1. Design Example Overview
The Multi Channel DMA for PCI Express IP Design Examples demonstrate a MultiChannel DMA solution for Intel® Stratix® 10 devices using the H-Tile Gen3 x16 hard IPand soft IP implemented in the FPGA fabric.
You can generate the design example from the Example Designs tab of the MultiChannel DMA for PCI Express IP Parameter Editor. For user interface, you can chooseeither Avalon-ST or Avalon-MM Interface. You can allocate up to 8 DMA channels whenAvalon-MM Interface type is selected. For Avalon-ST Interface, DMA channel andAvalon-ST port has 1:1 mapping. You can also configure the PCIe BAR2 size that ismapped to the Avalon-MM PIO Master port.
2.2. Hardware and Software Requirements
• Intel Quartus® Prime Pro Edition Software version 20.2
• Modelsim, VCS, or NCSim
• Intel Stratix 10 MX or GX FPGA Development Kit
For details on the design example simulation steps and running Hardware test, refer tothe Quick Start Guide.
For more information on development kits, refer to Intel Stratix 10 FPGA DevelopmentKits on Intel website.
UG-20303 | 2020.08.05
Send Feedback
Intel Corporation. All rights reserved. Agilex, Altera, Arria, Cyclone, Enpirion, Intel, the Intel logo, MAX, Nios,Quartus and Stratix words and logos are trademarks of Intel Corporation or its subsidiaries in the U.S. and/orother countries. Intel warrants performance of its FPGA and semiconductor products to current specifications inaccordance with Intel's standard warranty, but reserves the right to make changes to any products and servicesat any time without notice. Intel assumes no responsibility or liability arising out of the application or use of anyinformation, product, or service described herein except as expressly agreed to in writing by Intel. Intelcustomers are advised to obtain the latest version of device specifications before relying on any publishedinformation and before placing orders for products or services.*Other names and brands may be claimed as the property of others.
ISO9001:2015Registered
2.3. Avalon-ST PIO using MCDMA Bypass mode
Figure 1. Avalon-ST PIO using MCDMA Bypass mode
ninit_done
AVMM
Design Example Platform Designer System
Multi Channel DMAfor PCI Express
rx_pio_master
H2DDMA
PCIeHIP
Hosthip
_ser
ial
D2HDMA
resetIP
MEM_PIO
h2d_st_0h2d_st_1h2d_st_2h2d_st_3
d2h_st_0d2h_st_1d2h_st_2d2h_st_3
This design example enables Avalon-MM PIO master which bypasses the DMA path.The Avalon-MM PIO master allows application to perform single, non-bursting registerread/write operation with on-chip memory.
The design example includes the Multi Channel DMA for PCI Express IP Core with theparameters you specified and following components:
• resetIP – Reset Release IP that holds the Multi Channel DMA in reset until theentire Intel Stratix 10 FPGA fabric enters user mode
• MEM_PIO – On-chip memory for the PIO operation. Connected to the MCDMAAvalon-MM PIO Master (rx_pio_master) port that is mapped to PCIe BAR2
Transfer mode option supported in test application software (perfq_app) commandline:
• PIO test: -o
2.3.1. Simulation Result
Testbench writes 4 KB of incrementing pattern to on-chip memory and read back viaAvalon-MM PIO interface. This design example testbench doesn’t simulate H2D/D2Hdata movers.
Figure 2. Simulation Log
2. Design Example Detailed Description
UG-20303 | 2020.08.05
Send Feedback Multi Channel DMA for PCI Express IP Design Example User Guide
5
Figure 3. Simulation Waveform
2.3.2. Hardware Test Result
Figure 4. PIO Test-o option
2.4. Avalon-ST Packet Generate/Check
Figure 5. Avalon-ST Packet Generate/Check
ninit_done
AVMM
AVSTAVSTAVSTAVST
Design Example Platform Designer System
Multi Channel DMAfor PCI Express
rx_pio_master
H2DDMA
PCIeHIP
Host
hip_s
erial
AVSTAVSTAVSTAVST
D2HDMA
resetIP
MEM_PIO
h2d_st_0h2d_st_1h2d_st_2h2d_st_3
GEN_CHKd2h_st_0d2h_st_1d2h_st_2d2h_st_3
This design example performs H2D and D2H multi channel DMA via Avalon-STstreaming. The Multi Channel DMA for PCI Express IP core provides four independentAvalon-ST Source/Sink ports. DMA channel and Avalon-ST port has 1:1 mapping.
For H2D streaming, Multi Channel DMA sends the data to Avalon-ST packet checkervia four Avalon-ST Source ports. The Packet Checker validates the received data. ForD2H streaming, Multi Channel DMA receives the data from Avalon-ST packet generatorvia Avalon-ST Sink ports.
2. Design Example Detailed Description
UG-20303 | 2020.08.05
Multi Channel DMA for PCI Express IP Design Example User Guide Send Feedback
6
In addition, the design example enables Avalon-MM PIO master which bypasses theDMA path. It allows application to perform single, non-bursting register read/writeoperation with on-chip memory block. Also, test application software, perfq_app,uses the Avalon-MM PIO Master port to configure the Packet Generator and Checker.
The design example includes the Multi Channel DMA for PCI Express IP Core with theparameters you specified and following components:
• resetIP – Reset Release IP that holds the Multi Channel DMA in reset until theentire Intel Stratix 10 FPGA fabric enters user mode
• MEM_PIO – On-chip memory for the PIO operation. Connected to the MCDMAAvalon-MM PIO Master (rx_pio_master) port that is mapped to PCIe BAR2
• GEN_CHK – Packet Generator and Checker for MCDMA. Connected to the MCDMAAvalon-ST Source (h2d_st_x) and Avalon-ST Sink (d2h_st_x) ports
Transfer mode Options supported in test application software (perfq_app) commandline:
• PIO test: -o
• DMA test: -t (Tx), -r (Rx), -z (Bidirection)
2.4.1. Simulation Results
Note: For detailed description about the testbench for this design example, refer to ExampleTestbench Flow for DMA Test with Avalon-ST Packet Generate/Check Design Exampleon page 26.
Figure 6. H2D Simulation Log
2. Design Example Detailed Description
UG-20303 | 2020.08.05
Send Feedback Multi Channel DMA for PCI Express IP Design Example User Guide
7
Figure 7. H2D Simulation Waveform
Figure 8. D2H Simulation Log
2. Design Example Detailed Description
UG-20303 | 2020.08.05
Multi Channel DMA for PCI Express IP Design Example User Guide Send Feedback
8
Figure 9. D2H Simulation Waveform
2.4.2. Hardware Test Results
Figure 10. PIO Test-o option
Figure 11. H2D Avalon-ST Streaming-t option
Note: The perfq_app command transfers 1 GB of total transfer size with payload of 8192bytes in each descriptor in H2D direction (-t) for four channels. The File size is 127,i.e. sof is on the 1st descriptor and eof on the 127th descriptor. Without -v (datavalidtion) option, the command displays the bandwidth.
2. Design Example Detailed Description
UG-20303 | 2020.08.05
Send Feedback Multi Channel DMA for PCI Express IP Design Example User Guide
9
Figure 12. H2D Avalon-ST Streaming with Data Validation Enabled-t with -v option
Figure 13. D2H Avalon-ST Streaming-r option
2. Design Example Detailed Description
UG-20303 | 2020.08.05
Multi Channel DMA for PCI Express IP Design Example User Guide Send Feedback
10
Figure 14. D2H Avalon-ST Streaming with Data Validation Enabled-r with -v option
Figure 15. Bidirectional Avalon-ST Streaming-z option
2. Design Example Detailed Description
UG-20303 | 2020.08.05
Send Feedback Multi Channel DMA for PCI Express IP Design Example User Guide
11
Figure 16. Bidirectional Avalon-ST Streaming with Data Validation Enabled-z with -v option
2.5. Avalon-ST Device-side Packet Loopback
Figure 17. Avalon-ST Device-side Packet Loopback
ninit_done
AVMM
AVSTAVSTAVSTAVST
Design Example Platform Designer System
Multi Channel DMAfor PCI Express
rx_pio_master
H2DDMA
PCIeHIP
Host
hip_s
erial
AVSTAVSTAVSTAVST
D2HDMA
resetIP
MEM_PIO
h2d_st_0h2d_st_1h2d_st_2h2d_st_3
FIFO_ST0FIFO_ST1FIFO_ST2FIFO_ST3
d2h_st_0d2h_st_1d2h_st_2d2h_st_3
2. Design Example Detailed Description
UG-20303 | 2020.08.05
Multi Channel DMA for PCI Express IP Design Example User Guide Send Feedback
12
This design example performs H2D and D2H multi channel DMA via Avalon-STstreaming. The Multi Channel DMA for PCI Express IP core provides four independentAvalon-ST Source/Sink ports. DMA channel and Avalon-ST port has 1:1 mapping.
For H2D streaming, Multi Channel DMA sends the data to Avalon-ST loopback FIFOsvia four Avalon-ST Source ports. For D2H streaming, Multi Channel DMA receives thedata from Avalon-ST loopback FIFOs via Avalon-ST Sink ports.
In addition, the design example enables Avalon-MM PIO master which bypasses theDMA path. It allows application to perform single, non-bursting register read/writeoperation with on-chip memory block.
The design example includes the Multi Channel DMA for PCI Express IP Core with theparameters you specified and following components:
• resetIP – Reset Release IP that holds the Multi Channel DMA in reset until theentire Intel Stratix 10 FPGA fabric enters user mode
• MEM_PIO – On-chip memory for the PIO operation. Connected to the MCDMAAvalon-MM PIO Master (rx_pio_master) port that is mapped to PCIe BAR2
• FIFO_ST0, FIFO_ST1, FIFO_ST2, and FIFO_ST3 – Avalon-ST FIFOs forstreaming loopback. Connected to the MCDMA Avalon-ST Source (h2d_st_x) andAvalon-ST Sink (d2h_st_x) ports
Transfer mode options supported in test application software (perfq_app) commandline:
• PIO test: -o
• DMA test: -i (performance loopback operation where the Tx and Rx are run in twodifferent threads), -v (enable data validation)
— -i without -v flag displays the throughput per channel
2. Design Example Detailed Description
UG-20303 | 2020.08.05
Send Feedback Multi Channel DMA for PCI Express IP Design Example User Guide
13
2.5.1. Simulation Results
Figure 18. Simulation Log
2. Design Example Detailed Description
UG-20303 | 2020.08.05
Multi Channel DMA for PCI Express IP Design Example User Guide Send Feedback
14
Figure 19. H2D Simulation Waveform
Figure 20. D2H Simulation Waveform
2.5.2. Hardware Test Results
Figure 21. PIO Test-o option
2. Design Example Detailed Description
UG-20303 | 2020.08.05
Send Feedback Multi Channel DMA for PCI Express IP Design Example User Guide
15
Figure 22. Performance Test-i option
Figure 23. Data Validation Test-i with -v option
2. Design Example Detailed Description
UG-20303 | 2020.08.05
Multi Channel DMA for PCI Express IP Design Example User Guide Send Feedback
16
2.6. Avalon-MM PIO using MCDMA Bypass mode
Figure 24. Avalon-MM PIO using MCDMA Bypass mode
ninit_done
AVMM
Design Example Platform Designer System
Multi Channel DMAfor PCI Express
rx_pio_master
h2ddm_master
H2DDMA
PCIeHIP
Hosthip
_ser
ial
d2hdm_master
D2HDMA
resetIP
MEM_PIO
This design example enables Avalon-MM PIO master which bypasses the DMA path.The Avalon-MM PIO master allows application to perform single, non-bursting registerread/write operation with on-chip memory.
The design example includes the Multi Channel DMA for PCI Express IP Core with theparameters you specified and other supporting components:
• resetIP – Reset Release IP that holds the Multi Channel DMA in reset until theentire Intel Stratix 10 FPGA fabric enters user mode
• MEM_PIO – On-chip memory for the PIO operation. Connected to the MCDMAAvalon-MM PIO Master (rx_pio_master) port that is mapped to PCIe BAR2
Transfer mode option supported in test application software (perfq_app) commandline:
• PIO test: -o
2.6.1. Simulation Results
Testbench writes 4 KB of incrementing pattern to on-chip memory and read back viaAvalon-MM PIO interface. This design example testbench doesn’t simulate H2D/D2Hdata movers.
Figure 25. Simulation Log
2. Design Example Detailed Description
UG-20303 | 2020.08.05
Send Feedback Multi Channel DMA for PCI Express IP Design Example User Guide
17
Figure 26. Simulation Waveform
2.6.2. Hardware Test Results
Figure 27. PIO Test-o option
2.7. Avalon-MM DMA
Figure 28. Avalon-MM DMA
ninit_done
AVMM
Design Example Platform Designer System
Multi Channel DMAfor PCI Express
rx_pio_master
h2ddm_master
H2DDMA
PCIeHIP
Host
hip_s
erial
d2hdm_master
D2HDMA
resetIP
MEM_PIO
AVMM
AVMM
MEM
This design example performs H2D and D2H multi channel DMA via Avalon-MMmemory-mapped interface. The Multi Channel DMA for PCI Express IP core providesone Avalon-MM Write/Read Master port. You can allocate up to eight DMA channelswhen generating this example design.
For H2D DMA, Multi Channel DMA H2D data mover writes the data to on-chip memoryvia Avalon-MM Write Master port. For D2H DMA, Multi Channel DMA D2H data moverreads the data from on-chip memory via Avalon-MM Read Master port.
In addition, the design example enables Avalon-MM PIO master which bypasses theDMA path. It allows application to perform single, non-bursting register read/writeoperation with on-chip memory block.
2. Design Example Detailed Description
UG-20303 | 2020.08.05
Multi Channel DMA for PCI Express IP Design Example User Guide Send Feedback
18
The design example includes the Multi Channel DMA for PCI Express IP Core with theparameters you specified and following components:
• resetIP – Reset Release IP that holds the Multi Channel DMA in reset until theentire Intel Stratix 10 FPGA fabric enters user mode
• MEM_PIO – On-chip memory for the PIO operation. Connected to the MCDMAAvalon-MM PIO Master (rx_pio_master) port that is mapped to PCIe BAR2
• MEM – Dual port on-chip memory. One port is connected to the Avalon-MM WriteMaster (h2ddm_master) and the other port to Avalon-MM Read Master(d2hdm_master)
Transfer mode options supported in test application software (perfq_app) commandline:
• PIO test: -o
• DMA test: -t (Tx), -r (Rx)
2.7.1. Simulation Results
Testbench writes 4 KB of incrementing pattern to on-chip memory and read back viaAvalon-MM PIO interface. In the current release, this design example testbench doesnot simulate data movement through Avalon-MM Write and Read Master ports.
Figure 29. Simulation Log
Figure 30. Simulation Waveform
2.7.2. Hardware Test Results
Figure 31. PIO Test-o option
2. Design Example Detailed Description
UG-20303 | 2020.08.05
Send Feedback Multi Channel DMA for PCI Express IP Design Example User Guide
19
Figure 32. H2D Avalon-MM Write-t option
Figure 33. H2D Avalon-MM Write with Data Validation Enabled-t -v option
2. Design Example Detailed Description
UG-20303 | 2020.08.05
Multi Channel DMA for PCI Express IP Design Example User Guide Send Feedback
20
Figure 34. D2H Avalon-MM Read-r option
2. Design Example Detailed Description
UG-20303 | 2020.08.05
Send Feedback Multi Channel DMA for PCI Express IP Design Example User Guide
21
3. Design Example Quick Start GuideUsing Intel Quartus Prime software, you can generate a design example for the MultiChannel DMA for PCI Express (PCIe) IP core.
The generated design example reflects the parameters that you specify. The designexample automatically creates the files necessary to simulate and compile in the IntelQuartus Prime software. You can download the compiled design to your FPGADevelopment Board. To download to custom hardware, update the Intel Quartus PrimeSettings File (.qsf) with the correct pin assignments.
Figure 35. Design Example Development Steps
DesignExample
Generation
Compilation(Simulator)
FunctionalSimulation
HardwareTesting
Compilation(Quartus Prime)
3.1. Design Example Directory Structure
Table 2. Directory Structure
Directory / File Sub-directory /File
Sub-directory /File
Sub-directory /File
Sub-directory /File
Note
pcie_ed
sim
pcie_ed.v Design exampletop-level HDL
<simulators> <simulation scripts> pcie_edsimulationdirectory
synth pcie_ed.v Design exampletop-level HDL
<Components automatically generated by Platform Designer>
pcie_ed_tb pcie_ed_tb simpcie_ed_tb.v Testbench
including IntelFPGA BFM
continued...
UG-20303 | 2020.08.05
Send Feedback
Intel Corporation. All rights reserved. Agilex, Altera, Arria, Cyclone, Enpirion, Intel, the Intel logo, MAX, Nios,Quartus and Stratix words and logos are trademarks of Intel Corporation or its subsidiaries in the U.S. and/orother countries. Intel warrants performance of its FPGA and semiconductor products to current specifications inaccordance with Intel's standard warranty, but reserves the right to make changes to any products and servicesat any time without notice. Intel assumes no responsibility or liability arising out of the application or use of anyinformation, product, or service described herein except as expressly agreed to in writing by Intel. Intelcustomers are advised to obtain the latest version of device specifications before relying on any publishedinformation and before placing orders for products or services.*Other names and brands may be claimed as the property of others.
ISO9001:2015Registered
Directory / File Sub-directory /File
Sub-directory /File
Sub-directory /File
Sub-directory /File
Note
<simulators> <simulationscript>
Testbenchsimulationdirectory
ip pcie_ed_tb DUT_pcie_tb_ip Intel FPGA BFM(RP)
pcie_ed_tb.qsys TestbenchPlatform Designerfile
pcie_ed.ipx
software
kernel
common
driver kmod <kernel driverfiles>
Kernel driver
Licenses
user
cli
perfq_app
<test applicationsoftware>
Test Application
README Readme file
sample ref.c Reference APIflow
common
include regs MCDMA and PktGen/Chk registers
mk
src
libmqdma <user space library files> User space library
Licenses
Readme Readme file
readme Readme file
ip pcie_ed <Design example IP components>
pcie_ed.qpf Quartus projectfile
pcie_ed.qsf Quartus settingfile
pcie_ed.qsys Design examplePlatform Designerfile
3.2. Generating the Example Design using Intel Quartus Prime
Figure 36. Design Example Generation
Start ParameterEditor
Specify IP Variationand Select Device
SelectDesign Parameters
SpecifyExample Design andSelect Target Board
InitiateDesign Generation
3.2.1. Procedure
3. Design Example Quick Start Guide
UG-20303 | 2020.08.05
Send Feedback Multi Channel DMA for PCI Express IP Design Example User Guide
23
1. In the Intel Quartus Prime Pro Edition software, create a new project (File → NewProject Wizard).
2. Specify the Directory, Name, and Top-Level Entity.
3. For Project Type, accept the default value, Empty project. Click Next.
4. For Add Files click Next.
5. For Family, Device & Board Settings, select Intel Stratix 10(GX/SX/MX/TX) and the Target Device for your design.
Note: The selected device is only used if you select None in Step 10f below.
6. Click Finish.
7. In the IP Catalog locate and add the Multi Channel DMA for PCI Express*which brings up the IP Parameter Editor.
8. In the New IP Variant dialog box, specify a name for your IP. Click Create.
9. On the IP Settings tabs, specify the parameters for your IP variation.
10. On the Example Designs tab, make the following selections:
a. For Currently Selected Example Design, select a design example from apulldown menu.
b. Available design examples depends on the Interface type setting in MCDMASettings under IP Settings tab. Available design examples for Avalon-STInterface type:
• PIO using DMA Bypass Mode
• Packet Generate/Check
• Device-side Packet Loopback
Available design examples for Avalon-MM Interface type:
• PIO using DMA Bypass Mode
• Avalon-MM DMA
c. For Example Design Files, turn on the Simulation and Synthesis options.If you do not need these simulation or synthesis files, leaving thecorresponding option(s) turned off significantly reduces the example designgeneration time.
d. For Select simulation Root Complex BFM, choose the appropriate BFM:
i. Intel FPGA BFM: This bus functional model (BFM) supports x16configurations by down training to x8.
ii. Third-party BFM: If you want to simulate all 16 lanes, use a third-partyBFM. If you have an Avery BFM installed and need information aboutsimulating with the Avery BFM, contact your local FAE or salesrepresentative.
e. For Generated HDL Format, only Verilog is available in the current release.
f. For Target Development Kit, select the appropriate option.
Note: If you select None, the generated design example targets the devicespecified. Otherwise, the design example uses the device on theselected development board. If you intend to test the design inhardware, make the appropriate pin assignments in the .qsf file.
3. Design Example Quick Start Guide
UG-20303 | 2020.08.05
Multi Channel DMA for PCI Express IP Design Example User Guide Send Feedback
24
11. Select Generate Example Design to create a design example that you cansimulate and download to hardware. If you select one of the Intel Stratix 10development boards, the device on that board supersedes the device previouslyselected in the Intel Quartus Prime Pro Edition project if the devices are different.When the prompt asks you to specify the directory for your example design, youcan choose to accept the default directory ./intel_pcie_mcdma_0_example_design or choose another directory.
12. Click Close on Generate Example Design Completed message.
13. Close the IP Parameter Editor. Click File → Exit. When prompted with Savechanges?, you do not need to save the .ip. Click Don’t Save.
3.3. Simulating the Design Example
3.3.1. Testbench Overview
Figure 37. Testbench Platform Designer View
Testbench Platform Designer file path: pcie_ed_tb/pcie_ed_tb.qsys
The design example, pcie_ed_inst, is generated with x16. The Intel FPGA BFM,DUT_pcie_tb, can support up to x8 link. The BFM supports the testbench simulationby down-training to x8 link. If you want to simulate x16 link, you can use a third-partyBFM.
The testbench uses a Root Port driver module, altpcietb_bfm_rp_gen3_x8.sv(Path: pcie_ed_tb/ip/pcie_ed_tb/DUT_pcie_tb_ip/altera_pcie_s10_tbed_191/sim), to exercise the target memory and DMAchannel in the Endpoint. This is the module that you can modify to vary thetransactions sent to the example Endpoint design or your own design.
For more information about Intel FPGA BFM, refer to Intel Stratix 10 Avalon streamingand SR-IOV Interface for PCI Express Solutions User Guide (Section 9.3 Root Port BFMOverview).
Note: The Endpoint BAR0 is configured for 64-bit 4 MB address space used for the MultiChannel DMA control registers. The BAR2 is used for 64-bit PIO Access.
3. Design Example Quick Start Guide
UG-20303 | 2020.08.05
Send Feedback Multi Channel DMA for PCI Express IP Design Example User Guide
25
Related Information
Intel Stratix 10 Avalon streaming and SR-IOV Interface for PCI Express Solutions UserGuide (Section 9.3 Root Port BFM Overview)
3.3.2. Example Testbench Flow for DMA Test with Avalon-ST PacketGenerate/Check Design Example
The DMA testbench for the Avalon-ST Packet Generate/Check design exampledemonstrates the following two major tasks:
• Host-to-Device: Transferring packets stored in the host memory to the PacketChecker in the design example user logic, where a checker module verifies theintegrity of the packet
• Device-to-Host: Packets generated from a Generator module are transferred to thehost memory where the host checks the packet integrity
Note: This testbench implements transfer of one packet with length of 4096 bytes.
The DMA testbench for the design example completes the following tasks for each ofthe 4 ports supported by the DUT:
1. Set up 4096 bytes of incrementing data pattern for testing data movement fromthe host to the device and then back to the host.
2. Write the expected packet length value (4096 bytes) to the Packet Generation andChecker in the design example user logic through the PIO. This value is used bythe Packet checker module for testing packet integrity.
3. MSI-X is enabled and configured for launching a memory write to signal the end ofeach descriptor’s DMA transaction. Write-Back function is kept disabled for thesimulation.
4. Set up the H2D (Host-to-Device) queue in the Multi Channel DMA.
5. Set up three H2D descriptors in the host memory, with the source addresspointing to the incrementing data pattern locations in the host memory. The startof packet (SOF) and end of packet (EOF) markers along with packet length areindicated in the descriptors.
6. At the last step of the Queue programming, the Multi Channel DMA tail pointerregister is written, which triggers the Multi Channel DMA to start the H2D DMAtransaction.
7. The previous step instructs the H2D Data Mover to fetch the descriptors from thehost memory.
8. The Multi Channel DMA H2D Data Mover reads the data from the host memory andforwards the packet to the Packet Generator and Checker through the AVSTStreaming interface.
9. The checker module receives the packet and checks for integrity by testing thedata pattern, length as expected and proper receipt of the “end of packet” marker.If the packet is found to be proper, the good packet count is incremented by 1 elsethe bad packet count is incremented.
10. The testbench does a PIO read access of the Good Packet Count and Bad PacketCount registers and displays the test success or failure status.
11. MSI-X write commands are triggered for every description or completion which arechecked by the testbench for proper receipt.
3. Design Example Quick Start Guide
UG-20303 | 2020.08.05
Multi Channel DMA for PCI Express IP Design Example User Guide Send Feedback
26
12. Next, set up the D2H (Device-to-Host) Queue.
13. Setup three D2H descriptors in the host memory, with the destination addresspointing to a new address space in host memory which is pre-filled with all zeroes.
14. At the last step of the Queue programming, the Multi Channel DMA tail pointerregister is written, which triggers the Multi Channel DMA to start the D2H DMAtransaction.
15. The previous step instructs the H2D Data Mover to fetch the descriptors from thehost memory to start the D2H DMA transaction.
16. The Multi Channel DMA D2H Data Mover reads the incoming packet from thePacket Generator and writes the data to the host memory according to thedescriptors fetched in the previous step.
17. MSI-X write commands are triggered for every description completion which arechecked by the testbench for proper receipt.
18. Compares the data written back to the system memory in D2H task with thestandard incrementing pattern and declare test success/failure.
The simulation reports Simulation stopped due to successful completionif no errors occur.
3.3.3. Run the Simulation Script
Figure 38. Simulation Script
Change toTestbench Directory
Run<Simulation Script>
AnalyzeResults
1. Change to the testbench simulation directory, pcie_ed_tb/pcie_ed_tb/sim/<simulators>.
2. Run the simulation script for the simulator of your choice. Refer to the table below.
3. Analyze the results.
Table 3. Steps to run the simulation
Simulator Simulation Directory Instructions
ModelSim <example_design>/pcie_ed_tb/pcie_ed _tb/sim/mentor/
1. Invoke vsim (by typing vsim, whichbrings up a console window whereyou can run the followingcommands).
2. do msim_setup.tcl
continued...
3. Design Example Quick Start Guide
UG-20303 | 2020.08.05
Send Feedback Multi Channel DMA for PCI Express IP Design Example User Guide
27
Simulator Simulation Directory Instructions
Note: Alternatively, instead ofdoing Steps 1 and 2, youcan type: vsim -c -domsim_setup.tcl
3. ld_debug
4. run -all
5. A successful simulation ends withthe following message:"Simulation stopped due tosuccessful completion!"
VCS <example_design> /pcie_ed_tb/pcie_ed _tb/sim/synopsys/vcs
1. sh vcs_setup.sh USER_DEFINED_COMPILE_OPTIONS="" USER_DEFINED_ELAB_OPTIONS="-xlrm\ uniq_prior_final" USER_DEFINED_SIM_OPTIONS=""
2. A successful simulation ends withthe following message:"Simulation stopped due tosuccessful completion!"
NCSim <example_design>/pcie_ed_tb/pcie_ed_tb/sim/cadence
1. sh ncsim_setup.sh USER_DEFINED_SIM_OPTIONS="" USER_DEFINED_ELAB_OPTIONS ="-timescale\ 1ns/1ps"
2. A successful simulation ends withthe following message:"Simulation stopped due tosuccessful completion!"
3.3.4. View the Results
To view the Simulation Logs, Simulation Waveforms and Hardware Test Results foreach design example, refer to Design Example Detailed Descirption chapter of thisdocument.
3.4. Compiling the Example Design in Intel Quartus Prime
To compile the example design, follow these steps:
1. Navigate to the design example directory,intel_pcie_mcdma_0_example_design, and open the Intel Quartus Primeproject file, pcie_ed.qpf in Intel Quartus Prime Pro Edition software.
2. On the Processing menu, select Start Compilation.
Figure 39. Design Example Compilation
3. Design Example Quick Start Guide
UG-20303 | 2020.08.05
Multi Channel DMA for PCI Express IP Design Example User Guide Send Feedback
28
3.5. Running the Design Example Application
3.5.1. Program the FPGA
1. Connect a FPGA programming cable to the Intel Stratix 10 FPGA DevelopmentBoard
2. On the Tools menu, select Programmer
3. In the Programmer, click Hardware Setup and verify the Intel Stratix 10 FPGADevelopment Board is detected in Hardware Setting tab and JTAG Settings tab
4. Select Auto Detect to detect the JTAG device chain
5. Select the target FPGA device in the JTAG chain, select Change File, and selectthe pcie_ed.sof
6. Select Start to start programming
Figure 40. Programming Stratix 10 MX FPGA Development Board
3.5.2. Set Up the Linux Software
3.5.2.1. Set the default huge pages
1. Modify the default huge pages setting in grub files as follows:
Edit /etc/default/grub file. Add the following in GRUB_CMDLINE_LINUXparamenter:
"default_hugepagesz=1G hugepagesz=1G hugepages=40"
After the edit, the file will look as below:
GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved
3. Design Example Quick Start Guide
UG-20303 | 2020.08.05
Send Feedback Multi Channel DMA for PCI Express IP Design Example User Guide
29
GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console" GRUB_CMDLINE_LINUX="default_hugepagesz=1G intel_iommu=on iommu=pt intel_pstate=disable hugepagesz=1G hugepages=40 panic=1 crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet" GRUB_DISABLE_RECOVERY="true"
2. Generate GRUB configuration files. Check if /sys/firmware/efi exists. If itexists, the system is EFI based. Otherwise, the system is a legacy system.
In case of legacy system execute following command
$ grub2-mkconfig -o /boot/grub2/grub.cfg
In case of EFI based system, execute following command
$ grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg
3. Reboot the system.
4. Verify the above changes
$ cat /proc/cmdline
The output should include the following:
default_hugepagesz=1G hugepagesz=1G hugepages=40
5. Set the huge pages
$ echo 10 > /proc/sys/vm/nr_hugepages
3.5.2.2. Install the Linux Kernel Driver
1. Install the UIO driver
$ modprobe uio
2. Build the Multi Channel DMA kernel driver and load
$ cd software/kernel$ make -C driver/kmod/
3. Install the kernel driver
$ insmod driver/kmod/ifc_uio.ko
4. Verify if the kernel driver is loaded
$ lspci -d 1172:000 -v | grep ifc_uio
Kernel driver in use: ifc_uio
3.5.2.3. Build and Install User Space Library
1. Build the library
$ cd software/user $ make -C libmqdma/
2. Load the library
3. Design Example Quick Start Guide
UG-20303 | 2020.08.05
Multi Channel DMA for PCI Express IP Design Example User Guide Send Feedback
30
For 64 bit system:
$ rm -f /usr/lib64/libmqdmasoc.so$ cp libmqdma/libmqdmasoc.so /usr/lib64/
For 32 bit system:
$ rm -f /usr/lib/libmqdmasoc.so$ cp libmqdma/libmqdmasoc.so /usr/lib/
3. Verify that ldconfig output contains libmqdma
$ ldconfig -v | grep libmqdmasoc.so
3.5.3. Run the Test Application Software
1. Build the perfq_app test application software and check the available commandline options using -h.
$ cd cli/perfq_app/$ make clean$ make$ ./perfq_app -h
Note: For more information on perfq_app command options, refer to the READMEfile located in software/user/cli/perfq_app directory.
2. Perform the PIO test to check if the hardware setup is correct, if successful, theapplication will show as Pass status as shown below:
3. Perform the DMA test with design example. Refer to Hardware Test results foreach design example in the Design Example Detailed Description chapter.
3. Design Example Quick Start Guide
UG-20303 | 2020.08.05
Send Feedback Multi Channel DMA for PCI Express IP Design Example User Guide
31
4. Revision HistoryTable 4. Revision History for Multi Channel DMA for PCI Express IP Design Example
User Guide
Date Intel QuartusPrime Version
IP Version Changes
2020.08.05 20.2 20.0.0 Initial Release
UG-20303 | 2020.08.05
Send Feedback
Intel Corporation. All rights reserved. Agilex, Altera, Arria, Cyclone, Enpirion, Intel, the Intel logo, MAX, Nios,Quartus and Stratix words and logos are trademarks of Intel Corporation or its subsidiaries in the U.S. and/orother countries. Intel warrants performance of its FPGA and semiconductor products to current specifications inaccordance with Intel's standard warranty, but reserves the right to make changes to any products and servicesat any time without notice. Intel assumes no responsibility or liability arising out of the application or use of anyinformation, product, or service described herein except as expressly agreed to in writing by Intel. Intelcustomers are advised to obtain the latest version of device specifications before relying on any publishedinformation and before placing orders for products or services.*Other names and brands may be claimed as the property of others.
ISO9001:2015Registered