1
PCI Express Analyzer
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
2
הטכניון - מכון טכנולוגי לישראל
Technion - Israel institute of technology
Final Presentation
Samuel Amir , Danny Volkind
Mr. Orbach Mony
Analyzer Core
3
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
Agenda
• PCI - Express – Reminder
• Link and Lane Training - Overview
• Project Goals
• Project features and capabilities
• PCI-Express Generator - Brief
4
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
Agenda (cont.)
• Project Block Diagram
• Demonstration
• Future Improvement Procedures
5
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
PCI - Express
Reminder
6
PCI-Express Topology
Ref. Clock Ref. Clock
Dev
ice
A Device B
TX+TX-
RX+RX-
TX+TX-
RX-RX+
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
• Differential
• Dual Simplex
• Speeds of 2.5Gb/sec
7
Packet Formation
Packet formation reflects
layered architecture.
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
Transaction
Data Link
Physical
Transaction Layer
DataHeader
Data Link Layer
CRCSequenceNumber
Physical Layer
FrameFrame
8
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
8/10b conversion (4.2.1)
8b/10b Encoded Byte Value
Byte Value 00H
• Convert each byte to a 10bit character according to a pre-
defined table.
• Extra characters are used as control characters or not used.
• Embedded clocking-> No need to add clock traces.
• Error detection (running disparity).
• DC balancing
• Reduces ISI.
9
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
PCI Express scrambling (4.2.3)• Assures no constant pattern is transmitted.
• Spread the energy transmitted in one frequency to
different frequencies-> reduces EMI! (and gets FCC
approval)
• Only data characters are encoded.
• Encoding is done using a linear feedback shift register.
• Decoding is done using the same process at the receiver
side.
10
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
Frequency Mismatch (4.2.7)
How does the protocol cope with the fact that each
device is feed by a different clock source?
Ref. Clock Ref. Clock
Dev
ice
A Device B
TX+TX-
RX+RX-
TX+TX-
RX-RX+
11
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
Frequency Mismatch (4.2.7)
Input @ 2501MHz Output @ 2500MHz
Input Shift Register is filling up faster than it is emptied!
Input @ 2500MHz Output @ 2501MHz
Input Shift Register is emptied faster than it is filled!
If we take 600ppm difference between clocks, the
transmitter and receiver clocks can shift one clock
every 1666 clocks!
12
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
Frequency Mismatch-Solution• The protocol issues a SKIP ordered set that can be
skipped so that the input shift register can be partially
cleared!
• The SKIP ordered sets insertion time as dictated by the
protocol is between 1180 and 1538 symbol times.
• Calculated allowed deviance : ± 300ppm !
• This is known as “clock tolerance compensation
mechanism”.
13
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
Link & Lane
Training
14
Link and Lane Training
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
Lane - a single set of differential RX TX pairs
Link - a collection of lanes connecting two
PCI-Express Devices.
Dev
ice
A Device B
TX+TX-
RX+RX-
TX+TX-
RX-RX+
x1 Lane wide Link
15
Link and Lane Training (cont)
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
Training - a process aimed at turning a collection of
available lanes into a properly functioning link.
Elements established during training :• Physical Level
• SERDES lock.• Symbol alignment.
• Link Level link configuration• Link data rate• Link width• Etc.
16
Link and Lane Training (cont)
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
Configuration
Disable
Polling Loopback
Detect
HotReset
Recovery
L0s
L2
L3
L0(Full On)
L2/L3Ready
L1
Link Training States
Link Power Manaement States
Active Power Management States
To Detect
17
Training Ordered Sets
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
• Group of 16 symbols.• Used During Polling State (TS1,TS2)• Establish alignment• Exchange Physical layer parameters• Not scrambled!
18
Training Ordered Sets (cont)
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
TS1
19
Fast Training Sequence
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
Used to re-establish bit and symbol lock when transitioning out of the LO power management state.
K28.1K28.1K28.1K28.5
FTS pattern
20
Skipped Ordered Set (CTC)
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
Compensate for differences in frequencies between bit rates at two devices sharing a mutual Link (Clock Tolerance Compensation)
Skipped Ordered Set pattern
K28.0K28.0K28.0K28.5
21
Electrical Idle
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
• Steady state condition• Transmitter differential pair held at fixed value• Must remain at this state at least 20nsec• Must attempt to detect a receiver within 100msec
22
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
DLLP packet
DLLP ContentsDLLP Contents44bytebyte
CRCCRC2byte2byte
Data Link LayerData Link Layer
FrameFrame1byte1byte
FrameFrame1byte1byte
Physical Layer
23
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
TLP packet
Transaction LayerTransaction Layer
Data0-4Kbyte
Header12/16byte
Data Link LayerData Link Layer
LCRCLCRC4byte4byte
Sequence#Sequence#2byte2byte
Physical Layer
FrameFrame1byte1byte
FrameFrame1byte1byte
24
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
Packet Header
FmtFmt
Address/RoutingAddress/Routing
TypeType Requestor IDRequestor ID RsrvRsrv Traffic ClassTraffic Class
LengthLength AttrAttr TagTag
16/1216/12ByteByte
RsrvRsrv Byte EnablesByte Enables
Data PayloadData PayloadIndicator and Indicator and
16/12B header flag16/12B header flag
Memory, I/O, Config,Memory, I/O, Config, Message; RequestMessage; Request
or Completionor Completion
Bus#, Device#, Bus#, Device#, Function#,Function#,
Virtual ChannelVirtual ChannelSupportSupport
TransactionTransactionTagTag
ReservedReservedfor futurefor futureexpansionexpansion
Attributes:Attributes:Snoop,Snoop,
OrderingOrdering
First DW BE;First DW BE;Last DW BELast DW BE
RequestedRequestedLength or Length or
Payload SizePayload Size
Header
Transaction LayerTransaction Layer
Data
Data Link LayerData Link Layer
CRCSequenceNumber
Physical LayerPhysical Layer
FrameFrame
25
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
Project Goals
26
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
Project Goal Overview• PCI-Express packets capturing at wire speed of 2.5Gbps
• Selective filtering – work modes
• TLP analysis – header based filtering
• Simple register based user interface
• RS232 accessible.
27
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
Project Features
&
Capabilities
28
Project features
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
• Receiving PCIe communication at 2.5 Gbps.
• Full line synchronization capability according
to PCIe spec.
• Handling of realignment and clock tolerance
compensation events on the fly.
• Invalid symbol filtering.
29
Project features (cont.)
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
• Detection of and field extraction for all PCIe training
sequences.
• Selective descrambling of the received data.
• Accumulative user-controlled counters for statistic
purposes.
• Data stream marking and preliminary analysis.
30
Projects features (cont.)• 6 Available work modes:
• Wire Speed Capture.
• Capture TS1 / TS2
• Capture DLLP
• Capture TLP
• Selective TLP capture
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
31
Projects features (cont.)•TLP filtering versatility:
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
TLP Type field 5 Bit
TLP format field 2 Bit
TLP Traffic Class 3 Bit
TLP length 10 Bit
TLP Attribute 2 Bit
TLP Tag 8 Bit
TLP Byte Enables or Message 8 Bit
TLP Sequence num 12 Bit
TLP Bus num 8 Bit
TLP device number 5 Bit
TLP function number 3 Bit
TLP poisoned packet 1 Bit
TLP Type field Mask 5 Bit
TLP format field Mask 2 Bit
TLP Traffic Class Mask 3 Bit
TLP length Mask 10 Bit
TLP Attribute Mask 2 Bit
TLP Tag Mask 8 Bit
TLP Byte Enables or Message Mask 8 Bit
TLP Sequence num Mask 12 Bit
TLP Bus num Mask 8 Bit
TLP device number Mask 5 Bit
TLP function number Mask 3 Bit
TLP poisoned packet Mask 1 Bit
32
Project features (cont.)
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
•Storage
• Fully user controlled dynamically allocated
16K x 32 memory.
• Captured data is saved with preliminary
analysis.
• Bursts and capture events are separated
allowing selective extraction.
33
Project features (cont.)
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
• LCD Interface
• Built-in LCD micro-controller implementation.
• 1 Kbyte user memory for display commands.
• Display file can be loaded on the fly.
34
Project features (cont.)
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
• User Interface
• RS232 serial link operating at 115.2Kbps.
• Simple comm. protocol implementation
allowing read and write commands accessing
32bit address space with 16bit data.
• All the core features are accessible.
35
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
Generator - Overview• Capable of generating all PCIe traffic.
• Fully user-controlled.
• PCIe compatible including clock tolerance
compensation simulation.
• Can be set to continuous or single shot mode.
36
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
Generator - Overview
37
Basic Block Diagram
PCIe
MCL PDM
SDMPLA
Modular Single-channel UnitSDRAM
MCSU
AD
DR
DA
TA
CCIU
LQAM MGR
Gigabit Receiver1
1
Decryption Module2
Wrap Filter3
Packet Filter4
MSU Control5
Link Assessment6
2
34
5
6 Memory Controller7
Central Controller8
7
8
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
Physical
Physical
Transaction
Physical
Physical
Data Link
Transaction
Physical Data Link
38
Modular Single-channel Unit
MGR
SDM
LQAM
MCSU
PLA
PDM
PCIe
MCL
CCIU
Modular Single-channel Unit#2
PCIe
SDRAM8Mx16
SDRAM8Mx16
Addr
Data
Control
Data
Top Level
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
39
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
Project Block Diagram
40
PDM – LFSR
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
8 bit values generated by LFSR for repeated data value of 0
41
PDM – LFSR (cont)
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
Scrambling spectral power distribution for repeated data value of 0
42
PDM – LFSR (cont)
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
Reset Value (Comma)
Figure 1
43
PDM – LFSR (cont)
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
LFSR1_out
LFSR2_out
44
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
Demonstration
AnalyzerAnalyzer
Frame
Frame
Dev
ice
AD
evice B
45
Future Developments
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
• Additional functions.
• User interface development.
Open Code flexibility opens the door for development
46
Closing Words
Thank You !
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
47
Additional informationCertain images adopted from PCI-SIG PCI Express™ Architectural Overview Presented at the 2002 PCI-SIG Developers Conference and Intel Developers Forum, Fall 2001.
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות
48
Project Development Process
High Speed Communication Fundamentals
Semester A
Final ConceptRequirements Doc
PCI ExpressArchitecture Concepts
market surveyCurrent Available Products
Existing Infrastructure
Constructing Analyzer CoreBuilding Blocks
Semester B
Analyzer CoreDevelopment report
Debugging & TestingEach block
Final Core Integration
High speed digital systems laboratoryהמעבדה למערכות ספרתיות מהירות