Structure of Computer Systems Course 10 Interconnection systems

Structure of Computer Structure of Computer SystemsSystems

Course 10 Course 10

Interconnection systemsInterconnection systems


Purpose:Purpose: connect different components of a computer system connect different components of a computer system

(CPU, memory, interfaces, peripheral devices) - (CPU, memory, interfaces, peripheral devices) - busesbuses

interconnect multiple computer systems – networksinterconnect multiple computer systems – networks

why?why? exchange of data and instruction codesexchange of data and instruction codes synchronization and coordination of actionssynchronization and coordination of actions events signalingevents signaling


EvolutionEvolution Inside of a computer system:Inside of a computer system:

• first 3 generation of computers – dedicated connections first 3 generation of computers – dedicated connections between computer modulesbetween computer modules

• microprocessors (4microprocessors (4thth generation) – system bus generation) – system bus• high performance processors – multiple buses with different high performance processors – multiple buses with different

speeds and destinationsspeeds and destinations• multi-core processors – network-on-chipmulti-core processors – network-on-chip

Between computer systems:Between computer systems:• first generations – dedicated point-to-point serial connectionsfirst generations – dedicated point-to-point serial connections• the 80’s – network communication and Internetthe 80’s – network communication and Internet• last years – very high speed interconnection systems for Grids last years – very high speed interconnection systems for Grids

and clusters (e.g. InfiniBand)and clusters (e.g. InfiniBand)


Design decisionsDesign decisions general purpose or dedicated connectionsgeneral purpose or dedicated connections serial or parallelserial or parallel synchronous or asynchronoussynchronous or asynchronous speedspeed dimension/distance (in circuit, on board, on dimension/distance (in circuit, on board, on

system, inter-system)system, inter-system) single or multi-mastersingle or multi-master


Interconnection system examplesInterconnection system examples General purpose, parallel, asynchronous, single and General purpose, parallel, asynchronous, single and

multi-master bus (classical system bus) multi-master bus (classical system bus) General purpose, parallel, synchronous busGeneral purpose, parallel, synchronous bus Transactional parallel busesTransactional parallel buses Specialized, parallel busesSpecialized, parallel buses Serial, point-to-point, and multipoint asynchronous Serial, point-to-point, and multipoint asynchronous

busesbuses Serial, synchronous buses Serial, synchronous buses Peripheral serial buses Peripheral serial buses

General purpose, parallel, General purpose, parallel, asynchronous bus (classical bus)asynchronous bus (classical bus)

purpose – one interconnection environment for all purpose – one interconnection environment for all the components of a computerthe components of a computer

features:features: parallel bus – transfer is made on multiple parallel lines parallel bus – transfer is made on multiple parallel lines

(signals)(signals) asynchronous – the bus in not controlled by clock asynchronous – the bus in not controlled by clock

signal; signals travel on the bus with a limited speed signal; signals travel on the bus with a limited speed causing delays causing delays

single master – only one module (the CPU) can initiate single master – only one module (the CPU) can initiate transfers on the bustransfers on the bus

multi-master – multiple modules can initiate transfers multi-master – multiple modules can initiate transfers on the buson the bus

General purpose, parallel, General purpose, parallel, asynchronous bus (classical bus)asynchronous bus (classical bus)

signals (sub-buses):signals (sub-buses): address signals Aaddress signals A00- A- Ann – used for specifying the – used for specifying the

location of the transfer (memory location or I/O location of the transfer (memory location or I/O register/portregister/port

• 22nn – the maximum addressing space allowed by the bus – the maximum addressing space allowed by the bus• selecting an optimal n: selecting an optimal n:

too small – limit the addressing spacetoo small – limit the addressing space too big – wasted space on the boardtoo big – wasted space on the board

data bus Ddata bus D00-D-Dmm – used for transferring data or – used for transferring data or

instruction codesinstruction codes• m - the maximum width of the data, which can be transferred m - the maximum width of the data, which can be transferred

in a bus cycle; in accordance with the CPU structure (e.g. 8, in a bus cycle; in accordance with the CPU structure (e.g. 8, 16, 32 or 64 bits)16, 32 or 64 bits)

General purpose, parallel, asynchronous General purpose, parallel, asynchronous bus (classical bus)bus (classical bus)

signals (cont.)signals (cont.) control and command signals – used to control the traffic on the control and command signals – used to control the traffic on the

bus (examples from ISAx86)bus (examples from ISAx86)• command signals – determine the type of the transfer cyclecommand signals – determine the type of the transfer cycle

MRDC\ (memory read command), MWTC\, IORC\, IOWC\, INTA\MRDC\ (memory read command), MWTC\, IORC\, IOWC\, INTA\• control signals – enable and disable data and address amplifiers, control signals – enable and disable data and address amplifiers,

validate transfers, reset the systemvalidate transfers, reset the system DEN (data enable), ALE (address latch enable), Ready, RST (reset)DEN (data enable), ALE (address latch enable), Ready, RST (reset)

• interrupt signals – used for signaling eventsinterrupt signals – used for signaling events IRQIRQ00-IRQ-IRQ77 (interrupt request) (interrupt request)

• bus arbitration signals – in multi-master buses, used for deciding bus arbitration signals – in multi-master buses, used for deciding who has the control of the buswho has the control of the bus

BRQ, BGT or HOLD, HOLDABRQ, BGT or HOLD, HOLDA• clock signals – used for synchronization or for generating other clock signals – used for synchronization or for generating other

useful frequenciesuseful frequencies CLK, BCLK (bus clock), PCLK (peripheral clock)CLK, BCLK (bus clock), PCLK (peripheral clock)

power signalspower signals• GND, Vcc, +12V, -12VGND, Vcc, +12V, -12V


a single bus configuration with: a single bus configuration with: CPU(s), memory CPU(s), memory modules, Input/Output interfaces and devicesmodules, Input/Output interfaces and devices

CPU Memory Memory

System bus

Address bus

Data bus

Control bus

I/O int. I/O int.

I/O dev. I/O dev.

General purpose, parallel, asynchronous General purpose, parallel, asynchronous bus (classical bus) - time diagramsbus (classical bus) - time diagrams

Memory Read Cycle

A0-An

MRDC

MWTC

D0-Dm

valid address

valid data

tcycle

taccess

Ready

General purpose, parallel, asynchronous General purpose, parallel, asynchronous bus (classical bus) - time diagramsbus (classical bus) - time diagrams

A0-An

Memory Write Cycle

MRDC

MWTC

D0-Dm

valid address

valid data

tcycle

taccess

Ready


Advantages:Advantages: simple operation (easy to understand and debug)simple operation (easy to understand and debug) simple design of bus modulessimple design of bus modules no dimensional limitations (asynchronous mode)no dimensional limitations (asynchronous mode) single communication environment for all the single communication environment for all the

components of a computer components of a computer

Drawbacks:Drawbacks: low speed – limited to the slowest modulelow speed – limited to the slowest module limited number of modules connected on the bus (10-limited number of modules connected on the bus (10-

16 – see fan-out of a TTL circuit)16 – see fan-out of a TTL circuit)


Examples of general purpose, parallel, asynchronous Examples of general purpose, parallel, asynchronous buses:buses: 8086 bus8086 bus ISA (ISA (Industry Standard ArchitectureIndustry Standard Architecture), EISA (extended ISA)), EISA (extended ISA) S-100 S-100

EISA connectors and Interface board

General purpose, parallel, General purpose, parallel, synchronous bussynchronous bus

Why ? (Purpose): increase the speed through a Why ? (Purpose): increase the speed through a better control of timingbetter control of timing

How ? (Principles)How ? (Principles) every signal on the bus is related (synchronized) with every signal on the bus is related (synchronized) with

the clock signalthe clock signal modules may anticipate next steps (does not have to modules may anticipate next steps (does not have to

wait until a signal arrives to the module, as in wait until a signal arrives to the module, as in asynchronous mode)asynchronous mode)

modules on the bus must have some intelligencemodules on the bus must have some intelligence Examples:Examples:

PCIPCI P6 (Pentium Pro) busP6 (Pentium Pro) bus


Block read cycle (PCI bus)Block read cycle (PCI bus) request for a block of data (first period)request for a block of data (first period) memory generates data from consecutive addressesmemory generates data from consecutive addresses


Advantages:Advantages: higher transfer speedhigher transfer speed small average access timesmall average access time promotes block transfers (good for cache line transfers)promotes block transfers (good for cache line transfers)

Disadvantages:Disadvantages: dimension of the bus is limited by the clock frequency dimension of the bus is limited by the clock frequency

• if the bus is too long, clock signal is not synchronized with itself if the bus is too long, clock signal is not synchronized with itself at the two ends of the bus (the speed of the signal is limited)at the two ends of the bus (the speed of the signal is limited)

more complex design of modules connected on the busmore complex design of modules connected on the bus harder debugging processharder debugging process

Transactional, parallel, Transactional, parallel, synchronous busessynchronous buses

Why ? (Purpose): increase the speed of the bus Why ? (Purpose): increase the speed of the bus How ?(Principles):How ?(Principles):

pipeline implementation of a transfer on the buspipeline implementation of a transfer on the bus use transaction (set of operations) instead of transfer use transaction (set of operations) instead of transfer

cyclescycles a transaction divided into stages that use different a transaction divided into stages that use different

signal groups and therefore can be executed in a signal groups and therefore can be executed in a pipeline mannerpipeline manner

Example: Example: P6 (Pentium pro) busP6 (Pentium pro) bus


Example: the P6 busExample: the P6 bus Phases:Phases:

• ArbitrationArbitration – decides which master has access on the bus– decides which master has access on the bus

• Transfer requestTransfer request – specifies the request (read or write, – specifies the request (read or write, start address, number of bytes)start address, number of bytes)

• SnoopingSnooping – detect and solve cache inconsistencies– detect and solve cache inconsistencies

• Error Error – – detect and solve transmission errors (ECC – error detect and solve transmission errors (ECC – error correction code on data and parity on address and command correction code on data and parity on address and command signals)signals)

• Response Response – specifies the type of the answer (now, – specifies the type of the answer (now, delayed, refused)delayed, refused)

• TransferTransfer – data transfer in accordance with the request– data transfer in accordance with the request


P6 bus (cont.)P6 bus (cont.) 1 2 3 4 5 6 7 8 9 1

0 11

12

13

14

15

16

BCLK

Arbitration

Request

Error

Snooping

Response

Transfer

Concurrent transactions on the P6 bus

Specialized, parallel busesSpecialized, parallel buses buses specialized for a group of peripheral devices (e.g. buses specialized for a group of peripheral devices (e.g.

HDD, DVD, etc.)HDD, DVD, etc.) Examples: IDE, SCASI (read “scazi”), ATAExamples: IDE, SCASI (read “scazi”), ATA SCASI details:SCASI details:

assures communication between an initiator (computer) and a assures communication between an initiator (computer) and a target (peripheral device)target (peripheral device)

protocol steps:protocol steps:• initiator sends a command (command descriptor block) to the targetinitiator sends a command (command descriptor block) to the target• target respond with a status code (success, error or busy)target respond with a status code (success, error or busy)• target returns a Check condition and the initiator respond with SCI target returns a Check condition and the initiator respond with SCI

Request sense commandRequest sense command there are about 60 command types grouped in 4 categories: there are about 60 command types grouped in 4 categories:

• non-data, read, write and bidirectionalnon-data, read, write and bidirectional SCASI and ATA have serial versions tooSCASI and ATA have serial versions too

Multi-master parallel busesMulti-master parallel buses

Issue:Issue: the bus is a shared resource; only one master can control the the bus is a shared resource; only one master can control the

bus at a given momentbus at a given moment how to establish who has the control of the bushow to establish who has the control of the bus

Solutions:Solutions: centralized controlcentralized control

• the central CPU is controlling the access on the bus – the central CPU is controlling the access on the bus – example: DMA transfer (HOLD, HOLDA handshaking mechanism)example: DMA transfer (HOLD, HOLDA handshaking mechanism)

• bus arbiter circuit bus arbiter circuit example: I8289 – bus arbiter (BRQ, BGT, CBRQ)example: I8289 – bus arbiter (BRQ, BGT, CBRQ)

distributed controldistributed control• every master has an arbitration component:every master has an arbitration component:

serial linkserial link token basestoken bases

Serial busesSerial buses Serial bus v.s. parallel busSerial bus v.s. parallel bus

less signals (lines)less signals (lines) longer transmission distanceslonger transmission distances cheaper implementation (e.g. less wires, less space on cheaper implementation (e.g. less wires, less space on

the PCB - printed circuit board, less pins on the circuits)the PCB - printed circuit board, less pins on the circuits) speed:speed:

• old view – lower speed than parallel connectionold view – lower speed than parallel connection• new view – higher speed than parallel connectionnew view – higher speed than parallel connection• explanation - its easier to increase more than 10 times the explanation - its easier to increase more than 10 times the

transmission frequency on serial bus than on a parallel one (see transmission frequency on serial bus than on a parallel one (see electro-magnetic interferences in case of long parallel lines)electro-magnetic interferences in case of long parallel lines)

• consequence – most of the parallel buses are replaced with consequence – most of the parallel buses are replaced with serial ones:serial ones:

serial ATA and SCASIserial ATA and SCASI network-on-chipnetwork-on-chip serial system buses – e.g. I2C for microcontrollersserial system buses – e.g. I2C for microcontrollers

Serial busesSerial buses

Design decisions:Design decisions: synchronous or asynchronoussynchronous or asynchronous point-to-point or multipointpoint-to-point or multipoint character-based or message-basecharacter-based or message-base information coding: voltage levels, differential information coding: voltage levels, differential

voltages, light impulses, radio wavesvoltages, light impulses, radio waves flow control: hardware, software, protocol-flow control: hardware, software, protocol-

basedbased error detection and correctionerror detection and correction

Serial buses – Serial buses – Synchronous transmissionSynchronous transmission

synchronous – an extra clock signal synchronous – an extra clock signal controls the transmissioncontrols the transmission

Clk

Data

0 1 1 0 1 0 0 0

Shift reg.Data signal

Clock signal

Shift reg.

Sender ReceiverGND


FeaturesFeatures easy to implement, no other synchronization easy to implement, no other synchronization

mechanisms are neededmechanisms are needed requires an extra signal (clock), inefficient use requires an extra signal (clock), inefficient use

of wiresof wires hard to synchronize sender and receiver on hard to synchronize sender and receiver on

long distances long distances


Example: I2C protocol (read: eye to see) Example: I2C protocol (read: eye to see) multi-point, serial, synchronous busmulti-point, serial, synchronous bus used in microcontroller systems to connect external components: used in microcontroller systems to connect external components:

memory circuits, analog-digital convertermemory circuits, analog-digital converter master-slave protocol (1 master controls the traffic on the bus)master-slave protocol (1 master controls the traffic on the bus) uses two lines: SCL- clock and SDA - datauses two lines: SCL- clock and SDA - data

address

Serial buses Serial buses Asynchronous transmissionAsynchronous transmission

FeaturesFeatures no clock signalno clock signal synchronization made through the specific structure of synchronization made through the specific structure of

the transmitted data the transmitted data the sender and the transmitter must use the same the sender and the transmitter must use the same

protocol that specifies:protocol that specifies:• transmission frequency transmission frequency • number of bits/character or bytes/messagenumber of bits/character or bytes/message• coding of logical 0 and1coding of logical 0 and1• data-flow control mechanismsdata-flow control mechanisms• error detection methoderror detection method

Serial buses Serial buses Asynchronous transmissionAsynchronous transmission

best known protocol (standard): RS232 or V24best known protocol (standard): RS232 or V24 Specifications of RS232 protocol:Specifications of RS232 protocol:

point-to-point bidirectional transmission on characterspoint-to-point bidirectional transmission on characters standard frequencies: standard frequencies: 300,600, 1200 ...9600 ...Bauds300,600, 1200 ...9600 ...Bauds bits/character: 6,7, 8 bits bits/character: 6,7, 8 bits 1 START bit = 0 and 1or 2 STOP bits = 11 START bit = 0 and 1or 2 STOP bits = 1 error detection – optional parity bit, even or odderror detection – optional parity bit, even or odd flow-control protocols:flow-control protocols:

• software (XON/XOFF) – with ASCII codes for starting (XON) and software (XON/XOFF) – with ASCII codes for starting (XON) and stopping (XOFF) the transmissionstopping (XOFF) the transmission

• hardware – with 2 pairs of signals: RTS-CTS or DSR-DTRhardware – with 2 pairs of signals: RTS-CTS or DSR-DTR

Serial buses – Serial buses – Asynchronous transmissionAsynchronous transmission

Specifications of RS232 protocol (cont.):Specifications of RS232 protocol (cont.): signals:signals:

• RXD – receive dataRXD – receive data

• TXD – transmit dataTXD – transmit data

• GND – ground (voltage reference)GND – ground (voltage reference)

• RTS – request to sendRTS – request to send

• CTS – clear to sendCTS – clear to send

• DSR - data set readyDSR - data set ready

• DTR – data terminal readyDTR – data terminal ready

max. transmission distance: 100mmax. transmission distance: 100m data format: data format: Start (1 bit = 1), data (6-8 bits), Parity (1 bit), Stop (1-Start (1 bit = 1), data (6-8 bits), Parity (1 bit), Stop (1-

2 bits=1)2 bits=1)

RXDTXDGNDRTSCTSDSRDTR

RXDTXDGNDRTSCTSDSRDTR

Sender Receiver

Start data bits (6-8 bits) Parity Stop (1-2 bits)

Serial buses – Serial buses – Asynchronous transmissionAsynchronous transmission

Specifications of the RS485 protocol:Specifications of the RS485 protocol: multi-point, serial, asynchronous transmission on multi-point, serial, asynchronous transmission on

characterscharacters• transceivers (receiver-transmitter circuit) with three-state transceivers (receiver-transmitter circuit) with three-state

capabilitycapability transmission on two twisted wires (A and B) transmission on two twisted wires (A and B) bit coding: differential voltagebit coding: differential voltage

Multipoint interconnectionsMultipoint interconnections

RingRing busbus treetree matrixmatrix hyper-cubehyper-cube switch fabricsswitch fabrics


Crossbar switchCrossbar switch multiple connections between multiple componentsmultiple connections between multiple components

• multi-bus access of CPUs to memory modulesmulti-bus access of CPUs to memory modules


Issues:Issues: reduce the number of connectionsreduce the number of connections reduce the number of interfacesreduce the number of interfaces reduce communication delays (data latency)reduce communication delays (data latency) reduce the number of hops (nodes) involved reduce the number of hops (nodes) involved

in a transferin a transfer increase the bandwidthincrease the bandwidth

Multipoint interconnectionsMultipoint interconnections Implementations:Implementations:

latest Intel processorslatest Intel processors (Sand Bridge) (Sand Bridge) • internal ring between cache memoriesinternal ring between cache memories• QPI – QuickPath InterconnectQPI – QuickPath Interconnect – connection between CPUs – connection between CPUs

InfiniBandInfiniBand • a switched fabric communications link used in high-a switched fabric communications link used in high-

performance computing and enterprise data centersperformance computing and enterprise data centers• features:features:

high throughput, high throughput, low latency, low latency, quality of service and failover, quality of service and failover, scalable. scalable.

• The InfiniBand architecture specification defines a The InfiniBand architecture specification defines a connection between processor nodes and high performance connection between processor nodes and high performance I/O nodes such as storage devices. Infiniband host bus I/O nodes such as storage devices. Infiniband host bus adapters and network switches are manufactured by adapters and network switches are manufactured by Mellanox and Intel Mellanox and Intel

Multipoint interconnectionsMultipoint interconnections Wishbone Bus:Wishbone Bus:

an open source hardware computer an open source hardware computer bus intended to let the parts of an bus intended to let the parts of an integrated circuit communicate with integrated circuit communicate with each other. each other.

the aim is to allow the connection of the aim is to allow the connection of differing cores to each other inside differing cores to each other inside of a chip. of a chip.

is used by many designs in the is used by many designs in the OpenCores project.OpenCores project.

Documents

Structure of Computer Systems Course 10 Interconnection systems