CIS 371 Computer Organization and Designmilom/cis371-Spring13/lectures/10... · CIS 371: Comp. Org. | Prof. Milo Martin | Scheduling 1 CIS 371 Computer Organization and Design Unit

CIS 371: Comp. Org. | Prof. Milo Martin | Scheduling 1

CIS 371 Computer Organization and Design

Unit 10: Static & Dynamic Scheduling

Slides developed by Milo Martin & Amir Roth at the University of Pennsylvania with sources that included University of Wisconsin slides

by Mark Hill, Guri Sohi, Jim Smith, and David Wood.


This Unit: Static & Dynamic Scheduling

•  Code scheduling •  To reduce pipeline stalls •  To increase ILP (insn level parallelism)

•  Static scheduling by the compiler •  Approach & limitations

•  Dynamic scheduling in hardware •  Register renaming •  Instruction selection •  Handling memory operations

CPU Mem I/O

System software

App App App

CIS 371 (Martin): Scheduling 3

Readings

•  P&H •  Chapter 4.10 – 4.11

Code Scheduling & Limitations


Code Scheduling

•  Scheduling: act of finding independent instructions •  “Static” done at compile time by the compiler (software) •  “Dynamic” done at runtime by the processor (hardware)

•  Why schedule code? •  Scalar pipelines: fill in load-to-use delay slots to improve CPI •  Superscalar: place independent instructions together

•  As above, load-to-use delay slots •  Allow multiple-issue decode logic to let them execute at the

same time



Compiler Scheduling

•  Compiler can schedule (move) instructions to reduce stalls •  Basic pipeline scheduling: eliminate back-to-back load-use pairs •  Example code sequence: a = b + c; d = f – e;

• sp stack pointer, sp+0 is “a”, sp+4 is “b”, etc…

Before

ld [sp+4]➜r2 ld [sp+8]➜r3 add r2,r3➜r1 //stall st r1➜[sp+0] ld [sp+16]➜r5 ld [sp+20]➜r6 sub r6,r5➜r4 //stall st r4➜[sp+12]

After

ld [sp+4]➜r2 ld [sp+8]➜r3 ld [sp+16]➜r5 add r2,r3➜r1 //no stall ld [sp+20]➜r6 st r1➜[sp+0] sub r6,r5➜r4 //no stall st r4➜[sp+12]


Compiler Scheduling Requires

•  Large scheduling scope •  Independent instruction to put between load-use pairs +  Original example: large scope, two independent computations –  This example: small scope, one computation

•  Compiler can create larger scheduling scopes •  For example: loop unrolling & function inlining

Before

ld [sp+4]➜r2 ld [sp+8]➜r3 add r2,r3➜r1 //stall st r1➜[sp+0]

After (same!)

ld [sp+4]➜r2 ld [sp+8]➜r3 add r2,r3➜r1 //stall st r1➜[sp+0]

CIS 371: Comp. Org. | Prof. Milo Martin | Scheduling

Scheduling Scope Limited by Branches

r1 and r2 are inputs loop: jz r1, not_found ld [r1+0]➜r3 sub r2,r3➜r4 jz r4, found ld [r1+4]➜r1 jmp loop

Legal to move load up past branch? No: if r1 is null, will cause a fault

Aside: what does this code do? Searches a linked list for an element

8

bool search(list* lst, int v)!{! while (lst != NULL) {! if (lst->value == val) {! return true;! }! lst = lst->next;! }! return false;!}!


Compiler Scheduling Requires

•  Enough registers •  To hold additional “live” values •  Example code contains 7 different values (including sp) •  Before: max 3 values live at any time → 3 registers enough •  After: max 4 values live → 3 registers not enough

Original

ld [sp+4]➜r2 ld [sp+8]➜r1 add r1,r2➜r1 //stall st r1➜[sp+0] ld [sp+16]➜r2 ld [sp+20]➜r1 sub r2,r1➜r1 //stall st r1➜[sp+12]

Wrong!

ld [sp+4]➜r2 ld [sp+8]➜r1 ld [sp+16]➜r2 add r1,r2➜r1 // wrong r2 ld [sp+20]➜r1 st r1➜[sp+0] // wrong r1 sub r2,r1➜r1 st r1➜[sp+12]


Compiler Scheduling Requires •  Alias analysis

•  Ability to tell whether load/store reference same memory locations •  Effectively, whether load/store can be rearranged

•  Previous example: easy, loads/stores use same base register (sp) •  New example: can compiler tell that r8 != r9? •  Must be conservative

Before

ld [r9+4]➜r2 ld [r9+8]➜r3 add r3,r2➜r1 //stall st r1➜[r9+0] ld [r8+0]➜r5 ld [r8+4]➜r6 sub r5,r6➜r4 //stall st r4➜[r8+8]

Wrong(?)

ld [r9+4]➜r2 ld [r9+8]➜r3 ld [r8+0]➜r5 //does r8==r9? add r3,r2➜r1 ld [r8+4]➜r6 //does r8+4==r9? st r1➜[r9+0] sub r5,r6➜r4 st r4➜[r8+8]

Compiler Scheduling Limitations

•  Scheduling scope •  Example: can’t generally move memory operations past branches

•  Limited number of registers (set by ISA)

•  Inexact “memory aliasing” information •  Often prevents reordering of loads above stores by compiler

•  Caches misses (or any runtime event) confound scheduling •  How can the compiler know which loads will miss vs hit? •  Can impact the compiler’s scheduling decisions


Dynamic (Hardware) Scheduling



Can Hardware Overcome These Limits?

•  Dynamically-scheduled processors •  Also called “out-of-order” processors •  Hardware re-schedules insns… •  …within a sliding window of VonNeumann insns •  As with pipelining and superscalar, ISA unchanged

•  Same hardware/software interface, appearance of in-order

•  Examples: •  Pentium Pro/II/III (3-wide), Core 2 (4-wide),

Alpha 21264 (4-wide), MIPS R10000 (4-wide), Power5 (5-wide)

Motivating Example

•  In-order pipeline, two-cycle load-use penalty •  2-wide

•  Why not the following:


0 1 2 3 4 5 6 7 8 9 10 11 12

Ld [r1] ➜ r2 F D X M1 M2 W add r2 + r3 ➜ r4 F D d* d* d* X M1 M2 W xor r4 ^ r5 ➜ r6 F D d* d* d* X M1 M2 W ld [r7] ➜ r4 F D p* p* p* X M1 M2 W

0 1 2 3 4 5 6 7 8 9 10 11 12

Ld [r1] ➜ r2 F D X M1 M2 W add r2 + r3 ➜ r4 F D d* d* d* X M1 M2 W xor r4 ^ r5 ➜ r6 F D d* d* d* X M1 M2 W ld [r7] ➜ r4 F D X M1 M2 W

Motivating Example (“Renamed”)

•  In-order pipeline, two-cycle load-use penalty •  2-wide

•  Why not the following:


0 1 2 3 4 5 6 7 8 9 10 11 12

Ld [p1] ➜ p2 F D X M1 M2 W add p2 + p3 ➜ p4 F D d* d* d* X M1 M2 W xor p4 ^ p5 ➜ p6 F D d* d* d* X M1 M2 W ld [p7] ➜ p8 F D p* p* p* X M1 M2 W

0 1 2 3 4 5 6 7 8 9 10 11 12

Ld [p1] ➜ p2 F D X M1 M2 W add p2 + p3 ➜ p4 F D d* d* d* X M1 M2 W xor p4 ^ p5 ➜ p6 F D d* d* d* X M1 M2 W ld [p7] ➜ p8 F D X M1 M2 W

In-Order Pipeline


Fetc

h

Dec

ode

/ Rea

d-re

g

Exe

cute

Mem

ory

Writ

ebac

k

•  What stages can (or should) be done out-of-order?


Out-of-Order Pipeline Fe

tch

Dec

ode

Ren

ame

Dis

patc

h

Com

mit

Buffer of instructions

Issu

e

Reg

-rea

d

Exe

cute

Writ

ebac

k

17 M

emor

y

In-order front end Out-of-order execution

In-order commit

Out-of-Order to the Rescue

•  “Dynamic scheduling” done by the hardware •  Still 2-wide superscalar, but now out-of-order, too

•  Allows instructions to issues when dependences are ready

•  Longer pipeline •  In-order front end: Fetch, “Dispatch” •  Out-of-order execution core:

•  “Issue”, “RegisterRead”, Execute, Memory, Writeback •  In-order retirement: “Commit”


0 1 2 3 4 5 6 7 8 9 10 11 12

Ld [p1] ➜ p2 F Di I RR X M1 M2 W C add p2 + p3 ➜ p4 F Di I RR X W C xor p4 ^ p5 ➜ p6 F Di I RR X W C ld [p7] ➜ p8 F Di I RR X M1 M2 W C

Out-of-Order Execution

•  Also call “Dynamic scheduling” •  Done by the hardware on-the-fly during execution

•  Looks at a “window” of instructions waiting to execute •  Each cycle, picks the next ready instruction(s)

•  Two steps to enable out-of-order execution: Step #1: Register renaming – to avoid “false” dependencies Step #2: Dynamically schedule – to enforce “true” dependencies

•  Key to understanding out-of-order execution: •  Data dependencies


Types of Dependences

•  RAW (Read After Write) = “true dependence” (true) mul r0 * r1 ➜ r2 … add r2 + r3 ➜ r4

•  WAW (Write After Write) = “output dependence” (false) mul r0 * r1➜ r2 … add r1 + r3 ➜ r2

•  WAR (Write After Read) = “anti-dependence” (false) mul r0 * r1 ➜ r2 … add r3 + r4 ➜ r1

•  WAW & WAR are “false”, Can be totally eliminated by “renaming”



Step #1: Register Renaming •  To eliminate register conflicts/hazards •  “Architected” vs “Physical” registers – level of indirection

•  Names: r1,r2,r3 •  Locations: p1,p2,p3,p4,p5,p6,p7 •  Original mapping: r1→p1, r2→p2, r3→p3, p4–p7 are “available”

•  Renaming – conceptually write each register once + Removes false dependences + Leaves true dependences intact!

•  When to reuse a physical register? After overwriting insn done

MapTable FreeList Original insns Renamed insns r1 r2 r3 p1 p2 p3 p4,p5,p6,p7 add r2,r3➜r1 add p2,p3➜p4 p4 p2 p3 p5,p6,p7 sub r2,r1➜r3 sub p2,p4➜p5 p4 p2 p5 p6,p7 mul r2,r3➜r3 mul p2,p5➜p6 p4 p2 p6 p7 div r1,4➜r1 div p4,4➜p7

Register Renaming Algorithm

•  Two key data structures: •  maptable[architectural_reg] physical_reg •  Free list: allocate (new) & free registers (implemented as a queue)

•  Algorithm: at “decode” stage for each instruction: insn.phys_input1 = maptable[insn.arch_input1]!insn.phys_input2 = maptable[insn.arch_input2]!insn.old_phys_output = maptable[insn.arch_output]!new_reg = new_phys_reg()!maptable[insn.arch_output] = new_reg!insn.phys_output = new_reg

•  At “commit” •  Once all prior instructions have committed, free register free_phys_reg(insn.old_phys_output) !




tch

Dec

ode

Ren

ame

Dis

patc

h

Com

mit


Issu

e

Reg

-rea

d

Exe

cute

Writ

ebac

k

23 M

emor

y


In-order commit Have unique register names Now put into out-of-order execution structures


regfile

D$ I$ B P

insn buffer

S D

add p2,p3➜p4 sub p2,p4➜p5 mul p2,p5➜p6 div p4,4➜p7

Ready Table P2 P3 P4 P5 P6 P7 Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes

div p4,4➜p7 mul p2,p5➜p6 sub p2,p4➜p5 add p2,p3➜p4

and

Step #2: Dynamic Scheduling

•  Instructions fetch/decoded/renamed into Instruction Buffer •  Also called “instruction window” or “instruction scheduler”

•  Instructions (conceptually) check ready bits every cycle •  Execute earliest “ready” instruction, set output as “ready”

Tim

e

Dynamic Scheduling/Issue Algorithm

•  Data structures: •  Ready table[phys_reg] yes/no (part of “issue queue”)

•  Algorithm at “schedule” stage (prior to read registers): foreach instruction:!

if table[insn.phys_input1] == ready && table[insn.phys_input2] == ready then! insn is “ready”!

select the earliest “ready” instruction!table[insn.phys_output] = ready!

•  Multiple-cycle instructions? (such as loads) •  For an insn with latency of N, set “ready” bit N-1 cycles in future!


Register Renaming


Register Renaming Algorithm (Simplified)


•  Algorithm: at “decode” stage for each instruction: insn.phys_input1 = maptable[insn.arch_input1]!insn.phys_input2 = maptable[insn.arch_input2]!

new_reg = new_phys_reg()!maptable[insn.arch_output] = new_reg!insn.phys_output = new_reg



Renaming example

xor r1 ^ r2 ➜ r3 add r3 + r4 ➜ r4 sub r5 - r2 ➜ r3 addi r3 + 1 ➜ r1

r1 p1

r2 p2

r3 p3

r4 p4

r5 p5

Map table Free-list

p6

p7

p8

p9

p10

28


Renaming example

r1 p1

r2 p2

r3 p3

r4 p4

r5 p5

Map table Free-list

p6

p7

p8

p9

p10

xor p1 ^ p2 ➜ xor r1 ^ r2 ➜ r3 add r3 + r4 ➜ r4 sub r5 - r2 ➜ r3 addi r3 + 1 ➜ r1

29


Renaming example

r1 p1

r2 p2

r3 p3

r4 p4

r5 p5

Map table Free-list

p6

p7

p8

p9

p10

xor p1 ^ p2 ➜ p6 xor r1 ^ r2 ➜ r3 add r3 + r4 ➜ r4 sub r5 - r2 ➜ r3 addi r3 + 1 ➜ r1

30


Renaming example

r1 p1

r2 p2

r3 p6

r4 p4

r5 p5

Map table Free-list

p7

p8

p9

p10


31


Renaming example

r1 p1

r2 p2

r3 p6

r4 p4

r5 p5

Map table Free-list

p7

p8

p9

p10

xor p1 ^ p2 ➜ p6 add p6 + p4 ➜


32


Renaming example

r1 p1

r2 p2

r3 p6

r4 p4

r5 p5

Map table Free-list

p7

p8

p9

p10

xor p1 ^ p2 ➜ p6 add p6 + p4 ➜ p7


33


Renaming example

r1 p1

r2 p2

r3 p6

r4 p7

r5 p5

Map table Free-list

p8

p9

p10

xor p1 ^ p2 ➜ p6 add p6 + p4 ➜ p7


34


Renaming example

r1 p1

r2 p2

r3 p6

r4 p7

r5 p5

Map table Free-list

p8

p9

p10

xor p1 ^ p2 ➜ p6 add p6 + p4 ➜ p7 sub p5 - p2 ➜


35


Renaming example

r1 p1

r2 p2

r3 p6

r4 p7

r5 p5

Map table Free-list

p8

p9

p10

xor p1 ^ p2 ➜ p6 add p6 + p4 ➜ p7 sub p5 - p2 ➜ p8


36


Renaming example

r1 p1

r2 p2

r3 p8

r4 p7

r5 p5

Map table Free-list

p9

p10



37


Renaming example

r1 p1

r2 p2

r3 p8

r4 p7

r5 p5

Map table Free-list

p9

p10

xor p1 ^ p2 ➜ p6 add p6 + p4 ➜ p7 sub p5 - p2 ➜ p8 addi p8 + 1 ➜


38


Renaming example

r1 p1

r2 p2

r3 p8

r4 p7

r5 p5

Map table Free-list

p9

p10

xor p1 ^ p2 ➜ p6 add p6 + p4 ➜ p7 sub p5 - p2 ➜ p8 addi p8 + 1 ➜ p9


39


Renaming example

r1 p9

r2 p2

r3 p8

r4 p7

r5 p5

Map table Free-list

p10



40



tch

Dec

ode

Ren

ame

Dis

patc

h

Com

mit


Issu

e

Reg

-rea

d

Exe

cute

Writ

ebac

k

41 M

emor

y


In-order commit Have unique register names Now put into out-of-order execution structures

Dynamic Scheduling Mechanisms


Dispatch

•  Renamed instructions into out-of-order structures •  Re-order buffer (ROB)

•  All instruction until commit

•  Issue Queue •  Central piece of scheduling logic •  Holds un-executed instructions •  Tracks ready inputs

•  Physical register names + ready bit •  “AND” the bits to tell if ready


Insn Inp1 R Inp2 R Dst

Ready?

#

Dispatch Steps

•  Allocate Issue Queue (IQ) slot •  Full? Stall

•  Read ready bits of inputs •  Table 1-bit per physical reg

•  Clear ready bit of output in table •  Instruction has not produced value yet

•  Write instruction into Issue Queue (IQ) slot


Dispatch Example



Insn Inp1 R Inp2 R Dst #

Issue Queue

p1 y

p2 y

p3 y

p4 y

p5 y

p6 y

p7 y

p8 y

p9 y

Ready bits

Dispatch Example



xor p1 y p2 y p6 0

Issue Queue

p1 y

p2 y

p3 y

p4 y

p5 y

p6 n

p7 y

p8 y

p9 y

Ready bits xor p1 ^ p2 ➜ p6 add p6 + p4 ➜ p7 sub p5 - p2 ➜ p8 addi p8 + 1 ➜ p9

Dispatch Example



xor p1 y p2 y p6 0

add p6 n p4 y p7 1

Issue Queue

p1 y

p2 y

p3 y

p4 y

p5 y

p6 n

p7 n

p8 y

p9 y


Dispatch Example



xor p1 y p2 y p6 0

add p6 n p4 y p7 1

sub p5 y p2 y p8 2

Issue Queue

p1 y

p2 y

p3 y

p4 y

p5 y

p6 n

p7 n

p8 n

p9 y


Dispatch Example



xor p1 y p2 y p6 0

add p6 n p4 y p7 1

sub p5 y p2 y p8 2

addi p8 n --- y p9 3

Issue Queue

p1 y

p2 y

p3 y

p4 y

p5 y

p6 n

p7 n

p8 n

p9 n


Out-of-order pipeline

•  Execution (out-of-order) stages •  Select ready instructions

•  Send for execution

•  Wakeup dependents


Issue

Reg-read

Execute

Writeback

Dynamic Scheduling/Issue Algorithm

•  Data structures: •  Ready table[phys_reg] yes/no (part of issue queue)

•  Algorithm at “schedule” stage (prior to read registers): foreach instruction:!

if table[insn.phys_input1] == ready && table[insn.phys_input2] == ready then! insn is “ready”!

select the earliest “ready” instruction!table[insn.phys_output] = ready !


Issue = Select + Wakeup

•  Select earliest of “ready” instructions   “xor” is the earliest ready instruction below   “xor” and “sub” are the two earliest ready instructions below •  Note: may have resource constraints: i.e. load/store/floating point



xor p1 y p2 y p6 0

add p6 n p4 y p7 1

sub p5 y p2 y p8 2

addi p8 n --- y p9 3

Ready!

Ready!

Issue = Select + Wakeup •  Wakeup dependent instructions

•  Search for destination (Dst) in inputs & set “ready” bit •  Implemented with a special memory array circuit

called a Content Addressable Memory (CAM) •  Also update ready-bit table for future instructions

•  For multi-cycle operations (loads, floating point) •  Wakeup deferred a few cycles •  Include checks to avoid structural hazards



xor p1 y p2 y p6 0

add p6 y p4 y p7 1

sub p5 y p2 y p8 2

addi p8 y --- y p9 3

p1 y

p2 y

p3 y

p4 y

p5 y

p6 y

p7 n

p8 y

p9 n

Ready bits

Issue •  Select/Wakeup one cycle •  Dependent instructions execute on back-to-back cycles

•  Next cycle: add/addi are ready:

•  Issued instructions are removed from issue queue •  Free up space for subsequent instructions



add p6 y p4 y p7 1

addi p8 y --- y p9 3

OOO execution (2-wide)


p1 7

p2 3

p3 4

p4 9

p5 6

p6 0

p7 0

p8 0

p9 0

xor RDY add sub RDY addi



p1 7

p2 3

p3 4

p4 9

p5 6

p6 0

p7 0

p8 0

p9 0

add RDY

addi RDY

xor p

1^ p

2 ➜

p6

sub

p5 -

p2 ➜

p8



p1 7

p2 3

p3 4

p4 9

p5 6

p6 0

p7 0

p8 0

p9 0

add

p6 +

p4 ➜

p7

addi

p8

+1 ➜

p9

xor 7

^ 3 ➜

p6

sub

6 - 3

➜ p

8



p1 7

p2 3

p3 4

p4 9

p5 6

p6 0

p7 0

p8 0

p9 0

add

_ +

9 ➜

p7

addi

_ +

1 ➜

p9

4 ➜

p6

3 ➜

p8



p1 7

p2 3

p3 4

p4 9

p5 6

p6 4

p7 0

p8 3

p9 0

13 ➜

p7

4 ➜

p9



p1 7

p2 3

p3 4

p4 9

p5 6

p6 4

p7 13

p8 3

p9 4



p1 7

p2 3

p3 4

p4 9

p5 6

p6 4

p7 13

p8 3

p9 4

Note similarity to in-order

When Does Register Read Occur?

•  Current approach: after select, right before execute •  Not during in-order part of pipeline, in out-of-order part •  Read physical register (renamed) •  Or get value via bypassing (based on physical register name) •  This is Pentium 4, MIPS R10k, Alpha 21264, IBM Power4,

Intel’s “Sandy Bridge” (2011) •  Physical register file may be large

•  Multi-cycle read

•  Older approach: •  Read as part of “issue” stage, keep values in Issue Queue

•  At commit, write them back to “architectural register file” •  Pentium Pro, Core 2, Core i7 •  Simpler, but may be less energy efficient (more data movement)


Renaming Revisited


Re-order Buffer (ROB) •  ROB entry holds all info for recover/commit

•  All instructions & in order •  Architectural register names, physical register names, insn type •  Not removed until very last thing (“commit”)

•  Operation •  Dispatch: insert at tail (if full, stall) •  Commit: remove from head (if not yet done, stall)

•  Purpose: tracking for in-order commit •  Maintain appearance of in-order execution •  Done to support:

•  Misprediction recovery •  Freeing of physical registers


Renaming revisited

•  Track (or “log”) the “overwritten register” in ROB •  Freed this register at commit •  Also used to restore the map table on “recovery”

•  Branch mis-prediction recovery


Register Renaming Algorithm (Full)


•  Algorithm: at “decode” stage for each instruction: insn.phys_input1 = maptable[insn.arch_input1]!insn.phys_input2 = maptable[insn.arch_input2]!insn.old_phys_output = maptable[insn.arch_output]!new_reg = new_phys_reg()!maptable[insn.arch_output] = new_reg!insn.phys_output = new_reg

•  At “commit” •  Once all prior instructions have committed, free register free_phys_reg(insn. old_phys_output) !


Recovery

•  Completely remove wrong path instructions •  Flush from IQ •  Remove from ROB •  Restore map table to before misprediction •  Free destination registers

•  How to restore map table? •  Option #1: log-based reverse renaming to recover each instruction

•  Tracks the old mapping to allow it to be reversed •  Done sequentially for each instruction (slow) •  See next slides

•  Option #2: checkpoint-based recovery •  Checkpoint state of maptable and free list each cycle •  Faster recovery, but requires more state

•  Option #3: hybrid (checkpoint for branches, unwind for others) CIS 371: Comp. Org. | Prof. Milo Martin | Scheduling 67

Renaming example



r1 p1

r2 p2

r3 p3

r4 p4

r5 p5

Map table Free-list

p6

p7

p8

p9

p10

Renaming example


r1 p1

r2 p2

r3 p3

r4 p4

r5 p5

Map table Free-list

p6

p7

p8

p9

p10

xor p1 ^ p2 ➜ xor r1 ^ r2 ➜ r3 add r3 + r4 ➜ r4 sub r5 - r2 ➜ r3 addi r3 + 1 ➜ r1

[ p3 ]

Renaming example


r1 p1

r2 p2

r3 p6

r4 p4

r5 p5

Map table Free-list

p7

p8

p9

p10


[ p3 ]

Renaming example


r1 p1

r2 p2

r3 p6

r4 p4

r5 p5

Map table Free-list

p7

p8

p9

p10

xor p1 ^ p2 ➜ p6 add p6 + p4 ➜


[ p3 ] [ p4 ]

Renaming example


r1 p1

r2 p2

r3 p6

r4 p7

r5 p5

Map table Free-list

p8

p9

p10

xor p1 ^ p2 ➜ p6 add p6 + p4 ➜ p7


[ p3 ] [ p4 ]

Renaming example


r1 p1

r2 p2

r3 p6

r4 p7

r5 p5

Map table Free-list

p8

p9

p10

xor p1 ^ p2 ➜ p6 add p6 + p4 ➜ p7 sub p5 - p2 ➜


[ p3 ] [ p4 ] [ p6 ]

Renaming example


r1 p1

r2 p2

r3 p8

r4 p7

r5 p5

Map table Free-list

p9

p10



[ p3 ] [ p4 ] [ p6 ]

Renaming example


r1 p1

r2 p2

r3 p8

r4 p7

r5 p5

Map table Free-list

p9

p10

xor p1 ^ p2 ➜ p6 add p6 + p4 ➜ p7 sub p5 - p2 ➜ p8 addi p8 + 1 ➜


[ p3 ] [ p4 ] [ p6 ] [ p1 ]

Renaming example


r1 p9

r2 p2

r3 p8

r4 p7

r5 p5

Map table Free-list

p10



[ p3 ] [ p4 ] [ p6 ] [ p1 ]

Recovery Example


r1 p9

r2 p2

r3 p8

r4 p7

r5 p5

Map table Free-list

p10

bnz p1, loop xor p1 ^ p2 ➜ p6 add p6 + p4 ➜ p7 sub p5 - p2 ➜ p8 addi p8 + 1 ➜ p9

bnz r1 loop xor r1 ^ r2 ➜ r3 add r3 + r4 ➜ r4 sub r5 - r2 ➜ r3 addi r3 + 1 ➜ r1

[ ] [ p3 ] [ p4 ] [ p6 ] [ p1 ]

Now, let’s use this info. to recover from a branch misprediction

Recovery Example


r1 p1

r2 p2

r3 p8

r4 p7

r5 p5

Map table Free-list

p10

bnz p1, loop xor p1 ^ p2 ➜ p6 add p6 + p4 ➜ p7 sub p5 - p2 ➜ p8 addi p8 + 1 ➜ p9

bnz r1 loop xor r1 ^ r2 ➜ r3 add r3 + r4 ➜ r4 sub r5 - r2 ➜ r3 addi r3 + 1 ➜ r1

[ ] [ p3 ] [ p4 ] [ p6 ] [ p1 ]

p9

Recovery Example


r1 p1

r2 p2

r3 p6

r4 p7

r5 p5

Map table Free-list

p10

bnz p1, loop xor p1 ^ p2 ➜ p6 add p6 + p4 ➜ p7 sub p5 - p2 ➜ p8

bnz r1 loop xor r1 ^ r2 ➜ r3 add r3 + r4 ➜ r4 sub r5 - r2 ➜ r3

[ ] [ p3 ] [ p4 ] [ p6 ]

p9

p8

Recovery Example


r1 p1

r2 p2

r3 p6

r4 p4

r5 p5

Map table Free-list

p10

bnz p1, loop xor p1 ^ p2 ➜ p6 add p6 + p4 ➜ p7

bnz r1 loop xor r1 ^ r2 ➜ r3 add r3 + r4 ➜ r4

[ ] [ p3 ] [ p4 ]

p9

p8

p7

Recovery Example


r1 p1

r2 p2

r3 p3

r4 p4

r5 p5

Map table Free-list

p10

bnz p1, loop xor p1 ^ p2 ➜ p6

bnz r1 loop xor r1 ^ r2 ➜ r3

[ ] [ p3 ]

p9

p8

p7

p6

Recovery Example


r1 p1

r2 p2

r3 p3

r4 p4

r5 p5

Map table Free-list

p10

bnz p1, loop bnz r1 loop [ ]

p9

p8

p7

p6

Commit




[ p3 ] [ p4 ] [ p6 ] [ p1 ]

•  Commit: instruction becomes architected state

•  In-order, only when instructions are finished

•  Free overwritten register (why?)

Freeing over-written register




[ p3 ] [ p4 ] [ p6 ] [ p1 ]

•  P3 was r3 before xor

•  P6 is r3 after xor

•  Anything before (in program order) xor should read p3

•  Anything after (in program order) xor should p6 (until next r3 writing instruction

•  At commit of xor, no instructions before it are in the pipeline

Commit Example


r1 p9

r2 p2

r3 p8

r4 p7

r5 p5

Map table Free-list



[ p3 ] [ p4 ] [ p6 ] [ p1 ]

p10

Commit Example


r1 p9

r2 p2

r3 p8

r4 p7

r5 p5

Map table Free-list



[ p3 ] [ p4 ] [ p6 ] [ p1 ]

p3

p10

Commit Example


r1 p9

r2 p2

r3 p8

r4 p7

r5 p5

Map table Free-list

p10

add p6 + p4 ➜ p7 sub p5 - p2 ➜ p8 addi p8 + 1 ➜ p9

add r3 + r4 ➜ r4 sub r5 - r2 ➜ r3 addi r3 + 1 ➜ r1

[ p4 ] [ p6 ] [ p1 ]

p4

p3

Commit Example


r1 p9

r2 p2

r3 p8

r4 p7

r5 p5

Map table Free-list

p10

sub p5 - p2 ➜ p8 addi p8 + 1 ➜ p9

sub r5 - r2 ➜ r3 addi r3 + 1 ➜ r1

[ p6 ] [ p1 ]

p4

p3

p6

Commit Example


r1 p9

r2 p2

r3 p8

r4 p7

r5 p5

Map table Free-list

p10

addi p8 + 1 ➜ p9 addi r3 + 1 ➜ r1 [ p1 ]

p4

p3

p6

p1

Commit Example


r1 p9

r2 p2

r3 p8

r4 p7

r5 p5

Map table Free-list

p10

p4

p3

p6

p1

Dynamic Scheduling Example


Dynamic Scheduling Example

•  The following slides are a detailed but concrete example

•  Yet, it contains enough detail to be overwhelming •  Try not to worry about the details

•  Focus on the big picture take-away:

Hardware can reorder instructions to extract instruction-level parallelism


Recall: Motivating Example

•  How would this execution occur cycle-by-cycle?

•  Execution latencies assumed in this example: •  Loads have two-cycle load-to-use penalty

•  Three cycle total execution latency •  All other instructions have single-cycle execution latency

•  “Issue queue”: hold all waiting (un-executed) instructions •  Holds ready/not-ready status •  Faster than looking up in ready table each cycle


0 1 2 3 4 5 6 7 8 9 10 11 12

ld [p1] ➜ p2 F Di I RR X M1 M2 W C add p2 + p3 ➜ p4 F Di I RR X W C xor p4 ^ p5 ➜ p6 F Di I RR X W C ld [p7] ➜ p8 F Di I RR X M1 M2 W C

Out-of-Order Pipeline – Cycle 0 0 1 2 3 4 5 6 7 8 9 10 11 12

ld [r1] ➜ r2 F add r2 + r3 ➜ r4 F xor r4 ^ r5 ➜ r6

ld [r7] ➜ r4

Issue Queue

Insn Src1 R? Src2 R? Dest #

Ready Table p1 yes p2 yes p3 yes p4 yes p5 yes p6 yes p7 yes p8 yes p9 ---

p10 --- p11 --- p12 ---

Map Table

r1 p8

r2 p7

r3 p6

r4 p5

r5 p4

r6 p3

r7 p2

r8 p1

Insn To Free Done? ld no

add no

Reorder Buffer

Out-of-Order Pipeline – Cycle 1a 0 1 2 3 4 5 6 7 8 9 10 11 12

ld [r1] ➜ r2 F Di add r2 + r3 ➜ r4 F xor r4 ^ r5 ➜ r6

ld [r7] ➜ r4

Issue Queue


ld p8 yes --- yes p9 0

Ready Table p1 yes p2 yes p3 yes p4 yes p5 yes p6 yes p7 yes p8 yes p9 no

p10 --- p11 --- p12 ---

Map Table

r1 p8

r2 p9

r3 p6

r4 p5

r5 p4

r6 p3

r7 p2

r8 p1

Insn To Free Done? ld p7 no

add no

Reorder Buffer

Out-of-Order Pipeline – Cycle 1b 0 1 2 3 4 5 6 7 8 9 10 11 12

ld [r1] ➜ r2 F Di add r2 + r3 ➜ r4 F Di xor r4 ^ r5 ➜ r6

ld [r7] ➜ r4

Issue Queue



add p9 no p6 yes p10 1


p10 no p11 --- p12 ---

Map Table

r1 p8

r2 p9

r3 p6

r4 p10

r5 p4

r6 p3

r7 p2

r8 p1


add p5 no

Reorder Buffer

Out-of-Order Pipeline – Cycle 1c 0 1 2 3 4 5 6 7 8 9 10 11 12

ld [r1] ➜ r2 F Di add r2 + r3 ➜ r4 F Di xor r4 ^ r5 ➜ r6 F ld [r7] ➜ r4 F

Issue Queue





p10 no p11 --- p12 ---

Map Table

r1 p8

r2 p9

r3 p6

r4 p10

r5 p4

r6 p3

r7 p2

r8 p1


add p5 no xor no ld no

Reorder Buffer


ld [r1] ➜ r2 F Di I add r2 + r3 ➜ r4 F Di xor r4 ^ r5 ➜ r6 F ld [r7] ➜ r4 F

Issue Queue





p10 no p11 --- p12 ---

Map Table

r1 p8

r2 p9

r3 p6

r4 p10

r5 p4

r6 p3

r7 p2

r8 p1


add p5 no xor no ld no

Reorder Buffer


ld [r1] ➜ r2 F Di I add r2 + r3 ➜ r4 F Di xor r4 ^ r5 ➜ r6 F Di ld [r7] ➜ r4 F

Issue Queue




xor p10 no p4 yes p11 2


p10 no p11 no p12 ---

Map Table

r1 p8

r2 p9

r3 p6

r4 p10

r5 p4

r6 p11

r7 p2

r8 p1


add p5 no xor p3 no ld no

Reorder Buffer

Out-of-Order Pipeline – Cycle 2c 0 1 2 3 4 5 6 7 8 9 10 11 12

ld [r1] ➜ r2 F Di I add r2 + r3 ➜ r4 F Di xor r4 ^ r5 ➜ r6 F Di ld [r7] ➜ r4 F Di

Issue Queue







p10 no p11 no p12 no

Map Table

r1 p8

r2 p9

r3 p6

r4 p12

r5 p4

r6 p11

r7 p2

r8 p1


add p5 no xor p3 no ld p10 no

Reorder Buffer


ld [r1] ➜ r2 F Di I RR add r2 + r3 ➜ r4 F Di xor r4 ^ r5 ➜ r6 F Di ld [r7] ➜ r4 F Di I

Issue Queue








Map Table

r1 p8

r2 p9

r3 p6

r4 p12

r5 p4

r6 p11

r7 p2

r8 p1



Reorder Buffer


ld [r1] ➜ r2 F Di I RR X add r2 + r3 ➜ r4 F Di xor r4 ^ r5 ➜ r6 F Di ld [r7] ➜ r4 F Di I RR

Issue Queue



add p9 yes p6 yes p10 1



Ready Table p1 yes p2 yes p3 yes p4 yes p5 yes p6 yes p7 yes p8 yes p9 yes


Map Table

r1 p8

r2 p9

r3 p6

r4 p12

r5 p4

r6 p11

r7 p2

r8 p1



Reorder Buffer


ld [r1] ➜ r2 F Di I RR X M1

add r2 + r3 ➜ r4 F Di I xor r4 ^ r5 ➜ r6 F Di ld [r7] ➜ r4 F Di I RR X

Issue Queue




xor p10 yes p4 yes p11 2



p10 yes p11 no p12 no

Map Table

r1 p8

r2 p9

r3 p6

r4 p12

r5 p4

r6 p11

r7 p2

r8 p1



Reorder Buffer


ld [r1] ➜ r2 F Di I RR X M1

add r2 + r3 ➜ r4 F Di I xor r4 ^ r5 ➜ r6 F Di ld [r7] ➜ r4 F Di I RR X

Issue Queue







p10 yes p11 no p12 yes

Map Table

r1 p8

r2 p9

r3 p6

r4 p12

r5 p4

r6 p11

r7 p2

r8 p1



Reorder Buffer


ld [r1] ➜ r2 F Di I RR X M1 M2

add r2 + r3 ➜ r4 F Di I RR xor r4 ^ r5 ➜ r6 F Di I ld [r7] ➜ r4 F Di I RR X M1

Issue Queue







p10 yes p11 yes p12 yes

Map Table

r1 p8

r2 p9

r3 p6

r4 p12

r5 p4

r6 p11

r7 p2

r8 p1



Reorder Buffer


ld [r1] ➜ r2 F Di I RR X M1 M2 W add r2 + r3 ➜ r4 F Di I RR X xor r4 ^ r5 ➜ r6 F Di I RR ld [r7] ➜ r4 F Di I RR X M1 M2

Issue Queue








Map Table

r1 p8

r2 p9

r3 p6

r4 p12

r5 p4

r6 p11

r7 p2

r8 p1

Insn To Free Done? ld p7 yes


Reorder Buffer


ld [r1] ➜ r2 F Di I RR X M1 M2 W C add r2 + r3 ➜ r4 F Di I RR X xor r4 ^ r5 ➜ r6 F Di I RR ld [r7] ➜ r4 F Di I RR X M1 M2

Issue Queue






Ready Table p1 yes p2 yes p3 yes p4 yes p5 yes p6 yes p7 --- p8 yes p9 yes


Map Table

r1 p8

r2 p9

r3 p6

r4 p12

r5 p4

r6 p11

r7 p2

r8 p1



Reorder Buffer


ld [r1] ➜ r2 F Di I RR X M1 M2 W C add r2 + r3 ➜ r4 F Di I RR X W xor r4 ^ r5 ➜ r6 F Di I RR X ld [r7] ➜ r4 F Di I RR X M1 M2 W

Issue Queue






Ready Table p1 yes p2 yes p3 yes p4 yes p5 yes p6 yes p7 --- p8 yes p9 yes


Map Table

r1 p8

r2 p9

r3 p6

r4 p12

r5 p4

r6 p11

r7 p2

r8 p1


add p5 yes xor p3 no ld p10 yes

Reorder Buffer


ld [r1] ➜ r2 F Di I RR X M1 M2 W C add r2 + r3 ➜ r4 F Di I RR X W C xor r4 ^ r5 ➜ r6 F Di I RR X ld [r7] ➜ r4 F Di I RR X M1 M2 W

Issue Queue






Ready Table p1 yes p2 yes p3 yes p4 yes p5 --- p6 yes p7 --- p8 yes p9 yes


Map Table

r1 p8

r2 p9

r3 p6

r4 p12

r5 p4

r6 p11

r7 p2

r8 p1


add p5 yes xor p3 no ld p10 yes

Reorder Buffer


ld [r1] ➜ r2 F Di I RR X M1 M2 W C add r2 + r3 ➜ r4 F Di I RR X W C xor r4 ^ r5 ➜ r6 F Di I RR X W ld [r7] ➜ r4 F Di I RR X M1 M2 W

Issue Queue






Ready Table p1 yes p2 yes p3 yes p4 yes p5 --- p6 yes p7 --- p8 yes p9 yes


Map Table

r1 p8

r2 p9

r3 p6

r4 p12

r5 p4

r6 p11

r7 p2

r8 p1


add p5 yes xor p3 yes ld p10 yes

Reorder Buffer


ld [r1] ➜ r2 F Di I RR X M1 M2 W C add r2 + r3 ➜ r4 F Di I RR X W C xor r4 ^ r5 ➜ r6 F Di I RR X W C ld [r7] ➜ r4 F Di I RR X M1 M2 W C

Issue Queue






Ready Table p1 yes p2 yes p3 --- p4 yes p5 --- p6 yes p7 --- p8 yes p9 yes

p10 --- p11 yes p12 yes

Map Table

r1 p8

r2 p9

r3 p6

r4 p12

r5 p4

r6 p11

r7 p2

r8 p1



Reorder Buffer

Out-of-Order Pipeline – Done! 0 1 2 3 4 5 6 7 8 9 10 11 12

ld [r1] ➜ r2 F Di I RR X M1 M2 W C add r2 + r3 ➜ r4 F Di I RR X W C xor r4 ^ r5 ➜ r6 F Di I RR X W C ld [r7] ➜ r4 F Di I RR X M1 M2 W C

Issue Queue






Ready Table p1 yes p2 yes p3 --- p4 yes p5 --- p6 yes p7 --- p8 yes p9 yes

p10 --- p11 yes p12 yes

Map Table

r1 p8

r2 p9

r3 p6

r4 p12

r5 p4

r6 p11

r7 p2

r8 p1



Reorder Buffer

Handling Memory Operations


Recall: Types of Dependencies

•  RAW (Read After Write) = “true dependence” mul r0 * r1 ➜ r2 … add r2 + r3 ➜ r4

•  WAW (Write After Write) = “output dependence” mul r0 * r1➜ r2 … add r1 + r3 ➜ r2

•  WAR (Write After Read) = “anti-dependence” mul r0 * r1 ➜ r2 … add r3 + r4 ➜ r1

•  WAW & WAR are “false”, Can be totally eliminated by “renaming”


Also Have Dependencies via Memory

•  If value in “r2” and “r3” is the same… •  RAW (Read After Write) – True dependency

st r1 ➜ [r2] … ld [r3] ➜ r4

•  WAW (Write After Write) st r1 ➜ [r2] … st r4 ➜ [r3]

•  WAR (Write After Read) ld [r2] ➜ r1 … st r4 ➜ [r3]


WAR/WAW are “false dependencies” -  But can’t rename memory in same way as registers

-  Why? Address are not known at rename

- Need to use other tricks

Let’s Start with Just Stores

•  Stores: Write data cache, not registers •  Can we rename memory? •  Recover in the cache?

  No (at least not easily) •  Cache writes unrecoverable

•  Solution: write stores into cache only when certain •  When are we certain? At “commit”


Handling Stores

•  Can “st p4 ➜ [p6+8]” issue and begin execution? •  Its registers inputs are ready… •  Why or why not?

0 1 2 3 4 5 6 7 8 9 10 11 12

mul p1 * p2 ➜ p3 F Di I RR X1 X2 X3 X4 W C jump-not-zero p3 F Di I RR X W C st p5 ➜ [p3+4] F Di I RR X M W C st p4 ➜ [p6+8] F Di I?


Problem #1: Out-of-Order Stores

•  Can “st p4 ➜ [p6+8]” write the cache in cycle 6? •  “st p5 ➜ [p3+4]” has not yet executed

•  What if “p3+4 == p6+8” •  The two stores write the same address! WAW dependency! •  Not known until their “X” stages (cycle 5 & 8)

•  Unappealing solution: all stores execute in-order •  We can do better…

0 1 2 3 4 5 6 7 8 9 10 11 12

mul p1 * p2 ➜ p3 F Di I RR X1 X2 X3 X4 W C jump-not-zero p3 F Di I RR X W C st p5 ➜ [p3+4] F Di I RR X M W C st p4 ➜ [p6+8] F Di I? RR X M W C


Problem #2: Speculative Stores

•  Can “st p4 ➜ [p6+8]” write the cache in cycle 6? •  Store is still “speculative” at this point

•  What if “jump-not-zero” is mis-predicted? •  Not known until its “X” stage (cycle 8)

•  How does it “undo” the store once it hits the cache? •  Answer: it can’t; stores write the cache only at commit •  Guaranteed to be non-speculative at that point

0 1 2 3 4 5 6 7 8 9 10 11 12

mul p1 * p2 ➜ p3 F Di I RR X1 X2 X3 X4 W C jump-not-zero p3 F Di I RR X W C st p5 ➜ [p3+4] F Di I RR X M W C st p4 ➜ [p6+8] F Di I? RR X M W C


Store Queue (SQ)

•  Solves two problems •  Allows for recovery of speculative stores •  Allows out-of-order stores

•  Store Queue (SQ) •  At dispatch, each store is given a slot in the Store Queue •  First-in-first-out (FIFO) queue •  Each entry contains: “address”, “value”, and “#” (program order)

•  Operation: •  Dispatch (in-order): allocate entry in SQ (stall if full) •  Execute (out-of-order): write store value into store queue •  Commit (in-order): read value from SQ and write into data cache •  Branch recovery: remove entries from the store queue

•  Address the above two problems, plus more…


Memory Forwarding

•  Can “ld [p7] ➜ p8” issue and begin execution? •  Why or why not?

0 1 2 3 4 5 6 7 8 9 10 11 12

fdiv p1 / p2 ➜ p9 F Di I RR X1 X2 X3 X4 X5 X6 W C st p4 ➜ [p5+4] F Di I RR X W C st p3 ➜ [p6+8] F Di I RR X W C ld [p7] ➜ p8 F Di I? RR X M1 M2 W C


Memory Forwarding


•  If the load reads from either of the store’s addresses… •  Load must get correct value, but it isn’t written to the cache until commit…

0 1 2 3 4 5 6 7 8 9 10 11 12

fdiv p1 / p2 ➜ p9 F Di I RR X1 X2 X3 X4 X5 X6 W C st p4 ➜ [p5+4] F Di I RR X SQ C st p3 ➜ [p6+8] F Di I RR X SQ C ld [p7] ➜ p8 F Di I? RR X M1 M2 W C


Memory Forwarding


•  If the load reads from either of the store’s addresses… •  Load must get correct value, but it isn’t written to the cache until commit…

•  Solution: “memory forwarding” •  Loads also searches the Store Queue (in parallel with cache access) •  Conceptually like register bypassing, but different implementation

•  Why? Addresses unknown until execute

0 1 2 3 4 5 6 7 8 9 10 11 12

fdiv p1 / p2 ➜ p9 F Di I RR X1 X2 X3 X4 X5 X6 W C st p4 ➜ [p5+4] F Di I RR X SQ C st p3 ➜ [p6+8] F Di I RR X SQ C ld [p7] ➜ p8 F Di I? RR X M1 M2 W C


Problem #3: WAR Hazards

•  What if “p3+4 == p6 + 8”? •  Then load and store access same memory location

•  Need to make sure that load doesn’t read store’s result •  Need to get values based on “program order” not “execution order”

•  Bad solution: require all stores/loads to execute in-order •  Good solution: Track order, loads search SQ

•  Read from store to same address that is “earlier in program order” •  Another reason the SQ is a FIFO queue

0 1 2 3 4 5 6 7 8 9 10 11 12

mul p1 * p2 ➜ p3 F Di I RR X1 X2 X3 X4 W C jump-not-zero p3 F Di I RR X W C ld [p3+4] ➜ p5 F Di I RR X M1 M2 W C st p4 ➜ [p6+8] F Di I RR X SQ C


Memory Forwarding via Store Queue •  Store Queue (SQ)

•  Holds all in-flight stores •  CAM: searchable by address •  “Age” to determine which to

forward from

•  Store rename/dispatch •  Allocate entry in SQ

•  Store execution •  Update SQ (Address + Data)

•  Load execution •  Search SQ to find: most recent

store prior to the load (program order)

•  Match? Read SQ •  No Match? Read cache


value address == == == == == == == ==

age

Data cache

head

tail

load position

address data in

data out Store Queue (SQ)

Store Queue (SQ)

•  On load execution, select the store that is: •  To same address as load •  Prior to the load (before the load in program order)

•  Of these, select the “youngest” store •  The store to the address that most recently preceded the load


When Can Loads Execute?

•  Can “ld [p6+8] ➜ p7” issue in cycle 3 •  Why or why not?

0 1 2 3 4 5 6 7 8 9 10 11 12

mul p1 * p2 ➜ p3 F Di I RR X1 X2 X3 X4 W C jump-not-zero p3 F Di I RR X W C st p5 ➜ [p3+4] F Di I RR X SQ C ld [p6+8] ➜ p7 F Di I? RR X M1 M2 W C


When Can Loads Execute?

•  Aliasing! Does p3+4 == p6+8? •  If no, load should get value from memory

•  Can it start to execute? •  If yes, load should get value from store

•  By reading the store queue? •  But the value isn’t put into the store queue until cycle 9

•  Key challenge: don’t know addresses until execution! •  One solution: require all loads to wait for all earlier (prior) stores

0 1 2 3 4 5 6 7 8 9 10 11 12

mul p1 * p2 ➜ p3 F Di I RR X1 X2 X3 X4 W C jump-not-zero p3 F Di I RR X W C st p5 ➜ [p3+4] F Di I RR X SQ C ld [p6+8] ➜ p7 F Di I? RR X M1 M2 W C



Compiler Scheduling Requires •  Alias analysis

•  Ability to tell whether load/store reference same memory locations •  Effectively, whether load/store can be rearranged

•  Example code: easy, all loads/stores use same base register (sp) •  New example: can compiler tell that r8 != r9? •  Must be conservative

Before

ld [r9+4]➜r2 ld [r9+8]➜r3 add r3,r2➜r1 //stall st r1➜[r9+0] ld [r8+0]➜r5 ld [r8+4]➜r6 sub r5,r6➜r4 //stall st r4➜[r8+8]

Wrong(?)

ld [r9+4]➜r2 ld [r9+8]➜r3 ld [r8+0]➜r5 //does r8==r9? add r3,r2➜r1 ld [r8+4]➜r6 //does r8+4==r9? st r1➜[r9+0] sub r5,r6➜r4 st r4➜[r8+8]


Dynamically Scheduling Memory Ops •  Compilers must schedule memory ops conservatively •  Options for hardware:

•  Don’t execute any load until all prior stores execute (conservative) •  Execute loads as soon as possible, detect violations (optimistic)

•  When a store executes, it checks if any later loads executed too early (to same address). If so, flush pipeline

•  Learn violations over time, selectively reorder (predictive) Before ld [r9+4]➜r2 ld [r9+8]➜r3 add r3,r2➜r1 //stall st r1➜[r9+0] ld [r8+0]➜r5 ld [r8+4]➜r6 sub r5,r6➜r4 //stall st r4➜[r8+8]

Wrong(?) ld [r9+4]➜r2 ld [r9+8]➜r3 ld [r8+0]➜r5 //does r8==sp? add r3,r2➜r1 ld [r8+4]➜r6 //does r8+4==sp? st r1➜[r9+0] sub r5,r6➜r4 st r4➜[r8+8]

Conservative Load Scheduling

•  Conservative load scheduling: •  All earlier stores have executed

•  Some architectures: split store address / store data •  Only requires knowing addresses (not the store values)

•  Advantage: always safe •  Disadvantage: performance (limits out-of-orderness)


Conservative Load Scheduling 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

ld [p1] ➜ p4 F Di I Rr X M1 M2 W C ld [p2] ➜ p5 F Di I Rr X M1 M2 W C add p4, p5 ➜ p6 F Di I Rr X W C st p6 ➜ [p3] F Di I Rr X SQ C ld [p1+4] ➜ p7 F Di I Rr X M1 M2 W C ld [p2+4] ➜ p8 F Di I Rr X M1 M2 W C add p7, p8 ➜ p9 F Di I Rr X W C st p9 ➜ [p3+4] F Di I Rr X SQ C


Conservative load scheduling: can’t issue ld [p1+4] until cycle 7! Might as well be an in-order machine on this example Can we do better? How?

Optimistic Load Scheduling 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

ld [p1] ➜ p4 F Di I Rr X M1 M2 W C ld [p2] ➜ p5 F Di I Rr X M1 M2 W C add p4, p5 ➜ p6 F Di I Rr X W C st p6 ➜ [p3] F Di I Rr X SQ C ld [p1+4] ➜ p7 F Di I Rr X M1 M2 W C ld [p2+4] ➜ p8 F Di I Rr X M1 M2 W C add p7, p8 ➜ p9 F Di I Rr X W C st p9 ➜ [p3+4] F Di I Rr X SQ C


Optimistic load scheduling: can actually benefit from out-of-order! But how do we know when out speculation (optimism) fails?

Load Speculation

•  Speculation requires two things….. •  1. Detection of mis-speculations

•  How can we do this?

•  2. Recovery from mis-speculations •  Squash from offending load •  Saw how to squash from branches: same method


Load Queue

•  Detects load ordering violations

•  Load execution: Write LQ •  Write address into LQ •  Record which in-flight store

it forwarded from (if any)

•  Store execution: Search LQ •  For a store S, foreach load L:

•  Does S.addr = L.addr? •  Is S before L in program

order? •  Which store did L gets its

value from?


== == == == == == == ==

Data Cache

head

tail

load queue (LQ)

address == == == == == == == ==

tail

head

age

store position flush?

SQ

Store Queue + Load Queue

•  Store Queue: handles forwarding •  Entry per store (allocated @ dispatch, deallocated @ commit) •  Written by stores (@ execute) •  Searched by loads (@ execute) •  Read from to write data cache (@ commit)

•  Load Queue: detects ordering violations •  Entry per load (allocated @ dispatch, deallocated @ commit) •  Written by loads (@ execute) •  Searched by stores (@ execute)

•  Both together •  Allows aggressive load scheduling •  Stores don’t constrain load execution


Optimistic Load Scheduling Problem

•  Allows loads to issue before earlier stores •  Increases out-of-orderness +  Good: When no conflict, increases performance -  Bad: Conflict => squash => worse performance than waiting

•  Can we have our cake AND eat it too?


Predictive Load Scheduling

•  Predict which loads must wait for stores

•  Fool me once, shame on you-- fool me twice? •  Loads default to aggressive •  Keep table of load PCs that have been caused squashes

•  Schedule these conservatively +  Simple predictor -  Makes “bad” loads wait for all stores before it is not so great

•  More complex predictors used in practice •  Predict which stores loads should wait for •  “Store Sets”


Load/Store Queue Examples


Initial State 1. St p1 ➜ [p2] 2. St p3 ➜ [p4] 3. Ld [p5] ➜ p6

RegFile

p1 5

p2 100

p3 9

p4 200

p5 100

p6 ---

p7 ---

p8 ---

Store Queue

# Addr Val

RegFile

p1 5

p2 100

p3 9

p4 200

p5 100

p6 ---

p7 ---

p8 ---

Store Queue

# Addr Val

RegFile

p1 5

p2 100

p3 9

p4 200

p5 100

p6 ---

p7 ---

p8 ---

Store Queue

# Addr Val

Addr Val

100 13

200 17

Cache Addr Val

100 13

200 17

Cache Addr Val

100 13

200 17

Cache

(Stores to different addresses)

Load Queue

# Addr From

Load Queue

# Addr From

Load Queue

# Addr From

Good Interleaving 1. St p1 ➜ [p2] 2. St p3 ➜ [p4] 3. Ld [p5] ➜ p6

RegFile

p1 5

p2 100

p3 9

p4 200

p5 100

p6 ---

p7 ---

p8 ---

Store Queue

# Addr Val

1 100 5

RegFile

p1 5

p2 100

p3 9

p4 200

p5 100

p6 ---

p7 ---

p8 ---

Store Queue

# Addr Val

1 100 5

2 200 9

RegFile

p1 5

p2 100

p3 9

p4 200

p5 100

p6 5

p7 ---

p8 ---

Store Queue

# Addr Val

1 100 5

2 200 9

Addr Val

100 13

200 17

Cache Addr Val

100 13

200 17

Cache Addr Val

100 13

200 17

Cache

1.  St p1 ➜ [p2] 2. St p3 ➜ [p4] 3. Ld [p5] ➜ p6

(Shows importance of address check)

Load Queue

# Addr From

Load Queue

# Addr From

Load Queue

# Addr From

3 100 #1

Different Initial State 1. St p1 ➜ [p2] 2. St p3 ➜ [p4] 3. Ld [p5] ➜ p6

RegFile

p1 5

p2 100

p3 9

p4 100

p5 100

p6 ---

p7 ---

p8 ---

Store Queue

# Addr Val

RegFile

p1 5

p2 100

p3 9

p4 100

p5 100

p6 ---

p7 ---

p8 ---

Store Queue

# Addr Val

RegFile

p1 5

p2 100

p3 9

p4 100

p5 100

p6 ---

p7 ---

p8 ---

Store Queue

# Addr Val

Addr Val

100 13

200 17

Cache Addr Val

100 13

200 17

Cache Addr Val

100 13

200 17

Cache

(All to same address)

Load Queue

# Addr From

Load Queue

# Addr From

Load Queue

# Addr From

Good Interleaving #1 1. St p1 ➜ [p2] 2. St p3 ➜ [p4] 3. Ld [p5] ➜ p6

RegFile

p1 5

p2 100

p3 9

p4 100

p5 100

p6 ---

p7 ---

p8 ---

Store Queue

# Addr Val

1 100 5

RegFile

p1 5

p2 100

p3 9

p4 100

p5 100

p6 ---

p7 ---

p8 ---

Store Queue

# Addr Val

1 100 5

2 100 9

RegFile

p1 5

p2 100

p3 9

p4 100

p5 100

p6 9

p7 ---

p8 ---

Store Queue

# Addr Val

1 100 5

2 100 9

Addr Val

100 13

200 17

Cache Addr Val

100 13

200 17

Cache Addr Val

100 13

200 17

Cache

1. St p1 ➜ [p2] 2. St p3 ➜ [p4] 3. Ld [p5] ➜ p6

(Program Order)

Load Queue

# Addr From

Load Queue

# Addr From

Load Queue

# Addr From

3 100 #2


Store Queue

# Addr Val

2 100 9

Store Queue

# Addr Val

1 100 5

2 100 9

RegFile

p1 5

p2 100

p3 9

p4 100

p5 100

p6 9

p7 ---

p8 ---

Store Queue

# Addr Val

1 100 5

2 100 9

Addr Val

100 13

200 17

Cache Addr Val

100 13

200 17

Cache Addr Val

100 13

200 17

Cache

2. St p3 ➜ [p4] 1. St p1 ➜ [p2] 3. Ld [p5] ➜ p6

(Stores reordered, so okay)

RegFile

p1 5

p2 100

p3 9

p4 100

p5 100

p6 ---

p7 ---

p8 ---

RegFile

p1 5

p2 100

p3 9

p4 100

p5 100

p6 ---

p7 ---

p8 ---

Load Queue

# Addr From

Load Queue

# Addr From

Load Queue

# Addr From

3 100 #2

Bad Interleaving #1 1. St p1 ➜ [p2] 2. St p3 ➜ [p4] 3. Ld [p5] ➜ p6

RegFile

p1 5

p2 100

p3 9

p4 100

p5 100

p6 13

p7 ---

p8 ---

Store Queue

# Addr Val

RegFile

p1 5

p2 100

p3 9

p4 100

p5 100

p6 13

p7 ---

p8 ---

Store Queue

# Addr Val

2 100 9

Addr Val

100 13

200 17

Cache Addr Val

100 13

200 17

Cache

3. Ld [p5] ➜ p6 2. St p3 ➜ [p4]

(Load reads the cache, but should not)

Load Queue

# Addr From

3 100 --

Load Queue

# Addr From

3 100 --

Bad Interleaving #2 1. St p1 ➜ [p2] 2. St p3 ➜ [p4] 3. Ld [p5] ➜ p6

RegFile

p1 5

p2 100

p3 9

p4 100

p5 100

p6 ---

p7 ---

p8 ---

Store Queue

# Addr Val

1 100 5

RegFile

p1 5

p2 100

p3 9

p4 100

p5 100

p6 5

p7 ---

p8 ---

Store Queue

# Addr Val

1 100 5

RegFile

p1 5

p2 100

p3 9

p4 100

p5 100

p6 5

p7 ---

p8 ---

Store Queue

# Addr Val

1 100 5

2 100 9

Addr Val

100 13

200 17

Cache Addr Val

100 13

200 17

Cache Addr Val

100 13

200 17

Cache

1. St p1 ➜ [p2] 3. Ld [p5] ➜ p6 2. St p3 ➜ [p4]

(Load gets value from wrong store)

Load Queue

# Addr From

Load Queue

# Addr From

3 100 #1

Load Queue

# Addr From

3 100 --

Load Queue

# Addr From

3 100 #1


RegFile

p1 5

p2 100

p3 9

p4 100

p5 100

p6 ---

p7 ---

p8 ---

Store Queue

# Addr Val

2 100 9

RegFile

p1 5

p2 100

p3 9

p4 100

p5 100

p6 9

p7 ---

p8 ---

Store Queue

# Addr Val

2 100 9

RegFile

p1 5

p2 100

p3 9

p4 100

p5 100

p6 9

p7 ---

p8 ---

Store Queue

# Addr Val

1 100 5

2 100 9

Addr Val

100 13

200 17

Cache Addr Val

100 13

200 17

Cache Addr Val

100 13

200 17

Cache

2. St p3 ➜ [p4] 3. Ld [p5] ➜ p6 1.  St p1 ➜ [p2] Load Queue

# Addr From

(Using “From” field to prevent false squash)

Load Queue

# Addr From

3 100 #2

Load Queue

# Addr From

3 100 #2

Out-of-Order: Benefits & Challenges


Dynamic Scheduling Operation •  Dynamic scheduling

•  Totally in the hardware (not visible to software) •  Also called “out-of-order execution” (OoO)

•  Fetch many instructions into instruction window •  Use branch prediction to speculate past (multiple) branches •  Flush pipeline on branch misprediction

•  Rename registers to avoid false dependencies •  Execute instructions as soon as possible

•  Register dependencies are known •  Handling memory dependencies more tricky

•  “Commit” instructions in order •  Anything strange happens before commit, just flush the pipeline

•  How much out-of-order? Core i7 “Haswell”: •  192-entry reorder buffer, 168 integer registers, 60-entry scheduler






Out of Order: Benefits

•  Allows speculative re-ordering •  Loads / stores •  Branch prediction to look past branches

•  Done by hardware •  Compiler may want different schedule for different hw configs •  Hardware has only its own configuration to deal with

•  Schedule can change due to cache misses •  Different schedule optimal from on cache hit

•  Memory-level parallelism •  Executes “around” cache misses to find independent instructions •  Finds and initiates independent misses, reducing memory latency

•  Especially good at hiding L2 hits (~11 cycles in Core i7)


Challenges for Out-of-Order Cores •  Design complexity

•  More complicated than in-order? Certainly! •  But, we have managed to overcome the design complexity

•  Clock frequency •  Can we build a “high ILP” machine at high clock frequency? •  Yep, with some additional pipe stages, clever design

•  Limits to (efficiently) scaling the window and ILP •  Large physical register file •  Fast register renaming/wakeup/select/load queue/store queue

•  Active areas of micro-architectural research •  Branch & memory depend. prediction (limits effective window size)

•  95% branch mis-prediction: 1 in 20 branches, or 1 in 100 insn. •  Plus all the issues of build “wide” in-order superscalar

•  Power efficiency •  Today, even mobile phone chips are out-of-order cores


Redux: Hdw vs. Software Scheduling

•  Static scheduling •  Performed by compiler, limited in several ways

•  Dynamic scheduling •  Performed by the hardware, overcomes limitations

•  Static limitation ➜ dynamic mitigation •  Number of registers in the ISA ➜ register renaming •  Scheduling scope ➜ branch prediction & speculation •  Inexact memory aliasing information ➜ speculative memory ops •  Unknown latencies of cache misses ➜ execute when ready

•  Which to do? Compiler does what it can, hardware the rest •  Why? dynamic scheduling needed to sustain more than 2-way issue •  Helps with hiding memory latency (execute around misses) •  Intel Core i7 is four-wide execute w/ large scheduling window •  Even mobile phones have dynamic scheduled cores (ARM A9)



Summary: Scheduling

•  Code scheduling •  To reduce pipeline stalls •  To increase ILP (insn level parallelism)

•  Static scheduling by the compiler •  Approach & limitations

•  Dynamic scheduling in hardware •  Register renaming •  Instruction selection •  Handling memory operations

•  Up next: multicore

CPU Mem I/O

System software

App App App

Documents

CIS 371 Computer Organization and Designmilom/cis371-Spring13/lectures/10... · CIS 371: Comp. Org. | Prof. Milo Martin | Scheduling 1 CIS 371 Computer Organization and Design Unit