Dynamic Verification of Cache Dynamic Verification of Cache Coherence ProtocolsCoherence Protocols

Jason F. Cantin

Mikko H. Lipasti

James E. Smith

6/30/2001 Workshop on Memory Performance

Issues

IntroductionIntroduction

Multiprocessors are used for a variety commercial and mission-critical tasks

Reliability is a growing concernCoherence is a fundamental feature of

shared-memory MPs• High design complexity• Relatively low interconnect reliability

6/30/2001 Workshop on Memory Performance

Issues

Introduction:Introduction:Cache Coherence ProtocolsCache Coherence ProtocolsNotoriously difficult to design and verify

Often conceptually simple, but with complex implementations for efficiency and handling special cases

Multiple finite state machines operating concurrently

6/30/2001 Workshop on Memory Performance

Issues

Introduction:Introduction:A Simple ExampleA Simple ExampleMSI Protocol

“Architected” State Invalid / Not Present Shared (readable) Modified (read/write)

I S

M

Bus_RdX, Replace

Read

Write

Bus_RdX,Replace

Write

Bus_Rd

6/30/2001 Workshop on Memory Performance

Issues

M

S

PendRdI

I

PendWBM

PendRdX

S

PendRdX

I

Bus_RdX, Replace

ReadBus_Av

Replace

Bus_Av

Bus_Av

Bus_Av

Bus_Rd

Write

Bus_RdX

Write

Bus_RdX

Introduction:Introduction:Simple Example with a BusSimple Example with a Bus

MSI Protocol “Implementation” State

Transient states for pending operations

Arcs to satisfy requests while operations pending

6/30/2001 Workshop on Memory Performance

Issues

ProblemProblem

In practice, implementations can have dozens of states• Atomic memory operations• Split transaction buses• Protocol optimizations

Complexity grows exponentially with added states• Random testing: Low Coverage• Exhaustive testing: Too time consuming

6/30/2001 Workshop on Memory Performance

Issues

Dynamic VerificationDynamic Verification

Check the implementation at runtime• It is easier to check a computation than to

do the actual computation, provided there is a delay between the computation and the check (Rotenberg, AR-SMT)

• Simplified version of a processor implementation can be used for online verification (Austin, DIVA)

6/30/2001 Workshop on Memory Performance

Issues

Dynamic Verification of Cache Dynamic Verification of Cache CoherenceCoherence

A distributed form of dynamic verification for multiprocessor memory systems• Simplified version of protocol added to each node

• Maintains architected state

• Check completed transitions and actions against simple protocol

• Additional messages (assertions) sent between nodes to ensure coherence

6/30/2001 Workshop on Memory Performance

Issues

Conceptual View for Superscalar Conceptual View for Superscalar Processors (DIVA)Processors (DIVA) Single, centralized

check processor Receives

instructions serially in program order from implementation

Physical registersComplex ExecutionProcessor

CheckProcessor

Arch.registers

Prediction Tables

Committed results

R.O.B.

Arch.registers

6/30/2001 Workshop on Memory Performance

Issues

Conceptual View for CoherenceConceptual View for Coherence

Distributed checking hardware

Transitions received in parallel, in completion order

Shared Logical Bus

Shared Validation Bus

Implementation Protocol

Simple Protocol

Completed Transitions

6/30/2001 Workshop on Memory Performance

Issues

High Level OrganizationHigh Level Organization

Cache

Controller

P

Shared logical bus(addresses, data, control)

Memory

DV-CCChecker

Validation bus (assertions to be checked)

6/30/2001 Workshop on Memory Performance

Issues

BenefitsBenefits

Detects hardware faults• Redundant computation• Including intermittent network failures

Detects design mistakes• Checker is simple and easy to verify

6/30/2001 Workshop on Memory Performance

Issues

DrawbacksDrawbacks

Time is required for checking, but…• May be overlapped with other activities• Simple protocol requires fewer transitions

Assertions consume bandwidth• May need second bus / network

Additional hardware• But not much

6/30/2001 Workshop on Memory Performance

Issues

DV for coherence in an SMPDV for coherence in an SMP

Architected state stored in a second tag array

Transactions sent to the checker when architected state changes• Address• Initial State and Final States• Input (Request, Snoop Responses, etc)• Action (Send Data, Respond Shared, etc)

6/30/2001 Workshop on Memory Performance

Issues

DV for coherence in an SMP (2)DV for coherence in an SMP (2)

Checker compares the initial state of a transition against the architected state

Final state and action recomputed and compared to implementation’s result

Assertions broadcast to other nodes to check coherence and confirm completion of transactions

Watchdog timer detects deadlock, livelock, and other omission failures

6/30/2001 Workshop on Memory Performance

Issues

Init. stateFinal state Input ActionAddress

NextState Logic

=?

Arch.Tag

State

Action Logic

=?=?

Error Detection / Diagnosis

Transition From Implementation Protocol

UpdateTags

Validation Bus

AsstSendBuffer

Checking a State TransitionChecking a State Transition

Watchdogtimer

6/30/2001 Workshop on Memory Performance

Issues

Arch.Tag

State

Error Detection / Diagnosis

Watchdogtimer

Validation Bus

AssertRecv

Buffer

OK

Address

Remote State

Checking an AssertionChecking an Assertion

6/30/2001 Workshop on Memory Performance

Issues

When to Broadcast AssertionsWhen to Broadcast Assertions

For MSI:1. I S (readable copy loaded)2. I M (writeable copy loaded)3. S M (upgrade)4. M I (writeback)

Note: The MS transition results from remote reads, and doesn’t require an extra assertion. Replacements (SI) are not considered here.

6/30/2001 Workshop on Memory Performance

Issues

Preliminary Data (4-way SMP)Preliminary Data (4-way SMP)

Most memory references do not change cache state (checker need not have high bandwidth)

MSI Stable State Changes/Reference

0.54%1.19%

7.17%

1.71%2.50%

0%

2%

4%

6%

8%

barnes ocean db2 specweb tpc-w

Perc

enta

ge o

f Re

fere

nces

6/30/2001 Workshop on Memory Performance

Issues

Preliminary Data (4-way SMP)Preliminary Data (4-way SMP)

0%

20%

40%

60%

80%

100%

Ext

ra M

ess

ag

es I-->M, S-->M

I-->S, I-->M

I-->S, I-->M, S-->M

All Transactions

6/30/2001 Workshop on Memory Performance

Issues

Future WorkFuture Work

Performance impact for a real SMP protocol implementation• In progress

Directory-based protocolsDynamically verifying memory modelsRecovery

• Can stall to avoid error propagation• Can write checkpoints periodically

6/30/2001 Workshop on Memory Performance

Issues

In SummaryIn Summary

Dynamic verification can be applied to multiprocessor systems (in a distributed manner)

Improves fault-tolerance, and design verification may be relaxed

More to come