44
McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Marco Elver Vijay Nagarajan University of Edinburgh HPCA, March 2016 1 www.inf.ed.ac.uk

 · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

  • Upload
    others

  • View
    5

  • Download
    0

Embed Size (px)

Citation preview

Page 1:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

McVerSi: A Test Generation Framework forFast Memory Consistency Verification in

Simulation

Marco Elver Vijay Nagarajan

University of Edinburgh

HPCA, March 2016

1 www.inf.ed.ac.uk

Page 2:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Overview

Problem

• How to rigorously verify hardware implementation (full system functionaldesign) satisfies memory consistency model in simulation?

• Simulation-based verification slow: takes long to achieve high coverage→ low assurance.

McVerSi: Evolutionary Test Generation Approach (GP)

• Carefully recombines tests, favoring racy operations.

• Coverage directed.

Results

• Bugs found faster and more consistent than random test generation andlitmus tests.

• Found all tested bugs in less than 1 day, unlike alternatives.

• Found 2 new bugs in Gem5.

2 www.inf.ed.ac.uk

Page 3:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Motivation

Memory consistency verification still a problem

• Real silicon still ships with consistency bugs1 2.

• Major challenges:1 In testing based approaches: how to cover all relevant aspects of

implementation?2 In formal verification: does translated abstraction represent real

implementation sufficiently?3 Which components (regardless of method): can miss bugs due to

interaction if not modelled correctly!

• Want rigorous verification of functional design implementation, e.g.cycle accurate model of full system.

1Alglave et al., “Herding cats”, In: TOPLAS, 2014.2Alglave et al., “GPU concurrency: Weak behaviours and programming assumptions”, In: ASPLOS,

2015.3 www.inf.ed.ac.uk

Page 4:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Memory Consistency Models

Consistency: contract between programmer and hardware

⇒ Specifies how memory system should behave: conveys orderingguarantees (or lack of) among memory ops.

Some Common MCMs

• SC: all program orders maintained.

• TSO: relaxes write→ read order.

init: x = 0, y = 0Thread 1 Thread 2x← 1 r1← yy← 1 r2← xAllow?: r1 = 1 ∧ r2 = 0

4 www.inf.ed.ac.uk

Page 5:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Memory Consistency Models

Consistency: contract between programmer and hardware

⇒ Specifies how memory system should behave: conveys orderingguarantees (or lack of) among memory ops.

Some Common MCMs

• SC: all program orders maintained.

• TSO: relaxes write→ read order.

init: x = 0, y = 0Thread 1 Thread 2x← 1 y← 1r1← y r2← xAllow?: r1 = 0 ∧ r2 = 0

4 www.inf.ed.ac.uk

Page 6:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

What components can affect the MCM?

• Core pipeline (in-order,OoO).

• MMU+TLB.

• Caches + coherenceprotocol (eager, lazy).

• Interconnect.

• ...

P0

CorePn

Core

SB

LB

SB

LBMMU+TLB MMU+TLB

L2 cache (shared)

L1 cache(private)

L1 cache(private)

On-chip interconnect

Main Memory

5 www.inf.ed.ac.uk

Page 7:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

What components can affect the MCM?

• Core pipeline (in-order,OoO).

• MMU+TLB.

• Caches + coherenceprotocol (eager, lazy).

• Interconnect.

• ...

P0

CorePn

CoreSB

LB

SB

LB

MMU+TLB MMU+TLB

L2 cache (shared)

L1 cache(private)

L1 cache(private)

On-chip interconnect

Main Memory

5 www.inf.ed.ac.uk

Page 8:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

What components can affect the MCM?

• Core pipeline (in-order,OoO).

• MMU+TLB.

• Caches + coherenceprotocol (eager, lazy).

• Interconnect.

• ...

P0

CorePn

CoreSB

LB

SB

LBMMU+TLB MMU+TLB

L2 cache (shared)

L1 cache(private)

L1 cache(private)

On-chip interconnect

Main Memory

5 www.inf.ed.ac.uk

Page 9:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

What components can affect the MCM?

• Core pipeline (in-order,OoO).

• MMU+TLB.

• Caches + coherenceprotocol (eager, lazy).

• Interconnect.

• ...

P0

CorePn

CoreSB

LB

SB

LBMMU+TLB MMU+TLB

L2 cache (shared)

L1 cache(private)

L1 cache(private)

On-chip interconnect

Main Memory

5 www.inf.ed.ac.uk

Page 10:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

What components can affect the MCM?

• Core pipeline (in-order,OoO).

• MMU+TLB.

• Caches + coherenceprotocol (eager, lazy).

• Interconnect.

• ...

P0

CorePn

CoreSB

LB

SB

LBMMU+TLB MMU+TLB

L2 cache (shared)

L1 cache(private)

L1 cache(private)

On-chip interconnect

Main Memory

5 www.inf.ed.ac.uk

Page 11:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

What components can affect the MCM?

• Core pipeline (in-order,OoO).

• MMU+TLB.

• Caches + coherenceprotocol (eager, lazy).

• Interconnect.

• ...

P0

CorePn

CoreSB

LB

SB

LBMMU+TLB MMU+TLB

L2 cache (shared)

L1 cache(private)

L1 cache(private)

On-chip interconnect

Main Memory

5 www.inf.ed.ac.uk

Page 12:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

What components can affect the MCM?

• Core pipeline (in-order,OoO).

• MMU+TLB.

• Caches + coherenceprotocol (eager, lazy).

• Interconnect.

• ...

P0

CorePn

CoreSB

LB

SB

LBMMU+TLB MMU+TLB

L2 cache (shared)

L1 cache(private)

L1 cache(private)

On-chip interconnect

Main Memory

Composition+interaction of components must still satisfy MCM!

5 www.inf.ed.ac.uk

Page 13:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

How to verify?

Formal verification

• Provides proof; e.g. model checking (exhaustive search) [Dill, 1996].

• Works for high-level abstractions, individual components.

• Impractical for detailed (e.g. cycle accurate) full system design.

Post-silicon verification

• Random testing (e.g. TSOtool [Hangal et al., 2004]), etc.

• Fastest→ good coverage (high assurance).

• Expensive; hard to debug.

6 www.inf.ed.ac.uk

Page 14:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

How to verify?

Formal verification

• Provides proof; e.g. model checking (exhaustive search) [Dill, 1996].

• Works for high-level abstractions, individual components.

• Impractical for detailed (e.g. cycle accurate) full system design.

Simulation-based verification of full system

• Litmus tests, pseudo-random tests [Saha et al., 1995].

• Cheap; easy to debug.

• Extremely slow→ poor coverage (low assurance).

Post-silicon verification

• Random testing (e.g. TSOtool [Hangal et al., 2004]), etc.

• Fastest→ good coverage (high assurance).

• Expensive; hard to debug.

6 www.inf.ed.ac.uk

Page 15:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

How to verify in simulation?

Litmus tests Full System Simulation Coverage Report

Write new tests

MCM Checker

Random TestGenerator

Full System Simulation Coverage Report

Modify parameters

7 www.inf.ed.ac.uk

Page 16:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

How to automate?

????? Full System Simulation Coverage Report

MCM Checker

8 www.inf.ed.ac.uk

Page 17:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Problem Statement

How to generate efficient MCM tests for simulation, s.t. wall-clock time toexplore rare corner cases is reduced?

1 What are good MCM tests?

2 How to feed back coverage into test generator?

9 www.inf.ed.ac.uk

Page 18:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

McVerSi Approach

McVerSi Full System Simulation Coverage Report

MCM Checker

GP based evolution

MCM Checker

T2 =b

T1 a= T1 c=

T1 c=

T2 b=

T2 c= T1 a=

T2 =a T1 b=

T1 b=

T2 =c

T2 =aT2 =b

T1 a= T1 c=

T1 c=

T2 b=

T2 c= T1 a=

T2 =a T1 b=

T1 b=

T2 =c

T2 =aT2 =b

T1 a= T1 c=

T1 c=

T2 b=

T2 c= T1 a=

T2 =a T1 b=

T1 b=

T2 =c

T2 =aT2 =b

T1 a= T1 c=

T1 c=

T2 b=

T2 c= T1 a=

T2 =a T1 b=

T1 b=

T2 =c

T2 =a

Population

Fitness

How racy?

Custom crossoverand mutation

Reproduction+ replacementSelection

How racy?

10 www.inf.ed.ac.uk

Page 19:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

McVerSi Approach

McVerSi Full System Simulation Coverage Report

MCM Checker

GP based evolution

MCM Checker

T2 =b

T1 a= T1 c=

T1 c=

T2 b=

T2 c= T1 a=

T2 =a T1 b=

T1 b=

T2 =c

T2 =aT2 =b

T1 a= T1 c=

T1 c=

T2 b=

T2 c= T1 a=

T2 =a T1 b=

T1 b=

T2 =c

T2 =aT2 =b

T1 a= T1 c=

T1 c=

T2 b=

T2 c= T1 a=

T2 =a T1 b=

T1 b=

T2 =c

T2 =aT2 =b

T1 a= T1 c=

T1 c=

T2 b=

T2 c= T1 a=

T2 =a T1 b=

T1 b=

T2 =c

T2 =a

Population

Fitness

How racy?

Custom crossoverand mutation

Reproduction+ replacementSelection

How racy?

10 www.inf.ed.ac.uk

Page 20:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

McVerSi Approach

McVerSi Full System Simulation Coverage Report

MCM Checker

GP based evolution

MCM Checker

T2 =b

T1 a= T1 c=

T1 c=

T2 b=

T2 c= T1 a=

T2 =a T1 b=

T1 b=

T2 =c

T2 =aT2 =b

T1 a= T1 c=

T1 c=

T2 b=

T2 c= T1 a=

T2 =a T1 b=

T1 b=

T2 =c

T2 =aT2 =b

T1 a= T1 c=

T1 c=

T2 b=

T2 c= T1 a=

T2 =a T1 b=

T1 b=

T2 =c

T2 =aT2 =b

T1 a= T1 c=

T1 c=

T2 b=

T2 c= T1 a=

T2 =a T1 b=

T1 b=

T2 =c

T2 =a

Population

Fitness

How racy?

Custom crossoverand mutation

Reproduction+ replacementSelection

How racy?

10 www.inf.ed.ac.uk

Page 21:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

McVerSi Approach

McVerSi Full System Simulation Coverage Report

MCM Checker

GP based evolution

MCM Checker

T2 =b

T1 a= T1 c=

T1 c=

T2 b=

T2 c= T1 a=

T2 =a T1 b=

T1 b=

T2 =c

T2 =aT2 =b

T1 a= T1 c=

T1 c=

T2 b=

T2 c= T1 a=

T2 =a T1 b=

T1 b=

T2 =c

T2 =aT2 =b

T1 a= T1 c=

T1 c=

T2 b=

T2 c= T1 a=

T2 =a T1 b=

T1 b=

T2 =c

T2 =aT2 =b

T1 a= T1 c=

T1 c=

T2 b=

T2 c= T1 a=

T2 =a T1 b=

T1 b=

T2 =c

T2 =a

Population

Fitness

How racy?

Custom crossoverand mutation

Reproduction+ replacementSelection

How racy?

10 www.inf.ed.ac.uk

Page 22:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

McVerSi Approach

McVerSi Full System Simulation Coverage Report

MCM Checker

GP based evolution

MCM Checker

T2 =b

T1 a= T1 c=

T1 c=

T2 b=

T2 c= T1 a=

T2 =a T1 b=

T1 b=

T2 =c

T2 =aT2 =b

T1 a= T1 c=

T1 c=

T2 b=

T2 c= T1 a=

T2 =a T1 b=

T1 b=

T2 =c

T2 =aT2 =b

T1 a= T1 c=

T1 c=

T2 b=

T2 c= T1 a=

T2 =a T1 b=

T1 b=

T2 =c

T2 =aT2 =b

T1 a= T1 c=

T1 c=

T2 b=

T2 c= T1 a=

T2 =a T1 b=

T1 b=

T2 =c

T2 =a

Population

Fitness

How racy?

Custom crossoverand mutation

Reproduction+ replacementSelection

How racy?

10 www.inf.ed.ac.uk

Page 23:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Measuring Fitness/Coverage

• Simulation runs continuously, with new tests loaded on-the-fly.

• Test fitness: result of coverage of individual test-run.

• Evaluated with coherence protocol transition coverage.

P0

CorePn

CoreSB

LB

SB

LBMMU+TLB MMU+TLB

On-chip interconnect

Main Memory

L2 cache (shared)

L1 cache(private)

L1 cache(private)

11 www.inf.ed.ac.uk

Page 24:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Measuring Fitness/Coverage

• Simulation runs continuously, with new tests loaded on-the-fly.

• Test fitness: result of coverage of individual test-run.

• Evaluated with coherence protocol transition coverage.

One of many possible options—highly dependent on design!

11 www.inf.ed.ac.uk

Page 25:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Test Suitability: What is a good MCM test?

init: x = 0, y = 0Thread 1 Thread 2x← 1 r1← yy← 1 r2← xr1 = {0, 1} ∧ r2 = {0, 1}

Average non-determinism of test (NDT)

Average number of races observed per memory operation (NDe) across alliterations of a test-run→ higher implies better test.

12 www.inf.ed.ac.uk

Page 26:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Test Suitability: What is a good MCM test?

init: x = 0, y = 0, z = 0Thread 1 Thread 2z← 1 r1← yy← 1 r2← xr1 = {0, 1} ∧ r2 = {0}

Average non-determinism of test (NDT)

Average number of races observed per memory operation (NDe) across alliterations of a test-run→ higher implies better test.

12 www.inf.ed.ac.uk

Page 27:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Selective Crossover

Crossover

• Crossover operator crucial in GP to create new program(s) from 2parents with good fitness: offspring likely to outperform (in fitness).

Goal: Encode Domain Knowledge

• Use knowledge of what makes good MCM test.

Crossover Algorithm

1 Obtain fitaddrs: addresses of operations where NDe is larger thanrounded NDT of test.

2 Select all operations with address in fitaddrs (discard others).

3 Overlay both parent tests, resulting in new test.

13 www.inf.ed.ac.uk

Page 28:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Evaluation Methodology

System setup

• Gem5 full-system simulator; x86-64 (TSO).

• Ruby memory simulator with Garnet interconnect model.

• 8 out-of-order cores.

• Private L1s; NUCA organization for L2.

Coherence Protocols

• MESI: Standard Ruby implementation.

• TSO-CC: Lazy coherence protocol for TSO [Elver et al., 2014].

14 www.inf.ed.ac.uk

Page 29:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Test Generators

McVerSi• Test size: 1000 operations across all threads.

• Iterations: 10 test executions per test-run.

• GP params: population size 100; tournament selection (size 2);mutation probability 0.005; crossover probability 1.0.

Random• Test size: 1000 ops

• Iterations: 10

Litmus tests (diy-litmus)

• Diy [Alglave et al., 2012] generated litmus tests for TSO (38 tests).

15 www.inf.ed.ac.uk

Page 30:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Results

Bugs found within 1 day 5 days 10 days

McVerSi (large mem.) 100% N/A N/ARandom (small mem.) 73% 73% 73%Random (large mem.) 55% 73% 82%diy-litmus 18% 45% 45%

Bugs found, when running up to the equivalent of 10 days time.

16 www.inf.ed.ac.uk

Page 31:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Results

Bug McVerSi(small mem.)

McVerSi(large mem.)

Random(small mem.)

Random(large mem.)

diy-litmus

MESI,LQ+IS,Inv1 10 (0.01) 10 (0.49) 10 (0.01) 10 (0.89) NFMESI,LQ+SM,Inv1 10 (0.33) 10 (5.20) 10 (0.48) NF NFMESI,LQ+E,Inv 10 (2.97) 10 (0.09) 10 (4.34) 10 (0.10) NFMESI,LQ+M,Inv 10 (1.42) 10 (1.37) 10 (1.93) 10 (11.05) NFMESI,LQ+S,Replace NF 10 (2.69) NF 6 (10.10) NFMESI+PUTX-Race2 NF 10 (4.64) NF 3 (9.63) NFMESI+Replace-Race NF 10 (0.12) NF 10 (0.19) 5 (0.53)TSO-CC+no-epoch-ids 10 (0.90) 10 (7.40) 10 (0.96) NF 6 (5.93)TSO-CC+compare 10 (0.01) 10 (2.28) 10 (0.01) 1 (22.31) 10 (0.92)LQ+no-TSO3 10 (0.00) 10 (0.03) 10 (0.00) 10 (0.08) 10 (5.35)SQ+no-FIFO 10 (0.01) 10 (0.24) 10 (0.01) 10 (0.40) 9 (4.77)All 80 (0.71) 110 (2.23) 80 (0.97) 70 (3.41) 40 (3.60)

Bug coverage: bug found count out of 10 samples (arith. mean hours).

1Found with McVerSi (patches submitted).2Komuravelli et al., “Revisiting the Complexity of Hardware Cache Coherence and Some Implications”,

In: TACO, 2014.3Found in 2013 via litmus testing, patch sent in March 2014; also independently discovered by [Lustig

et al., 2014].17 www.inf.ed.ac.uk

Page 32:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Results

Bug McVerSi(small mem.)

McVerSi(large mem.)

Random(small mem.)

Random(large mem.)

diy-litmus

MESI,LQ+IS,Inv1 10 (0.01) 10 (0.49) 10 (0.01) 10 (0.89) NFMESI,LQ+SM,Inv1 10 (0.33) 10 (5.20) 10 (0.48) NF NFMESI,LQ+E,Inv 10 (2.97) 10 (0.09) 10 (4.34) 10 (0.10) NFMESI,LQ+M,Inv 10 (1.42) 10 (1.37) 10 (1.93) 10 (11.05) NFMESI,LQ+S,Replace NF 10 (2.69) NF 6 (10.10) NFMESI+PUTX-Race2 NF 10 (4.64) NF 3 (9.63) NFMESI+Replace-Race NF 10 (0.12) NF 10 (0.19) 5 (0.53)TSO-CC+no-epoch-ids 10 (0.90) 10 (7.40) 10 (0.96) NF 6 (5.93)TSO-CC+compare 10 (0.01) 10 (2.28) 10 (0.01) 1 (22.31) 10 (0.92)LQ+no-TSO3 10 (0.00) 10 (0.03) 10 (0.00) 10 (0.08) 10 (5.35)SQ+no-FIFO 10 (0.01) 10 (0.24) 10 (0.01) 10 (0.40) 9 (4.77)All 80 (0.71) 110 (2.23) 80 (0.97) 70 (3.41) 40 (3.60)

Bug coverage: bug found count out of 10 samples (arith. mean hours).

1Found with McVerSi (patches submitted).2Komuravelli et al., “Revisiting the Complexity of Hardware Cache Coherence and Some Implications”,

In: TACO, 2014.3Found in 2013 via litmus testing, patch sent in March 2014; also independently discovered by [Lustig

et al., 2014].17 www.inf.ed.ac.uk

Page 33:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Results

Bug McVerSi(small mem.)

McVerSi(large mem.)

Random(small mem.)

Random(large mem.)

diy-litmus

MESI,LQ+IS,Inv1 10 (0.01) 10 (0.49) 10 (0.01) 10 (0.89) NFMESI,LQ+SM,Inv1 10 (0.33) 10 (5.20) 10 (0.48) NF NFMESI,LQ+E,Inv 10 (2.97) 10 (0.09) 10 (4.34) 10 (0.10) NFMESI,LQ+M,Inv 10 (1.42) 10 (1.37) 10 (1.93) 10 (11.05) NFMESI,LQ+S,Replace NF 10 (2.69) NF 6 (10.10) NFMESI+PUTX-Race2 NF 10 (4.64) NF 3 (9.63) NFMESI+Replace-Race NF 10 (0.12) NF 10 (0.19) 5 (0.53)TSO-CC+no-epoch-ids 10 (0.90) 10 (7.40) 10 (0.96) NF 6 (5.93)TSO-CC+compare 10 (0.01) 10 (2.28) 10 (0.01) 1 (22.31) 10 (0.92)LQ+no-TSO3 10 (0.00) 10 (0.03) 10 (0.00) 10 (0.08) 10 (5.35)SQ+no-FIFO 10 (0.01) 10 (0.24) 10 (0.01) 10 (0.40) 9 (4.77)All 80 (0.71) 110 (2.23) 80 (0.97) 70 (3.41) 40 (3.60)

Bug coverage: bug found count out of 10 samples (arith. mean hours).

1Found with McVerSi (patches submitted).2Komuravelli et al., “Revisiting the Complexity of Hardware Cache Coherence and Some Implications”,

In: TACO, 2014.3Found in 2013 via litmus testing, patch sent in March 2014; also independently discovered by [Lustig

et al., 2014].17 www.inf.ed.ac.uk

Page 34:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Results

Bug McVerSi(small mem.)

McVerSi(large mem.)

Random(small mem.)

Random(large mem.)

diy-litmus

MESI,LQ+IS,Inv1 10 (0.01) 10 (0.49) 10 (0.01) 10 (0.89) NFMESI,LQ+SM,Inv1 10 (0.33) 10 (5.20) 10 (0.48) NF NFMESI,LQ+E,Inv 10 (2.97) 10 (0.09) 10 (4.34) 10 (0.10) NFMESI,LQ+M,Inv 10 (1.42) 10 (1.37) 10 (1.93) 10 (11.05) NFMESI,LQ+S,Replace NF 10 (2.69) NF 6 (10.10) NFMESI+PUTX-Race2 NF 10 (4.64) NF 3 (9.63) NFMESI+Replace-Race NF 10 (0.12) NF 10 (0.19) 5 (0.53)TSO-CC+no-epoch-ids 10 (0.90) 10 (7.40) 10 (0.96) NF 6 (5.93)TSO-CC+compare 10 (0.01) 10 (2.28) 10 (0.01) 1 (22.31) 10 (0.92)LQ+no-TSO3 10 (0.00) 10 (0.03) 10 (0.00) 10 (0.08) 10 (5.35)SQ+no-FIFO 10 (0.01) 10 (0.24) 10 (0.01) 10 (0.40) 9 (4.77)All 80 (0.71) 110 (2.23) 80 (0.97) 70 (3.41) 40 (3.60)

Bug coverage: bug found count out of 10 samples (arith. mean hours).

1Found with McVerSi (patches submitted).2Komuravelli et al., “Revisiting the Complexity of Hardware Cache Coherence and Some Implications”,

In: TACO, 2014.3Found in 2013 via litmus testing, patch sent in March 2014; also independently discovered by [Lustig

et al., 2014].17 www.inf.ed.ac.uk

Page 35:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Conclusion

Problem

• How to rigorously verify memory consistency model in full systemfunctional design in simulation?

McVerSi: Evolutionary Approach (GP)

• Fitness: coverage (primary objective).

• Crossover: favoring operations from tests that contribute highly tonon-determinism/races (more likely to expose MCM bugs).

• Special guest-host interface to improve throughput (speed upconvergence).

• Found 2 new bugs in Gem5 (patches sent upstream).

• Standalone library available: https://github.com/melver/mc2lib

18 www.inf.ed.ac.uk

Page 36:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Crossover Example

T2 =b

T1 a= T1 =c

T1 =c

T2 =b

T2 =c T1 a=

T2 =a T1 =b

T1 b=

T2 c=

T2 =a

=ca=

=b=c

b= =a

T1 T2

Parent-1

{a,b}fitaddrs

=a=c

=ba=

=b c=

T1 T2

Parent-2

{a,c}fitaddrs

crossover_mutate

T2 =b T1 =c T1 a= T1 b= T2 =aT2 c=

T2 c=T1 b=T2 =aT1 =cT1 a= T1 a=

c=a=

=b=c

b= =a

T1 T2

Child-1

c==c

=aa=

a=

b=

T1 T2

Child-2

19 www.inf.ed.ac.uk

Page 37:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Crossover Example

T2 =b

T1 a= T1 =c

T1 =c

T2 =b

T2 =c T1 a=

T2 =a T1 =b

T1 b=

T2 c=

T2 =a

=ca=

=b=c

b= =a

T1 T2

Parent-1

{a,b}fitaddrs

=a=c

=ba=

=b c=

T1 T2

Parent-2

{a,c}fitaddrs

crossover_mutate

T2 =b T1 =c T1 a= T1 b= T2 =aT2 c=

T2 c=T1 b=T2 =aT1 =cT1 a= T1 a=

c=a=

=b=c

b= =a

T1 T2

Child-1

c==c

=aa=

a=

b=

T1 T2

Child-2

19 www.inf.ed.ac.uk

Page 38:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Crossover Example

T2 =b

T1 a= T1 =c

T1 =c

T2 =b

T2 =c T1 a=

T2 =a T1 =b

T1 b=

T2 c=

T2 =a

=ca=

=b=c

b= =a

T1 T2

Parent-1

{a,b}fitaddrs

=a=c

=ba=

=b c=

T1 T2

Parent-2

{a,c}fitaddrs

crossover_mutate

T2 =b T1 =c T1 a= T1 b= T2 =aT2 c=

T2 c=T1 b=T2 =aT1 =cT1 a= T1 a=

c=a=

=b=c

b= =a

T1 T2

Child-1

c==c

=aa=

a=

b=

T1 T2

Child-2

19 www.inf.ed.ac.uk

Page 39:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Crossover Example

T2 =b

T1 a= T1 =c

T1 =c

T2 =b

T2 =c T1 a=

T2 =a T1 =b

T1 b=

T2 c=

T2 =a

=ca=

=b=c

b= =a

T1 T2

Parent-1

{a,b}fitaddrs

=a=c

=ba=

=b c=

T1 T2

Parent-2

{a,c}fitaddrs

crossover_mutate

T2 =b T1 =c T1 a= T1 b= T2 =aT2 c=

T2 c=T1 b=T2 =aT1 =cT1 a= T1 a=

c=a=

=b=c

b= =a

T1 T2

Child-1

c==c

=aa=

a=

b=

T1 T2

Child-2

19 www.inf.ed.ac.uk

Page 40:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Crossover Example

T2 =b

T1 a= T1 =c

T1 =c

T2 =b

T2 =c T1 a=

T2 =a T1 =b

T1 b=

T2 c=

T2 =a

=ca=

=b=c

b= =a

T1 T2

Parent-1

{a,b}fitaddrs

=a=c

=ba=

=b c=

T1 T2

Parent-2

{a,c}fitaddrs

crossover_mutate

T2 =b T1 =c T1 a= T1 b= T2 =a

T2 c=

T2 c=T1 b=T2 =aT1 =cT1 a= T1 a=

c=a=

=b=c

b= =a

T1 T2

Child-1

c==c

=aa=

a=

b=

T1 T2

Child-2

19 www.inf.ed.ac.uk

Page 41:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Crossover Example

T2 =b

T1 a= T1 =c

T1 =c

T2 =b

T2 =c T1 a=

T2 =a T1 =b

T1 b=

T2 c=

T2 =a

=ca=

=b=c

b= =a

T1 T2

Parent-1

{a,b}fitaddrs

=a=c

=ba=

=b c=

T1 T2

Parent-2

{a,c}fitaddrs

crossover_mutate

T2 =b T1 =c T1 a= T1 b= T2 =aT2 c=

T2 c=T1 b=T2 =aT1 =cT1 a= T1 a=

c=a=

=b=c

b= =a

T1 T2

Child-1

c==c

=aa=

a=

b=

T1 T2

Child-2

19 www.inf.ed.ac.uk

Page 42:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Crossover Example

T2 =b

T1 a= T1 =c

T1 =c

T2 =b

T2 =c T1 a=

T2 =a T1 =b

T1 b=

T2 c=

T2 =a

=ca=

=b=c

b= =a

T1 T2

Parent-1

{a,b}fitaddrs

=a=c

=ba=

=b c=

T1 T2

Parent-2

{a,c}fitaddrs

crossover_mutate

T2 =b T1 =c T1 a= T1 b= T2 =aT2 c=

T2 c=T1 b=T2 =aT1 =cT1 a=

T1 a=

c=a=

=b=c

b= =a

T1 T2

Child-1

c==c

=aa=

a=

b=

T1 T2

Child-2

19 www.inf.ed.ac.uk

Page 43:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Crossover Example

T2 =b

T1 a= T1 =c

T1 =c

T2 =b

T2 =c T1 a=

T2 =a T1 =b

T1 b=

T2 c=

T2 =a

=ca=

=b=c

b= =a

T1 T2

Parent-1

{a,b}fitaddrs

=a=c

=ba=

=b c=

T1 T2

Parent-2

{a,c}fitaddrs

crossover_mutate

T2 =b T1 =c T1 a= T1 b= T2 =aT2 c=

T2 c=T1 b=T2 =aT1 =cT1 a= T1 a=

c=a=

=b=c

b= =a

T1 T2

Child-1

c==c

=aa=

a=

b=

T1 T2

Child-2

19 www.inf.ed.ac.uk

Page 44:  · Title: McVerSi: A Test Generation Framework for Fast Memory Consistency Verification in Simulation Author: Marco Elver, Vijay Nagarajan Created Date

Crossover Example

T2 =b

T1 a= T1 =c

T1 =c

T2 =b

T2 =c T1 a=

T2 =a T1 =b

T1 b=

T2 c=

T2 =a

=ca=

=b=c

b= =a

T1 T2

Parent-1

{a,b}fitaddrs

=a=c

=ba=

=b c=

T1 T2

Parent-2

{a,c}fitaddrs

crossover_mutate

T2 =b T1 =c T1 a= T1 b= T2 =aT2 c=

T2 c=T1 b=T2 =aT1 =cT1 a= T1 a=

c=a=

=b=c

b= =a

T1 T2

Child-1

c==c

=aa=

a=

b=

T1 T2

Child-2

19 www.inf.ed.ac.uk