Michael Bond Katherine Coons Kathryn McKinley University of Texas at Austin

Pacer: Proportional Detection of Data RacesMichael BondKatherine CoonsKathryn McKinleyUniversity of Texas at Austin

Detecting data races in production

Overhead

FastTrack[Flanagan & Freund ’09]

80x 8x

Overhead

creads&writes + csync n

Number of threads

Overhead

Problemin future

Problemtoday

Overhead

Pacer (creads&writes + csync n) r + cnon-sampling (1 – r)

Sampling rate

Overhead

Sampling periods Non-sampling periods

Overhead

Probability (detecting any race)

FastTrack 1

Pacer r

Detect race first access sampled

Sampling period

Thread A Thread B

Non-sampling period

Sampling period

Non-sampling period

Thread A Thread B

write x

read x

read y

write y

Insight #1:Stop tracking variable after

non-sampled access

Thread A

write x

unlock m

Thread B

Thread A

write x

unlock m

Thread B

lock m

Thread A

write x

unlock m

Thread B

lock m

write x

Thread A

write x

unlock m

read x

Thread B

lock m

write x

Thread A

write x

unlock m

read x

Thread B

lock m

write xRace!

Thread A

write x

unlock m

read x

Thread B

lock m

write xRace!

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4A B A B

Vector clocks

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4A B A B

Vector clocks

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4A B A B

Vector clocks

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4A B A B

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4A B A B

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4

A B A B

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4

6 25 2

Incrementclock

A B A B

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4

Joinclocks

A B A B

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4

Happens before?

A B A B

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4

A B A B

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4

A B A B

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4

No work performed

A B A B

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4

Race uncaught

A B A B

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4

A B A B

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4

Happens before?Race!

A B A B

Insight #2: We only care whether“A happens before B”

if A is sampled

Thread A Thread B

Do these events happen before other events?We don’t care!

Increment clocks

Thread A Thread B

Don’t increment clocks

Increment clocks

Don’t increment clocks

Do these events happen before other events?We don’t care!

Thread A

unlock m1

unlock m2

Thread B

lock m1

lock m2

5 2 3 4A B A B

Thread A

unlock m1

unlock m2

Thread B

lock m1

lock m2

5 2 3 4

No clock increment

A B A B

Thread A

unlock m1

unlock m2

Thread B

lock m1

lock m2

5 2 3 4

A B A B

Thread A

unlock m1

unlock m2

Thread B

lock m1

lock m2

5 2 3 4

Unnecessary join

A B A B

Thread A

unlock m1

unlock m2

Thread B

lock m1

lock m2

5 2 3 4

O(n) O(1)

A B A B

Implementation

http://jikesrvm.org/Research+Archive

Performance

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%0

eclipsehsqldbxalanpseudojbb

Sampling rate

Performance

Qualitative improvementin time & space

Accuracy

Probability (detecting any race) = r?

Per-Race Accuracy(eclipse, r = 1%)

Distinct races (ordered by detection rate)

Related Work

LiteRace [Marino et al. ’09]

Cold-region hypothesis [Chilimbi & Hauswirth ’04]

Full analysis at synchronization operations

Deployable Race Detection

Accuracy, time, space sampling rateDetect race first access sampled

Qualitative improvement

Qualitative improvementHelp developers fix difficult-to-reproduce bugs

Thank you

Backup

Thread A

unlock m1

unlock m2

Thread B

lock m1

lock m2

5 2 3 4

A B A B

Example: “Timeless” Non-Sampling Periods

Vector clock versions

Thread A

unlock m1

unlock m2

Thread B

lock m1

lock m2

5 2 3 4

A B A B

5 2 v6

Thread A

unlock m1

unlock m2

Thread B

lock m1

lock m2

5 2 3 4A B A B

5 2 v6

Join unnecessary

Per-Race Accuracy (Eclipse)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 270%

r = 25%r = 10%r = 5%r = 3%r = 1%

Distinct races (each line sorted by detection rate)

Space Performance (eclipse)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

r=100%r=25%r=5%r=1%Base

Fraction of program execution

Performance (0-10% sampling rate)

0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10%0

Sampling rate

0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10%0

Sampling rate

33% base overhead

52% over-head

0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10%0

Sampling rate

33% base overhead

52% over-head

Qualitative improvement

Methodology

Core 2 Quad (4 cores) Multithreaded benchmarks (DaCapo & SPECjbb2000)

Evaluating sampling-based race detection Need 100s of trials to evaluate Some races are rare Evaluate only frequent races

Data Races

Two accesses to same variable (one is a write)

One access doesn’t happen before the other Program order Synchronization order▪ Acquire-release▪ Wait-notify▪ Fork-join▪ Volatile read-write

Data Races

Thread A

write x

unlock m

Thread B Two accesses to same variable (one is a write)

Data Races

Thread A

write x

unlock m

Thread B

lock m

write x

Data Races

Thread A

write x

unlock m

read x

Thread B

lock m

write x

Data Races

Thread A

write x

unlock m

read x

Thread B

lock m

write xRace!

Why Do We Care?

Races indicate Atomicity violations Order violations

Why Do We Care?

Races lead to Sequential consistency violations

No races sequential consistency (Java/C++) Races writes observed out of order

Why Do We Care?

Races lead to Sequential consistency violations

No races sequential consistency (Java/C++) Races writes observed out of order

Most races potentially harmful [Flanagan & Freund ’10]

Producer-Consumer Example

class ProducerConsumer { boolean ready; int x;

produce() { x = … ; ready = true; }

consume() { while (!ready) { } … = x; }}

Does It Race?

class ProducerConsumer { boolean ready; int x; T1 T2 produce() { x = … ; ready = true; }

Does It Race?

So What?

Can read old value

So What?

consume() { … = x; while (!ready) { } }}

Legal reordering by compiler or hardware

How to Fix?

Properly Synchronized

class ProducerConsumer { volatile boolean ready; int x; T1 T2 produce() { x = … ; ready = true; }

Happens- before edge

Example #2

class LibraryBook { Set<Person> borrowers;}

Initialization on Demand

class LibraryBook { Set<Person> borrowers;

addBorrower(Person p) { if (borrowers == null) { borrowers = new HashSet<Person>(); } borrowers.add(p); }}

Synchronized but Slow?

addBorrower(Person p) { synchronized (this) { if (borrowers == null) { borrowers = new HashSet<Person>(); } } borrowers.add(p); }}

Double-Checked Locking

addBorrower(Person p) { if (borrowers == null) { synchronized (this) { if (borrowers == null) { borrowers = new HashSet<Person>(); } } } borrowers.add(p); }}

Does It Race?

addBorrower(Person p) { if (borrowers == null) { synchronized (this) { if (borrowers == null) { borrowers = new HashSet<Person>(); } } } borrowers.add(p); }}

Does It Race?

addBorrower(Person p) { if (borrowers == null) { synchronized (this) { if (borrowers == null) { borrowers = new HashSet(); } } }

borrowers.add(p);}

addBorrower(Person p) {

if (borrowers == null) { ...

borrowers.add(p);

So What?

addBorrower(Person p) { if (borrowers == null) { synchronized (this) { if (borrowers == null) { HashSet obj = alloc HashSet; obj.<init>(); borrowers = obj; } } }

borrowers.add(p);}

borrowers.add(p);

So What?

addBorrower(Person p) { if (borrowers == null) { synchronized (this) { if (borrowers == null) { HashSet obj = alloc HashSet; borrowers = obj; obj.<init>(); } } }

borrowers.add(p);}

borrowers.add(p);

So What?

addBorrower(Person p) { if (borrowers == null) { synchronized (this) { if (borrowers == null) { HashSet obj = alloc HashSet; borrowers = obj;

obj.<init>(); }}} ... borrowers.add(p);}

borrowers.add(p);

Performance vs. Accuracy

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%0

Detection rate ( sampling rate)

Performance vs. Accuracy

-1% 1% 3% 5% 7% 9% 11% 13% 15%0

Detection rate ( sampling rate)

33% base overhead

~50% overhead

Accuracy & Performance

Program alone FastTrack Pacer

Detection rate 0 occurrence rate occurrence rate × r

Running time t t(c1 + c2n) t[(c1 + c2n)r + c3]

Evaluate only frequent races Evaluate scaling with r Don’t evaluate scaling with n

Northeast Blackout of 2003

50 million people

Energy Management System Alarm and Event Processing Routine (1 MLOC)

http://www.securityfocus.com/news/8412

Energy Management System Alarm and Event Processing Routine (1 MLOC)

Post-mortem analysis: 8 weeks"This fault was so deeply embedded, it took them weeks of poring through millions of lines of code and data to find it.” –Ralph DiNicola, FirstEnergy

Race condition Two threads writing to data structure simultaneously

Usually occurs without error Small window for causing data corruption

Tracks happens-before: sound & precise 80X slowdown Each analysis step: O(n) time (n = # of threads)

Vector Clock-Based Race Detection

FastTrack [Flanagan & Freund ’09] Reads & writes (97%): O(1) time Synchronization (3%): O(n) time 8X slowdown

Problem today

Problem in future

Thread A Thread B

5 2 3 4A B A B

Vector clocks

Thread A Thread B

5 2 3 4A B A B

Vector clocks

Thread A’s logical time Thread B’s logical time

Thread A Thread B

5 2 3 4A B A B

Vector clocks

Last logical time “received” from B

Last logical time “received” from A

5 2 3 4A B A B

Thread A

unlock m

Thread B

lock m6 2Increment

5 2 3 4A B A B

Thread A

unlock m

Thread B

lock m6 2

Joinclocks

5 2 3 4A B A B

Thread A

unlock m

Thread B

lock m6 2

5 4 n = # of threads

O(n) time

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4A B A B

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4A B A B

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4A B A B

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4A B A B

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4A B A B

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4A B A B

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4A B A B

6 2Happens before?5 2

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4A B A B

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4A B A B

Happens before?

Thread A

write x

unlock m

read x

Thread B

lock m

write x

5 2 3 4A B A B

Happens before?

4@BRace!

Detection rate occurrence rate occurrence rate × r

Prior Work Isn’t Deployable

Sampling rate

Running time t(c1 + c2n)

(Theoretical) Accuracy & Performance

No. of threads

Reads & writes Synchronization

Reads & writes

Problem today Problem in future

Synchronization

Running time t(c1 + c2n) t[(c1 + c2n)r + c3]

Overhead in sampling periods

Running time t(c1 + c2n) t[(c1 + c2n)r + c3]

Overhead in sampling periods

Overhead in non-sampling periods (small)

Detecting Data Races in Production

Data race occurs extremely rarely

Data race occurs periodically

Pre-deployment Deployed

“We test exhaustively … we had in excess of three million online operational hours [342 years] in which

nothing had ever exercised that bug.”–Mike Unum, manager of commercial solutions, GE Energy

Data race buggy execution

Michael Bond Katherine Coons Kathryn McKinley University of Texas at Austin

Documents

The Yin and Yang of Hardware Heterogeneity: Can Software Survive? Kathryn S McKinley

1 The Job Search Ellen Spertus Mills College Kathryn McKinley University of Texas at Austin Kathryn McKinley University of Texas at Austin

Networking New and Sustaining Professional Relationships Kathryn S McKinley, Microsoft Research

P ath & E dge P rofiling Michael Bond, UT Austin Kathryn McKinley, UT Austin Continuous Presented by: Yingyi Bu

Michael Bond Kathryn McKinley The University of Texas at Austin

Networking New and Sustaining Professional Relationships Kathryn S McKinley, Microsoft Research CRA-W co-chair

Systematic Editing: Generating Program Transformations from an Example Na Meng Miryung Kim Kathryn S. McKinley The University of Texas at Austin

Michael Bond Varun Srivastava Kathryn McKinley Vitaly Shmatikov University of Texas at Austin

Polar Opposites: Next Generation Languages & Architectures Kathryn S McKinley The University of Texas at Austin

Bell: Bit-Encoding Online Memory Leak Detection Michael D. Bond Kathryn S. McKinley University of Texas at Austin

Beltway: Getting Around GC Gridlock Steve Blackburn, Kathryn McKinley Richard Jones, Eliot Moss

Luca’s spreekbeurt over Maine coons

Fast Garbage Collection without a Long Wait Steve Blackburn – Kathryn McKinley Presented by: Na Meng Ulterior Reference Counting:

Ulterior Reference Counting: Fast Garbage Collection without a Long Wait Author: Stephen M Blackburn Kathryn S McKinley Presenter: Jun Tao

Hosted By Michelle Coons

1 Fast and Efficient Partial Code Reordering Xianglong Huang (UT Austin, Adverplex) Stephen M. Blackburn (Intel) David Grove (IBM) Kathryn McKinley (UT

Practical Path Profiling for Dynamic Optimizers Michael Bond, UT Austin Kathryn McKinley, UT Austin

Michael coons visual resume

University of Washingtonbornholt/papers/estimates... · 2019-08-15 · University of Washington. Kathryn McKinley Matthias Felleisen Shan Lu Steve Blackburn Michael Carbin Matthew

Jerrica Schreib And Donald Coons