Synthesis and Veriﬁcation for All - Racketcon.racket-lang.org/2016/torlak.pdf · Synthesis and Veriﬁcation for All. Emina ... (list out opcode in ...) (define op (eval opcode))

Emina TorlakUniversity of Washington

[email protected]

homes.cs.washington.edu/~emina/

Synthesis and Verification for All

http://homes.cs.washington.edu/~emina/

Emina

Racket

a programmable programming language

Racket

Racket



synthesis

verification ROSETTE


synthesis

verification ROSETTE

solver-aided

visiona little programming for everyone

A little programming for everyone

8

Every knowledge worker wants to program …

‣ spreadsheet data manipulation [Flashfill, POPL’11]


8


social scientist


‣ models of cell fates [SBL, POPL’13]


8


biologist social scientist



‣ cache coherence protocols [Transit, PLDI’13]

‣ memory models [MemSAT, PLDI’10]


8


hardware designer







8


hardware designer







8


hardware designer


time

expertise


9

solver-aided languages

We all want to build programs …

‣ spreadsheet data manipulation‣ models of cell fates‣ cache coherence protocols‣ memory models

hardware designer


less time

less expertise

outlinesolver-aided tools, languages and beyond

outlinesolver-aided tools, languages and beyondsolver-aided tools

outlinesolver-aided tools, languages and beyondsolver-aided tools, languages

outlinesolver-aided tools, languages and beyondsolver-aided tools, languages, and applications

toolssolver-aided tools

P(x) {……

}

Programming …

12

specification

P(x) {……

}

Programming …

12

specificationtest case

assert safe(P(2))

translator

P(x) {……

}assert safe(P(2))

Programming with a solver-aided tool

13

?

SAT/SMT solver

∃x . ¬ safe(P(x))

CBMC [Kroening et al., DAC’03]Dafny [Leino, LPAR’10]Miniatur [Vaziri et al., FSE’07]Klee [Cadar et al., OSDI’08]

P(x) {……

}assert safe(P(x))


13

Find an input on which the program fails.

? verify

42

SAT/SMT solver

BugAssist [Jose & Majumdar, PLDI’11]

P(x) {v = x + 2…

}assert safe(P(x))


13


Localize bad parts of the program.? verify debug

42

SAT/SMT solverx = 42 ⋀ safe(P(x))

Kaplan [Koksal et al, POPL’12]PBnJ [Samimi et al., ECOOP’10]Squander [Milicevic et al., ICSE’11]

P(x) {v = choose()…

}assert safe(P(x))


13


Localize bad parts of the program.

Find values that repair the failing run.? verify debugsolve

42 40

SAT/SMT solver∃v . safe(P(42, v))

Sketch [Solar-Lezama et al., ASPLOS’06]Comfusy [Kuncak et al., CAV’10]

P(x) {v = ??…

}assert safe(P(x))


13


Localize bad parts of the program.

Find values that repair the failing run.

Find code that repairs the program.? verify

debugsolvesynth

x − 2

SAT/SMT solver∃e . ∀x . safe(Pe(x))

translator

The standard (hard) way to build a tool

14

?P(x) {

……

}assert safe(P(x))

verify debugsolvesynth

expertise in PL, FM, SE

translator SAT/SMT solver

A new, easy way to build tools

15

programming

?P(x) {

……

}assert safe(P(x))


an interpreter or a library

ROSETTE


15

?P(x) {

……

}assert safe(P(x))



Implement a language for an application domain, get the tools for free!

ROSETTE


15

?P(x) {

……

}assert safe(P(x))




symbolic virtual machine

ROSETTE


15

?P(x) {

……

}assert safe(P(x))




symbolic virtual machine [Torlak & Bodik,

PLDI’14, Onward’13]

Hard technical challenge: how to efficiently translate a program and its interpreter?

designsolver-aided languages

domain-specific language (DSL)

Layers of languages

17

host language

A formal language that is specialized to a particular application domain and often limited in capability.

A high-level language for implementing DSLs, usually with meta-programming features.

interpreterlibrary

Scala, Racket, JavaScript


artificial intelligenceChurch, BLOG

databasesSQL, Datalog

hardware designBluespec, Chisel, Verilog, VHDL

math and statisticsEigen, Matlab, R

layout and visualizationLaTex, dot, dygraphs, D3

Layers of languages

17

host language

interpreterlibrary


Layers of languages

17

host language

C = A * B

C / Java

Eigen / Matlab

[associativity]C = A * B

for (i = 0; i < n; i++) for (j = 0; j < m; j++) for (k = 0; k < p; k++) C[i][k] += A[i][j] * B[j][k]

interpreterlibrary

solver-aided domain-specific language (SDSL)

Layers of solver-aided languages

18

solver-aided host language


interpreterlibrary

ROSETTE[Torlak & Bodik, Onward’13, PLDI’14]



18



interpreterlibrary

ROSETTE[Torlak & Bodik, Onward’13, PLDI’14]



18



interpreterlibrary

spatial programmingChlorophyll

intelligent tutoring RuleSynth

memory modelsMemSynth

optimal synthesisSynapse

radiotherapy controllersNeutrons

BGP router configurationsBagPipe

Anatomy of a solver-aided host language

19

Racket

(define-symbolic id type)

(assert expr)

(verify expr) (debug [expr] expr) (solve expr) (synthesize [expr] expr)

ROSETTE

Anatomy of a solver-aided host language

19

A tiny example SDSL

20

def bvmax(r0, r1) : r2 = bvge(r0, r1) r3 = bvneg(r2) r4 = bvxor(r0, r2) r5 = bvand(r3, r4) r6 = bvxor(r1, r5) return r6

BV: A tiny assembly-like language for writing fast, low-level library functions.

debugsynth

A tiny example SDSL

20



test verify

debugsynth

A tiny example SDSL

20



test verify

debugsynth

1. interpreter [10 LOC]

2. verifier [free]

3. debugger [free]

4. synthesizer [free]


> bvmax(-2, -1)

A tiny example SDSL:

21

ROSETTE


> bvmax(-2, -1)


21

(define bvmax `((2 bvge 0 1) (3 bvneg 2) (4 bvxor 0 2) (5 bvand 3 4) (6 bvxor 1 5)))

parse

ROSETTE


> bvmax(-2, -1)


21


parse

ROSETTE

(out opcode in ...)



> bvmax(-2, -1) -1

(define (interpret prog inputs) (make-registers prog inputs) (for ([stmt prog]) (match stmt [(list out opcode in ...) (define op (eval opcode)) (define args (map load in)) (store out (apply op args))])) (load (last)))


22

ROSETTE

interpret

(define bvmax `((2 bvge 0 1) (3 bvneg 2) (4 bvxor 0 2) (5 bvand 3 4) (6 bvxor 1 5))) `(-2 -1)


> bvmax(-2, -1) -1



22

ROSETTE

interpret


0 -21 -12 03 04 -25 06 -1


> bvmax(-2, -1) -1



22

ROSETTE

interpret


0 -21 -12 03 04 -25 06 -1

(2 bvge 0 1)


> bvmax(-2, -1) -1



22

ROSETTE

interpret


0 -21 -12 03 04 -25 06 -1

(2 bvge 0 1)


> bvmax(-2, -1) -1



22

ROSETTE

interpret


0 -21 -12 03 04 -25 06 -1

(2 bvge 0 1)


> bvmax(-2, -1) -1



22

ROSETTE

interpret


0 -21 -12 03 04 -25 06 -1

(2 bvge 0 1)


> bvmax(-2, -1) -1



22

ROSETTE

interpret


0 -21 -12 03 04 -25 06 -1

(2 bvge 0 1)(define bvmax `((2 bvge 0 1) (3 bvneg 2) (4 bvxor 0 2) (5 bvand 3 4) (6 bvxor 1 5)))


> bvmax(-2, -1) -1



22

ROSETTE

interpret


0 -21 -12 03 04 -25 06 -1

(2 bvge 0 1)(define bvmax `((2 bvge 0 1) (3 bvneg 2) (4 bvxor 0 2) (5 bvand 3 4) (6 bvxor 1 5)))


> bvmax(-2, -1) -1


23

ROSETTE


‣ pattern matching‣ dynamic evaluation‣ first-class &

higher-order procedures

‣ side effects


(define-symbolic n0 n1 integer?) (define inputs (list n0 n1)) (verify (assert (= (interpret bvmax inputs) (interpret max inputs))))


> verify(bvmax, max) (0, -2)

> bvmax(0, -2) -1


24

ROSETTE

query





> bvmax(0, -2) -1


24

ROSETTE

Creates two fresh symbolic constants of type number and binds them to variables n0 and n1.

query






> bvmax(0, -2) -1


24

ROSETTE

Symbolic values can be used just like concrete values of the same type.

query







> bvmax(0, -2) -1


24

ROSETTE

(verify expr) searches for a concrete interpretation of symbolic constants that causes expr to fail.

query







> bvmax(0, -2) -1


24

ROSETTE

query

(define inputs (list 0 -2)) (debug [input-register?] (assert (= (interpret bvmax inputs) (interpret max inputs))))


> debug(bvmax, max, (0, -2))


25

ROSETTE

query

(define inputs (list 0 -2)) (debug [input-register?] (assert (= (interpret bvmax inputs) (interpret max inputs))))


> debug(bvmax, max, (0, -2))


25

ROSETTE

query


(define-symbolic n0 n1 integer?) (define inputs (list n0 n1)) (synthesize [inputs] (assert (= (interpret bvmax inputs) (interpret max inputs)))))

def bvmax(r0, r1) : r2 = bvge(r0, r1) r3 = bvneg(r2) r4 = bvxor(??, ??) r5 = bvand(r3, ??) r6 = bvxor(??, ??) return r6

> synthesize(bvmax, max)


26

ROSETTE

query

(define-symbolic n0 n1 integer?) (define inputs (list n0 n1)) (synthesize [inputs] (assert (= (interpret bvmax inputs) (interpret max inputs)))))

def bvmax(r0, r1) : r2 = bvge(r0, r1) r3 = bvneg(r2) r4 = bvxor(??, ??) r5 = bvand(r3, ??) r6 = bvxor(??, ??) return r6

> synthesize(bvmax, max)



26

ROSETTE

query

techsymbolic virtual machine (SVM)

ROSETTE

How it all works: a big picture view

28

symbolicvirtual machine

SDSL

program

query

solver

[Torlak & Bodik, Onward’13]

[Torlak & Bodik, PLDI’14]

ROSETTE


28


SDSL

program

result

solver



ROSETTE


28


SDSL

program

result

solver

‣ pattern matching‣ dynamic evaluation‣ first-class procedures ‣ higher-order procedures‣ side effects‣ macros

theories of bitvectors, integers, reals, and uninterpreted functions



(3, 1, -2) (1, 3)

Translation to constraints by example

29

solve: ps = () for v in vs: if v > 0: ps = insert(v, ps) assert len(ps) == len(vs)

reverse and filter, keeping only positive numbers

vs ps

(3, 1, -2) (1, 3)


29


vs ps


29


vs psconstraints

(a, b)


29


vs psconstraints

a>0 ∧ b>0 (a, b)


29


vs psconstraints

Design space of precise symbolic encodings

30

symbolic execution

bounded model checkingsolve: ps = () for v in vs: if v > 0: ps = insert(v, ps) assert len(ps) == len(vs)

{ }a > 0b ≤ 0false


30

a > 0

b ≤ 0

ps ↦ (a)

ps ↦ (a)

vs ↦ (a, b)ps ↦ ( )

symbolic execution


{ }a > 0b ≤ 0false

∨ ∨ ∨


30

a > 0

b ≤ 0

ps ↦ (a)

ps ↦ (a)

vs ↦ (a, b)ps ↦ ( )

b > 0b > 0

ps ↦ (b) ps ↦ (b, a)

{ }a ≤ 0b > 0false

{ }a > 0b > 0true

a ≤ 0

b ≤ 0

ps ↦ ( )

ps ↦ ( )

{ }a ≤ 0b ≤ 0false

symbolic execution


{ }a > 0b ≤ 0false

∨ ∨ ∨


30

vs ↦ (a, b)ps ↦ ( )

ps ↦ (a)

ps ↦ ps0

a > 0a ≤ 0

ps0 = ite(a > 0, (a), ( ))ps1 = insert(b, ps0)ps2 = ite(b > 0, ps0, ps1)assert len(ps2) = 2

a > 0

b ≤ 0

ps ↦ (a)

ps ↦ (a)

vs ↦ (a, b)ps ↦ ( )

b > 0b > 0

ps ↦ (b) ps ↦ (b, a)

{ }a ≤ 0b > 0false

{ }a > 0b > 0true

a ≤ 0

b ≤ 0

ps ↦ ( )

ps ↦ ( )

{ }a ≤ 0b ≤ 0false

symbolic execution

bounded model checking

ps ↦ ( )


{ }a > 0b ≤ 0false

∨ ∨ ∨


30

vs ↦ (a, b)ps ↦ ( )

ps ↦ (a)

ps ↦ ps0

a > 0a ≤ 0


a > 0

b ≤ 0

ps ↦ (a)

ps ↦ (a)

vs ↦ (a, b)ps ↦ ( )

b > 0b > 0

ps ↦ (b) ps ↦ (b, a)

{ }a ≤ 0b > 0false

{ }a > 0b > 0true

a ≤ 0

b ≤ 0

ps ↦ ( )

ps ↦ ( )

{ }a ≤ 0b ≤ 0false

symbolic execution


ps ↦ ( )

ps ↦ ps1

b > 0


{ }a > 0b ≤ 0false

∨ ∨ ∨


30

vs ↦ (a, b)ps ↦ ( )

ps ↦ (a)

ps ↦ ps0

a > 0a ≤ 0


a > 0

b ≤ 0

ps ↦ (a)

ps ↦ (a)

vs ↦ (a, b)ps ↦ ( )

b > 0b > 0

ps ↦ (b) ps ↦ (b, a)

{ }a ≤ 0b > 0false

{ }a > 0b > 0true

a ≤ 0

b ≤ 0

ps ↦ ( )

ps ↦ ( )

{ }a ≤ 0b ≤ 0false

symbolic execution


ps ↦ ( )

ps ↦ ps1

ps ↦ ps2

b > 0b ≤ 0

ps ↦ ps0


A new design: type-driven state merging

31

{ }a > 0b > 0true


Merge values of ‣ primitive types: symbolically‣ immutable types: structurally‣ all other types: via unions


31

{ }a > 0b > 0true



ba


31

{ }a > 0b > 0true


⁊g g

c


ba (c, d)(a, b)


31

{ }a > 0b > 0true


⁊g g

c(e, f)


ba (c, d)


31

{ }a > 0b > 0true


⁊g g

c(e, f){ ¬g ⊦ a, g ⊦ () }

()



32



a > 0a ≤ 0


32

vs ↦ (a, b)ps ↦ ( )

ps ↦ (a)ps ↦ ( )



Symbolic union: a set of guarded values, with disjoint guards.

g0 = a > 0g1 = b > 0 g2 = g0 ∧ g1

g3 = ¬(g0 ⇔ g1)g4 = ¬g0 ∧ ¬g1

c = ite(g1, b, a)assert g2

a > 0a ≤ 0 g0¬ g0


32

vs ↦ (a, b)ps ↦ ( )

ps ↦ (a)ps ↦ ( )


ps ↦ { g0 ⊦ (a), ¬g0 ⊦ ( ) }


Execute insert concretely on all lists in the union.

g0 = a > 0g1 = b > 0 g2 = g0 ∧ g1

g3 = ¬(g0 ⇔ g1)g4 = ¬g0 ∧ ¬g1


a > 0a ≤ 0

g1

g0¬ g0


32

vs ↦ (a, b)ps ↦ ( )

ps ↦ (a)ps ↦ ( )


ps ↦ { g0 ⊦ (a), ¬g0 ⊦ ( ) }

ps ↦ { g0 ⊦ (b, a), ¬g0 ⊦ (b) }


g0 = a > 0g1 = b > 0 g2 = g0 ∧ g1

g3 = ¬(g0 ⇔ g1)g4 = ¬g0 ∧ ¬g1


a > 0a ≤ 0

¬ g1 g1

g0¬ g0


32

vs ↦ (a, b)ps ↦ ( )

ps ↦ (a)ps ↦ ( )


ps ↦ { g0 ⊦ (a), ¬g0 ⊦ ( ) }

ps ↦ { g0 ⊦ (b, a), ¬g0 ⊦ (b) }

ps ↦ { g0 ⊦ (a), ¬g0 ⊦ ( ) }


g0 = a > 0g1 = b > 0 g2 = g0 ∧ g1

g3 = ¬(g0 ⇔ g1)g4 = ¬g0 ∧ ¬g1


a > 0a ≤ 0

¬ g1 g1

g0¬ g0


32

vs ↦ (a, b)ps ↦ ( )

ps ↦ (a)ps ↦ ( )


ps ↦ { g0 ⊦ (a), ¬g0 ⊦ ( ) }

ps ↦ { g0 ⊦ (b, a), ¬g0 ⊦ (b) }

ps ↦ { g0 ⊦ (a), ¬g0 ⊦ ( ) }

ps ↦ { g2 ⊦ (b, a), g3 ⊦ (c), g4 ⊦ ( ) }


Evaluate len concretely on all lists in the union; assertion true only on the list guarded by g2.

g0 = a > 0g1 = b > 0 g2 = g0 ∧ g1

g3 = ¬(g0 ⇔ g1)g4 = ¬g0 ∧ ¬g1


a > 0a ≤ 0

¬ g1 g1

g0¬ g0


32

vs ↦ (a, b)ps ↦ ( )

ps ↦ (a)ps ↦ ( )


ps ↦ { g0 ⊦ (a), ¬g0 ⊦ ( ) }

ps ↦ { g0 ⊦ (b, a), ¬g0 ⊦ (b) }

ps ↦ { g0 ⊦ (a), ¬g0 ⊦ ( ) }

ps ↦ { g2 ⊦ (b, a), g3 ⊦ (c), g4 ⊦ ( ) }


g0 = a > 0g1 = b > 0 g2 = g0 ∧ g1

g3 = ¬(g0 ⇔ g1)g4 = ¬g0 ∧ ¬g1


a > 0a ≤ 0

¬ g1 g1

g0¬ g0


32

vs ↦ (a, b)ps ↦ ( )

ps ↦ (a)ps ↦ ( )


ps ↦ { g0 ⊦ (a), ¬g0 ⊦ ( ) }

ps ↦ { g0 ⊦ (b, a), ¬g0 ⊦ (b) }

ps ↦ { g0 ⊦ (a), ¬g0 ⊦ ( ) }

ps ↦ { g2 ⊦ (b, a), g3 ⊦ (c), g4 ⊦ ( ) }

concrete evaluation

polynomial encoding

appssolver-aided programming for everyone

Investigating Safety of a Radiotherapy Machine (CAV’16)Stuart Pernsteiner, Calvin Loncaric, Emina Torlak, Zachary Tatlock, Xi Wang, Michael Ernst, and Jon Jacky

Scaling up Superoptimization (ASPLOS’16)Phitchaya Mangpo Phothilimthana, Aditya Thakur, Rastislav Bodík, and Dinakar Dhurjati.

Recent applications of ROSETTE

34

Synthesizing Memory Models from Litmus TestsJames Bornholt and Emina Torlak (under submission)

Synthesizing Custom Tutoring Rules for Introductory AlgebraEric Butler, Emina Torlak, and Zoran Popovic (under submission)

Scalable Verification of BGP Configurations (OOPSLA’16)Konstantin Weitz, Doug Woos, Emina Torlak, Michael D. Ernst, Arvind Krishnamurthy, and Zachary Tatlock

Investigating Safety of a Radiotherapy Machine (CAV’16)Stuart Pernsteiner, Calvin Loncaric, Emina Torlak, Zachary Tatlock, Xi Wang, Michael Ernst, and Jon Jacky

A Framework for Synthesis and Parameterized Design of Rule Systems Applied to Algebra (ITS’16)

Eric Butler, Emina Torlak, and Zoran Popovic

Specifying and Checking File System Crash-Consistency Models (ASPLOS’16)

James Bornholt, Antoine Kaufmann, Jialin Li, Arvind Krishnamurthy, Emina Torlak, and Xi Wang.

Scaling up Superoptimization (ASPLOS’16)Phitchaya Mangpo Phothilimthana, Aditya Thakur, Rastislav Bodík, and Dinakar Dhurjati.

Synapse: Optimizing Synthesis with Metasketches (POPL’16)James Bornholt, Emina Torlak, Dan Grossman, and Luis Ceze.

…

Investigating Safety of a Radiotherapy Machine (CAV’16)

Stuart Pernsteiner, Calvin Loncaric, Emina Torlak, Zachary Tatlock, Xi Wang, Michael Ernst, and Jon Jacky

Scaling up Superoptimization (ASPLOS’16)

Phitchaya Mangpo Phothilimthana, Aditya Thakur, Rastislav Bodík, and Dinakar Dhurjati.

35

Recent applications of ROSETTE

36

Clinical Neutron Therapy System (CNTS) at UW

Recent applications of ROSETTE: Neutrons





• Used Rosette to build a verifier for the EPICS DSL.

• Found safety-critical defects in a pre-release version of the therapy control software.

• Used by CNTS staff to verify changes to the control software.





37

Recent applications of ROSETTE: Enlearn

• Education technology company founded by Zoran Popovic.

• Uses Rosette to develop educational math games, released to thousands (and eventually millions) of students.

• Building a Rosette-based DSL to enable educators to develop their own educational tools.





38

Recent applications of ROSETTE: Greenthumb

• A Rosette-based framework for creating efficient superoptimizers for new ISAs.

• Requires only an emulator for the ISA and a few ISA-specific search utility functions!

• Hosts superoptimizers for ARM and GreenArrays (GA) ISA.

• Up to 82% speedup over gcc -O3 for ARM; within 19% of hand optimized code for GA.

thanks

thanksROSETTE

your SDSL


synthesize

debug

solve

verify

Documents

Synthesis and Veriﬁcation for All - Racketcon.racket-lang.org/2016/torlak.pdf · Synthesis and Veriﬁcation for All. Emina ... (list out opcode in ...) (define op (eval opcode))