Program Transformationwith Stratego/XT

— rules, strategies, tools, systems —

Eelco Visser

Center for Software Technology, Utrecht University, Utrecht, The Netherlands

visser@cs.uu.nl, http://www.cs.uu.nl/~visser

Program Transformation 2003

1

Stratego/XT

Ambition

• framework for specification of wide range of transformations

Stratego

• language for program transformation

• based on paradigm of programmable rewriting strategies

XT

• a bundle of transformation tools

• infrastructure for constructing/generating transformation systems

• parsing, pretty-printing, interoperability

XT Orbit: expanding with packages for specific languages

• xml-tools, java-front, c-front, stratego-doc, ...

2

Applications

• Compilers

– translation– desugaring– instruction selection∗ maximal munch, burg-style dynamic programming

– optimization∗ data-flow optimizations, vectorization∗ ghc-style simplification∗ deforestation∗ domain-specific optimization (codeboost)∗ partial evaluation

– typechecking– specialization of dynamic typing

3

Applications

• Program generators

– pretty-printer and signature generation from syntax defini-tions

– application generation from DSL (Java, C++)

• Program migration

– grammar conversion (yacc to sdf, ...)

• Program understanding

– documentation generation

• Document generation/transformation

– Web/XML programming (server-side ‘scripts’)

4

Abstraction Levels

Composition of transformation systems at all abstraction levels

• Rule

– basic transformation step

• Strategy

– controls application of a set of rules

• Tool

– complete transformation on abstract syntax tree

– standalone binary component

• Transformation system

– complete ‘source-to-source’ transformation system

– composition of (multiple) transformation tools with parser and pretty-printer

5

Abstraction Levels

Some examples

• Rule

– Constant folding rule

• Strategy

– Simplifier algorithm applying constant folding, inlining, etc.

• Tool

– Simplifier for whole program

• Transformation system

– Compiler translating high-level source to target, applying sim-plifier and other optimization tools

6

Abstract Syntax of Programs

textually represented programs

f(a + 10) - 3

are turned into abstract syntax tree by parser

Minus

Call Int

Var [] "3"

"f" Plus

Var Int

"a" "10"

abstract syntax trees are equivalent to terms

Minus(Call(Var("f"),[Plus(Var("a"),Int("10"))]),Int("3"))

7

Transformation Rules

EvalBinOp :Plus(Int(i), Int(j)) -> Int(k)where <add>(i, j) => k

LetSplit :Let([d1, d2 | d*], e*) ->Let([d1], Let([d2 | d*], e*))

rules express basic (local) modifications of programs

8

Concrete Object Syntax

EvalBinOp :|[ i + j ]| -> |[ k ]|where <add>(i, j) => k

LetSplit :|[ let d1 d2 d* in e* end ]| ->|[ let d1

in let d2 d* in e* endend ]|

the use of the concrete syntax of the object language decreases thegap between meta-program and object program

9

Concrete Object Syntax

TraceFunction :|[ function f(x*) : tid = e ]| ->|[ function f(x*) : tid =

(enterfun(s);let var x : tidin x := e;

exitfun(s);x

end) ]|where new => x ; !f => s

especially when program fragments become larger

10

Transformation Strategies

Strategies control the application of rules

constfold = bottomup(try(EvalPlus + EvalMul + ... ))dnf = innermost(DAOL + DAOR + DN + DMA + DMO)

Strategies are programmable using combinators for control

try(s) = s <+ idrepeat(s) = try(s; repeat(s))

and (generic) traversal

topdown(s) = s; all(topdown(s))bottomup(s) = all(bottomup(s)); salltd(s) = s <+ all(alltd(s))oncetd(s) = s <+ one(oncetd(s))

to avoid overhead of tree traversal

11

Transformation Idioms

Control over strategy admits wide variety of transformation idioms

• Cascading transformations (rewriting, exhaustive application)

simplify = innermost(R1 <+ ... <+ Rn)innermost(s) = bottomup(try(s; innermost(s)))

• Staged transformations (sequence of normal forms)

simplify =innermost(A1 <+ ... <+ Ak); innermost(B1 <+ ... <+ Bl); ...; innermost(C1 <+ ... <+ Cm)

12

Transformation Idioms

• ‘Local’ transformations

transformation =alltd(

trigger-transformation; innermost(A1 <+ ... <+ An)

)alltd(s) = s <+ all(alltd(s))

• Application-specific transformations

• Recursive patterns

• Dynamic programming

• ...

13

First-Class Pattern Matching

• Distinction between rules and strategies is only methodological

• Rules

L : t1 -> t2 where s

are just syntactic sugar for strategies

L = ?t1; where(s); !t2

• Conclusion: composite transformations (e.g., instantiated strate-gies) can be used as rules

14

Scoped Dynamic Rewrite Rules

InlineFun :|[ f(a*) ]| -> |[ let d* in e2 end ]|where |[ function f(x2*) ta = e2 ]|

; <zip(BindVar)>(x2*, a*) => d*

BindVar :(FArg |[ x ta ]|, e) -> |[ var x ta := e ]|

rewrite rules are context-free

15

Scoped Dynamic Rewrite Rules

DeclareFun =?fdec@|[ function f(x1*) ta = e1 ]|;rules(

InlineFun :|[ f(a*) ]| -> |[ let d* in e2 end ]|where <rename>fdec => |[ function f(x2*) ta = e2 ]|

; <zip(BindVar)>(x2*, a*) => d*)

BindVar :(FArg |[ x ta ]|, e) -> |[ var x ta := e ]|

dynamic rules support the run-time generation context-sensitive rewriterules

16

Transformation Tools

main = iowrap(simplify-options, simplify)

simplify-options =ArgOption("-O", where(<set-config> ("-O", <id>)), !"-O n Set optimization level (1 default)")

simplify = ...

tool makes strategy into deployable transformation

• handle command line options

• read input term, write output term

17

ATerm Exchange Format

• The Annotated Term (ATerm) Format is a language independentformat for exchange of structured data between tools

• Stratego terms are equivalent to ATerms

• Term under transformation can easily be exchanged with othertransformation tools

• Transformation systems can be divided into smaller, reusabletools

• ⇒ open compilers

18

XTC: Transformation Tool Composition

• Composition of transformation tools using conventional means(make, shell scripts) is awkward

– where is the tool installed?

– no possibility to deal with term

• XTC is a light-weight composition model for tool composition

– repository-based registration and search of tool locations

– library of abstractions for calling tools

– apply tool as basic transformation step

19

XTC Example

io-tiger-pe =xtc-io-wrap(tiger-pe-options,

parse-tiger; tiger-desugar; tiger-partial-eval; if-switch(!"elim-dead", tiger-elim-dead); if-switch(!"ensugar", tiger-ensugar); if-switch(!"pp", pp-tiger))

tiger-partial-eval =xtc-transform(!"Tiger-Partial-Eval", pass-verbose)

...

20

Transformation Services

XTC scales naturally to a web service model

io-calc-client =xtc-io-wrap(

xtc-http-transform(!URL("http://127.0.0.1/cgi-bin/calculator")

))

Applications?

• Provide optimization/generation service without deploying code

• Access website from program instead of using browser (Package-Base)

21

Units of Reuse

Rule

• can be used in different transformations

Strategy

• generic strategy can be instantiated with different rules

Tool

• can be reused in different transformation systems

Transformation system

• complete ‘source-to-source’ transformation system

• composition of (multiple) transformation tools with parser andpretty-printer

22

Summary

• High-level specification of transformations

• Applicable to many kinds of transformation

– compilation, generation, analysis, migration, ...

• Separation of rules and strategy

– specify rules separately from strategy that applies them

• Reuse of transformations at different levels of granularity

– rule, strategy, tool, system

• No strict separation between abstractions

– mix rules, strategies, tools

23

Availability

• www.stratego-language.org

– download StrategoXT (LGPL)– mailinglists– applications– documentation– package base– buildfarm

• Drafts of book available on request

• Fifth Stratego Users Day — Spring 2004

– presentations about new developments– meet users and developers

• I’m always looking for new applications!

24