Creating own language made easy

Creating own language made easy

Ingvar Stepanyan

@RReverser

Everything sucks

Everything sucks – human version

Everything sucks – developer version

Scary magic

?

?????

Not so scary magic

JS

JS

JS

ParserGenerator

Parsers

Parser generators

jison Bison in javascript, used by Coffeescript PEG.js parser generator for JavaScript based on the parsing expression grammar formalism OMeta/JS (source) metacompiler for many languages to many targets, including js.

languagejs - PEG parser written in JavaScript with first class errors Canopy Self-hosting PEG parser compiler for JavaScript, influenced by Ruby parser generators such as Treetop and Citrus JS/CC LALR(1) parser generator

jsparse PEG by Grandmaster Chris Double ReParse parser combinator library for Javascript like Haskell's Parsec p4js Monadic parser library for JavaScript JSGLR Scannerless, Generalized Left-to-right Rightmost (SGLR) derivation parser for JavaScript

antlr has a javascript target Cruiser.Parse LL(k) parser

Parser generators

Bottom-up (PEG.js)start = additive

additive = left:multiplicative "+" right:additive { return left + right; } / multiplicative

multiplicative = left:primary "*" right:multiplicative { return left * right; } / primary

primary = integer / "(" additive:additive ")" { return additive; }

integer "integer" = digits:[0-9]+ { return parseInt(digits.join(""), 10); }

Top-down (Jison)%left "+"%left "*"

%start program;

%%

program : expression { return $$ } ;

expression : expression "+" expression -> $1 + $3 | expression "*" expression -> $1 * $3 | "(" expression ")" -> $2 | NUMBER -> $1 ;

Choice ordering

Bottom-up (PEG.js)start = additive

additive = left:multiplicative "+" right:additive { return left + right; } / multiplicative

multiplicative = left:primary "*" right:multiplicative { return left * right; } / primary

primary = integer / "(" additive:additive ")" { return additive; }

integer "integer" = digits:[0-9]+ { return parseInt(digits.join(""), 10); }

Top-down (Jison)%left "+"%left "*"

%start program;

%%

program : expression { return $$ } ;

expression : expression "+" expression -> $1 + $3 | expression "*" expression -> $1 * $3 | "(" expression ")" -> $2 | NUMBER -> $1 ;

Ambiguity

Bottom-up (PEG.js)start = additive

additive = left:multiplicative "+" right:additive { return left + right; } / multiplicative

multiplicative = left:primary "*" right:multiplicative { return left * right; } / primary

primary = integer / "(" additive:additive ")" { return additive; }

integer "integer" = digits:[0-9]+ { return parseInt(digits.join(""), 10); }

Top-down (Jison)%left "+"%left "*"

%start program;

%%

program : expression { return $$ } ;

expression : expression "+" expression -> $1 + $3 | expression "*" expression -> $1 * $3 | "(" expression ")" -> $2 | NUMBER -> $1 ;

if a then (if b then s) else s2 orif a then (if b then s else s2)

if a then if b then s else s2:

Left recursion

Bottom-up (PEG.js)start = additive

additive = left:multiplicative "+" right:additive { return left + right; } / multiplicative

multiplicative = left:primary "*" right:multiplicative { return left * right; } / primary

primary = integer / "(" additive:additive ")" { return additive; }

integer "integer" = digits:[0-9]+ { return parseInt(digits.join(""), 10); }

Top-down (Jison)%left "+"%left "*"

%start program;

%%

program : expression { return $$ } ;

expression : expression "+" expression -> $1 + $3 | expression "*" expression -> $1 * $3 | "(" expression ")" -> $2 | NUMBER -> $1 ;

x = 1 - 2 - 3

Left recursion

Bottom-up (PEG.js)[ "x", "=", [ "1", "-", [ "2", "-", "3" ] ]]

Top-down (Jison)[ "x", "=", [ [ "1", "-", "2" ], "-", "3" ]]

x = 1 - 2 - 3

Summary choice

Bottom-up (PEG.js)start = additive

additive = left:multiplicative "+" right:additive { return left + right; } / multiplicative

multiplicative = left:primary "*" right:multiplicative { return left * right; } / primary

primary = integer / "(" additive:additive ")" { return additive; }

integer "integer" = digits:[0-9]+ { return parseInt(digits.join(""), 10); }

Top-down (Jison)%left "+"%left "*"

%start program;

%%

program : expression { return $$ } ;

expression : expression "+" expression -> $1 + $3 | expression "*" expression -> $1 * $3 | "(" expression ")" -> $2 | NUMBER -> $1 ;

Jison syntax: helpers

%{

var scope = {};

%}

…

Jison syntax: lexer

…

%lex

%%

\s+ /* skip whitespace */

[A-Za-z_]\w+ return 'ID';

\d+ return ‘NUMBER’;

[+*;=] return yytext;

<<EOF>> return 'EOF';

/lex

…

Jison syntax: operator precedence

…

%left ';‘

%right ‘=‘

%left ‘+’

%left ‘*’

…

/*

“x=a*b+c”

-> assign(“x”, “a*b+c”)

-> assign(“x”, add(“a*b”, “c”))

-> assign(“x”, add(mul(“a”, “b”), “c”))

*/

Jison syntax: rules

%start program

program

: stmt* EOF { return $1 }

;

stmt

: expr ‘;’ -> $1

;

expr

: expression "+" expression -> $1 + $3

| expression "*" expression -> $1 * $3

| NUMBER -> $1

;

Code generation

Methods

string concatenation

building AST object + escodegen (http://github.com/Constellation/escodegen)

using SourceNode from source-map (https://github.com/mozilla/source-map)

Debugging: source maps

Methods

string concatenation

building AST object + escodegen (http://github.com/Constellation/escodegen)

using SourceNode from source-map (https://github.com/mozilla/source-map)

AST way

Methods

string concatenation

building AST object + escodegen (http://github.com/Constellation/escodegen)

using SourceNode from source-map (https://github.com/mozilla/source-map)

SourceNode

new SourceNode(line, column, filename, jsChunk)

line, column – position in original file

filename – name of original file

jsChunk – JavaScript code string, another SourceNode instance or array of those

Resulting stack

JS

JS

JS

Jisonsource-map

Live demo