B U I L D I N G A P Y T H O N I D E W I T H X T E X T

S E B A S T I A N Z A R N E K O W

T H I S I S N O T A B O U T

T H I S I S A B O U T

A N D A N T L R

A N D

[\n\r\t ]

I N D E N TAT I O N A W A R E

Block: OPEN stmt+=Statement+ CLOSE;

C H A L L E N G E SI N D E N TA T I O N A W A R E L A N G U A G E S

PA R S I N G C O N T E N T A S S I S T F O R M AT T I N G F O L D I N G O U T L I N E

C H A L L E N G E SI N D E N TA T I O N A W A R E L A N G U A G E S

PA R S I N GI N D E N TA T I O N A W A R E L A N G U A G E S - C H A L L E N G E S

PA R S I N GI N D E N TA T I O N A W A R E L A N G U A G E S - C H A L L E N G E S

Stateless, Upfront Lexing Reentrant Lexing CommonToken Oddities Token Types Only Basic Logic in Terminal Rules

S O M E P Y T H O N E S Q U E C O D E

if name==“Sebastian”: println “Hello”

S O M E P Y T H O N E S Q U E C O D E

if.name==“Sebastian”:\n ....println.“Hello”\n

PA R S I N G - T O K E N I Z I N G

i f . n a m e = = “ S e b a s t i an ” : \n . . . . p r i n t l n . “ H e l l o ” \n

PA R S I N G - T O K E N I Z I N G

i f . n a m e = = “ S e b a s t i a n ” : \n . . . . p r i n t l n . “ H e l l o ” \n

CharStream

o.a.IntStream

AntlrStringStream

PA R S I N G - T O K E N I Z I N G if.name==“Sebastian”:\n....println.“Hello”\n

CharStream

o.a.IntStream

AntlrStringStream

PA R S I N G - T O K E N I Z I N G if.name==“Sebastian”:\n....println.“Hello”\n

CharStream

o.a.IntStream

AntlrStringStream

1 2

9

1 0

7

8

6

9

1 0

9

9

8

- 1

PA R S I N G - T O K E N I Z I N G if.name==“Sebastian”:\n....println.“Hello”\n

CharStream

o.a.IntStream

TokenSource

Lexer

AntlrStringStream

1 2

9

1 0

7

8

6

9

1 0

9

9

8

- 1

PA R S I N G - T O K E N I Z I N G if.name==“Sebastian”:\n....println.“Hello”\n

1 2

9

1 0

7

8

6

9

1 0

9

9

8

TokenStream

- 1

TokenSource

Lexer

o.a.IntStream

if.name==“Sebastian”:\n....println.“Hello”\n

1 2

9

1 0

7

8

6

9

1 0

9

9

8

- 1

Parser

TokenStream

TokenSource

PA R S I N G - T O K E N I Z I N G

Parser

TokenStream

TokenSource

PA R S I N G - T O K E N I Z I N G

9

9

9

9

- 1

if.name==“Sebastian”:\n....println.“Hello”\n

if.name==“Sebastian”:\n....println.“Hello”\n

9

9

9

9

- 1

TokenSource

TokenStream

?

PA R S I N G - T O K E N I Z I N G

Parser

R E C A P

Block: OPEN stmt+=Statement+ CLOSE;

IfStatement: ‘if’ cond=Condition ‘:’ then=Block;

R E C A P

if name==“Sebastian”: println “Hello”

if.name==“Sebastian”:\n....println.“Hello”\n

\n OPEN . . . .

9

9

9

9

- 1

1 3 9

\n CLOSE1 4

PA R S I N G - T O K E N S P L I T T I N G

if.name==“Sebastian”:\n....println.“Hello”\n

9

9

9

9

- 1

1 3 9

1 4

PA R S I N G - T O K E N S P L I T T I N G

if.name==“Sebastian”:\n....println.“Hello”\n

1 3 9

1 4

PA R S I N G - T O K E N S P L I T T I N G

1 2

9

1 0

7

8

6

9

1 0

9

9

8

- 1

if.name==“Sebastian”:\n....println.“Hello”\n

PA R S I N G - T O K E N S P L I T T I N G

1 3 9

1 4

TokenStream

Parser

SplittingTokenSource

TokenSource1 2

9

1 0

7

8

6

9

1 0

9

9

8

- 1

Lexer

L E T ’ S D O T H I SI N D E N TA T I O N A W A R E L A N G U A G E S

Block: OPEN stmt+=Statement+ CLOSE;

1. D E F I N E I N D E N TAT I O N T O K E N S

1. D E F I N E I N D E N TAT I O N T O K E N S

terminal OPEN: ‘synthetic:OPEN’; terminal CLOSE: ‘synthetic:CLOSE’;

fragment = parser.antlr.ex.rt.AntlrGeneratorFragment {} .. fragment = parser.antlr.ex.ca.ContentAssistParserGeneratorFragment {}

2. A D J U S T M W E 2 W O R K F L O W

1. GENERATE SPLITTING TOKEN SOURCE 2. DISABLE PARTIAL PARSING

public class MyDslTokenSource extends AbstractIndentationTokenSource { public MyDslTokenSource(TokenSource delegate) { super(delegate); } @Override protected boolean shouldSplitTokenImpl(Token token) { return token.getType() == InternalMyDslParser.RULE_WS; }

@Override protected int getBeginTokenType() { return InternalMyDslParser.RULE_OPEN; }

@Override protected int getEndTokenType() { return InternalMyDslParser.RULE_CLOSE; } }

3. M A R K T O K E N S A S S P L I T TA B L E

3. M A R K T O K E N S A S S P L I T TA B L E

protected Token createEndToken(int offset) { CommonToken result = new CommonToken(getEndTokenType()); result.setText(""); result.setChannel(Token.DEFAULT_CHANNEL); result.setStartIndex(offset); result.setStopIndex(offset-1); return result; }

public class AbstractIndentationTokenSource extends AbstractSplittingTokenSource { …

… }

@Override protected boolean shouldSplitTokenImpl(Token token) { return token.getType() == InternalMyDslParser.RULE_WS; }

@Override protected int getBeginTokenType() { return InternalMyDslParser.RULE_OPEN; }

@Override protected int getEndTokenType() { return InternalMyDslParser.RULE_CLOSE; }

3. M A R K T O K E N S A S S P L I T TA B L E

@Override protected boolean shouldSplitTokenImpl(Token token) { return token.getType() == InternalMyDslParser.RULE_WS; }

@Override protected int getBeginTokenType() { return InternalMyDslParser.RULE_OPEN; }

@Override protected int getEndTokenType() { return InternalMyDslParser.RULE_CLOSE; }

3. M A R K T O K E N S A S S P L I T TA B L E

@Override protected boolean shouldEmitPendingEndTokens() { return false; }

4. M A R K T O K E N S A S S P L I T TA B L E A G A I N1. Configure Content-Assist Parser 2. Almost the same, but:

CharStream

o.a.IntStream TokenSource

Lexer

AntlrStringStream

P U T T I N G E V E R T H I N G T O G E T H E RI N D E N TA T I O N A W A R E L A N G U A G E S

TokenStream Parser

AbstractSplittingTokenSource

AbstractIndentationTokenSource

O N E M O R E T H I N GI N D E N TA T I O N A W A R E L A N G U A G E S

def void format(Block block, extension IFormattableDocument document) { val open = block.regionForRuleCallTo(OPENRule) open.prepend[ newLineAndIncreaseIndentation ] val close = block.regionForRuleCallTo(CLOSERule) if (block.isLast) close.append[ decreaseIndentation ] else close.append[ newLineAndDecreaseIndentation ] }

def void format(Block block, extension IFormattableDocument document) { val open = block.regionForRuleCallTo(OPENRule) open.prepend[ newLineAndIncreaseIndentation ] val close = block.regionForRuleCallTo(CLOSERule) if (block.isLast) close.append[ decreaseIndentation ] else close.append[ newLineAndDecreaseIndentation ] }

5. F O R M AT T I N G

1. Indent Blocks 2. Adjust NewLines 3. Remove superfluous spaces

PA R S I N G C O N T E N T A S S I S T F O R M AT T I N G F O L D I N G O U T L I N E

D O N EI N D E N TA T I O N A W A R E L A N G U A G E S

Q & A