redefine.digital.design: Helping you deal with complexity in VHDL and Verilog.

Xtending our VHDL Xtextformatter with the formatter2 APIir. Titouan Vervack

EclipseCon France2017-06-21

redefine.digital.design: Helping you deal with complexity in VHDL and Verilog.

Xtending our VHDL Xtextformatter with the formatter2 APIir. Titouan Vervack

EclipseCon France, Toulouse2017-06-21

Ghent

2

What do we do?

VHDL & SV IDE

3

What do we do?

Advanced analysis and linting

4

What do we do?

Visualisation

5

What do we do?

Visualisation

6

What do we do?

Eat cake

7

What do we do?

And cookies...

8

What do we do?

And occasionally… formatting

9

What do we do?

And occasionally… formatting

9

Our formatter problems

10

Our formatter problems

● Big grammar○ ~1.2k LoC○ ~200 rules

10

Our formatter problems

● Big grammar○ ~1.2k LoC○ ~200 rules

● Hard to fix bugs

10

Our formatter problems

● Big grammar○ ~1.2k LoC○ ~200 rules

● Hard to fix bugs

● Near impossible to add new features

10

Formatter 1.0 vs Formatter 2.0

11

Formatter 1.0 Formatter 2.0

● No access to○ Node model○ AST○ Text

● => Specify everything purely on the grammar

● => Lots of hacks

● Access to○ Node model○ AST○ Grammar

● Can’t take context into account○ Can’t adjust to the user

● Context aware formatting○ Conditional formatting○ Table formatting

● Not customizable ● Very customizable

Formatter 2.0 region example

variable _ foo _ : _ bit _ ; _-- bar\n

variable foo : bit; -- bar

= ISemanticRegion

= IHiddenRegion

_ -- bar \n

= IHiddenRegionPart

12https://de.slideshare.net/meysholdt/xtexts-new-formatter-api

Formatting 2.0 usage example

https://de.slideshare.net/meysholdt/xtexts-new-formatter-api13

Formatting 2.0 usage example

https://de.slideshare.net/meysholdt/xtexts-new-formatter-api

class AwesomO4000 extends AbstractFormatter2 {

}

13

Formatting 2.0 usage example

https://de.slideshare.net/meysholdt/xtexts-new-formatter-api

class AwesomO4000 extends AbstractFormatter2 {

def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) {

}}

13

Formatting 2.0 usage example

https://de.slideshare.net/meysholdt/xtexts-new-formatter-api

class AwesomO4000 extends AbstractFormatter2 {

def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) { interior(fd.regionFor.keyword(“{”), fd.regionFor.keyword(“}”))[indent]

}}

13

Formatting 2.0 usage example

https://de.slideshare.net/meysholdt/xtexts-new-formatter-api

class AwesomO4000 extends AbstractFormatter2 {

def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) { interior(fd.regionFor.keyword(“{”), fd.regionFor.keyword(“}”))[indent]

fd.declarations.forEach[format]

}}

13

Formatting 2.0 usage example

https://de.slideshare.net/meysholdt/xtexts-new-formatter-api

class AwesomO4000 extends AbstractFormatter2 {

def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) { interior(fd.regionFor.keyword(“{”), fd.regionFor.keyword(“}”))[indent]

fd.declarations.forEach[format]

fd.regionFor.keyword(“;”)

}}

13

Formatting 2.0 usage example

https://de.slideshare.net/meysholdt/xtexts-new-formatter-api

class AwesomO4000 extends AbstractFormatter2 {

def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) { interior(fd.regionFor.keyword(“{”), fd.regionFor.keyword(“}”))[indent]

fd.declarations.forEach[format]

fd.regionFor.keyword(“;”) .prepend[noSpace highPriority]

}}

13

Formatting 2.0 usage example

https://de.slideshare.net/meysholdt/xtexts-new-formatter-api

class AwesomO4000 extends AbstractFormatter2 {

def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) { interior(fd.regionFor.keyword(“{”), fd.regionFor.keyword(“}”))[indent]

fd.declarations.forEach[format]

fd.regionFor.keyword(“;”) .prepend[noSpace highPriority] .append[setNewlines(1, 1, 2)] }}

13

Time for a plan

14

Time for a plan

1. Make tests compatible

14

Time for a plan

1. Make tests compatible2. Start out easy: only indentation

14

Time for a plan

1. Make tests compatible2. Start out easy: only indentation3. Fix tests one by one

14

Time for a plan

1. Make tests compatible2. Start out easy: only indentation3. Fix tests one by one4. Fix old bugs

14

Time for a plan

1. Make tests compatible2. Start out easy: only indentation3. Fix tests one by one4. Fix old bugs5. Do not introduce new bugs

14

Time for a plan

1. Make tests compatible2. Start out easy: only indentation3. Fix tests one by one4. Fix old bugs5. Do not introduce new bugs6. Feature parity & extend with new features

14

Time for a plan

1. Make tests compatible2. Start out easy: only indentation3. Fix tests one by one4. Fix old bugs5. Do not introduce new bugs6. Feature parity & extend with new features7. Performance tweaking to same or better level than old

14

Actual timeline

formatter.format(node, node.offset, node.length).formattedText

1. Make tests compatible

15

Actual timeline

formatter.format(node, node.offset, node.length).formattedText

1. Make tests compatible

15

Actual timeline

formatter.format(node, node.offset, node.length).formattedText

1. Make tests compatible

val request = new FormatterRequest()

val parsed = new XtextResource()

parsed.contents.add(parseResult.rootASTElement)

parsed.setParseResult(parseResult)

val access = builder.forNodeModel(resource).create()

request.setTextRegionAccess(access)

val replacements = formatter.format(request) access.rewriter.renderToString(replacements)

15

interior(entity.regionFor.keyword(“is”), entity.regionFor.keyword(“end”))[indent]

Actual timeline

2. Start out easy: only indentation

16

interior(entity.regionFor.keyword(“is”), entity.regionFor.keyword(“end”))[indent]

Actual timeline

2. Start out easy: only indentation

● Indentation requires newline definitions

16

interior(entity.regionFor.keyword(“is”), entity.regionFor.keyword(“end”))[indent]

Actual timeline

2. Start out easy: only indentation

● Indentation requires newline definitions

● => replace all newlines with… the same amount of newlines

16

Actual timeline

3. Fix tests one by one4. Fix old bugs

17

Actual timeline

3. Fix tests one by one4. Fix old bugs

17

Actual timeline

5. Do not introduce new bugs

18

Actual timeline

5. Do not introduce new bugs

18

Actual timeline

6. Feature parity & extend formatter with new features

19

Actual timeline

6. Feature parity & extend formatter with new features

● Parity:

19

Actual timeline

6. Feature parity & extend formatter with new features

● Parity:○ Keyword casing

19

Actual timeline

6. Feature parity & extend formatter with new features

● Parity:○ Keyword casing○ Comment alignment to columns

19

Actual timeline

6. Feature parity & extend formatter with new features

● Parity:○ Keyword casing○ Comment alignment to columns○ Vertical alignment on keywords “:”, “:=”,...

19

Actual timeline

6. Feature parity & extend formatter with new features

● Parity:○ Keyword casing○ Comment alignment to columns○ Vertical alignment on keywords “:”, “:=”,...○ Preserve newline

19

Actual timeline

6. Feature parity & extend formatter with new features

● Parity:○ Keyword casing○ Comment alignment to columns○ Vertical alignment on keywords “:”, “:=”,...○ Preserve newline

● New features:

19

Actual timeline

6. Feature parity & extend formatter with new features

● Parity:○ Keyword casing○ Comment alignment to columns○ Vertical alignment on keywords “:”, “:=”,...○ Preserve newline

● New features:○ Correct indentation

19

Actual timeline

6. Feature parity & extend formatter with new features

● Parity:○ Keyword casing○ Comment alignment to columns○ Vertical alignment on keywords “:”, “:=”,...○ Preserve newline

● New features:○ Correct indentation○ Formatter tags

19

Actual timeline

6. Feature parity & extend formatter with new features

● Parity:○ Keyword casing○ Comment alignment to columns○ Vertical alignment on keywords “:”, “:=”,...○ Preserve newline

● New features:○ Correct indentation○ Formatter tags

Easy once we have alignment19

Alignment?

20

Alignment?

20

Alignment?

● Replacements are added in random order, applied in correct order

21

Alignment?

● Replacements are added in random order, applied in correct order

● Alignment needs to be done on post-formatted input

21

Alignment?

● Replacements are added in random order, applied in correct order

● Alignment needs to be done on post-formatted input

● How do I know which region to format?

21

Alignment!

22

Alignment!

22

Alignment!

22

Alignment!

22

Alignment!

start end

22

Alignment!

start end

22

Alignment!

Find smallest encapsulating object that formats region between start and end

start end

22

Alignment!

Find smallest encapsulating object that formats region between start and end

start end

22

Alignment!

clk : in std_logic := ‘X’;

23

Alignment!

clk : in std_logic := ‘X’;

start end

23

PortDeclaration

Alignment!

clk : in std_logic := ‘X’;

start end

name direction type value

end.getSemanticElement()

PortDeclaration

PortList

23

PortDeclaration

Alignment!

clk : in std_logic := ‘X’;

start end

name direction type value

offset x

x + 4

end.getSemanticElement()

PortDeclaration

PortList

23

val rule = portDecl.getParserRule()val oldAccess = createTextRegionAccess(rule, portDecl.text)

val formatted = portDecl.format()

Alignment!

Format smallest encapsulating object

24

val rule = portDecl.getParserRule()val oldAccess = createTextRegionAccess(rule, portDecl.text)

val formatted = portDecl.format()

Alignment!

val newAccess = createTextRegionAccess(rule, formatted.text)val oldIndices = oldAccess.getIndices(start, end)val length = newAccess.countBetween(oldIndices)

Format smallest encapsulating object

Map unformatted string to formatted string

24

Align on first parameter?

→return (→→foo((5,→→·····6),→→····3));

25

Align on first parameter?

→return (→→foo((5,→→·····6),→→····3));

25

Align on first parameter?

→return (→→foo((5,→→·····6),→→····3));

→return (5,→········foo((5,→·············6),→············3));

25

Align on first parameter?

→return (→→foo((5,→→·····6),→→····3));

→return (5,→········foo((5,→·············6),→············3));

25

Align on first parameter?

→return (→→foo((5,→→·····6),→→····3));

→return (5,→········foo((5,→·············6),→············3));

→return foo((5,→············6),→···········3));

25

Align on first parameter?

→return (→→foo((5,→→·····6),→→····3));

→return (5,→········foo((5,→·············6),→············3));

→return foo((5,→············6),→···········3));

25

Align on first parameter?

→return (→→foo((5,→→·····6),→→····3));

→return (5,→········foo((5,→·············6),→············3));

→return foo((5,→············6),→···········3));

→return foo(→→(5,→→·6),→→3));

25

Align on first parameter?

→return (→→foo((5,→→·····6),→→····3));

→return (5,→········foo((5,→·············6),→············3));

→return foo((5,→············6),→···········3));

→return foo(→→(5,→→·6),→→3));

25

Align on first parameter?

→return (→→foo((5,→→·····6),→→····3));

→return (5,→········foo((5,→·············6),→············3));

→return foo((5,→············6),→···········3));

→return foo(→→(5,→→·6),→→3));

Find largest encapsulating element on same line

25

Actual timeline

6. Feature parity & extend formatter with new features

Correct indentation

26

Actual timeline

6. Feature parity & extend formatter with new features

Correct indentation

val replacements = formatter.format(request)

26

Actual timeline

6. Feature parity & extend formatter with new features

Correct indentation

val replacements = formatter.format(request)replacements .filter[ r|

]

26

Actual timeline

6. Feature parity & extend formatter with new features

Correct indentation

val replacements = formatter.format(request)replacements .filter[ r| r.lineCount > 1

]

26

Actual timeline

6. Feature parity & extend formatter with new features

Correct indentation

val replacements = formatter.format(request)replacements .filter[ r| r.lineCount > 1 && r.replacementText.newlines + 1 == r.lineCount ]

26

Actual timeline

6. Feature parity & extend formatter with new features

Formatter tags

27

Actual timeline

6. Feature parity & extend formatter with new features

Formatter tags

-- @formatter:off Turns formatter off-- @formatter:on Turns formatter on

27

Actual timeline

6. Feature parity & extend formatter with new features

Formatter tags

-- @formatter:off Turns formatter off-- @formatter:on Turns formatter on

val originalRegions = request.getRegions() ?: document.region

27

Actual timeline

6. Feature parity & extend formatter with new features

Formatter tags

-- @formatter:off Turns formatter off-- @formatter:on Turns formatter on

val originalRegions = request.getRegions() ?: document.regionval betweenRegions = document.findRegionsBetween(off, on)

27

Actual timeline

6. Feature parity & extend formatter with new features

Formatter tags

-- @formatter:off Turns formatter off-- @formatter:on Turns formatter on

val originalRegions = request.getRegions() ?: document.regionval betweenRegions = document.findRegionsBetween(off, on) .addAll(document.findRegionsBetween(off, EOF))

27

Actual timeline

6. Feature parity & extend formatter with new features

Formatter tags

-- @formatter:off Turns formatter off-- @formatter:on Turns formatter on

val originalRegions = request.getRegions() ?: document.regionval betweenRegions = document.findRegionsBetween(off, on) .addAll(document.findRegionsBetween(off, EOF))val newRegions = originalRegions.removeRegions(betweenRegions)

27

Actual timeline

6. Feature parity & extend formatter with new features

Formatter tags

-- @formatter:off Turns formatter off-- @formatter:on Turns formatter on

val originalRegions = request.getRegions() ?: document.regionval betweenRegions = document.findRegionsBetween(off, on) .addAll(document.findRegionsBetween(off, EOF))val newRegions = originalRegions.removeRegions(betweenRegions)request.setRegions(newRegions)

27

Performance

7. Performance tweaking to same or better level than old

28

Performance

7. Performance tweaking to same or better level than old

● Cache ITextRegionAccess in subformats

28

Performance

7. Performance tweaking to same or better level than old

● Cache ITextRegionAccess in subformats

● Cache IParser.createAllRules(Grammar)

28

Performance

7. Performance tweaking to same or better level than old

● Cache ITextRegionAccess in subformats

● Cache IParser.createAllRules(Grammar)

● Merge all replacements into one big replacement

28

Performance

7. Performance tweaking to same or better level than old

emacs/xtend > 25m

29

Performance

7. Performance tweaking to same or better level than old

emacs/xtend > 25m

30

Biggest problems in hindsight

31

Biggest problems in hindsight

● Tests pass, close enough?

31

Biggest problems in hindsight

● Tests pass, close enough?○ Nope, old testset isn’t nearly enough

31

Biggest problems in hindsight

● Tests pass, close enough?○ Nope, old testset isn’t nearly enough

● Comments are an enormous pain: eat newlines, add spaces, eat spaces, fake empty lines

31

Biggest problems in hindsight

● Tests pass, close enough?○ Nope, old testset isn’t nearly enough

● Comments are an enormous pain: eat newlines, add spaces, eat spaces, fake empty lines

● Alignment interactions with normal and first param alignments and comments

31

Biggest problems in hindsight

● Tests pass, close enough?○ Nope, old testset isn’t nearly enough

● Comments are an enormous pain: eat newlines, add spaces, eat spaces, fake empty lines

● Alignment interactions with normal and first param alignments and comments

31

Biggest problems in hindsight

● Tests pass, close enough?○ Nope, old testset isn’t nearly enough

● Comments are an enormous pain: eat newlines, add spaces, eat spaces, fake empty lines

● Alignment interactions with normal and first param alignments and comments

31

Biggest problems in hindsight

● Tests pass, close enough?○ Nope, old testset isn’t nearly enough

● Comments are an enormous pain: eat newlines, add spaces, eat spaces, fake empty lines

● Alignment interactions with normal and first param alignments and comments

31

Biggest problems in hindsight

● Tests pass, close enough?○ Nope, old testset isn’t nearly enough

● Comments are an enormous pain: eat newlines, add spaces, eat spaces, fake empty lines

● Alignment interactions with normal and first param alignments and comments

3031

Biggest problems in hindsight

● Tests pass, close enough?○ Nope, old testset isn’t nearly enough

● Comments are an enormous pain: eat newlines, add spaces, eat spaces, fake empty lines

● Alignment interactions with normal and first param alignments and comments

3031

Biggest problems in hindsight

● Tests pass, close enough?○ Nope, old testset isn’t nearly enough

● Comments are an enormous pain: eat newlines, add spaces, eat spaces, fake empty lines

● Alignment interactions with normal and first param alignments and comments

3031

Warning to Xtext users

Default ExceptionHandler logs DSL source code to stderr

32

Sigasi future work

33

Sigasi future work

● Smooth out remaining bugs

33

Sigasi future work

● Smooth out remaining bugs● Some more context awareness

33

Sigasi future work

● Smooth out remaining bugs● Some more context awareness● Autowrap

33

Sigasi future work

● Smooth out remaining bugs● Some more context awareness● Autowrap● Performance

33

Sigasi future work

● Smooth out remaining bugs● Some more context awareness● Autowrap● Performance● Upstream features

33

Sigasi future work

● Smooth out remaining bugs● Some more context awareness● Autowrap● Performance● Upstream features

○ Formatter tags

33

Sigasi future work

● Smooth out remaining bugs● Some more context awareness● Autowrap● Performance● Upstream features

○ Formatter tags○ Correct indentation

33

Sigasi future work

● Smooth out remaining bugs● Some more context awareness● Autowrap● Performance● Upstream features

○ Formatter tags○ Correct indentation○ Performance optimizations

33

Xtext wish list

34

Xtext wish list

● Accept my PRs (tags, correct indent, performance)

34

Xtext wish list

● Accept my PRs (tags, correct indent, performance)● Fix/discuss my issues

34

Xtext wish list

● Accept my PRs (tags, correct indent, performance)● Fix/discuss my issues● Fix Xtend formatter issues

34

Xtext wish list

● Accept my PRs (tags, correct indent, performance)● Fix/discuss my issues● Fix Xtend formatter issues● Very slow Xtend formatter source file

34

Xtext wish list

● Accept my PRs (tags, correct indent, performance)● Fix/discuss my issues● Fix Xtend formatter issues● Very slow Xtend formatter source file● Add pre- and post format hooks in format(FormatterRequest)

34

Conclusion

35

Conclusion

● Context aware

35

Conclusion

● Context aware● Debugable

35

Conclusion

● Context aware● Debugable● Good performance

35

Conclusion

● Context aware● Debugable● Good performance● Very customizable

35

Conclusion

● Context aware● Debugable● Good performance● Very customizable● User friendly

35

Conclusion

● Context aware● Debugable● Good performance● Very customizable● User friendly

○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend

35

Conclusion

● Context aware● Debugable● Good performance● Very customizable● User friendly

○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend

● Happy with the switch

35

Conclusion

● Context aware● Debugable● Good performance● Very customizable● User friendly

○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend

● Happy with the switch○ New features

35

Conclusion

● Context aware● Debugable● Good performance● Very customizable● User friendly

○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend

● Happy with the switch○ New features○ Very readable

35

Conclusion

● Context aware● Debugable● Good performance● Very customizable● User friendly

○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend

● Happy with the switch○ New features○ Very readable○ Conditional formatting

35

Evaluate the Sessions

-1 0 +1Sign in and vote at eclipsecon.org