LR Parsing - Lecture 7kjleach.eecs.umich.edu/c18/l7.pdf · LR Parsing An LR Parser reads tokens from left to right and constructs a bottom-up rightmost derivation. LR parsers shift

LR ParsingLecture 7

January 30, 2018

Midterm 1 Next Week

É in-class examÉ One sheet letter paper, two sidesÉ Some review questions onlineÉ Things to know

É Cool SyntaxÉ Compiler/language processor stepsÉ Regular LanguagesÉ NFA/DFAsÉ Regular ExpressionsÉ ParsingÉ Context-Free Grammars

Compiler Construction 2/41

From Last Week

É Top-down Parsing strategiesÉ Tells us how to get from the start symbol of a

grammar to a sequence of terminals following asequence of productions


Top Down Parsing

S → a B cB → C x BB → εC → d

| a B c

Input string:“adxdxc”

S

a d x d x c


Top Down Parsing


| a B c


S

BB

aa d x d x cc


Top Down Parsing


| a B c


S

B

CC

BB

a d xx d x c


Top Down Parsing


| a B c


S

B

C

B

a dd x d x c


Top Down Parsing


| a B c


S

B

C

CC

B

BB

a d x d xx c


Top Down Parsing


| a B c


S

B

C

C

B

B

a d x dd x c


Top Down Parsing


| a B c


S

B

C

C

B

B

a d x d x cε


Top Down Parsing (2)

E → T + E | TT → ( E ) | i n t | i n t ∗ T

Input: int * int

E



E → T + E | TT → ( E ) | i n t | i n t ∗ T

Input: int * int

E

TT ++ EE



E → T + E | TT → ( E ) | i n t | i n t ∗ T

Input: int * int

E

T + E

(( EE ))



E → T + E | TT → ( E ) | i n t | i n t ∗ T

Input: int * int

E

T + E

( E )

int



E → T + E | TT → ( E ) | i n t | i n t ∗ T

Input: int * int

E

T + E



E → T + E | TT → ( E ) | i n t | i n t ∗ T

Input: int * int

E

T + E

int



E → T + E | TT → ( E ) | i n t | i n t ∗ T

Input: int * int

E

T + E

int *



E → T + E | TT → ( E ) | i n t | i n t ∗ T

Input: int * int

E

T + E



E → T + E | TT → ( E ) | i n t | i n t ∗ T

Input: int * int

E

T + E

int * T



E → T + E | TT → ( E ) | i n t | i n t ∗ T

Input: int * int

E

T + E



E → T + E | TT → ( E ) | i n t | i n t ∗ T

Input: int * int

E



E → T + E | TT → ( E ) | i n t | i n t ∗ T

Input: int * int

E

intint ** TT



E → T + E | TT → ( E ) | i n t | i n t ∗ T

Input: int * int

E

intint * T



E → T + E | TT → ( E ) | i n t | i n t ∗ T

Input: int * int

E

intint ** T



E → T + E | TT → ( E ) | i n t | i n t ∗ T

Input: int * int

E

intint ** TT



E → T + E | TT → ( E ) | i n t | i n t ∗ T

Input: int * int

E

int * T

int



E → T + E | TT → ( E ) | i n t | i n t ∗ T

Input: int * int

E

int * T

int * int


Top-Down Summary

É Exhaust all grammatical rule options at eachstepÉ Notice significant backtracing!

É Backtracing results from having multiple optionsfor rules when considering tokens!

É If we restrict our grammar slightly, we can useLL(1) parsingÉ Key idea: deterministically select grammar rule

based on one token to eliminate backtracing


LL(1) Summary

É We want a parsing strategy that lets us parse aninput string by considering only one token ata timeÉ Intuition: If input string x ∈ L(G), there must be a

parse tree. We want to pick a grammar rule onetoken at a time so we can quickly find if the stringis in the language!

É Implementation: Parse tableÉ Intuition: 2-d table indexed by ‘current’

non-terminal and ‘lookahead’ token. Each cellcontains the production rule corresponding to thelookahead.


Parse Table Construction

É Constructing the Parse table requires aPREDICT set of every non-terminal in thegrammar.É We need a way to know which rule must be used

to successfully construct the next part of the parsetree

S → a| b

Consider! You know exactly which rule to takebased on the first thing you see as part of the S rule.


Parse Table ConstructionS → a A a

| b AA → c c A

| b b A

É Same idea! We always start with S. You can seethat the first non-terminal in the S rules tellsyou unambiguously which rule is used!É Similarly, after you consume one ‘a’, the next

non-terminal you consider is A. You can decidewhich A rule to choose based on the firstterminal in the A rules!



S → A aA → B bB → c B

| a A

É A bit trickier, but you can still decide whichrule to take with only one terminal



É Sounds like we need to know the first terminalthat can appear as a result of expanding anon-terminal!

É This is called the FIRST setFIRST(A) is the set of terminals that canappear first in the expansion of thenon-terminal AÉ Note this means recursing through other

nonterminals!

É FIRST(X) = {b|X→∗ bα} ∪ {ε|X→∗ ε}Compiler Construction 11/41


É That includes ε rules

S → a A aA → b A

| ε



É Recall we’re trying to make a parse tableÉ Almost enough to know FIRST set of all

non-terminals. The token will tell us which rule totake next.

É However, how do we know when the tokenshould be an ε?



S → a A B aA → b B

| εB → c A d

| ε

If my input string is aba, after consuming the ‘a’token, how do we know that the next ‘b’ tokenresults in B→ ε?


Parse Table Construction: FOLLOWÉ We also need to know the terminals that could

come after the previous non-terminal

É FOLLOW(X) = {b|S→∗ βXbω}

Computing FOLLOW set

É Compute FIRST sets for all non-terminalsÉ Add $ to FOLLOW(S)

É start symbol always ends with end-of-inputÉ For all productions Y → ...XA1...An

É Add FIRST(Ai)-{ε} to FOLLOW(X). Stop ifε 6∈ FIRST (Ai).

É Add FOLLOW(Y) to FOLLOW(X)Compiler Construction 15/41

Example FOLLOW Set

E → T XX → + E | εT → ( E ) | i n t YY → ∗ T | ε

FOLLOW(“+”) = { int, ( }FOLLOW(“(”) = { int, ( }FOLLOW(X) = { $, ) }FOLLOW(Y) = { +, ), $ }


Back to Parsing Tables

É Recall: We want to build a LL(1) Parsing Table

For each production A→ α in G do:É For each terminal b ∈ FIRST(α) do

É T[A][b] = αÉ If α→∗ ε, for each b ∈ FOLLOW(A) do

É T[A][b] = α


Parsing TableE → T XX → + E | εT → ( E ) | i n t YY → ∗ T | ε

Where do we put Y →∗T ?

É Well, FIRST(*T) = {*}, thus column * of row Y gets *T

int * + ( ) $

T int Y ( E )E T X T XX + E ε εY *T ε ε ε


Parsing TableE → T XX → + E | εT → ( E ) | i n t YY → ∗ T | ε

Where do we put Y → ε?É Well, FOLLOW(Y) = {$, +, )}, thus columns $, +, and )

in row Y get Y → ε

int * + ( ) $

T int Y ( E )E T X T XX + E ε εY *T ε ε ε


Another Parse Table Example

E → T X R → ∗ F RX → + T X | ε

| ε F → i n tT → F R | ( E )

+ * ( ) int $

EXTRF

Production Prediction

E→ TX (, intX→+TX +X→ ε $, )T → FR (, int)R→∗FR *R→ ε +, $, )F→ int intF→ (E) (


Another Parse Table Example

+ * ( ) int $

E TX TXX +TX ε εT FR FRR ε ∗FR ε εF (E) int

Production Prediction

E→ TX (, intX→+TX +X→ ε $, )T → FR (, int)R→∗FR *R→ ε +, $, )F→ int intF→ (E) (


Notes on LL(1) Parsing Tables

É If any entry is multiply defined then G is notLL(1)É G is ambiguousÉ G is left-recursiveÉ G is not left-factored


Ambiguity in parse tables

E → E + T T → FE → T F → i dT → T ∗ F F → ( E )

For the E productions, we need FIRST(T) = {(, id} andFIRST(E) = {(, id}

But now, which rule ( E→ E+T or E→ T ) gets put inT[E][(] and T[E][id]??

+ * ( ) id $

E ? ?TF


LR ParsingÉ LL(1) parsing is simple and fastÉ However, what if we wanted more

expressiveness in the grammar?

É Enter LR Parsing, a bottom-up parsingapproachÉ Equally efficient as LL(1)É Not quite as easy by handÉ Preferred practical method (see bison/yacc)


LR Parsing

An LR Parser reads tokens from left to right andconstructs a bottom-up rightmost derivation. LRparsers shift terminals and reduce input byapplication of productions in reverse. LR parsing isfast and easy, and uses a finite automaton (a.k.a. aparse table) augmented with a stack. LR works finewith grammars that are left-recursive or notleft-factored.


LR Parsing

É LR parsers do not require left-factoredgrammars and can also handle left-recursivegrammars

Consider

E→ E+(E)|int

Can you see why this is not LL(1)?


LR Parsing

É Consider input string xÉ Loop

É Identify β in x such that A→β isa production(i.e., x= αβγ )É Replace β by A in x

(i.e., x becomes αAγ )

É until x= SCompiler Construction 27/41

LR Parsing Example

S → a T R eT → T b c | bR → d

Rightmost derivation:

S → a T R e→ a T d e→ a T b c d e→ a b b c d e


LR Parsing Example





LR Parsing Example





LR Parsing Example





LR Parsing Example

Recall E→ E+(E)|int

Input = int + (int) + (int)

EE + (E)E + (int)E + (E) + (int)E + (int) + (int)int + (int) + (int)

int + ( int ) + ( int )


LR Parsing Example




E

int + ( int ) + ( int )


LR Parsing Example




E E

int + ( int ) + ( int )


LR Parsing Example




E

E E

int + ( int ) + ( int )


LR Parsing Example




E

E E E

int + ( int ) + ( int )


LR Parsing Example




E

E

E E E

int + ( int ) + ( int )


LR Parsing

Important fact about LR parsing:An LR parser traces a rightmostderivation in reverse.


Notation

Idea: Split the string into two substrings

É Right substring (a string of terminals)is as yet unexamined by the parserÉ Left substring has terminals and

non-terminals

The dividing point is marked by a •Initially, all input is new: • x1x2x3...xn


LR = shift-reduce Parsing

Bottom-up parsing uses only twoactions:

1. Shift2. Reduce


Shift

Shift: Move • one place to the rightÉ Shifts a terminal to the left string

E + ( • int )⇒

E + ( int • )


Reduce

Apply an inverse production at the right end ofthe left stringIf T → E+(E) is a production, then

E + ( E + ( E ) • )E + ( T • ) Reductions can

only happen here!


Shift-Reduce Example• int + (int) + (int)$ shift


Shift-Reduce Example• int + (int) + (int)$ shiftint • + (int) + (int)$ reduce E→ int


Shift-Reduce Example• int + (int) + (int)$ shiftint • + (int) + (int)$ reduce E→ intE • + (int) + (int)$ shift (3 times)


Shift-Reduce Example• int + (int) + (int)$ shiftint • + (int) + (int)$ reduce E→ intE • + (int) + (int)$ shift (3 times)E + (int • ) + (int)$ reduce E→ int


Shift-Reduce Example• int + (int) + (int)$ shiftint • + (int) + (int)$ reduce E→ intE • + (int) + (int)$ shift (3 times)E + (int • ) + (int)$ reduce E→ intE + (E • ) + (int)$ shift


Shift-Reduce Example• int + (int) + (int)$ shiftint • + (int) + (int)$ reduce E→ intE • + (int) + (int)$ shift (3 times)E + (int • ) + (int)$ reduce E→ intE + (E • ) + (int)$ shiftE + (E) • + (int)$ reduce E→ E+(E)


Shift-Reduce Example• int + (int) + (int)$ shiftint • + (int) + (int)$ reduce E→ intE • + (int) + (int)$ shift (3 times)E + (int • ) + (int)$ reduce E→ intE + (E • ) + (int)$ shiftE + (E) • + (int)$ reduce E→ E+(E)E • + (int)$ shift (3 times)


Shift-Reduce Example• int + (int) + (int)$ shiftint • + (int) + (int)$ reduce E→ intE • + (int) + (int)$ shift (3 times)E + (int • ) + (int)$ reduce E→ intE + (E • ) + (int)$ shiftE + (E) • + (int)$ reduce E→ E+(E)E • + (int)$ shift (3 times)E + (int • )$ reduce E→ int


Shift-Reduce Example• int + (int) + (int)$ shiftint • + (int) + (int)$ reduce E→ intE • + (int) + (int)$ shift (3 times)E + (int • ) + (int)$ reduce E→ intE + (E • ) + (int)$ shiftE + (E) • + (int)$ reduce E→ E+(E)E • + (int)$ shift (3 times)E + (int • )$ reduce E→ intE + (E • )$ shift


Shift-Reduce Example• int + (int) + (int)$ shiftint • + (int) + (int)$ reduce E→ intE • + (int) + (int)$ shift (3 times)E + (int • ) + (int)$ reduce E→ intE + (E • ) + (int)$ shiftE + (E) • + (int)$ reduce E→ E+(E)E • + (int)$ shift (3 times)E + (int • )$ reduce E→ intE + (E • )$ shiftE + (E) • $ reduce E→ E+(E)


Shift-Reduce Example• int + (int) + (int)$ shiftint • + (int) + (int)$ reduce E→ intE • + (int) + (int)$ shift (3 times)E + (int • ) + (int)$ reduce E→ intE + (E • ) + (int)$ shiftE + (E) • + (int)$ reduce E→ E+(E)E • + (int)$ shift (3 times)E + (int • )$ reduce E→ intE + (E • )$ shiftE + (E) • $ reduce E→ E+(E)E • $ accept


StackThe left string can be implemented with astackÉ Top of the stack is •

ShiftÉ pushes a terminal onto the stack

ReduceÉ pop 0 or more symbols from the stack

(production RHS)É push a non-terminal on the stack

(production LHS)Compiler Construction 36/41

When to Shift/Reduce?

É Decide based on the left string (stack)É Idea: Use a DFA to decide when to shift or

reduceÉ The DFA input is the stackÉ DFA language consists of terminals and

non-terminals

É We run the DFA on the stack and we examinethe resulting state X and the token t after •É If X has a transition labeled t, then shiftÉ If X is labeled with “A→β on t”, then reduce


LR(1) Parsing Example

0 1

23 4 5

67

8 9

10 11

E→ int on $, +

accept on $E→ int on ), +

E→ E + (E) on $, +

E→ E+(E) on ), +

int

E

+( int

E

)

+

(

E+

)

• int + (int) + (int)$ shift



0 1

23 4 5

67

8 9

10 11

E→ int on $, +


E→ E + (E) on $, +

E→ E+(E) on ), +

int

E

+( int

E

)

+

(

E+

)

• int + (int) + (int)$ shiftint • + (int) + (int)$ reduce E→ int



0 1

23 4 5

67

8 9

10 11

E→ int on $, +


E→ E + (E) on $, +

E→ E+(E) on ), +

int

E

+( int

E

)

+

(

E+

)

• int + (int) + (int)$ shiftint • + (int) + (int)$ reduce E→ intE • + (int) + (int)$ shift (3 times)



0 1

23 4 5

67

8 9

10 11

E→ int on $, +


E→ E + (E) on $, +

E→ E+(E) on ), +

int

E

+( int

E

)

+

(

E+

)

• int + (int) + (int)$ shiftint • + (int) + (int)$ reduce E→ intE • + (int) + (int)$ shift (3 times)E + (int • ) + (int)$ reduce E→ int



0 1

23 4 5

67

8 9

10 11

E→ int on $, +


E→ E + (E) on $, +

E→ E+(E) on ), +

int

E

+( int

E

)

+

(

E+

)

• int + (int) + (int)$ shiftint • + (int) + (int)$ reduce E→ intE • + (int) + (int)$ shift (3 times)E + (int • ) + (int)$ reduce E→ intE + (E • ) + (int)$ shift



0 1

23 4 5

67

8 9

10 11

E→ int on $, +


E→ E + (E) on $, +

E→ E+(E) on ), +

int

E

+( int

E

)

+

(

E+

)

• int + (int) + (int)$ shiftint • + (int) + (int)$ reduce E→ intE • + (int) + (int)$ shift (3 times)E + (int • ) + (int)$ reduce E→ intE + (E • ) + (int)$ shiftE + (E) • + (int)$ reduce E→ E+(E)



0 1

23 4 5

67

8 9

10 11

E→ int on $, +


E→ E + (E) on $, +

E→ E+(E) on ), +

int

E

+( int

E

)

+

(

E+

)

• int + (int) + (int)$ shiftint • + (int) + (int)$ reduce E→ intE • + (int) + (int)$ shift (3 times)E + (int • ) + (int)$ reduce E→ intE + (E • ) + (int)$ shiftE + (E) • + (int)$ reduce E→ E+(E)E • + (int)$ shift (3 times)



0 1

23 4 5

67

8 9

10 11

E→ int on $, +


E→ E + (E) on $, +

E→ E+(E) on ), +

int

E

+( int

E

)

+

(

E+

)

• int + (int) + (int)$ shiftint • + (int) + (int)$ reduce E→ intE • + (int) + (int)$ shift (3 times)E + (int • ) + (int)$ reduce E→ intE + (E • ) + (int)$ shiftE + (E) • + (int)$ reduce E→ E+(E)E • + (int)$ shift (3 times)E + (int • )$ reduce E→ int



0 1

23 4 5

67

8 9

10 11

E→ int on $, +


E→ E + (E) on $, +

E→ E+(E) on ), +

int

E

+( int

E

)

+

(

E+

)

• int + (int) + (int)$ shiftint • + (int) + (int)$ reduce E→ intE • + (int) + (int)$ shift (3 times)E + (int • ) + (int)$ reduce E→ intE + (E • ) + (int)$ shiftE + (E) • + (int)$ reduce E→ E+(E)E • + (int)$ shift (3 times)E + (int • )$ reduce E→ intE + (E • )$ shift



0 1

23 4 5

67

8 9

10 11

E→ int on $, +


E→ E + (E) on $, +

E→ E+(E) on ), +

int

E

+( int

E

)

+

(

E+

)

• int + (int) + (int)$ shiftint • + (int) + (int)$ reduce E→ intE • + (int) + (int)$ shift (3 times)E + (int • ) + (int)$ reduce E→ intE + (E • ) + (int)$ shiftE + (E) • + (int)$ reduce E→ E+(E)E • + (int)$ shift (3 times)E + (int • )$ reduce E→ intE + (E • )$ shiftE + (E) • $ reduce E→ E+(E)



0 1

23 4 5

67

8 9

10 11

E→ int on $, +


E→ E + (E) on $, +

E→ E+(E) on ), +

int

E

+( int

E

)

+

(

E+

)

• int + (int) + (int)$ shiftint • + (int) + (int)$ reduce E→ intE • + (int) + (int)$ shift (3 times)E + (int • ) + (int)$ reduce E→ intE + (E • ) + (int)$ shiftE + (E) • + (int)$ reduce E→ E+(E)E • + (int)$ shift (3 times)E + (int • )$ reduce E→ intE + (E • )$ shiftE + (E) • $ reduce E→ E+(E)E • $ accept


Parsing Tables

That’s right! Represent that DFA as a table

É Parsers represent the DFA as a 2D tableÉ Rows correspond to DFA statesÉ Columns correspond to

terminals/non-terminalsÉ Cells contain actions

É Terminals shift or reduceÉ Non-temrinals goto subsequent DFA states

É Critically Restart DFA with stack as inputevery time you reduce


Parsing Tablesint + ( ) $ E

0 s1 g21 rE→int rE→int2 s3 Accept3 s44 s5 g65 rE→int rE→int6 s8 s77 rE→E+(E) rE→E+(E)...

0 1

23 4 5

67

8 9

10 11

E→ int on $, +


E→ E + (E) on $, +

E→ E+(E) on ), +

int

E

+( int

E

)

+

(

E+

)


Parsing Tablesint + ( ) $ E

0 s1 g21 rE→int rE→int2 s3 Accept3 s44 s5 g65 rE→int rE→int6 s8 s77 rE→E+(E) rE→E+(E)...

0 1

23 4 5

67

8 9

10 11

E→ int on $, +


E→ E + (E) on $, +

E→ E+(E) on ), +

int

E

+( int

E

)

+

(

E+

)


Next time

É Parser generators construct the parsingDFA/table for youÉ For PA3, you use a yacc/bison-based parser

generator to implement the COOL grammar!É What about the abstract syntax tree


Documents

LR Parsing - Lecture 7kjleach.eecs.umich.edu/c18/l7.pdf · LR Parsing An LR Parser reads tokens from left to right and constructs a bottom-up rightmost derivation. LR parsers shift