Parser s

8/8/2019 Parser s

1/47

8/8/2019 Parser s

2/47

Prof. Busch - LSU 2

Compiler

Progra m Filev = 5;if (v>5)

x = 12 +v;while (x !=3) {

x = x - 3;

v = 10;}......

Add v,v,5cmp v,5 jmplt ELSE

THEN:add x, 12,v

ELSE:WHILE:cmp x,3...

Machine Code

8/8/2019 Parser s

3/47

Prof. Busch - LSU 3

Lexical

analyzer parser

Compiler

Progra m

file

machine

code

I nput String Out put

8/8/2019 Parser s

4/47

Prof. Busch - LSU 4

Lexical analyzer :

Recognizes t he lexemes of t he input progra m file :Keywords (if, t hen, else, while, ), I ntegers,Id entifiers (variables ), et c

It is built wit h DFAs (base d on t he t heory of reg ular languages )

8/8/2019 Parser s

5/47

Prof. Busch - LSU 5

Knows t he gra mmar of t he progra mming language t o be compiled

Parser :

Constr ucts derivati on (and derivati on tree )f or input progra m file (input string )

Converts derivati on t o machine code

8/8/2019 Parser s

6/47

Prof. Busch - LSU 6

Example ParserPROGRAM STMT_LIST STMT_LIST STMT ; STMT_LIST | STMT ;STMT EXPR | IF_STMT | WHILE_STMT

| {STMT_LIST }

EXPR EXPR +EXPR | EXPR - EXPR | ID

IF_STMT if (EXPR) t hen STMT | if (EXPR) t hen STMT else STMT WHILE_STMT while (EXPR) do STMT

pp

p

pp

p

8/8/2019 Parser s

7/47

Prof. Busch - LSU 7

The parser finds t he derivati onof a parti cular input file

10 + 2 * 5

Example Parser

E ->E + E| E * E

| INT

E =>E + E=>E + E * E=>10 + E*E

=>10 + 2 * E=>10 + 2 * 5

I nput string

derivati on

8/8/2019 Parser s

8/47

Prof. Busch - LSU 8

E =>E + E=>E + E * E=>10 + E*E

=>10 + 2 * E=>10 + 2 * 5

derivati on derivati on tree

10

E

2 5

E E

E E

+

*

mult a, 2, 5add b, 10, a

machine codeDerivati on treesare use d t o build

Machine code

a

b

8/8/2019 Parser s

9/47

Prof. Busch - LSU 9

A simple (exhaustive ) parser

8/8/2019 Parser s

10/47

Prof. Busch - LSU 10

gra mmar

Exhaustive Parserinputstring

derivati on

We will build an exhaustive sear ch parser

t hat examines all possible derivati ons

8/8/2019 Parser s

11/47


Example:

Exhaustive Parserderivati on

Ppp

p

p

S bSaS

aSbS

SS S I nput string

?aa bb

aa bbFind derivati on of string

8/8/2019 Parser s

12/47


Exhaustive Sear ch

P||| bS aa SbSS S p

Phase 1:

PS

bS aS

a SbS

SS S

All possible derivati ons of lengt h 1

Find derivati on

of aa bb

8/8/2019 Parser s

13/47

8/8/2019 Parser s

14/47


aSbS

SS S

Phase 1

P||| bS aa SbSS S p

I n Phase 2, explore t he next ste p

of each derivati on fr om Phase 1

8/8/2019 Parser s

15/47


Phase 2

aSbS

SS S S SS S

bSaS SS S

aSbS SS S

SSS SS S

Phase 1

abaSbS

abSabaSbS

aaSbbaSbS

aSSbaSbS

P||| bS aa SbSS S p

Find derivati onof aa bb

8/8/2019 Parser s

16/47


Phase 2

S SS S

a SbS SS S

SSS SS S

aa Sbba SbS

a SSba SbS

Find derivati onof aabb

I n Phase 3 explore all possible derivati ons

P||| bS aa SbSS S p

8/8/2019 Parser s

17/47


Phase 2

S SS S

a SbS SS S

SSS SS S

aa Sbba SbS

a SSba SbS

A possible derivati on of Phase 3

aabbaaSbbaSbS

Find derivati onof aabb

P||| bS aa SbSS S p

8/8/2019 Parser s

18/47


Final res ult of exhaustive sear ch

Exhaustive Parser

derivati on

Pp

p

p

p

S

bSaS

aSbS

SS S I nput stringaabb

aabbaaSbbaSbS

8/8/2019 Parser s

19/47


( -producti ons)

Suppose t hat t he gra mmar does not haveproducti ons of t he f orm

Pp A

B A p (unit producti ons)

P

T ime Complexity

8/8/2019 Parser s

20/47


w x x x S k

.21

|||| w x i e

For any der i vat i on of a str i ng of ter mi nals )(G w

i t holds t hat f or all i

PSi nce t he are no -product i ons

8/8/2019 Parser s

21/47


Since t he are no unit producti ons

1. At most derivati on ste ps are neededt o produce a string wit h at mostvariables

||w jx ||w

2. At most derivati on ste ps are neededt o convert t he variables of t o t he string of ter minals

||w jx

w

8/8/2019 Parser s

22/47


Theref ore, at most derivati on

ste ps are req uire d t o produce

||2 w w

The exhaustive sear ch req uires at most

||2 w phases

8/8/2019 Parser s

23/47

8/8/2019 Parser s

24/47


Choices f or phase 2: at most 2k k k !v

Choices of phase 1

Number ofProducti ons

Choices f or phase i: at mostii k k k !v)1(

Choices ofphase i-1

Number ofProducti ons

I n General

8/8/2019 Parser s

25/47


Total explorati on choices f or string :w

)( ||2||22 ww k O k k k !.

Extre mely bad!!!

phase 1 phase 2 phase 2|w|

Exponential t o t he string lengt h

8/8/2019 Parser s

26/47


Faster Parsers

8/8/2019 Parser s

27/47


There exist faster parsing algorit hmsf or specialized gra mmars

S-gra mmar : avpS ymbol String of variables

),( W X

appears once in a producti on

Each pair of variable, ter minal

(a restri cte d versi on of Greinbach Normal f orm)

wW p

8/8/2019 Parser s

28/47


S-gra mmar example:

cS

bSS S

aS S

p

p

p

ab ccab c S abSS aS S

Each string has a unique derivati on

8/8/2019 Parser s

29/47


I n t he exhaustive sear ch parsingt here is only one choice in each phase

For S-gra mmars :

Total ste ps f or parsing string :w || w

Ste ps f or a phase : 1

8/8/2019 Parser s

30/47


For general conte xt -free gra mmars :

Next, we give a parsing algorit hmt hat parses a string in ti me w )|(| 3wO

(t his ti me is very close t o t he worst caseopti mal since parsing can be used t o solve t he matri x multiplicati on problem)

8/8/2019 Parser s

31/47


The CYKParsing Algorit hm

I nput : Arbitrary Grammar in Chomsky Normal FormG

String

Out put : Deter mine if )(G Lw

w

Number of Ste ps: )|(|3

wO

Can be easily converte d t o a Parser

8/8/2019 Parser s

32/47


Basic Id ea

Denote by t he set of variablest hat generate a string

)(w F w

Consider a gra mmar I n Chomsky Normal Form

G

)(w F X w*

if

8/8/2019 Parser s

33/47


Suppose t hat we have compute d )(wF

Check if :)(wFS

YES

NO

)(G w

)(G w )(

*

wS

8/8/2019 Parser s

34/47


and t here is producti on

u vw !

)(u F

)(w F can be compute d re cursively :

)(v F Y

XY H p

Then )(w F H

prefi x suffi x

I f

)(*

vY )( * u X

and

)(

**

wuv uYXYH!

Write

8/8/2019 Parser s

35/47


Examine all prefi x-suffi xdecompositi ons of w

1||1!

w v uw

2||2!

w v uw

11|| v uw w !

1

2

| w |-1

Lengt h Set of Variablest hat generate w

1 H

2 H

1||w H

1||21)( ! ww F .Result :

8/8/2019 Parser s

36/47


At t he basis of t he re cursi onwe have strings of lengt h 1

}symbolgeneratethatVariables{)( W W ! F

symbol W p X

Very easy t o find

8/8/2019 Parser s

37/47


The whole algorit hm can be implemente dwit h d ynamic progra mming:

Remark :

First compute f or smallersubstrings and t hen use t hist o compute t he res ult f or larger substrings of

)(wF dw d

w

8/8/2019 Parser s

38/47


Grammar :G

b A

a BB A

A BS

|

|

p

p

p

Deter mine if )(G Laabbbw !

Example:

8/8/2019 Parser s

39/47


a a b b b

aa ab bb bb

aab abb bbb

aabb abbb

aabbb

aa bbbDecompose t he string t o all possible substrings

Lengt h1

23

4

5

8/8/2019 Parser s

40/47


a {A}

a {A}

b {B}

b {B}

b {B}

aa ab bb bb

aab abb bbb

aabb abbb

aabbb

b A B Ba BB A A BS | ,| , ppp

)(W F

8/8/2019 Parser s

41/47

8/8/2019 Parser s

42/47


)(aa F

}{)( Aa F !

AA X pThere is no producti on of f orm

aaprefi x suffi x}{)( Aa F !


Thus, {})(!aa F

)(a b F

}{)( Aa F !

A B X pThere are t wo producti ons of f orm

abprefi x suffi x}{)( Bb F !

Thus, },{)( BS a b F ! A B B A BS pp ,

8/8/2019 Parser s

43/47


a

{ }

a

{ }

b

{B}

b

{B}

b

{B}aa{}

ab { ,B}

bb { }

bb { }

aab { ,B}

abb { }

bbb { ,B}

aabb abbb

aabbb


8/8/2019 Parser s

44/47


)(F

}{)(F !

AS X pThere is no producti on of f orm

prefi x suffi x},{)( BS ab F !


AB X p

A B B A BS pp ,

There are 2 producti ons of f orm

Decompositi on 1

},{1 S H !

8/8/2019 Parser s

45/47


)(aab F

{})( !aa F

B X pThere is no producti on of f orm

aa b prefi x suffi x}{)( B b F !


Decompositi on 2

}{2 !H

},{}{},{)( 21 B S B S HHaab F !!!

8/8/2019 Parser s

46/47


a

{A}

a

{A}

b

{B}

b

{B}

b

{B}aa {}

ab {S,B}

bb {A}

bb {A}

aab {S,B}

abb {A}

bbb {S,B}

aabb {A} abbb {S,B}aabbb

{S,B}


!)(aabbb F

)(G Laabbb

)(w F S Since

8/8/2019 Parser s

47/47

P f B h LSU 47

Approximate ti me complexity :)|(||)||(| 32 wOwwO !

Number of substrings

Number of Prefi x-suffi x

decompositi onsf or a string

Documents

Parser s