Embed Size (px)
Citation preview
Lexical Analysis IIIRecognizing Tokens
Lecture 4CS 4318/5331Apan Qasem
Texas State University
Spring 2015
Announcements
• Assg 1 due this Friday at 11:59 PM
• Test instances on github
• No lecture at RRC this week
Lexical Analysis
int main() {
int i
for (i = 0; i < MAX; i++)
printf(“Hello World”);
}
Scanner
<KEYWORD,int> <ID,main> <OP,(> <OP,)> <OP,{> <KEYWORD,int> <ID,i>
<SEP,;> <KEYWORD, for> <OP,(> <ID,I> <OP,=> <CONST,0> <SEP,;>
<ID,i> <ID,<> <ID,MAX> <SEP,;> <ID,I> <OP,++> <ID,printf> <OP,(> <OP,“> <STR, Hello World> <OP,”> <OP,)> <SEP,;> <OP,{>
What do we do if we encounter a missing semi-colon?
Nothing!
Lexical Analysis
int main() {
int i;
for (i = 0; i < MAX; i++)
abcprintf(“Hello World”);
}
Scanner
<KEYWORD, int> <ID,main> <OP,(> <OP,)> <OP,{> <KEYWORD,int> <ID,i>
<SEP,;> <KEYWORD,for> <OP,(> <ID,I> <OP,=> <CONST,0> <SEP,;> <ID,i>
<ID,<> <ID,MAX> <SEP,;> <ID,I> <OP,++> <ID,printf> <OP,(> <OP,“>
<STR, Hello World> <OP,”> <OP,)> <SEP,;><OP,}>
What do we do if we encounter an undefined function name?
Nothing!
Lexical Analysis
int main() {
int i;
for (i = 0; i < MAX; i++)
abcprintf(“Hello World”);
}
Scanner
<KEYWORD, int> <ID,main> <OP,(> <OP,)> <OP,{> <KEYWORD,int> <ID,i>
<SEP,;> <KEYWORD,for> <OP,(> <ID,I> <OP,=> <CONST,0> <SEP,;> <ID,i>
<ID,<> <ID,MAX> <SEP,;> <ID,I> <OP,++> <ID,abcprintf> <OP,(> <OP,“>
<STR, Hello World> <OP,”> <OP,)> <SEP,;><OP,}>
What do we do if we encounter an undefined function name?
Nothing!
Lexical Analysis
intmain(){inti;for(i=0;i<MAX;i++)printf(“Hello World”);}
Scanner
<ID,intmain> <OP,(> <OP, )> <OP,{> <KEYWORD,inti> <SEP,;>
<KEYWORD,for> <OP,(> <ID,i> <OP,=> <CONST,0> <SEP,;> <ID,i>
<ID,<> <ID,MAX> <SEP,;> <ID,i> <OP,++> <ID,printf> <OP,(> <OP,“>
<STR, Hello World> <OP,”> <OP,)> <SEP,;><OP,}>
Legal C program? Passes Scanner?
No Yes
Lexical Analysis
intmain(){inti;for(i=0;i<MAX;i++)printf(“Hello World”);}
Scanner
<ID,intmain> <OP,(> <OP, )> <OP,{> <ID,inti> <SEP,;>
<KEYWORD,for> <OP,(> <ID,i> <OP,=> <CONST,0> <SEP,;> <ID,i>
<ID,<> <ID,MAX> <SEP,;> <ID,i> <OP,++> <ID,printf> <OP,(> <OP,”>
<STR, Hello World> <OP,”> <OP,)> <SEP,;><OP,}>
Legal C program? Passes Scanner?
No Yes
Lexical Analysis
int main() {
int %$*&i;
for (i = 0; i < MAX; i++)
printf(“Hello World”);
}
Scanner
What’s an illegal C program at the scanner phase?
Very Few!C/C++ has become too large!
<KEYWORD,int> <ID,main> <OP,(> <OP, )> <OP,{> <KEYWORD,int> <OP,%>
<ID,$> <OP,*> <OP,&> <ID,i> <SEP,;> <KEYWORD, for> <OP,(> <ID,i>
<OP,=> <CONST,0> <SEP,;> <ID, i> <ID,<> <ID,MAX> <SEP,;> <ID,I> <OP,++> <ID,printf> <OP,(> <OP,“> <STR, Hello World> <OP,”> <OP,)> <SEP,;><OP,{>
Breaking Down Lexical Analysis Further …
1. Specify patterns for tokens• Look at language description and identify the types of
tokens needed for the language• usually trivial
• Use regular expressions to specify a pattern for each token• patterns for some tokens are trivial
2. Recognize patterns in the input stream and generate tokens for the parser
Recognizing Tokens
• We can specify the regular expression while
for the while keyword in C
• How do we recognize it if we see it in the input stream?• Essentially a pattern-matching algorithm
Code for Recognizing while
if (nextchar() == ‘w’) if (nextchar() == ‘h’) if (nextchar() == ‘i’) if (nextchar() == ‘l’) if (nextchar() == ‘e’)
return KEYWORD_WHILE; else
// do something else // do something else // do something else // do somethingelse
// do something
This approach works for more complex REs as well while (nextchar() == ‘a’ || …)
Need to decide what to do for strings like when
Need to account for strings like whileabc
Need to account for strings like abcwhile
Can we generate this code automatically?
Code for Recognizing while
if (nextchar() == ‘w’) if (nextchar() == ‘h’) if (nextchar() == ‘i’) if (nextchar() == ‘l’) if (nextchar() == ‘e’)
return KEYWORD_WHILE; else
// do something else // do something else // do something else // do somethingelse // do something
Each ‘if clause’ represents a state
The state is determined solely based on what we have seen so far in the input stream
No need to go back and rescan input
At each state we make a decision to move to a new state based on the next input symbol
This is exactly the idea behind (deterministic) finite state
machines
Recognizing Tokens
General idea• Consume a character from the input stream
• Based on the value of the character move to a new state • If the character just consumed
• produces a valid token and no more characters to consume then DONE
• leads to a valid token, move to a valid state• produces an invalid token go to error state and finish
• Repeatabove recognizes one token
Recognizing Tokens
• Need to construct a recognizer based on regular expressions
• A recognizer for a regular expression is a machine that recognizes the language described by the RE
• Given an input string constructed from the alphabet, the recognizer will
• Say “yes” if the string is in the language (ACCEPT)• Say “no” if the string is not in the language (REJECT)
• Implications • Must produce a yes or no answer on every input• Cannot say yes when the string is not in the language (false
positives)
RE and DFA
For every RE there is a recognizer that recognizes the corresponding RL
If you build it … it will be recognizable!
The recognizers are called deterministic finite automata (DFAs)
Kleene’s Theorem (1952)
Deterministic Finite Automata
Formal mathematical construct • Abstract state machines that can recognize regular
languages• A set of states with transitions defined on each input
symbol on every state• Formal definition in Text (Section 2.2.1) • Convenient to reason about DFAs using state transition
diagrams
DFA Diagram
s0 s2s1 s3i n t
E
initial state
input
error state
final state
error states sometimes implicit
only one initial state
can have multiple final states
i
n t
Acceptance Criteria for DFAs
• A DFA accepts a string if and only if the DFA ends up in a final state after consuming all input symbols
• Implications • A DFA built to recognize int will _______ intmain
• A DFA built to recognize intmain will _______ int
reject
reject
Easy fix if we want the machine to recognize int AND intmain
DFA Example : if
s0 s1i f s2
DFA Example: int | if
s0 s1i f s3
s2
n
s4
t
DFA for if | int
s0
s1if s3
s4ns2i t s5
Non-determinism
DFA Example : Integers
Σ = {0-9}Digit : 0|1|2|3|… |9 Integer : 0 | (1|2|3|… |9)(Digit)*
s0 s2
E
s11-9
0-9
0
REs and DFAs
every RL has a DFA that recognizes it and every DFA has a corresponding RL
there are algorithms that allow us to convert an RE to a DFA and vice versa
we can automate scanning!
to convert REs to DFAs we need to first look at non-deterministic finite
automata (NFA)
Non-determinism
DFAs do not allow non-determinism• Must have a transition defined on every state
on every possible input symbol• Cannot move to a new state without
consuming an input symbol• Cannot have multiple transitions on the same
input symbol
NFA
• DFAs with transitions
• To run NFAs, start at the initial state and guess the right transition at each step• Always guess correctly• If some sequence of correct guesses leads to a
final state then accept
Sounds dubiousBut works!
NFA for if | int
s0
s1if s3
s4ns2
i t s5
NFA, multiple transitions on i in state s0
NFA and DFA
• Although NFAs allow non-determinism it has been shown that NFAs and DFAs are equivalent!
Scott and Rabin (1959)
• DFAs are just specialized forms of NFAs• NFAs and DFAs both recognize the same set of languages • Can simulate a DFA with an NFA• Can construct corresponding DFAs for any NFA
• Implication• For every RE there is also an NFA
Relatively easy to construct an NFA from an RE
RE to NFA : Empty String
1. is a regular expression that denotes { }, the set that contains the empty string
s0 s1
RE to NFA : Symbol
2. For each , a is a regular expression denoting {a}, the set containing the string a.
s0 s1a
RE to NFA : Union
3. r | s is an RE denoting L(r) U L(s) e.g., RE = a | b L(RE) = {a, b}
s0 s1b
s0 s1a s1 s3a
s2 s4b
s5s0
RE to NFA : Concatenation
4. rs is an RE denoting L(r)L(s) e.g., RE = ab L(RE) = {ab}
s0 s1b
s0 s1a
s1 s3s0 a bs2
RE to NFA : Closure
5. r* is an RE denoting L(r)* e.g., RE = a* L(RE) = { , a, aa, aaa, aaaa, …}
s1 s3s0 a s2s0 s1a
RE to NFA
• The algorithm for converting REs to NFAs is known as Thompson’s construction• Repeated application of the five conversion
rules!• Named after Ken Thompson (1968)
Example : NFA for a(b|c)*
Work inside parentheses b|c
s0 s1c
s0 s1b
s0 s5
Example : NFA for a(b|c)*
Work inside parentheses b|c
s2 s4c
s1 s3b
s0 s5
Adjust final statesRename states
Example : NFA for a (b|c)*
Step 3: * (closure)
(b | c)*
s1 s3b
s2 s4c
s5s0 s5s0
Example : NFA for a (b|c)*
Step 3: * (closure)
(b | c)*
s2 s4b
s3 s5c
s6s0 s7s1
Example : NFA for a (b|c)*
Step 4: concatenation
s4 s5b
s6 s7c
s8s1 s9s3s2s0a
Cycle of Construction
RE
MinimizedDFA
DFA
NFA
Code
Thompson’s Construction
SubsetConstructionHopcroft’s
Algorithm
