View
241
Download
0
Category
Tags:
Preview:
Citation preview
CodeGenerator
Translator Architecture
ParserTokenizer
string ofcharacters
(source code)
string oftokens
abstractprogram
string ofintegers
(object code)
A Tokenizing Machine
Another greatmachine!
Ready to Dispense?
In(Chars)
Out(Tokens)
Tokenizing Machine Continued…
Ready toDispense?
In Out
3: Tokenscome outhere
2: Light comes on
1: Charactersgo in here
Tokenizing Machine Continued… Type
BL_Tokenizing_Machine_Kernel is modeled by (
buffer : string of character ready_to_dispense : boolean
) constraint …
Initial Value (empty_string, false)
Tokenizing Machine Continued…
Operations m.Insert (ch) m.Dispense (token_text, token_kind) m.Is_Ready_To_Dispense () m.Flush_A_Token (token_text, token_kind) m.Size ()
A State-Transition View
not readyto dispense
readyto dispense
Insert
Flush_A_Token
Insert
Size
Size
Is_Ready_To_DispenseIs_Ready_To_Dispense
Dispense
Tokenizing BL Programs
Token Types KEYWORD IDENTIFIER CONDITION WHITE_SPACE COMMENT ERROR
A Very Useful Extensionprocedure_body Get_Next_Token ( alters Character_IStream& str, produces Text& token_text, produces Integer& token_kind ){
}
while (not self.Is_Ready_To_Dispense () and not str.At_EOS ()) { object Character ch; str.Read (ch); self.Insert (ch); } if (self.Is_Ready_To_Dispense ()) { self.Dispense (token_text, token_kind); } else { self.Flush_A_Token (token_text, token_kind); }
Another Useful Extensionprocedure_body Get_Next_Non_Separator_Token ( alters Character_IStream& str, produces Text& token_text, produces Integer& token_kind ){
}
self.Get_Next_Token (str, token_text, token_kind); while ((token_kind == WHITE_SPACE) or (token_kind == COMMENT)) { self.Get_Next_Token (str, token_text, token_kind); }
How Does Insert Work?
buffer: “PROGRAM”ready_to_dispense: false
buffer: “PROGRAM ”ready_to_dispense: true
#m:
m:
Here’s anothercharacter.
‘ ’
The Specification of Insert
procedure Insert ( preserves Character ch ) is_abstract; /*! requires self.ready_to_dispense = false ensures self.buffer = #self.buffer * <ch> and self.ready_to_dispense = IS_COMPLETE_TOKEN_TEXT (#self.buffer, ch) !*/
An Important Math Operation
math definition IS_COMPLETE_TOKEN_TEXT (
s: string of character
c: character
): boolean is
( s is in OK_STRINGS and
s * <c> is not in OK_STRINGS ) or
( <c> is in PREFIX (OK_STRINGS) and
s * <c> is not in PREFIX (OK_STRINGS) )
s is a complete“valid” token
c can start a “valid” token, buts*<c> doesn’t start a “valid” token
Other Math Definitions
OK_STRINGS ={s: string of character (IS_KEYWORD (s))} union{s: string of character (IS_IDENTIFIER (s))} union{s: string of character (IS_CONDITION_NAME (s))} union{s: string of character (IS_WHITE_SPACE (s))} union{s: string of character (IS_COMMENT (s))}
PREFIX (s_set) ={x: string of character (there exists y: string of character (x * y is in s_set))}
PREFIX Examples
s_set = {“abc”} PREFIX (s_set) =
{“”, “a”, “ab”, “abc”}
s_set = {“abc”, “de”} PREFIX (s_set) =
{“”, “a”, “ab”, “abc”, “d”, “de”}
Tokenizing Machine: Implementation
Obvious Representation Text buffer_rep Boolean token_ready
Insert (ch)? check if IS_COMPLETE_TOKEN_TEXT
(self[buffer_rep], ch), andset self[token_ready] accordingly
append ch at end of self[buffer_rep]
Tokenizing Machine: Implementation Continued…
Dispense (token_text, token_kind)? set token_text to all but the last
character of self[buffer_rep] set token_kind to the value of
WHICH_KIND (token_text) set self[token_ready] to false
Tokenizing Machine: Implementation Continued…
How do we “check if IS_COMPLETE_TOKEN_TEXT (self[buffer_rep], ch)”?
How do we determine“WHICH_KIND (token_text)”?
How do we do these things quickly?
Making Decisions Quickly
Keep track of the “state” of the buffer by adding one field to the representation: Text buffer_rep Boolean token_ready Integer buffer_state
Possible Buffer States
How many interestingly different buffer states do you think there may be?
Let’s start enumerating them…
Buffer States Continued… Initial state (empty buffer) How many states after inserting
the first character? ‘B’, ‘D’, ‘E’, ‘I’, ‘P’, ‘T’, ‘W’, ‘n’, ‘r’, ‘t’,
identifier (any other letter) white_space (‘ ’, ‘\n’, ‘\t’) comment (‘#’) error (any other character)
Buffer States Continued…
How many states after inserting the second character? “BE”, “DO”, “EL”, “EN”, “IF”, “IN”, “IS”,
“PR”, “TH”, “WH”, “ne”, “ra”, “tr”, identifier (any other id character)
white_space (‘ ’, ‘\n’, ‘\t’) comment (any other character but ‘\n’) error (any character that cannot start a
new “good” token)
A State Transition Diagram:Transitions Out of ‘empty’ Only
B
DE I
PT
W
n
r
t
identifierwhite_space
comment
error
empty
‘B’
‘D’
‘E’ ‘I’‘P’
‘T’
‘W’
‘n’
‘r’
‘t’
any otherletter
‘ ’,’\n’,’\t’‘#’
any othercharacter
Structure of Body of Insertcase_select (self[buffer_state]){ case empty: // case for buffer = empty_string case B: // case for buffer = “B” case D: // case for buffer = “D” case E: // case for buffer = “E” . . . case error: // case for buffer holding an error token}
A Simplified View
Buffer States EMPTY_BS ID_OR_KEYWORD_OR_CONDITION_BS WHITE_SPACE_BS COMMENT_BS ERROR_BS
The State Transition Diagram
EMPTY_BS
ID_OR_KEYWORD_OR_CONDITION_BS
COMMENT_BS
WHITE_SPACE_BS
ERROR_BS
‘ ’, ‘\n’, ‘\t’
‘#’
‘a’..’z’,‘A’..’Z’
any othercharacter
any characterexcept ‘\n’
‘ ’, ‘\n’, ‘\t’
‘a’..’z’,‘A’..’Z’,‘0’..’9’,
‘-’
any character except‘a’..’z’, ‘A’..’Z’, ‘ ’, ‘\n’, ‘\t’, ‘#’
Useful Private Functions Is_White_Space_Character (ch) Is_Digit_Character (ch) Is_Alphabetic_Character (ch) Is_Identifier_Character (ch) Can_Start_Token (ch) Id_Or_Keyword_Or_Condition (t) Buffer_Type (ch) Token_Kind (bs, str)
Recommended