1

Automata and Formal Languages

Winter 2009-2010

Yacov Hel-Or

2

What this course is all about

• This course is about mathematical modelsof computation

• We’ll study different “machine models” (finite automata, pushdown automata, Turing machine)

• … and characterize what they can compute

3

Why should I care ?

• To understand the limits of computationsSome problems require more resources to compute

and others cannot be computed at all.

• To learn some programming tools Automata show up in many different settings:

compilers, text editors, hardware design, communication protocols, program proofing,…

• To learn to think in a formal way about computing

4

Computability and Complexity

• Are there any problems that can not be solved by a (very powerful) computer?

• What makes some problems computationally hard and other easy?

• Can we partition the problems into classes such that problems in one class share the same computational properties?

Complete answers: next semester.

Introduction, basic tools, models, intuition: this semester.

5

This Course: Formal computational models

• The basic computational model : Finite State Automata

• Additional models:– Pushdown automaton

– Turing machine

• Formal languages– Regular languages

– Context free languages

6

Administration

• Course Home page: http://www1.idc.ac.il/toky/Automata

• Lecture notes will appear at the course web-page the day before the lecture (the latest).

• Proofs, examples, technical details will usually be presented on the board.

• No recitations, part of lecture time will be dedicated to solving problems.

• Grade calculation: Exam 70%, HW 30%

• Exam: Must pass the exam (60) in order to have the HW component.

7

Administration (cont.)

• HW: Will be given every week.

• Submission: Wednesdays before your lecture.

• Appeals: No more than two weeks after return.

• Grader + TA: Ilit Raz (raz.ilit@post.idc.ac.il)

• Newsgroup: news://news.idc.ac.il/automata

• Book: Sipser (see web-page for details).

8

Unit 1

IntroductionMathematical Background

Reading: Sipser, chapter 1

Today

• Set theory (review?)

• Logic, proofs (review?)

• Words and their operations:

• Languages and their operations:

9

wwwww i ,,, *

21 LLLLL i ,,, *

21

10

Set TheoryA Set is a group of objects.The objects are called elements.

Notation:

1. By listing the elements. Examples: {3,5,7} ,{Alice, Bob, {1,2} } (for finite sets only)

2. By providing a rule. Examples:

{x | x is an odd integer between 2 and 8} ;

{x | name of a student in the Automata class}

A name of a set is usually a capital letter of theEnglish alphabet (A, B, C) or a capital letter withan index (X1 , X2 , X3)

איבר, קבוצה

11

The Empty Set

• A set with no elements is called an empty set.

• Notation – or { }

Example:

S - a set of all odd numbers that can be dividedby two without any remainder.

S is an empty set.

S = {xN | x is odd and x mod 2=0}=

הקבוצה הריקה

12

Membership in a set

• s S , means that an element s is a memberof the set S

• b S, means that an element b is not a member of the set S

Examples: 7 {21,7,30} and 8 {21,7,30}

Let N be the set of natural numbers.2 N and 3.2 N

איבר בקבוצה

13

Subsets

• Given two sets A and B we say that A is asubset of B, if each element of A is also anelement of B.

• The notation: A B• Formally: A B x A x B

Example:A set of natural numbers N is a subset of a

set of all real numbers R.N R

תת קבוצה

מוכל

14

Proper Subset

• Given two sets A and B we say that A is aproper subset of B if each element of A isalso an element of B and there exists atleast one element in B that does not belongto A.

• The notation: A B

Example:{1,3} {1, 2, 3, 8}

תת קבוצה ממש

מוכל ממש

15

Some Facts

• Each set is a subset of itself:

A A

• The empty set is a subset of every set:

A

16

More on Sets

• Two groups of elements that have the same elements but in different order form the same set.

Example: If A = { 1,2,3,4}, B = { 2,1,4,3} then A = B

• Repetitions in a set are irrelevant

Example: {1,2,3,4,2} = {1,2,3,4}

17

More on Sets

• The cardinality of a set is the number ofelements in the set. Notation: |A|

Example: Let A = { 1,2,4,8,16}, then |A|=5

• A set can be:

– Finite A={Even integers smaller than 100}

– Infinite A={Even integers dividing 7 with no reminder}

עצמת הקבוצה

18

Operations on sets• union

• intersection

• complement

• difference

A visual model, called Venn diagram can be used.

A

19

Venn Diagram

Start-t End-z Start-j

terrifictheory topaz jazz

20

Union

• A binary operation

• The notation : A B

• Formally:

A B = { x | xA or xB }

A B

אחוד

21

Intersection

• A binary operation

• The notation: A B

• Formally:

A B = { x | xA and xB }

A B

חיתוך

22

Complement

• An unary operation

• The notation : ~A

• Formally:

~A = { x | x A}

A

משלים

23

Set Difference

• An binary operation

• The notation : A-B

• Formally: A-B = { x | xA and x B}

Example: A = { 1,2,3 }, B={3,4,5}, A-B={1,2}

A B

חיסור

24

Power Set

• The power set is the set of all subsets of a given set.

• Notation: P(S) or 2S is a power set of S

• Note, that sets may appear as elements of other sets.

• The cardinality of a power set is: 2|S|

(why?)

Example:S = { 1,2 }P(S) = 2S = { ,{1},{2},{1,2}}

קבוצת החזקה

25

Sequences and Tuples

• A sequence of objects is a list of objects in some order.

• The notation : (7,21,57,…)

• Unlike sets, the order and repetitions in the sequence do matter, thus

(7,21,57) (7,57,21) and (7,21,57,57) (7,21,57)

• A sequence with k elements is a k-tuple

Example: (7,21,57) is a 3-tuple.

סדרית

איה-ק

26

Cartesian product

• A binary operation

• The Cartesian product (or cross product) is a set of all pairs (2-tuple) where the first element of the pair is in A and the second element of the pair is in B.

• The notation: AB

• A Cartesian product of a set with itself:

AAA… (k times)=Ak

Example:

A = {2,3}, B = {b,c}, AB ={ (2,b), (2,c), (3,b), (3,c) }

מכפלה חיצונית

Summary: sets

• ,{ } empty set

• a A membership

• |A| cardinality

• AB subset

• AB union

• AB intersection

• ~A complement

• A-B set difference

• 2A power set

• (a,b,..) a sequence (k-tuple)

• AxB cartesian product 27

28

Letters and Alphabet

• Any finite set of letters (symbols) is called an alphabet.

• Notation: alphabet- ; Letter (symbol)-

Examples:

1 = { 0,1 } , 1=0 ; 2=1;

2 = { a,b,c,d,e,… }, 1=a ; 2=b ; 3=c ; …

3 = { 0,1,x,y,z}

אותיות , אלפבית

29

Strings (and words)

• A string (word) over some alphabet is a finite sequence of letters from the alphabet.

Example: = {0, 1}, w = 101

• The length of a word, w, denoted |w|, is the number of letters in it.

Example:

w1 = abracadabra; |w1| = 11

w2 = 001011 ; |w2| = 6

סדרית, מילה, מחרוזת

30

More on Strings• An empty word is a string without letters.

• The notation of an empty word is

• || = 0

• The number of occurrences of some letter in word w is denoted by #(W)

Example:

Let w=aaba, then #a(w)=3, #b(w)=1, #c(w)=0

המילה הריקה

31

Operations on Strings

• reverse

• sub-string

• prefix

• suffix

• concatenation

32

Reversing strings

• A reverse string is a string in which all letters are written in the opposite order

• Notation: wR

Examples:w = 10, wR = 01

s = abcb, sR = bcba

A palindrome: a string w such that w=wR.

Examples: aba, 010010,

היפוך

33

Sub-strings

• A sub-string is a subsequence of consecutiveletters from a string

Example: Let w = 101. All sub-strings of w are:

B(w) = { , 1, 0, 10, 01, 101 }.

Note: 11 B(w)

תת מחרוזת

34

Prefixes• A prefix is a sub-string which starts from

the first letter of the word (or an empty word).

• A proper prefix of a string is a prefix that is not equal to the complete word.

Example: Let w = acdb. The prefixes of w are { , a,ac,acd,acdb}. acdb is not a proper prefix.

ממש רישא, רישא

35

Suffixes• A suffix is a substring ending with the last

letter of the word or an empty word.

• A proper suffix of the string is a suffix that is not the whole word.

Example: Let w = acdb. The suffixes of w are { ,b, db,cdb,acdb}. acdb is not a proper suffix

Note: The prefixes of w are the reversed suffixes of wR.

ממש סיפא, סיפא

36

Concatenation

• A binary operation (over two words)

• The concatenation of two words x and yplaces them one after the other such thatthe first word is a prefix and a second oneis a suffix.

• Notation: concatenation of two words x andy: xy

Example: Let x= 01, y=012, z=10

Then xy=01012, yx=01201, xyz=0101210

שרשור

37

Concatenation (Cont.)

• The result of concatenating a word with anempty word is the string itself. Forexample, let w=ab, then w = w = w = ab

• Concatenating an empty word to itselfresults in an empty word: = = …

38

Languages

• A language is a set of words (strings).

• A language can be finite or infinite.

• Notation: L (or with an index: Li)

• The language of all words over some alphabet Σ is denoted Σ* (sigma star).

• An empty language – A language with zero words .

שפה

השפה הריקה

39

Examples

Infinite languages:

L1= The language of all natural numbers over thedecimal alphabet.

L2 = The language of all even length words over thebinary alphabet.

L3 = The language of all strings over the binaryalphabet that ends with 0.

Finite languages:

L4 = {abc, bc}

L5 = The Language of natural numbers smaller than 5.

L6 = {words over Σ={0,1} whose binary value is as an oddnumber smaller than 325}

40

Operations over languages

• reverse

• concatenation

• iteration

• positive closure

• Kleene closure

41

Reverse Languages

• The reverse language – LR is the language with all reversed words in it.

Formally: LR ={ w | wR L}

Examples:

L = {abc, bc}; LR = {cba, cb}

L={0,00,0010}; LR = {0,00,0100}

L= all binary words ={0,1}* ; LR = L (why?)

שפה הפוכה

42

Concatenation• A concatenation of two languages is a language

in which each word is a concatenation of twowords - one from the first language and asecond from the second language.

• Formally: A◦B={w=ab| aA and b B}

Examples: L1= {ab, cd }; L2 = {00, 1} ; L3={0,10,110}

L1◦L2 = { ab00, ab1, cd00, cd1 }

L3◦L2 = { 000, 01, 1000, 101, 11000, 1101}

Note: the order is important (A◦B is different from B◦A )

שרשור שפות

43

Concatenation v.s Cartesian Product

• Do not confuse concatenation of languages with Cartesian Product of sets.

For example, let A = {0,00} then

A◦A = { 00, 000, 0000 } with |A◦A|=3,

AA = { (0,0), (0,00), (00,0), (00,00) } with |AA|=4

• What is the cardinality of |AxA| v.s. |A◦A| ?

44

Concatenation with special languages

• Concatenation with an empty language – the result is an empty language.

L◦ =◦L =

• Concatenation with a language that includesonly an empty word – the result is thelanguage itself.

L◦{} = {}◦L = L

45

Iterations

• K-th iteration – concatenation of the language with itself k times : L◦L◦…L◦L (k times)

• Notation: LK

• Definition: L0 = {} (for each L!)

Examples: Let L1={ ,00, 1} ; L2={01, 1}

L12= { ,00, 1, 0000, 001, 100, 11}

L22 = {0101, 011, 101, 11}

L23 = {010101, 01011, 01101, 0111, 10101, 1011, 1101, 111}

46

Kleene (star) closure

• A Kleene closure is the union of all possible iterations of L:

• Notation: L*

Examples:

1. Let L={a}, then L*= { , a, aa, aaa, aaaa …. }

2. Let L={0,1} then L*={all binary words}

Note: for all L , L*

...}{ 210

0

LLLLL i

i

קליןסגור של

47

Positive closures

• A positive closure is a union of all positive iterations of L, not including the zero iteration:

• Notation: L+

Examples:

1. Let L={a}, then L+= { a, aa, aaa, aaaa …. }

2.Let L={0,1} then L+={all binary words of length > 0}

...}{ 21

1

LLLL i

i

סגור חיובי

48

Problems concerning formal languages

• Is a given word a member of the language?

• Is the language infinite?

• Does the sequence of operations (steps) create (derive) a given word?

• Given a word and a sequence of basic steps does the result belongs to a given language?

• Given a grammar (a set of rules) what language does it create?

49

Practice Problems

1. Prove that L+ = L* if and only if L

2. Prove that for any three languages

(L1 L2) ◦ L3 = L1 ◦ L3 L2 ◦ L3

3. For a given word w, define

L1 = {prefixes of w}; L2={suffixes of wR}

Prove that L1=L2R

Answers: In class.

50

Types of Proofs

• Four main types:– direct proof (syllogism)

– proof by construction

– proof by contradiction

– proof by induction

51

Direct proof

axioms + theorems + rules of deduction theorems

All students at IDC are nice.

Danny is a student at IDC.

Danny is nice.

modus ponens:

ab, a b

ab, ~b ~a

52

Proof by Construction

prove by building a solution (algorithm, automaton)

Example:

Claim: There exists a set with 4 elements.

Proof: Here is such a set: A = {01, 001, 1, 1100}

53

Proof by Contradiction

The idea:• assume the opposite of the theorem• derive a contradiction

Example:Claim: There is an infinite number of integers.Proof: Assume the opposite, therefore, there

is some largest integer. Denote it N. But N+1is also an integer, and it is bigger than N – Acontradiction.

54

Another Example

If there are more girls than boys and every girl is dating a boy, there must be a boy that’s cheating

Proof:Assume no boy is cheating, that is, each boy dates at

most one girl, therefore, the total number ofdating girls is at most the number of boys. Sinceevery girl is dating a boy, the total number of girlsis less or equal the number of boys. Contradictingthe fact that there are more girls than boys.

Note: This is called THE PIGEONHOLE PRINCIPLE:If you put 6 pigeons in 5 holes then at least onehole will have more than one pigeon.

55

Induction has many appearances:

• Formal Arguments

• Loop Invariants

• Recursion

• Algorithm Design

Proof by Induction

56

Review: Induction

• Suppose– S(k) is true for fixed constant k (often k=0)

– S(n) S(n+1) for all n >= k

• Then S(n) is true for all n >= k

57

Proof By Induction

• Claim:S(n) is true for all n >= k

• Base:– Show S(n) is true for n = k

• Inductive hypothesis:– Assume S(n) is true for an arbitrary n

• Step:– Show that S(n+1) is true

58

Induction Example:Geometric Closed Form

• S(n)=a0 + a1 + … + an

• Prove S(n)= (an+1-1)/(a-1) for all a 1

• Proof:

– Base: S(0)=a0 = (a0+1 - 1)/(a - 1)=1

– Inductive hypothesis:

• Assume S(n)= (an+1 - 1)/(a - 1)

– Step (show true for n+1):

S(n+1)=a0 + a1 + … + an+1 = S(n) + an+1

= (an+1 - 1)/(a - 1) + an+1 = (an+1+1 - 1)/(a - 1)

59

Induction

• Another variation:– Basis: show S(0), S(1)

– Hypothesis: assume S(n) and S(n+1) are true

– Step: show S(n+2) follows

• Another variation:– Basis: show S(k)

– Hypothesis: assume S(n)

– Step: show S(n-1) follows

E N D

60