Survey: String Matching with k Mismatches Moshe Lewenstein Bar Ilan University

Survey: String Matching with k Mismatches

Moshe Lewenstein Bar Ilan University

String Matching with k Mismatches

Landau – Vishkin 1986

Galil – Giancarlo 1986

Abrahamson 1987

Amir - Lewenstein - Porat 2000

Exact String Matching

Input: T = t1 . . . tn

P = p1 … pm

Output: All locations i of T where P appears Example:

P = A B C A A B T = A B A B C A A B C A A B C A A B A A…


P = p1 … pm


P = A B C A A B T = A B A B C A A B C A A B C A A B A A… 3



P = p1 … pm


P = A B C A A B T = A B A B C A A B C A A B C A A B A A… 3 7



P = p1 … pm


P = A B C A A B T = A B A B C A A B C A A B C A A B A A… 3 7 11



P = p1 … pm


P = A B C A A B T = A B A B C A A B C A A B C A A B A A…

Answer: {3,7,11,..}



Problem: Matching not exact in applications of:

• Computational Biology

• Musicology

• Text Editing

• Meteorology

• etc.

Need other definitions of string matching!

Approximate String Matching

Idea: Find all text locations where distance from pattern is sufficiently small.

distance metric: HAMMING DISTANCE

Let S = s1s2…sm

R = r1r2…rm

Ham(S,R) = The number of locations j where sj rj

Example: S = ABCABC R = ABBAAC

Ham(S,R) = 2

String Matching with Mismatches


P = p1 … pm

Output: For each i in T Ham(P, titi+1…ti+m-1)

Example:

P = A B B A A C T = A B C A A B C A C…



P = p1 … pm


Example:

P = A B B A A C T = A B C A A B C A C… 2

Ham(P,T1) = 2



P = p1 … pm


Example:

P = A B B A A C T = A B C A A B C A C… 2, 4

Ham(P,T2) = 4



P = p1 … pm


Example:

P = A B B A A C T = A B C A A B C A C… 2, 4, 6

Ham(P,T3) = 6



P = p1 … pm


Example:

P = A B B A A C T = A B C A A B C A C… 2, 4, 6, 2

Ham(P,T4) = 2



P = p1 … pm


Example:

P = A B B A A C T = A B C A A B C A C… 2, 4, 6, 2, …

Input: T = t1 . . . tn, P = p1 … pm


Output: Every i in T s.t. Ham(P, titi+1…ti+m-1) k

Example: k = 2


Input: T = t1 . . . tn, P = p1 … pm



Example: k = 2


Input: T = t1 . . . tn, P = p1 … pm



Example: k = 2

P = A B B A A C T = A B C A A B C A C… 2, 4, 6, 2, … Y,N,N,Y,…

Naïve Algorithm(for counting mismatches or k-mismatches problem)

Running Time: O(nm) n = |T|, m = |P|

- Goto each location of text and compute hamming distance of P and Ti

The Kangaroo Method(for k-mismatches)

Landau – Vishkin 1986

Galil – Giancarlo 1986

Trie

• A tree representing a set of strings.

ab

c

e

e

f

d b

f

e g

{ aeef ad bbfe bbfg c }

Trie (Cont)

• Assume no string is a prefix of another

ab

c

e

e

f

d b

f

e g

Each string corresponds to a leaf.

Compressed Trie • Compress unary nodes, label edges by strings

ab

c

e

e

f

d b

f

e g

a

bbf

c

eefd

e g

Suffix tree

Suffix tree of string s:a compressed trie of all suffixes of s

Prefix-free: add a special character, say $, at the end of s

Suffix tree (Example) Let s = abab, a suffix tree of s is a compressed trie of all suffixes of s=abab$

{ $ b$ ab$ bab$ abab$ }

ab

ab$

ab$

b

$

$

$

Suffix Tree properties

- Succint in space - O(n).

- Can be built in O(n) time. McCreight, Weiner,

Ukkonen, Farach-Colton

b

12

ab

a

b$

a

b$

3

$ 4

$

5

$

Exact string matching

12

ab

ab

$

ab$

b

3

$ 4

$

5

$

Given a pattern P = ab we traverse the tree according to the pattern.

s=abab$


12

ab

ab

$

ab$

b

3

$ 4

$

5

$

Leaves correspond to locations of appearance!

s=abab$ 1 3


12

ab

ab

$

ab$

b

3

$ 4

$

5

$

Prepare Tree: O(n) time

Find matches: O(m + occ) time occ = # of matches

s=abab$ 1 3

Lowest common ancestors

A lot more can be gained from the suffix tree if we preprocess it so that we can answer LCA queries on it

Why?The LCA of two leaves represents the longest common prefix (LCP) of these 2 suffixes

s = abbaab$

1

3

a

b

aab

ab$

b

5

$

2

b

4

b$a

6

$

7

$

b

$

aaa

b$


1

3

a

b

aab

ab$

b

5

$

2

b

4

b$a

6

$

7

$

b

$

aaa

b$

s = abbaab$ aab$


1

3

a

b

aab

ab$

b

5

$

2

b

4

b$a

6

$

7

$

b

$

aaa

b$

s = abbaab$ aab$ abbaab$


1

3

a

b

aab

ab$

b

5

$

2

b

4

b$a

6

$

7

$

b

$

aaa

b$

s = abbaab$

aab$ abbaab$

LCA/LCP propertiesa

1

3

b

aa

b

ab$

b

5

$

2

b

4

b$

a6

$

7

$

b

$

aa

ab

$

Preprocesssing time : O(n)

Query Time: O(1)

Harel & Tarjan 1984, Schieber & Vishkin 1988, Berkman & Vishkin 1993


- Create suffix tree for: s = P#T

-Check P at each location i of T by kangrooing

Example:

P = A B A B A A B A C A BT = A B B A C A B A B A B C A B B C A B C A … i




Example:





Example:





Example:





Example:





Example:





Example:





Example:





Example:



Preprocess:

Build suffix tree of both P and T - O(n+m) timeLCA preprocessing - O(n+m) time

Check P at given text location

Kangroo jump till next mismatch - O(k) time

Overall time: O(nk)

a b a c c a c b a c a b a c c P =

a b b b c c c a a a a b a c b ......T =

Boolean Convolutions (FFT) Method


a b a c c a c b a c a b a c c a-mask




a b a c c a c b a c a b a c c a-mask

Pa 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0




Pa 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0




Pa 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0


a b b b c c c a a a a b a c b ......not-amask



Pa 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0


a b b b c c c a a a a b a c b ......not-amask

0 1 1 1 1 1 1 0 0 0 0 1 0 1 1 Tnot a



Pa 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0


0 1 1 1 1 1 1 0 0 0 0 1 0 1 1 Tnot a



Pa 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0


0 1 1 1 1 1 1 0 0 0 0 1 0 1 1 Tnot a

Multiply Pa and Tnot a to count mismatches (use FFT)

Pa 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0

0 1 1 1 1 1 1 0 0 0 0 1 0 1 1 Tnot a ... ...



Pa 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0


0 1 1 1 1 1 1 0 0 0 0 1 0 1 1 Tnot a

Multiply Pa and Tnot a to count mismatches (use FFT)

Pa 1 0 1 0 0 1 0 0 1 0 1 0 1 0 0

0 1 1 1 1 1 1 0 0 0 0 1 0 1 1 Tnot a ... ...



Running Time: One boolean convolution - O(n log m) time

# of matches of all symbols - O(n| | log m) timeΣ


Counting Method

Input: Text: T = t1…tn

Pattern: P = p1…pm

Max # of allowed mismatches: k

Assumption: Each pattern element is distinct

a b c d e f g h

b g d e f h d c c a b g h h ...

...

...

...

Count matches (instead of mismatches)

P =

T =

counter

increment

O(n log m) Algorithm

Frequent Symbol: a symbol that appears at least times in P.

Case 1: At least frequent symbols.

- Consider first frequent symbols.

- For each of them construct a mask for first appearances.

k2

k

k

k2

k

We distinguish between two cases:

Case 2: Less than frequent symbols.k

Case 1: At least frequent symbols.k

Example of Masked Countingk = 4, = 4k2


a b a c c a c b a c a b a c c


a-mask

c-mask

a b b b c c c a a a a b a c b ......

T =

use a-mask





a-mask

c-mask


T = a b a c c a c b a c a b a c c

d

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ...... 1counter





a-mask

c-mask


T = a b a c c a c b a c a b a c c

d

0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 ...... 1counter

Counting Stage:

Run through text and count occurrencesof all marks.

Time: O(n ).k

For location i of T, if counteri < k then no match at location i.

Why? The total # of elements in all masks is 2 = 2k.

Important Observations:

1) Sum of all counters 2 n 2) Every counter whose value is less than k already has more than k errors.

k k

k

How many locations remain?

Sum of all counters: 2n

Value of potential matches > k

k

kn

kkn 22

The Kangaroo Method.

How do we check these locations?

Use

Kangaroo Method Time: O(k) per location

Overall Time: O( ) = O( )kkn kn

# of potential matches:

Case 2: X frequent symbols, x < k

a) Count all matches of frequent symbols - one boolean convolution per symbol.

k

b) For non-frequent symbols, build full masks.

Time: O(x n log m) = O( n log m)

Symbol non-frequent appears < 2 in P mask size < 2kk

Count time: O(n )k

So, Case 2 is O(n log m)k

Overall Algo. Time: O(n log m)k

c) Add results of a) & b) and get total number of matches at every text location.

Time:a) O(n log m)b) O(n )c) O(n)

k

k

Additional Points

1. O(n log k)k

mknkn log

3

mk 31

For there is a linear time

algorithm - O( )

2. O( n )kk log

Better tradeoff:

Define frequent symbol > kk log

O( ) time algorithmmknkn log

3

Outline:

1. Find 2k special substrings of pattern.

2. Construct forest data structure combining info of special pattern substrings and text.

3. Use local counting arguments and quick queries to forest data structure to prune candidates.

4. Use kangaroo method to check leftover potential candidates.

k-Mismatches and Matrix Multiplication

“Or-And” matrix multiplication:

AxB = C, cij = aik bkj nk 1

Pattern all-mismatch problem: Find all text locations where the pattern mismatches at every character.

Indyk: If there is an algorithm faster than O(n ) for the Pattern all-mismatch problem then there is a new method for solving “Or-And” matrix multiplication faster than O(n3)

m

OPEN PROBLEMS

Hamming Distance in time:O(n log m)

Edit Distance?

Other metrics?

Documents

Survey: String Matching with k Mismatches Moshe Lewenstein Bar Ilan University