1 Efficient Merging and Filtering Algorithms for Approximate String Searches Jiaheng Lu, University...

Efficient Merging and Filtering Algorithms

for Approximate String Searches

Jiaheng Lu,

University of California, Irvine

Joint work with Chen Li, Yiming Lu

2Chen Li, Jiaheng Lu, Yiming Lu Chen Li, Jiaheng Lu, Yiming Lu 2

Example: a movie database

Star Title Year GenreKeanu Reeves The Matrix 1999 Sci-Fi

Samuel Jackson Iron man 2008 Sci-Fi

Schwarzenegger The Terminator 1984 Sci-Fi

Samuel Jackson The man 2006 Crime

Find movies starred Schwarrzenger.

Data may not clean

Keanu Reeves

Samuel Jackson

Schwarzenegger

Relation R Relation S

Data integration and cleaning:

Keanu Reeves

Samuel L. Jackson

Schwarzenegger

Problem definition: approximate string searches

Schwarzenger

Samuel Jackson

Keanu ReevesStar

Query q:

Collection of strings s

Search

Output: strings s that satisfy Sim(q,s)≤δOutput: strings s that satisfy Sim(q,s)≤δSim functions: edit distance, Jaccard Coefficient and Cosine similaritySim functions: edit distance, Jaccard Coefficient and Cosine similarity

SchwarrzengerSchwarrzenger

Outline Problem motivation Preliminaries

Grams Inverted lists

Merge algorithms Filtering techniques Conclusion

String Grams q-grams

(un),(ni),(iv),(ve),(er),(rs),(sa),(al)

For example: 2-gram

u n i v e r s a l

Inverted lists Convert strings to gram inverted lists

id strings01234

richstickstichstuckstatic

2-grams

atchckicristtatituuc

201 30 1 2 4

41 2 433

Main ExampleQuery

Data Grams

stick (st,ti,ic,ck)

count >=2

id strings

0 rich

1 stick

2 stich

3 stuck

4 static

1,2,3,4

ed(s,q)≤1

0,0,1,2,41,2,4

Candidates

Problem definition:

Find elements whose occurrences ≥ T

Ascending

MergeMerge

Example T = 4

Result: 13

11Chen Li, Jiaheng Lu, Yiming Lu Chen Li, Jiaheng Lu, Yiming Lu

Contributions

Three new merge algorithms

New finding: wisely using filters

Outline Problem motivation Preliminaries Merge algorithms

Two previous algorithms Our proposed three algorithms

Filtering techniques Conclusion

Five Merge Algorithms

HeapMerger[Sarawagi,SIGMOD

MergeOpt[Sarawagi,SIGMOD

Heap-based Algorithm

Min-heap

Count # of the occurrences of each element by a heap

Push to heap ……

MergeOpt Algorithm

Long Lists: T-1 Short Lists

Binary

search

Example of MergeOpt [Sarawagi et al 2004]

Count threshold T≥ 4

Long Lists: 3Short Lists: 2

Can we run faster?

HeapMerger MergeOpt

# of occurrences# of occurrences

Increment by 1

Increment by 111

String idsString ids

Result!Result!

HeapMerger MergeOpt

Min-heap ……Pop T-1

Jump Greater or

equals

Greater or

equals

Example of MergeSkip

minHeap10

JumpJump

15151515

13131313

17171717

Skip is safe

Min-heap ……

# of occurrences of skipped elements ≤T-1

HeapMerger MergeOpt

Long Lists Short Lists

Binary

searchMergeSkip

How many lists are treated as long lists?

Short ListsMerge

Long ListsLookup

Decide L valueA good balance in the tradeoff:

# of long lists = T / ( μ logM +1)

Experimental data sets

DBLP data IMDB data Google Web corpus

Performance (DBLP)

DivideSkip is the best one

# of access elements (DBLP)

DivideSkip is the best one

Outline Problem motivation Preliminaries Merge algorithms Filtering techniques

Length, positional filters Filter tree

Conclusion and future work

Length Filtering

Ed(s,t) ≤ 2

Length: 19Length: 19

Length: 10Length: 10

By length only!

Positional Filtering

Ed(s,t) ≤ 2

(ab,1)

(ab,12)

Filter tree

Length level

Gram level

Position level

Inverted list512172844

2 n1 3

… zy zzabaa

Surprising experimental results (DBLP)

No filter (ms)

Length (ms)

Length+Pos (ms)

DivideSkip 2.23 0.76 1.96

Why adding position filter increases the running time?Why adding position filter

increases the running time?

Filters fragment inverts lists

Applying filters

MergeMergeMergeMerge MergeMerge MergeMerge

(1)Tree traversal

(2)More merging

(1)Tree traversal

(2)More merging

Saving:

reduce total lists size

Saving:

reduce total lists size

Conclusion

Three new merge algorithms We run faster

Interesting finding:

Do not abuse filters!Do not abuse filters!

Related work

Approximate string matching

[Navarro 2001]

Varied length Grams

[Li et al 2007]

Fuzzy lookup in

References1. [Arasu 2006] A. Arasu and V. Ganti and R.

Kaushik “Efficient Exact Set-similarity Joins” in VLDB 2006

2. [Chaudhuri 2003] S. Chaudhuri ,K Ganjam, V. Ganti and R. Motwani “Robust and Efficient Fuzzy Match for online Data Cleaning” in SIGMOD 2003

3. [Gravano 2001] L. Gravano, P.G. Ipeirotis, H.V. Jagadish, N. Koudas, S. Muthukrishnan and D. Srivastava “Approximate string joins in a database almost for free” in VLDB 2001

References4. [Li 2007] C. Li, B Wang and X. Yang

“VGRAM:Improving performance of approximate queries on string collections using variable-length grams ” in VLDB 2007

5. [Navarro 2001] G. Navarro, “A guided tour to approximate string matching” in Computing survey 2001

6. [Sarawagi 2004] S. Sarawagi and A. Kirpal, “Efficient set joins on similarity predicates” in ACM SIGMOD 2004

1 Efficient Merging and Filtering Algorithms for Approximate String Searches Jiaheng Lu, University...

Documents

Merging Spirit Presentation

Image merging

Merging Arrays Example

Merging Storms & Tornadogenesis

Merging Technology Ui2009

Yiming Tian i dr. Reguluiranje topologije VANET -a

Présentation Merging-Spirit

Advanced data management Jiaheng Lu Department of Computer Science Renmin University of China

DARTS: Differentiable Architecture SearchDARTS: Differentiable Architecture Search Hanxiao Liu CMU hanxiaol@cs.cmu.edu Karen Simonyan DeepMind simonyan@google.com Yiming Yang CMU yiming@cs.cmu.edu

Ehab A. Etellisi & Yiming Deng Departments of Electrical ...Ehab A. Etellisi & Yiming Deng Departments of Electrical Engineering University of Colorado Denver ASPRS Conference, Milwaukee,

wTips: Dupes & Merging

Merging funnels

Merging Pictures

Merging Technology Teams

Waad Subber , Sayan Ghosh, Piyush Pandita, Yiming Zhang

Yucai Middle School Xie Yiming

Merging Instructions

Yiming Qiu yiming.qiu@nist.gov 4/2016

Merging Banks & Merging Cultures, Texas Bankers Association Presentation

Yiming Zhang SUNY at Buffalo TRAFFIC SIGN RECOGNITION WITH COLOR IMAGE