The Longest Common Substring

Problema.k.a Long Repeat

by Donnie Demuth

Sections

1. MapReduce and Hadoop2. Map and Reduce3. Mappers and Reducers4. Using Tools (Amazon)5. Conclusions

1. MapReduce and Hadoop

• What is it?• And how do I get it?

Google MapReduce

• Circa 2003• Based on Map and Reduce (go figure)– and Functional Programming!

• Proprietary

Apache Hadoop

• Circa 2006, released 2009• Named after an Elephant Toy• Seconds, maybe a minute, to install

Installing Hadoop on OSX

• Single Cluster setup is a piece of cake• Download the archive (tar.gz)• Modify conf/hadoop-env.sh:

– # export JAVA_HOME=/usr/lib/j2sdk1.6-sun – export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/

• Modify bin/hadoop:– JAVA=$JAVA_HOME/bin/java– JAVA=$JAVA_HOME/Commands/java

• Just run bin/hadoop with arguments

STOP!

• Actually, installing Hadoop wasn’t necessary• We can write parallel code without it

2. Map and Reduce

• What is it?– Quick Primer to Functional Programming• Higher-Order Functions• Alonzo Church (Lamba Calculus)

• Haskell Curry (Spicy Food)

• How do I use it?

(x (y x*x + y*y))(5)(2)↦ ↦

Code w/ Side-Effects

>>> thing = {'name':'Donald'}>>> def change_name(object): object['name'] = 'Donnie'... >>> change_name(thing)>>> thing{'name': 'Donnie'}

Pure Code, Side-effect Free

>>> thing = {'name':'Donald'}>>> def change_name(object): ... new_obj = {'name': 'Donnie'}... # copy any other values... return new_obj... >>> thing = change_name(thing)>>> thing{'name': 'Donnie'}

Benefits of Pure Code / FP

• easy to understand– Local vars = easy– Global vars + side-effects = hard

• it’s easy to parallelize– We only care about what we know RIGHT NOW

Map

1

2

3

1

4

6

f(x)

Map in Python

• Use the map(<function>, <list>) built-in

>>> map(lambda x: x*x, range(1,100))[1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289, 324, 361, 400, 441, 484, 529, 576, 625, 676, 729, 784, 841, 900, 961, 1024, 1089, 1156, 1225, 1296, 1369, 1444, 1521, 1600, 1681, 1764, 1849, 1936, 2025, 2116, 2209, 2304, 2401, 2500, 2601, 2704, 2809, 2916, 3025, 3136, 3249, 3364, 3481, 3600, 3721, 3844, 3969, 4096, 4225, 4356, 4489, 4624, 4761, 4900, 5041, 5184, 5329, 5476, 5625, 5776, 5929, 6084, 6241, 6400, 6561, 6724, 6889, 7056, 7225, 7396, 7569, 7744, 7921, 8100, 8281, 8464, 8649, 8836, 9025, 9216, 9409, 9604, 9801]

Reduce

1

2

3

0f(x, y)

f(x, y)

f(x, y) = 6

Reduce in Python

• Use the map(<function>, <list>, <unit>) built-in

>>> reduce(lambda x, y: x+y, [1,2,3], 0)6

>>> reduce(lambda x, y: x+y, (map(lambda x: x*x, range(1,100)), 0)

328350

3. Mappers and Reducers

• How do I write them?– Word Count (Hello World for Distrib. Comp.)– Longest Repeat

• Show me how to pipe them

Mappers

• Pseudo-Code– Take some input– Process it– And emit a Key – Value pair

Word Count Mapper

• For some input:– Donald Demuth Donald Draper

• The output should be:– Donald 1– Demuth 1– Donald 1– Draper 1

Word Count Mapper Code

• wordcount/mapper.py

#!/usr/bin/env pythonimport sys, re

word_re = re.compile('[a-zA-Z]+')for line in sys.stdin: line = line.strip().lower() for word in word_re.findall(line): print '%s\t%s' % (word, 1)

Reducers

• Dependant on the Mapper’s emissions• Pseudo-Code for word count– Read an emission from the mapper– Find the key and the value– Store the key in a dictionary with it’s value• But if the key already exists, add the value with the

pre-existing value!

– Emit the dictionary

Word Count Reducer Code

• wordcount/reducer.py#!/usr/bin/env pythonimport sys

counts = {}for line in sys.stdin: line = line.strip() word, count = line.split('\t', 1) count = int(count) counts[word] = counts.get(word, 0) + count

for word, count in counts.items(): print '%s\t%s'% (word, count)

Unix Pipes

• Does this really work??

$ cat books/*.txt | wordcount/mapper.py | wordcount/reducer.py | sort | heada 10526ab 3aback 1abaft 2abaht 1abandon 2abandoned 10abandonment 1abasement 1abash 1

Longest Repeat (LCS)

• Many problems can be solved with a series of Maps and Reduces

• However, Hadoop Streaming is a single Map and Reduce step

• After much trial and error my solution involves a pre-processing step

Pre-processing

• fasta_to_line.py

• gen_suffixes.py

ecoli.fasta.line

ecoli.fasta.line.0

ecoli.fasta.line.100000

ecoli.fasta.line.200000

4.6 megs

4.5 megs

4.4 megs

4.3 megs

ecoli.fasta ecoli.fasta.line

LCS Mapper

• Pseudo-code– Read a line from a suffix file– Determine the index (first chars)– Cycle through the first 100,000 positions• Cycle through possible lengths (10 3000)

– Emit the Length (Key) and the Position (Val)

• Emit (-1) and (-1) to STAY ALIVE

LCS Reducer

• Pseudo-Code– Simple– Find the largest KEY emitted by any mapper– Display it

LCS w/ Murmur.txt$ cat murmur.txt.line.0 | lcs/mapper.py | lcs/reducer.py length(63) pos(128)

$ python>>> text = open('murmur.txt.line').read()>>> text[128:128+63]'Dance the cha chaOr the can canShake your pom pomTo Duran Duran'

>>> seq = text[128:128+63]>>> text.index(seq)128>>> text[129:].index(seq) + 1291777>>> text[128:128+63] == text[1777:1777+63]True>>> text[1777:1777+63]'Dance the cha chaOr the can canShake your pom pomTo Duran Duran'

4. Using Tools, Amazon

• Harness the power of many machines at once– Easy to use 20

• Need to sign up for:– Amazon Elastic MapReduce Service (EMS)– Amazon Elastic Compute Cloud (EC2)– Amazon Simple Storage Service (S3)– Amazon SimpleDB

Deploying Data/Code

• First you’ll need to upload it to S3– Create a new bucket (or global folder) named ecoli-lcs

– Create a new path named input, ecoli-lcs/input– Upload all of the generated suffixes to the input

folder– Upload mapper.py and reducer.py to ecoli-lcs

Creating a Job (Flow)

Creating a Job Flow (…)

RESULTS!

• Need to download the output$ cd output$ cat * | sort (...)length(2815) pos(4166641)

$ python>>> text = open('ecoli.fasta.line').read()>>> seq = text[4166641:4166641+2815]>>> text.index(seq)4166641>>> text[4166642:].index(seq) + 41666424208043>>> text[4166641:4166641+2815] == text[4208043:4208043+2815]

5. Conclusions

• Costs– It’s about 3 cents an hour for a “medium” VM– One run took 840 instance hours (20+ actual)

• Approx. $25– Used about 2000 instance hours in total

• Hadoop Streaming is EASY– Though requires many (easy) tools– But costly if you have “bugs”

A Better Solution?

• Jeff Parker’s program used the following approach:– Cycle through the sequence and find all repeats of

a given size– Emit the location– Increase the size and use the previously known

locations to find larger matches

• Looks good for MapReduce (Core)