December 15, 2014

Luigi

NYC Data Science meetup

What is Luigi

Luigi is a workflow engine If you run 10,000+ Hadoop jobs every day, you need one If you play around with batch processing just for fun, you want one Doesn’t help you with the code, that’s what Scalding, Pig, or anything else is good at It helps you with the plumbing of connecting lots of tasks into complicated pipelines,

especially if those tasks run on Hadoop

2

What do we use it for?

Music recommendations A/B testing Top lists Ad targeting Label reporting Dashboards … and a million other things!

3

Currently running 10,000+ Hadoop jobs every day

On average a Hadoop job is launched every 10s There’s 2,000+ Luigi tasks in production

4

Some history

… let’s go back to 2008!

5

The year was 2008

I was writing my master’s thesis about music recommendations

Had to run hundreds of long-running tasks to compute the output

6

Toy example: classify skipped tracks

$ python subsample_extract_features.py /log/endsongcleaned/2011-01-?? /tmp/subsampled $ python train_model.py /tmp/subsampled model.pickle $ python inspect_model.py model.pickle

7

Log d

Log d+1

...

Log d+k-1

Subsample and extract

features

Subsampled features

Train classifier

Classifier

Look at the output

Reproducibility matters

…and automation. !The previous code is really hard to run again

8

Let’s make into a big workflow

9

$ python run_everything.py

Reality: crashes will happen10

How do you resume this?

Ability to resume matters

When you are developing something interactively, you will try and fail a lot Failures will happen, and you want to resume once you fixed it You want the system to figure out exactly what it has to re-run and nothing else Atomic file operations is crucial for the ability to resume

11

So let’s make it possible to resume

12

13

But still annoying parts

Hardcoded junk

Generalization matters

You should be able to re-run your entire pipeline with a new value for a parameter Command line integration means you can run interactive experiments

14

… now we’re getting something

15

$ python run_everything.py --date-first 2014-01-01 --date-last 2014-01-31 --n-trees 200

16

… but it’s hardly readable

BOILERPLATE

Boilerplate matters!

We keep re-implementing the same functionality Let’s factor it out to a framework

17

A lot of real-world data pipelines are a lot more complex

The ideal framework should make it trivial to build up big data pipelines where dependencies are non-trivial (eg depend on date algebra)

18

So I started thinking

Wanted to build something like GNU Make

19

What is Make and why is it pretty cool?

Build reusable rules Specify what you want to build and then

backtrack to find out what you need in order to get there

Reproducible runs

20

# the compiler: gcc for C program, define as g++ for C++ CC = gcc ! # compiler flags: # -g adds debugging information to the executable file # -Wall turns on most, but not all, compiler warnings CFLAGS = -g -Wall ! # the build target executable: TARGET = myprog ! all: $(TARGET) ! $(TARGET): $(TARGET).c $(CC) $(CFLAGS) -o $(TARGET) $(TARGET).c ! clean: $(RM) $(TARGET)

We want something that works for a wide range of systems

We need to support lots of systems “80% of data science is data munging”

21

Data processing needs to interact with lots of systems

Need to support practically any type of task: Hadoop jobs Database dumps Ingest into Cassandra Send email SCP file somewhere else

22

My first attempt: builder

Use XML config to build up the dependency graph!

23

Don’t use XML

… seriously, don’t use it

24

Dependencies need code

Pipelines deployed in production often have nontrivial ways they define dependencies between tasks

!!!!!!!! … and many other cases

25

Recursion (and date algebra)

BloomFilter(date=2014-05-01)

BloomFilter(date=2014-04-30)

Log(date=2014-04-30)

Log(date=2014-04-29)

...

Date algebra

Toplist(date_interval=2014-01)

Log(date=2014-01-01)

Log(date=2014-01-02)

...

Log(date=2014-01-31)

Enum types

IdMap(type=artist) IdMap(type=track)

IdToIdMap(from_type=artist, to_type=track)

Don’t ever invent your own DSL

“It’s better to write domain specific code in a general purpose language, than writing general purpose code in a domain specific language” – unknown author

!!Oozie is a good example of how messy it gets

26

2009: builder2

Solved all the things I just mentioned - Dependency graph specified in Python - Support for arbitrary tasks - Error emails - Support for lots of common data plumbing stuff: Hadoop jobs, Postgres, etc - Lots of other things :)

27

Graphs!

28

More graphs!

29

Even more graphs!

30

What were the good bits? !Build up dependency graphs and visualize them Non-event to go from development to deployment Built-in HDFS integration but decoupled from the core library !!

What went wrong? !Still too much boiler plate Pretty bad command line integration

31

32

Introducing Luigi

A workflow engine in Python

33

Luigi – History at Spotify

Late 2011: Me and Elias Freider build it, release it into the wild at Spotify, people start using it

“The Python era” !Late 2012: Open source it Early 2013: First known company outside of Spotify:

Foursquare !

34

Luigi is your friendly plumber

Simple dependency definitions Emphasis on Hadoop/HDFS integration Atomic file operations Data flow visualization Command line integration

35

Luigi Task 36

Luigi Task – breakdown 37

The business logic of the task Where it writes output What other tasks it depends on

Parameters for this task

Easy command line integration

So easy that you want to use Luigi for it

38

$ python my_task.py MyTask --param 43 INFO: Scheduled MyTask(param=43) INFO: Scheduled SomeOtherTask(param=43) INFO: Done scheduling tasks INFO: [pid 20235] Running SomeOtherTask(param=43) INFO: [pid 20235] Done SomeOtherTask(param=43) INFO: [pid 20235] Running MyTask(param=43) INFO: [pid 20235] Done MyTask(param=43) INFO: Done INFO: There are no more tasks to run at this time INFO: Worker was stopped. Shutting down Keep-Alive thread $ cat /tmp/foo/bar-43.txt hello, world $

Let’s go back to the example

39

Log d

Log d+1

...

Log d+k-1

Subsample and extract

features

Subsampled features

Train classifier

Classifier

Look at the output

Code in Luigi

40

Extract the features

41

$ python demo.py SubsampleFeatures --date-interval 2013-11-01 DEBUG: Checking if SubsampleFeatures(test=False, date_interval=2013-11-01) is complete INFO: Scheduled SubsampleFeatures(test=False, date_interval=2013-11-01) DEBUG: Checking if EndSongCleaned(date_interval=2013-11-01) is complete INFO: Done scheduling tasks DEBUG: Asking scheduler for work... DEBUG: Pending tasks: 1 INFO: [pid 24345] Running SubsampleFeatures(test=False, date_interval=2013-11-01) ... INFO: 13/11/08 02:15:11 INFO streaming.StreamJob: Tracking URL: http://lon2-hadoopmaster-a1.c.lon.spotify.net:50030/jobdetails.jsp?jobid=job_201310180017_157113 INFO: 13/11/08 02:15:12 INFO streaming.StreamJob: map 0% reduce 0% INFO: 13/11/08 02:15:27 INFO streaming.StreamJob: map 2% reduce 0% INFO: 13/11/08 02:15:30 INFO streaming.StreamJob: map 7% reduce 0% ... INFO: 13/11/08 02:16:10 INFO streaming.StreamJob: map 100% reduce 87% INFO: 13/11/08 02:16:13 INFO streaming.StreamJob: map 100% reduce 100% INFO: [pid 24345] Done SubsampleFeatures(test=False, date_interval=2013-11-01) DEBUG: Asking scheduler for work... INFO: Done INFO: There are no more tasks to run at this time INFO: Worker was stopped. Shutting down Keep-Alive thread $

Run on the command line

42

Step 2: Train a machine learning model

43

Let’s run everything on the command line from scratch$ python luigi_workflow_full.py InspectModel --date-interval 2011-01-03 DEBUG: Checking if InspectModel(date_interval=2011-01-03, n_trees=10) id" % self) INFO: Scheduled InspectModel(date_interval=2011-01-03, n_trees=10) (PENDING) INFO: Scheduled TrainClassifier(date_interval=2011-01-03, n_trees=10) (PENDING) INFO: Scheduled SubsampleFeatures(test=False, date_interval=2011-01-03) (PENDING) INFO: Scheduled EndSongCleaned(date=2011-01-03) (DONE) INFO: Done scheduling tasks INFO: Running Worker with 1 processes INFO: [pid 23869] Worker Worker(salt=912880805, workers=1, host=lon3-edgenode-a22.lon3.spotify.net, username=erikbern, pid=23869) running SubsampleFeatures(test=False, date_interval=2011-01-03) INFO: 14/12/17 02:07:20 INFO mapreduce.Job: Running job: job_1418160358293_86477 INFO: 14/12/17 02:07:31 INFO mapreduce.Job: Job job_1418160358293_86477 running in uber mode : false INFO: 14/12/17 02:07:31 INFO mapreduce.Job: map 0% reduce 0% INFO: 14/12/17 02:08:34 INFO mapreduce.Job: map 2% reduce 0% INFO: 14/12/17 02:08:36 INFO mapreduce.Job: map 3% reduce 0% INFO: 14/12/17 02:08:38 INFO mapreduce.Job: map 5% reduce 0% INFO: 14/12/17 02:08:39 INFO mapreduce.Job: map 10% reduce 0% INFO: 14/12/17 02:08:40 INFO mapreduce.Job: map 17% reduce 0% INFO: 14/12/17 02:16:30 INFO mapreduce.Job: map 100% reduce 100% INFO: 14/12/17 02:16:32 INFO mapreduce.Job: Job job_1418160358293_86477 completed successfully INFO: [pid 23869] Worker Worker(salt=912880805, workers=1, host=lon3-edgenode-a22.lon3.spotify.net, username=erikbern, pid=23869) done SubsampleFeatures(test=False, date_interval=2011-01-03) INFO: [pid 23869] Worker Worker(salt=912880805, workers=1, host=lon3-edgenode-a22.lon3.spotify.net, username=erikbern, pid=23869) running TrainClassifier(date_interval=2011-01-03, n_trees=10) INFO: [pid 23869] Worker Worker(salt=912880805, workers=1, host=lon3-edgenode-a22.lon3.spotify.net, username=erikbern, pid=23869) done TrainClassifier(date_interval=2011-01-03, n_trees=10) INFO: [pid 23869] Worker Worker(salt=912880805, workers=1, host=lon3-edgenode-a22.lon3.spotify.net, username=erikbern, pid=23869) running InspectModel(date_interval=2011-01-03, n_trees=10) time 0.1335% ms_played 96.9351% shuffle 0.0728% local_track 0.0000% bitrate 2.8586% INFO: [pid 23869] Worker Worker(salt=912880805, workers=1, host=lon3-edgenode-a22.lon3.spotify.net, username=erikbern, pid=23869) done InspectModel(date_interval=2011-01-03, n_trees=10)

44

Let’s make it more complicated – cross validation

45

Log d

Log d+1

...

Log d+k-1

Subsample and extract

features

Subsampled features

Train classifier

Classifier

Log e

Log e+1

...

Log e+k-1

Subsample and extract

features

Subsampled features

Cross validation

Cross validation implementation

$ python xv.py CrossValidation --date-interval-a 2012-11-01 --date-interval-b 2012-11-02

46

Run on the command line $ python cross_validation.py CrossValidation --date-interval-a 2011-01-01 --date-interval-b 2011-01-02 INFO: Scheduled CrossValidation(date_interval_a=2011-01-01, date_interval_b=2011-01-02) (PENDING) INFO: Scheduled TrainClassifier(date_interval=2011-01-01, n_trees=10) (DONE) INFO: Scheduled SubsampleFeatures(test=False, date_interval=2011-01-02) (DONE) INFO: Scheduled SubsampleFeatures(test=False, date_interval=2011-01-01) (DONE) INFO: Done scheduling tasks INFO: Running Worker with 1 processes INFO: [pid 18533] Worker Worker(salt=752525444, workers=1, host=lon3-edgenode-a22.lon3.spotify.net, username=erikbern, pid=18533) running CrossValidation(date_interval_a=2011-01-01, date_interval_b=2011-01-02) 2011-01-01 (train) AUC: 0.9040 2011-01-02 ( test) AUC: 0.9040 INFO: [pid 18533] Worker Worker(salt=752525444, workers=1, host=lon3-edgenode-a22.lon3.spotify.net, username=erikbern, pid=18533) done CrossValidation(date_interval_a=2011-01-01, date_interval_b=2011-01-02) INFO: Done INFO: There are no more tasks to run at this time INFO: Worker Worker(salt=752525444, workers=1, host=lon3-edgenode-a22.lon3.spotify.net, username=erikbern, pid=18533) was stopped. Shutting down Keep-Alive thread

47

… no overfitting!

More trees!$ python cross_validation.py CrossValidation --date-interval-a 2011-01-01 --date-interval-b 2011-01-02 --n-trees 100 INFO: Scheduled CrossValidation(date_interval_a=2011-01-01, date_interval_b=2011-01-02, n_trees=100) (PENDING) INFO: Scheduled TrainClassifier(date_interval=2011-01-01, n_trees=100) (PENDING) INFO: Scheduled SubsampleFeatures(test=False, date_interval=2011-01-02) (DONE) INFO: Scheduled SubsampleFeatures(test=False, date_interval=2011-01-01) (DONE) INFO: Done scheduling tasks INFO: [pid 27835] Worker Worker(salt=539404294, workers=1, host=lon3-edgenode-a22.lon3.spotify.net, username=erikbern, pid=27835) running TrainClassifier(date_interval=2011-01-01, n_trees=100) INFO: [pid 27835] Worker Worker(salt=539404294, workers=1, host=lon3-edgenode-a22.lon3.spotify.net, username=erikbern, pid=27835) done TrainClassifier(date_interval=2011-01-01, n_trees=100) INFO: [pid 27835] Worker Worker(salt=539404294, workers=1, host=lon3-edgenode-a22.lon3.spotify.net, username=erikbern, pid=27835) running CrossValidation(date_interval_a=2011-01-01, date_interval_b=2011-01-02, n_trees=100) 2011-01-01 (train) AUC: 0.9074 2011-01-02 ( test) AUC: 0.8896 INFO: [pid 27835] Worker Worker(salt=539404294, workers=1, host=lon3-edgenode-a22.lon3.spotify.net, username=erikbern, pid=27835) done CrossValidation(date_interval_a=2011-01-01, date_interval_b=2011-01-02, n_trees=100) INFO: Done INFO: There are no more tasks to run at this time INFO: Worker Worker(salt=539404294, workers=1, host=lon3-edgenode-a22.lon3.spotify.net, username=erikbern, pid=27835) was stopped. Shutting down Keep-Alive thread

48

… overfitting!

The nice things about Luigi

50

Overhead for a task is about 5 lines (class def + requires + output + run) Easy command line integration

Section name

Minimal boiler plate

51

Everything is a directed acyclic graph

Makefile style Tasks specify what they are dependent on not what other things depend on them

52

Luigi’s visualizer

53

Dive into any task

54

Run with multiple workers

$ python dataflow.py --workers 3 AggregateArtists --date-interval 2013-W08

55

Error notifications

56

Process synchronization

Luigi worker 1 Luigi worker 2

A

B C

A C

F

Luigi central planner

Prevents the same task from being run simultaneously, but all execution is being done by the workers.

57

Luigi is a way of coordinating lots of different tasks

… but you still have to figure out how to implement and scale them!

58

Do general-purpose stuff

Don’t focus on a specific platform !… but comes “batteries included”

59

Built-in support for HDFS & Hadoop

At Spotify we’re abandoning Python for batch processing tasks, replacing it with Crunch and Scalding. Luigi is a great glue!

!Our team, the Lambda team: 15 engs, running 1,000+ Hadoop jobs daily, having 400+ Luigi Tasks in

production. !Our recommendation pipeline is a good example: Python M/R jobs, ML algos in C++, Java M/R jobs,

Scalding, ML stuff in Python using scikit-learn, import stuff into Cassandra, import stuff into Postgres, send email reports, etc.

60

The one time we accidentally deleted 50TB of data

We didn’t have to write a single line of code to fix it – Luigi rescheduled 1000s of task and ran it for 3 days

61

Some things are still not perfect

62

The missing parts

Execution is tied to scheduling – you can’t schedule something to run “in the cloud” Visualization could be a lot more useful There’s no built scheduling – have to rely on crontab These are all things we have in the backlog

63

Source:

What are some ideas for the future?

64

Separate scheduling and execution

65

Luigi central scheduler

Slave

Slave

Slave

Slave

...

Luigi in Scala?

66

Luigi implements some core beliefs

The #1 focus is on removing all boiler plate The #2 focus is to be as general as possible The #3 focus is to make it easy to go from test to production !!

67

Join the club!

Questions?

69