© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Jim Donahue | Principal Scientist Adobe Systems Technology LabFlint: Making Sparks (and Sharks and HDFSs too!)

© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Flint: Bring BDAS to the AWS Masses @ Adobe

How to effectively evangelize BDAS @ Adobe?

Looking for intrepid, curious users who want to experiment

Curiosity is always tempered by cost of startup

Most of the data for experimental applications likely in AWS

2

© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Flint: Design Principles

Shared Nothing

Get your own AWS account and go

Simple Configuration

Write a little JSON, run a couple of scripts

Efficient, flexible scaling

As simple or complex as you want/need

Full access to tools

Batch, Spark/Shark shells, Shark Server, web UIs, …

Access to all the Spark/Shark tuning parameters

Very simple hardwired “spark-env.sh”

Tuned to Adobe environment

Port choices determined by our firewall

3

© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Flint: Architecture

4

• Local Spark/Shark, Slaves can use S3 storage for files

• Remote Access runs shells on SSH Server

• Components use S3, SimpleDB for state management

• Flint distributes shared AWS credentials among components

• Flint manages master, SSHServer startup

• Slave elasticity managed by master, can leverage spot pricing

LocalSpark/Shark

RemoteAccess

ClusterSetup

Local Flint

Server

S3

Spark Master

SSHServer

(Shells)

SimpleDB

SparkSlave(s)

Flint

AWS

© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Flint: Setup

5

• Flint instance manages encrypted AWS credentials

• Create S3 buckets to hold JAR files

• Create SimpleDB tables to hold state

• Create key pair, security group for instances

LocalSpark/Shark

RemoteAccess

ClusterSetup

Local Flint

Server

S3

SimpleDB

Flint

AWS

© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Flint: Provisioning

6

• Define clusters through JSON spec

(“master instance configuration is x, slave instance configuration is y, scaling rule is …”)

• Define configurations through JSON spec

(“spark master uses AMI x, running service y, with properties a, b, …”) and JAR file containing services code

• “Getting started” set of clusters, configurations provided

• AMI provided with all the requisite Spark / Shark / Hadoop / Kafka bits

LocalSpark/Shark

RemoteAccess

ClusterSetup

Local Flint

Server

S3

SimpleDB

Flint

AWS

© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Flint: Cluster Start

7

• Local Flint Instance launches “master” instance (using cluster definition in SimpleDB)

• Master reads SimpleDB and S3 for configuration and code, installs master services

• Starting services launches Spark and/or HDFS masters through command line

• Master puts “connect URL” in SimpleDB

LocalSpark/Shark

RemoteAccess

ClusterSetup

Local Flint

Server

S3

Spark Master

SimpleDB

Flint

AWS

© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Flint: Slave(s) Start

8

• Master “scaling service” launches slave instance(s)

• Slave reads SimpleDB and S3 for configuration and code, installs worker services

• Slave gets master “connect URL” from SimpleDB

• Slave launches Spark and/or HDFS workers through command line

LocalSpark/Shark

RemoteAccess

ClusterSetup

Local Flint

Server

S3

Spark Master

SimpleDB

SparkSlave(s)

Flint

AWS

© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Flint: Client Start

9

• Flint instance launches “client” instance (using cluster definition in SimpleDB)

• Client reads SimpleDB and S3 for configuration and code, installs (SSHServer) services

• Client reads SimpleDB for authentication info, master connect URL

• Service startup starts SSHServer connected to right “shell factory”

LocalSpark/Shark

RemoteAccess

ClusterSetup

Local Flint

Server

S3

Spark Master

SSHServer

(Shells)

SimpleDB

SparkSlave(s)

Flint

AWS

© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Flint: Client Connect (Remote Shells)

10

• Flint server finds “appropriate client”

• SSH client launched to connect

• SSHServer connects to master on client’s behalf

LocalSpark/Shark

RemoteAccess

ClusterSetup

Local Flint

Server

S3

Spark Master

SSHServer

(Shells)

SimpleDB

SparkSlave(s)

Flint

AWS

© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Flint: Client Asynchronous Requests

Flint clients can also make asynchronous requests

Each Flint master runs service that pulls request from SQS queue

Request progress/results stored in SDB

Requests include:

Move data between HDFS and S3

Mount EBS volume and cache in HDFS (AWS public data sets)

Run batch job

Client can make request even if cluster not alive

Simplifies startup sequencing

Can use monitoring of “cluster queues” to start cluster “on demand”

11

© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Flint: Where We Are Now

Have some intrepid, curious users

The big issue is always “Do I really want to use Spark/Shark?”

SQL is a big selling point

Scala is a mild put-off

Spark Streaming may help settle the issue

Open Sourcing is under discussion

If you’re interested, let me know!

12

© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.