Writing your own RDD for fun and profit

Writing your own RDD for fun and profit

by Paweł Szulc @rabbitonweb

Writing my own RDD? What for?

● To write your own RDD, you need to understand to some extent internal mechanics of Apache Spark

● Writing your own RDD will prove you understand them well● When connecting to external storage, it is reasonable to

create your own RDD for it

Outline

1. The Recap

Outline

1. The Recap2. The Internals

Outline

1. The Recap2. The Internals3. The Fun & Profit

Part I - The Recap

RDD - the definition


RDD stands for resilient distributed dataset



Dataset - initial data comes from some distributed storage



Distributed - stored in nodes among the cluster




Resilient - if data is lost, data can be recreated

Distributed - stored in nodes among the cluster


RDD - example

RDD - example

val logs = sc.textFile("hdfs://logs.txt")

RDD - example


From Hadoop DistributedFile System

RDD - example


From Hadoop DistributedFile SystemThis is the RDD

RDD - example

val numbers = sc.parallelize(List(1, 2, 3, 4))

Programmatically from a collection of elementsThis is the RDD

RDD - example

val logs = sc.textFile("logs.txt")

RDD - example


val lcLogs = logs.map(_.toLowerCase)

RDD - example



Creates a new RDD

RDD - example



val errors = lcLogs.filter(_.contains(“error”))

RDD - example




And yet another RDD

RDD - example




And yet another RDDPerformance Alert?!?!

RDD - Operations

1. Transformationsa. Mapb. Filterc. FlatMapd. Samplee. Unionf. Intersectg. Distincth. GroupByKeyi. ….

2. Actionsa. Reduceb. Collectc. Countd. Firste. Take(n)f. TakeSampleg. SaveAsTextFileh. ….

RDD - example




RDD - example




val numberOfErrors = errors.count

RDD - example





This will trigger the computation

RDD - example





This will the calculated value (Int)

This will trigger the computation

Partitions?

Partitions?

A partition represents subset of data within your distributed collection.

Partitions?


Number of partitions tightly coupled with level of parallelism.

Partitions evaluationval counted = sc.textFile(..).count


node 1

node 2

node 3


node 1

node 2

node 3


node 1

node 2

node 3


node 1

node 2

node 3


node 1

node 2

node 3


node 1

node 2

node 3


node 1

node 2

node 3


node 1

node 2

node 3


node 1

node 2

node 3


node 1

node 2

node 3


node 1

node 2

node 3


node 1

node 2

node 3


node 1

node 2

node 3

Pipeline

Pipelinemap

Pipelinemap count

Pipelinemap count

task

Pipelinemap count

task

Pipelinemap count

task

But what if...val startings = allShakespeare

.filter(_.trim != "")

.groupBy(_.charAt(0))

.mapValues(_.size)

.reduceByKey {

case (acc, length) =>

acc + length

}

But what if...val startings = allShakespeare



.mapValues(_.size)

.reduceByKey {


acc + length

}

But what if...filter

val startings = allShakespeare



.mapValues(_.size)

.reduceByKey {


acc + length

}

And now what?filter




.mapValues(_.size)

.reduceByKey {


acc + length

}

And now what?filter mapValues




.mapValues(_.size)

.reduceByKey {


acc + length

}

And now what?filter




.mapValues(_.size)

.reduceByKey {


acc + length

}

Shufflingfilter groupBy




.mapValues(_.size)

.reduceByKey {


acc + length

}

Shufflingfilter mapValuesgroupBy




.mapValues(_.size)

.reduceByKey {


acc + length

}

Shufflingfilter reduceByKeygroupBy




.mapValues(_.size)

.reduceByKey {


acc + length

}

mapValues

Shufflingfilter reduceByKeygroupBy mapValues

Shufflingfilter reduceByKey

task

groupBy mapValues


task

groupBy mapValues


task

groupBy mapValues


task

Wait for calculations on all partitions before moving on

groupBy mapValues


task

groupBy mapValues


task

groupBy

Data flying around through cluster

mapValues


task

groupBy mapValues


task taskgroupBy mapValues

Shufflingfilter reduceByKeygroupBy mapValues

stage1

Stagefilter reduceByKeygroupBy mapValues

sda

stage2stage1

Stagefilter reduceByKeygroupBy mapValues

Part II - The Internals

What is a RDD?

What is a RDD?

Resilient Distributed Dataset

What is a RDD?

Resilient Distributed Dataset

...10 10/05/2015 10:14:01 UserInitialized Ania Nowak10 10/05/2015 10:14:55 FirstNameChanged Anna12 10/05/2015 10:17:03 UserLoggedIn12 10/05/2015 10:21:31 UserLoggedOut …198 13/05/2015 21:10:11 UserInitialized Jan Kowalski

What is a RDD?

node 1


node 2 node 3

What is a RDD?

node 1



node 2 node 3



What is a RDD?

What is a RDD?

What is a RDD?

RDD needs to hold 3 chunks of information in order to do its work:

What is a RDD?


1. pointer to his parent

What is a RDD?


1. pointer to his parent2. how its internal data is partitioned

What is a RDD?


1. pointer to his parent2. how its internal data is partitioned3. how to evaluate its internal data

What is a RDD?



What is a partition?




override def getPartitions: Array[Partition] = ???



override def getPartitions: Array[Partition] = ???

How this subset is defined depends on type of the RDD

example: HadoopRDD

val journal = sc.textFile(“hdfs://journal/*”)

example: HadoopRDD


How HadoopRDD is partitioned?

example: HadoopRDD


How HadoopRDD is partitioned?

In HadoopRDD partition is exactly the same as file chunks in HDFS

example: HadoopRDD

10 10/05/2015 10:14:01 UserInit3 10/05/2015 10:14:55 FirstNa12 10/05/2015 10:17:03 UserLo4 10/05/2015 10:21:31 UserLo5 13/05/2015 21:10:11 UserIni




example: HadoopRDD

node 1


node 2 node 3




example: HadoopRDD

node 1


node 2 node 3




example: HadoopRDD

node 1


node 2 node 3




example: HadoopRDD

node 1


node 2 node 3




example: HadoopRDD

node 1


node 2 node 3




example: HadoopRDD

class HadoopRDD[K, V](...) extends RDD[(K, V)](sc, Nil) with Logging {...override def getPartitions: Array[Partition] = { val jobConf = getJobConf()

SparkHadoopUtil.get.addCredentials(jobConf) val inputFormat = getInputFormat(jobConf) if (inputFormat.isInstanceOf[Configurable]) { inputFormat.asInstanceOf[Configurable].setConf(jobConf) } val inputSplits = inputFormat.getSplits(jobConf, minPartitions) val array = new Array[Partition](inputSplits.size) for (i <- 0 until inputSplits.size) { array(i) = new HadoopPartition(id, i, inputSplits(i)) } array

}

example: HadoopRDD



}

example: HadoopRDD



}

example: MapPartitionsRDD


val fromMarch = journal.filter {

case (date, size) => LocalDate.parse(date) isAfter LocalDate.of(2015,3,1)

}





}

How MapPartitionsRDD is partitioned?





}

How MapPartitionsRDD is partitioned?

MapPartitionsRDD inherits partition information from its parent RDD


class MapPartitionsRDD[U: ClassTag, T: ClassTag](...) extends RDD[U](prev) {

...

override def getPartitions: Array[Partition] = firstParent[T].partitions

What is a RDD?



What is a RDD?



RDD parent

sc.textFile(“hdfs://journal/*”)

.groupBy(extractDate _)

.map { case (date, events) => (date, events.size) }

.filter {


}

.take(300)

.foreach(println)

RDD parent

sc.textFile(“hdfs://journal/*”)

.groupBy(extractDate _)

.map { case (date, events) => (date, events.size) }

.filter {


}

.take(300)

.foreach(println)

RDD parent

sc.textFile()

.groupBy()

.map { }

.filter {

}

.take()

.foreach()

Directed acyclic graphsc.textFile() .groupBy() .map { } .filter { } .take() .foreach()

Directed acyclic graph

HadoopRDD

sc.textFile() .groupBy() .map { } .filter { } .take() .foreach()


HadoopRDD

ShuffeledRDD



HadoopRDD

ShuffeledRDD MapPartRDD



HadoopRDD

ShuffeledRDD MapPartRDD MapPartRDD



HadoopRDD




HadoopRDD



Two types of parent dependencies:

1. narrow dependency2. wider dependency


HadoopRDD






HadoopRDD






HadoopRDD





Stage 1Stage 2


What is a RDD?


1. pointer to his parent2. how its internal data is partitioned3. how evaluate its internal data

What is a RDD?



Stage 1Stage 2

Running Job aka materializing DAGsc.textFile() .groupBy() .map { } .filter { }

Stage 1Stage 2

Running Job aka materializing DAGsc.textFile() .groupBy() .map { } .filter { } .collect()

Stage 1Stage 2


action

Stage 1Stage 2


action

Actions are implemented using sc.runJob method

Running Job aka materializing DAG

/**

* Run a function on a given set of partitions in an RDD and return the results as an array.

*/

def runJob[T, U](

): Array[U]


/**


*/

def runJob[T, U](

rdd: RDD[T],

): Array[U]


/**


*/

def runJob[T, U](

rdd: RDD[T],

func: Iterator[T] => U,

): Array[U]


/**

* Return an array that contains all of the elements in this RDD.

*/

def collect(): Array[T] = {

val results = sc.runJob(this, (iter: Iterator[T]) => iter.toArray)

Array.concat(results: _*)

}


/**

* Return the number of elements in the RDD.

*/

def count(): Long = sc.runJob(this, Utils.getIteratorSize _).sum

Multiple jobs for single action

/*** Take the first num elements of the RDD. It works by first scanning one partition, and use the results from that partition to estimate the number of additional partitions needed to satisfy the limit.*/def take(num: Int): Array[T] = { while (buf.size < num && partsScanned < totalParts) { (….) val left = num - buf.size val res = sc.runJob(this, (it: Iterator[T]) => it.take(left).toArray, p, allowLocal = true) (….) res.foreach(buf ++= _.take(num - buf.size)) partsScanned += numPartsToTry (….) } buf.toArray }


/**


*/

def runJob[T, U](

rdd: RDD[T],


): Array[U]


/**


*/

def runJob[T, U](

rdd: RDD[T],


): Array[U]


/** * :: DeveloperApi :: * Implemented by subclasses to compute a given partition. */@DeveloperApidef compute(split: Partition, context: TaskContext): Iterator[T]

What is a RDD?



What is a RDD?

RDD needs to hold 3 + 2 chunks of information in order to do its work:

1. pointer to his parent2. how its internal data is partitioned3. how evaluate its internal data4. data locality5. paritioner

What is a RDD?



Data Locality: HDFS example

node 1


node 2 node 3




What is a RDD?



What is a RDD?



Spark performance - shuffle optimization


join


join


map groupBy


map groupBy


map groupBy join


map groupBy join


map groupBy join

Optimization: shuffle avoided if data is already partitioned


map groupBy


map groupBy map


map groupBy map


map groupBy map join


map groupBy map join


map groupBy mapValues


map groupBy mapValues


map groupBy mapValues join


map groupBy mapValues join

Part III - The Fun & Profit

It’s all on github!

http://bit.do/scalapolis



RandomRDD

RandomRDD

sc.random()

.take(3)

.foreach(println)

RandomRDD

sc.random()

.take(3)

.foreach(println)

210

-321

21312

RandomRDD

sc.random()

.take(3)

.foreach(println)

RandomRDD

sc.random()

.take(3)

.foreach(println)

sc.random(maxSize = 10, numPartitions = 4)

.take(10)

.foreach(println)

CensorshipRDD

CensorshipRDD

val statement =

sc.parallelize(List("We", "all", "know that", "Hadoop rocks!"))

CensorshipRDD

val statement =


.censor()

.collect().toList.mkString(" ")

println(statement)

CensorshipRDD

CensorshipRDD


.censor().collectLegal().foreach(println)

CensorshipRDD


.censor().collectLegal().foreach(println)

We

all

know that

Fin

Fin

Paweł Szulc

Fin

Paweł Szulc

[email protected]

mailto:[email protected]


Fin

Paweł Szulc

[email protected]

Twitter: @rabbitonweb



Fin

Paweł Szulc

[email protected]


http://rabbitonweb.com





Fin

Paweł Szulc

[email protected]



http://github.com/rabbitonweb







Fin

Paweł Szulc

[email protected]













Software

Writing your own RDD for fun and profit