1ST TECHNICAL WEBINAR

BDE Platform Architecture02.03.2016

Targets of the BDE platform

1-mars-16www.big-data-europe.eu

Easy to use

1-mars-16

Flexible

Applications running on the platform

1-mars-16www.big-data-europe.eu

Pipeline

o Communicating components

to solve a specific problem

Example

o HDFS nodes

o Spark master/workers

o Spark map-reduce algorithm

Platform architecture

1-mars-16www.big-data-europe.eu

Platform architecture

1-mars-16www.big-data-europe.eu

Platform architecture

1-mars-16www.big-data-europe.eu

1-mars-16www.big-data-europe.eu

Installation

Manual installation

o https://github.com/big-data-europe/README/wiki/Base-platform

Chef recipes

o Temporarily outdated

o https://github.com/big-data-europe/chef-base-platform-cookbook

1-mars-16www.big-data-europe.eu

1-mars-16www.big-data-europe.eu

Docker in a nutshell

Docker

o Build, ship and run any application on any platform

Docker containers

o =~ lightweight virtual machine

Docker image

o Template of a Docker container

1-mars-16www.big-data-europe.eu

Developing a component

1-mars-16www.big-data-europe.eu

Base Docker images

provided by BDE

Extend image with

custom algorithm/data

with minimal effort

Developing a pipeline

1-mars-16www.big-data-europe.eu

Pipeline = collection

of communicating

components

Describe pipeline

in 1 file using

Docker Compose

Supported components

- https://github.com/big-data-

europe/README/wiki/Components

- https://hub.docker.com/u/bde2020

1-mars-16www.big-data-europe.eu

GitHub

DockerHub

1-mars-16www.big-data-europe.eu

1-mars-16

docker-compose upCustom GUI is being built

Docker Swarm

Resources are managed by Swarm

Integrates with Docker Compose

Automatic download/install/run

1-mars-16www.big-data-europe.eu

1-mars-16www.big-data-europe.eu

Future work: Monitoring

1-mars-16www.big-data-europe.eu

1-mars-16

THANK YOU