9
Concepts and Technologies for Distributed Systems and Big Data Processing Guido Salvaneschi 1

Concepts and Technologies for Distributed …stg-tud.github.io/ctbd/2017/CTBD_01_organization.pdfConcepts and Technologies for Distributed Systems and Big Data Processing •Provide

  • Upload
    others

  • View
    2

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Concepts and Technologies for Distributed …stg-tud.github.io/ctbd/2017/CTBD_01_organization.pdfConcepts and Technologies for Distributed Systems and Big Data Processing •Provide

ConceptsandTechnologiesforDistributedSystemsand

BigDataProcessingGuidoSalvaneschi

1

Page 2: Concepts and Technologies for Distributed …stg-tud.github.io/ctbd/2017/CTBD_01_organization.pdfConcepts and Technologies for Distributed Systems and Big Data Processing •Provide

People

GuidoSalvaneschi

PascalWeisenburgerMatthiasEichholz

2

Page 3: Concepts and Technologies for Distributed …stg-tud.github.io/ctbd/2017/CTBD_01_organization.pdfConcepts and Technologies for Distributed Systems and Big Data Processing •Provide

Motivation

• Distributedsystemsareubiquitous• Emails,WWW,…

• Areaisquicklychanging- sorryforthebuzzwords:)• Bigdataanalytics• Complexeventprocessing/streamprocessing• Cloudcomputing• Newprogrammingmodels(streams,actors)• ...

• Yet,manyconceptsremainthesame• Abstractionoverlow-leveldetails• Faulttolerance• Performance:throughput,latency• Asynchonouscommunication• ...

3

Page 4: Concepts and Technologies for Distributed …stg-tud.github.io/ctbd/2017/CTBD_01_organization.pdfConcepts and Technologies for Distributed Systems and Big Data Processing •Provide

ConceptsandTechnologiesforDistributedSystemsandBigDataProcessing

• ProvideanoverviewofrecentdevelopmentindistributedsystemsandBigDataprocessing

• Focusonconcepts,notontechnology.• Technologywasdifferent5yearsago,willbedifferentin5years• Wewilllookattechnologyonlyasawaytobettergrasptheconcepts

• Selectedtopics• Differenttopics,selectedamongthe“mostinteresting”ones• Guestlectures

4

Page 5: Concepts and Technologies for Distributed …stg-tud.github.io/ctbd/2017/CTBD_01_organization.pdfConcepts and Technologies for Distributed Systems and Big Data Processing •Provide

Aboutthiscourse

WhatthiscourseIS• Anintroductiononselected topics• …ondistributedsystemsandbigdataprocessing• Acourseaboutconcepts…andabitabouttechnologies

WhatitisNOT• Itisnot(only)acourseaboutrecenttrendsinBigData.• Itisnotasystematic courseondistributedsystems

• DistributedSystems:PrinciplesandParadigmsAndrewS.Tanenbaum,MaartenvanSteen

• Itisnotatutorial onhowtoprogrambigdatasystems

5

Page 6: Concepts and Technologies for Distributed …stg-tud.github.io/ctbd/2017/CTBD_01_organization.pdfConcepts and Technologies for Distributed Systems and Big Data Processing •Provide

April21- Intro,motivationApril28- Introtobigdata,mapreduceMay5- hadoop,hdfsMay12- Futures,actors,streamsMay19- Futures,actors,streamsMay26- ComplexEventProcessingJune2- SparkJune9- SparkstreamingJune16- Exampreparation…

JulyTBA- Exam

Tentativecourseschedule

Bigdataandcomplexeventprocessing

Architecturesandprogrammingmodelsfordistr.systems

Subjecttochange!Seecoursewebsiteforupdates

6

Page 7: Concepts and Technologies for Distributed …stg-tud.github.io/ctbd/2017/CTBD_01_organization.pdfConcepts and Technologies for Distributed Systems and Big Data Processing •Provide

Exercises

• Nogradedexercises• Exerciseswillbeprovidedaftersomelectures.• Solutionsdiscussedinthenextlectureifneeded

• Why• Getmoreconfidentonthetopicsofthelectures• Getafeelingthetypeofquestionsthatcancomeupintheexam

• Examples• Readapaper• Answerquestionsbasedonthecontentofthelecture• Discussacasestudy• Smallcodingexercises

7

Page 8: Concepts and Technologies for Distributed …stg-tud.github.io/ctbd/2017/CTBD_01_organization.pdfConcepts and Technologies for Distributed Systems and Big Data Processing •Provide

RegistrationandGrading

• RegisterinTucan forthecourse!

• Writtenexam– July,dateTBA• 90mins

• Questionsaboutthetopicscoveredinthelectures• Simple“programming”tasks• Understandacodesnippetthatisrelevantforoneofthecoveredtopics• Completeacodesnippet

8

Page 9: Concepts and Technologies for Distributed …stg-tud.github.io/ctbd/2017/CTBD_01_organization.pdfConcepts and Technologies for Distributed Systems and Big Data Processing •Provide

Resources

• Website(slides,exercises)• http://stg-tud.github.io/ctbd/• Thisistheofficialplaceslides,exercises,updatedschedule,etc.

• Forum• https://www.fachschaft.informatik.tu-darmstadt.de/forum/viewforum.php?f=580• Pleaseusetheforumforyourquestions.Answerswillremainasareferenceforotherpeople.

• Pleaseaskassistantsforquestionsthatcannotbepostedontheforum• Butthinkifthisisreallythecase:)

9


An Introduction to Reactive Programmingstg-tud.github.io/sedc/Lecture/ws13-14/I-1.1-ReactiveProgramming.pdfGuido Salvaneschi: introduction to reactive programming December 2013 Map

An Introduction to Reactive Programmingstg-tud.github.io/sedc/Lecture/ws13-14/I-1.1-ReactiveProgramming.pdfGuido Salvaneschi: introduction to reactive programming December 2013 Map

Documents
Software Engineering Design & Constructionstg-tud.github.io/.../ss18/2-PL-Design-Style-notes.pdfSoftware Engineering Design & Construction Dr. Michael Eichberg Fachgebiet Softwaretechnik

Software Engineering Design & Constructionstg-tud.github.io/.../ss18/2-PL-Design-Style-notes.pdfSoftware Engineering Design & Construction Dr. Michael Eichberg Fachgebiet Softwaretechnik

Documents
Dr.-Ing. Michael Eichberg - GitHub Pagesstg-tud.github.io/ctbd/2016/CTBD_10_components.pdf · 2019. 6. 18. · Dr.-Ing. Michael Eichberg eichberg@informatik.tu-darmstadt.de The Interceptor

Dr.-Ing. Michael Eichberg - GitHub Pagesstg-tud.github.io/ctbd/2016/CTBD_10_components.pdf · 2019. 6. 18. · Dr.-Ing. Michael Eichberg [email protected] The Interceptor

Documents
Distributed Transactions 7. Transaction …Distributed Transactions 7. Transaction Management for distributed databases 28 Distributed Transactions 29 Distributed Commit 30 Distributed

Distributed Transactions 7. Transaction …Distributed Transactions 7. Transaction Management for distributed databases 28 Distributed Transactions 29 Distributed Commit 30 Distributed

Documents
The Factory Method Design Pattern - GitHub Pagesstg-tud.github.io/...Factory...and_Abstract_Factory_Design_Pattern.pdf · The Factory Method Design Pattern ... The Abstract Factory

The Factory Method Design Pattern - GitHub Pagesstg-tud.github.io/...Factory...and_Abstract_Factory_Design_Pattern.pdf · The Factory Method Design Pattern ... The Abstract Factory

Documents
Resource Management - stg-tud.github.iostg-tud.github.io/ctbd/2016/CTBD_14_resource_management.pdf · Resource Management • Typically implemented by a system deployed across nodes

Resource Management - stg-tud.github.iostg-tud.github.io/ctbd/2016/CTBD_14_resource_management.pdf · Resource Management • Typically implemented by a system deployed across nodes

Documents
Introduction to Big Data - GitHub Pagesstg-tud.github.io/ctbd/2017/CTBD_03_bigdata_intro.pdf · Big Data – Business perspective It is a new business model • People pay with data,

Introduction to Big Data - GitHub Pagesstg-tud.github.io/ctbd/2017/CTBD_03_bigdata_intro.pdf · Big Data – Business perspective It is a new business model • People pay with data,

Documents
CTBD X preparation - GitHub Pages · •Hive, HBase, Yarn •Futures, Promises, Actors •Spark •Spark streaming 2 Papers •MapReduce •GFS •Spark Exercises •MapReduce •Futures,

CTBD X preparation - GitHub Pages · •Hive, HBase, Yarn •Futures, Promises, Actors •Spark •Spark streaming 2 Papers •MapReduce •GFS •Spark Exercises •MapReduce •Futures,

Documents
Concepts and Technologies for Distributed Systems and Big ...stg-tud.github.io/ctbd/2017/CTBD_10_spark_streaming.pdf · Spark Streaming •Framework for large scale stream processing

Concepts and Technologies for Distributed Systems and Big ...stg-tud.github.io/ctbd/2017/CTBD_10_spark_streaming.pdf · Spark Streaming •Framework for large scale stream processing

Documents
Concepts and Technologies for Distributed Systems and Big ...stg-tud.github.io/ctbd/2016/CTBD_16_spark_streaming.pdf · •Integrates with Spark’s batch and interactive processing

Concepts and Technologies for Distributed Systems and Big ...stg-tud.github.io/ctbd/2016/CTBD_16_spark_streaming.pdf · •Integrates with Spark’s batch and interactive processing

Documents
Advanced Big Data Systemsstg-tud.github.io/ctbd/2016/CTBD_12_spark.pdf•Matei Zaharia et al. Spark: Cluster Computing with Working Sets,. HotCloud 2010. •Matei Zaharia et al. Resilient

Advanced Big Data Systemsstg-tud.github.io/ctbd/2016/CTBD_12_spark.pdf•Matei Zaharia et al. Spark: Cluster Computing with Working Sets,. HotCloud 2010. •Matei Zaharia et al. Resilient

Documents
Futures, Actors, and Streamsstg-tud.github.io/ctbd/2016/CTBD_09_futures_actors.pdf · Futures, Actors, and Streams. Philipp Haller Programming a Concurrent World ... • Contributed

Futures, Actors, and Streamsstg-tud.github.io/ctbd/2016/CTBD_09_futures_actors.pdf · Futures, Actors, and Streams. Philipp Haller Programming a Concurrent World ... • Contributed

Documents
Futures, Async, and Actors - stg-tud.github.iostg-tud.github.io/sedc/Lecture/ws17-18/Futures_Async_Actors.pdf · • Contributed to Akka (Typesafe/Lightbend) ... Anatomy of an Actor

Futures, Async, and Actors - stg-tud.github.iostg-tud.github.io/sedc/Lecture/ws17-18/Futures_Async_Actors.pdf · • Contributed to Akka (Typesafe/Lightbend) ... Anatomy of an Actor

Documents
Distributed Database: Part 2. Distributed DBMS Distributed database requires distributed DBMS Distributed database requires distributed DBMS Functions

Distributed Database: Part 2. Distributed DBMS Distributed database requires distributed DBMS Distributed database requires distributed DBMS Functions

Documents
Concurrent Programming - GitHub Pagesstg-tud.github.io/ctbd/2016/CTBD_08_concurrency.pdf · 2019-06-18 · Concurrency Communication in Java Using Monitors ‣Via reading and writing

Concurrent Programming - GitHub Pagesstg-tud.github.io/ctbd/2016/CTBD_08_concurrency.pdf · 2019-06-18 · Concurrency Communication in Java Using Monitors ‣Via reading and writing

Documents
Intro to distributed systems - GitHub Pagesstg-tud.github.io/ctbd/2016/CTBD_02_intro.pdf · Intro to distributed systems 1. What is a distributed system? A distributed system consists

Intro to distributed systems - GitHub Pagesstg-tud.github.io/ctbd/2016/CTBD_02_intro.pdf · Intro to distributed systems 1. What is a distributed system? A distributed system consists

Documents
Modeling Dynamic Behavior - stg-tud.github.iostg-tud.github.io/...Diagrams-Sequence_Diagrams.pdf · UML Sequence Diagrams | Asynchronous messages are messages that don’t block

Modeling Dynamic Behavior - stg-tud.github.iostg-tud.github.io/...Diagrams-Sequence_Diagrams.pdf · UML Sequence Diagrams | Asynchronous messages are messages that don’t block

Documents
WS18-SE-15-Composite Design Pattern - stg-tud.github.iostg-tud.github.io/eise/WS18-SE-15-Composite_Design_Pattern.pdf · (Interaction diagram for context & interaction.) ... May facilitate

WS18-SE-15-Composite Design Pattern - stg-tud.github.iostg-tud.github.io/eise/WS18-SE-15-Composite_Design_Pattern.pdf · (Interaction diagram for context & interaction.) ... May facilitate

Documents
Geo-Distributed Big Data Processing - GitHub Pagesstg-tud.github.io/.../CTBD_13_geo_distributed_big_data.pdfGeo-Distributed Big Data • Many large datasets geo-distributed, i.e.,

Geo-Distributed Big Data Processing - GitHub Pagesstg-tud.github.io/.../CTBD_13_geo_distributed_big_data.pdfGeo-Distributed Big Data • Many large datasets geo-distributed, i.e.,

Documents
Modeling Dynamic Behavior - GitHub Pagesstg-tud.github.io/...Diagrams-Sequence_Diagrams.pdf · UML Sequence Diagrams | Asynchronous messages are messages that don’t block. An active

Modeling Dynamic Behavior - GitHub Pagesstg-tud.github.io/...Diagrams-Sequence_Diagrams.pdf · UML Sequence Diagrams | Asynchronous messages are messages that don’t block. An active

Documents
The Composite Design Pattern - GitHub Pagesstg-tud.github.io/eise/WS11-EiSE-16-Composite_Design_Pattern.pdf · The Composite Design Pattern Applicability Use composite when... you

The Composite Design Pattern - GitHub Pagesstg-tud.github.io/eise/WS11-EiSE-16-Composite_Design_Pattern.pdf · The Composite Design Pattern Applicability Use composite when... you

Documents
Distributed Systems: Distributed algorithms

Distributed Systems: Distributed algorithms

Documents
WS18-SE-16-Observer Design Pattern - GitHub Pagesstg-tud.github.io/eise/WS18-SE-16-Observer_Design_Pattern.pdf · Observer Design Pattern Communication without Coupling • Change

WS18-SE-16-Observer Design Pattern - GitHub Pagesstg-tud.github.io/eise/WS18-SE-16-Observer_Design_Pattern.pdf · Observer Design Pattern Communication without Coupling • Change

Documents
Introduction to Map Reducestg-tud.github.io/ctbd/2016/CTBD_04_mapreduce.pdf · 2019-06-18 · Basic MapReduce Program Design •Tasks that can be performed independently on a data

Introduction to Map Reducestg-tud.github.io/ctbd/2016/CTBD_04_mapreduce.pdf · 2019-06-18 · Basic MapReduce Program Design •Tasks that can be performed independently on a data

Documents
Data Analytics Expert Knowledge - stg-tud.github.iostg-tud.github.io/ctbd/2016/20160630_Lecture_Darmstadt_Part1.pdf · Splunk Rapid Miner Unstructured Data Business Objects Spark

Data Analytics Expert Knowledge - stg-tud.github.iostg-tud.github.io/ctbd/2016/20160630_Lecture_Darmstadt_Part1.pdf · Splunk Rapid Miner Unstructured Data Business Objects Spark

Documents
NDG Security: Distributed Governance, Distributed Access Control, Distributed Data

NDG Security: Distributed Governance, Distributed Access Control, Distributed Data

Documents
What is Software Engineering? - GitHub Pagesstg-tud.github.io/eise/WS11-EiSE-03-What_is_Software_Engineering.pdf · The Software Engineering Process: Definition and Scope; ACM SIGSOFT

What is Software Engineering? - GitHub Pagesstg-tud.github.io/eise/WS11-EiSE-03-What_is_Software_Engineering.pdf · The Software Engineering Process: Definition and Scope; ACM SIGSOFT

Documents
Alessandro Margara Politecnico di Milanostg-tud.github.io/ctbd/2016/CTBD_07_cep.pdf · reactive applications • Focus on streams as unbounded collections of elements – ... –

Alessandro Margara Politecnico di Milanostg-tud.github.io/ctbd/2016/CTBD_07_cep.pdf · reactive applications • Focus on streams as unbounded collections of elements – ... –

Documents
Back to Hadoop - GitHub Pagesstg-tud.github.io/ctbd/2016/CTBD_05_mapreduce.pdf•Hadoop is an ecosystem of tools for processing Big Data. •Hadoop is an open source project. 2 A family

Back to Hadoop - GitHub Pagesstg-tud.github.io/ctbd/2016/CTBD_05_mapreduce.pdf•Hadoop is an ecosystem of tools for processing Big Data. •Hadoop is an open source project. 2 A family

Documents
What is Software Engineering? - GitHub Pagesstg-tud.github.io/eise/WS14-EiSE-03-What_is_Software_Engineering.… · hardware. Properties of ... “ What is Software Engineering? |

What is Software Engineering? - GitHub Pagesstg-tud.github.io/eise/WS14-EiSE-03-What_is_Software_Engineering.… · hardware. Properties of ... “ What is Software Engineering? |

Documents