ExamPreparationGuidoSalvaneschi

1

Warning!

• Thesearejustexamplesofthekindofquestionsthatcanappearintheexam.

• Theyarenotsupposedtobecomplete(ofcourse).

• Theyarenotrepresentativeofthecoverageofthecoursetopicsintheexam.

• Theydonotcoverquestionsaboutcoding(but“simple”exercisesprovidegoodexamplesforthat).

2

Explain3reasonsthatmotivatebuildingasysteminadistributedway

3

WhyDistributedSystems

• Functionaldistribution• Computershavedifferent functional capabilities(e.g.,Fileserver,printer )yetmayneedtoshareresources

• Client /server• Datagathering/dataprocessing

• Incrementalgrowth• Easiertoevolvethesystem• Modularexpandability

• Inherentdistributioninapplicationdomain• Banks,reservationservices,distributed games,mobileapps• physicallyoracrossadministrativedomains• cashregisterandinventorysystemsforsupermarket chains• computersupportedcollaborativework

WhyDistributedSystems

• Economics• collections ofmicroprocessorsofferabetterprice/performanceratiothanlargemainframes.

• Lowprice/performance ratio:costeffectivewaytoincrease computingpower.

• Betterperformance• Loadbalancing• Replicationofprocessingpower• Adistributedsystemmayhavemoretotalcomputingpowerthanamainframe.Ex.10,000CPUchips,each runningat50MIPS.Notpossible tobuild500,000MIPSsingleprocessorsince itwouldrequire0.002nsec instructioncycle.Enhancedperformancethroughloaddistributing.

• IncreasedReliability• Exploit independentfailuresproperty• Ifonemachinecrashes,thesystemasawholecanstill survive.

• Anotherdriving force:theexistenceoflargenumberofpersonalcomputers, theneedforpeople tocollaborateandshareinformation.

Explain3goals(andchallenges)ofdistributedsystems

Goalsandchallengesofdistributedsystems

• Transparency• Howtoachieve thesingle-systemimage

• Performance• Thesystemprovideshigh(computing,storage,..)performance

• Scalability• Theability toservemoreusers,provideacceptableresponsetimeswithincreasedamountofdata

• Openness• Anopendistributedsystemcanbeextendedandimprovedincrementally• Requirespublicationofcomponentinterfacesandstandardsprotocolsforaccessinginterfaces

• Reliability/faulttolerance• Maintainavailabilityevenwhenindividual componentsfail

• Heterogeneity• Network,hardware,operatingsystem,programminglanguages,differentdevelopers

• Security• Confidentiality,integrityandavailability

Whichtechniquescanbeusedtomakeasystemscalable?Brieflyexplainthem.

8

Scalingtechniques

Distribution• Splittingaresource(suchasdata)intosmallerparts,andspreading thepartsacrossthesystem(cf DNS)

9

Scalingtechniques

• Replication• Replicateresources(services,data)acrossthesystem,canaccesstheminmultipleplaces

• Cachingtoavoidrecomputation• Increasedavailabilityreducestheprobability thatabigger systembreaks

• Hidingcommunicationlatencies• Avoidwaitingforresponses toremoteservicerequests

• Useasynchronouscommunication

10

ShowthesignatureoftheMapfunctionandtheReducefunctioninMapReduce.WhatistheMapphaseandwhataretheReducephaseresponsiblefor?

Functionalprogramming“foundations”

• mapinMapReduce↔mapinFP• map::(a→b)→[a]→[b]• Example:Doubleallnumbers inalist.• >map((*)2)[1,2,3]>[2,4,6]

• Inapurelyfunctionalsetting,anelementofalistbeingcomputedbymapcannotseetheeffectsofthecomputationsonotherelements.

• Iftheorderofapplicationofafunctionftoelementsinalistiscommutative,thenwecanreorderorparallelizeexecution.

12

Note:Thereisnoprecise1-1correspondence.Pleasetakethisjustasananalogy.

Functionalprogramming“foundations”

• Moveoverthelist,applyftoeachelementandanaccumulator. freturnsthenextaccumulatorvalue,whichiscombinedwiththenextelement.

• reduceinMapReduce↔foldinFP• foldl ::(b→a→b)→b→[a]→b• Example:Sumofallnumbers inalist.• >foldl (+)0[1,2,3]foldl (+)0[1,2,3]>6

13

Note:Thereisnoprecise1-1correspondence.Pleasetakethisjustasananalogy.

MapReduceBasicProgrammingModel

• Transformasetofinputkey-valuepairstoasetofoutputvalues:• Map:(k1,v1)→list(k2,v2)• MapReducelibrarygroupsallintermediatepairswithsamekeytogether.

• Reduce:(k2,list(v2))→list(v2)

14

Whatistheproblemwith“stragglers”(slowworkers)andwhatcanbedonetosolvethisproblem?

Stragglers&BackupTasks

• Problem:“Stragglers”(i.e.,slowworkers)significantlylengthenthecompletiontime.

• Solution:Closetocompletion,spawnbackupcopiesoftheremainingin-progresstasks.

• Whicheveronefinishes first,“wins”.

• Additionalcost:afewpercentmoreresourceusage.• Example:Asortprogramwithoutbackup=44%longer.

16

SketchtheGFSarchitecturepresentingthecomponentsthatconstitutesitandthemaininteractions.

GFS- Overview

18

Whatisthe“operatorplacement”problem?

Deploymentmodel

• Automaticdistributionofprocessing introducestheoperatorplacementproblem

• Givenasetofrules(composedofoperators)andasetofnodes

• Howtosplittheprocessingload• Howtoassignoperatorstoavailablenodes

• Inotherwords• Givenaprocessingnetwork• Howtomapitontothephysicalnetworkofnodes

20

ExplainthesemanticsofthefollowingEsper query

select name, avg(price) as averagePricefrom StockTickEvent.win:length(100)group by name

ExplainthesemanticsofthefollowingEsper query

select name, avg(price) as averagePricefrom StockTickEvent.win:length(100)group by name

• Returnstheaveragepricepernameforthelast100stockticks

Inthecontextofconcurrency,whatisthe”happens-before”relationship?

23

Whatisahappens-beforerelationship?

24

• Whenasynchronizedmethodexits,itestablishesahappens-beforerelationshipwithanysubsequentinvocationofasynchronizedmethodforthesameobject.

• Whenthehappens-before relationisestablishedbyaprogrammer,e.g.,bymeansofsynchronization,wehavetheguaranteethatmemorywritesbystatementAexecutedbyThreadTAarevisibletoanotherspecificstatementBexecutedbyThreadTB.

Explainwhatafutureis

25

Explainwhatafutureis

26

• Placeholderobjectforavaluethatmaynotyetexist• ThevalueoftheFutureissuppliedconcurrentlyandcansubsequentlybeused

ExplainwhatOSGi is

27

OSGi

28

• TheOSGi specificationsdefineastandardized,componentoriented,computingenvironmentfornetworkedservices thatisthefoundationofanenhancedserviceorientedarchitecture

• TheOSGi ServicePlatformisaJavabasedapplicationserver fornetworkeddevices...

• TheOSGi ServicePlatformis[...] consideredtobethecheapest,fastestandeasiestwaytoenablethedynamicdeploymentofWeb2.0servicesandmashups inthenextgenerationJavaServicePlatform

• TheOSGi specifications[...] formasmalllayerthatallowsmultiple,Javabased,componentstoefficientlycooperateinasingleJavaVirtualMachine

Whatisawebservice?

29

Whatisawebservice?

30

• [...] self-contained,modularbusinessapplicationsthathaveopen,internet-oriented, standards-basedinterfaces.

Whatisawebservice?

31

Whatisawebservice?

32

Whatisawebservice?

• Thecomponents:• Servicerequester:Thepotentialuserofaservice

• Serviceprovider:Theentitythatimplements theserviceandofferstocarryitoutonbehalfoftherequester

• Serviceregistry:Aplacewhere• availableservicesarelisted

33

WhichunderlyingdatastructureisusedbyApacheSpark?Showaminimal exampleandindicatewheresuchdatastructureisused.

34

RDD (ResilientDistributedDatasets)

• Restrictedformofdistributedsharedmemory• immutable,partitionedcollectionof records• canonlybebuiltthrough coarse-graineddeterministictransformations

(map, filter, join...)

• Efficientfault-toleranceusinglineage• Logcoarse-grainedoperations insteadoffine-graineddataupdates• AnRDDhasenough information abouthowit’sderivedfromother

dataset• Recompute lostpartitionsonfailure

35

SparkandRDDs

• ImplementsResilientDistributedDatasets(RDDs)

• OperationsonRDDs• Transformations:definesnewdatasetbasedonpreviousones• Actions:startsajobtoexecuteoncluster

• Well-designedinterfacetorepresentRDDs• Makesitveryeasyto

implement transformations• MostSparktransformation

implementation<20LoC

36

MoreonRDDs

Workwithdistributedcollectionsasyouwouldwithlocalones

• Resilientdistributeddatasets(RDDs)• Immutablecollectionsofobjectsspreadacrossacluster• Builtthroughparalleltransformations (map, filter,etc)• Automaticallyrebuiltonfailure• Controllablepersistence(e.g.,cachinginRAM)

• Differentstoragelevelsavailable,fallback todiskpossible

• Operations• Transformations (e.g.map,filter,groupBy, join)

• LazyoperationstobuildRDDsfromotherRDDs• Actions (e.g.count,collect,save)

• Returnaresultorwriteittostorage

WorkflowwithRDDs

• CreateanRDDfromadatasource: <list>• ApplytransformationstoanRDD:mapfilter• ApplyactionstoanRDD:collectcount

distFile = sc.textFile("...", 4) • RDDdistributed in4partitions• Elementsarelinesofinput• Lazyevaluationmeansnoexecutionhappensnow

38

GiveapossibleexplanationwhythecomputationofthePageRankissignificantlydifferentbetweenHadoop andSpark

171

80

23

14

020406080

100120140160180200

30 60

Iteratio

ntim

e(s)

Numberofmachines

HadoopSpark

Spark

• Fast,expressiveclustercomputingsystemcompatiblewithApacheHadoop

• WorkswithanyHadoop-supported storagesystem(HDFS,S3,Avro,…)

• Improvesefficiency through:• In-memorycomputing primitives• Generalcomputationgraphs

• Improvesusability through:• RichAPIsinJava,Scala,Python• Interactiveshell

Up to 100× faster

Often 2-10× less code

PageRank

• Givepagesranks(scores)basedonlinkstothem• Linksfrommanypagesè high rank• Linkfromahigh-rankpageè high rank

• Goodexampleofamorecomplexalgorithm• Multiple stagesofmap&reduce

• BenefitsfromSpark’sin-memorycaching• Multiple iterationsoverthesamedata

Image: en.wikipedia.org/wiki/File:PageRank-hi-res-2.png

Explaintworeasonswhydatacanbegeodistributed

Explaintwomotivationswhydatacanbegeodistributed

• Manylargedatasetsgeo-distributed,i.e.,splitacrosssites• Storednearresp.sources,frequentlyaccessingentities• Gatheredandstoredbydifferent (sub-)organizations yetsharedtowardsacommongoal

• E.g.,UScensus,Google“buckets”• Replicatedacrossdatacentersforavailability,incompletelytolimittheoverheadofupdates

• Manyanalysistasksinvolveseveraldatasets,whichmaybedistributed

• Legalconstraintsmayconfinecertaindatasetstospecificlocations

• The“cloud”isnotasingledatacenter• Inter-DClatency≠intra-DClatency

Whatisaresourcemanagementsystem,e.g.,ApacheYARN?

ResourceManagement

• Typicallyimplementedbyasystemdeployedacrossnodesofacluster

• Layerbelow“frameworks”likeHadoop• Onanynode, thesystemkeepstrackofavailabilities• Applicationsontopuseinformationandestimationsofownrequirements tochoosewheretodeploysomething

• RMsystems(RMSs)differinabstractions/interfaceprovidedandactual schedulingdecisions

Whatarebyzantinefailures?Howcantheybesolved?

ByzantineFaultTolerance

• Protect“computation”withBFT[Lamport,Shostak, Pease;TOPLAS’82]replication

• Processes(statemachines)withbenignandmaliciousfailures• 3f +1replicasforf failuresinasynchronousdistributedsystems

• Safetywithf +1• Liveness with2f +1insynchronoussystem

• Comparisonofoutputs

BFTReplication

BFTReplication

Whatishomomorphicencryption?Besidesthetechnicalities,explainthekeygoalitachieves.

FullyHomomorphic Encryption

GiventhescenarioX,whatisthetechnology/approachthatyouwouldrecommendforsolvingproblemY?• MapReduce• HDFS• Adatabase• Complexeventprocessing• ApacheSpark• OSGi• REST• …• ...

52

MapReducevs.TraditionalRDBMS

MapReduce TraditionalRDBMSDatasize Petabytes GigabytesAccess Batch Interactiveandbatch

Updates Writeonce,readmanytimes

Readandwritemanytimes

Structure Dynamicschema StaticschemaIntegrity Low High(normalizeddata)

Scaling Linear Non-linear(generalSQL)

53

ASummary

54

55

ProgrammingModel

DataOrganization

Decla

rativ

e

StructuredFlatRawTypes

Proced

ural

Event-drivenapplications

• Canweuseexistingtechnologiesforbatchprocessing?• Theyarenotdesigned tominimizelatency• Weneedawholenewmodel!

Esper inanutshell

• EPL:richlanguagetoexpressrules• Grounded ontheDSMSapproach

• Windowing• Relational select, join,aggregate,…• Relation-to-streamoperatorstoproduceoutput• Sub-queries

• Queriescanbecombined toformagraph• IntroducessomefeaturesofCEPlanguages

• Patterndetection

• Designedforperformance• High throughput• Lowlatency

Goals

Batch

Interactive Streaming

One stack to

rule them all?

§ Easy to combine batch, streaming, and interactive computations§ Easy to develop sophisticated algorithms§ Compatiblewith existing open source ecosystem (Hadoop/HDFS)

59