Practical Aggregate Programming in Scala

Practical Aggregate Programming in Scala

Roberto CasadeiPhD Student in CS&Eng

[email protected]

Department of Computer Science and Engineering

University of Bologna

Student talk at Scala Symposium, Amsterdam 2016

Slides available at http://www.slideshare.net/RobertoCasadei/presentationsSample code at https://bitbucket.org/metaphori/scafi-tutorial

R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 1 / 30

[email protected]

http://www.slideshare.net/RobertoCasadei/presentations

https://bitbucket.org/metaphori/scafi-tutorial

Outline

1 Aggregate Computing: The Basics

2 SCAFI: Practical Aggregate Programming in Scala

3 Conclusion


Aggregate Computing: The Basics

Outline



3 Conclusion



Problem: design/programming CASs

Collective/Complex Adaptive Systems (CASs)

Structure: Environment + (Mobile, Large-scale) Networks of { people + devices }

Global interpretation: embedded devices collectively form a “diffused” computational system



An approach to CAS development

Issues⇒ approach

• Large-scale⇒ decentralised coordination• Situatedness + distributed autonomy⇒ substantial unpredictability⇒ self-*• Complex collective behavior⇒ good abstractions, layered approach, compositionality

Shifting the mindset: from local to global

• Declarativeness and the global viewpoint• Crowd-aware services• Failure recovery of enterprise services• Distributed monitoring and reacting (e.g., temperature, fire)

• Expected global behavior vs. traditional device-centric interface

⇒ Aggregate Programming [BPV15]: a paradigm for programming whole aggregates of devices.



Aggregate programming [BPV15]From the local/device-centric viewpoint to the global/aggregate viewpoint

Aggregate programming: what

Goal: programming the collective behaviour of aggregates (of devices)⇒ global-to-local

Aggregate programming: how

Prominent approach (generalizing over several prior approaches and strategies [BDU+12])founded on field calculus and self-org patterns• Computational fields as unifying abstraction of local/global viewpoints



Aggregate Programming Stack



Aggregate (computing) systems & Execution model

Structure⇒ (network/graph)

• A set of devices (aka nodes/points/things).• Each device is able to communicate with a subset of devices known as its neighbourhood.

Dynamics

Each device is given the same aggregate program and works at async / partially-sync rounds:

(1) Retrieve context⇐ Messages from neighbours⇐ Sensor values

(2) Aggregate program execution⇒ export (a tree-like repr of computation) + output (result of last expr in body)

(3) Broadcast export to neighbourhood(4) Execute actuators


SCAFI: Practical Aggregate Programming in Scala

Outline



3 Conclusion



SCAFI: Scala with Computational FieldsGoal: bring Aggregate Computing to the field of mainstream software development

WhatSCAFI [CV16] is an integrated framework for building systems with aggregate programming.

• Scala-internal DSL for expressing aggregate computations.• Linguistic support + execution support (interpreter/VM)• Correct, complete, efficient impl. of the Higher-Order Field Calculus

semantics [DVPB15]• Distributed platform for execution of aggregate systems.

• Support for multiple architectural styles and system configurations.• Actor-based implementation (based on Akka).

Where• https://bitbucket.org/scafiteam/scafi

libraryDependencies += "it.unibo.apice.scafiteam" % "scafi-core_2.11" % "0.1.0" // on Maven Central


https://bitbucket.org/scafiteam/scafi


Computational fields [DVB16]

• (Abstract interpretation) Mapping space-time to computational objects• (Concrete interpretation) Mapping devices to values: φ : δ 7→ `

• “Distributed” data structure working as the global abstraction• The bridge abstraction between local behavior and global behavior

Discrete systems as an approximation ofspacetime



Field/aggregate computations

Global viewpoint

• Aggregate interpretation• Natural/denotational semantics• Program: computation over whole fields• Output (at a given time): system-wide

snapshot of a computational field• Geometric view: properties of collections

of points

Local viewpoint

• Device-centric interpretation• Operational semantics• Program: steps of a single device• Output (at a given time): latest value

yielded by the device• Geometric view: properties of a single

point



So, what is an aggregate program?

• The global program– "Local programs" obtained via global-to-local mapping

• May take the form of field calculus programs (in the representation given by some PL)– Actually, the field calculus is like FJ for Java, or the lambda calculus for Haskell

• An aggregate program consists of1) A set of function definitions2) A body of expressions.

• Example: an aggregate program in SCAFIclass MyProgram extends AggregateProgram with MyAPI {def isSource = sense[Boolean]("source")

// Entry point for executionoverride def main() = gradient(isSource)

}

– Each device of the aggregate system is given an instance of MyProgram.– Each device repeatedly runs the main method at async rounds of execution.



Computing with fields

Expressing aggregate/field computations in SCAFItrait Constructs {def rep[A](init: A)(fun: (A) => A): Adef nbr[A](expr: => A): Adef foldhood[A](init: => A)(acc: (A,A)=>A)(expr: => A): Adef aggregate[A](f: => A): A

// Not primitive, but foundationaldef sense[A](name: LSNS): Adef nbrvar[A](name: NSNS): Adef branch[A](cond: => Boolean)(th: => A)(el: => A): A

}

• Mechanisms for context-sensitiveness: nbr, nbrvar, sense

• Mechanisms for field evolution: rep

• Mechanisms for interaction: nbr

• Mechanisms for field domain restriction and partitioning: aggregate, branch

• Reference formal system: field calculus [DVB16, DVPB15]



Simple fields

0

(x)=>x+1

true t<0,1>

Constant, uniform field: 5

– Local view: evaluates to 5 in the context of a single device– Global view: yields a uniform constant field that holds 5 at any point (i.e., at any device)

Constant, non-uniform field: mid()

– mid() is a built-in function that returns the ID of the running device



rep: dynamically evolving fields

rep

0

(x)=>x+1t

v0

t

v1

..rep(0){(x)=>x+1}

// Signature: def rep[A](init: A)(fun: (A) => A): A// Initially 0; state is incremented at each roundrep(0){ _+1 }

– Notice: the frequency of computation can vary over time and from device to device– In general, the resulting field will be heterogeneous in time and space



nbr: interaction, communication, observation

nbr de

nbr{e}

φd=[d1→v1,..,dn→vn]

– Local view: nbr returns a field from neighbors to their corresponding value of the given expr e– Global view: a field of fields– Needs to be reduced using a *hood operation– foldhood works by retrieving the value of expr for each neighbour and then folding over the

resulting structure as you’d expect from FP.

// Signature: def nbr[A](expr: => A): A// Signature: def foldhood[A](init: => A)(acc: (A,A)=>A)(expr: => A): Afoldhood(0)(_+_){ nbr{1} }



Context-sensitiveness and sensorsLocal context:

1) The export of the previous computation2) Messages received from neighbours3) Values perceived from the physical/software environment

Sensing// Query a local sensorsense[Double]("temperature")

// Compute the maximum distance from neighboursfoldhood(Double.MinValue)(max(_,_)){ // Also: maxHood {...}nbrvar[Double](NBR_RANGE_NAME)

}

– nbr queries a local sensor– nbrvar queries a "neighbouring sensor" (a sort of "environmental probe")



Field domain restriction

Alignment

• Aggregate computations can be represented as a trees• Device exports are "paths" along these trees• When two devices execute the same tree node, they are said to be aligned• Interaction is possible only between aligned devices

Use cases for branch

• Partitioning the space into subspaces performing subcomputations• Regulating admissible interactions (i.e., further restricting the neighbourhood)

branch(sense[Boolean]("flag")){compute(...) // sub-computation

}{Double.MaxValue // stable value (i.e., not computing)

}



Functions

Two kinds of functions in SCAFI:

1) "Normal" Scala functions: serve as units for encapsulating behavior/logicdef foldhoodMinus[A](init: => A)(acc: (A,A) => A)(ex: => A): A =foldhood(init)(acc){ mux(mid()==nbr(mid())){ init }{ ex } }

def isSource = sense[Boolean]("source")

2) First-class "aggregate" functions [DVPB15] – which also work as units for alignmentdef branch[A](cond: => Boolean)(th: => A)(el: => A): A =mux(cond, ()=>aggregate{ th }, ()=>aggregate{ el })()



Example: the gradient [BBVT08]def nbrRange = nbrvar[Double](NBR_RANGE_NAME)

def gradient(source: Boolean): Double =rep(Double.PositiveInfinity){ distance =>mux(source) {0.0

}{minHood { nbr{distance} + nbrRange }

}}



Example: the channel I



Example: the channel IIEach device is given the same aggregate program:class ChannelProgram extends AggregateProgram with ChannelAPI {

def main = channel(isSource, isDestination, width)}

def channel(src: Boolean, dest: Boolean, width: Double) =distanceTo(src) + distanceTo(dest) <= distBetween(src, dest) + width



Example: the channel IIItrait ChannelAPI extends Language with Builtins {def channel(src: Boolean, dest: Boolean, width: Double) =

distanceTo(src) + distanceTo(dest) <= distBetween(src, dest) + width

def G[V:OB](src: Boolean, field: V, acc: V=>V, metric: =>Double): V =rep( (Double.MaxValue, field) ){ dv =>mux(src) { (0.0, field) } {minHoodMinus {val (d, v) = nbr { (dv._1, dv._2) }(d + metric, acc(v))

}}

}._2

def broadcast[V:OB](source: Boolean, field: V): V =G[V](source, field, x=>x, nbrRange())

def distanceTo(source: Boolean): Double =G[Double](source, 0, _ + nbrRange(), nbrRange())

def distBetween(source: Boolean, target: Boolean): Double =broadcast(source, distanceTo(target))

def nbrRange(): Double = nbrvar[Double](NBR_RANGE_NAME)def isSource = sense[Boolean]("source"); def isDestination = sense[Boolean]("destination")

}



Scaling with complexity

General coordination operators [VBDP15]

• Gradient-cast: accumulates values “outward” along a gradient starting from source nodes.def G[V:OB](src: Boolean, init: V,

acc: V=>V, metric: =>Double): V

• Converge-cast: collects data distributed across space “inward” by accumulating values fromedge nodes to sink nodes down a “potential” field.def C[V:OB](potential: V, acc: (V,V)=>V, local: V, Null: V): V

• Time-decay: supports information summarisation across time.def T[V:Numeric](initial: V, floor: V, decay: V=>V): V

• Sparse-choice: supports creation of partitions and selection of sparse subsets of devices inspacedef S(grain: Double, metric: => Double): Boolean



A case study: crowd engineering [CPV16]

val (high,low,none) = (2,1,0) // crowd level

def crowdWarning(p: Double, r: Double, warn: Double, t: Double):Boolean =

distanceTo(crowdTracking(p,r,t) == high) < warn

def crowdTracking(p: Double, r: Double, t: Double) = {val crowdRgn = recentTrue(densityEst(p, r)>1.08, t)branch(crowdRgn){ dangerousDensity(p, r) }{ none }

}

def dangerousDensity(p: Double, r: Double) = {val mr = managementRegions(r*2, () => { nbrRange })val danger = average(mr, densityEst(p, r)) > 2.17 &&

summarize(mr, (_:Double)+(_:Double), 1 / p, 0) > 300mux(danger){ high }{ low }

}

// Auxiliary functionsdef recentTrue(state: Boolean, memTime: Double): Booleandef managementRegions(grain: Double,

metric: => Double): Boolean = S(gran,metric)def densityEst(p: Double, range: Double): Doubledef summarize(sink: Boolean, acc: (Double,Double)=>Double,

local: Double, Null: Double): Doubledef average(sink: Boolean, value: Double): Double



Quick platform setup// STEP 1: CHOOSE INCARNATIONimport it.unibo.scafi.incarnations.{ BasicActorP2P => Platform }

// STEP 2: DEFINE AGGREGATE PROGRAM SCHEMAclass Demo_AggregateProgram extends Platform.AggregateProgram {override def main(): Any = foldhood(0){_ + _}(1)

}

// STEP 3: DEFINE MAIN PROGRAMobject Demo_MainProgram extends Platform.CmdLineMain

1) Demo_MainProgram -h 127.0.0.1 -p 9000-e 1:2,4,5;2;3 --subsystems 127.0.0.1:9500:4:5--program "demos.Demo_AggregateProgram"

2) Demo_MainProgram -h 127.0.0.1 -p 9500-e 4;5:4--program "demos.Demo_AggregateProgram"



Manual node setup// STEP 1: CHOOSE INCARNATIONimport scafi.incarnations.{ BasicActorP2P => Platform }import Platform.{AggregateProgram,Settings,PlatformConfig}

// STEP 2: DEFINE AGGREGATE PROGRAM SCHEMAclass Program extends AggregateProgram with CrowdAPI {// Specify a "dangerous density" aggregate computationoverride def main(): Any = crowdWarning(...)

}

// STEP 3: PLATFORM SETUPval settings = Settings()val platform = PlatformConfig.setupPlatform(settings)

// STEP 4: NODE SETUPval sys = platform.newAggregateApplication()val dm = sys.newDevice(id = Utils.newId(),

program = Program,neighbours = Utils.discoverNbrs())

val devActor = dm.actorRef // get underlying actor


Conclusion

Outline



3 Conclusion


Conclusion

Summary: key ideas

Aggregate programming

• A "macro" (programmingengineering) approach to CASs, formally grounded in the FieldCalculus.

• Allows to compose “emergent” phenomena & defines layers of (self-stabilizing) building blocks.

SCAFI: a Scala framework for Aggregate Programming

• Provides an internal DSL for field-based computations• Provides an actor-based platform for building aggregate systems

Future work• Evolve SCAFI to support scalable computations in cluster- and cloud-based systems.• What does it take to set up a framework for adaptive execution strategies?


Conclusion

Question time

Questions?


Appendix References

References I

[BBVT08] Jacob Beal, Jonathan Bachrach, Dan Vickery, and Mark Tobenkin.Fast self-healing gradients.In Proceedings of the 2008 ACM symposium on Applied computing, pages 1969–1975.ACM, 2008.

[BDU+12] Jacob Beal, Stefan Dulman, Kyle Usbeck, Mirko Viroli, and Nikolaus Correll.Organizing the aggregate: Languages for spatial computing.CoRR, abs/1202.5509, 2012.

[BPV15] Jacob Beal, Danilo Pianini, and Mirko Viroli.Aggregate Programming for the Internet of Things.IEEE Computer, 2015.


Appendix References

References II

[CPV16] Roberto Casadei, Danilo Pianini, and Mirko Viroli.Simulating large-scale aggregate mass with alchemist and scala.In Maria Ganzha, Leszek Maciaszek, and Marcin Paprzycki, editors, Proceedings of theFederated Conference on Computer Science and Information Systems (FedCSIS2016), Gdansk, Poland, 11-14 September 2016. IEEE Computer Society Press.To appear.

[CV16] Roberto Casadei and Mirko Viroli.Towards aggregate programming in Scala.In First Workshop on Programming Models and Languages for Distributed Computing,PMLDC ’16, pages 5:1–5:7, New York, NY, USA, 2016. ACM.

[DVB16] Ferruccio Damiani, Mirko Viroli, and Jacob Beal.A type-sound calculus of computational fields.Science of Computer Programming, 117:17 – 44, 2016.


Appendix References

References III

[DVPB15] Ferruccio Damiani, Mirko Viroli, Danilo Pianini, and Jacob Beal.Code mobility meets self-organisation: A higher-order calculus of computational fields.volume 9039 of Lecture Notes in Computer Science, pages 113–128. SpringerInternational Publishing, 2015.

[VBDP15] Mirko Viroli, Jacob Beal, Ferruccio Damiani, and Danilo Pianini.Efficient engineering of complex self-organising systems by self-stabilising fields.2015.


Technology

Practical Aggregate Programming in Scala