View
521
Download
0
Category
Preview:
DESCRIPTION
This presentation introduced Vortex by means of a running example. Throughout the presentation we will show how Vortex makes it easy to build a micro-blogging platform a la Twitter.
Citation preview
Angelo Corsaro, PhD Chief Technology Officer
angelo.corsaro@prismtech.com
Vortex Tutorial Part I
Cop
yrig
ht P
rism
Tech
, 201
4
Vortex enables seamless, ubiquitous, efficient and timely data sharing across mobile, embedded, desktop, cloud and web applications
Vortex is based on the OMG DDS standard
The Vortex PlatformVortex Device
Tools
Integration
Vortex Cloud
MaaS
Cop
yrig
ht P
rism
Tech
, 201
4
Vortex Architecture
Security Model
Cop
yrig
ht P
rism
Tech
, 201
4
Support for transport level security via TLS and DTLS
Support for fine grained access control
Access control plug-in compliant with DDS Security
Vortex Security Model
Arthur Dent
Arthur Dent
Ford Prerfect
Zaphod Beeblebrox
Marvin
Trillian
A(r,w), B(r)
A(r,w), B(r,w), X(r)
*(r,w)
*(r)
A(r,w), B(r,w), C(r,w)
Ford Prerfect
Zaphod Beeblebrox
Trillian
Marvin
A
B
A,BX
*
*
A,B,C
Identity Access RightsSessions are authenticated and communication is encrypted
Only the Topic included as part of the access rights are visible and accessible
Use Cases
Cop
yrig
ht P
rism
Tech
, 201
4
Smart Grids
AnalyticsBig Data Store
Cop
yrig
ht P
rism
Tech
, 201
4
E-Health
Hospital A Private Cloud
AnalyticsBig Data Store
Hospital B
Cop
yrig
ht P
rism
Tech
, 201
4
Smart City
http://bit.ly/connected-boulevard
Cop
yrig
ht P
rism
Tech
, 201
4
SESAR
PENS
PENS: Pan European Network Service FDP: Flight Data Processor CWP: Controller Working Position
IPv6 network SSM support
FDPFDPFDP
CWP
Tower
DeviceFDPFDPFDP Tower
Device
CWP
FDPFDPFDP Tower
Device
CWP
FDPFDPFDP
CWP
Tower
Device
EU Nation 4
EU Nation 1
EU Nation 2EU Nation 3
Building ChirpIt
Cop
yrig
ht P
rism
Tech
, 201
4
To explore the various features provided by the Vortex platform we will be designing and implementing a micro-blogging platform called ChirpIt. Specifically, we want to support the following features:
ChirpIt users should be able to “chirp”, “re-chirp”, “like” and “dislike” trills as well as get detailed statistics
The ChirpIt platform should provide information on trending topics — identified by hashtags — as well as trending users
Third party services should be able to flexibly access slices of produced trills to perform their own trend analysis
ChirpIt should scale to millions of users
ChirpIt should be based on a Lambda Architecture
ChirpIt Requirements
Cop
yrig
ht P
rism
Tech
, 201
4
ChirpIt Architecture
analytics
data centre
chirp
emotionsstats
batch layer
chirpit apps
Cloud Messaging
serving layer
chirp
emotionsstats
master dataset
view view view
chirp
emotionsstats
3rd party svcs
Speed layer
The Data Distribution Service
Cop
yrig
ht P
rism
Tech
, 201
4
DDS provides a Global Data Space abstraction that allows applications to autonomously, anonymously, securely and efficiently share data
DDS’ Global Data Space is fully distributed (decentralised), highly efficient and scalable
Data Distribution Service (DDS)
DDS Global Data Space
...
Data Writer
Data Writer
Data Writer
Data Reader
Data Reader
Data Reader
Data Reader
Data Writer
TopicAQoS
TopicBQoS
TopicCQoS
TopicDQoS
Cop
yrig
ht P
rism
Tech
, 201
4DataWriters and DataReaders are automatically and dynamically matched by the DDS Discovery
A rich set of QoS allows to control existential, temporal, and spatial properties of data
Data Distribution Service (DDS)
DDS Global Data Space
...
Data Writer
Data Writer
Data Writer
Data Reader
Data Reader
Data Reader
Data Reader
Data Writer
TopicAQoS
TopicBQoS
TopicCQoS
TopicDQoS
Cop
yrig
ht P
rism
Tech
, 201
4
Elegant and High Level Data Sharing Abstraction
Polyglot and platform independent
• Java, Scala, C, C++, C#, JavaScript, CoffeeScript etc.
• Android, Windows, Linux, VxWorks, etc.
Peer-to-Peer by nature, Brokered when useful
Key Highlights
Cop
yrig
ht P
rism
Tech
, 201
4
Content and Temporal Filtering (both sender and receiver filtering supported)
Queries
20+ QoS to control existential, temporal, and spatial properties of data
Key Highlights
Cop
yrig
ht P
rism
Tech
, 201
4
High Performance and Scalable
• ~30 usec latency*
• 7M msgs/sec node-to-node throughput*
Key Highlights
(*) Performance measured on Linux Host with i7 processor on a 1Gbps Ethernet network
Decomposing DDS
Information Organisation
Cop
yrig
ht P
rism
Tech
, 201
4
DDS data lives within a domain
A domain is identified with a non negative integer, such as 1, 3, 31
The number 0 identifies the default domain
A domain represent an impassable communication plane
Domain
DDS Domain
Cop
yrig
ht P
rism
Tech
, 201
4
Partitions are the mechanism provided by DDS to organise information within a domain
Access to partitions is controlled through QoS Policies
Partitions are defined as strings: - “system:telemetry” - “system:log” - “data:row-2:col-3”
Partitions addressed by name or regular expressions: - ”system:telemetry” - “data:row-2:col-*”
Partitions
Partitions
Information Definition
Cop
yrig
ht P
rism
Tech
, 201
4
A Topic defines a domain-wide information’s class
A Topic is defined by means of a (name, type, qos) tuple, where
• name: identifies the topic within the domain
• type: is the programming language type associated with the topic. Types are extensible and evolvable
• qos: is a collection of policies that express the non-functional properties of this topic, e.g. reliability, persistence, etc.
Topic
TopicTypeName
QoS
...
TopicAQoS
TopicBQoS
TopicCQoS
TopicDQoS
Cop
yrig
ht P
rism
Tech
, 201
4
As explained in the previous slide a topic defines a class/type of information
Topics can be defined as Singleton or can have multiple Instances
Topic Instances are identified by means of the topic key
A Topic Key is identified by a tuple of attributes -- like in databases
Remarks: - A Singleton topic has a single domain-wide instance - A “regular” Topic can have as many instances as the number of different key
values, e.g., if the key is an 8-bit character then the topic can have 256 different instances
Topic and Instances
ChirpIt Data Model
Cop
yrig
ht P
rism
Tech
, 201
4
Chirp Actions union ChirpBody switch (ChirpActionKind) { case CHIRP_KIND: string<128> chirp; case RECHIRP_KIND: string<128> chirp; string user; case LIKE_KIND: string luser; case DISLIKE_KIND: string duser; }; ! struct ChirpAction { ChirpHeader header; ChirpBody body; }; #pragma keylist ChirpAction header.id.cid
struct ChirpId { string uid; string cid; }; ! enum ChirpActionKind { CHIRP_KIND, RECHIRP_KIND, LIKE_KIND, DISLIKE_KIND }; struct ChirpHeader { ChirpId id; Location location; unsigned long long timestamp; ChirpActionKind kind; };
Cop
yrig
ht P
rism
Tech
, 201
4
ChirpIt Statistics
struct ChirpStats { ChirpId id; unsigned long rechirps; unsigned long likes; }; #pragma keylist ChirpStats id.cid
struct UserStats { string userId; unsigned long followers; unsigned long chirps; unsigned long followed; }; #pragma keylist UserStats userId
Cop
yrig
ht P
rism
Tech
, 201
4
User Actions
struct enum UserActionKind { FOLLOW, UNFOLLOW }; struct UserAction { UserActionKind kind; string fromUser; string toUser; };
Cop
yrig
ht P
rism
Tech
, 201
4
App Config
typedef sequence<string> Partitions; ! struct AppConfig { string userId; Partitions ps; unsigned short history; }; #pragma keylist UserConfig userId
ChirpIt Information Scoping
Cop
yrig
ht P
rism
Tech
, 201
4
To make the system scale properly we need to ensure that the chirps, rechirps, etc., are distributed only to the interested parties
As we have a single topic representing chirp actions, how can we ensure that information is received only by followers?
DDS Provides two mechanisms:
- Partitions: Allow to create information scopes
- Content Filters: Allow to receive the data that satisfy certains predicates
Scoping Information
Cop
yrig
ht P
rism
Tech
, 201
4
ChirpIt Architecture
ChirpAction
@wolverine
@drxchirp:wolverine
ChirpAction
@drx follows @wolverine
Cop
yrig
ht P
rism
Tech
, 201
4
@drx
ChirpAction
ChirpIt Architecture
data centre
ChirpAction
@wolverine
chirp:wolverine
ChirpAction
@magneto
chirp:magneto
@drx follows @wolverine and @magneto
Cop
yrig
ht P
rism
Tech
, 201
4
In summary partitions are used to scope information
Each user will “join” a partition for each followed party
Example:
- If @drx follows @magneto, @wolverine, @cyclops and @mistique then he will receive ChirpActions from the partitions:
• chirp:@magneto, chirp:@wolverine, chirp:@cyclops and chirp:@mistique
- On the other hand, regardless of the people that follows @wolverine he will always and only write ChirpActions in the partitions
• chirp:@wolverine
Using Partitions to Scope Chirps
Producing Information
Cop
yrig
ht P
rism
Tech
, 201
4
A DataWriter (DW) is a strongly typed entity used to produce samples for one or more instances of a Topic, with a given QoS
Conceptually, the DataWriter QoS should be the same as the Topic QoS or more stringent
However, DDS does enforce a specific relationship between the Topic and DataWriter QoS
DataWriter
DWTypeTopic
QoS
Cop
yrig
ht P
rism
Tech
, 201
4
The DataWriter controls the life-cycle of Topic Instances and allows to:
Define a new topic instance
Write samples for a topic instance
Dispose the topic instance
DataWriter
DWTypeTopic
QoS
Consuming Information
Cop
yrig
ht P
rism
Tech
, 201
4
A DataReader (DR) is a strongly typed entity used to access and/or consume samples for a Topic, with a given QoS
Conceptually, the DataReader QoS should be the same as the Topic QoS or less stringent
However, DDS does enforce a specific relationship between the Topic and DataReader QoS
DataReader
DRTypeTopic
QoS
Putting all Together
Cop
yrig
ht P
rism
Tech
, 201
4
DomainParticipant: Provides access to a data cloud -- called a domain in DDS
Topic: Domain-wide definition of a kind of Information
Publisher/Subscriber: Provide scope to data sharing through the concept of partitions DataReader/DataWriter: Allow to read/write data for a given topic in the partitions their Subscriber/Publisher are associated with.
DDS EntitiesDomain (e.g. Domain 123)
Domain Participant
Topic
Publisher
DataWrter
Subscriber
DataReader
Partition (e.g. “Telemetry”, “Shapes”, )
Topic Instances/Samples
TaTb
Tc
Tx
Ty
T1
T1 T3
ChirpIt Implementation — Chirping —
Cop
yrig
ht P
rism
Tech
, 201
4
ChirpOut App // Domain Participant defined by dependency injection // Define the topic val topic = Topic[ChirpAction](Config.ChirpActionTopic) // Define the publisher and link-‐it up to the “chirp:@user” partition val pubQos = PublisherQos() .withPolicy(Partition(Config.ChirpRootPartition + user)) implicit val pub = Publisher(pubQos) // Define the data writer and make it reliable and for the time being // Transient Local… Will refined this in the Part II val dwQos = DataWriterQos().withPolicies( Reliability.Reliable, Durability.TransientLocal) // Create the DataWriter val dw = DataWriter[ChirpAction](pub, topic, dwQos) // Chirp!!! val chirp = … dw.write(chirp)
Cop
yrig
ht P
rism
Tech
, 201
4
ChirpIn // Domain Participant defined by dependency injection // Define the topic val topic = Topic[ChirpAction](Config.ChirpActionTopic) // Define the subscribe and link-‐it up to the partition associated to // the user followed val subQos = PublisherQos() .withPolicy(Partition(followerList.map(Config.ChirpRootPartition + _)) implicit val sub = Subscriber(subQos) // Define the data reader and make it reliable and for the time being // Transient Local… Will refined this in the Part II val drQos = DataReaderQos().withPolicies( Reliability.Reliable, Durability.TransientLocal) // Create the DataWriter val dr = DataReader[ChirpAction](sub, topic, drQos)
Cop
yrig
ht P
rism
Tech
, 201
4
ChirpIn
// React to new Chirp dr listen { case DataAvailable(_) => { // take vs. read will become clear in Part II val samples = dr.take() // For the time being simply print-‐out the new chirps samples .filter(s => (s.getData != null)) .foreach(s => displayChirp) } }
Cop
yrig
ht P
rism
Tech
, 201
4
ChirpOut App// Define the topic — assuming the topic is registered with the server…!topic = new dds.Topic(0, 'Chirp', ‘com.ChirpIt’)!// Define the data writer Qos!dwQos = new dds.DataWriterQos(! dds.Reliability.Reliable, ! dds.Durability.TranslientLocal,! dds.Partition(userPartition))!!// Use an Option monad to maintain the writer!chirpWriter = z_.None!!// Create the writer when the runtime is connected!runtime.onconnect(() ->! dw = new dds.DataWriter(runtime, topic, dwQos)! chirpWriter = z_.Some(dw)!)!!// …!// Write chirps!chirp = …!chirpWriter.map((dw) -> dw.write(chirp))!
Cop
yrig
ht P
rism
Tech
, 201
4
ChirpIn App// Define the topic — assuming the topic is registered with the server…!topic = new dds.Topic(0, 'Chirp', ‘com.ChirpIt’)!// Define the data reader Qos!drQos = new dds.DataWriterQos(! dds.Reliability.Reliable, ! dds.Durability.TranslientLocal,! dds.Partition(followedList))!!// Use an Option monad to maintain the reader!chirpReader = z_.None!!// Create the writer when the runtime is connected!runtime.onconnect(() ->! dr = new dds.DataReader(runtime, topic, dwQos)! // Attach a listener! dr.addListener((chirp) -> diplayChirp(chirp))! chirpReader = z_.Some(dr)!)
Cop
yrig
ht P
rism
Tech
, 201
4
ChirpIt Thus Far…data centre
chirp
emot
ions
stat
s
chirp
emot
ions
stat
s
chirp
emot
ions
stat
s
chirp
emot
ions
stat
s
Live Demo
Cop
yrig
ht P
rism
Tech
, 201
4
$ ./chirpOut @wolverin chirp>> @magneto will win!$ ./chirpOut @magneto
@magneto>> @wolverine chirping while asleep… again!
Download the package zip or tar.gz package available at:
- https://dl.dropboxusercontent.com/u/19238968/webcast/2014/10/chirpit.zip
- https://dl.dropboxusercontent.com/u/19238968/webcast/2014/10/chirpit.tar.gz
Extract the package somewhere in your file-system and open a terminal at that location and run the app as shown below
ChirpIt Demo
$ ./chirpIn @magneto @wolverinse @mystique @wolverin>> @magneto will win! @magneto>> @wolverine chirping while asleep… again!
Cop
yrig
ht P
rism
Tech
, 201
4
In Part II we extend our implementation to manage application configuration storage and analytics
To do so, we will need to introduce additional features provided by the DDS and the Vortex Platform, such as Durability and Integration with Big-Data Stores
What’s Next
Cop
yrig
ht P
rism
Tech
, 201
4
In this presentation we have performed the first step toward implementing a micro-blogging platform
The combination of Vortex Device and Vortex Cloud made it very easy to create and deploy our Internet-Scale, multi-device micro-blogging platform
Some of the DDS QoS, such as Durability have already shown how historical data, i.e. chirps made while we were offline — can be provided by the platform
The Part II will illustrate how the complete application fits together and what is the role of Vortex in the Lambda Architecture
Summary
Cop
yrig
ht P
rism
Tech
, 201
4
Online Resources
Recommended