OpenSplice DDS Tutorial -- Part II

OpenSplice | DDS Tutorial-- Part II --

Angelo CORSARO, Ph.D. Chief Technology Officer OMG DDS Sig Co-Chair

PrismTech [email protected]

Part I Recap

DDS is a standard technology for ubiquitous, interoperable, secure, platform independent, and real-time data sharing

across network connected devices

Copyrig

ht 2013, PrismTech – A

ll Rights Reserved.

Data Distribution Service (DDS)

• DDS provides a Global Data Space abstraction that allow applications to autonomously, anonymously, securely and efficiently share data

• DDS’ Global Data Space is fully distributed, highly efficient and scalable

DDS Global Data Space

...

Data Writer

Data Writer

Data Writer

Data Reader

Data Reader

Data Reader

Data Reader

Data Writer

TopicAQoS

TopicBQoS

TopicCQoS

TopicDQoS

Copyrig


ll Rights Reserved.

Copyrig


ll Rights Reserved.

DDS Entities

• DomainParticipant: Provides access to a data cloud -- called a domain in DDS

• Topic: Domain-wide definition of a kind of Information

• Publisher/Subscriber: Provide scope to data sharing through the concept of partitions

• DataReader/DataWriter: Allow to read/write data for a given topic in the partitions their Subscriber/Publisher are associated with.

Domain (e.g. Domain 123)

Domain Participant

Topic

Publisher

DataWrter

Subscriber

DataReader

Partition (e.g. “Telemetry”, “Shapes”, )

Topic Instances/Samples

TaTb

Tc

Tx

Ty

T1

T1 T3

Part II

Quality of Service

Copyrig


ll Rights Reserved.

QoS Model• QoS-Policies control local and end-

to-end properties of DDS entities

• Local properties controlled by QoS are related resource usage

• End-to-end properties controlled by QoS are related to temporal and spatial aspects of data distribution

• Some QoS-Policies are matched based on a Request vs. Offered (RxO) Model

Publisher

DataWriter

Topic

Type

QoS

Name

writes

QoS

DataWriter

Topic

Typewrites

Subscriber

DataReaderreads

DataReaderreads

...

QoS

Name

QoS

QoS QoS

QoS matching

......

QoS QoS

Type Matching

DomainParticipant DomainParticipant

QoS QoS

Copyrig


ll Rights Reserved.

QoS Policies

QoS Policy Applicability RxO Modifiable

USER_DATA DP, DR, DW N Y

ConfigurationTOPIC_DATA T N Y

GROUP_DATA P, S N Y

DURABILITY T, DR, DW Y N

Data AvailabilityDURABILITY

SERVICET, DW N N

HISTORY T, DR, DW N N

PRESENTATION P, S Y N

Data Delivery

RELIABILITY T, DR, DW Y N

PARTITION P, S N Y

DESTINATION ORDER

T, DR, DW Y N

LIFESPAN T, DW N Y

[T: Topic] [DR: DataReader] [DW: DataWriter] [P: Publisher] [S: Subscriber] [DP: Domain Participant]

Copyrig


ll Rights Reserved.

QoS Policies


DEADLINE T, DR, DW Y Y

Temporal/Importance

Characteristics

LATENCY BUDGET T, DR, DW Y Y

TRANSPORT PRIORITY

T, DW N Y

TIME BASED FILTER DR N Y

OWNERSHIP T, DR, DW Y N

ReplicationOWNERSHIP STRENGTH

DW N Y

LIVELINESS T, DR, DW Y N Fault-Detection

[T: Topic] [DR: DataReader] [DW: DataWriter] [P: Publisher] [S: Subscriber] [DP: Domain Participant]

Copyrig


ll Rights Reserved.

QoS DSL

• The ISO C++ and Java 5 APIs provide DSL for dealing with QoS Policies configuration

• The DSL uses language specific idioms, such as fluid interfaces, as well as specific features of the languages

• Policies as well as Entity QoS are immutable — this allows for better safety and object sharing

• Policies are treated as algebraic data types and the DSL provide constructors of each of the cases

• A QoS Provider can now be used to retrieve QoS settings from external sources, e.g. a file, an HTTP server, DDS durability

Copyrig


ll Rights Reserved.

C++ QoS Policy DSL

!

!

// == ISO C++ DDS API == !DataWriterQos dwqos = pub.default_datawriter_qos() << History.KeepLast(10) << Durability.Transient(); !

Copyrig


ll Rights Reserved.

Java 5 QoS Policy DSL

!

!

// == Java 5 DDS API == !final PolicyFactory pf = ... !DataWriterQos dwqos = pub.getDefaultDataWriterQos() .withPolicies ( pf.History.withKeepLast(10), pf.Durability.withTransient(), ); !

Copyrig


ll Rights Reserved.

Data Delivery

Data Delivery

Reliability

Presentation

Destination OrderPartition

Copyrig


ll Rights Reserved.

Reliability QoS Policy

The Reliability Policy controls the level of guarantee offered by the DDS in delivering data to subscribers

• Reliable. In steady-state, and with no data writer crashes, guarantees that all samples in the DataWriter history will eventually be delivered to all the DataReader

• Best Effort. Indicates that it is acceptable not to retry propagation of samples


RELIABILITY T, DR, DW Y N

Copyrig


ll Rights Reserved.

Setting Reliability Policy

!

!

// == ISO C++ DDS API == !// Reliable Reliability DataWriterQos dwqos = pub.default_datawriter_qos() << Reliability.Reliable(); !// Best-‐Effort Reliability DataWriterQos dwqos = pub.default_datawriter_qos() << Reliability.BestEffort();

Copyrig


ll Rights Reserved.

Setting Reliability Policy

!

!

// == Java 5 DDS API == !final PolicyFactory pf = ... !// Reliable Reliability DataWriterQos dwqos = pub.getDefaultDataWriterQos() .withPolicies(pf.Reliability.withReliable()); !// Best-‐Effort Reliability DataWriterQos dwqos = pub.getDefaultDataWriterQos() .withPolicies(pf.Reliability.withBestEffort());

Copyrig


ll Rights Reserved.

Partitions

• Partitions are the mechanism provided by DDS to organise information within a domain

• Access to partitions is controlled through QoS Policies

• Partitions are defined as strings:

• “system:telemetry”

• “system:log”

• “data:row-‐2:col-‐3”

• Partitions addressed by name or regular expressions:

• ”system:telemetry”

• “data:row-‐2:col-‐*”

Partitions

Copyrig


ll Rights Reserved.

Partition QoS Policy

• The Partition QoS Policy can be used as subjects for organising the flows of data

• The Partition QoS Policy is used to connect Publishers/Subscribers to a Partitions’ List which might also contain wildcards, e.g. tracks.*

• Topics instances are published and subscribed across one or more Partitions

Partitions

Copyrig


ll Rights Reserved.

Setting Partition QoS Policy

!

!

// == ISO C++ DDS API == !// Setting a Partition PublisherQos pqos = dp.default_publisher_qos() << Partition(“MyPartition”); !// Setting a List of Partitions PublisherQos pqos = dp.default_publisher_qos() << Partition(myPartitionList);

Copyrig


ll Rights Reserved.

Setting Partition QoS Policy

!

!

// == Java 5 DDS API == !final PolicyFactory pf = ... !// Setting a Partition PublisherQos pqos = dp.getDefaultPublisherQos() .withPolicies(pf.Partition(“MyPartition”)); !// Setting a List of Partitions PublisherQos pqos = dp.getDefaultPublisherQos() .withPolicies(pf.Partition(myPartitionList));

Copyrig


ll Rights Reserved.

Data Availability

Data Availability

History

Ownership

DurabilityLifespan

OwnershipStrength

Copyrig


ll Rights Reserved.

History QoS PolicyThe DataWriter HISTORY QoS Policy controls the amount of data that can be made available to late joining DataReaders under TRANSIENT_LOCAL Durability

The DataReader HISTORY QoS Policy controls how many samples will be kept on the reader cache

• Keep Last. DDS will keep the most recent “depth” samples of each instance of data identified by its key

• Keep All. The DDS keep all the samples of each instance of data identified by its key -- up to reaching some configurable resource limits

QoS Policy Applicability RxO ModifiableHISTORY T, DR, DW N N

0 1 2 3

0

1

2

3

4

Pressure time

Pressure time

Pressure time

KeepLast(3)

KeepLast(1)

KeepAll

Copyrig


ll Rights Reserved.

Durability QoS PolicyThe DURABILITY QoS controls the data availability w.r.t. late joiners, specifically the DDS provides the following variants:

• Volatile. No need to keep data instances for late joining data readers

• Transient Local. Data instance availability for late joining data reader is tied to the data writer availability

• Transient. Data instance availability outlives the data writer

• Persistent. Data instance availability outlives system restarts

QoS Policy Applicability RxO ModifiableDURABILITY T, DR, DW Y N

Copyrig


ll Rights Reserved.

Data Writer

Data Reader

TopicAQoS

Volatile Durability

• No Time Decoupling

• Readers get only data produced after they joined the Global Data Space

1

Copyrig


ll Rights Reserved.

!

!

Volatile Durability



Data Writer

Data Reader

TopicAQoS

Data Reader

1

Late Joiner

Copyrig


ll Rights Reserved.

!

!

Volatile Durability



Data Writer

Data Reader

TopicAQoS

Data Reader

1

Late Joiner

22

Copyrig


ll Rights Reserved.

Data Writer

Data Reader

TopicAQoS

Transient Local Durability

• Some Time Decoupling

• Data availability is tied to the availability of the data writer and the history settings

11

Copyrig


ll Rights Reserved.

!

!

Transient Local Durability



Data Writer

Data Reader

TopicAQoS

Data Reader

1

Late Joiner

11

Copyrig


ll Rights Reserved.

Data Writer

Data Reader

TopicAQoS

Data Reader

2

Transient-Local Durability



1

2

1

12

Copyrig


ll Rights Reserved.

Data Writer

Data Reader

TopicAQoS

Transient Durability

• Time Decoupling

• Data availability is tied to the availability of the durability service

1

1

Copyrig


ll Rights Reserved.

!

!

Data Writer

Data Reader

TopicAQoS

Data Reader

1


• Time Decoupling


1

Late Joiner

1

Copyrig


ll Rights Reserved.

Data Writer

Data Reader

TopicAQoS

Data Reader

2


• Time Decoupling


1

2

1

1

2

Copyrig


ll Rights Reserved.

Data Reader

TopicAQoS

Data Reader


• Time Decoupling


1

1

2

2

12

Copyrig


ll Rights Reserved.

!

!


• Time Decoupling


1

Late Joiner

1

Data Reader

TopicAQoS

Data Reader

Data Reader

2

2

12 12

Copyrig


ll Rights Reserved.

Setting the Durability Policy

!

!

// == Java 5 DDS API == !final PolicyFactory pf = ... !// Setting TransientLocalDurability PublisherQos pqos = dp.getDefaultPublisherQos() .withPolicies(pf.Durability().withTransientLocal()); !

Copyrig


ll Rights Reserved.

Ownership QoS Policy

Availability of data producers can be controlled via two QoS Policies

• OWNERSHIP (SHARED vs. EXCLUSIVE)

• OWNERSHIP STRENGTH

• Instances of exclusively owned Topics can be modified (are owned) by the higher strength writer

• Writer strength is used to coordinate replicated writersQoS Policy Applicability RxO Modifiable

OWNERSHIP T, DR, DW Y NSTRENGTH DW N Y

Copyrig


ll Rights Reserved.

Temporal Properties

Throughput

TimeBasedFilter

[Inbound]

[Outbound]Latency

Deadline

TransportPriority

LatencyBudget

Copyrig


ll Rights Reserved.

Latency Budget

• The LATENCY_BUDGET QoS policy specifies the maximum acceptable delay from the time the data is written until the data is inserted in the receiver's application-cache

• A non-zero latency-budget allows a DDS implementation to batch samples and improve CPU/Network utilisation

QoS Policy Applicability RxO ModifiableLATENCY BUDGET

T, DR, DW Y Y

DataWriter DataReaderT1

T2

T3

Latency = T1+T2+T3

Batching

Copyrig


ll Rights Reserved.

Deadline QoS Policy

The DEADLINE QoS policy defines the maximum inter-arrival time between data samples

• DataWriter indicates that the application commits to write a new sample at least once every deadline period

• DataReaders are notified when the DEADLINE is violated

DataWriter DataReaderDeadlineDeadlineDeadlineDeadlineDeadline

Deadline Violation

QoS Policy Applicability RxO ModifiableDEADLINE T, DR, DW Y Y

Copyrig


ll Rights Reserved.

Transport Priority QoS Policy

• The TRANSPORT_PRIORITY QoS policy is a hint to the infrastructure as to how to set the priority of the underlying transport used to send the data.

QoS Policy Applicability RxO ModifiableTRANSPORT

PRIORITYT, DW N Y

Copyrig


ll Rights Reserved.

Time-Based Filter QoS Policy

• The Time Based Filter allows to control the throughput at which data is received by a data reader

• Samples produced more often than the minimum inter-arrival time are not delivered to the data reader

QoS Policy Applicability RxO ModifiableTIME BASED FILTER DR N Y

DataWriter DataReader

mit

T2

T3

Latency = T1+T2+T3

discarded sample

mit

mit

mit

mit = minimum inter-arrival time

produced sample delivered sample

Copyrig


ll Rights Reserved.

!

!

QoS Provider

• The new C++ and Java APIs introduce the concept of a QoS Provider

• This class allows to externally define policies and decouples the mechanism used to define and access policy definition with policy creation

// QosProvider... QosProvider qos_provider( "http://www.opensplice.org/demo/config/qos.xml", "ishapes-‐profile"); ! DataReader<ShapeType> dr(sub, topic, qos_provider.datareader_qos());

Demo: QoS in Action

Copyrig


ll Rights Reserved.

Copyrig


ll Rights Reserved.

Shapes Application

• The iShapes application is used by DDS vendors to demonstrate some of the basic mechanism as well as product interoperability

• Three Topics

• Circle, Square, Triangle

• One Type

struct ShapeType { string color; long x; long y; long shapesize; }; #pragma keylist ShapeType color

Spotted shapes represent subscriptions

Pierced shapes represent publications

QoS Idioms

Copyrig


ll Rights Reserved.

Soft State

• In distributed systems you often need to model soft-state -- a state that is periodically updated

• Examples are the reading of a sensor (e.g. Temperature Sensor), the position of a vehicle, etc.

• The QoS combination to model Soft-State is the following:

Reliability => BestEffort Durability => Volatile History => KeepLast(n) [with n = 1 in most of the cases] Deadline => updatePeriod LatencyBudget => updatePeriod/3 [rule of thumb] DestinationOrder => SourceTimestamp [if multiple writers per instance]

Copyrig


ll Rights Reserved.

Hard State

• In distributed systems you often need to model hard-state -- a state that is sporadically updated and that often has temporal persistence requirements

• Examples are system configuration, a price estimate, etc.

• The QoS combination to model Hard-State is the following:

Reliability => Reliable Durability => Transient | Persistent History => KeepLast(n) [with n = 1 in most of the cases] DestinationOrder => SourceTimestamp [if multiple writers per instance]

Copyrig


ll Rights Reserved.

Events

• In distributed systems you often need to model events -- the occurrence of something noteworthy for our system

• Examples are a collision alert, the temperature beyond a given threshold, etc.

• The QoS combination to model Events is the following:

Reliability => Reliable Durability => any [depends on system requirements] History => KeepAll [on both DataWriter and DataReader!] DestinationOrder => SourceTimestamp

Accessing the Reader Cache

Copyrig


ll Rights Reserved.

Reading Samples

• Samples can be read from the Data Reader History Cache

• The action of reading a sample is non-destructive. Samples are not removed from the cache

DataReader Cache

DataReader

...

DataReader Cache

DataReader

... read

Copyrig


ll Rights Reserved.

Taking Samples

• Samples can be taken from the Data Reader History Cache

• The action of taking a sample is destructive. Samples are removed from the cache

DataReader Cache

DataReader

...

DataReader Cache

DataReader

... take

Copyrig


ll Rights Reserved.

Sample Selectors

• DDS provides some very flexible mechanisms for selecting the samples to be read/take:

• Content

• Status

• These mechanisms are composable

Content-Based Selection

Copyrig


ll Rights Reserved.

Filters and Queries• DDS Filters allow to control what gets

into a DataReader cache

• DDS Queries allow to control what gets out of a DataReader cache

• Filters are defined by means of ContentFilteredTopics

• Queries operate in conjunction with read operations

• Filters and Queries are expressed as SQL where clauses

DataReader Cache

DataReader

...... ... ...

Filter

Query

Application

Copyrig


ll Rights Reserved.

!

!

Filters

// == ISO C++ DDS API !// Create a Topic auto topic = Topic<ShapeType>(dp, “Circle”); !// Define filter expression and parameters auto filter = Filter(“x < 100 AND y < 200”); !// Define content filtered topic auto cftopic = ContentFilteredTopic<ShapeType>(“CFCircle”, topic, filter) !// Create a DataReader for the content-filtered Topic auto dr = DataReader<ShapeType>(sub,cftopic)


Copyrig


ll Rights Reserved.

!

!

Filters

// == Java 5DDS API == !final PolicyFactory pf = runtime.policyFactory(); final DataReaderQos drqos = sub.getDefaultDataReaderQos() .withPolicy ( pf.ContentFilter() .withFilter( new JavaScriptFilter<ShapeType>("data.x > data.y")) ); !final DataReader<ShapeType> dr = sub.createDataReader(shape, drqos);


Copyrig


ll Rights Reserved.

!

!

Query

// == ISO C++ DDS API == !// Define filter expression and parameters auto dr = DataReader<ShapeType>(sub, topic) val query = Query(dr, “x < 100 AND y < 200”); !dr.select() .content(query) .read();


Copyrig


ll Rights Reserved.

!

!

Query

// == Java 5 DDS API == !Filter<ShapeType> filter = new JavaScriptFilter<ShapeType>("data.x > data.y")) !!dr.select() .content(filter) .read();


Copyrig


ll Rights Reserved.!

!

Instances

• DDS provides a very efficient way of reading data belonging to a specific Topic Instance

• Obviously, one could use queries to match the key’s value, but this is not as efficient as the special purpose instance selector

// == ISO C++ DDS API == !auto handle = dr.lookup_instance(ShapeType(“RED”, 0, 0, 0)); !auto data = dr.select() .instance(handle) .read();

State Based Selection

Copyrig


ll Rights Reserved.

Sample, Instance, and View State

• The samples included in the DataReader cache have associated some meta-information which, among other things, describes the status of the sample and its associated stream/instance

• The Sample State (READ, NOT_READ) allows to distinguish between new samples and samples that have already been read

• The View State (NEW, NOT_NEW) allows to distinguish a new instance from an existing one

• The Instance State (ALIVE, NOT_ALIVE_DISPOSED, NOT_ALIVE_NO_WRITERS) allows to track the life-cycle transitions of the instance to which a sample belongs

Copyrig


ll Rights Reserved.

!

!

State Selector in Action

// == ISO C++ DDS API == !// Read only new samples auto data = dr.read() !// Read any samples from live instances auto data = dr.select() .state(DataState::any_data()) .read();

Copyrig


ll Rights Reserved.

!

!

State Selector in Action

// == Java 5 DDS API == !// Read only new samples auto data = dr.read() !// Read any samples from live instances auto data = dr.select() .dataState(sub.createDataState().withAnySampleState()) .read();

Copyrig


ll Rights Reserved.

!

!

// == ISO C++ DDS API == !auto data = dr.select() .content(query) .state(data_state) .instance(handle) .read();

Putting all Together

• Selectors can be composed in a flexible and expressive manner

Application / DDS Interaction Models

Copyrig


ll Rights Reserved.

Interaction ModelsPolling

• The application proactively polls for data availability as well as special events, such as a deadline being missed, etc. Notice that all DDS API calls, exclusion made for wait operations, are non-blocking

Synchronous Notification

• The application synchronously waits for some conditions to be verified, e.g., data availability, instance lifecycle change, etc.

Asynchronous Notification

• The application registers the interest to be asynchronously notified when specific condition are satisfied, e.g. data available, a publication matched, etc.

Copyrig


ll Rights Reserved.

Synchronous Notifications

• DDS provides a mechanism known as WaitSet to synchronously wait for a condition

• Condition can predicate on:

• communication statuses

• data availability

• data availability with specific content

• user-triggered conditions

Copyrig


ll Rights Reserved.

WaitSet// == Java 5 DDS API == !// Create the waitset WaitSet ws = runtime.createWaitSet(); !Subscriber.DataState ds = sub.createDataState(); !// Create the condition Condition c = dr.createReadCondition( ds.withAnyViewState() .with(InstanceState.ALIVE) .with(SampleState.NOT_READ)); !// Attach the condition ws.attachCondition(c); !// Wait for the condition ws.wait();

!

!

Copyrig


ll Rights Reserved.

Asynchronous Notifications

• DDS provides a mechanism known as Listeners for asynchronous notification of a given condition

• Listener interest can predicate on:

• communication statuses

• data availability

Copyrig


ll Rights Reserved.

!

!

Listener Declaration// == ISO C++ DDS API == !class ShapeListener : public dds::sub::NoOpDataReaderListener<ShapeType> { public: ShapeListener() {} ! virtual void on_data_available(dds::sub::DataReader<ShapeType>& dr) { auto samples = dr.read(); std::for_each(samples.begin(), samples.end(), [](const dds::sub::Sample<ShapeType>& sample) { if (sample.info().valid()) // Check if sample contains valid data std::cout << sample.data() << std::endl; }); } ! virtual void on_liveliness_changed(dds::sub::DataReader<ShapeType>& the_reader, const dds::core::status::LivelinessChangedStatus& status) { std::cout << ">> Liveliness Changed! " << std::endl; } };

Copyrig


ll Rights Reserved.

Listener Registration

!

!

// == ISO C++ DDS API == !auto l = new ShapeListener(); !// Create a “nothing” status mask StatusMask mask = StatusMask::none(); !// Add the statuses we are interested in. mask << StatusMask::data_available() << StatusMask::liveliness_changed() << StatusMask::liveliness_lost(); !// Register the listener with the associated mask dr.listener(l, mask);

Common Bootstrap Questions

Copyrig


ll Rights Reserved.

Can I use C++11?

• OpenSplice default configuration does not require C++11

• If you want to use C++11 in your DDS applications then you need to enable support for it. Check:

• http://dev.opensplice.org/releases/downloads/docs/isocpp/html/isocpp_customlibs.html

http://dev.opensplice.org/releases/downloads/docs/isocpp/html/isocpp_customlibs.html

Copyrig


ll Rights Reserved.

Which DDS Implementation

• If you are planning to use C/C++ or C# you may consider OpenSplice DDS

• If you are developing Java Applications or are interested in targeting Android then you may consider OpenSplice Mobile

Copyrig


ll Rights Reserved.

Where can I find the Java 5 API?

• The Java 5 API is currently implemented only as part of OpenSplice Mobile

• OpenSplice Mobile is a pure Java implementation of DDS

Copyrig


ll Rights Reserved.

Can I “Script” DDS Applications?Assuming you deploy on the JVM then you have two options

• Use the DDS API from JavaScript (or any other script language supported by the JVM) and execute the script through the javax.script.ScriptEngine

• Use the DDS Java 5 API through Scala, and then run your scala program as a script

• The Molière Scala API for DDS and run the resulting program as a script

http://docs.oracle.com/javase/7/docs/api/javax/script/package-summary.html

https://github.com/nuvo-io/moliere

Copyrig


ll Rights Reserved.

How can I Debug my DDS App?

OpenSplice DDS

• Use the Tuner to inspect entities and QoS

!

OpenSplice Mobile

• Use OpenSplice Monitor to inspect entities and QoS

Copyrig


ll Rights Reserved.

OpenSplice Monitor

Part II Summary & Concluding Remarks

Copyrig


ll Rights Reserved.

Summary & Remarks

• DDS provides an elegant Data Space abstraction that allows application to share data ubiquitously and efficiently -- the Data Space implementation is fully distributed

• DDS a rich set of QoS to control the key non-functional properties of data

• DDS provides idiomatic API which are highly expressive and simple to use

¥@prismtech

¥@acorsaro

¥youtube.com/opensplicetube ¥slideshare.net/angelo.corsaro

¥ [email protected] ¥[email protected]

¥opensplice.com ¥forums.opensplice.org

¥opensplice.org ¥[email protected]

:: Connect with Us ::

http://twitter.com/%2523!/prismtech

http://twitter.com/%2523!/prismtech

http://twitter.com/%2523!/acorsaro

http://www.youtube.com/opensplicetube

http://www.youtube.com/opensplicetube

http://www.slideshare.net/angelo.corsaro



mailto:[email protected]


http://www.opensplice.com

http://www.opensplice.com

http://forums.opensplice.org/

http://www.opensplice.org

http://www.opensplice.org

