Upload
sumant-tambe
View
457
Download
5
Tags:
Embed Size (px)
DESCRIPTION
In this presentation you will see why Reactive Extensions (Rx) is a powerful technology for asynchronous stream processing. RTI Data Distribution Service (DDS) will be used as the source of data and as a communication channel for asynchronous data streams. On top of DDS, we'll use Rx to subscribe, observe, project, filter, aggregate, merge, zip, and correlate one or more data streams (Observables). The live demo will be very visual as bouncing shapes of different colors will be transformed in front of you using C# lambdas, Rx.NET, and Visual Studio. You will also learn about the new Rx4DDS.NET library that integrates RTI DDS with Rx.NET. Rx and DDS are a great match because both are reactive. Rx is based on the subject-observer pattern, which is quite analogous to the publish-subscribe pattern of DDS. When used together they support distributed dataflows seamlessly. If time permits, we will touch upon advanced Rx concepts such as stream of streams (IGroupedObservable) and how it captures DDS "keyed topics". The DDS applications using Rx4DDS.NET dramatically simplify concurrency to the extent that it can be simply configured.
Citation preview
Your systems. Working as one.
Reactive Stream Processing
using DDS and Rx
www.rti.com
Sumant Tambe, Ph.D.
Senior Software Research Engineer and Microsoft MVP
Real-Time Innovations, Inc.
@sutambe
Oct. 11, 2014
Outline
• Reactive Systems
• Stream Processing
• Overview of Reactive Extensions
• Overview of DDS
• Streaming Shapes Demo in C#
10/10/2014 2Real-Time Innovations, Inc.
Systems Everyone Wants to Build
• Event-Driven: Modular, pipelined, asynchronous
• Elasic: Scales easily up/down with load and cpu cores.
• Resilient: fault-tolerant
• Responsive: Reacts to events at the speed of environment
10/10/2014 Real-Time Innovations, Inc. 3
Networking Middleware
App
Stream Processing
• Stream Processing is the term used to describe an architectural style that operate on a continuous sequence of data.
10/10/2014 Real-Time Innovations, Inc. 4
Shape of an application
o/p
Where Once CombineLatest Select Scan Merge Raw Data
i/p
*nix command line (pipes and filter)
$ ls -1 | grep “search” | grep “research”
research
$
10/10/2014 Real-Time Innovations, Inc. 6
$ cat - | grep “Real”
R
Real
Real
[Ctrl+C]
$
Reactive Extensions• The Reactive Extensions (Rx) is a
library for composing asynchronous and event-based programs using observable sequences
• Rx = Observables + Composition + Schedulers
• Streams are first-class
• Filter, project, aggregate, compose and perform time-based operations on multiple streams
• Uses Functional Programming Style
• Rx.NET, RxJava, RxJS, RxCpp, RxRuby, RxPyton, and more…
10/10/2014 Real-Time Innovations, Inc. 7
10/10/2014 Real-Time Innovations, Inc. 8
Data Distribution Service (DDS)
A Reactive Middleware
DDS: Standards-based Integration Infrastructure for Critical Applications
© 2009 Real-Time Innovations, Inc.
Streaming
DataSensors Events
Real-Time
Applications
Enterprise
ApplicationsActuators
Systems that interact with the Real World
• Must adapt to changing environment• Cannot stop processing the information• Live within world-imposed timing
Beyond traditional interpretation of real-time
© 2010 Real-Time Innovations, Inc.
Real-Time Data Distribution
10/10/2014 © 2012 RTI • COMPANY 11
RPC over
DDS
2014
DDS
Security
2012
Family of Specifications
© 2009 Real-Time Innovations, Inc. COMPANY 12
DDSImplementation
Network / TCP / UDP / IP
App
DDSImplementation
App
DDSImplementation
DDS Spec
2004
DDS
Interoperablity
2006
UML Profile
for DDS
2008
DDS for
Lw CCM
2009
DDS
X-Types
2010 2010
DDS-STD-C++
DDS-JAVA5
App
Evolution of DataBus
Data-centricity basics
Everyday Example: Schedule Meeting via Emails
Alternative Process #1 (message-centric):
1. Email: “Meeting Monday at 10:00.”
2. Email: “Here’s dial-in info for meeting…”
3. Email: “Meeting moved to Tuesday”
4. You: “Where do I have to be? When?”
5. You: (sifting through email messages…)
14
Everyday Example: Schedule Meeting Using a Calendar
Alternative Process #2:
1. Calendar: (add meeting Monday at 10:00)
2. Calendar: (add dial-in info)
3. Calendar: (move meeting to Tuesday)
4. You: “Where do I have to be? When?”
5. You: (check calendar. Contains consolidated-state)
15
The difference is state!
The infrastructure consolidates changes and maintains it
10/10/2014 © 2012 RTI • COMPANY 16
DDS Communication Model
DDS for Distribution, Rx for Processing
10/10/2014 Real-Time Innovations, Inc. 17
DR
DR
DR
Observable Observer
DW
DW
Processing
Rx4DDS.NET = DDS + Rx
10/10/2014 © 2012 RTI • COMPANY 18
• DDS wrapper for Rx.NET
• In C#
• Anything that produces data is an Observable
– Topics, Discovery, Statuses, statuses, etc.
DDS and Rx: A Great Match
10/10/2014 © 2012 RTI • COMPANY 19
DDS Concept Rx Concept/Type/Operator
Topic of type T An object that implements IObservable<T>, which internally creates a DataReader<T>
Communication status, Discovery event streams
IObservable<SampleLostStatus>
IObservable<SubscriptionBuiltinTopicData>
Topic of type T with key type=Key
IObservable<IGroupedObservable<Key, T>>
Detect a new instance Notify Observers about a new IGroupedObservable<Key, T> with key==instance. Invoke IObserver<IGroupedObservable<Key, T>>.OnNext()
Dispose an instance Notify Observers through IObserver<IGroupedObservable<Key,T>>.OnCompleted()
Take an instance update of type T
Notify Observers about a new value of T using Iobserver<T>.OnNext()
Read with history=N IObservable<T>.Replay(N) (Produces a new IObservable<T>)
DDS and Rx: A Great Match
10/10/2014 © 2012 RTI • COMPANY 20
DDS Concept Rx Concept/Type/Operation
Query Conditions Iobservable<T>.Where(…) OR
Iobservable<T>.GroupBy(…)
SELECT in CFT expression IObservable<T>.Select(...)
FROM in CFT expression DDSObservable.FromTopic(“Topic1”)
DDSObservable.FromKeyedTopic(“Topic2”)
WHERE in CFT expression IObservable<T>.Where(...)
ORDER BY in CFT expression IObservable<T>.OrderBy(...)
MultiTopic (INNER JOIN) IObservable<T>.Join(...)
.Where(...)
.Select(...)
Join between DDS and non-DDS data
Join, CombineLatest, Zip
Living Demo in C#
10/10/2014 Real-Time Innovations, Inc. 21
Background Code
10/12/2014 Real-Time Innovations, Inc. 22
public class ShapeType : ICopyable<ShapeType>{
public string color;public int shapesize;public int x;public int y;
public ShapeType();}
public class ShapeTypeExtended : ShapeType{
public float angle;public ShapeFillKind fillKind;
public ShapeTypeExtended();}
private DDS.TypedDataWriter<ShapeTypeExtended> triangle_writer;
Flower Demo
10/12/2014 Real-Time Innovations, Inc. 23
IDisposable flower(DDS.DomainParticipant participant){
int a = 30, b = 30, c = 10;
returnObservable
.Interval(TimeSpan.FromMilliseconds(1), Scheduler.Immediate)
.Select((long x) =>{
int angle = (int)(x % 360);return new ShapeTypeExtended{
x = (int)(120 + (a + b) * Math.Cos(angle) + b * Math.Cos((a / b - c) * angle)),y = (int)(120 + (a + b) * Math.Sin(angle) + b * Math.Sin((a / b - c) * angle)),color = "GREEN",shapesize = 5
};}).Subscribe(triangle_writer);
}
Simple Square to Triangle Transformation
10/12/2014 Real-Time Innovations, Inc. 24
IDisposable forward_verbose(DDS.DomainParticipant participant){
var rx_reader = DDSObservable.FromTopic<ShapeTypeExtended>(participant, "Square");
IDisposable disposable =rx_reader.OnDataAvailable((ShapeTypeExtended shape) =>{
DDS.InstanceHandle_t handle = DDS.InstanceHandle_t.HANDLE_NIL;triangle_writer.write(shape, ref handle);
});
return disposable;}
IDisposable forward_shortest(DDS.DomainParticipant participant){
return DDSObservable.FromTopic<ShapeTypeExtended>(participant, "Square").OnDataAvailable(triangle_writer);
}
Swap Square’s x and y and Propagate Stream Dispose Event
10/12/2014 Real-Time Innovations, Inc. 25
IDisposable swap(DDS.DomainParticipant participant){
return DDSObservable.FromTopic<ShapeTypeExtended>(participant, "Square").Select(shape => new ShapeTypeExtended{
x = shape.y,y = shape.x,color = shape.color,shapesize = shape.shapesize
}).SubscribeAndDisposeOnCompleted(triangle_writer,
new ShapeTypeExtended { color = "BLUE" });}
Aggregator (Square + Circle = Triangle!)
10/12/2014 Real-Time Innovations, Inc. 26
IDisposable aggregator(DDS.DomainParticipant participant){
var rx_square_reader =DDSObservable.FromTopic<ShapeTypeExtended>(participant, "Square");
var rx_circle_reader =DDSObservable.FromTopic<ShapeTypeExtended>(participant, "Circle");
return new CompositeDisposable(new IDisposable[] {
rx_square_reader.Subscribe(triangle_writer),rx_circle_reader.Subscribe(triangle_writer)
});
}
Square/Circle Correlator using LINQ
10/12/2014 Real-Time Innovations, Inc. 27
IDisposable selectmany_correlator(DDS.DomainParticipant participant, bool useLinq){
var rx_circle_reader =DDSObservable.FromTopic<ShapeTypeExtended>(participant,
"Circle", Scheduler.Default);
var rx_square_reader =DDSObservable.FromTopic<ShapeTypeExtended>(participant,
"Square", Scheduler.Default);
var correlator =from square in rx_square_readerfrom circle in rx_circle_reader.Take(1)where square.color == circle.colorselect new ShapeTypeExtended{
x = square.x,y = square.y,color = square.color,shapesize = circle.x
};
return correlator.Subscribe(triangle_writer);}
Thank You!
• Rx4DDS.NET– https://github.com/rticommunity/rticonnextdds-reactive
• Data Distribution Service– http://portals.omg.org/dds
• Real-Time Innovations, Inc.– www.rti.com
10/12/2014 Real-Time Innovations, Inc. 28