Communication Patterns Using Data-Centric Publish/Subscribe

Sumant Tambe, Ph.D.

Senior Software Research Engineer

Real-Time Innovations, Inc.

www.rti.com

http://www.rti.com/

Agenda

• Communication Models

• Data Distribution Service Standard

• Data-Centricity 101 – Demo

• Communication Patterns – Request/Reply

– Guaranteed Delivery

10/11/2012 2

Middleware Communication Models

10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 3

RPC Vs. Pub-Sub/Messaging/Data-Distribution


• RPC (WebServices, CORBA, DCOM) offer a remote method abstraction

– Familiar OO programming model

– Results in a tightly-coupled system • Forces synchronous invocations

• Imposes global object model

• Limited QoS(appearance of local method call)

• Lack robustness: cascading points of failure

– Typically built on top of TCP: • Impacts scalability and time-determinism

– Best-suited to smaller, closely-coupled systems

• Pub-Sub (Messaging Data-Distribution) offer a queue-based and/or replicated-data model

– Subsystems are decoupled in time, space, and synchronization • Contracts established by verifying QoS compatibility

– Supports a variety of transports including multicast UDP

– Better suited for high-performance and real-time

Queue Vs. Pub/Sub


• Queue – Message sent to Queue

– Multiple readers can read from the queue

– Each message is delivered to ONLY one reader

• Readers “affect each other”

– Apps:

• Job Scheduling, Load Balancing, Collaboration

• Pub/Sub – Message sent to Topic

– Multiple readers can subscribe to Topic with or without filters

– Each message delivered to ALL subscribers that pass filter

• Readers are decoupled

– Apps:

• Notifications, Information Distribution

Pub/Sub Vs. Data-Distribution


• Pub-Sub – Only messages no concept of data

– Each message is interpreted without context

– Messages must be delivered FIFO or according to some “priority” attribute

– No Caching of data

– Simple QoS: filters, durability, lifespan

• Data-Distribution – Messages represent update to data-objects

– Data-Objects identify by a key

– Middleware maintains state of each object

– Objects are cached. Applications can read at leisure

– Smart QoS

• Ownership, History (per key), Deadline

– Subsumes Pub-Sub

Data Bus

DDS: Standards-based Integration Infrastructure for Critical Applications

© 2009 Real-Time Innovations, Inc.

Streaming Data

Sensors Events

Real-Time Applications

Enterprise Applications

Actuators

Systems that interact with the Real World

• Must adapt to changing environment • Cannot stop processing the information • Live within world-imposed timing Beyond traditional interpretation of real-time


Real-Time Data Distribution


RPC over DDS

2013

DDS Security

2012

Family of Specifications

© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL 10

DDS Implementation

Network / TCP / UDP / IP

App

DDS Implementation

App

DDS Implementation

DDS Spec

2004

DDS Interoperablity

2006

UML Profile for DDS

2008

DDS for Lw CCM

2009

DDS X-Types

2010 2010

DDS-STD-C++ DDS-JAVA5

App

© 2009 Real-Time Innovations, Inc. 11

Broad Adoption

• Vendor independent – API for portability – Wire protocol for interoperability

• Multiple implementations – 10 of API – 8 support RTPS

• Heterogeneous – C, C++, Java, .NET (C#, C++/CLI) – Linux, Windows, VxWorks, other

embedded & realtime • Loosely coupled

DDS Real-Time Publish-Subscribe Wire Protocol (RTPS)

Middleware

DDS API

Cross-vendor portability

Cross-vendor interoperability

DDS adopted by key programs

• European Air Traffic Control – DDS used to interoperate ATC centers

• UK Generic Vehicle Architecture – Mandates DDS for vehicle comm.

– Mandates DDS-RTPS for interop.

• DISR (DoD Information Technology Standards Registry) – Mandates DDS for Pub-Sub API and Interop.

• US Navy Open Architecture – Mandates DDS for Pub-Sub

• SPAWAR NESI – Mandates DDS for Pub-Sub SOA


Evolution of DataBus Data-centricity basics

Everyday Example: Schedule Meeting via Emails

Alternative Process #1 (message-centric):

1. Email: “Meeting Monday at 10:00.”

2. Email: “Here’s dial-in info for meeting…”

3. Email: “Meeting moved to Tuesday”

4. You: “Where do I have to be? When?”

5. You: (sifting through email messages…)

14

Everyday Example: Schedule Meeting Using a Calendar

Alternative Process #2:

1. Calendar: (add meeting Monday at 10:00)

2. Calendar: (add dial-in info)

3. Calendar: (move meeting to Tuesday)

4. You: “Where do I have to be? When?”

5. You: (check calendar. Contains consolidated-state)

15

The difference is state! The infrastructure consolidates changes and maintains it

What’s the Difference? State.

• Objects have identity and attributes – The meeting will run 1:00–

2:00 in the conference room. – My friend’s phone number is

555-1234 his email is… – The car is blue and is

traveling north from Sunnyvale at 65 mph.

• …whether they exist in the real world, in the computer, or both

• …whether or not we observe or acknowledge them

“State” (“data”) is a snapshot of those attributes and characteristics.

If the infrastructure maintains the state, the application does not need to re-construct it…

Why is it better to have the (data-centric) middleware manage the state?

• Reconstructing the state of an object is hard – Must infer based on all previous messages – Maintaining all these messages is expensive – Each app makes these inferences duplicate effort

• Reconstructing state is not robust – Many copies of state may be different bugs vs. Uniform operations on state fewer bugs – Any missing change compromises integrity

• State awareness results in better performance – Middleware can be smart about what to send and when

• Data-type awareness simplifies programming – Middleware supports direct definition and instantiation of the data-

model


DDS Communication Model

Demo: Publish-Subscribe


DDS Real-Time Quality of Service (QoS) Attributes

QoS Policy

DURABILITY

HISTORY

READER DATA LIFECYCLE

WRITER DATA LIFECYCLE

LIFESPAN

ENTITY FACTORY

RESOURCE LIMITS

RELIABILITY

TIME BASED FILTER

DEADLINE

CONTENT FILTERS

Vo

lati

lity

Use

r Qo

S

De

live

ry

Pre

sen

tation

R

ed

un

dan

cy

Infr

astr

uct

ure

Transp

ort

QoS Policy

USER DATA

TOPIC DATA

GROUP DATA

PARTITION

PRESENTATION

DESTINATION ORDER

OWNERSHIP

OWNERSHIP STRENGTH

LIVELINESS

LATENCY BUDGET

TRANSPORT PRIORITY


Communication Patterns Beyond DDS

Why Patterns

• From app programming POV

– Patterns solve specific problems nicely

• From platform or systems POV

– Handling each pattern in independent ad hoc ways results in brittle, poorly performing systems

• Solution: Build platform in terms of fundamental patterns

– Derive high-level patterns from basic patterns; expose those to apps

© 2012 RTI 22

Communication Pattern Request/Reply

Introduction to Request/Reply


• Publish/Subscribe (Pub/Sub) – Fundamental pattern in RTI Connext – Pub/Sub shines where there is

• Push model • Unidirectional stream of data • One-to-many (multicast)

• Request/Reply – Client-server model built on top of pub/sub – Request/Reply shines where there is

• Pull model • Bidirectional communication • A need for remote code execution

– E.g., Get/set configuration, send commands

Request-Reply

© 2011 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL

Requestor Replier

Request (Foo)

Request Topic

Reply Topic Reply

(Bar)

Correlation


Replier Requests

Replies

3

3

2 1

2 1

Message ID

Correlation ID

1

1

Requestor

Correlation


How Does Correlation Work


Requester App Replier App

Data Writer (VGUID1)

Data Reader

Data Reader

Data Writer (VGUID2)

VGUID1 VSN1 Foo

VGUID2 VSN2 Bar VGUID1 VSN1

sample_id1

sample_id-id2 related_sample_id1

Content-based Filter “@related_sample_id.VGUID == VGUID1”

Read

Bar

Index Condition “@related_sample_id.VSN == VSN1”

Write Foo

VSN1

Foo

Bar

Processing

VGUID1

Filtering Implementation details

Part of SampleInfo

Single-Request Multiple-Reply


Requestor Replier

Replies

3 2 1

Sequence ID

Request

Multiple Repliers


Requester

Replier A

Replier C

Replier B

Request

Reply

Replier “Hello, World!” (Server-side)


using namespace connext; DDS::DomainParticipantFactory factory = DDS::DomainParticipantFactory::get_instance(); DDS::DomainParticipant * participant = factory->create_participant(domain_id, DDS::PARTICIPANT_QOS_DEFAULT, NULL, DDS::STATUS_MASK_NONE); std::auto_otr<Replier<Foo, Bar> > replier (new Replier<Foo, Bar>(participant, "TestService")); Sample<Foo> request; // Receive one request bool received = replier->receive_request(request, MAX_WAIT); if (!received) { std::cout << "Requests not received" << std::endl; return false; } // A WriteSample automatically initializes and finalizes // the data using the TypeSupport (in this case BarTypeSupport) WriteSample<Bar> reply; if (request.info().valid_data) { sprintf(reply.data().message, "Reply for %s", request.data().message); // Send a reply for that request replier->send_reply(reply, request.identity()); // Note: a replier can send more than one reply for the same request }

Requester “Hello, World!” (Client-side)


using namespace connext;

// Create a DomainParticipant

DDS::DomainParticipant * participant = ...;

// Create a Requester

std::auto_ptr<Requester<Foo, Bar> > requester (new Requester<Foo, Bar>(participant, "TestService")); // Send request

WriteSample<Foo> request;

strcpy(request.data().message, "A Request");

requester->send_request(request); if (requester->wait_for_replies(2, MAX_WAIT, request.identity())) { LoanedSamples<Bar> replies = requester->take_replies(request.identity());

// If there are no replies, the container is empty

for(LoanedSamples<Bar>::const_iterator it = replies.begin();

it != replies.end();

++it) {

if (it->info().valid_data) {

std::cout << "Received reply: " << it->data().message << std::endl;

}

} }

Implementing Request/Reply Pattern


Define request and reply types Manual Manual

Steps With Vanilla DDS API With BIGPINE API

Create Participant Manual Manual

Pollute application data model with a correlation - id

Manual Not necessary

Create publisher/subscriber Manual Automatic

Create datareader / datawriter Manual Automatic

Register request/reply types Manual Automatic

Create request/reply topics Manual Automatic

Setup CFT to filter unwanted replies Manual Automatic

Correlation of request/reply samples Manual Semi - Automatic

Memory management of samples Manual Automatic

Lifecycle m anagement of entities Manual Automatic

Language friendly (idiomatic API) Partially Yes

Scalable Depends on impl Yes

Salient Features of Connext Request/Reply


• New API, built on top of DDS

• Data-centric

• Scalable

• Simple to use, abstracts many DDS details

• Idiomatic, language-friendly API (C++, Java, and C#)

• It’s not RPC, though can be the foundation for an RPC solution

Scalable Request-Reply


Replier

Requester 1

Requester N

Requester 2

Replies

Reply 1

Reply 2

Reply N

…

Thousands of clients!

Combining Patterns


Requestor Replier

Request Request

Topic

Reply Topic Reply

Subscriber

Subscriber

Wire Tap

Communication Pattern Guaranteed Delivery

The Guaranteed Delivery Problem

© 2012 RTI • COMPANY CONFIDENTIAL

• Delivery of samples from a Producer to the Intended Consumers such that the delivery is robust to multiple kind of failures

• Guaranteed delivery does not imply complete certainty Continuum of Delivery Guarantees

The Guaranteed Delivery Scenario


Producer ‘P’ (DataWriter)

Consumer ‘C’ (DataReader)

Sample ‘S’

• Producer ‘P’ publish a Sample ‘S’ that should be received by Consumer ‘C’

Producer process failure

(the producer process crashes after the sample has been written)

Network and transport failures (packets are lost due to temporary

network failures or transport/middleware buffer overflows)

Late joiner consumer (the consumer is not running when

the sample is written)

Consumer process failure (the process crashes after the sample is

received by the middleware and before it is processed by the application)

Guaranteed Delivery


Publisher

Message

Disk

Message

Message

Subscriber

Durable Subscriber

Message

App-level ack

DDS Durability QoS Primer


DDS Durability controls data availability with respect to late joining datareaders

• VOLATILE

– No need to keep data samples for late joiners

• TRANSIENT_LOCAL

– Data instances available for late joiners

– Data does not survive beyond datawriter

• TRANSIENT

– Data is not tied to the lifecycle of the datawriter.

– A separate service (persistence service) keep data in memory

• PERSISTENT

– Data is kept on permanent storage, so that they can outlive a system session

• DataWriter can be configured with:

– a list of required subscriptions • Required subscriptions are named subscriptions identified by a

role name

• DataWriter must store a sample until both:

– Acknowledged by all active reliable readers

– Acknowledged by all required subscriptions

• DataReader identifies itself:

– As serving a required subscription

– Uniquely via a virtual GUID

Beyond DDS: Required Subscriptions

• Concept

– It is a required subscription with durability >= TRANSIENT

– Features that allows to keep data until received by durable subscriptions for which readers may or may not exist at any given time

– Data is not lost even if the DataWriter crashes

• Benefits and Use Cases

• In combination with other features durable subscriptions provides guaranteed messaging

• Requires persistence service – To persist data

– To persist existence of durable subscriptions and their stats

Beyond DDS: Durable Subscriptions

• Concept: – Multiple DataWriters can relay samples from a common

stream (data source). The DataReaders reconstruct the original stream and deliver to the application the complete set of samples

• Benefits & Use cases: – High availability and fault tolerance

• Multiple DataWriters publishing the same samples

– Load balancing and data partition • Multiple DataWriters publishing different sets of samples from the

same source

– Scalability • Partition across different machines and cores

Beyond DDS: Collaborative DataWriters

Delivery Models: Best-Effort to Guaranteed!


Use Case Features in RTI Connext Messaging 5.0.0

Periodic data from live producers Best effort reliability

Event data from live producers Reliable reliability

Last value cache from live producers (entire state of the datawriter cache)

Transient local durability + Reliable reliability

Durable last value cache (datawriter may come and go)

Persistent durability + Reliable reliability + Persistence Service

Guaranteed messaging from live producers Transient local durability + Reliable reliability + Required subscriptions + Application level ACK

Durable Guaranteed messaging Persistent durability + Reliable reliability + Collaborative DataWriters + Persistence Service + Durable subscriptions + Application level ACK

References


• Real-Time Innovations, Inc. – www.rti.com

• RTI Connext DDS – www.rti.com/products/dds

• RTI Connext Request-Reply API – Just Google!

• OMG DDS Specifications – www.omg.org/technology/documents/dds_spec_catalog.htm

http://www.rti.com/

http://www.rti.com/products/dds

http://www.google.com/search?q=RTI Connext Request-Reply API

http://www.omg.org/technology/documents/dds_spec_catalog.htm

Thank you
