Message Passing and thePubSub Pattern

November 25, 2017

Joe DeBruycker, Rohan Khante

Montana State University

Message Passing

Why is it important?

• All computing involves communication - between processes,machines, users, nodes, etc.

◦ Local: In-process, interprocess (shared memory)◦ Network: Unicast, multicast (protocols)

• Many of our interesting problems involve how to consistentlytransfer information

• 1-to-1 isn’t too hard, but it gets a lot more interesting N-to-M

Message

• An abstraction for any piece of information being transportedfrom one place to another

• Requires a Sender and a Receiver

PubSub DeBruycker-Khante November 25, 2017 2 / 17

Message Passing

PubSub DeBruycker-Khante November 25, 2017 3 / 17

Message Passing

Messages

• Modular - Sender and Receiver are black boxes with acommon defined interface

1 Ease of use/understanding2 Plug-and-play3 Helps debugging

Connectivity

• Spectrum from 1:1 to M:N

• Client-server to Peer-to-peer

PubSub DeBruycker-Khante November 25, 2017 4 / 17

Message Passing

Message Patterns

• Messaging protocols define rules for encoding and decodingmessages

◦ Examples: HTTP, TCP, UDP, WebSub

• Messaging patterns describe how sender and receiverconnect and communicate

◦ Examples: Request-reply, fan-out, task distribution (workqueues, competing consumers),pub-sub

PubSub DeBruycker-Khante November 25, 2017 5 / 17

Publish-Subscribe

Pattern

• One-to-many (or many-to-many)

• One-way communication from publisher to subscriber

PubSub DeBruycker-Khante November 25, 2017 6 / 17

Publish-Subscribe

Applications

• Error logging

• News feeds

• Change APIs

Topologies

• Message Queues - middleware brokers

• Direct connections (i.e. sockets)

PubSub DeBruycker-Khante November 25, 2017 7 / 17

Publish-Subscribe Message Filtering Types

Topic Based

• Subs get all messages published to the channel they sub to

• Publisher is responsible for posting to channels

Content Based• Subs only get those messages which match the filters they set

• Subscriber is responsible for filtering

Hybrid

• Publishers post to topic while subs have a content basedsystem

PubSub DeBruycker-Khante November 25, 2017 8 / 17

Publish-Subscribe

Pros

• Non-blocking (asynchronous)

• Decoupled architecture

• Scalable...to a point

Cons• One way communication

• No message delivery guarantees without extra overhead

• Very large networks increase frequency of instabilities◦ Load-surges: bursty traffic◦ Slowdowns: too many publishers, subscribers can’t keep up

PubSub DeBruycker-Khante November 25, 2017 9 / 17

Database Implementations

Redis• subscribe < key >

• publish < key > < message >

PostgresQL

• Called Asynchronous Notifications

• LISTEN, UNLISTEN

• NOTIFY

Change APIs

• Alternative to polling to get the changes

• CouchDB, RethinkDB

PubSub DeBruycker-Khante November 25, 2017 10 / 17

Cloud Pub-sub Services

Google Cloud Pub/Sub

• Topic based model. Sub to a channel

• Highly scalable with easy addition of servers

• Firebase API

Amazon Simple Notification Service

• Notifications are pushed to clients

• Cloud based middleware

• Added features for mobile apps

PubSub DeBruycker-Khante November 25, 2017 11 / 17

WebSub (AKA PubSubHubbub)

Pub-Sub Protocol• Open source protocol for distributed pub-sub via HTTP

• Goal is real-time change notifications without polling

• Consumers request content from a server, and if it referencesa ”hub” they can subscribe to receive updates as long as theyrun a server

• Supported types - Any HTTP supported type, e.g. text,audio, video, HTML

• Notable ”pushers” - CNN, FOX, Myspace.com, WordPress

PubSub DeBruycker-Khante November 25, 2017 12 / 17

ZeroMQ

Overview• Asynchronous messaging library

• Bindings for multiple languages

• Primary protocol is TCP, also supports various others

Psuedo-sockets• Binding vs connecting to a port

• Socket context - 0MQ’s magic

• No middleware

• Functionality baked into ZMQ Socket Types

PubSub DeBruycker-Khante November 25, 2017 13 / 17

ZeroMQ - PubSub

PubSub DeBruycker-Khante November 25, 2017 14 / 17

0MQ - Server Code

import zmq

from random import randrange

context = zmq.Context()

socket = context.socket(zmq.PUB)

socket.bind("tcp://*:5556")

while True:

zipcode = randrange(1, 100000)

temperature = randrange(-80, 135)

relhumidity = randrange(10, 60)

socket.send_string("%i %i %i" % (zipcode, temperature, relhumidity))

PubSub DeBruycker-Khante November 25, 2017 15 / 17

0MQ - Client Code

import sys

import zmq

# Socket to talk to server

context = zmq.Context()

socket = context.socket(zmq.SUB)

print("Collecting updates from weather server")

socket.connect("tcp://localhost:5556")

# Subscribe to zipcode, default is NYC, 10001

zip_filter = sys.argv[1] if len(sys.argv) > 1 else "10001"

# Python 2 - ascii bytes to unicode str

if isinstance(zip_filter, bytes):

zip_filter = zip_filter.decode(’ascii’)

socket.setsockopt_string(zmq.SUBSCRIBE, zip_filter)

# Process 5 updates

total_temp = 0

for update_nbr in range(5):

string = socket.recv_string()

zipcode, temperature, relhumidity = string.split()

total_temp += int(temperature)

print("Average temperature for zipcode ’%s’ was %dF" % (

zip_filter, total_temp / (update_nbr+1))

)

PubSub DeBruycker-Khante November 25, 2017 16 / 17

ZeroMQ

Exercises• How many messages does the server send before the client

receives 5 messages from the same topic (in this case, ZipCode)?

◦ Hint: zip filter = ”” subscribes to all messages

• How long does it take on your computer before the subscriberreceives all 5 messages?

◦ Hint: import time

• What order do messages arrive on the client when it issubscribed to multiple publishers?

◦ Hint: A socket can make multiple connect calls

PubSub DeBruycker-Khante November 25, 2017 17 / 17