AMQP, Message BrokerBabu Ram Dawadi

overview

• Why MOM architecture?

• Messaging broker like RabbitMQ in brief

• RabbitMQ

• AMQP – What is it ?

Message Oriented Middleware Architecture

• Method of communication between different applications or software components between distributed systems.

• The predominant approaches

RMI, CORBA, SOAP web services

Calls remote methods with parameters …. And waits for response

• Used for

• Scalability and load balancing

• Decoupling

• Reliability

• Availability

• Asynchronous hand off

• Queuing and buffering

• Monitoring and management

AMQP 1.0 Scope

AMQP is Message Oriented Middleware (MOM)

Transfers application data units from senders to receivers – layer 7

An expectation that the message transfer is via trusted intermediaries

– An expectation that messages will be delivered unchanged

– An expectation of security

– Applications can be separated by (large amounts) of space and time

Abstract from the underlying technology

– Physical network limits should be hidden (message size, node location)

– Technology concerns should be hidden (platform, language, OS)

The intermediaries offer various delivery options, as defined by either the sender or the receiver (s)

The intermediaries provided various defined qualities of service for the sender and the receiver (s)

AMQP (Advanced Messaging Queue Protocol)

• Originated in 2003 by John O’Hara at JPMorgan Chase.

• A protocol to communicate between clients and messaging middleware servers (brokers)

• Open internet protocols like TCP, SMTP, HTTP.

• Open wire protocol standard for message-queuing communication.

• Enables conforming client applications to communicate with conforming messaging middleware services (aka message brokers)

• Defines

• Networking protocol

• Semantics of broker services

• Use cases

• Real time feeds for constantly updating information

• Encrypted assured transaction

AMQP Design Requirement

•To accommodate existing messaging API standards (for example, Sun's JMS)

•To allow complete configuration of server wiring via the protocol

•To handle messages of any size without significant limit

•To carry multiple channels across a single connection

•To allow asynchronous command pipelining

To get there …

• Queues

• Messages are consumed from queues

• Stores messages in memory or disk

• Deliver in sequence

• Exchanges

• Messages are routed to queues through exchanges through pre-arranged criteria

• Matching and routing engines, rather can do more than routing messages.

• How is the message routed ??

Messages are routed to queues by matching routing keys against binding pattern.

Routing key Binding key

AMQP Communication Model

• Provides a “Shared Queue Space” that is accessible to all interestedapplications.

• Message are sent to an Exchange

• Each message has an associated Routing Key

• Brokers forward messages to one or more Queues based on theRouting Key

• Subscriber get messages from named Queues

• Only one subscriber can get a given message from each Queue

AMQL Object Model

• Exchange – Receives messages and routes to a set of message queues

• Queue – Stores messages until they can be processed by the application(s)

• Binding – Routes messages between Exchange and Queue. Configured externally to the application.

• Default binding maps routing-key to Queue name

• Routing Key – label used by the exchange to route Content to the queues

•Content – Encapsulates application data and provides the methods to send receive,acknowledge, etc.

Messages

• Carry content (header + body)

• Content body is opaque block of binary data.

• Broker never modifies the content body.

• Have client specified properties

• Content-type, Encoding, Timestamp, App-Id, User-Id, Headers

• Delivery mode

• Non-Persistent

• Persistent

• Producers send messages to exchanges with a routing key e.g. “abc”, or ordered set of keys e.g. “abc.cloud.aws”

• Exchanges route messages to queues whose binding pattern matches the message routing key or keys.

• Consumers create queues; these buffer messages for push to consumers

• Queues are stateful, ordered, and can be persistent, transient, private, shared. Exchanges are stateless routing tables.

• Consumers tell queues to bind to named exchanges; each binding has a pattern e.g. “abc” or “*.cloud.*”

Example of pub-sub message flow

Messages

Traditional Topologies Built from Parts

Queues are used both for Persistent stores and transient buffers.

Link model unifies point-to-point and publish/subscribe

– example shows client messages being routed to various Queues

– Example mixes traditional Store & Forward and Transient Pub/Sub

Queue1link/transferClient ASession

Client BSession

Queue:“StockTicker”

Queue:“US-Payments”

Queue:“ServiceBus”

Subject REGEXP “stocks.ny.*”

PREDICATE

Subject REGEXP “stocks.uk.*”

Subject REGEXP “stocks.tk.*”

BusEvt=“Pay” and Ccy!=“USD”

BusEvt = “Unwind”

usaQ

Queue1worldQ

Queue1usPayQ

link/transfer

link/transfer

link/transfer

BusEvt=“Pay” and Ccy=“USD”

Queue1wrldPayQ

Queue1unwindQ

Well-Known Queue

In-Broker Links Work QueueSession

StockTicker worldQ

StockTicker

StockTicker

SOURCE TARGET

worldQ

usaQ

Subject REGEXP “stocks.ny.*”

PREDICATE

StockTicker

SOURCE TARGET

usaQ

StockTicker

StockTicker

StockTicker

PREDICATESOURCE TARGET

unwindQ

worldPayQ

usPayQ

AMQP Architectural Model

• Centralized Multi Broker

• Each Queue is materialized in a Broker

AMQP1.0 Typical Usage Patterns

ClientProducer

AMQP BrokerAMQP Broker

ClientConsumer

Entry 1

Entry 2

Entry 3

Session

LinkLink

Session

LinkLink

Queue (source)

-Persistent

Head

Tail

Highlights:• Only “Source” queue is required and can be

read directly by consumer over Link (i.e. dedicated consumer Worker queue and bridging between Source and Worker unnecessary).

Point-to-point Queue Delivery

ClientProducer

AMQP BrokerAMQP Broker

Entry 1

Entry 2

Entry 3

Session

LinkLink

Queue (Source)

-Persistent

Head

Tail

Entry 1

Entry 2

Head

LinkLinkTail

Queue (worker)

-Persistent

Abstracted Point-to-point Queue

Highlights:• One Queue performs the role of holding the

“Well Known” name for the outside world.• All messages are automatically forward on

to the real worker queue.• Allows internal topology to change without

the outside world seeing (this PO Box)

ClientProducer

AMQP BrokerAMQP Broker

ClientConsumer

Entry 1

Entry 2

Entry 3

Session

LinkLinkSession

LinkLink

Queue (source)

-Persistent

1 Head or 2 ?

Tail

ClientConsumer

Session

LinkLink

Load-Balanced Point-to-point Queue Delivery

ClientPublisher

AMQP BrokerAMQP Broker

ClientSubscriber

Entry 1

Entry 2

Entry 3

Session

LinkLink

Session

LinkLink

Queue (Source)

-Non-persistent

Head

Tail

ClientSubscriber

Session

LinkLink

ClientSubscriber

Session

LinkLink

Head

Head

Dynamic (non-persistent) Pub/Sub Delivery

Highlights:• Messages are “garbage collected” in an implementation

specific manner after delivery.• AMQP makes some guarantees about how long

messages are valid for.

ClientPublisher

AMQP BrokerAMQP Broker

Entry 1

Entry 2

Entry 3

Session

LinkLink

Queue (Source)

- persistent

Head

Tail

ClientSubscriber

Session

LinkLink

ClientSubscriber

Session

LinkLink

Head

Entry 1

Entry 2 Head

HeadEntry 1

Tail

LinkLink

LinkLink

Queue (Worker)

- persistent

Queue (Worker)

- persistent

Durable (persistent) Pub/Sub Delivery