Upload
datakurre
View
2.341
Download
3
Tags:
Embed Size (px)
Citation preview
Speakers' backgroundUniversity of Jyväskyläthe largest Plone user in Finland● 83 Plone sites● 2.5 TB data served through Plone● 700 content editors
Why message queues?We needed a reliable, effective and scalable solution for:
● running asynchronous tasks● being a common integration point
between services (decoupling).
Why message queues?
”None of the existing ZODB-based mechanism really scales. When you are really in need for a real and working queuing mechanism one would use some external queuing solution (e.g. something AMQP compatible like RabbitMQ).”
– Andreas Jung
Advanced Message Queue Protocol (AMQP)
Think of it as an highly advancedpost office which can delivermessage to multiple recipientsbased on address.
AMQP jargon 1/2
● Message consists of label and payload.
● Producer is a program which sends messages to exchange.
● Consumer is a program which mostly waits to receive messages.
AMQP jargon 2/2
● Exchange receives the messages from producers and pushes them to queues.
● Bindings are how the messages get routed from the exchange to a queue.
● Queue is a buffer that stores messages.
Message broker
Basic messaging
Producer
Consumer
Exchange(direct) Queue
Work queues
Consumer
M1M2
M1 M1M2
M1
M2
M2
Basic messaging
Producer
Message broker
Exchange(fanout)
Queue
Publish/Subscribe queues
Queue
Consumer
Consumer
M1M1
M1
M1
M1
M1
Basic messaging
Client/producer
Message broker
RPC queue
RPC implementation
Exchange
Server
M1 M1Exchange M1
Reply_to queue
R1R1
M1
R1
R1
RabbitMQ● Implements AMQP (Advanced Message
Queue Protocol) open standard
● Written in Erlang○ made for messaging○ easy to cluster○ fast (enough)
● Industry tested and reliable
Example withcollective.zamqp
Site A
Message broker
Exchange(fanout)
Queue
Publish and subscribe announcements
Queue
Site B
Site C
M1M1
M1
M1
M1
M1
Produce messages withcollective.zamqpfrom zope.component import getUtility
from collective.zamqp.interfaces import IProducer
producer = getUtility( IProducer, name="announcer")
producer.register() # for transaction
producer.publish("Hello World!")
Handle messages withcollective.zamqpfrom five import grok
from collective.zamqp.interfaces import IMessageArrivedEvent
@grok.subscribe(IAnnouncement,
IMessageArrivedEvent)
def handleMessage(message, event):
logger.info(message.body)
message.ack()
Overview ofcollective.zamqp
Producers
Message broker
AMQP Broker Connections
Message handlers(as event subscribers)
Consuming Servers
IMessageArrivedEvent
produce messages
consume messages
Consumers
stamp messages
Connections incollective.zamqpbuildout.cfg:
[instance]
zope-conf-additional =
%import collective.zamqp
<amqp-broker-connection>
connection_id example
...
</amqp-broker-connection>
Producers incollective.zamqpclass Announcer(Producer):
grok.name("announcements")
connection_id = "example"
exchange = "announcements"
exchange_type = "fanout"
serializer = "text/plain"
durable = False
Consuming servers incollective.zamqpbuildout.cfg:
zope-conf-additional =
%import collective.zamqp
....
<amqp-consuming-server>
connection_id example
site_id Plone
</amqp-consuming-server>
Consumers incollective.zamqpclass Announcements(Consumer):
grok.name("announcements")
connection_id = "example"
exchange = "announcements"
exchange_type = "fanout"
queue = "" # anonymous temp. queue
durable = False
marker = IAnnouncement
Message handlers incollective.zamqp
@grok.subscribe(IAnnouncement, IMessageArrivedEvent)
def handleMessage(message, event):
logger.info(message.body)
message.ack()
Overview ofcollective.zamqp
Producers
Message broker
AMQP Broker Connections
Message handlers(as event subscribers)
Consuming Servers
IMessageArrivedEvent
produce messages
consume messages
Consumers
stamp messages
Real world exampleMoniviestin video publishing service● Plone 4.1 site● Distributed encoder servers (48 CPUs
combined)● First version used XML-RPC based
communication - lots of error handling code.● New version uses AMQP-messaging. ● Very fast and scalable. Recovers from
network outages.
Thank youc.zamqp and examples are available at:● https://github.com/datakurre/collective.zamqp● https://github.com/datakurre/collective.zamqpdemos● https://github.com/datakurre/chatbehavior● https://github.com/datakurre/pubsubannouncementsother relevant links:● https://www.rabbitmq.com/getstarted.html● https://www.amqp.org/about/examples● http://www.manning.com/videla/ (”RabbitMQ in Action”)