Queue Everything and Please Everyone

Queue Everything&

Please Everyone

* * *

Vaidik Kapoor

Who am I?

➔ Undergraduate student.➔ Involved with:

✔ Mozilla as a Contributor and Rep✔ Drupal as a Contributor

➔ Twitter: @vaidikkapoor➔ Github: vaidikkp➔ Web: vaidikkapoor.info

Web Applications

Responsiveness

Why do everything at once?

Task 1 Task 2 Task 3 Task N

ClientRequest Response

Break the Request-Response Cycle

Task 1

Task 2

Task 3

Task N

ClientRequest Response

Meet Queues

What can Queues help with?

➔ Background Processing✔ Data Processing✔ Media Processing✔ Updating Caches

➔ Anything that you want to offload off your server.

➔ Improve the overall User Experience

Redis+

HotQueue / PyRes

Redis + HotQueue

PRODUCER:

from hotqueue import HotQueuequeue = HotQueue("myqueue", host="localhost", port=6379, db=0)

queue.put(message)

CONSUMER / WORKER:

from hotqueue import HotQueuequeue = HotQueue("myqueue", host="localhost", port=6379, db=0)

while True: message = queue.get() # do something awesome

Redis + PyRes

➔ Python clone of Github's Resque➔ Offers a lot more features than HotQueue

✔ Job failure handling✔ Queue status information

➔ Comes with Monitoring System written in Ruby

Redis + PyRes

PRODUCER:

from pyres import ResQr = Resq()

class WebPage: queue = “screenies”

@staticmethod def perform(urll): # save the screenshot

r.enqueue(WebPage, 'http://python.org')

CONSUMER:

./pyres_worker screenies

Message Queue

A system for enabling asynchronous processing of discrete tasks.

What can MQs help with?

➔ Everything that generic Queues can help with.➔ Increase Reliability➔ Cron Jobs (Celery)

Available MQ Solutions

➔ RabbitMQ➔ Amazon Simple Queue➔ Apache MQ➔ Gearman➔ Starling➔ OpenAMQ➔ Sun Java Message Queue System

Message Queue Protocols

➔ AMQP: Advanced Message Queue Protocol➔ JMS: Java Messaging Service➔ STOMP: Streaming Text Oriented Messaging

Protocol

Criteria for Broker Selection

➔ Difficulty in Recovery➔ Relatively low level of required maintenance➔ Ease of Deployment➔ Durability➔ Persistence➔ Community Support➔ Cluster Support➔ What language is it written in?

RabbitMQ

➔ A Message Broker➔ Implements AMQP

Pika & Other AMQP Libraries

Celery

➔ An amazing Task Manager➔ Batteries Included➔ Uses RabbitMQ as broker (or almost anything)➔ Libraries for most of the common web frameworks

like Django, Flask, Pyramid➔ Supports multiprocessing.➔ Distribute tasks easily.➔ Makes life easy

Celery Example

from celery.task import task

@task():def take_screenshot(url): # magical code to take screenshot comes here

How to put it all together?

Web App BrokerTask

Manager

WorkerServer (s)Database

The General Setup

Then what?

Notify Your Users

Django's Messaging System

Flask's Flashes

Poll using AJAX

WebSockets

Email

Things to Remember

Isolate

Isolate code, make reusable components.

Recycle

Remove Request Dependency

Unit Testing?

Review

➔ Choose a system according to needs➔ Build a robust system➔ Integrate seamlessly with your UI➔ Isolate & Recycle: make it a habit➔ Improve the overall UX!➔ Please everyone, even yourself!

References

➔ PyRes - http://pyres.readthedocs.org/

➔ RabbitMQ - http://www.rabbitmq.com/

➔ Celery - http://celeryproject.org/

➔ Amazing Article - http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes

Thank You!