Redis & ZeroMQ: How to scale your application

Follow this topic:

@rjsmelo

Redis & ZeroMQ: How to scale your application

RICARDO MELO

Presented at #PHPLX – 11 July 2013

@rjsmelo 2

RICARDO MELO

● CTO @ DRI● PHP, Mysql, Linux and lots of other OSS

● ZCE, RHCE, LPI 3, ITIL, etc● +10 years building (and breaking) things

@rjsmelo 3

About

● 14 Year old academic spin-off● Pragmatic OSS Orientation● PHP, Mysql, SugarCRM, Drupal, JavaScript, Linux, etc.

● Crafters, Integrators

● Always looking for software developers– Yes, right now!

1999 - 2013 DRI. Some Rights Reserved. 4

Outline

● Redis● ZeroMQ● Use Cases● Conclusions


Redis

“Redis is an open source, BSD licensed, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets.”

source: http://redis.io


Predis

● Pure PHP Redis Client– https://github.com/nrk/predis

● Use composer:– composer install

{ "name": "rjsmelo/talk-redis-zmq", "description": "Sample code for Redis & ZeroMQ presentation", "require": { "ext-zmq": "*", "predis/predis": "dev-master" }, "license": "Apache-2.0", "authors": [ { "name": "Ricardo Melo", "email": "[email protected]" } ]}


Strings

1 <?php 2 //redis_strings.php 3 require_once __DIR__ . '/../vendor/autoload.php'; 4 $redis = new Predis\Client('tcp://localhost:6379'); 5 6 // Using SET 7 $redis->set("SampleKey", "Throw me any thing here...."); 8 echo $redis->get("SampleKey") . "\n"; // Throw me any thing here.... 9 10 // Remove Key11 $redis->del("SampleKey");12 13 // Using APPEND14 $redis->append("SampleKey", "Hello"); // Hello15 $redis->append("SampleKey", " World!"); // Hello World!16 echo $redis->get("SampleKey") . "\n";17 18 // Other commands: incr, decr, incrby, getrange, setrange19 // http://redis.io/commands/#string


Lists

1 <?php 2 //redis_lists.php 3 require_once __DIR__ . '/../vendor/autoload.php'; 4 $redis = new Predis\Client('tcp://localhost:6379'); 5 6 $redis->del('SampleList'); 7 8 // Using {L,R}PUSH 9 $redis->lpush('SampleList', 'a'); // a10 $redis->lpush('SampleList', 'b'); // b, a11 $redis->rpush('SampleList', 'c'); // b, a, c12 13 // Using LLEN14 echo $redis->llen('SampleList') . "\n"; // 315 16 // Using LINDEX (note: zero indexed)17 echo $redis->lindex('SampleList', 1) . "\n"; // a18 19 // Using {L,R}POP20 21 echo $redis->lpop('SampleList') . "\n"; // b22 echo $redis->rpop('SampleList') . "\n"; // c23 24 // Other commands: lrange, rpoplpush25 // http://redis.io/commands#list


Sets

1 <?php 2 //redis_sets.php 3 require_once __DIR__ . '/../vendor/autoload.php'; 4 $redis = new Predis\Client('tcp://localhost:6379'); 5 6 $redis->del('SampleSet'); 7 $redis->del('OtherSampleSet'); 8 9 // Using SADD10 $redis->sadd('SampleSet', 'Hello'); // Hello11 $redis->sadd('SampleSet', 'World'); // Hello, World12 $redis->sadd('SampleSet', 'World'); // Hello, World13 14 // Using SMEMBERS15 var_dump($redis->smembers('SampleSet')); // Hello, World16 17 // Using SINTER18 $redis->sadd('OtherSampleSet', 'Hello');19 $redis->sadd('OtherSampleSet', 'All');20 var_dump($redis->sinter('SampleSet', 'OtherSampleSet')); // Hello21 22 // Using SUNION23 var_dump($redis->sunion('SampleSet', 'OtherSampleSet')); // Hello, World, All24 25 // Other commands: smove, srandmember, srem, scard26 // http://redis.io/commands#set


Hashes

1 <?php 2 //redis_hashes.php 3 require_once __DIR__ . '/../vendor/autoload.php'; 4 $redis = new Predis\Client('tcp://localhost:6379'); 5 6 $redis->del('SampleHash'); 7 8 // Using HMSET 9 $redis->hmset("SampleHash", array(10 'prop_a' => 'aaa',11 'prop_b' => 'bbb',12 ));13 14 // Using HGETALL15 var_dump($redis->hgetall("SampleHash")); // prop_a=>aaa, prop_b=>bbb16 17 // Using HSET18 $redis->hset('SampleHash', 'prop_b', 'ccc');19 20 //USING HGET21 echo $redis->hget("SampleHash", 'prop_b') ."\n"; // ccc22 23 // Other commands: hexists, hdel, hlen, hkeys, hvals24 // http://redis.io/commands/#hash


Sorted Sets

1 <?php 2 //redis_sorted_sets.php 3 require_once __DIR__ . '/../vendor/autoload.php'; 4 $redis = new Predis\Client('tcp://localhost:6379'); 5 6 $redis->del('SampleSortedSet'); 7 8 // Using SADD 9 $redis->zadd('SampleSortedSet', 1, 'Hello'); // Hello(1)10 $redis->zadd('SampleSortedSet', 2, 'World'); // Hello(1), World(2)11 $redis->zadd('SampleSortedSet', 3, 'World'); // Hello(1), World(3)12 13 // Using ZRANGE14 var_dump($redis->zrange('SampleSortedSet', 0, -1, array('withscores'=>true))); // Hello(1), World(3)15 16 // Using ZSCORE17 echo $redis->zscore('SampleSortedSet', 'World') . "\n"; // 318 19 // Other commands: zrank, zrevrange, zrangebyscore, zremrangebyscore20 // http://redis.io/commands#sorted_set


ZeroMQ

“ØMQ is a high-performance asynchronous messaging library aimed at use in scalable distributed or concurrent applications. It provides a message queue, but unlike message-oriented middleware, a ØMQ system can run without a dedicated message broker. With bindings for 30+ languages”

source: https://en.wikipedia.org/wiki/%C3%98MQ


PHP Module - zmq

## ZMQ instalation - http://www.zeromq.org/intro:get-the-software#

tar xzvf zeromq-3.2.2.tar.gzcd zeromq-3.2.2./configuremakemake installecho "/usr/local/lib" > /etc/ld.so.conf.d/usr_local_lib.confldconfig

## ZMQ PHP Binding Instalation#

pear channel-discover pear.zero.mqpecl install pear.zero.mq/zmq-betaecho "extension=zmq.so" > /etc/php.d/zmq.ini


Socket Types

● Threads in a process (inproc://)● Processes in a box (ipc://)● Processes over the network (tcp://)● Multicast group (pgm://)


Pattern: Request - Reply


Pattern: Request - Reply

1 <?php 2 // zmq_request.php 3 $context = new ZMQContext(); 4 5 $requester = new ZMQSocket($context, ZMQ::SOCKET_REQ); 6 $requester->connect("tcp://localhost:5555"); 7 8 for ($number = 0 ; $number <= 10 ; $number++) { 9 $mensage = "Hello " . $number . "!";10 echo "Sending - " . $mensage . "\n";11 $requester->send($mensage);12 $reply = $requester->recv();13 echo "Received - " . $reply . "\n";14 }

1 <?php 2 //zmq_reply.php 3 $context = new ZMQContext(); 4 5 $responder = new ZMQSocket($context, ZMQ::SOCKET_REP); 6 $responder->bind("tcp://*:5555"); 7 8 while (true) { 9 $request = $responder->recv();10 echo $request . "\n";11 sleep (1); // some work12 $responder->send("Reply to: " . $request);13 }


Pattern: Publish - Subscribe


Pattern: Publish - Subscribe 1 <?php 2 // zmq_publisher.php 3 $context = new ZMQContext(); 4 5 $publisher = $context->getSocket(ZMQ::SOCKET_PUB); 6 $publisher->bind("tcp://*:5556"); 7 8 $timezones = array('UTC', 'EST'); 9 while (true) {10 foreach($timezones as $tz) {11 date_default_timezone_set($tz);12 $message = date('T:c'); // the message to broadcast13 $publisher->send($message);14 }15 sleep(1);16 } 1 <?php

2 // zmq_subscriber.php 3 $context = new ZMQContext(); 4 5 $subscriber = $context->getSocket(ZMQ::SOCKET_SUB); 6 $subscriber->connect("tcp://localhost:5556"); 7 8 $filter = $_SERVER['argc'] > 1 ? $_SERVER['argv'][1] : "UTC"; 9 $subscriber->setSockOpt(ZMQ::SOCKOPT_SUBSCRIBE, $filter);10 11 while (true) {12 $message = $subscriber->recv();13 echo substr($message,4) . "\n"; // remove prefix14 }


Pattern: Pipeline

20

Pattern: Pipeline 1 <?php 2 // zmq_ventilator.php 3 // extracted from: http://zguide.zeromq.org/ 4 5 $context = new ZMQContext(); 6 7 // Socket to send messages on 8 $sender = new ZMQSocket($context, ZMQ::SOCKET_PUSH); 9 $sender->bind("tcp://*:5557");10 11 echo "Press Enter when the workers are ready: ";12 $fp = fopen('php://stdin', 'r');13 $line = fgets($fp, 512);14 fclose($fp);15 echo "Sending tasks to workersâÀ¦", PHP_EOL;16 17 // The first message is "0" and signals start of batch18 $sender->send(0);19 20 // Send 100 tasks21 $total_msec = 0; // Total expected cost in msecs22 for ($task_nbr = 0; $task_nbr < 100; $task_nbr++) {23 // Random workload from 1 to 100msecs24 $workload = mt_rand(1, 100);25 $total_msec += $workload;26 $sender->send($workload);27 28 }29 30 printf ("Total expected cost: %d msec\n", $total_msec);31 sleep (1); // Give 0MQ time to deliver


Pattern: Pipeline

1 <?php 2 // zmq_worker.php 3 // extracted from: http://zguide.zeromq.org/ 4 5 $context = new ZMQContext(); 6 7 // Socket to receive messages on 8 $receiver = new ZMQSocket($context, ZMQ::SOCKET_PULL); 9 $receiver->connect("tcp://localhost:5557");10 11 // Socket to send messages to12 $sender = new ZMQSocket($context, ZMQ::SOCKET_PUSH);13 $sender->connect("tcp://localhost:5558");14 15 // Process tasks forever16 while (true) {17 $string = $receiver->recv();18 19 // Simple progress indicator for the viewer20 echo $string, PHP_EOL;21 22 // Do the work23 usleep($string * 1000);24 25 // Send results to sink26 $sender->send("");27 }

22

Pattern: Pipeline 1 <?php 2 // zmq_sink.php 3 // extracted from: http://zguide.zeromq.org/ 4 5 // Prepare our context and socket 6 $context = new ZMQContext(); 7 $receiver = new ZMQSocket($context, ZMQ::SOCKET_PULL); 8 $receiver->bind("tcp://*:5558"); 9 10 // Wait for start of batch11 $string = $receiver->recv();12 13 // Start our clock now14 $tstart = microtime(true);15 16 // Process 100 confirmations17 $total_msec = 0; // Total calculated cost in msecs18 for ($task_nbr = 0; $task_nbr < 100; $task_nbr++) {19 $string = $receiver->recv();20 if ($task_nbr % 10 == 0) {21 echo ":";22 } else {23 echo ".";24 }25 }26 27 $tend = microtime(true);28 29 $total_msec = ($tend - $tstart) * 1000;30 echo PHP_EOL;31 printf ("Total elapsed time: %d msec", $total_msec);32 echo PHP_EOL;


Pattern: Pipeline


Use Case: Service Cluster

Reference: http://zguide.zeromq.org/page:all#Service-Oriented-Reliable-Queuing-Majordomo-Pattern


Use Case: “Unix Style” Application

● Lots of simple, “focused” programs– Or in different servers, different languages, etc

● All components collaborate to get the job done– cat file | sort | uniq -c

● Glue everything with ZeroMQ


Use Case: Cache

● We need to cache some values for speed

● Use SET / GET on Redis● I could use memcache, but this is a Redis Talk :-)

● They have similar performance– http://redis.io/topics/benchmarks


Use Case: Buffer

● Use Redis as a FIFO (or LIFO)● Hidden Goods

– Decoupling

– Flow control

– rpoplpush


Use Case: Background Tasks

● Your application needs to do some heavy work– Send Email

– Image Resizing

– Mega Huge map-reduce query to your pentabyte cluster :-)

● You don't want to keep your user waiting

● Send things to background


Use Case: Background Tasks

● Use Redis as your Job Queue– Your application should send job to be run and parameters to the

queue

● Use workers to POP jobs from the queue and do the heavy work.

● Don't reinvent the well– Look for php-resqueue


Conclusions

● Both ZeroMQ and Redis are extremely fast

● ZeroMQ is great to “glue things” as well as adding flexibility to scale dynamical

● Redis is great as a queue and allows you to cope with your load.

Thank you

Follow this topic:

@rjsmelo

QA

Code: https://github.com/rjsmelo/talk-redis-zmq

Feedback: https://joind.in/talk/view/8937

www.dri-global.com

@rjsmelo

[email protected]

Technology

Redis & ZeroMQ: How to scale your application