#DevoxxFR

3 degrees of MoM kafka, rabbitmq, nsq

Guillaume Arnaud @guillarnaud

voyages-sncf.com

1

#DevoxxFR 2

P publisher

C consumer

A Message-oriented Middleware (MoM)just a queue ?

#DevoxxFR 3

ActiveMQ RabbitMQ Nats NSQ ZeroMQ Kafka AmazonSQS Disque Aeron Mosquitto

Beanstalk Apollo Nanomsg …

JMS Stomp MQTT AMQP XMPP …

Flume Redis Logstash Syslog …

Brokers Protocol/API Almost

https://github.com/mfornos/awesome-microservices#messaginghttp://queues.io/

#DevoxxFR 4

ActiveMQ RabbitMQNats NSQZeroMQ KafkaAmazonSQS Disque Aeron Mosquitto

Beanstalk Apollo Nanomsg …

JMS Stomp MQTT AMQP XMPP …

Flume Redis Logstash Syslog …

Brokers Protocol/API Almost

https://github.com/mfornos/awesome-microservices#messaginghttp://queues.io/

ROUTING

DECENTRALIZED

DISTRIBUTED

#DevoxxFR

RabbitMQ.id

5

parents: Pivotal age: v3.6.1 (march 2016) license: mozilla public license native languages: erlang/amqp fluently spoken: python, java, ruby, .Net, nodejs big brother: activemq

#DevoxxFR 6

P

C

decouplingexchange queue routing/binding keys

C

exchange

routing key: devoxx.2016.newuser

binding key: devoxx.2016.*

routing

queue: devoxx2016_service

#DevoxxFR 7

P

C C

exchange: fanout

routing key: devoxx.2016.newuser

binding key: devoxx.2016.*

message 1

fanoutbroadcast no routing rules

message 1 message 1

#DevoxxFR 8

P

C C

exchange: direct

routing key: devoxx.2016.newuser

binding key: devoxx.2016.newuser

message 1

directrouting key=binding key

message 1

queue: new_user_servicequeue: update_user_service

binding key: devoxx.2016.updateuser

#DevoxxFR 9

P

C C

exchange: topic

routing key: devoxx.2016.newuser

binding key: devoxx.2016.*

message 1

message 1 message 1

queue: devoxx_servicequeue: trigger_new_user

binding key: devoxx.2016.newuser

topicwildcard subscription

#DevoxxFR

RabbitMQ.model

10

exchange routing key

queue service

event type

binding key aggregator

business domain

#DevoxxFR

RabbitMQ.microservices

11

exchange'user' / fanout

cv.*

microservicesrouting between exchanges and services

specify_user_action

audit

check_fraud

exchange'cv' / topic

alert_recruiters

crop_photo

cv.photo

cv.employer.new

check_employer

exchange'mailer' / direct

generate_and_send_mail

#DevoxxFR

RabbitMQ.ops

12

• virtual hosts / security • plugins: management, ldap, mqtt, consistent hashing • cluster/federation • queue size/disk • cpu

#DevoxxFR

RabbitMQ.admin

13

$ rabbitmq-plugins enable rabbitmq-management

#DevoxxFR

Kafka.id

14

parents: LinkedIn/confluent age: v0.9.0.1 (feb. 2016) license: apache native language: java/scala family: hdfs, spark streaming, storm, flume

#DevoxxFR 15

P publisher

C consumer

5 4 3 2 1

offset

lag

log file(s)append only zero copy ~ k*100 000 msg/s

#DevoxxFR 16

P

C

5 4 3 2 1

Partitionsplit the topic partition key

4 3 2 1

keys 1, 3, 5… keys 2, 4, 6…

#DevoxxFR 17

P

C

distribution

5 4 3 2 1

4 3 2 1

4 3 2 1

4 3 2 1

node 1 node 2

#DevoxxFR 18

replication

5 4 3 2 1

4 3 2 1

4 3 2 1

P

replica lag

partition leader

partition follower

partition follower

#DevoxxFR 19

P

C

5 4 3 2 1

4 3 2 1

C

Scalingone consumer per partition add partitions for scaling

#DevoxxFR 20

P

C

5 4 3 2 1

4 3 2 1

C

Scalingone consumer per partition add partitions for scaling

C

#DevoxxFR 21

P

C C

Scalingone consumer per partition add partitions for scaling

C

#DevoxxFR 22

P

C C

Groupconsume same messages ~ JMS topic C

group 1

group 2

#DevoxxFR

Kafka.event sourcing

23

domain object

domain object

eventview 1

view 2event store

#DevoxxFR

Kafka.event sourcing

24

1 id:1234 my order (…)

2 id:abcd an other order

3 id:wxyz and again…

120 id:1234 my other order

…

log compaction

offset key payload

#DevoxxFR

Kafka.ops

25

• clients are part of the distributed system (heartbeat, tcp config, etc…)

• lag sender/consumer • replica lags • preferred replica election • repartition master/slave • don't forget Zookeeper • size and time retention • use kafka tools

~ equivalent to a NoSQL database

properties #

server 127

publisher 47

consumer 45

#DevoxxFR

Kafka.admin

26

https://github.com/yahoo/kafka-manager

#DevoxxFR

NSQ.id

27

parents: nsq.io age: v0.3.7 (feb. 2016) license: open source native language: go fluently spoken: go, python, javascript, http(!) zodiacal sign: syslog++

#DevoxxFR 28

P

C

topic: server_start_eventnsqd

channel: server_subscribe

topic: server_start_eventnsqd

channel: server_subscribe

P

consumer: server_subscribe

Cnsqlookupd

http http

tcptcp

#DevoxxFR 29

server

nsqd

server

nsqd

server

nsqd

server

nsqd

push

pull

decentralizedlocal push remote pull

#DevoxxFR 30

nsqd

nsqd

nsqd

haproxy slave

daemon

reload

admin

tomcat

new_ip_port1

2

3

5

6

7

use caseservice discovery

nsqd

tomcat

#DevoxxFR

NSQ.service discovery

31

nsqd

nsqd

nsqd

haproxy slave

daemon

reload

admin

tomcat

new_ip_port1

2

3

5

6

7

nsqd

tomcat

#DevoxxFR

NSQ.ops

32

$ tree bin/ bin/ ├── nsq_pubsub ├── nsq_stat ├── nsq_tail ├── nsq_to_file ├── nsq_to_http ├── nsq_to_nsq ├── nsqadmin ├── nsqd ├── nsqlookupd └── to_nsq

#DevoxxFR

NSQ.ops

33

$ nsqd —help -e2e-processing-latency-percentile value … -max-bytes-per-file int -max-heartbeat-interval duration -max-msg-size int -max-msg-timeout duration -max-output-buffer-size int -max-output-buffer-timeout duration -max-rdy-count int -max-req-timeout duration -mem-queue-size int -msg-timeout string -snappy …

#DevoxxFR

MoM.diversity

34

#DevoxxFR

NSQ.admin

35

http://nsq.io/components/nsqadmin.html

#DevoxxFR

your MoM is a part of your application

36

#DevoxxFR

your application is a part of your distributed system

37

#DevoxxFR 38

http://www.meetup.com/fr-FR/Meetup-Messaging-Paris/

#DevoxxFR

Questions ?

39

#DevoxxFR

Thanks

40