Kafka in practice

KAFKA in practiceCaique Lima @ Easy Taxi

Apache Kafka

distributed, partitioned, replicated commit log service

Apache Kafka

Apache Kafka

Apache Kafka

Kafka at Easy Taxi

• +600 mi unique events/day

• Currently on 0.9.0

• 3 brokers

• 10 topics

• X consumers

Architecture

Api Kafka

Consumer 1

Consumer 2

Consumer 3

S3

Redis

Consumer N

BQ

Api

…

Redash

Do’s

• Rsyslog as a scalable simple producer

• Use BigQuery to allow SQL

• Start building the event based culture

• Right-time before real-time

Dont’s

• Doesn't run stress tests

• Increase architecture’s complexity

• Produce useless data

• Go live with default configs

Easy Explorer

Easy Taxi Point

Supply demand analysis

• Allow us to understand the area behaviour

• Generate a “health indicator” based on

predefined parameters

• Take actions without any human interaction

References

• Confluent blog

• Kafka Summit

• Jay Kreps interview

• Jepsen: Kafka

• Documentation

• Dados Brasil Slack

THANKS!

QUESTIONS