Upload
xebia-france
View
483
Download
1
Embed Size (px)
Citation preview
#DevoxxFR
Hand’s on
Kafka : http://kafka.apache.org/downloads.htmlHand’s on : https://github.com/mblanc/hands_on_kafka.git
#DevoxxFR
@matthieublanc
@slequeux
Matthieu BlancSylvain Lequeux
#DevoxxFR
Messaging System?
#DevoxxFR
Jay KrepsNeha NarkhedeJun Rao
History
#DevoxxFR
WebAppRelationalDB
NoSQLDB
DWH
Hadoop
Monitoring Logs
#DevoxxFR
WebApp
RelationalDB
NoSQLDB
DWH
Hadoop
ActiveMQ
WebApp
Logs
Monitoring
WebApp
Search
Big Data?
#DevoxxFR
WebApp
RelationalDB
NoSQLDB
DWH
Hadoop ETL
ActiveMQ
WebApp
Logs
Monitoring
WebApp
Search
BIGMESS!
#DevoxxFR
Stream Data Platform
#DevoxxFR
● Decoupling Systems● High throughput● Distributed - Horizontal scaling● Multi consumers● Persistence● Automatic recovery from broker failure
Features
#DevoxxFR
● Cost● Persistence● Batch system -> perfs down● Large scale stream processing● Ordering guarantees
RabbitMQ/ActiveMQ?
#DevoxxFR
Consumer
Broker
Consumer
Consumer
Kafka Cluster
Broker Broker
Broker Broker Broker
Zookeeper
Producer
Producer
Producer
Architecture
#DevoxxFR
Distributed Commit Logs
10 11 12 13 14 15 16 17 18987654321 19
1st recordNext recordWritten
Reads(sequential access = high performance)
#DevoxxFR
Producer
10 11 12 13 14 15 16 17 18987654321
10 11 12 13 14987654321 15
10 11 12 13 14 15987654321 16
Partition #1
Partition #2
Partition #3
ProducerProducer
19
16
17
offset
Old New
Writes
Writes
Writes
message : (key bytes[ ], value bytes[ ])
#DevoxxFR
Topic storage
10 11 12 13 14 15 16 17 18987654321
Partition #1
directory segment = file
#DevoxxFR
Fast
● Sequential Access● PageCache● Linux : sendfile()● Compression
Source : http://queue.acm.org/detail.cfm?id=1563874
#DevoxxFR
Fast
#DevoxxFR
Consumer Group
10 11 12 13 14 15 16 17 18987654321
10 11 12 13 14987654321 15
10 11 12 13 14 15987654321 16
19
16
17
Producer
Consumer Group A
Consumer Group A
Consumer Group A
Consumer Group B
Consumer Group B
Partition #1
Partition #2
Partition #3
Writes
Consumption
#DevoxxFR
Fault tolerant consumption
10 11 12 13 14 15 16 17 18987654321
10 11 12 13 14987654321 15
10 11 12 13 14 15987654321 16
19
16
17
Producer
Consumer Group A
Consumer Group A
Consumer Group A
Consumer Group B
Consumer Group B
Partition #1
Partition #2
Partition #3
Writes
Automatic rebalancing on failure
#DevoxxFR
Consumer group
10 11 12 13 14 15 16 17 18987654321
10 11 12 13 14987654321 15
10 11 12 13 14 15987654321 16
Partition #1
Partition #2
Partition #3
Group Topic # Offset
1 log 1 18
1 log 2 12
1 log 3 14
2 log 1 1
2 log 2 0
2 log 3 3
Consumer group 2 Consumer group 1
Old New
#DevoxxFR
Replicas/ISRs
Partition #0
Partition #2
Topic : fooPartitions : 3Replicas : 3
Partition #1
Partition #0
Partition #2
Producer
Broker #0 Broker #1 Broker #2
WritesConsumer
Leader
Leader
Leader
Partition #1
Partition #2
Partition #0
Partition #1
#DevoxxFR
Ka ka 0.9 - New Consumer
● Unified consumer API● Much simpler and thinner● Allows for larger groups with far faster
rebalancing● Decouple Kafka clients from Zookeeper!!!
#DevoxxFR
Security
● Authentication : Kerberos / TLS certificate● Authorization : unix-like permissions
system ● Encryption on the wire : SSL● Encryption at rest : encrypting individual
fields / filesystem security features● User defined quota
#DevoxxFR
Ka ka Connect
Kaf
ka C
onne
ct
DataSource
Kaf
ka C
onne
ct
DataSink
Kafka
#DevoxxFR
Ka ka Streams
Kaf
ka C
onne
ct
DataSource
Kaf
ka C
onne
ct
DataSink
Kafka
KafkaStreams
#DevoxxFR
Jay KrepsNeha NarkhedeJun Rao
Ka ka Enterprise Ready
2011 2012
2014
#DevoxxFR
● User behaviour, click stream analysis● Infrastructure monitoring and security ● Telemetry data from mobile/sensors● IoT● Log analysis● ...
Use cases
#DevoxxFR
Used by
● LinkedIn : activity stream, metrics● Netflix : Real-time Monitoring● Twitter : Real-time data pipeline● Spotify : log delivery● Loggly : log collection and processing● Mozilla : telemetry data● Microsoft : Ads, Bing, Office● Airbnb, Square, Uber, Criteo, OVH ...
#DevoxxFR
GL HF !
● Download Kafka : http://kafka.apache.org/downloads.html
● Git Clone : https://github.com/mblanc/hands_on_kafka.git
● Open : reveal.js/index_java.html