Apache Kafka & Kafka Connect
( ETL )
Future Architect ,IncKeigo Suda
2016/11/25D&S Data Night vol.04
12
!Kafka Connect ETL( )
( ) /
13
!Future Architect ,Inc ( )
!
! IoT
16
!Kafka ver 0.9!Kafka!
!
! /
Kafka Connect
17
Connectors
https://www.confluent.io/product/connectors/
22
!
!
!
Kafka Connect ( )
23
ETL Kakfa Connect( )
P T
// / /
AC
KM
I
M
KM
25http://docs.confluent.io/2.0.0/connect/userguide.html#getting-started
Kakfa Connect
Kafka Connect
26
Kakfa Connect
http://docs.confluent.io/2.0.0/connect/userguide.html#getting-started
27
Stream & Partition(RDB )
http://www.slideshare.net/KaufmanNg/data-pipelines-with-kafka-connect
28
N:1
Stream & Partition(RDB )
http://www.slideshare.net/KaufmanNg/data-pipelines-with-kafka-connect
29
Kakfa Connect
http://docs.confluent.io/2.0.0/connect/userguide.html#getting-started
30
Worker & Connector
http://docs.confluent.io/2.0.0/connect/userguide.html#getting-started
31
CONNECTERWORKER
STREAM
PARTITIONSTANDALONE MODE
DISTRIBUTED MODE
TASK
32
$ bin/connect-standalone.sh config/connect-standalone.properties connector1.properties
(Standalone mode)
!
!
! Worker! Connector
Worker Connector
33
name=local-file-source connector.class=FileStreamSource tasks.max=1 file=test.txt topic=connect-test
bootstrap.servers=localhost:9092 key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter ...
Worker
Connector
36
!Kafka Connect Source Sink
!Sink/Source Sink
Kafka Connect
Source Sink
37
! Connector/Task OK! API OK
( )
( )
39
!!(source pull)
Task
40
! Task Connector
!
Deploy
name=sample-sink connector.class=SampleSinkConnector tasks.max=1 topic=connect-test
Connector
42
!Kafka
!
Kafka Connect
43
!
API
Kafka Connect
44
(kafka-connect-jdbc)
45
! ConfluentConfuluent Platform
Kafka Connect
46
!Kakfa ( )!Confluent….