Embed Size (px)
Citation preview
• • Kafka Streams • State for Kafka Streams • Queryable State for Kafka Streams • •
2
• 2011/04
• 2015/09
• • Druid (KDP, 2015)
• RDB NoSQL ( , 2016; : HBase )
• ESP8266 Wi-Fi IoT (KDP, 2016)
• • (DEIM 2014)
• (WebDB Forum 2014)
• Spark Streaming (Spark Meetup December 2015)
• Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016)
• (FutureOfData; 2016)
3
Apache Storm
PipelineDB
Druid
Apache Kafka
4
Apache Storm
Memcached
Redis
Apache Kafka
5
Apache Kafka
6
Kafka
7
Apache Kafka Version 0.10.1.0
• New Features • [KAFKA-1464] - Add a throttling option to the Kafka replication tool • [KAFKA-3176] - Allow console consumer to consume from
particular partitions when new consumer is used. • [KAFKA-3492] - support quota based on authenticated user name • [KAFKA-3776] - Unify store and downstream caching in streams • [KAFKA-3858] - Add functions to print stream topologies • [KAFKA-3909] - Queryable state for Kafka Streams • [KAFKA-4015] - Change cleanup.policy config to accept a list of
valid policies • [KAFKA-4093] - Cluster id
8
Kafka Streams
• Version 0.10.0.0
• Kafka API
9
Kafka Streams
Stateless transformations Stateful transformations
10
State for Kafka Streams
• Stateful transformations
•
11
State
• Kafka Streams
• 2
• State Store RocksDB
• Kafka StateChangelog
• Kafka Streams
• Key Value
• Window12
• Record Stream
• Changelog Stream
• State (Table)
Record Changelog State
13
Queryable State for Kafka Streams
• Version 0.10.1.0
• State Store API
14
Queryable State
• ReadOnlyKeyValueStore • Window Streams State
• Key
• Value
• ReadOnlyWindowStore • Window Streams State
• Key, From (Long), To (Long)
• Iterator<KeyValue<Long, Value>>
15
Kafka Streams
• Stateful transformations count
KafkaStreams streams = null; KStreamBuilder builder = new KStreamBuilder(); KStream<String, String> textLines = builder.stream(stringSerde, stringSerde, inputStream); KStream<String, Long> wordCounts = textLines .flatMapValues(value -> ...) .groupBy((key, word) -> word) .count(storeName) .toStream(); wordCounts.to(stringSerde, longSerde, outputStream); streams = new KafkaStreams(builder, conf);
17
Queryable State Streams
• Streams o.a.k.streams.KafkaStreams
KafkaStreams streams = ...;
String key = "kafka"; ReadOnlyKeyValueStore<String, Long> store = streams.store( storeName, QueryableStoreTypes.<String, Long>keyValueStore()); Long value = store.get(key); return value;
18
Queryable State Streams
• API
KafkaStreams streams = ...;
get("/results/:key", (req, res) -> { String key = req.params(":key"); ReadOnlyKeyValueStore<String, Long> store = streams.store( storeName, QueryableStoreTypes.<String, Long>keyValueStore()); Long value = store.get(key); return value; });
19
github:laclefyoshi/kafka_streams_example
Apache Kafka
21
HTTP
$ java -cp target/kafka_streams_example-1.0-SNAPSHOT.jar \ org.saekiyoshiyasu.App kafka -> null ... kafka -> 5
$ ./bin/kafka-console-producer.sh --topic input-stream \ --broker-list 127.0.0.1:9092 What is Kafka good for? ...
22
Queryable State for Kafka Streams
•
Kafka Streams ?
23
Queryable State for Kafka Streams
•
Kafka Streams ?
State
24
• Queryable State
• Kafka Streams
25
