Upload
others
View
21
Download
0
Embed Size (px)
Citation preview
IBM Event Streams
© 2018 IBM Corporation
Apache Kafka as an event backbone
Event Streaming with Apache Kafka in IBM Cloud Private
Please note
2Event Driven Enterprise Workshop / © 2018 IBM Corporation
IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice and at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract.
The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.
Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
Becoming event-driven to deliver engaging customer experiences
3
Phone company has existing data around customers’ usage
and buying preferences
Combined with events generated when phones connect to in-store wi-fi
Enables a more engaging and personal customer experience
4Event Driven Enterprise Workshop / © 2018 IBM Corporation
The next wave of digital transformation is event-driven
Source: Gartner - From data-centric to event-centric IT priority
Isn’t that just “messaging”?
5Event Driven Enterprise Workshop / © 2018 IBM Corporation
Stream History
Transient Data Persistence
Immutable DataScalable Consumption
TargetedReliable Delivery
Operations
Events
Request / Reply
�
Event Driven Enterprise Workshop / © 2018 IBM Corporation
Apache Kafka is an Open-Source Streaming Platform
Kafka Cluster
Producer ProducerProducer
Consumer ConsumerConsumer
Stream Processor
Stream Processor
Source Connector
Sink Connector
External System
External System
Use cases
Pub/sub messaging
Website activity tracking
Metrics
Log aggregation
Stream processing
Event sourcing
Commit log
6
Kafka is built for scale
Event Driven Enterprise Workshop / © 2018 IBM Corporation
0 1 2 3 4 5
Topic
Partition 0
7
Kafka is built for scale
Event Driven Enterprise Workshop / © 2018 IBM Corporation
0 1 2 3 4 5
Topic
6 7Partition 0
8
Kafka is built for scale
Event Driven Enterprise Workshop / © 2018 IBM Corporation
0 1 2 3 4 5
Topic
6 7
0 1 2 3
0 1 2 3 4 5
Partition 0
Partition 1
Partition 2
9
Kafka is built for scale
Event Driven Enterprise Workshop / © 2018 IBM Corporation
Broker 1 Broker 2 Broker 3
10
Kafka is built for scale
Event Driven Enterprise Workshop / © 2018 IBM Corporation
Broker 1 Broker 2 Broker 3
11
Kafka is built for scale
Event Driven Enterprise Workshop / © 2018 IBM Corporation
Broker 4 Broker 5Broker 1 Broker 2 Broker 3
12
Kafka is built for scale
Event Driven Enterprise Workshop / © 2018 IBM Corporation
Broker 4 Broker 5Broker 1 Broker 2 Broker 3
13
Kafka is built for availability
Event Driven Enterprise Workshop / © 2018 IBM Corporation
Broker 1 Broker 2 Broker 3
Leader Leader Leader
14
Broker 4
Kafka is built for availability
Event Driven Enterprise Workshop / © 2018 IBM Corporation
Broker 1 Broker 2 Broker 3
Leader
Leader
Leader
15
Broker 4
Topics are partitioned for scale
Event Driven Enterprise Workshop / © 2018 IBM Corporation
0 1 2 3 4 5
Topic
6 7
0 1 2 3
0 1 2 3 4 5
Partition 0
Partition 1
Partition 2
Writes
Old New
16
Each partition is an immutable sequence of records
Event Driven Enterprise Workshop / © 2018 IBM Corporation
0 1 2 3 4 5
Topic
6 7Partition 0 8 9 10
Producer
writes
Consumer A(offset 6)
Consumer B(offset 9)
reads
17
Consumer groups share the partitions
Event Driven Enterprise Workshop / © 2018 IBM Corporation
0 1 2 3 4 5
Topic
6 7
0 1 2 3
0 1 2 3 4 5
Partition 0
Partition 1
Partition 2
Consumer Group A
Consumer Group B
Consumer
Consumer
Consumer
Consumer
Consumer
Consumer
p0, offset 7
p1, offset 3
p2, offset 4
p0, offset 6
p1, offset 1
p2, offset 5
18
Reliability
Event Driven Enterprise Workshop / © 2018 IBM Corporation
Producer can choose acknowledgement level0 Fire-and-forget, fast but risky1 Waits for 1 broker to acknowledgeAll Waits for all replica brokers to acknowledge
Producer can choose whether to retry0 Do not retry, lose messages on error>0 Retry, might results in duplicates on error
Producer can also choose idempotenceMeans that retries can be used without risk of duplicates
Consumer can choose how to commit offsetsAutomatic
Commits might go faster than processingManual, asynchronous
Fairly safe, but could re-process messagesManual, synchronous
Safe, but slows down processing
A common pattern is to commit offsets on a timer
Exactly-once semanticsCan group sending messages and committing offsets into transactionsPrimarily aimed at stream processing applications
19
Compacted topics are evolving data stores
Event Driven Enterprise Workshop / © 2018 IBM Corporation
0
key:aval:A1
1
key:bval:B1
2
key:aval:A2
3
key:cval:C1
Partition 0
20
4
key:cval:C2
5
key:bval:
Producer
writes
Consumer takes latest values and builds own data store
readsKey Value
a A2
b <deleted>
c C2
Old New
Source change log Read
Compacted topics are evolving data stores
Event Driven Enterprise Workshop / © 2018 IBM Corporation
0
key:aval:A1
1
key:bval:B1
2
key:aval:A2
3
key:cval:C1
Partition 0
21
4
key:cval:C2
5
key:bval:
2
key:aval:A2
4
key:cval:C2
5
key:bval:
Periodic compaction eliminates duplicate keys to minimize storage
Partition 0(rewritten)
Key Value
a A2
b <deleted>
c C2
Event Driven Enterprise Workshop / © 2018 IBM Corporation
Kafka Streams
!!! �
J
O
I
N
TRANSFORM
FILTER AGGREGATE
22
Kafka Streams concepts
Event Driven Enterprise Workshop / © 2018 IBM Corporation
A stream is an unbounded, continuously updating data set of ordered, replayable, immutable records
A topology is a graph of processors (nodes) joined by streams (edges)
A stream processor is a node in a topology which receives input records processes them to produce output records and state
A state store provides the capability to store and query data
Created automatically by some operations
Two kinds of streamKStream
Just a topic
KTableA changelog stream of <key,value>Each record is an update for a keyActually 2 topics
Table (compacted)Changelog (not compacted)
There’s a nice duality between streams and tables
23
Kafka Streams example
Event Driven Enterprise Workshop / © 2018 IBM Corporation
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "my-streams-application");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka1:9092");
StreamsConfig config = new StreamsConfig(props);
KStreamBuilder builder = new KStreamBuilder();
KStream<String, String> source = builder.stream("my-input")
.through("my-logger")
.filter((key,value) -> key.equals("bingo"))
.map((key,value) -> KeyValue.pair(key, value.toUpperCase()))
.to("my-output");
KafkaStreams streams = new KafkaStreams(builder, config);
streams.start();
24
my-input
!!!
FILTER
my-output
MAP
my-logger
THROUGH
Event Driven Enterprise Workshop / © 2018 IBM Corporation
Kafka Connect
Source
Sink
External System
External System
Over 80 connectors
HDFS
Elasticsearch
MySQL
JDBC
IBM MQ
MQTT
CoAP
+ many others
https://www.confluent.io/product/connectors/
Connector (Transform) Converter
Connector (Transform) Converter
25
Kafka Connect MQ Source connector
Event Driven Enterprise Workshop / © 2018 IBM Corporation
MQ Source connectorOpen-source – build it yourself
Use any supported MQ release
Uses JMS client internally
Client connections
Supports TLS, authentication
MQ queue->Kafka topic
Support for binary, text, JSON
Easy to extend
RecordBuilder Converter
https://github.com/ibm-messaging/kafka-connect-mq-source
FROM.MQ.TOPICTO.KAFKA.Q
MQ Message
MQMD
(MQRFH2)
Payload
Kafka Record
Null key
Valuebyte[]
SourceRecord
Schema
Value(may be
complex)
26
Kafka Connect MQ Sink connector
Event Driven Enterprise Workshop / © 2018 IBM Corporation
MQ Sink connectorOpen-source – build it yourself
Use any supported MQ release
Uses JMS client internally
Client connections
Supports TLS, authentication
Kafka topic -> MQ queue
Support for binary, text, JSON
Easy to extend
Converter MessageBuilder
https://github.com/ibm-messaging/kafka-connect-mq-sink
FROM.KAFKA.QTO.MQ.TOPIC
Kafka Record
Key
Valuebyte[]
MQ Message
MQMD
Payload
SinkRecord
Schema
Value(may be
complex)
(MQRFH2)
27
Event Driven Enterprise Workshop / © 2018 IBM Corporation
Disaster recovery
Primary Datacenter
T1
Backup Datacenter
T1
mirrormaker
Consumer config
Producer config
28
Event Driven Enterprise Workshop / © 2018 IBM Corporation
Active-active replication
Datacenter 1
DC1.T2
DC2.T2
Datacenter 2
DC1.T2
DC2.T2
mirrormaker
Consumer config
Producer config
mirrormaker
Consumer config
Producer config
29
IBM and Apache Kafka
30
IBM Expertise
• Running Apache Kafka as a single and multi-tenant offering since December 2015
• Continue to make contributions to the open-source project
• Team located across 3 time zones
IBM Expertise IBM Cloud
Apache Kafka
Apache Kafka
• Built with open-source technology and therefore open protocols
• Clients benefit from innovation within IBM but also across the wider community
IBM Cloud
• Deployment to 50+ locations worldwide
• Integrations to key services
IBM Event Streams [Tech Preview]
© 2018 IBM Corporation
Apache Kafka 1.1
New UI focused on developer needs
Try it for free on IBM Cloud Private
Released 23 May 2018
https://developer.ibm.com/messaging/event-streams/
32
We invite youto become a Sponsor User Become part of our team and help shape
the future of our offerings!
Tell us about your work through workshops and
interviews. Be first to preview the latest designs, test
new offerings, and more.
[email protected] Williamson –Design Researcher
To find out more, and for details on how to get involved, email:
At IBM, we work with real users who can bring
their experience and expertise to our team…
Thank you
Event Driven Enterprise Workshop / © 2018 IBM Corporation
Andrew SchofieldChief Architect, Event Streams—[email protected]
33
Notices and disclaimers
Event Driven Enterprise Workshop / © 2018 IBM Corporation
© 2018 International Business Machines Corporation. No part of this document may be reproduced or transmitted in any form without written permission from IBM.
U.S. Government Users Restricted Rights — use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. This document is distributed “as is” without any warranty, either express or implied. In no event, shall IBM be liable for any damage arising from the use of this information, including but not limited to, loss of data, business interruption, loss of profit or loss of opportunity. IBM products and services are warranted per the terms and conditions of the agreements under which they are provided.
IBM products are manufactured from new parts or new and used parts. In some cases, a product may not be new and may have been previously installed. Regardless, our warranty terms apply.”
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those
customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer follows any law.
34
Notices and disclaimerscontinued
Event Driven Enterprise Workshop / © 2018 IBM Corporation
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products about this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM expressly disclaims all warranties, expressed or implied, including but not limited to, the implied warranties of merchantability and fitness for a purpose.
The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right.
IBM, the IBM logo, ibm.com and [names of other referenced IBM products and services used in the presentation] are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.
.
35
Event Driven Enterprise Workshop / © 2018 IBM Corporation 36