The Enterprise Kafka Summit 2016

Rimas Silkaitis

how Heroku leveraged Kafka to realize the original vision of the enterprise service bus (esb)

Rimas Silkaitis

Product

Heroku Postgres Heroku Redis

What is an ESB?

Analyst Definitions

ESBs receive service consumption requests, route the requests to the correct service provider, transform the requests to a format compatible with the service provider, wait for the results, and deliver them back to the service consumer.

IDChttp://www.idc.com/2010st/esbcm.html

Uses a bus architecture for scalability and reliability, establishes a unified communications channel, provides integration, mediation and communication services and is standards based.

http://redmonk.com/public/OpenSourceESBs.pdf

Redmonk

Vendor Definitions

Oracle Service Bus transforms complex and brittle architectures into agile integration networks by connecting, virtualizing, and managing interactions between services and applications.

http://www.oracle.com/technetwork/middleware/service-bus/overview/index.html

Oracle

Mule ESB is a lightweight Java-based enterprise service bus (ESB) and integration platform that allows developers to connect applications together quickly and easily, enabling them to exchange data.

https://www.mulesoft.com/resources/esb/what-mule-esb

Mulesoft

How should we define it?

¯\_(ツ)_/¯

App

App

App

App

Finance

HR

BizOps

Inventory

ESBRouting

Mediation

Messaging

EventProcessing

ServiceOrchestration

ESB

Push

Centralized

Architecture

API

CLI GIT Dashboard

Data Runtime BizOps

$ heroku addons:create heroku-postgresql:standard-0 -a sushiCreating postgresql-reticulated-74861... done, ($50.00/month)Adding postgresql-reticulated-74861 to neovintage... doneSetting HEROKU_POSTGRESQL_GRAY_URL and restarting sushi... done, v249The database should be available in 3-5 minutes. ! The database will be empty. If upgrading, you can transfer ! data from another database with pg:copy.Use `heroku pg:wait` to track statusUse `heroku addons:docs heroku-postgresql` to view documentation.

API

CLI GIT Dashboard

Data Runtime BizOps

API

CLI GIT Web

Data Runtime BizOps

THE API IS OUR ESB!

Problems Emerge

• Bottleneck in getting features released that touch API

• Difficult to get Analytics across the entire Org

• Hiring has become difficult

• API is down. So is the entire platform.

PUSH vs PULLhttp://kafka.apache.org/documentation.html#design_pull

PUSH

PULLProducers Consumers

ESB Redefinition

• Pull over Push

• Consumers dictate their own consumption rate

• Independent development, scaling and scheduling

• Stop the world processing doesn’t exist

CLI GIT Dashboard

API Data Runtime BizOps

Other Use Cases Around Heroku

Operational Experience

aggregation

router

aggregation

router

metrics-ingest

router

metrics-ingest

router

ZK postgres

metrics-ingest

router

aggregation

router

aggregation

router

aggregation

router

sink

router

aggregation

router

aggregation

router

metrics-api

router

$ heroku kafka:list -a metaas-addons=== Kafka Topics on KAFKA_URL

Name Messages Traffic─────────────────────────────────── ──────────────

─────────────ROUTER_REQUEST.v1 132,823.12/sec 14.6 mb/sec

DYNO_MEMORY.v1 1,478.97/sec 187 kb/secDYNO_LOAD.v1 1,468.01/sec 153 kb/sec

ROUTER_STATUS_SUMMARY.v1 987.4/sec 28.3 kb/secSYSLOG.v1 691.9/sec 76.8 kb/sec

ROUTER_LATENCY_SUMMARY.v1 678.84/sec 37.5 kb/secROUTER_ERROR.v1 476.12/sec 93.9 kb/sec

DYNO_ERROR.v1 354.06/sec 38.5 kb/secDYNO_LOAD_SUMMARY.v1 241.33/sec 9.73 kb/sec

DYNO_MEMORY_SUMMARY.v1 219.26/sec 25.1 kb/secDYNO_ERROR_SUMMARY.v1 73.49/sec 1.83 kb/sec

ROUTER_ERROR_SUMMARY.v1 12.35/sec 276 bytes/sec__consumer_offsets 4.1/sec 405 bytes/sec

ROUTER_STATUS_PARTIAL_SUMMARY.v1 0.0/sec 0 bytes/secROUTER_LATENCY_PARTIAL_HISTOGRAM.v1 0.0/sec 0 bytes/sec

ROUTER_REQUESTS.v1 0.0/sec 0 bytes/sec

What do we need to do next?

Schema Registry

Security

as a Service

Creating kafka-lively-27896... done, (free)Adding kafka-lively-27896 to sushi... doneKafka cluster has been created and will be available shortlyUse `heroku addons:docs heroku-kafka` to view documentation.

=== KAFKA_URLName: kafka-silhouetted-79119Created: 2016-03-24 21:21 UTCPlan: Beta 3Status: availableVersion: 0.9.0.0Topics: 0 topics (see heroku kafka:list)Connections: 0 consumers (0 applications)Messages: 0.0 messages/secTraffic: 0 Bytes/sec in / 0 Bytes/s out

$ heroku kafka:info -a sushi

$ heroku addons:create heroku-kafka:beta-3 -a sushi

Public Betahttps://heroku.com/kafka

Come See Us At Our BoothWe’re Hiring!