Powering Microservices with Docker, Kubernetes, Kafka, & MongoDB

Andrew Morgan – MongoDB Product Marketing

POWERING MICROSERVICES

WITH DOCKER,

KUBERNETES, KAFKA, &

MONGODB

andrewmorgan

AGENDA

Microservice

s

What, Why,

How?

Containers

Docker, Kafka

Orchestratio

n

Kubernetes,

Mesos, …

MongoDB

Why, How?

When to use Use cases

Who, Why?

1 2 3 4 5 6

MICROSERVICES

WHY USE MICROSERVICES? (TL;DRWEBSCALE)

Speed Change Maintain Scale Empower

Build MVP quickly Rapid iterations Simple components Product Team == Component

React to market Isolated impact Team Committees

WHY USE MICROSERVICES? (TL;DRWEBSCALE)

Speed Change Maintain Scale Empower

Build MVP quickly Rapid iterations Simple components Product Team == Component

React to market Isolated impact Team Committees

WHY USE MICROSERVICES? (TL;DRWEBSCALE)

Speed Change Maintain Scale Empower

Build MVP quickly Rapid iterations Simple components Product Team == Component

React to market Isolated impact Team Committees

WHY USE MICROSERVICES? (TL;DRWEBSCALE)

Speed Change Maintain Scale Empower

Build MVP quickly Rapid iterations Simple components Product Team == Component

React to market Isolated impact Team Committees

WHY USE MICROSERVICES? (TL;DRWEBSCALE)

Speed Change Maintain Scale Empower

Build MVP quickly Rapid iterations Simple components Product Team == Component

React to market Isolated impact Team Committees

WHY USE MICROSERVICES? (TL;DRWEBSCALE)

Speed Change Maintain Scale Empower

Build MVP quickly Rapid iterations Simple components Product Team == Component

React to market Isolated impact Team Committees

Monolithic

Cross-Team

Tight Coupling

Small Change

Huge Impact

System Retests

Microservices

Decoupled

Independent Dev

Isolated Impact

MICROSERVICES EXAMPLE

Twitter

IngestGoogle+

Ingest

Snapcha

t Ingest

Feed

merge

Faceboo

k Ingest

MICROSERVICES EXAMPLE

Twitter

Ingest

Snapcha

t Ingest

Feed

merge

Faceboo

k Ingest

MICROSERVICES EXAMPLE

Twitter

Ingest

Snapcha

t Ingest

Feed

merge

Faceboo

k Ingest

MICROSERVICES EXAMPLE

Twitter

Ingest

Snapcha

t Ingest

Feed

merge

Faceboo

k Ingest

Whatsap

p Ingest

MICROSERVICES EXAMPLE

Twitter

Ingest

Snapcha

t Ingest

Feed

merge

Faceboo

k Ingest

Whatsap

p Ingest

Snapcha

t IngestSnapcha

t Ingest

DEVELOPMENT TEAMS

CONTAINERS

CONTAINERS – POWERING MICROSERVICES

Real world shipping containers

• Road, rail & sea

• Contents untouched

• Ubiquitous & standardized

• Simple

• Contents protected

• Constraints

CONTAINERS – POWERING MICROSERVICES

Software containers

• 1 image -> Many containers‒ Laptop, DC, cloud

‒ Dev, QA, production, support

• Simple, efficient

• Isolation

• Constraints

VMS VS. CONTAINERS

VM VMVM

Bare Metal

Host Operating System

Hypervisor

Guest OS

Libraries

Apps

Service

Guest OS

Libraries

Apps

Service

Guest OS

Libraries

Apps

Service

Container ContainerContainer

Bare Metal

Host Operating System

Docker Engine

Libraries

Libraries

Apps

Libraries

Apps

Service ServiceService

VMS VS. CONTAINERS

VM VMVM

Bare Metal

Host Operating System

Hypervisor

Guest OS

Libraries

Apps

Service

Guest OS

Libraries

Apps

Service

Guest OS

Libraries

Apps

Service

Container ContainerContainer

Bare Metal

Host Operating System

Docker Engine

Libraries

Libraries

Apps

Libraries

Apps

Service ServiceService

VMS VS. CONTAINERS

VM VMVM

Bare Metal

Host Operating System

Hypervisor

Guest OS

Libraries

Apps

Service

Guest OS

Libraries

Apps

Service

Guest OS

Libraries

Apps

Service

Container ContainerContainer

Bare Metal

Host Operating System

Docker Engine

Libraries

Libraries

Apps

Libraries

Apps

Service ServiceService

DOCKER

• Simple to use

• 100K+ images on Docker Hub

• Build images from images

• Platforms‒ Linux, OS X, Windows

‒ Laptop, VM, Cloud,…

‒ Cloud services

RUN MONGODB

docker run -d mongo

TITLE ONLY

TITLE ONLY

MICROSERVICE ARCHITECTURES BUILT ON CONTAINERS

Many small, focused containers ->

sophisticated services

• Well defined APIs

• Independent languages &

libraries

• Modular: easy maintenance +

reuse

• Fault tolerant

• Scalable

CONNECTING THE MICROSERVICES –KAFKA

Producer

9

8

7

123...

Topic A

Consumer

CONNECTING THE MICROSERVICES –KAFKA

Producer

9

8

7

123...

Topic A

Consumer

Producer Consumer

CONNECTING THE MICROSERVICES –KAFKA

Producer

9

8

7

123...

Partition 0

Topic A

Consumer

Producer Consumer

4

3

5

123...

Partition 1

CONNECTING THE MICROSERVICES –KAFKA

Producer

LEADER

Topic A / Partition 0

Broker 1

FOLLOWER

Topic A / Partition 1

FOLLOWER

Topic A / Partition 0

Broker 2

LEADER

Topic A / Partition 1

CONNECTING THE MICROSERVICES –KAFKA

Producer

Producer

Producer

9

8

7

123...

Partition 0

4

3

5

123...

Partition 1

7

3

2

123...

Partition N

Topic A

Topic B

7

6

5

123...

Partition 0

New Old

Consumer

Consumer

ORCHESTRATION

ORCHESTRATION

Automated deployment, connecting, and maintenance of multiple containers

• Provision hosts

• Containers‒ Instantiate

‒ Reschedule

‒ Link

‒ Scale Out/In

• Expose services

KUBERNETES

Created by Google, feature-rich and widely adopted

• Deployment and ‘replication’

• On-line scale out/in

• Rolling upgrades

• High Availability

• Persistence

• Ports

• Load balancing

• Google Compute Engine

APACHE MESOS

10,000s of physical servers; used by

Twitter, Airbnb & Apple

• Code (“frameworks) vs.

declarative

• Less feature rich than Kubernetes

• Kubernetes as a Mesos

framework

• Foundation for distributed systems‒ Apache Aurora, Chronos, Marathon

CHOOSING AN ORCHESTRATION FRAMEWORK

• What you have:‒ Skills?

‒ DevOps frameworks?

‒ Number of hosts?

‒ Bare metal, VMs, or cloud?

• Lifecycle

• Features‒ Automated High Availability?

‒ Grouping and load balancing?

‒ As a service?

MONGODB

WHY MONGODB IS A GOOD FIT FOR MICROSERVICES

Monitoring

&

Automation

Flexible Data

Model

Redundancy Scalability Simplicity

WHY MONGODB IS A GOOD FIT FOR MICROSERVICES

Monitoring

&

Automation

Flexible Data

Model

Redundancy Scalability Simplicity

WHY MONGODB IS A GOOD FIT FOR MICROSERVICES

Monitoring

&

Automation

Flexible Data

Model

Redundancy Scalability Simplicity

WHY MONGODB IS A GOOD FIT FOR MICROSERVICES

Monitoring

&

Automation

Flexible Data

Model

Redundancy Scalability Simplicity

WHY MONGODB IS A GOOD FIT FOR MICROSERVICES

Monitoring

&

Automation

Flexible Data

Model

Redundancy Scalability Simplicity

WHY MONGODB IS A GOOD FIT FOR MICROSERVICES

Monitoring

&

Automation

Flexible Data

Model

Redundancy Scalability Simplicity

HOW TO USE MONGODB WITH CONTAINERS

Twitter

Ingest

Snapch

at Ingest

Feed

merge

Faceboo

k Ingest

Whatsap

p Ingest

Snapch

at IngestSnapch

at Ingest

MongoDB Atlas

HOW TO USE MONGODB WITH CONTAINERS

Twitter

Ingest

Snapch

at Ingest

Feed

merge

Faceboo

k Ingest

Whatsap

p Ingest

Snapch

at IngestSnapch

at Ingest

Kubernetes

Ops Mgr

agent

Ops Mgr

agent

Ops Mgr

agent

HOW TO USE MONGODB WITH CONTAINERS

Twitter

Ingest

Snapch

at Ingest

Feed

merge

Faceboo

k Ingest

Whatsap

p Ingest

Snapch

at IngestSnapch

at Ingest

Kubernetes

mongod

mongod

mongod

ORCHESTRATING MONGODB USING KUBERNETES

Distributed, stateful application

• Persistent volumes

• External IP addresses for internal

comms

• Init MongoDB replica set

• Monitor

• Backup

Volumename:

mongo-persistent-storage1

pdName: mongodb-disk1

mongodb

-disk1

Containername: mongo-node1

image: mongo

command: mongod –replSet my_replica_set

containerPort: 27017

volumeMounts:

name: mongo-persistent-storage1

mountPath: /data/db

DockerHubRegistry

Podlabels: [name: mongo-node1; instance: rod]

ReplicationControllername: mongo-rc1

labels: [name: mongo-rc]

replicas: 1

selector: [name: mongo-node1]

Service:LoadBalancername: mongo-svc-a

labels: [name: mongo-svc-a]

ports: [port: 27017, targetPort: 27017]]

selector: [name: mongo-node, instance: rod]ExternalIP

Address

104.1.1.1

App

104.1.1.1:27017

ReplicationControlername: mongo-rc-europe

labels: [name: mongo-europe]

replicas: 1

selector: [name: mongo-node]

Volumename:

mongo-persistent-storage

pdName: mongodb-disk-europe

mongodb-

disk-europe

Containername: mongo-node

image: mongo

command: mongod –replSet my_replica_set

containerPort: 27017

volumeMounts:

name: mongo-persistent-storage

mountPath: /data/db

Podlabels: [name: mongo-node]

Service:LoadBalancername: mongo-svc-europe

labels: [name: mongo-svc-europe]

ports: [port: 27017, targetPort: 27017]]

selector: [name: mongo-node]

ReplicationControlername: mongo-rc-asia

labels: [name: mongo-asia]

replicas: 1

selector: [name: mongo-node]

Volumename:

mongo-persistent-storage

pdName: mongodb-disk-asia

mongodb-

disk-asia

Containername: mongo-node

image: mongo

command: mongod –replSet my_replica_set

containerPort: 27017

volumeMounts:

name: mongo-persistent-storage

mountPath: /data/db

Podlabels: [name: mongo-node]

Service:LoadBalancername: mongo-svc-asia

labels: [name: mongo-svc-asia]

ports: [port: 27017, targetPort: 27017]]

selector: [name: mongo-node]

ReplicationControlername: mongo-rc-us

labels: [name: mongo-us]

replicas: 1

selector: [name: mongo-node]

Volumename:

mongo-persistent-storage

pdName: mongodb-disk-us

mongodb-

disk-us

Containername: mongo-node

image: mongo

command: mongod –replSet my_replica_set

containerPort: 27017

volumeMounts:

name: mongo-persistent-storage

mountPath: /data/db

Podlabels: [name: mongo-node]

Service:LoadBalancername: mongo-svc-us

labels: [name: mongo-svc-us]

ports: [port: 27017, targetPort: 27017]]

selector: [name: mongo-node]

STATEFULSETS

Beta in Kubernetes 1.5/6

• Stable, predictable, unique network identifiers.‒ IP addresses may change

• Stable, persistent storage

• Ordered, graceful deployment and scaling (0 N-1)

• Ordered, graceful deletion and termination (N-1 0)

PUTTING IT INTO PRACTICE

WHEN TO USE

MICROSERVICES

WHEN TO USE MICROSERVICES

USE CASES

MONGODB & MICROSERVICES IN THE WILD

REFERENCES

• Enabling Microservices – Containers & Orchestration Explained

https://www.mongodb.com/collateral/microservices-containers-and-orchestration-explained

• Microservices: The Evolution of Building Modern Applications

https://www.mongodb.com/collateral/microservices-the-evolution-of-building-modern-

applications

• Data Streaming with Apache Kafka & MongoDB

https://www.mongodb.com/collateral/data-streaming-with-apache-kafka-and-mongodb

PUTTING IT INTO PRACTICE