Containerizing Couchbase with Swarm & Kubernetes – Couchbase Connect 2016

©2016 Couchbase Inc. 1

Containerizing Couchbase with Swarm and Kubernetes


Eric CooperManager, Test [email protected]

IMAGE GOES HERE


Tommie McAfeeSenior Engineer, Test [email protected]

IMAGE GOES HERE

©2016 Couchbase Inc.©2016 Couchbase Inc.

Agenda

• Containers and their popularity • Orchestration Frameworks • Functional Testing at Couchbase with Docker• Interfacing with Docker • System testing with Sequoia Framework and Swarm •Demo

• Orchestrating Couchbase with Kubernetes • System testing with Swarm

• Best practices

4


Containers and their popularity

•Customizable images • e.g. the same Couchbase Server build and OS, mix and match• Include all necessary libraries in the image

•One solution from Dev -> production• Deploy the same image in dev, preproduction and production across all platforms

• Packageable • can share a container for troubleshooting

• Lightweight • time to bring up a container with Couchbase• “clean slate”

• Popular• 500K+ Couchbase pulls from Dockerhub

5


Orchestration and their frameworks

• Aggregation• Deploy apps implemented with multiple containers in pods

• Scaling• Increased the replication count across multiple servers• Computing resources per container

•Robustness• The ability to restart failing containers and hosts

• Services• Service discovery load balancing

6


Orchestration and Couchbase Test Environments


Functional Testing at Couchbase with Docker

• Couchbase Server Functional Testing Overview• Multiple test suites per component (e.g. Views, XDCR, N1QL etc.)• Each test suite:

• Contains 5 to 50 test cases• May use between 1 and 8 Couchbase Servers• Runs from 30 minutes to 10 hours

• Total of 150 test suites across 15 components• All regression suites runs in 12 hours

8


Test Infrastructure - historical

Test client - takes as input a test suite

descriptor and cluster config

(Python)

Couchbase Server 1

Couchbase Server n

REST/CLI/memcached

VM

VM

Jenkins Job


Challenges

• A multitude of Jenkins jobs (one per test suite)• Ordering is difficult• Test suite and cluster configuration implicitly stored in Jenkins jobs

• Dedicated VMs• Hard to maintain - if a VM is replaced with a new IP many Jenkins jobs must be edited• Fixed OS - testing with a different OS is cumbersome


Docker solution

11

Test Suite Descriptors (Couchbase)

Test Suite Dispatcher

Test Suite Executor

Jenkins Job (multiple instances dependent on hardware capacity)

Couchbase Server 1 on Docker

Test Client

Test Suite N

Couchbase Server n on Docker

Couchbase Server 1 on Docker

Test Client

Test Suite 1

Couchbase Server n on Docker


Docker based Jenkins Test Executor


Couchbase Functional Test Suites and Kubernetes (Work in progress)

• Test suites are heterogeneous - differing number of Couchbase servers depending on the suite• Test suite runs are not persistent - when they are done the containers are not reused• Test client and Couchbase servers pod configuration:

• in 1 pod, or • spread across pods?• Pods per common cluster configs?

• Kubernetes Jobs may be a solution:• Job per test suites? Auto generated• Control parallelism?


Interfacing With DockerContainer Based Testing


Requirements

15

Topology Changes

Data Loading

Query

• Building a framework for system testing

• JSON and binary documents with CRUD operations

• Map/Reduce, N1QL, Full Text Search, with index operations

• Add and remove servers, rebalance and failover cluster


The Good News

16

Topology Changes

Data Loading

Query

• No problem...we already have tools to perform these operations

• pillowfight, cbdocloader, spring, gideon

• cbq, rest_client, vegetta

• Couchbase_cli, rest_client


The Good News

17

Topology Changes

Data Loading

Query

• No problem...we already have tools to perform these operations

Package Together and we’re done!


The Bad News

18

Topology Changes

Data Loading

Query

• Problem...tools are written if different languages

GOC

PythonJava

Bash


The Docker News

19

Topology Changes

Data Loading

Query

• That’s fine...free containers for everyone!

GOC

PythonJava

Bash


The Docker News

20

Topology Changes

Data Loading

Query

• From Implementing to Interfacing

GOC

PythonJava

Bash

INTERFACE


The Docker News

21

• For Example

Topology Changes

Data Loading

Query

TEST

• run pillowfight -U 172.23.100.10

• run cbq -script=’create primary index on default’

• run couchbase-cli rebalance -c 172.23.100.10


More Requirements

22

• What if I want to test a different cluster?

Topology Changes

Data Loading

Query

TEST





More Requirements

23

• Just change IP’s?

Topology Changes

Data Loading

Query

TEST





More Requirements

24

• Scope Abstraction

SCOPE

Topology Changes

Data Loading

Query

TEST


• run couchbase-cli rebalance -c {{.Orchestrator}}

• run pillowfight -U {{.Orchestrator}}

Orchestrator: 10.1.2.10


More Requirements

25

• What if I wanted to test against a cluster with 100 vms?

SCOPE

Topology Changes

Data Loading

Query

TEST



• run pillowfight -U {{.NthDataNode 70}}

…DataNode[0]: 10.1.2.10 QueryNode[0]: 10.1.2.11 DataNode[98]: 10.1.2.110


Houston

26

• How about 100 containers?

SCOPE

Topology Changes

Data Loading

Query

TEST




…DataNode[0]: ??? QueryNode[0]: ??? DataNode[98]: ???


Solution

27

• Provider Abstraction

SCOPE

Topology Changes

Data Loading

Query

TEST




PROVIDER


TADA!

28

• Sequoia Framework

Topology Changes

Data Loading

Query

• Container Pluggability = More Tests + Less Coding• Scope Portability = Define Once + Run Anywhere• Resource Scalability = Your Laptop || The Lab


Testing With DockerUsing Sequoia Framework


Test expression

• Tests are a series of docker commands

30

• docker run cbq -script=’create primary index on default’

• docker run couchbase-cli rebalance -c 172.10.10.2

• docker run pillowfight -U 172.10.10.2


Test expression

• With sequoia we express this using YAML files

31


Altogether

• Scale it up!

32


Altogether

•With concurrency

33


Scaling With SwarmDeploying Containers As a Service With Swarm


• New Swarm Provider • Containers are encapsulated in services (networking, multi-host)

Scaling With Swarm

35

DOCKER SWARM

containers services


•Spread Strategy• Default behavior is to evenly distribute containers across couchbase servers

Scaling With Swarm

36

Manager 1

Manager 2

Manager 3

CB CLCB

Manager 4

CB CLCB CB CLCB CB CLCB


•Spread Strategy• Default behavior is to evenly distribute containers across couchbase servers

Scaling With Swarm

37

Manager 1

Manager 2

Manager 3

CB CLCB

Manager 4

CB CLCB CB CLCB CB CLCB


•Adding Constraints• Apply Labels for workload isolation

Scaling With Swarm

38

Manager 1

Manager 2

Manager 3

CB CB

Manager 4

CB CB CB CB CLCL

CL CL

CB CB CB

Server Server Server Client

• ‘docker node update --label-add zone=client manager-4’



Scaling With Swarm

39

Manager 1

Manager 2

Manager 3

D D

Manager 4

D D D D CLCL

CL CL

I I Q




Scaling With Swarm

40

Manager 1

Manager 2

Manager 3

D D

Manager 4

D D D D CLCL

CL CL

I I Q



•Service Level Constraints• We can further isolate nodes by service

Scaling With Swarm

41

Manager 1

Manager 2

Manager 3

D D

Manager 4

I I CLCLQ

CL CL

D

Data Index Query Client

DDD


Big Wins

•Multi-Host Load-Balanced Scalability with Isolation• Container automatically distributed evenly across Swarm nodes• Labeling allows service load balancing• Easy scale out• For FREE! We don’t have to rewrite a single test!

42


Demo

• Kubernetes and Couchbase service • System test with Swarm


Best Practices


Best Practices

• Isolate Isolate!• Consider CPU Pinning

• Set Quotas

• Share the memory

45


Best Practices (continued)

• Be aware of when system calls reflect the host perspective e.g. available RAM, set time• Google and stackoverflow are your friends

46

©2016 Couchbase Inc.

Thank You!

47

Software

Containerizing Couchbase with Swarm & Kubernetes – Couchbase Connect 2016