59
In containers Alban Crequy Testing applications with traffic control HighLoad++ - November 2016 https://goo.gl/knaz6m

Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

  • Upload
    ontico

  • View
    158

  • Download
    4

Embed Size (px)

Citation preview

Page 1: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

In containers

Alban Crequy

Testing applicationswith traffic control

HighLoad++ - November 2016https://goo.gl/knaz6m

Page 2: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Alban Crequy

∘ Worked on traffic control for multimedia applications in cars (tcmmd)

∘ Worked on the container run-time rkt∘ Working on Weave Scope

https://github.com/alban

Page 3: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Berlin-based software company building foundational Linux technologies

Some examples of what we worked on...

OSTreegit for operating system binaries

Page 4: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Who works with...

Page 5: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Find out more about us…

Blog: https://kinvolk.io/blog

Github: https://github.com/kinvolk

Twitter: https://twitter.com/kinvolkio

Email: [email protected]

Page 6: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

∘ What is traffic control and how does it work on Linux∘ How it can be used for testing a microservices application∘ Demo

∘ With Minikube, Kubernetes, Weave Scope, Weave Socks

Plan

Page 7: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

What is traffic control?

How does it work on Linux?

Page 8: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Traffic control, why?

web server client

client

client

THEINTERNET

∘ fair distribution of bandwidth

∘ reserve bandwidth to specific applications

∘ avoid bufferbloat

Page 9: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

∘ Network scheduling algorithm∘ which packet to emit next?∘ when?

∘ Configurable at run-time:∘ /sbin/tc∘ Netlink

∘ Default on new network interfaces: sysctl net.core.default_qdisc

Queuing disciplines(qdisc)

eth0 THE INTERNETqdisc

Page 10: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Stochastic FairnessQueueing (sfq)

eth0

THE INTERNET

FIFO n

FIFO 1

FIFO 0

...

round robin

Page 11: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Traffic control for testing?

Page 12: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Developers & InternetDevelopers usually have good Internet connection

- Low latency- High bandwidth- Not much packet loss or packet corruption

Difficult to see problems

Page 13: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Network emulator(netem)

eth0 THE INTERNETnetem

bandwidth

latency packet loss

corrupt...

Page 14: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Testing with containers

container 1 container 2

eth0eth0

Testing framework

configure “netem” qdiscs:bandwidth, latency, packet drop...

Page 15: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

The demo application

Page 16: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

microservices-demo

https://github.com/microservices-demo/microservices-demo

Page 17: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Some micro-services

front-end Firefox

catalogue

ordersorders-db

payment

Page 18: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Kubernetes

Page 19: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

PodsGroup of container(s) running together, in the same context

Kubernetesnode 1

Kubernetesnode 2

pod

container

container

podcontainer

podcontainer

podcontainer

Each pod has

- Its own network namespace- Its own IP address

Page 20: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Replica sets- Control the number of “Pods” replicas running- Deploy pods

Kubernetesnode 1

Kubernetesnode 2

podcatalogue

podfront-end

podorders

In this example:

- 2x front-end- 1x catalogue- 1x orders

podfront-end

Page 21: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Replica set“catalogue”

ServicesDirecting the traffic to “Pods”

- Cluster IP- DNAT catalogue1

catalogue2

catalogue3

Service“catalogue”

front-end

Page 22: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Testing with traffic control in Kubernetes

Kubernetesnode 1

pod

pod

Kubernetesnode 2

pod

pod

tc tccontrols∘ Latency∘ Bandwidth∘ Packet drop

∘ configure network simulator

∘ play scenarios

Page 23: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Weave Scope

Page 24: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Weave Scope

Page 25: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Testing with Weave Scope

Kubernetesnode 1

tcScopeProbe

pod pod

pod pod

KubernetesNode 2

tcScopeProbe

pod pod

pod pod

ScopeApp

Page 26: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

<Demo>

Reproduce the demo yourself: https://github.com/kinvolk/demo

Page 27: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
Page 28: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
Page 29: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
Page 30: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
Page 31: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
Page 32: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
Page 33: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
Page 34: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
Page 35: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
Page 36: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
Page 37: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
Page 38: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
Page 39: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
Page 40: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
Page 41: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
Page 42: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
Page 43: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
Page 44: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
Page 45: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
Page 46: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
Page 47: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

</Demo>

Reproduce the demo yourself: https://github.com/kinvolk/demo

Page 48: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Testing frameworkfor web apps

Selenium

Page 49: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Plugins in Scope

tc pluginScopeProbe

ScopeApp

report

reportcontrol

control

- Unix socket in /var/run/scope/plugins/- Protocols: report and control- Write your own plugins for your testing needs

Page 50: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
Page 51: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Testing more complex scenarios

(my “wishlist”)

Page 52: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Add latency on a specific connection

front-end Firefox

catalogue

ordersorders-db

payment

latency=100ms

Page 53: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

How to define classes of traffic

eth0

netem

interface

latency=100ms

dest_ip=10.0.4.* dest_ip=10.0.5.* other

Page 54: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

u32: filter on contenteth0

HTB

HTB

HTBHTB HTB

netemnetem netem

interface

root qdisc (type = HTB)

root class (type = HTB)

leaf qdiscs (type = netem)

leaf classes (type = HTB)

filters (type=u32)

otherip=10.0.5.*ip=10.0.4.*

latency=10ms

Page 55: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Filtering with cBPF/eBPF

eth0

BPF

netemnetem

kernel

userspace

BPF_JMP...BPF_LD...BPF_RET...

if (skb->protocol…) return TC_H_MAKE(TC_H_ROOT, mark); compilation

clang... -march=bpf

uploadin the kernel:

- bpf()- Netlink

x86_64 codeJIT compilation

Page 56: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

eBPF maps

eth0

BPF

netemnetem

kernel

userspace

x86_64 code

eBPF map

tc

∘ Build statistics∘ Make them available to

the testing framework

Page 57: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

- Another Weave Scope plugin could handlemore filters: u32 or eBPF

Advanced tc plugin

Anotherplugin

ScopeProbe

ScopeApp

control

controleth0

u32,eBPF, ...

Page 58: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

The EndTry the demos yourself: https://github.com/kinvolk/demo

Weave Scope’s traffic control plugin: https://github.com/weaveworks-plugins/scope-traffic-control

The slides: https://goo.gl/knaz6m

Page 59: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)

Questions?