Datadog• Monitoring service
• Made for the cloud
• Aggregates everything
• Support for Docker (since 1.0)
Agenda1.A (very) brief history of containers
2.Operational complexity
3.Monitoring Docker effectively
4.Demo
Containers in a nutshell• Been around for a long time
– jails, zones, cgroups
• No full-virtualization overhead
• Used for runtime isolation (e.g. jails)
• Docker is an Escape from Dependency Hell
Mini-host or über-process?
Process Container Host
Spec Source Dockerfile Kickstart
On disk .TEXT /var/lib/docker /
In memory PID Container ID Hostname
In the network Socket veth* eth*
Runtime
context
server core host data center
Combinatorial multiplication
Hardware
OS
Off-the-shelf
Your Application
Hardware
Hypervisor
Off-the-
shelf
App
OS OS
Off-the-
shelf
App
Hardware
Hypervisor
OS OS
A A A A
Containers
O O O O
Operational complexity• Average containers per host: N (N=5, 10/2014)
• N-times as many “hosts” to manage
• Affects
– provisioning: prep’ing & building containers
– configuration: passing config to containers
– orchestration: deciding where/when containers
run
– monitoring: making sure containers run
properly
Number of things to measure• 1 Amazon EC2 instance
– 10 CloudWatch metrics
• 1 operating system (e.g. linux)
– 100 metrics
•N containers
– 100*N metrics
•110 + 100*N metrics per instance
Combinatorial multiplication
160 610metrics
per hostmetrics
per host
Assuming only 5 containers per
instance
Aggravating factors• Registry-based provisioning
– new images as fast as you can git commit
• Autonomic orchestration
– from imperative to declarative
– automated
– individual containers don’t matter
– e.g. kubernetes, mesos
Layers of monitoring
CloudWatch
Infrastructure
Monitoring
APM
Hypervisor
OS OS
A A A A
Containers
O O O O
Layers of monitoring
cpu/net/io
filesystem
docker mem
docker cpu
db queries
web requests
app throughput
CloudWatch
Infrastructure
Monitoring
APM
e.g
.
Hypervisor
OS OS
A A A A
Containers
O O O O
Layers of monitoring• Access to metrics from all the layers
• Amazon CloudWatch, OS metrics, Docker metrics,
app metrics in 1 place
• Shared timeline
Tags• Monitoring is like Auto-Scaling Groups
• Monitoring is like Docker orchestration
• From imperative to declarative
• Query-based
• Queries operate on tags
Monitoring with tags and queries
“Monitor all Docker containers running image web”
“… in region us-west-2 across all availability zones”
“… and make sure resident set size < 1GB on c3.xl”
Monitoring with tags and queries
“Monitor all Docker containers running image web”
“… in region us-west-2 across all availability zones”
“… and make sure resident set size < 1GB on c3.xl”
Monitoring with tags and queries
“Monitor all Docker containers running image web”
“… in region us-west-2 across all availability zones”
“… that use more than 1.5x the average on c3.xl”