Upload
n-masahiro
View
9.687
Download
0
Embed Size (px)
Citation preview
andDocker meetup Tokyo #5 Aug 19, 2015
Who are you?
• Masahiro Nakagawa • github: @repeatedly
• Treasure Data Inc. • Fluentd / td-agent developer • https://jobs.lever.co/treasure-data
• I love OSS :) • D Language, MessagePack, The organizer of several meetups, etc…
Logging for containers
• Storage: • should be outside of containers / hosts
• Transferring: • should be over network
• Aggregation: • should be done per container / per service
Real world…
Container-based world needs unified logging layer
Structured logging
Reliable forwarding
Pluggable architecturehttp://fluentd.org/
What’s Fluentd?• Data collector for unified logging layer
• Streaming data transfer based on JSON • Written in Ruby
• Gem based various plugins • http://www.fluentd.org/plugins
• Working on lots of productions • http://www.fluentd.org/testimonials
Before…
After!
M x N → M + NNagios
MongoDB
Hadoop
Alerting
Amazon S3
Analysis
Archiving
MySQL
Apache
Frontend
Access logs
syslogd
App logs
System logs
Backend
Databasesbuffering / processing /
Core Plugins• Divide & Conquer
• Buffering & Retrying
• Error handling
• Message routing
• Parallelism
• Read data
• Parse data
• Filter data
• Format data
• Write data
Architecture
EngineInput
Filter Output
Buffer
> grep > record_transfomer > …
> Forward > File tail > ...
> Forward > File > ...
Output> File > Memory
not pluggable
FormatterParser
Aggregation patterns1) 1-level aggregation 2) 2-level aggregation
Pros:
• simple configuration
Cons:
• fixed aggr, address
• many connections
• high load for aggr
Pros:
• less connections
• lower load for aggr
Cons:
• more containers
Aggregation patterns1) 1-level aggregation 2) 2-level aggregation
Apps sends logs over fluent-logger
or …
There are some patterns - via udp / tcp - tail file - etc…
2-level aggr w/ Fluentd
1. Transfer logs over network
2. Container logs & tail generated log files
1) Network transferring• Apps sends logs:
• using fluent-logger-* • over inter-container network
• Pros: • no performance penalty
• Cons: • docker logs is out of scope • a bit complex for logging
over TCP
app
2) Tail container logs• Apps write logs to
STDOUT • docker writes it to log file • fluentd container reads its file
• Pros: • simple conf for apps & docker • logs include container logs
• Cons: • in_tail performance penalty &
a bit complex configuration
log file
app
http://www.fluentd.org/guides/recipes/docker-logging
Seems good? But we now have smarter way!
Logging drivers• New from docker v1.6
• We can get docker logs directly via drivers
• “fluentd" driver is coming from v1.8 • enabled by “—log-driver=fluentd” • contributed by @tagomoris
https://github.com/docker/docker/pull/12876
• https://docs.docker.com/reference/logging/fluentd/
3) Logging driver• Apps write logs to STDOUT
• Docker sends its logs to fluentd directly
• Pros: • simple conf for apps & docker • Logs includes container logs
• Cons: • Do you see any concerns?
over TCP
app
Demo?
Fluentd docker image
• Official image by fluent organization • Main maintainer is @tagomoris • https://registry.hub.docker.com/u/fluent/fluentd/
• Or build your container! • https://github.com/fluent/fluentd-docker-image
With Kubernetes
• Kubernetes provides fluentd + Elasticsearch + Kibana image officially • https://hub.docker.com/r/kubernetes/fluentd-
elasticsearch/
• http://www.dasblinkenlichten.com/logging-in-kubernetes-with-fluentd-and-elasticsearch/
Cloud service for entire data pipeline!