Upload
ilkka-anttonen
View
218
Download
0
Embed Size (px)
DESCRIPTION
Presentation about opinionated Docker based platform as a service (PAAS) with capabilities including Service Registration & Discovery, Log aggregation and running multiple processes in one container.
Citation preview
Media Partner
Organizers
Top Media Partner
General Partner
Supporters
Ilkka AnttonenAccenture
Opinionated PAAS on DockerOctober 17, 2014
Getting the demo• The demo can be checked out from https://
github.com/SirIle/openslava-docker• README.md file contains instructions to get it
running
About me• Started with basic and assembler on C64
almost exactly 30 years ago• 16 years work experience• Emerging Technology Nordics Lead at
Accenture
• I can be contacted at [email protected]
Ask questions or rate the Speaker
www.sli.do/openslava
Docker• Containers are nothing new, but Docker has popularized them
• CHROOT -> VMware/VBox, XEN, KVM -> Containers, LXC -> Docker• Containers share the same kernel, but run isolated processes on partitioned
resources• Very light and quick to start
• Docker runs (after 0.9) on a multitude of execution environments• libcontainer (default), LXC, OpenVZ, libvirt-lxc, qemu/kvm, Solaris
Zones, chroot...• From a developer perspective Docker
• Allows easily running lots of containers on a single VM• Allows for simulation of large distributed services• Container can be built using CI and shared using a private registry
Approaching PAAS• Running multiple processes in a container
• Supervisord (init.d, upstart, launchd)• Service registration and discovery
• Consul (etcd)• Log aggregation
• Rsyslog + Logstash + ElasticSearch + Kibana• Advanced networking
• Weave (Pipeworks, iptables)• Orchestration
• MaestroNG (Kubernetes, Fleet, Stampede, …)
http://www.mindmeister.com/389671722/docker-ecosystem
Multiple Processes per Container• Running some services (as processes) helps
• SSHD for connectivity and running remote commands • nsenter would also work, but is more complex• You can ssh directly to containers from outside the host if needed• You get security log entries for accesses
• Rsyslog for log aggregation• Consul for service registration, discovery and local DNS service
• Supervisord is simple and does the trick
[supervisord]nodaemon=true
[program:rsyslogd]command=/usr/sbin/rsyslogd -nautorestart=true
[program:sshd]command=/usr/sbin/sshd -Dstdout_logfile=/var/log/supervisor/%(program_name)s.logstderr_logfile=/var/log/supervisor/%(program_name)s.logautorestart=true
[program:consul]command=/bin/bash /usr/local/bin/startConsul.shstdout_logfile=/var/log/supervisor/%(program_name)s.logstderr_logfile=/var/log/supervisor/%(program_name)s.logautorestart=true
[program:cassandra]command=/bin/bash /usr/local/bin/startCassandra.sh
Example of supervisord.conf
Service Registration…• Containers enable quick and easy scaling• Applications in containers should be agnostic about the environment
• Helps with portability and ease of building• Consul provides distributed framework with an agent on every container• Configuring with a file per service or dynamically with API• Supports multiple failure detection methods• Core container is found through container linking for joining
{ "service": { "name": "cassandra", "tags": ["database"], "port": 9160 }}
Example service configuration:
…and Discovery• Support for DNS based discovery which makes things really simple
• Supports also REST API and other record types than A record for DNS for extra information if default ports can be used
• Datacenter aware, routes between datacenters only with explicit requests• List of matching nodes is randomized on each query to provide simple load
balancing• Services and nodes are named with specific syntax:
• <node>.node.<datacenter>.<domain>• [tag].<service>.service[.datacenter].<domain>
Inside the datacenter:ping cassandra.service.consulping cassandra1.node.consulIn a specific datacenter:ping cassandra.service.east.consulping cassandra4.node.east.consul
Example service discovery queries
Log Aggregation• Unless explicitly configured, container file system isn’t preserved
• In case of failures accessing the logs may be difficult• Easy solution is to ship the logs to a central place for storage and analysis• Logstash backed by ElasticSearch is simple yet powerful• One core container per datacenter that stores the logs• Rsyslog used to ship the logs
• Not too much overhead, robust system tolerant of temporary failures
$ModLoad imuxsock # for reading local syslog messages$ModLoad imfile # Load the imfile input module
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat$ActionResumeInterval 1$ActionResumeRetryCount -1
# Cassandra log$InputFileName /var/log/cassandra/system.log$InputFileTag cassandra:$InputFileStateFile state-cassandra$InputRunFileMonitor
# Send everything to a logstash server named 'log.service.consul'*.* @@log.service.consul:5000auth,authpriv.* /var/log/auth.log*.*;auth,authpriv.none -/var/log/syslog
Example of rsyslog configuration
Orchestration• Orchestration handles container lifecycle• Fetches (some systems can also build) containers into local registry• Starts containers taking into account the dependencies and start order• Kubernetes has lots of traction, but currently works well only on GAE• Lots of options, some works across multiple nodes
name: cassandra
ships: vagrant-docker: {ip: 10.10.10.30}
services: core: image: registry.local/core:latest instances: core: ship: vagrant-docker ports: {ssh: '22:1000', elasticsearch: 9200, kibana: 80 , consului: 8500} volumes: /etc/localtime: /etc/localtime /etc/timezone: /etc/timezone lifecycle: running: [{type: tcp, port: consului}] dns: localhost
Example service definition for MaestroNG
Advanced Networking• Docker networking is evolving rapidly as people are building more complex
environments• Basic networking gives changing IP addresses from a configurable range
• In a single node with service registration this is good enough• In multi-node setups greater control of addresses and ranges and the
ability to join the networks between nodes is required• Several projects have emerged to help with advanced networking
• Stampede supports inter-node secure communication, but is very alpha• With pipework and iptables doing complex stuff is possible
• Weave does the same with simplicity replacing the normal docker command
sudo weave launch 10.0.0.1/16# Launch the container sudo weave run 10.0.1.2/22 -t -p 9160:9160 -p 1021:22 \ --dns 127.0.0.1 -h cassandra1 --link core:core \–v /etc/localtime:/etc/localtime:ro \-v /etc/timezone:/etc/timezone:ro -e DC=west -i registry.local/cassandra
Example of container start with Weave networking
Example on a single node• Modified MaestroNG used for orchestration
• Added support for DNS parameter as Consul provides a local DNS server for every container
• YAML based readable configuration• One core container
• Core container is discovered through container linking• LogStash + ElasticSearch• Consul server for the node
• Datastax Opscenter container for monitoring Cassandra instances
• Three Cassandra containers• Node container for running
application to demonstrate Cassandra connectivity
• All containers have Consul agents• Connect to Consul server at
the start• Gossip from there on
Host
<Insert picture>
Node1 (10.10.10.30)
CoreConsul + UILogstash etc
rsyslogsshd (1000)
Cassandra3Consulrsyslog
sshd (1023)Cassandra
Cassandra2Consulrsyslog
sshd (1022)Cassandra
Cassandra1Consulrsyslog
sshd (1021)Cassandra
OpscenterConsulrsyslog
sshd (1121)Opscenter
NodeConsulrsyslog
sshd (1001)
MaestroN
G
Example on two nodes• Weave is used to provide advanced networking between nodes
• Orchestration tools don’t support it yet, so shell scripts used• Private registry used to transfer images, no need to build twice
Host
Node1 / West (10.10.10.30)
10.0.0.1Core
10.0.1.1Consul + UILogstash etc
rsyslogsshd(1000)
Cassandra310.0.1.4Consulrsyslog
sshs(1023)Cassandra
Cassandra210.0.1.3Consulrsyslog
sshd(1022)Cassandra
Cassandra110.0.1.2Consulrsyslog
sshd(1021)Cassandra
Opscenter10.0.1.10
Consulrsyslog
sshd(1121)Opscenter
Node10.0.1.20
Consulrsyslog
sshd(1001)
Node2 / East (10.10.10.40)
10.0.0.2Core
10.0.2.1Consul + UILogstash etc
rsyslogsshd(1000)
Cassandra510.0.2.3Consulrsyslog
sshd(1022)Cassandra
Cassandra410.0.2.2Consulrsyslog
sshd(1021)Cassandra
Registry (10.10.10.100)
Q & AThank you for listening!
Questions?