Upload
john-willis
View
35
Download
1
Embed Size (px)
Citation preview
About Me
• One of the founding members of the “Devops” movement. • Author of the “Devops Handbook”. • Author of the “Introduction to Devops” on Linux
Foundation edX. • Podcaster at devopscafe.org • Devops Enterprise Summit - Cofounder • Found of Socketplane (Acquired by Docker) • Formally Director of Devops at Dell • Formally Director at Chef • 10 Startups over 25 years
2
Github: botchagalupe/my-presentationsTwitter: botchagalupeWechat: botchagalupe
Devops Practices and Patterns• Continuous Delivery
• Everything in version control • Small batch principle • Trunk based deployments • Manage flow (WIP) • Automate everything
• Culture • Everyone is responsible • Done means released • Stop the line when it breaks • Remove silos
3itrevolution.com/devops-handbookhttp://www.bookdepository.com/
A commercial product,
built on a development platform,
built on infrastructure,
built on standards.
Docker is building a stack to program the Internet
8
Docker Project Sponsor •Primary authors, contributor maintainer •6B+ Downloads, 3000+ Contributors, 500,000+ Applications •100’s of ecosystem partners •Millions of developers use Docker. Millions of servers run Docker Commercial Docker Solutions •Integrated solutions to build, ship, run Docker at scale
• Orchestration, registry, security, workflow, control plane • CaaS (containers as a service)
•Official providers of commercial technical support •10K’s cloud customers, 300+ F500 customers
About Docker, Inc.
Gerber, Anna. “The State of Containers and the Docker Ecosystem: 2015” O’Reilly, September 2015
Docker users already running in production
60%
China is part of Worldwide Docker Community
Meetups in Beijing, Changsha, Chengdu, Chongqing, Dalian, Fuzhou, Guangzhou, Hangzhou, Hong Kong, Nanjing, Qingdao, Shanghai, Shenzen, Suzhou, Tianjin, Wuhan, Xi’an, Xiamen, and Zhuhai
Cloud and Docker Status Update in China
China cloud relative to the West – DB
! China’s cloud spending to reach 24.5% of the West by 2018 from 5.7% in 2015
! More than 80% respondents are already using or plan to use containers, but only 10% used in production environment.
Survey result from 350+ feedbacks by Alibaba Cloud, June 2016
Docker Adoption in China
Other
Government
Sample of Docker CustomersFinancial Services Healthcare
Consumer Education Tech Services
11
Dockerizing applications
12
Legacy App: One container per app. Microservices: one service per container. App comprised of many containers linked together
Enterprises Are Containerizing both Legacy and Microservices Applications
13
80%46% plan to build new microservices in 2016
#4 container workload is traditional databases
1 in 3 have already containerized legacy apps
Will leverage Docker to enable hybrid cloud initiatives
Docker platform standardizes environment AND enables workload portability
3 Paths to Containers Adoption
1Containerize Monoliths Build-Test for CI; Migrate to the Cloud; Get Better CapEx/OpEx than VM
2
3
Containerize Monolith; Transform to Microservices Look for Shared Services to Transform
Enable New Microservices and Apps Greenfield CaaS
Migrate any workload anywhere
Infrastructure agnostic platform • Standardize: Docker abstracts away
the infrastructure and virtualization away from the standardized app containers
• Portability: Containers move without friction from one environment to another – no recoding needed
• Lift and Shift: Containerize legacy and microservices to gain portability
15
Bins/Libs
App
OS
Bins/Libs
App
OS
Bins/Libs
App
OS
Before: VM formats are proprietary
Bins/Libs
App
Bins/Libs
App
Bins/Libs
App
OS
After: Docker abstracts above VMs for portability
Docker Engine
OS
Docker Engine
Infrastructure optimization with Docker
Swisscom • 20:1 VM consolidation ratio • Running the same 400 MongoDB
instances in 400 containers in 20 VMs
• Reduce CapEx and OpEx costs
Leading Energy Company • Containerize legacy apps for
portability • Entire cloud to datacenter site
migration in 5 months • Dramatically accelerated release
process
16
17
Enable Modern App Initiatives with CaaS
Cloud Migration Hybrid Cloud Multi Cloud
Modernizing Apps DevOps CI/CD
DevOps
Cloud DevOpsApps
18
Enterprises Can Decide How To Adopt
Enable CI and DevOps Better Resource Usage Enable Cloud Initiatives
Transform iteratively Ship faster, with better
reliability
Enable developers to self service Enable business transformations
Containerize Build New AppsContainerize & Transform
The Docker ecosystem
Dev Tools
Official Repositories
Operating Systems
Big Data
Service Discovery
Build / Continuous Integration
Configuration ManagementConsulting &Training
Management
Storage
Clustering & Scheduling
Networking
Infrastructure & Service Providers
Storage
Security
Monitoring & Logging
19
Docker and Alibaba Announce Commercial Agreement
• Localized Docker image store and distribution for Docker Hub on Alibaba Cloud
• Alibaba will resell Commercially Supported (CS) Docker Engine and Docker Datacenter, enabling enterprises to manage their production workloads across the entire application lifecycle.
• Alibaba will provide enterprise support options for CS Docker Engine and Docker Datacenter, backed by Docker, Inc.
20
Isolation using Linux kernel featuresnamespaces ● pid ● mnt ● net ● uts ● ipc ● user
cgroups ● memory ● cpu ● blkio ● devices
Docker Engine extensibility and plugins• Built in orchestration expands the opportunity for
the plugin to manage swarm wide vs a single Engine
• Updated architecture standardizes plugin process for ecosystem partners
– Benefits users and vendors – Standardized process of granting plugin permissions – Containerized plugins on roadmap
Docker Engine
Networking
Swarm Mode
Volumes
Plugins
Introducing the best way to orchestrate Docker: Docker.
Docker 1.12: now with orchestration built-in.
Swarm mode
Service API
Node identity
Built-in routing mesh
Docker 1.12: now with orchestration built-in.
Combine your engines in swarms of any scale Self-organizing, self-healing No external data store requiredNo single points of failure Infrastructure-agnostic topology
Swarm mode
Desired state reconciliation ScalingRolling updates Advanced scheduling Application-specified health checks Rescheduling on node failure
Docker Service API
How service deployment works
Declare
ScheduleReconcile
$ docker service create declares the service name, network, port, image:tag and scale
Managers break down service into tasks, schedules them and workers execute tasks
Engines check to see what is running and compared to what was declared to “true up” the environment
API
Allocator
Orchestrator
Scheduler
Dispatcher
R A F T
Manager Node
Worker
ExecutorWorker Node
Accepts command from client and creates service object
Reconciliation loop for service objects and creates tasks
Allocates IP addresses to tasks
Assigns tasks to nodes
Checks in on workers
docker service create
Connects to dispatcher to check on assigned tasks
How service deployment works
Executes the tasks assigned to worker node
Manager ManagerManager
Worker WorkerWorkerWorker Worker
Example service on a swarm
The declarative command describes a new service: • Named Frontend • Made of 5 containers based
on the latest my_frontend_image
• Connected on an overlay network called “my overlay”
• Assigned to port 80
$ docker service create --replicas 5 --name frontend --network myoverlay \
--publish 80/tcp mikegoelzer/my_frontend_image:latest
Manager ManagerManager
Worker WorkerWorkerWorker Worker
Example service on a swarm
This state is what swarm mode and the service deployment API will maintain.
Check to ensure 5 containers are always running for the frontend service$ docker service create --replicas 5 --name frontend
--network myoverlay \
--publish 80/tcp mikegoelzer/my_frontend_image:latest
Manager ManagerManager
Worker WorkerWorkerWorker Worker
Example service on a swarm
This command creates a new service to join an existing overlay network to communicate with frontend
$ docker service create --name redis --network myoverlay redis:latest
Built-in Routing Mesh
Swarm-wide overlay networking Container-native load-balancing DNS-based service discovery No separate cluster to setup Works with your existing load-balancers Rock-solid kernel-only data path with IPVS
Routing mesh for application services • Container-aware dynamic
load balancing • Assign ports to service that
do not change • Built in load balancing into
the Engine • Automatic service discovery
Worker
Load Balancing
Worker
Load Balancing
Worker
Load Balancing
Port 1000Port 1000 Port 1000
Manager ManagerManager
Docker user assigns a global port for a service
End user accessing www.website.com
How to get Docker 1.12
Mac OS X
Developer Workstations Cloud Providers
Windows PC
AWS
Azure
Commercially Supported
Docker Datacenter
CS Docker Engine
Docker Cloud
Open Source Docker Engine installers for the following Linux distros
Optimized for and integrated directly into the underlying platform with custom plugins and drivers
Deep integration with native load-balancers, templates, SSH keys, ACLs, scaling groups, firewall rules…
beta.docker.com
Distributed Application Bundle
experimental
www.docker.com/dab
A portable format for multi-container applications
5
Goldilocks and the 3 XaaS
Platform As A Service
Infrastructure As A Service
Software As A Service
Too high
Too low
Just right
Container As A Service
Docker Datacenter core values
48
Management at scale Integrated Content Trust Secure Access (RBAC) Integrates with existing
systems
Full support of Docker API Seamless dev to prod
workflow Infrastructure, network and
storage portability
Easy to setup and use Native Docker solution Extend existing Docker developer experience
+ +Agility Portability Control
Agility, portability and control for developers and IT
Developers IT Operations
• Freedom to create and deploy apps fast
• Define and package application needs
• Quickly and flexibly respond to changing needs
• Standardize, secure, and manage
Frictionless portability across teams, environments, infrastructure
49
Docker Datacenter platform
Management Universal Control Plane
Security Content Trust, RBAC, LDAP/AD
Orchestration Swarm
Container Runtime Engine
Registry Service Trusted Registry
BUILD Development Environments
SHIP Registry: Secure Content &
Collaboration
RUN Control Plane: Deploy,
Orchestrate, Manage, Scale
Networking Volumes MonitoringLoggingConfig MgtCI/CD
IT OperationsDevelopers IT Operations
Docker CaaS Workflow
Docker Containers as a Service platform
52
BUILD Developer Workflows
SHIP Registry Services
RUN Management
Docker for Mac and Windows Docker Trusted Registry Docker Universal Control Plane
Docker Cloud
Docker Container Engine
Ecosystem Plugins and Integrations
2013
- 05
2013
- 06
2013
- 07
2013
- 08
2013
- 09
2013
- 10
2013
- 11
2013
- 12
2014
- 01
1,000,0000
2014
- 02
2014
- 03
2014
- 04
2014
- 05
2014
- 06
2014
- 07
2014
- 08
2014
- 09
2014
- 10
2014
- 11
2014
- 12
2015
- 01
2015
- 02
2015
- 03
2015
- 04
2015
- 05
2015
- 06
2015
- 07
2015
- 08
2015
- 09
2015
- 10
2015
- 11
2015
- 12
2016
- 0
1
1,000,000,000~
10,000,0009,000,0008,000,0007,000,0006,000,0005,000,0004,000,0003,000,0002,000,000
6,000,000,0005,750,000,0005,500,000,0005,250,000,0005,000,000,0004,750,000,0004.500,000,0004,250,000,0004,000,000,0003,750,000,0003,500,000,0003,250,000,0003,000,000,0002,750,000,0002,500,000,0002,250,000,0002,000,000,0001,750,000,0001,500,000,0001,250,000,000
2013
- 05
2013
- 06
2013
- 07
2013
- 08
2013
- 09
2013
- 10
2013
- 11
2013
- 12
2014
- 01
1,000,0000
2014
- 02
2014
- 03
2014
- 04
2014
- 05
2014
- 06
2014
- 07
2014
- 08
2014
- 09
2014
- 10
2014
- 11
2014
- 12
2015
- 01
2015
- 02
2015
- 03
2015
- 04
2015
- 05
2015
- 06
2015
- 07
2015
- 08
2015
- 09
2015
- 10
2015
- 11
2015
- 12
2016
- 0
1
~
2016
- 09
1,000,000,000~
10,000,0009,000,0008,000,0007,000,0006,000,0005,000,0004,000,0003,000,0002,000,000
6,000,000,0005,750,000,0005,500,000,0005,250,000,0005,000,000,0004,750,000,0004.500,000,0004,250,000,0004,000,000,0003,750,000,0003,500,000,0003,250,000,0003,000,000,0002,750,000,0002,500,000,0002,250,000,0002,000,000,0001,750,000,0001,500,000,0001,250,000,000
NotaryrunC •
containerd •
HyperKit , VPNKit, DataKit •
SwarmKit •
libcontainer •
libnetwork • • Docker 1.8 : Docker Content Trust
• Docker for Mac Docker for Windows
• Docker 1.12 with built-in orchestration
• Docker 0.9 : Pluggable execution
• Docker 1.7 : Multi-Host Networking
• Docker 1.11: OCI support
Declarative
• JSON configuration for desired infrastructure state: • Specification of instances — vm image, instance type, etc. • Group properties — size, logical identifiers, etc.
• Design patterns encourage • encapsulation • composition
• Config is input to all operations — system figures out what to do66
Self-healing
• Composed of a set of active components / processes that • monitor infrastructure state • detect state divergence • take actions
• Continuous monitoring and reconciliation — always on
• No downtime — rolling update67
Toolkit• Primitives for managing collections of resources
• create, scale, destroy • rolling update
• Abstractions & Developer SPI • Group - manages collection of resources • Instance - describes the physical resource • Flavor - extra semantics for handling instances
• A collection of executable, active components — plugins • Initially, Go daemons in the toolkit • Soon, easy management via Docker Plugins (runc)
Instance Plugin
• Spec: specification / model of an instance (e.g. vagrant, EC2): • Logical ID, Init, Tags, and attachment • Platform-specific properties
• Methods: • /Instance.Validate • /Instance.Provision • /Instance.Destroy • /Instance.DescribeInstances
• Examples: instance plugins for EC2, Azure VM, Vagrant, … 70
Flavor Plugin
• Gives more context about the group members: • Size, or list of Logical ID’s (e.g. IP addresses for ‘pets’) • Application-specific notions of ‘health’
Is the node not only present but also joined a swarm? • Methods:
• /Flavor.Validate • /Flavor.Prepare • /Flavor.Healthy
• Examples: flavor for Zookeeper members, Docker swarm nodes71
Group Plugin
• Main entry point for user interaction: • Create, describe update, update, destroy • Config JSON is always the input
• Composed of Instance and Flavor — mix and match to manage cattle (fungible) or pets (special)
• Methods: • /Group.Watch • /Group.Unwatch • /Group.Inspect
72
• /Group.DescribeUpdate • /Group.Update • /Group.StopUpdate
• /Group.Destroy
ConfigurationExample config file (zk.conf): Group configuration = Instance + Flavor
{ "Properties": {
/* raw configuration */
} }
{ "groups" : { "my_zookeeper_nodes" : { "Properties" : { "Instance" : { "Plugin": "instance-vagrant", "Properties": { "Box": "bento/ubuntu-16.04" } }, "Flavor" : { "Plugin": "flavor-zookeeper", "Properties": { "type": "member", "IPs": ["192.168.1.200", "192.168.1.201", "192.168.1.202"] } } } } } }
Operations
• Make sure the plugins are running: • infrakit/group &; infrakit/zookeeper &; infrakit/vagrant &;
• “Watch” the group starts management: • infrakit/cli group watch zk.conf
• Update the config, e.g. change size or add IP address • Describe changes before committing —
infrakit/cli group describe zk.conf • Begin update —
infrakit/cli group update zk.conf74
Today
75
• InfraKit is just getting started… only primitives for working with groups like clusters of hosts
• But we have big plans • Improve group management strategies • More resource types — networking, load
balancers, storage…
• A cohesive framework for active management of infrastructure — physical, virtual, or containers
Get Involved
• Help define and implement new and interesting plugins • Instance plugins for different infrastructure providers • Flavor plugins for systems like etcd or mysql clusters • Group controller plugins — metrics-driven auto scaling
and more
• Help define interfaces and implement new infrastructure resource types — load balancers, networks and storage volume provisioners
76
More Info
• Github: https://github.com/docker/infrakit
• A quick tutorial: https://github.com/docker/infrakit/blob/master/docs/tutorial.md
77