Upload
vmworld
View
232
Download
0
Embed Size (px)
Citation preview
Containers without Compromise:Persistent Storage for Docker Containers with
VMwareVivek Saraswat, VMware, Inc
CNA5860
#CNA5860
• This presentation may contain product features that are currently under development.
• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.
• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.
• Technical feasibility and market demand will affect final delivery.
• Pricing and packaging for any new technologies or features discussed or presented have not been determined.
Disclaimer
CONFIDENTIAL 2
CONFIDENTIAL 3
Agenda
1 Intro to Cloud-Native Applications
2 Containers 101
3 Persistent Data in Containers
4 VMware Solutions for Cloud-Native Storage
5 Looking Towards the Future
CONFIDENTIAL 4
What are Cloud-Native Applications?
Developer access via APIs
Microservices, not monolithic stacks
Continuous integration and deployment
App-defined Availability
Built for scale
Decoupled from infrastructure
Application
CONFIDENTIAL
Hardware
OS Kernel
OS File system
Use
rspa
ce
Container
App
pro
cess
App
pro
cess
App
pro
cess
App
pro
cess
App
pro
cess
Container
App
pro
cess
App
pro
cess
Linux Containers
55
• App-level Isolation– Isolation at individual kernel
subsystem level (e.g. filesystem, CPU resources, etc.)
– User-level process (LXC, libcontainer) orchestrates these subsystems to create a container
• Existed for Many Years– Solaris Zones, FreeBSD Jails, etc.
• Why?– Process isolation– Reproducible environment– Enables management at scale
CONFIDENTIAL 6
is a “Shipping Container” for Code
Frictionless deployment andmaximum portability
On developer laptop:
Then on server:
That’s it!!
A natural fit for 3rd Platform, 12 factor, microservicesIt makes DevOps much much easier
Developers because …
~# docker build my_app~# docker push my_app~#
~# docker pull my_app~# docker run my_app~#
CONFIDENTIAL 7
Containers Are Stateless “Cattle”
Source: “CERN Data Centre Evolution”http://www.slideshare.net/gmccance/cern-data-centre-evolution
CONFIDENTIAL 8
But…What about Your Data?• If you start a new container, you might lose
all the data from the old one!• “Stateful” data needs to be accessed and
protected separately
• Original model: Persist stateful data to noncontainerized managed storage
• But, problems arise:– No local control over storage management– Latency/access issues– Issues at scale
CONFIDENTIAL 9
Containerized Storage Apps are Rapidly Increasing
Thousands of DB appsMillions of downloads
CONFIDENTIAL 10
Bring in the Container Data Volumes!• Usage
– Contains persistent data for local containers– Appears as directory within host file system
(e.g. “docker run –v /mount/mydata/”)– Can store on external storage and
mount/unmount from a host
• Benefits– Manage and preserve your stateful data– Utilize storage platform data services
Volumes open up new possibilities for containerized applications!
Host
Container
Storage Platform
Container
DataVolumes
CONFIDENTIAL 11
Container Storage Use CasesUnshared Volumes Shared Volumes Persist to External Storage
Use Case: Running container-based SQL or noSQL DB
Use Case: Sharing a set of tools or content across app instances
Use Case: Object store for retention/archival, DBaaS for config/transactions
Host
C C
Storage Platform
Host
C C
Host
C C
Host
C C
Storage Platform
Host
C C
Host
C C
CloudStorage
API API
CONFIDENTIAL 12
Containerized Storage in Practice• Application composed of series of
loosely coupled microservices– Scheduled by an application orchestrator
(e.g. Kubernetes, Mesos)– Microservices coordinate via REST API
• Each microservice consists of multiple stateless and stateful containers– API frontend container– Database engine container– Actual data is persisted to a container
data volume (supported byexternal storage)
ApplicationOrchestrator
Microservice 1
API Front-End
Database Engine
Data Volume
Microservice 2
API Front-End
Database Engine
Data Volume
Microservice 2
API Front-End
Database Engine
Data Volume
CONFIDENTIAL 13
Container Volumes and Virtual Disks: Look Familiar?
CONFIDENTIAL 14
Announcing vSphere Driver for Flocker 1.0!
Luke Marsden, CTO
CONFIDENTIAL 16
The Problem ClusterHQ’s Flocker Solves
CONFIDENTIAL 17
ClusterHQ Flocker Data Volume Migration (before)
CONFIDENTIAL 18
ClusterHQ Flocker Data Volume Migration (after)
CONFIDENTIAL 19
• Integrate “upwards” to Orchestration Frameworks– Docker Swarm and Compose via Flocker Docker Plugin– Mesosphere/Marathon– Kubernetes (coming later), CoreOS
• Integrate “downwards” to storage vendors– vSphere driver enables awesome coverage– Growing ClusterHQ partner network– Also support OpenStack Cinder– Integrate with public cloud storage backends
• Add depth of storage capabilities– Make data portable: enable migration of data volumes between clouds,
different stages of software development lifecycle– Enable snapshotting, cloning, backup/restore, HA, DR…
Get in touch! clusterhq.com/contact @clusterhq
ClusterHQ Flocker Roadmap
ClusterHQ
CONFIDENTIAL 20
vSphere Driver for Flocker Details• Run containerized stateful apps on your current vSphere deployment using open-source
Flocker software• Move containers + data volumes from one ESX VM to another• Compatible with ALL vSphere storage (VSAN, VVOL, VMFS, NFS)
• Straightforward install/configure/deploy process
• Free! • Available at https://github.com/vmware/vsphere-flocker-driver
21
Native Docker on vSphere
CONFIDENTIAL
ESX VM2
Container DB App
ESX VM1
Container DB App
VMDK1+ Container Volume
VMDK2
ESX VM2
Container DB App
ESX VM1
Container DB App
vSphere + Flocker
MoveContainer
MoveContainer
VMDK1 VMDK2Container VolumeVMDK
When container moves, data volume stays on host VMDK. Database starts on new
VM without any of its data.
Data Volume stored on separate VMDK. When container moves, VMDK moves with
it. Database keeps its data!
vSphere Flocker Demo
22
CONFIDENTIAL 23
VMware Virtual SAN 6.1Radically Simple Hyperconverged Storage for VMs
vSphere + Virtual SAN…
• Software-defined storage optimized for VMs
• Hypervisor-converged architecture
• Runs on any standard x86 server
• Pools HDD/SSD into a shared datastore
• Delivers enterprise-level scalability and performance
• Managed through per-VM storage policies
• Deeply integrated with the VMware stack
Overview
Hard disksSSDHard disks
SSDHard disks
SSD
Virtual SAN Datastore
CONFIDENTIAL 24
But Don’t Take Our Word for It…
“ClusterHQ and VMware supported our evaluation of Flocker with vSphere and Virtual SAN, providing us building blocks for container persistence in our Docker-based PaaS.”
CONFIDENTIAL 25
CONFIDENTIAL 26
Unified Hybrid Platform
vSphere Integrated Containers• Give developers the flexibility, portability
and speed containers deliver• Leverage existing investments in VMware
infrastructure, people, processes and management tools
• DRS, vMotion, HA/DR• Storage and Network Integration (VSAN
and NSX)
• Apply same isolation, data persistence, networking, management and robust service levels you have today
• No rebuilding or re-architecture required
• Full compatibility with broad ecosystem of existing tools
NSX
VSAN
vRealize
vSphere
vSphere Integrated Containers(Instant Clone, Project Bonneville, Photon OS)
Cloud-Native Platform
VMware Photon Platform
CONFIDENTIAL 27
Photon Controller
Photon Machine
Photon Platform
Host controller & scheduler:High scale distributed control plane,includes Lightwave
Compute host:Lightweight hypervisor, based on ESXIncludes Photon OS
Deep integration with modern, open source frameworks & app platforms
CONFIDENTIAL 28
Looking Towards the Future of Storage
• Distributed infrastructure– Global management– Automation friendly
• Scalable troubleshooting– Decentralized analytics– Information gathering, processing,
prediction
• IT-friendly GUI, scripting– Infrastructure operations
• Dev-friendly APIs, CLI– Application integration
Physical Servers
Infrastructures
Pools of Resources
Virtual SAN Datastore
Magnetic Devices
Flash Devices
Magnetic Devices
Flash Devices
Storage Abstraction Distributed storage platform
stor
age
infr
astr
uctu
rem
anag
emen
t
dist
ribut
ed m
onito
ring
UI/A
PIs
CONFIDENTIAL 29
VDI files
A Distributed File System for Cloud-Native Apps• Hyper-converged scale-out file system
• Relies on Object Storage backend– Hardware management– Resource provisioning, discovery– Distributed parallel data path– High availability
• Backend: VSAN, others…– Can span multiple backend “clusters”
• Design Requirements– POSIX file system– Cloud scale: files, clients, clones– Per file / directory access control– O(1) snapshot / clone creation
Distributed storage management service
VSAN (Object) store VSAN (Object) store
Distributed File system
NoSQL
Distributed File System Demo
30
CONFIDENTIAL 31
What’s Now?
• vSphere driver for Flocker 1.0 available now for free at: https://github.com/vmware/vsphere-flocker-driver
• Extending functionality for VMware-based container data volumes• Storage integration with vSphere Integrated Containers• Storage integration with Photon Platform
Follow along on Twitter: @theVSaraswat
What’s Next?
Containers without Compromise:Persistent Storage for Docker Containers with
VMwareVivek Saraswat, VMware, Inc
CNA5860
#CNA5860