Embed Size (px)
Citation preview
CloudStack Orchestrationby Alex Huang
A little bit about me
• Cloud.Com Founding Engineer• Software architect for CloudPlatform• Responsible for overall architecture,
performance, and scalability• Committer and PPMC member• BS from UC Berkeley and MS from Stanford
Outline
• Difference between automation, orchestration, and provisioning
• Networking Example• Anatomy of a plugin• Myth busting
Automation, Orchestration, & Provisioning
What is the difference?
• The key is the data center abstraction layer– Virtual Machine, Template, Nic, IP Address, Volume,
Network, Rules, Snapshot• Orchestration orchestrates within this
abstraction layer• Provisioning manifests concepts in abstraction
layer on physical resources• Automation orchestrates above the data center
abstraction layer to bring greater functionality
Examples
• Orchestration– VM deploy, Volume creation, Network creation,
Network rules propagation• Provisioning– Starting a VM on a hypervisor– The actual movement of a volume from one storage
pool to another• Automation– HA– Putting a physical resource into maintenance mode
Enabling Innovation
• CloudStack Orchestration must not define the innovation.– Partners define their own APIs.– Partners and CloudStack can work together on unified APIs
through design process on Apache.• Differentiate between orchestration and provisioning.
– CloudStack only orchestrates.– Provisioning is always pushed to the partner.
• Clearly defined data center abstraction layer.– Changes in this layer are broadcasted to partners.
• Utilize CloudStack’s orchestration to deploy and auto-scale partners’ technologies.
Networking Example
CloudStack Terminology (End User)
• Network– A single concept to encapsulate multiple network technologies to simplify
representation to the end user.– Each Network always carries its Network Traffic Type.– CloudStack DOESN’T understand how to provision this conceptual
network on to the physical network.• NetworkService
– L2-L7 network services that partners have written to operate within a Network.
– Currently defined: Load Balancing, Port Forwarding, Firewall, Gateway, DNS, DHCP, Static NAT, VPN, Source NAT, User Data.
• NetworkOffering– A packaging of the NetworkServices provided to the end user on a
particular Network.– NetworkOfferings are put together by cloud service provider.
CloudStack Terminology (Provider)
• Network Traffic Type– Traffic types are mapped to the underlying physical network by
the cloud service provider.– Traffic type is not the same as network (Guest traffic type can
actually be carried on multiple networks)– Currently defined: Public, Guest, Storage (Backup really),
Management• NetworkServiceProvider
– Plugin that understands how to provide one or more NetworkServices by using VPX or physical resource.
• PhysicalNetwork– Actual wiring of the data center.
CloudStack Terminology (Partner)
• NetworkGuru– Plugin that understands the network isolation technology, mac addressing scheme,
and IP addressing scheme deployed and how to map Network Traffic Types to the underlying physical network.
– CloudStack passes Network to NetworkGuru to “implement” before the network is needed by a virtual machine.
– CloudStack asks the NetworkGuru to issue ip, mac, and isolation to a virtual machine before it starts.
– CloudStack informs the NetworkGuru when a virtual machine stops so it can collect resources.
– When all virtual machines in a Network are stopped, CloudStack garbage-collects the Network by asking the NetworkGuru to shutdown the network.
– CloudStack provides a default implementation for VLAN based isolation technology.• NetworkElement
– Interface that specifies the events CloudStack signals to the NetworkServiceProviders when a Network needs to be “implemented” and shutdown and when a virtual machine joins and leaves a Network.
“Architect” Model
• The builder offers multiple blueprints for the owner to build the house.• Owner chooses on a blueprint and then adds on with additional enhancements
such as hardwood floors, granite counter tops, etc.• General contractor builds to the blueprint by orchestrating between different sub-
contractors to build different parts of the blueprint.• There are two general category of contractors.
– Rough-in sub-contractors who take care of plumbing, electricity, framing, foundation.– Finish sub-contractors who put in flooring, kitchen cabinets etc.
• Each sub-contractor is responsible for only their work but looks over the entire blueprint to make sure their work can actually be done.– E.g. A lighting plan may conflict or needs to change depending on the framing plan.
• General contractor is responsible for sequencing the sub-contractors to make sure everything the sub-contractor is dependent on is ready when the sub-contractor arrives to do his work.
• Every change requires a the blueprint to be republished so every sub-contractor can make their appropriate changes.
Comparison
Building a house• Owner• Builder• General Contractor• Rough-in Sub-Contractors• Finish Sub-Contractors• Blueprint• Cabinets, Flooring, Counter
Tops, etc
Building a network• End user• Cloud Service Provider• CloudStack• NetworkGurus• NetworkServiceProviders• Abstraction Layer• NetworkServices
Architectural Principles
• CloudStack clearly defines the difference between orchestration and provisioning.– Orchestration the ordering of what needs to happen in CloudStack’s
abstraction layer.– Provisioning is the actual work performed at the resource.
• CloudStack clearly defines the difference between network definition and network services.– Network definition is handled by NetworkGuru.– Network services is handled by NetworkServiceProvider.
• CloudStack broadcasts changes in the network every time NetworkServices and virtual machines changes in the Network.
• CloudStack allows the Cloud Service Provider to setup the appropriate mappings between virtual concepts such as Network and Network Traffic Type to the underlying physical network.
Engine
Sequence Flow for VM CreationEnd User Rest API
SecurityCheckers
User VM Mgr
Network Mgr
Storage MgrJob
SchedulingVirtualMachine Mgr
Network Guru
Deploy VM
ACL Checks
Allocate Entity in CS
Allocate VM
Allocate NIC
Allocate Volume
Allocate IP
Schedules Deploy Job
Returns with job id, VM id
Query Job Result
Returns with job status
Sequence Flow for VM DeploymentJob Threads
Network Element
User VM Mgr
Network Mgr
Storage Mgr
VirtualMachine Mgr
Network Guru
Start VM
Start VM
Prepare Nics
Notify that Nic is about to be started in network
Reserve resources for Nic
Services APIServer
Resources
Start User VM
Agent Calls
Prepare Volumes
Template Mgr
Deployment
Planner
Get a Deployment Plan (Host and StoragePool)
Prepare template on Primary Storage
Agent Calls
Agent Start VM Call
Stores job result
Anatomy of a plugin
CloudStack Orchestration
Accounts
CloudStack WebServices API
Security Manager
Events Manager
Usage Manager
Domain Manager
Account Manager
Limits Manager
Virt
ual M
achi
ne
Man
ager
Stor
age
Man
ager
Net
wor
k
Man
ager
Tem
plat
e
Man
ager
Snap
shot
M
anag
er
CloudStack Plugins
OAM&P API End User API AWS API Pluggable Service API Engine
AdaptersNetwork Guru
Network Element
Deployment Planner
Hypervisor Guru
Business Logic
Net
Scal
er N
etw
rok
Serv
ice
Prov
ider
Virt
ual R
oute
r Net
wro
k Se
rvic
e Pr
ovid
er
Firs
t Fit
Dep
loym
ent P
lann
er
Use
r Dis
pers
ing
Dep
loym
ent P
lann
er
Reso
urc
e M
anag
er
Rule
s M
anag
er
Upd
ate
Man
age
r HA
Man
age
rCa
paci
ty
Man
age
r
Framework
Agent Manager Cluster Manager Data Access Layer
Anatomy of a Plugin
• Server Component:– Deployed on management server– Can implement multiple Plugin
APIs to affect its feature– Can expose its own API through
Pluggable Service so administrators can configure the plugin
– Can access the database
• ServerResource:– Deployed co-located with the
physical resource– Cannot access the database
ServerResource- Optional. Required if Plugin needs to be co-
located with the resource- Implements translation layer to talk to resource- Communicates with server component via JSON
ServerResource- Optional. Required if Plugin needs to be co-
located with the resource- Implements translation layer to talk to resource- Communicates with server component via JSON
Rest API
Plug
in A
PI
Data Access Layer
Implementation
Anatomy of a Plugin
• Can be two jars: server component to be deployed on management server and an optional ServerResource component to be deployed co-located with the resource
• Server component can implement multiple Plugin APIs to affect its feature
• Can expose its own API through Pluggable Service so administrators can configure the plugin
• As an example, OVS plugin actually implements both NetworkGuru and NetworkElement
Plugin Interfaces Available• NetworkGuru – Implements various physical network technologies
and ip address• NetworkElement – Facilitate network services on network elements
to support a VM (i.e. DNS, DHCP, LB, VPN, Port Forwarding, etc)• DeploymentPlanner – Different algorithms to place a VM and
volumes.• Investigator – Ways to find out if a host is down or VM is down.• Fencer – Ways to fence off a VM if the state is unknown• UserAuthenticator – Methods of authenticating a user• SecurityChecker – ACL access• HostAllocator – Provides different ways to allocate host• StoragePoolAllocator – Provides different ways to allocate volumes
Adding a Plugin to CloudStack
• Components are configured though components.xml
• Supports DAO, Manager, and Adapter patterns
Components.xml Example<components.xml> <system-integrity-checker class="com.cloud.upgrade.DatabaseUpgradeChecker"> <checker name="ManagementServerNode" class="com.cloud.cluster.ManagementServerNode"/> <checker name="EncryptionSecretKeyChecker" class="com.cloud.utils.crypt.EncryptionSecretKeyChecker"/> <checker name="DatabaseIntegrityChecker" class="com.cloud.upgrade.DatabaseIntegrityChecker"/> <checker name="DatabaseUpgradeChecker" class="com.cloud.upgrade.PremiumDatabaseUpgradeChecker"/> </system-integrity-checker> <interceptor library="com.cloud.configuration.DefaultInterceptorLibrary"/> <management-server class="com.cloud.server.ManagementServerExtImpl" library="com.cloud.configuration.PremiumComponentLibrary"> <adapters key="com.cloud.storage.allocator.StoragePoolAllocator"> <adapter name="LocalStorage" class="com.cloud.storage.allocator.LocalStoragePoolAllocator"/> <adapter name="Storage" class="com.cloud.storage.allocator.FirstFitStoragePoolAllocator"/> </adapters> <pluggableservice name="VirtualRouterElementService" key="com.cloud.network.element.VirtualRouterElementService" class="com.cloud.network.element.VirtualRouterElement"/> </management-server></components.xml>
Extending CloudStack Networking
Network Manager
Network Element
DnsService
MyDnsElementMyDnsDeviceMa
nager
MyDnsDeviceService
PluggableService
MyDnsDeviceResource
AgentManager Queue
1. prepare (part of start vm)2. prepare (Network, Nic, DeployDestination, VmInfo)
3. addDnsRecord(ip, fqdn)
4.Enqueue AddDnsRecord
5.API call to Dns Device
Device ConfigurationAdmin API (CRUD)
MySQLDemonstrates one way to inform an external DNS server when an instance starts.Classes shaded blue form a plugin / service bundle to integrate an external DNS server. Clients of the instance can then use DNS names to access the instance.
ServerResource
Let’s do some myth busting
Myth 1
• Network is Layer 2• Network is Layer 3• Network confines how I implement my
physical network.
Busted
• No. Network is just a concept to represent where virtual machines can access each other.– One Network to rule them all, One Network to
define them, One Network to bring them all and in the cloud bind them.
• What defines a network to be layer 2 or layer 3 is the NetworkGuru.
Myth 2
• I must convert my service to run inside CloudStack’s plugin.
• CloudStack controls the deployment of my service.
Busted
• CloudStack’s plugin interfaces specify the provisioning events and data associated with the events.
• Plugin implementations can be client stubs that call out to your actual service upon receiving these events.
• If you so desired, plugin implementations can also expose APIs to configure the plugins.
Myth 3
• My service cannot scale independently from CloudStack.
• My service’s availability is tied together with CloudStack’s availability.
Busted
• CloudStack does not need to know how a service scales.– CloudStack does not manage instances of a
service. – CloudStack does not manage instances of the
same plugin. Plugin code is always a singleton.
The future needs you!
Project web site: http://incubator.apache.org/projects/cloudstack.html
Mailing lists: [email protected] [email protected]
IRC: #CloudStack on irc.freenode.net
