Managing Heterogeneous Cloud Infrastructure with OpenStack, Alex Glikson & Ezra Silvera

1

Managing Heterogeneous Cloud Infrastructure with OpenStack

Example: Hosting ‘Hybrid’ Applications Comprising Bare-Metal and Virtualized Resources

OpenStack Israel, June 2nd, 2014, Hertzliya

IBM Research – Haifa

Alex Glikson

Ezra Silvera

2 © 2014 IBM Corporation

Outline

� Heterogeneous Clouds

‒ Background & Motivation

‒ Opportunities and Challenges

� Hosting ‘Hybrid’ (Bare-Metal + Virtualized) Applications

‒ Goals

‒ Scenarios

‒ Solution Approach

‒ Technical Challenges and Design Considerations

‒ Example

� Summary, Q&A

3

Background & Motivation


Evolution of IaaS Usage Patterns and Offerings

� Large variety of configurations

‒ E.g., AWS now offers >60 instance configurations

� Rise of bare-metal providers

‒ IBM/SoftLayer, Internap, etc

� Small set of standardized (virtual) hardware configurations

‒ e.g., 1 instance type in AWS EC2 (2006)

Commonly Offered Hardware Configurations

� Growing spectrum of workloads

‒ E.g., ‘legacy’ business applications, HPC, analytics, etc

� Dev & test

� Web applications (‘cloud-native’)

Common Usage Patterns

� Agility, efficiency, elasticity, self-service, pay-per-useValue proposition

NowEarly Days

Trend: Increased Cloud Infrastructure Heterogeneity (including bare-metal)to Support Larger Variety of Applications with Special

Requirements


Embracing Infrastructure Heterogeneity for the Benefit of the Application: Example

Virtualization

Network

Storage

Compute

CPU/mem/disk balance

CPU-rich mem-rich disk-rich …

CPUs GPGPUs Accelerators

Local HDD SSD SAN NAS

…

…1 Gbit shared 10 Gbit shared 10 Gbit reserved

…

bare-metal KVM LXC VMware …

Tier-1: web app

Tier-2: 3D rendering

Tier-4: Designs DB Tier-3: Hadoop

balanced

Example: Application for Collaborative 3D Design


Embracing Infrastructure Heterogeneity for the Benefit of the Application: Example

Virtualization

Network

Storage

Compute

CPU/mem/disk balance

CPU-rich mem-rich disk-rich …

CPUs GPGPUs Accelerators

Local HDD SSD SAN NAS

…

…1 Gbit shared 10 Gbit shared 10 Gbit reserved

…

bare-metal KVM LXC VMware …

Tier-1: web app

Tier-2: 3D rendering

Tier-4: Designs DB Tier-3: Hadoop

balanced

Example: Application for Collaborative 3D Design

4

4

4

4 3

3

3

3

2

2

2

2

2

3

41

1

1

1

1

7

Opportunities and Challenges



� Business opportunities enabled by infrastructure heterogeneity

‒ Service/application providers:

• Cloud adoption in ‘traditional’ market segments � agility and efficiency

– E.g., manufacturing, finance, scientific, etc

– IDC: the (extended) HPC market alone is larger than IaaS ($15B vs $9B)

‒ Cloud/infrastructure providers:

• Ability to better fit to specific classes of applications � differentiation

– Especially relevant given the overall commoditization of the IaaS market, recent price wars, etc



Challenge: cloud infrastructure natively designed for:

‒ Underlying infrastructure heterogeneity

‒ Ability to seamlessly host applications with special requirements

‒ Still preserve the characteristics of the cloud model

• Agility, efficiency, elasticity, etc


Heterogeneous Cloud Requirements: Cloud User

� Cloud user perspective

‒ Access to a wider range of offered HW/SW configurations

• Performance, security/isolation

• Compute, storage, network

‒ Same or better user experience as with ‘regular’ cloud, e.g.:

• Same: single/unified catalog of configuration ‘flavors’, images, etc

• Same: seamless network connectivity, storage access, etc

• Same: higher-level services (orchestration, LBaaS, DBaaS, etc)

• Better: higher-level application abstraction, hiding the details of the underlying HW (critical with increased complexity of underlying HW)

– Specify goals/requirements, rather than specific resource types/models etc


Heterogeneous Cloud Requirements: Cloud Provider

� Cloud provider perspective

‒ Minimize management cost by unified processes and APIs

• Provisioning, inventory, monitoring, life cycle operations, visualization, etc

– Unified resource model

‒ Secure multi-tenancy across heterogeneous resources

‒ Elasticity & efficiency

• Smart scheduling, to overcome resource fragmentation

• Dynamic balancing between HW-compatible pools by “repurposing”(subject to compatibility)

– Capacity monitoring & prediction

– Automated evacuation & re-provisioning


Heterogeneous Cloud: OpenStack Perspective

� OpenStack is following a similar pattern‒ Started primarily with dev & test and cloud-native applications

• Simplistic approach to flavors, scheduling, performance optimization, etc

‒ Extensive efforts to enable adoption by additional applications, e.g.:

• Hadoop (Sahara)

• DBaaS (Trove)

• Bare-metal (Ironic, TripleO, Tuskar)

• Scheduling (Gantt)

• Reservations (Climate)

• Policies (Congress)

• Etc

‒ Growing community interest in supporting heterogeneous clouds, e.g.:• Multi-Tenant Bare Metal Provisioning (Mon, 11:15, B206; Tue, 2:50pm, B301)

• Deploying OpenStack in a Multi-Hypervisor Environment (Wed, 11am, B101)

• IBM, SoftLayer and OpenStack - Present and Future (Wed, 11:50am, B312)

• Many more related sessions…


Heterogeneous Cloud: OpenStack to the Rescue!

Claim: OpenStack provides a good basis for building heterogeneous clouds

• But few gaps yet to be addressed


Hosting ‘Hybrid’ (Bare-Metal + Virtualized) Applications


Hybrid Applications – Background and Goals

� Hybrid environment – includes both physical and virtual resources

‒ Pools: Resources are grouped into bare-metal and virtual pools

� Hybrid application – a composite (e.g., multi-tier) application that spans both virtual and physical server(s)

� Goals

‒ Admin: Minimize management complexity and overhead. Unified management across bare-metal and virtualized pools.

‒ User: Simplified ongoing management and application life cycle.

� Design guideline: If possible try to use OpenStack native solution

‒ E.g., no external coordinators, orchestration, management


Pool 3: Virt Type 2 Pool 4: Bare metalPool 1: Spare (BM) Pool 2: Virt Type 1

Tenant ATenant B

Hybrid Applications – Background and Goals


Scenarios


Scenario 1: Node Repurposing

Support native policies that dynamically balance the virtual andphysical pools (e.g., moving servers from one to another)

� Repurposing Virtualized node (Hypervisor) � Physical node

1. Detect resource congestion in the physical pool

2. Identify a candidate host to be re-purposed into physical

3. Trigger “evacuate“ for all VMs on the selected host

4. Assign physical host to the BM pool

‒ Server is ready to be deployed as bare-metal application (later on)

Bare metal pool

Bare metal pool


Scenario 2: Runtime Decision on Target Pool

Scenario: decide on the server type (BM/VM) upon deployment

� E.g., according to performance requirements

1. User deploys an application without specifying the server type

2. The system automatically choose server type

3. Automatically choose/construct actual image for deployment‒ Option 1: dynamic adaptation

‒ Option 2: Maintain multiple ‘versions’

4. Deploy using corresponding provisioning mechanism‒ leveraging Ironic

‒ Apply adaptation upon boot (e.g., leveraging Heat)


Solution Approach


Management Approaches

� Possible solutions range from 'share nothing to 'share everything'

‒ 'share nothing’ - dedicated services for each pool

‒ ‘share everything’ - Use single copy of each service

‒ Some of the services are shared (e.g., regions, cells)

� OpenStack can natively handle bare metal and virtual hosts, so why not combining them together in a single OpenStack?

� We focused on the 'shared everything' architecture

‒ Defined as 'Integrated Management‘


Integrated Management: Basic Architecture

� Special resource types, mapped to specially designed resource pools� Use multiple host aggregates, including ‘bare-metal’

‒ Use aggregate filters for scheduling� Networking

‒ Admin network for provisioning and management of compute nodes‒ Separate management network for BM machines hosting user applications‒ Data network combining both VMs and BM nodes

Controller node

mysql-server

rabbit-server

neutron-server

glance

Network node

Neutron.*-plugin-agent

Neutron-l3-agent

Neutron-dhcp-agent


Neutron-l3-agent

BM computeCompute node

nova-compute


Compute node

nova-compute


Compute node

nova-compute


BM nodeBM node

BM node

External

Data/Guests

Management

API

Internet

BM MgtBM nodeBM nodeBM spare


Advantages of Integrated Management approach

� Native and simple OpenStack-based solution

‒ Doesn’t require external orchestration

‒ No need to coordinate between services of different pools

� Simplified administration

‒ Controlling all pools/resources from one place in a unified manner

‒ Reduce configuration complexity and overhead

‒ Better diagnostics and root cause analysis

� Improved user experience and manageability

‒ Maintain/expose combined topology view for different server types

‒ Hide the underlying infrastructure details from the user (if needed)


Technical Challenges and Design Considerations


Challenges Examples

� OpenStack has some gaps when managing bare-metal and virtual machines simultaneously

� Networking: Simultaneously manage both virtual and physical networks

‒ Network isolation for tenants should combine both BM and VMs

‒ E.g., Use tunnels/overlays connecting physical nodes and VMs

‒ E.g., Use multiple plugins (e.g., OVS, openflow) simultaneously to gain advantages for each pool (if possible)

� Managing Bare-Metal workloads

‒ Security in multi-tenant environments (e.g., device firmware)


Challenges Examples (Cont.) - Compute

� Enhanced scheduler to better support heterogeneity‒ Use different placement policies for different pools (e.g., look at

GPU consumption rather then CPU/memory)

� Support instance management for BM‒ Actions for BM instances may differ from actions on VMs

‒ E.g., implement: Console, Explore usage of: Suspend, Resize

� Add hierarchical relations between BM instance and the compute node running on it‒ E.g., Propagate BM delete command to the VM level

� Support aggregates for specific BM nodes


Challenges Examples (Cont.)

� Image Management‒ Enhance access control for images in Glance

‒ Support run-time image selection

‒ Support dynamic adaptation of images

� UI: extend UI to support Hybrid applications/instances‒ Add bare-metal node management in Horizon

‒ Physical machines are shown as Hypervisors


Hybrid Application example


Example – Deploying Hybrid WordPress

� Apache+WordPress on VM and MySQL on physical node

� Setup and configuration‒ A single HEAT template used for the two nodes

‒ Using host aggregates for scheduling

‒ Used diskimage-builder to build the baremetal images• cloud-init element installed


Example – Deploying Hybrid WordPress

� Example gaps reflected in Horizon:‒ BM nodes are shown as Hypervisors

‒ Deployment images are exposed to users


Summary


Summary

� Heterogeneous cloud environments are gaining momentum

‒ Critical in order to host a broad spectrum of applications

� OpenStack is a promising solution to manage hybrid clouds

‒ By using integrated management we can have

• Simple native OpenStack solution

• Simplified administration and enhanced user experience

� Need a careful design to ensure all requirements (e.g., security, isolation) are maintained across the hybrid environment

� Some gaps need to be addressed in OpenStack


Q & A

Technology

Managing Heterogeneous Cloud Infrastructure with OpenStack, Alex Glikson & Ezra Silvera