Distributing OpenStack ... and compute node n-sched n-cond n-api n-net n-cpu horizon Nova ... OpenStack Cloud Distributed Broker IaaS Distributed IaaS Broker

  • View
    213

  • Download
    1

Embed Size (px)

Transcript

  • Distributing OpenStack on top of a Key/Value store

    Jonathan Pastor(Inria, Ascola, Ecole des Mines de Nantes)

    Journe Cloud 2015

    1

    PhD candidate, under the supervision of Adrien Lebre and Frdric Desprez

  • Discovery Initiative The Discovery Initiative targets the

    delivery of Locality based Utility Computing (LUC). http://beyondtheclouds.github.io

    Leveraging network backbones: extend any point of presence of network backbones with UC servers.

    Bring UC as close as possible to the end users more efficient (latency aware application ++).

    2

  • In short Building an infrastructure composed of

    100/1000 remote sites (10/100 servers per site).

    Target collaboration without any broker.

    But without reinventing the wheel to limit the development effort, we propose to leverage existing successful mechanisms: OpenStack.

    3

    GeographicalSite1

    GeographicalSite2

    GeographicalSite3

    IaaSIaaSIaaS

    IaasIaaS

    IaaS

    IaaSIaaSIaaS

    IaaSIaaS

    IaaS

    IaaSIaaSIaaS

    IaaSIaaS

    IaaS

  • OpenStack Open source IaaS manager

    with a large community.

    Composed of several services dedicated to each aspect of a cloud.

    Services collaborate via:

    Messaging queue (RabbitMQ)

    Shared database (MySQL)

    4

    Nova Nova

    Gestionnaire des machines virtuelles

    Swift Swift

    Glance Glance

    Gestionnaire du stockage

    Neutron Neutron

    Gestionnaire des rseaux

    Gestionnaire des rseaux

    KeyStone KeyStone

    Horizon Horizon

    Outils Administratifset Gestionnaire dinformation

  • Distributing OpenStack Few proposals to federate/operate distinct

    OpenStack DCS

    Flat approach: Leveraging HaProxy and Galera (Active replication) Complexity and scalability issues

    Hierarchical approaches: Cells based (CERN: 3 Sites / 50K cores) the top cell is a SPOF

    Cascading OpenStack.

    You know others!? please mail us! http://beyondtheclouds.github.io/dcc.html

    5

    http://beyondtheclouds.github.io/dcc.html

  • Leveraging Key/value stores

    Alternate solutions exists for storing states over a highly distributed infrastructure (more relevant?).

    Key/value store in place of the classical relational database (with active replication)?

    We focus on the compute service (Nova) by studying the replacement of MySQL by Redis.

    6

  • Replacing MySQL in Nova? Nova services dont access directly the database: they

    communicate with nova-conductor, which in turn calls functions located in:nova/nova/db/api.py

    These functions manipulate databases and return python objects from:

    7

  • Replacing MySQL Nova already includes the possibility of using a different

    DB backend through ORMs (only SQLAlchemy yet).

    nova/nova/db/api.py => provides functions that manipulate the DB via an ORM.

    8

    It means that we can add a new backend:

  • Relational

    9

    How Nova is linked with MySQL?

    SQLAlchemy

    Nova Network

    Nova Compute

    Nova Scheduler

    Nova Conductor

    db.api MySQLDB

  • Relational Object Mapping Extension for key/value stores. https://github.com/badock/rome

    Enables the query of KVS (key/value store) with the same interface as SQLAlchemy.

    Enables to use KVS in Nova by limiting the break of code.

    10

    Rome

  • Non-Relational Relational

    11

    Integrating Nova with Redis

    ROMESQLAlchemy

    Nova Network

    Nova Compute

    Nova Scheduler

    Nova Conductor

    db.api MySQLDBKey/Value DB

  • Proof of concept We can use the Key/

    Value as the cervical spin of a deployment.

    The Key/value store is clustered on controllers

    Compute nodes connect to the Key/value cluster.

    12

    AMQPbus

    AMQPbus

    AMQPbus

    Key/Value Store

    NovaController 3

    n-schedn-condn-apin-netn-cpuhorizon

    NovaController 2n-sched

    n-condn-apin-netn-cpuhorizon Nova

    ComputeNodes

    Nova Compute Nodes

    NovaController 1

    n-schedn-condn-apin-netn-cpuhorizon

    NovaController 5 n-sched

    n-condn-apin-netn-cpuhorizon

    Nova Controller 4and compute node

    n-schedn-condn-apin-netn-cpuhorizon

    NovaComputeNode

    Site 1

    Site 2

    Site 3

    Site 4

  • Experiments

    13

    Preliminary experiments have been conducted on Grid5000.

    mono-site experiments: to evaluate the overhead of using REDIS and the network impact.

    multi-site experiments: To determine the impact of latency.

    Ask for the creation of 500 VMs, fairly distributed on each controller.

  • Preliminary results Time measured for creating 500 VMs in parallel.

    Experiments performed on servers with homogeneous hardware.

    For a fair comparison (routing issues can disturb Galera): use servers on the same site (Rennes).

    Clusters were simulated by adding latency between nodes with TC.

    We followed configuration advised by OpenStack multi-site documentation.

    14

    Redis MySQL(no replication) Galera

    1 cluster (no replication) 298 357 -

    2 clusters 723 268 1361

    3 clusters 518 210 2202

    4 clusters 427 203 1253

    Redis MySQL(no replication) Galera

    1 cluster (no replication) 298 357 -

    2 clusters 271 209 2199

    3 clusters 280 157 3243

    4 clusters 263 139 2011

    10 ms intersite latency 50 ms intersite latency

  • Future work Increase the scale of the experiments

    (more nodes, more sites, more VMs ).

    Make Rome works with other NoSQL DB (Cassandra).

    Apply the same strategy with remaining services: Glance, Neutron?

    Propose leads to bring locality in an OpenStack deployment (host aggregates?)

    15

  • Thanks for your attention

    Questions?

    16

    http://beyondtheclouds.github.io/adrien.lebre@inria.fr / jonathan.pastor@inria.fr

    and many others!

    http://beyondtheclouds.github.iomailto:adrien.lebre@inria.frmailto:jonathan.pastor@inria.fr

  • Bibliography http://openstack-in-production.blogspot.fr/2014/03/cern-cloud-

    architecture-update-for.html

    [Greenberg2009] A. Greenberg, J. Hamilton, D. A. Maltz, and P. Patel. The cost of a cloud: research problems in data center networks. ACM SIGCOMM Computer Communication, Review, 39(1):6873, 2008.

    [Moreno2012] R. Moreno-Vozmediano, R. S. Montero, and I. M. Llorente. Iaas cloud architecture: From virtualized datacenters to federated cloud infrastructures. Computer, 45(12):6572, 2012.

    [IEEE2012] I. . E. W. Group. IEEE 802.3TM Industry Conne

    17

    http://openstack-in-production.blogspot.fr/2014/03/cern-cloud-architecture-update-for.html

  • 18

  • Federation?

    Is the LUC-OS a distributed cloud federation broker?

    19

  • 20

    CS 6CS 5CS 4

    CS 3CS 2

    CS 1

    CloudStack Cloud

    OS 6OS 5OS 4

    OS 3OS 2

    OS 1

    OpenStack Cloud

    OS 6OS 5OS 4

    OS 3OS 2

    OS 1

    OpenStack Cloud

  • 21

    CS 6CS 5CS 4

    CS 3CS 2

    CS 1

    CloudStack Cloud

    OS 6OS 5OS 4

    OS 3OS 2

    OS 1

    OpenStack Cloud

    Central Broker

    OS 6OS 5OS 4

    OS 3OS 2

    OS 1

    OpenStack Cloud

  • 22

    CS 6CS 5CS 4

    CS 3CS 2

    CS 1

    CloudStack Cloud

    OS 6OS 5OS 4

    OS 3OS 2

    OS 1

    OpenStack Cloud

    DistributedBroker

    DistributedBroker

    DistributedBroker

    OS 6OS 5OS 4

    OS 3OS 2

    OS 1

    OpenStack Cloud

  • 23

    CS 6CS 5CS 4

    CS 3CS 2

    CS 1

    CloudStack Cloud

    OS 6OS 5OS 4

    OS 3OS 2

    OS 1

    OpenStack Cloud

    DistributedBroker

    DistributedBroker

    DistributedBroker

    OS 6OS 5OS 4

    OS 3OS 2

    OS 1

    OpenStack Cloud

  • 24

    CS 6CS 5CS 4

    CS 3CS 2

    CS 1

    CloudStack Cloud

    OS 6OS 5OS 4

    OS 3OS 2

    OS 1

    OpenStack Cloud

    DistributedBroker

    IaaS

    DistributedBrokerIaaS

    DistributedBroker

    IaaS

    OS 6OS 5OS 4

    OS 3OS 2

    OS 1

    OpenStack Cloud

  • 25

    OpenStack Cloud

    OpenStack Cloud

    OpenStack Cloud

    OS 6OS 5OS 4

    OS 3OS 2

    OS 1

    OS 6OS 5OS 4

    OS 3OS 2

    OS 1

    OS 6OS 5OS 4

    OS 3OS 2

    OS 1

  • 26

  • How data are stored?

    Each entity (ie VM, Image, SecurityGroup) is modelised by an entity class.

    With ROME, each entity class corresponds to a bucket.

    With current implementation, the data model is quite normalised.

    27

  • How data are represented?

    28

  • How links are modelised?

    29

  • How data are stored?

    30

  • How data are queried?

    if no multiget available, query of objects is sequential.

    if multiget available, query of objects can be parallelised.

    31

  • 32

  • Host aggregates for locality? This information can be used in the scheduler

    to enable advanced scheduling, to set up xen hypervisor resources pools or to define logical groups for migration

    33

    http://docs.openstack.org/developer/nova/devref/aggregates.html

    Host aggregates could be set-up dynamically in an automatic way (Vivaldi?).

    http://docs.openstack.org/developer/nova/devref/aggregates.html