A Deep Dive Into Trove

  • Published on
    16-Jul-2015

  • View
    577

  • Download
    4

Embed Size (px)

Transcript

PowerPoint Presentation

A Deep Dive into TroveOpenStack MeetupMarch, 2015Amrith Kumar

Who Am I?Amrith Kumar, Founder & CTO of TesoraATC on several OpenStack projectsCore reviewer on TroveTesora: the Trove Company#1 contributor to Trove projectTesora DBaaS Platformpackaged, supported version of TroveNearly all work is upstream first Database & distribution certificationsDatabase Virtualization Engine (DVE)Open source, transparent sharding for MySQL

Trove Contributors (Juno)Diverse and growing community128 contributors from 28 companies1723 commits, and 231829 lines of code2A Deep Dive into Trove OpenStack MeetupMarch 2015

Traditional ITProvisioning by DBAsDatabase management by specialistsWaterfall developmentFew large machines / bare metalOracle enterprise licenses

Captive audienceCloud Self-service provisioningDevelopers manage their own databasesAgile development Many small machines / virtualizationMany data management technologiesCompetition with AWS

Transformation of Cloud Data Management

A Deep Dive into Trove OpenStack MeetupMarch 20153

What is Trove?Database as a Service for OpenStackAPIs for both development and operationsSelf service database provisioningFull database lifecycle managementMulti-database supportBoth Relational and NoSQL

4A Deep Dive into Trove OpenStack MeetupMarch 2015

What is Trove? RDS for OpenStackand DynamoDB and RedShiftAmazon AWSOpenStackEC2SwiftS3DynamoDBRedShiftCinderEBSNovaDeveloper APIDeveloper APIOperations APIAmazon Internal APITroveRDBMSNoSQLDWRDSand APIs for DevOPs5A Deep Dive into Trove OpenStack MeetupMarch 2015

What is Trove? The Trove MissionOpenStack Trove mission statement:To provide scalable and reliable Cloud Database as a Service provisioning functionality for both relational and non-relational database engines, and to continue to improve its fully-featured and extensible open source framework.

6A Deep Dive into Trove OpenStack MeetupMarch 2015

TuningAutomatically tune my.cnfBuffer Pool SizeLog file sizemax_connectionsSane defaultsInnoDB onlyDisable load data infileDisable select into outfileAPI to programmatically set configuration groupsManagementCreate database / schemaCreate usersGrant permissions to a SchemaEnable root userResize flavorResize volumeFull and incremental backupsCreate replicaWhat is Trove? More than just provisioningSecuritySecurity groupsTurn off SSHRemove anonymous userRemove non-localhost usersRemove local file accessMangle root user passwordApply security patches automatically7A Deep Dive into Trove OpenStack MeetupMarch 2015

Trove In Production TodayPublic CloudHP Cloud Relational DatabaseLaunched May 2012

Rackspace Cloud Databases Launched August 2012Private Cloud: eBayBegan mid 2013Multiple DatabasesMySQL, MongoDB, Redis Cassandra, CouchbaseMulti-region + HAWorking on Clustering

Key Use CasesDevelopment & testWeb application hostingOn-demand analytics

Critical CapabilitiesSelf-service provisioning & managementFleet wide configurationMulti-datastore architecture

8A Deep Dive into Trove OpenStack MeetupMarch 2015

What is Trove? OpenStack Architecture

OpenStackTroveA Deep Dive into Trove OpenStack MeetupMarch 2015

Datastore agnostic code in Trove Controller & Dashboard

Datastore specific code isolated to Guest Agents

What is Trove? Multi-Datastore ArchitectureTrove ControllerGuest Agent

Guest Agent

Guest Agent

Guest Agent

Trove Dashboard (Horizon)

Guest Agent

Guest Agent

Guest Agent

Guest Agent

Guest Agent

Guest Agent

Guest Agent

A Deep Dive into Trove OpenStack MeetupMarch 201510

10trove/guestagent/strategiesbackup/[]replication/restore/storage/

trove/guestagent/[]backup/common/datastore/[]strategies/[]

trove/guestagent/strategies/backupbase.pybase.pyccouchbase_impl.pycouchbase_impl.pyc__init__.py__init__.pycmysql_impl.pymysql_impl.pycpostgresql_impl.py

grep class guestagent/strategies/backup/mysql_impl.py

class MySQLDump(base.BackupRunner):class InnoBackupEx(base.BackupRunner):class InnoBackupExIncremental(InnoBackupEx):

Trove Architecture: Code ModularityA Deep Dive into Trove OpenStack MeetupMarch 2015

Lets shift gears for a couple of minutes and talk briefly about the architecture from a different perspective.

Trove must be able to address many different kinds of datastores. Some are NoSQL, some are SQL (and Im playing with a graph database as well). Each of these implements a certain capability in a different way. Lets talk about backup and restore as an example.

Once you get the software, in the trove/guestagent directory, youll find a directory called strategies. And under strategies, one called backup, and under backup youll see the actual implementation for the specific datastore. Now looking at the MySQL implementation, youll see that there are two backup capabilities (full backup) and one incremental backup method.

At configuration you can specify which implementation you would like to use. And you could implement a new strategy for your datastore thereby making Trove extensible and modular.

11

Using OpenStack TroveGetting started with OpenStack TroveAs a Trove userOpenStack distribution that includes Trove (such as HP Helion Dev Platform)http://www8.hp.com/us/en/cloud/hphelion-openstack.htmlTesora DBaaS platform, a Trove packaging tailored for the enterprisehttp://tesora.com/download-tesora-dbaas-platform-community-edition

As a Trove developer$ git clone http://github.com/openstack/trove-integration$ cd trove-integration/scripts$ ./redstack install$ ./redstack kick-start mysql

On top of DevStackAdd to localrc: ENABLED_SERVICES+=,trove,tr-api,tr-tmgr,tr-condSwift should also be enabled for Backup and Restore.ENABLED_SERVICES+=,s-proxy,s-object,s-container,s-accountSWIFT_HASH=A Deep Dive into Trove OpenStack MeetupMarch 201513

Provisioning a database instance with Trove$ trove create --size

Support for flavorsSupport for volumes using CinderOptional parameters to createImage per datastore type and versionSupport AZs using --availability_zoneSupport for Neutron using --nicA Deep Dive into Trove OpenStack MeetupMarch 201514

{ "instance": { "status": "BUILD", "updated": "2014-06-05T19:33:46", "name": "test", "links": [ { "href": "https://region-a.geo-1.database.hpcloudsvc.com/v1.0/39745227274204/instances/ce629494-a64d-41ed-b73c-04cb28bb33bb", "rel": "self" }, { "href": "https://region-a.geo-1.database.hpcloudsvc.com/instances/ce629494-a64d-41ed-b73c-04cb28bb33bb", "rel": "bookmark" } ], "created": "2014-06-05T19:33:46", "id": "ce629494-a64d-41ed-b73c-04cb28bb33bb", "flavor": { "id": "1002", "links": [ { "href": "https://region-a.geo-1.database.hpcloudsvc.com/v1.0/39745227274204/flavors/1002", "rel": "self" }, { "href": "https://region-a.geo-1.database.hpcloudsvc.com/flavors/1002", "rel": "bookmark" } ] }, "datastore": { "version": "5.5", "type": "mysql" } }}

14Managing a database with TroveResize flavor

Resize volume

Datastore specific extensions:Create Database / SchemaCreate UsersGrant Users Permissions on Databases

Enable a Root UserA Deep Dive into Trove OpenStack MeetupMarch 201515

$ trove backup-create

Optional params:Description through --descriptionIncremental backups using parent

To Restore backup use create:$ trove create --size --backup

Fully managed

Triggered and tracked via API

Streamed to Swift (OpenStack Object Storage)

Incremental & full backups

Multiple formats per datastore supported via strategies:XtraBackup (Percona)mysqldump

Backup and RestoreA Deep Dive into Trove OpenStack MeetupMarch 201516

OpenStack Trove: Completed in JunoAsync MySQL replication (master-slave)Clusters for MongoDBNeutron SupportSupport for PostgreSQLConfig-groups enhancementsConfiguration groups per datastore / versionConfig-groups for MongoDBBackups for Cassandra and CouchbaseAdditional Tempest TestsA Deep Dive into Trove OpenStack MeetupMarch 201517

New in Juno: Replication$ trove create --size --replica_of

Support for async MySQL replication (MySQL slave instances)

Manual detach using$ trove update --detach-replica-source

A Deep Dive into Trove OpenStack MeetupMarch 201518

New in Juno: Clusters$ trove cluster-create

Optional parameters to cluster-create--instance Specify multiple times to create multiple instances for your cluster

Initial support in Juno added for MongoDB ClustersSets up mongo config server, and mongo query routersTransparent and driven by configuration options Support for adding shards to existing cluster for horizontal scale out.A Deep Dive into Trove OpenStack MeetupMarch 201519

Planned for KiloBuilding out clusters Semi Synchronous MySQL clusters (Galera)Async Replication v2 GTID based replication Manual failover supportAssociate flavors with datastoresAccess datastore logs via APIRemoving deprecated oslo-incubator codeUpgrade testing through grenadeA Deep Dive into Trove OpenStack MeetupMarch 201520

20Where can I get OpenStack Trove?Get the sourceOpenStack Trove Projecthttps://git.openstack.org/cgit/openstack/trove / http://github.com/openstack/trove.git

Trove python client binding and command line clienthttps://git.openstack.org/cgit/openstack/python-troveclient / http://github.com/openstack/python-troveclient.git

Trove design specifications for blueprintshttps://git.openstack.org/cgit/openstack/trove-specs / http://github.com/openstack/trove-specs

Trove scripts for installation and testing, and elements for building guest imageshttps://git.openstack.org/cgit/openstack/trove-integration / http://github.com/openstack/trove-integration.git

Installable packages and guest images from TesoraCommunity Edition: http://tesora.com/download-tesora-dbaas-platform-community-editionEnterprise Edition trial: http://tesora.com/download-tesora-dbaas-platform-enterprise-edition

A Deep Dive into Trove OpenStack MeetupMarch 2015

More about TroveTrove Wikihttps://wiki.openstack.org/wiki/TroveTrove Sourcehttps://git.openstack.org/cgit/openstack/troveOn IRC#openstack-troveTrove DayTesora.com/trovedaySlideshare.net/TesoracorpLinkedInOpenStack Trove Group

Contact informationContact Tesorainfo@tesora.comwww.tesora.com@tesoracorpContact Kenkrugg@tesora.comwww.tesora.com@kenruggA Deep Dive into Trove OpenStack MeetupMarch 201522

Thank You!