Scaling mysql with python (and Docker)

Scaling MySQL with pythonEuroPython 2015, Bilbao

Roberto Polli - [email protected]

Par-Tec Spa - Rome Operation UnitP.zza S. Benedetto da Norcia, 33

00040, Pomezia RM - www.par-tec.it

20 July 2015


mailto:[email protected]


Agenda

IntroMySQLPython UtilitiesConnectors

ReplicationFailoverFabricProvisioningFabric in the Cloud



Who? What? Why?

Roberto Polli - Solutions Architect @ par-tec.it. Loves writing in C, Java andPython. Red Hat Certified Engineer and Virtualization Administrator.Par-Tec – Proud sponsor of this talk ;) Contributes to various FLOSS.Provides:

• IT Infrastructure & Services expertise;• Business Intelligence solutions;• Vertical Applications for the financial market.

Manage, replicate, scale MySQL is easy with python (and Docker).

Intro Roberto Polli - [email protected]


MySQL

• Single-process, multi-thread• 2-tier architecture

• Connection + SQL• Storage Backend

• Greatly improved in 5.6• More to come in 5.7: Dynamic Buffers & Multi-source replication

MySQL Roberto Polli - [email protected]


MySQL

Many Backends aka Storage Engines:• InnoDB1: ACID, MVCC2, Redo|Undo logs;• Memory: fast & volatile;• NDB: MySQL Cluster, network distributed, auto-sharded;

Replication based on changelog files (binary logs).

1default2Multi Versioning Concurrency Control



Manage & Use a database

Monitor:• Database size: Tables, Indexes,

Binary Logs• Replication inconsistencies• Failover

Connect:• Native Drivers

• for Python 2.6, 2.7 and 3.3• SSL & compression

• OrchestrateDownload scripts from some blog, add random queries from docs.mysql.com,Shake.



MySQL Utilities

A python package with utilities, drivers, fabric orchestrator

$ wget http://bit.ly/1CxNuZe$ tar xf mysql-utilities-1.6.1.tar.gz$ cd mysql-utilities-1.6.1$ python setup.py install

mysql-utilities-1.6.1|-- mysql|-- connector| |-- django| ‘-- fabric|-- fabric| |-- protocols| |-- providers| ‘-- services‘-- utilities

Python Utilities Roberto Polli - [email protected]


MySQL Utilities:

Start with mysqluc for a single entrypoint• entrypoint for all utilities• contextual help• TAB completion

Or access each utility separately.Specify server URIs as user:pass@host[:port].



MySQL Utilities

mysqluc> help utilities

Utility Description----------------- --------------------------------------------------------mysqlauditadmin audit log maintenance utilitymysqlauditgrep audit log search utility...mysqldbcompare compare databases for consistency...mysqldiskusage show disk usage for databasesmysqlfailover automatic replication health monitoring and failover



Disk Usage

A single command to show all disk usage infos (excluded system logs)$ mysqldiskusage --all --server=$SERVER...| db_name | total || akonadi | 969150919 |...Total database disk usage = 971233529 bytes or 926,24 MB...| ib_logfile0 | 67108864 || ib_logfile1 | 67108864 || ibdata1 | 44040192 |Total size of InnoDB files = 178257920 bytes or 170,00 MB



Connectors

from django.db.backends import BaseDatabaseValidation

if django.VERSION < (1, 7):from django.db import models

else:from django.core import checksfrom django.db import connection

class DatabaseValidation(BaseDatabaseValidation):if django.VERSION < (1, 7):

Connectors Roberto Polli - [email protected]


Log-based Replication: Master

Replication is asynchronous or semi − synchronous.A Master with a Unique Server ID

• produces a changelog named binary log;• assign each transaction a Global Transaction ID;• grants access to a replica user;

Optionally:• stores replication infos in ACID tables;• may track slave-updates;• waits for one slave ack on semi − synchronous replication;

Replication Roberto Polli - [email protected]


Log-based Replication: Slave

Slave• connects to the master as the replica user;• retrieves the binlog;• applies the changes;• ∀ slave ∃! master;

Optionally• forbids local changes from non-root users;

If binlogs have been purged, you need to import the master database first!Replication Roberto Polli - [email protected]


Benefits of replication

• Availability.• Scaling reads with R/W split.• Scaling reads differencing indexes.• Backup and data warehouse

strategies.

# mysqlrplshow output## Replication Topology Graphs-1.docker:3306 (MASTER)

|+--- s-3.docker:3306 - (SLAVE)|+--- s-4.docker:3306 - (SLAVE)



mysqlreplicate

Configuring replication with one command:mysqlreplicate -b --pedantic \

--master=$MASTER --slave=$SLAVE --rpl-user=repl:rpass

# master uuid = 2cda700...# slave uuid = 7520cf0...# Checking for binary logging on master...# Granting replication access to# replication user...# Connecting slave to master...# CHANGE MASTER TO MASTER_HOST = ’172.17.0.5’,...

• preliminary checks;• create replica user on the master;• point the slave to the master;• start loading the first available

transaction in binlogs;



Initialize new slavesBeware!

1. Avoid replicating user provisioning and other host-related [email protected]_LOG_BIN=0;

2. Binlogs are usually retained for a fixed period - eg 15days.To provision new slave after that period you must:

• start from a backup or a snapshot;• apply the binlogs.

mysqldbexport >> data.sql --server=root:pass@master --rpl-user=repl:rpass \--export=both --rpl=master --all

mysqldbimport --server=root:pass@slave data.sql



Failover BasicsFrom replication to high availability, automagically discovering slaves.

• promote the most updated slave• reconfigure the others• disable the master• eventually fix ip-address

# Read carefully the docs of# mysqlfailover and of all# the supported parameters!mysqlfailover --master=$MASTER \--discover-slaves-login=root:password \--candidates=$SLAVE1,$SLAVE2 \--exec-before=/pre-fail.sh \--exec-after=/post-fail.sh

Failover Roberto Polli - [email protected]


Fabric Orchestrator

You can try the content of those slides downloading code and baked dockerimages from here:

git clone https://github.com/ioggstream/mysql-communitycd mysql-community/fabricdocker-compose scale fabric=1 slave=4

Let me know after the talk!

Fabric Roberto Polli - [email protected]


Fabric - HLA

A python framework for managing, replicating and scaling mysql servers.• aggregate servers in high availability groups• configure single-master replication topologies• monitor and heal failures• director for rw/split• fabric connectors (caching db topology)



Fabric - HLA



mysqlfabric

mysqlfabric commands:• manage - setup and manage fabric service• group - administer high availability groups• server - manage, clone, provision and decommission servers• provider - manage server providers (eg. openstack, ...)

Further functionalities:• statistics, dump, role, user,• threat, snapshot, event



ManageSetup and administration is done viamysqlfabric manage [setup|start|ping|stop|teardown]

Configure in /etc/fabric.cfg

• listening ports for xmlrpc service• admin user (eg. fabric to manage servers)• applicative user credentials• db to store fabric data

Setup and start the servicemysqlfabric manage setupmysqlfabric manage start --daemon



Replication Groups

Groups are managed viamysqlfabric group [create|add|promote|activate] $GROUP

Crete an High Availability group and add a bunch of servers.

mysqlfabric group create $GROUPmysqlfabric group add $GROUP my-host-1:3306mysqlfabric group add $GROUP my-host-2:3306mysqlfabric group add $GROUP my-host-3:3306



Replication Groups

Promote one server as a master, eventually picking one.mysqlfabric group promote $GROUP [--slave_id=UUID]

Show the server groupmysqlfabric group lookup_servers haFabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249eTime-To-Live: 1server_uuid address status mode weight------------ ----------- --------- --------- ------2cda..110015 172.17.0.21 PRIMARY READ_WRITE 1.02cda..110016 172.17.0.22 SECONDARY READ_ONLY 1.02cda..110017 172.17.0.23 SECONDARY READ_ONLY 1.0



Replication GroupsFabric support spare servers.mysqlfabric server set_status $UUID spare

Monitoring failover and deactivating the master.mysqlfabric group activate $GROUPmysqladmin -h $MASTER shutdown

Checking group health we’ll see that...mysqlfabric group health ha

uuid is_alive status ..error status..io_error sql_error----------- -------- --------- --- --- --- --- -------- ---------2cd..110015 0 FAULTY 0 0 0 0 False False2cd..110016 1 SECONDARY 0 0 0 0 False False2cd..110017 1 PRIMARY 0 0 0 0 False False



Connecting programmatically

from mysql.connector import connect, fabric# Ask for a connection to the fabric server.c = connect(fabric={host: .., port: .., user: ..},

autocommit=True,database=’sample’, **sql_user)

# Fabric will point you to a suitable host# - in this case the master - of the given# groupc.set_property(mode=fabric.MODE_READWRITE, group="my-cluster-1")



Provisioning a new slave

Fabric uses mysqldump + mysql internally to clone a new slave.# Always reset TARGET configuration# before reinitializingmysql -e ’SET @SESSION.SQL_LOG_BIN=0;STOP SLAVE;RESET MASTER;’

Cloning doesn’t attach the server to a group, nor starts the replica.mysqlfabric server clone $GROUP $TARGET

Provisioning Roberto Polli - [email protected]


Reingesting a failed master

Reingesting a failed master is not trivial.• non-replicated transactions;• corrupt data;

Always reset it!mysqlfabric server set_status f484...110039 sparemysqlfabric server set_status f484...110039 secondary

Provisioning Roberto Polli - [email protected]


Fabric in the CloudFabric can provision new machines via eg. Openstack API.We implemented a DockerProvider: deploy containers, not machines.

# mysql.fabric.providers.dockerproviderclass MachineManager(AbstractMachineManager):

"""Manage Docker Containers."""def create(self, parameters, wait_spawning):

...def search(self, generic_filters, meta_filters):

...def destroy(self, machine_uuid):

...

Fabric in the Cloud Roberto Polli - [email protected]


Docker Provisioning# Register a provider (requires docker in http)mysqlfabric provider register mydocker

user passwordhttp://172.17.42.1:2375--provider_type=DOCKER

# Create or remove Containersmysqlfabric server create mydocker

--image=name=ioggstream/mysql-community--flavor=name=v1--meta=command="mysqld --log-bin=foo"

# List serversmysqlfabric server list docker



Next steps

Openstack interface supports machine snapshots via NovaClient.Docker volumes doesn’t support snapshots, but something is moving (in theexperimental tree).VM Snapshot alternatives require root access to docker host and a FLUSHTABLES on the source container:

• rsync on volumes• implement snapshot via docker volume plugins



Wrap Up

• Replication is easier with Fabric and MySQL 5.6• You can clone servers• Failover is just one command ahead• Don’t re-ingest failed masters (luckily;)• Try Fabric with Docker!• Play with docker volumes



That’s all folks!

Thank you for the attention!Roberto Polli - [email protected]




Internet

Scaling mysql with python (and Docker)