MariaDB for DBAs · PDF fileMariaDB for DBAs Installation Introduction Architecture Installation Client Utilities Conﬁguration Resource Usage Storage Engines Table Schema

MariaDB for DBAsMariaDB Training

MariaDB for DBAs

Introduction

IntroductionArchitecture

Installation

Client Utilities

Configuration

Resource Usage

Storage Engines

Table Schema

Views

Partitioning

User & Server Security

Log Files

Monitoring & Troubleshooting

Back-Ups & Restoration

Customizing

MariaDB Replication

High Availability

Optimization

[email protected]

Version 3.3, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB for DBA's

Introducing MariaDB AbFounders from MySQL the

Company and Community Funded by Founders, Employees,

and Venture Capital Over 100 Employees Several former MySQL Employees

and Community Members in over 14 Countries

3

[email protected]


Personal IntroductionsInstructor

Name and Background

Participants Name and Company MariaDB Experience How You Use MariaDB Needs Related to Course Topics

4

[email protected]


Class Schedule & Personal ConcernsStarting and Ending Times Planned Breaks On- Site

Location of Rest Rooms Smoking Areas Snacks and Drinks

LVC Classes Chat with Everyone

5

[email protected]


Course OutlineOverview & Server

Architecture Installation Client Utilities Configuration Resource Usage

Schema Choices Storage Engines Table Schema Views Partitioning

Core Admin. Tasks User & Server Security Logs Files Monitoring &

Troubleshooting Back-Ups & Restoration

Advanced Topics Customizing MariaDB Replication High Availability Optimization

6

MariaDB for DBAs

Architecture

Introduction

ArchitectureInstallation

Client Utilities

Configuration

Resource Usage

Storage Engines

Table Schema

Views

Partitioning


Log Files



Customizing

MariaDB Replication

High Availability

Optimization

[email protected]


MariaDB Terrain

mysql

application (e.g., php)

mysqladmin

mysqld

Archive

InnoDB

MyISAM

cron

DBA

web user

socket file

tcp/ip

socket file

ssh & tcp/ip

Memory

SQL Tier — ClientsStorage Engine

TierServer Tier

Handler Interface

8

[email protected]


Client ConnectionsTCP/IP Connections available on All Platforms

--skip-networking (disables)

Socket Files available on Unix Platforms - fastest choice Named Pipe and Shared Memory available on Windows

--enable-named-pipe, --shared-memory

MariaDB Connections require little resources and easy to open. Most use External Connection Pools (not needed usually) Set Global Client Connection Limit (max_connections=n)

9

[email protected]


Client Libraries and the Like

Information on MariaDB Connectors: https://mariadb.com/products/connectors-plugins

Various Libraries for connecting to MariaDB — Most are Wrappers for MariaDB C API, which uses MariaDB Network Protocol

PHP API, Perl DBI, Python, Ruby MariaDB Connector for Java Native C/C++ driver MariaDB ODBC Drive Embedded MariaDB (libmysqld)

10

https://mariadb.com/products/connectors-plugins

[email protected]


MariaDB Query Process

Thread Cache

Archive

InnoDB

Aria

Memory

Storage Engines

Query Cache

SQL Parser

Optimizer

Clients

11

[email protected]


Thread Cache

Thread Cache

Query Cache

SQL Parser

Optimizer

Storage Engines

Thread is Assigned to Each Connection

Threads may be Reused from the Thread Cache or Created

12

[email protected]


Thread Pool

Thread Pool

Query Cache

SQL Parser

Optimizer

Storage Engines

Main Thread Listens for Client Connections User Authentication based on Host, User, and

Password Client Buffers for Session Variables and Network

Communications

13

thread_handling = pool-of-threads thread_pool_size = 3

Excerpt from /etc/my.cnf.d/server.cnf

Documentation on Thread Pool: https://mariadb.com/kb/en/mariadb/thread-pool-in-mariadb/#using-thread-pool-scheduler

https://mariadb.com/kb/en/mariadb/thread-pool-in-mariadb/#using-thread-pool-scheduler

[email protected]


Query Cache

Thread Cache

Query Cache

SQL Parser

Optimizer

Storage Engines

Documentation on Query Cache: https://mariadb.com/kb/en/mariadb/query-cache/

SHOW VARIABLES LIKE 'query_cache_type';

Possible to Stipulate in a Query

Stores SELECT Query Result Sets

Useful for High Read, Low Write Servers

Cache Purged when Related Data Changed

SELECT SQL_CACHE * FROM ...

SELECT SQL_NO_CACHE * FROM ...

14

No Longer Recommended

https://mariadb.com/kb/en/mariadb/query-cache/

[email protected]


SQL Parser

Thread Cache

Query Cache

SQL Parser

Optimizer

Storage Engines

Converts SQL Text to Binary Format Parses SQL into Tokens (i.e., keyword, table, field, value) Applies Grammar Rules to check Validity (Lexical

Scanner, Grammar Rules Module) Construct a Parse Tree to be passed to Optimizer

15

[email protected]


Optimizer

Thread Cache

Query Cache

SQL Parser

Optimizer

Storage Engines

The Optimizer Reads the Parse Tree and Determines an Execution Plan

Locate any Related Indexes Compare Efficiency of Index Access to Table Scan Determine JOIN order of Table

Eliminate Unnecessary Tables and WHERE clauses

Find Indexes that can handle GROUP BY and ORDER BY

16

MariaDB for DBAs

Installation

Introduction

Architecture

Installation

Client Utilities

Configuration

Resource Usage

Storage Engines

Table Schema

Views

Partitioning


Log Files



Customizing

MariaDB Replication

High Availability

Optimization

[email protected]


Installation Methods & PackagesInstallation Tools (MariaDB, Oracle, Percona)

Built and Tested by Experts Installer (e.g., yum) Use Up-to-Date Versions

Distribution Packages (rpm, deb, pkg, dmg)

Easy and Quick to Install Need to Configure Manually May be Old Versions

Build from Source — Useful for Custom Patches

Required for Non-Standard Platforms

18

[email protected]


Installing with Binary Files

Download MariaDB: https://downloads.mariadb.org/mariadb/ Repository Configuration Tool: https://downloads.mariadb.org/mariadb/repositories/

Package Managers rpm -Ivh yum install dpkg -i apt-get install

Configuration File (datadir, etc.)

Execute manually mysql_install_db

InnoDB Log File Sizes in my.cnf

Service Scripts (i.e., /etc/init.d/mysql start|stop|restart)

19

https://downloads.mariadb.org/mariadb/

https://downloads.mariadb.org/mariadb/repositories/

[email protected]


Basics Steps to Install Tar FilesDownload and Extract Tar Zip File Configuration File (/etc/my.cnf or c:\my.ini) Add System User, mysql

Data Directory (owner, mysql)

Install the System Tables (scripts/mysql_install_db)

Install the mysql.server script (e.g., /etc/init.d)

MariaDB binaries in $PATH

Start Server Run mysql_secure_installation

20

[email protected]


Installing with Source FilesSystem Build Environment Install a Compiler (e.g., gcc), Libraries, Headers, etc. Build Scripts for various Platforms (see ./build/) Unix — Normal Configure and make Process (./configure --help)

Windows — get Visual C++ 6.0 or Higher Configure System to use Same Process as Binary

21

[email protected]


Upgrade PlanConsideration

Review Change Logs (major and minor version upgrades) Major Upgrade is Safest — with Backup and Reload

MySQL 5.1 to 5.5, 5.6; MySQL 5.5 to MariaDB 10.1; MySQL 5.5 to MariaDB 5.5

Minor Upgrade (mysql_upgrade) Required Downtime

Preparation Back-Up Data Update Applications and Libraries Use a Test Server Use Replication — Upgrade Slave, Promote It, then Upgrade Master

22

MariaDB for DBAs

Client Utilities

Introduction

Architecture

Installation

Client UtilitiesConfiguration

Resource Usage

Storage Engines

Table Schema

Views

Partitioning


Log Files



Customizing

MariaDB Replication

High Availability

Optimization

[email protected]


mysql Client — Command Line Tool (CLI)

shell> mysql --user root -p

mysql> SHOW DATABASES;

Shell or ssh

Execute Manually any SQL statement Display mysqld settings and status counters

Use as Interactive Session, or Pipe SQL via Shell

24

mysql -p -u user_name --execute "SHOW DATABASES"

[email protected]


mysqladmin — Command Line Tool (CLI)

Documentation on mysqladmin: https://mariadb.com/kb/en/mariadb/mysqladmin/

mysqladmin processlist mysqladmin -u root -p create employees mysqladmin extended-status

Shell or ssh

Manage User Accounts, Passwords, Permissions Display mysqld settings and status counters View and Kill Active Connections

Ping or Shutdown mysqld Create and Drop Databases

25

https://mariadb.com/kb/en/mariadb/mysqladmin/

[email protected]


Other Client Utilitiesmysqldump mysql_upgrade

my_print_defaults mysqlslap

sysbench

mysqlcheck

Percona Toolkit

26

Documentation on mysqldump: https://mariadb.com/kb/en/mariadb/mysqldump/ Documentation on mysql_upgrade: https://mariadb.com/kb/en/mariadb/mysql_upgrade/ Documentation on my_print_defaults: https://mariadb.com/kb/en/mariadb/my_print_defaults/ Documentation on mysqlslap: https://mariadb.com/kb/en/mariadb/mysqlslap/ Documentation on mysqlslap: https://mariadb.com/kb/en/mariadb/mysqlcheck/ Link to Percona Toolkit: https://www.percona.com/software/database-tools/percona-toolkit

https://mariadb.com/kb/en/mariadb/mysqldump/

https://mariadb.com/kb/en/mariadb/mysql_upgrade/

https://mariadb.com/kb/en/mariadb/my_print_defaults/

https://mariadb.com/kb/en/mariadb/mysqlslap/

https://mariadb.com/kb/en/mariadb/mysqlcheck/

[email protected]


Other Command-Line Tools

* Caution: These MyISAM Tools Directly Access the Files

mysql.server

mysqld_safe

mysql_fix_privilege_tables

mysql_install_db

mysql_tz_to_sql

myisampack *

mysqlbinlog

mysqldumpslow

myisamchk *

pt-query-digest

innochecksum

27

Documentation on mysql.server: https://mariadb.com/kb/en/mariadb/mysqlserver/ Documentation on mysqld_safe: https://mariadb.com/kb/en/mariadb/mysqld_safe/ Documentation on mysql_install_db: https://mariadb.com/kb/en/mariadb/mysql_install_db/ Documentation on mysqldumpslow: https://mariadb.com/kb/en/mariadb/mysqldumpslow/ Documentation on pt-query-digest: https://www.percona.com/doc/percona-toolkit/2.2/pt-query-digest.html

https://mariadb.com/kb/en/mariadb/mysqlserver/

https://mariadb.com/kb/en/mariadb/mysqld_safe/

https://mariadb.com/kb/en/mariadb/mysql_install_db/

https://mariadb.com/kb/en/mariadb/mysqldumpslow/

https://www.percona.com/doc/percona-toolkit/2.2/pt-query-digest.html

[email protected]


HeidiSQL

Site for HeidiSQL: http://www.heidisql.com/

GUI Application (MS Windows)

Connect via SSL or SSH tunnel

Schema and Query Construction Optimization and Profiling User Account Management Task and Backup Scheduling Data Migration and Cleaning Schema Sync and Diff Tools Notifications

28

http://www.heidisql.com/

[email protected]


SQLyog

Site for SQLyog: https://www.webyog.com/product/sqlyog

Graphical Interface for Executing Queries

Schema Changes Make Back-Ups Bulk Import Data

29

https://www.webyog.com/product/sqlyog

MariaDB for DBAs

Configuration

Introduction

Architecture

Installation

Client Utilities

ConfigurationResource Usage

Storage Engines

Table Schema

Views

Partitioning


Log Files



Customizing

MariaDB Replication

High Availability

Optimization

[email protected]


Configuration File LocationsLinux Systems /etc/my.cnf

/etc/my.cnf.d/*.cnf

--defaults-extra-file

~/.my.cnf (clients)

Windows c:\Windows\my.ini

c:\my.ini

%PROGRAMDATA%\MySQL\MySQL Server x.x\my.cnf

%WINDIR%\my.cnf

INSTALLDIR\my.cnf

--defaults-extra-file

31

[email protected]


Server Defaults

[mysqld] datadir=/data/mysql socket=/var/lib/mysql/mysql.sock user=mysql symbolic-links=0

[mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid

!includedir /etc/my.cnf.d

Configuration File (my.cnf):

mysqld --print-defaults

mysqld would have been started with the following arguments:

--datadir=/data/mysql --socket=/var/lib/mysql/mysql.sock --user=mysql --symbolic-links=0

Finding Defaults:

32

[email protected]


Global Variables

SET GLOBAL tmp_table_size = 32*1024*1024;

Can be Changed with SET Statement

Set in Configuration File at Start

Requires SUPER privilege for mysql.plugin table

SHOW GLOBAL VARIABLES;

Global Variables are System Wide

33

[email protected]


Global StatusSHOW STATUS provides Counters and

Thresholds of Internal Statistics

Check Status Variables to Decide on Adjustments to Global Variables

SHOW STATUS LIKE '%innodb_buffer_pool%';

34

[email protected]


Session Variables

SHOW SESSION VARIABLES LIKE '%character_set_client%';

+----------------------+-------+ | Variable_name | Value | +----------------------+-------+ | character_set_client | greek |

+----------------------+-------+

SHOW GLOBAL VARIABLES LIKE '%character_set_client%';

+----------------------+--------+ | Variable_name | Value | +----------------------+--------+ | character_set_client | latin1 | +----------------------+--------+

Session Variables are Values for Current Connection Global Variable Values is Default for Session

35

MariaDB for DBAs

Resource Usage

Introduction

Architecture

Installation

Client Utilities

Configuration

Resource Usage

Storage Engines

Table Schema

Views

Partitioning


Log Files



Customizing

MariaDB Replication

High Availability

Optimization

[email protected]


Global Memory

37

Global Memory Buffers are Allocated once when mysqld Starts table_open_cache table_definition_cache query_cache_size thread_cache

Permissions Tables Global Memory Buffers are also Allocated for Storage Engines

MyISAM: key_buffer_size InnoDB: innodb_buffer_pool_size, innodb_additional_mem_pool_size, innodb_log_buffer_size

Storage Engines may Allocate internally Other Memory

[email protected]


Session Memory

38

Buffers Allocated as required for each Client Connection Some Allocated more than once for Joins and Sorting Released when Query is Finished or Client Session Closed Persistent Connections need to be Reset Client Session Memory Variables:

thread_stack read_buffer_size join_buffer_size bulk_insert_buffer_size

binlog_cache_size net_buffer_length

read_rnd_buffer_size max_heap_table_size max_allowed_packet sort_buffer_size

tmp_table_size myisam_sort_buffer_size

[email protected]


Estimating Memory Usage

39

(sum(session_buffers) * max_connections)

+ sum(global_buffers) = total memory usage

Worst Case, Maximum Memory Usage Scenario:

Chances of All Clients Allocating All Possible Session Buffers at once is usually Small

Normal Practice is to Commit Over Memory Requirements

Out-of-Memory (OOM) may occur if mysqld is near System Memory Limits

Add Memory or Use a Crash-Safe Storage Engine

Use SWAP space, but Avoid letting MySQL Swap for Normal Load

Basic Formula for Estimating Memory Usage

[email protected]


Temporary Disk Space

40

MariaDB Uses Temporary Disk Space Large Implicit Temporary Tables for Resolving Queries Some Sort Operations (ORDER BY, GROUP BY)

tmpdir defaults to system temporary location Some Unix platforms mount /tmp as tmpfs

slave_load_tmpdir for Replicating LOAD DATA INFILE Defaults to Location of tmpdir — Never use tmpfs Slave Exports from binlog to here, then LOAD DATA

INFILE

[email protected]


Disk Speed

41

Disk Speed is important when a Data Set won’t Fit in Memory

Disk Reads are Alleviated by Caching Some Storage Engines Implement their own

Caches Some Rely on OS Disk Caching Mechanism

[email protected]


Disk Redundancy — Preparing for Disk Failure

42

RAID, SAN or other Redundant, Distributed Storage Solutions

Good Disk Failure Protection

MariaDB Replication Slave

Because Asynchronous, will lose some data (see sync_binlog)

DRBD Slower Writes

[email protected]


File Handles

43

Logs and Replication Files need to be Open Client Connections each need a File (socket = file_handle)

Storage Engines may have Specific Needs: MyISAM uses a file per table.frm, and multiple for

table.MYI and table.MYD InnoDB needs only a few for Global Table-Space and

Logs (unless using innodb_file_per_table)

Implicit Disk-Based Temporary Tables are Files SELECT INTO OUTFILE uses a file

[email protected]


Disk Data Directory

44

[email protected]


Disk Data Directory

45

[email protected]


Thread Cache

46

SHOW GLOBAL STATUS LIKE 'threads%';

+-------------------+-------+

| Variable_name | Value |

+-------------------+-------+

| Threads_cached | 0 |

| Threads_connected | 3 |

| Threads_created | 604 |

| Threads_running | 1 |

+-------------------+-------+

Use the SHOW STATUS for Thread Cache information:

SHOW VARIABLES LIKE 'thread_cache%';

+-------------------+-------+


+-------------------+-------+

| thread_cache_size | 0 |

+-------------------+-------+

Use the SHOW VARIABLES for Thread Cache Variables:

[email protected]


Server CPU

47

Since mysqld is multi-threaded, it can use many CPU cores

A Client Connection is a single thread — a Query uses only One Core

Query Concurrency is very Schema and Traffic Dependent

MariaDB for DBAs

Storage Engines

Introduction

Architecture

Installation

Client Utilities

Configuration

Resource Usage

Storage EnginesTable Schema

Views

Partitioning


Log Files



Customizing

MariaDB Replication

High Availability

Optimization

[email protected]


Storage Engine Basics

49

CREATE TABLE table1 (col1 INT) ENGINE = AriaDB;

ALTER TABLE table1 ENGINE = MyISAM;

Multiple Storage Engines Allowed They Determine Storage Medium (disk,

memory, etc.) Control Atomicity Consistency

Isolation Durability (ACID) Lock at certain Levels (table, page, row)

Some offer Special Features (Foreign Keys, GIS, ColumnStore)

Some Provide Optimization Storage Engine is Set for Table:

SHOW ENGINES;

[email protected]


Pluggable Storage Engines

[mysqld] plugin_dir=/usr/lib/mysql/plugin/ plugin_dir=/usr/lib64/mysql/plugin/

UNINSTALL PLUGIN BLACKHOLE;

Excerpt from my.cnf or my.ini configuration file

Requires INSERT privilege for mysql.plugin table

Documentation on INSTALL PLUGIN: https://mariadb.com/kb/en/mariadb/install-plugin/

INSTALL PLUGIN BLACKHOLE SONAME 'ha_blackhole.so';

There are Storage Engines with different Features and Purposes

Multiple Engines on a Server and within a Query are Permitted

Engines can be Installed and Removed

50

https://mariadb.com/kb/en/mariadb/install-plugin/

[email protected]


Advantages & Disadvantages of InnoDB

51

Read Uncommitted Read Committed Repeatable Read (no fantom rows) Serializable

Fully Transactional,

ACID Compliant

Data & Indexes Cached by mysqld

in Buffer Pool

Row-Level Locking for High-

Concurrency

Supports Four Isolation Levels

Supports Foreign Keys and Multi-

Version Concurrency

Control

Reliable Crash Recovery

Slower than MyISAM, but

Higher Concurrency

Advantages Disadvantages

Isolation Levels

Documentation for InnoDB & XtraDB: https://mariadb.com/kb/en/mariadb/xtradb-and-innodb/

https://mariadb.com/kb/en/mariadb/xtradb-and-innodb/

[email protected]


InnoDB Architecture

52

Additional Memory Pool

Buffer Pool *

Data Files * **

Redo Log (File 1/2)

Redo Log (File 2/2)

Memory

Disk

* Cached Data & Indexes, Stored in Pages ** Transaction Data Writes *** Data, Index, and Undo Log Files

INSERT, UPDATE, DELETE

1

COMMIT

CHECKPOINT

Log Buffer * *

2

3

[email protected]


InnoDB Clustered Index

53

root

node node

leafleaf leaf

PAGE (16KB) PAGE (16KB) PAGE (16KB)

B-Tree

HEADER PRIMARY KEY ROW DATA

leaf

[email protected]


InnoDB Secondary Index

54

root

node node

leafleaf leaf

PAGE (16KB) PAGE (16KB) PAGE (16KB)

B-Tree

HEADER VALUES OF INDEXED COLUMNS PRIMARY KEY

leaf

[email protected]


Pros & Cons of MyISAM

55

Documentation for InnoDB & XtraDB: https://mariadb.com/kb/en/mariadb/xtradb-and-innodb/

Indexes are Cached by mysqld in key_buffer Don't Disable — System Tables use MyISAM Storage Engine (InnoDB in Future)

Not Transactional

Table-Level Locking

Disadvantages

Poor Crash Recovery

(Relies on OS to Flush Data to

Disk)

Data Caching Relies on OS Disk Cache

High Read, Low Write

Traffic

Concurrent Writes

Advantages

Many Column Data Types

https://mariadb.com/kb/en/mariadb/xtradb-and-innodb/

[email protected]


Aria

56

Replacement for MyISAM Buffers Pages before Writing Rows

Index & Pages Cached (aria_pagecache_buffer_size)

Crash Safe — Statement Commit & Rollback (notes a log file)

Used for Temporary Tables Variable aria_used_for_temp_tables should be ON

Documentation for Aria: https://mariadb.com/kb/en/mariadb/aria-storage-engine/

https://mariadb.com/kb/en/mariadb/aria-storage-engine/

[email protected]


Memory

57

Temporary, but Fast Table Structure on Disk Data and Indexes in RAM and the Like Uses Table-Level Locking

Data won’t survive mysqld Restart — Schema Remains

Allows Only Fixed-Length Records — Variable-Length Columns Padded to Full Width

Some Data Types are Not Supported (e.g., VARCHAR(255), TEXT, BLOB)

Uses a Hash Index Type — Only Equality Comparisons (e.g., No LIKE Operator)

Documentation for Memory: https://mariadb.com/kb/en/mariadb/memory-storage-engine/

https://mariadb.com/kb/en/mariadb/memory-storage-engine/

[email protected]


Blackhole — /dev/null

58

Useful for Development Servers for Testing only Query Execution

Can be Used for Replication since Queries are written to Binary Log

Master can Replicate Data without Storing Locally Relay Slave can use Filters Rules Use on Slave to Prevent Storing Locally Replicated Data

Useful for Solving Performance Problems Remove Temporarily and Instantly Excessive Log Table

Activity Benchmark to determine Non-Storage-Engine Bottlenecks

Documentation for Blackhole: https://mariadb.com/kb/en/mariadb/blackhole/

https://mariadb.com/kb/en/mariadb/blackhole/

[email protected]


Archive

59

Useful for Storing Large Amounts of Archived Data or Logging Tables with Frequent Writes and Infrequent reads

Minimal Memory Footprint All Data is Compressed on Disk with zlib Supports only INSERT and SELECT statements Doesn’t support DELETE, UPDATE or REPLACE Indexes Are Not Stored No Data Caching, Except the OS Disk Cache Non-Transactional

Documentation for Archive: https://mariadb.com/kb/en/mariadb/archive/

https://mariadb.com/kb/en/mariadb/archive/

[email protected]


Spider

60

Partitions Large Tables across Multiple Servers Transparent — No Changes Necessary to Client Applications Allows for more Parallelism, Scale-out.

Parallel Partition Scanning — Make MariaDB Faster than Other RDBMS

ACID Compliant & Transactional Not Yet Production Ready Schema Creation & Changes Cumbersome

Documentation on Spider: https://mariadb.com/kb/en/mariadb/spider/

https://mariadb.com/kb/en/mariadb/spider/

[email protected]


ColumnStore (Columnar)

61

Massively Parallel, Distributed Data Engine Powerful Analytics Data Warehouse Fast Import (ETL)

cpimport

Columnar Architecture No Indexes Each Column is Stored Separately

WHERE Clauses Parsed Differently

Documentation on ColumnStore: https://mariadb.com/kb/en/mariadb/mariadb-columnstore/

https://mariadb.com/kb/en/mariadb/mariadb-columnstore/

[email protected]


Connect

62

Maps Diverse Data to Tables Flat Files including CSV Tables in External ODBC Databases

JOIN Mapped Data to Database Tables

Generated tables (PIVOT, etc.) Plug-in API for Customized Mapping

Documentation on Connect: https://mariadb.com/kb/en/mariadb/introduction-to-the-connect-engine/

https://mariadb.com/kb/en/mariadb/introduction-to-the-connect-engine/

[email protected]


Other Storage Engines

63

EXAMPLE — Example Storage Engine CSV — Comma-Separated Value Storage Engine FEDERATED — Federated Storage Engine

Documentation on EXAMPLE: https://mariadb.com/kb/en/mariadb/example-storage-engine/ Documentation on CSV: https://mariadb.com/kb/en/mariadb/csv-overview/ Documentation on FEDERATED: https://mariadb.com/kb/en/mariadb/federated-storage-engine/

https://mariadb.com/kb/en/mariadb/example-storage-engine/

https://mariadb.com/kb/en/mariadb/csv-overview/

https://mariadb.com/kb/en/mariadb/federated-storage-engine/

MariaDB for DBAs

Table Schema & Maintenance

Introduction

Architecture

Installation

Client Utilities

Configuration

Resource Usage

Storage Engines

Table SchemaViews

Partitioning


Log Files



Customizing

MariaDB Replication

High Availability

Optimization

[email protected]


Creating Databases & Tables

65

Documentation for CREATE DATABASE: https://mariadb.com/kb/en/mariadb/create-database/ Documentation for CREATE TABLE: https://mariadb.com/kb/en/mariadb/create-table/

CREATE TABLE table1

(col1 INT AUTO_INCREMENT KEY, col2 INT DEFAULT '100', col3 VARCHAR(255) CHARACTER SET 'greek', col4 DATETIME) ENGINE = Aria;

Creating Databases is Simple CREATE DATABASE db1;

Creating Tables can be Simple or Complex

List Column Names, Data Types, Default Values, Character Sets and Collation, etc.

Define Indexes and Specify Storage Engine

http://dev.mysql.com/doc/refman/5.5/en/create-database.html

https://mariadb.com/kb/en/mariadb/create-table/

[email protected]


Altering Tables

66

Documentation for ALTER TABLE: https://mariadb.com/kb/en/mariadb/alter-table/

ALTER TABLE table1

ADD COLUMN col5 CHAR(8), DROP COLUMN col3, CHANGE COLUMN col4 col4 DATE;

ALTER TABLE is used to change a Table's Schema ADD COLUMN to Add a Column DROP COLUMN to Drop a Column — Deletes Data CHANGE COLUMN and MODIFY COLUMN to Alter a Column

Basic Syntax Example for ALTER TABLE Statement

https://mariadb.com/kb/en/mariadb/alter-table/

[email protected]


InnoDB & MyISAM Row Formats

67

Fixed-Length Data Types Less Data File Fragmentation Heaviest Disk Space Usage

Static

Fixed and Variable Length Data Types More Data File Fragmentation More Efficient Disk Space Usage

Dynamic

Decompressed as Needed Minimal Disk Space Usage Read-Only

Compressed

INT, CHAR, etc.

VARCHAR, TEXT, etc.

[email protected]


Setting SQL Mode

68

List of SQL Modes: https://mariadb.com/kb/en/mariadb/sql_mode/

SET sql_mode='TRADITIONAL';

SHOW VARIABLES LIKE 'sql_mode' \G

Variable_name: sql_mode Value: STRICT_TRANS_TABLES,

STRICT_ALL_TABLES, NO_ZERO_IN_DATE, NO_ZERO_DATE, ERROR_FOR_DIVISION_BY_ZERO,

TRADITIONAL, NO_AUTO_CREATE_USER, NO_ENGINE_SUBSTITUTION

Determines the SQL Server Environment

Supported SQL Syntax Type of Data Validation

SQL Mode is Initially Empty Use SET Statement to set

SQL Mode

https://mariadb.com/kb/en/mariadb/sql_mode/

[email protected]


Column Attributes

69

Documentation on Default Column Values: https://mariadb.com/kb/en/mariadb/data-types/

CREATE TABLE people (name VARCHAR(20) DEFAULT 'unknown');

Default Values can be Specified in the CREATE TABLE Statement

Columns can be NULL, unless defined NOT NULL NULL means "No Value", "Not Applicable", or

"Unknown" Use NULL when Value is Not an Empty String or 0 NOT NULL Reduces Storage in some Engines

https://mariadb.com/kb/en/mariadb/data-types/

[email protected]


Data Types for Integers

70

Documentation on Integer Data Types: https://mariadb.com/kb/en/mariadb/data-types/

Use UNSIGNED when Appropriate

INT(n) Specifies Display Precision, Not Storage Precision

Size and Precision is Storage Engine Dependent Define Handling of Out-of-Range Values with sql_mode

Default Mode: Values are Truncated Silently Strict Mode: Errors are Generated

TINYINT

SMALLINT

MEDIUMINT

INTEGER,INT

BIGINT

https://mariadb.com/kb/en/mariadb/data-types/

[email protected]


AUTO_INCREMENT

71

Documentation on AUTO_INCREMENT: https://mariadb.com/kb/en/mariadb/auto_increment/

SELECT MAX(auto_inc_field) FROM table;

Use LAST_INSERT_ID()to get Value Generated for Client Connection

SERIAL is a synonym for "BIGINT UNSIGNED NOT NULL AUTO_INCREMENT UNIQUE"

MyISAM the Counter can be Set Back Manually or If Counter Value Wraps

InnoDB prepares AUTO_INCREMENT counters when mysqld starts

https://mariadb.com/kb/en/mariadb/auto_increment/

[email protected]


Floating and Fixed Point

72

Documentation on FLOAT Data Type: https://mariadb.com/kb/en/mariadb/float/ Documentation on DECIMAL Data Type: https://mariadb.com/kb/en/mariadb/decimal/ Documentation on DOUBLE Data Type: https://mariadb.com/kb/en/mariadb/double/

FLOAT and DOUBLE are Approximate Types - uses 4 and 8 bytes IEEE storage format

DECIMAL(m,d) - maximum total number of digits, number of decimal points

An Exact Value type, up to 65 digits precision, 4 bytes storage for each multiple of nine digits

NUMERIC is a synonym for DECIMAL

REAL is a synonym for DOUBLE - Unless in REAL_AS_FLOAT SQL mode

FLOAT

DOUBLE

DECIMAL

NUMERIC

REAL

https://mariadb.com/kb/en/mariadb/float/

https://mariadb.com/kb/en/mariadb/decimal/

https://mariadb.com/kb/en/mariadb/double/

[email protected]


String Data Types

73

Documentation on String Data Types: https://mariadb.com/kb/en/mariadb/string-data-types/

All String Data Types have a Character Set CHAR(n) — number of characters, not bytes, wide VARCHAR(n)— Changes to CHAR in Implicit Temporary Tables and mysqld internal buffers

TEXT —Not Supported by the MEMORY Storage Engine; Implicit Temporary Tables may Convert to MyISAM

CHAR

VARCHAR

TINYTEXT

TEXT

MEDIUMTEXT

LONGTEXT

https://mariadb.com/kb/en/mariadb/string-data-types/

[email protected]


Character Set & Collation

74

Documentation on Character Sets and Collation: https://mariadb.com/kb/en/mariadb/data-types-character-sets-and-collations/

SELECT * FROM table1 ORDER BY col1 COLLATE latin1_german2_ci;

Character Set may be Global or for Schema, Table or Column

Multi-Byte Character Sets Increase Disk Storage and Working Memory Requirements (e.g, UTF-8 Requires 3 or 4 bytes per Character)

Collations affect String Comparison (Character Order) Collations can be Changed for Query:

https://mariadb.com/kb/en/mariadb/data-types-character-sets-and-collations/

[email protected]


Binary Data Types

75

Documentation on BINARY Data Type: https://mariadb.com/kb/en/mariadb/binary/ Documentation on VARBINARY Data Type: https://mariadb.com/kb/en/varbinary/ Documentation on BLOB Data Type: https://mariadb.com/kb/en/blob/

BINARY and VARBINARY are Case-Sensitive Versions of CHAR and VARCHAR

No Character Set and Collation for Binary Types - ordered by bytes

Blobs are Used often to Store Files in a Database Files on Disk are often Faster Blobs are Included in Transactions, Replication, and

Backups Blobs Inflate mysqld memory usage

BINARY

VARBINARY

TINYBLOB

BLOB

MEDIUMBLOB

LONGBLOB

https://mariadb.com/kb/en/mariadb/binary/

https://mariadb.com/kb/en/varbinary/

https://mariadb.com/kb/en/blob/

[email protected]


Date & Time Data Types

76

DATE — from 1000-01-01 to 9999-12-31 (YYYY-MM-DD)

TIME — from -838:59:59 to 838:59:59

DATETIME — Same Range with Time (YYYY-MM-DD HH:mm:ss) TIMESTAMP — Unix timestamp, in seconds from

1970-01-01 Many Apps Store UNIX_TIMESTAMP() values in unsigned integer

field

YEAR — Accepts YY or YYYY

DATE

TIME

DATETIME

TIMESTAMP

YEAR

Documentation on Date and Time Data Types: https://mariadb.com/kb/en/mariadb/date-and-time-data-types/Documentation on Microseconds in MariaDB: https://mariadb.com/kb/en/mariadb/microseconds-in-mariadb/

SELECT CURTIME(4); +---------------+ | CURTIME(4) |

+---------------+ | 05:33:09.1061 | +---------------+

https://mariadb.com/kb/en/mariadb/date-and-time-data-types/

https://mariadb.com/kb/en/mariadb/microseconds-in-mariadb/

[email protected]


Manipulating Date & Time

77

Documentation on Date and Time Functions: https://mariadb.com/kb/en/mariadb/date-and-time-functions/

ADDDATE()

ADDTIME()

CONVERT_TZ()

CURDATE()

CURTIME()

DATE()

DATE_ADD()

DATE_FORMAT()

DATE_SUB()

DATEDIFF()

DAYNAME()

DAYOFMONTH()

DAYOFWEEK()

DAYOFYEAR()

EXTRACT()

FROM_DAYS()

FROM_UNIXTIME()

GET_FORMAT()

HOUR()

LAST_DAY()

MAKEDATE()

MAKETIME()

MICROSECOND()

MINUTE()

MONTH()

MONTHNAME()

NOW()

PERIOD_ADD()

PERIOD_DIFF()

QUARTER()

SEC_TO_TIME()

SECOND()

STR_TO_DATE()

SUBDATE()

SUBTIME()

SYSDATE()

TIME()

TIME_FORMAT()

TIME_TO_SEC()

TIMEDIFF()

TIMESTAMP()

TIMESTAMPADD()

TIMESTAMPDIFF()

TO_DAYS()

UNIX_TIMESTAMP()

UTC_DATE()

UTC_TIME()

UTC_TIMESTAMP()

WEEK()

WEEKDAY()

WEEKOFYEAR()

YEAR()

YEARWEEK()

Functions for Date and Time Manipulation

https://mariadb.com/kb/en/mariadb/date-and-time-functions/

[email protected]


Listing Acceptable Values

78

Documentation on ENUM: https://mariadb.com/kb/en/mariadb/enum/ Documentation on SET: https://mariadb.com/kb/en/mariadb/set-data-type/

CREATE TABLE colors (primary_colors ENUM('red', 'yellow', 'blue'));

ENUM is an Enumerated List of String Values - uses a 2-byte integer index

CREATE TABLE colors

(primary_colors SET('red','yellow','blue'));

INSERT INTO colors VALUES('red'), ('red,blue');

Simple Example for an ENUM Column

Simple Example for a SET Column

SET is a Specified List of String Values — Can Hold Multiple Specified Values

https://mariadb.com/kb/en/mariadb/enum/

https://mariadb.com/kb/en/mariadb/set-data-type/

[email protected]


Dynamic Columns

79

SELECT item_name, COLUMN_GET(item_attributes, 'color' AS CHAR) AS color FROM clothes;

+-----------------+-------+ | item_name | color | +-----------------+-------+

| MariaDB T-shirt | blue | +-----------------+-------+

COLUMN_GET()

COLUMN_ADD()

COLUMN_LIST()

COLUMN_DELETE()

Documentation on Dynamic Columns: https://mariadb.com/kb/en/dynamic-columns/ Tutorial on Dynamic Columns: http://radar.oreilly.com/2015/04/dynamic-columns-in-mariadb.html

CREATE TABLE clothes

(item_name VARCHAR(32) PRIMARY KEY,

item_attributes BLOB);

INSERT INTO clothes

VALUES ('MariaDB T-shirt',

COLUMN_CREATE('color', 'blue', 'size', 'XL'));

Stores Key/Value Pairs within a BLOB Column with COLUMN_CREATE()

Used to Store many Attributes which are Unknown in Advance

Manipulated with Special Dynamic Column Functions:

https://mariadb.com/kb/en/dynamic-columns/

http://radar.oreilly.com/2015/04/dynamic-columns-in-mariadb.html

[email protected]


Virtual Columns

80

Documentation on Virtual Columns: https://mariadb.com/kb/en/virtual-columns/

CREATE TABLE sales (sales_id INT AUTO_INCREMENT KEY, qty INT,

price DECIMAL(6,2), total DECIMAL(6,2) AS (qty * price) VIRTUAL);

Two Virtual Column Types: PERSISTENT (stored) VIRTUAL (generated only)

All Data Types Supported Use PERSISTENT for Indexes - Cannot

be Primary Key

Used with InnoDB, Aria, MyISAM, CONNECT

https://mariadb.com/kb/en/virtual-columns/

[email protected]


Table Maintenance SQL Statements

81

ANALYZE TABLE — Check and Store Index Cardinal Values

Run Periodically OPTIMIZE TABLE — Defragment MyISAM Data

Files and Sort Indexes Maps Simply to a Full Table Rebuild

for InnoDB CHECK TABLE — Check Table for Errors

Arguments: QUICK, FAST, CHANGED, MEDIUM, EXTENDED

REPAIR TABLE — Fix a Corrupted Table Arguments: QUICK, EXTENDED, USE_FRM

CHECKSUM TABLE

Arguments: QUICK, EXTENDED

Documentation on CHECK TABLE: https://mariadb.com/kb/en/mariadb/sql-commands-check-table/ Documentation on CHECKSUM TABLE: https://mariadb.com/kb/en/checksum-table/ Documentation on REPAIR TABLE: https://mariadb.com/kb/en/repair-table/ Documentation on ANALYZE TABLE: https://mariadb.com/kb/en/analyze-table/ Documentation on OPTIMIZE TABLE: https://mariadb.com/kb/en/optimize-table/

https://mariadb.com/kb/en/mariadb/sql-commands-check-table/

https://mariadb.com/kb/en/checksum-table/

https://mariadb.com/kb/en/repair-table/

https://mariadb.com/kb/en/analyze-table/

https://mariadb.com/kb/en/optimize-table/

MariaDB for DBAs

Views

Introduction

Architecture

Installation

Client Utilities

Configuration

Resource Usage

Storage Engines

Table Schema

Views

Partitioning


Log Files



Customizing

MariaDB Replication

High Availability

Optimization

[email protected]


Purpose of Views

83

Adapt and Standardize Table Schema for Applications or Across Application Upgrades

Restricting Visible Table Data to Specific Users and Applications

Useful to Simplify, Split, or Factor Complex Reporting Queries

[email protected]


View Example & Caveats

84

CREATE VIEW emp_names AS

SELECT emp_id, name_first, name_last

FROM employees;

SELECT * FROM view_example LIMIT 5;

+--------+------------+-----------+

| emp_no | name_first | name_last |

+--------+------------+-----------+

| 10021 | Ramzi | Erde |

| 10087 | Xinglin | Eugenio |

| 10122 | Ohad | Esposito |

| 10131 | Magdalena | Eldridge |

| 10145 | Akemi | Esposito |

+--------+------------+-----------+

An SQL Statement Represented as a Table

Views are Not Materialized The SELECT Query always Re-

Executes

[email protected]


Creating and Changing Views

85

Documentation on CREATE VIEW: https://mariadb.com/kb/en/create-view/

CREATE VIEW view_loans AS SELECT id, loan_amount, (loan_months / 12) AS loan_years FROM loan_table;

CREATE VIEW Statement Creates a View based on a SELECT statement

https://mariadb.com/kb/en/create-view/

[email protected]


Creating and Changing Views

86

ALTER VIEW view_loans AS

SELECT ...;

ALTER VIEW Statement Deletes View and Creates New One

DROP VIEW view_loans;

DROP VIEW Statement Deletes a View

Documentation on ALTER VIEW: https://mariadb.com/kb/en/mariadb/alter-view/ Documentation on DROP VIEW: https://mariadb.com/kb/en/mariadb/drop-view/

https://mariadb.com/kb/en/mariadb/alter-view/

https://mariadb.com/kb/en/mariadb/drop-view/

[email protected]


Showing Views

87

SHOW CREATE VIEW view_loans \G

Use SHOW TABLES to List Views

SHOW FULL TABLES WHERE Table_type = 'VIEW';

Use SHOW CREATE to Inspect Schema of a View

[email protected]


View Security

88

CREATE VIEW view_loans ...

SQL SECURITY = INVOKER;

Views may be Writable — INSERT, UPDATE and DELETE Affect Underlying Table

Conditions of Non-Writable View ALGORITHM Set to TEMPTABLE UNION, HAVING, some Joins, and Sub-Queries Data Aggregation (AVG, GROUP BY, DISTINCT) Derived Fields — Field using a Function or Calculation

Updatable Views WITH CHECK OPTION Constrains Writes to Rows

in View

Set Security Based on View Creator (DEFINER) or User (INVOKER)

[email protected]


Setting the Algorithm

89

Documentation on View Algorithms: https://mariadb.com/kb/en/mariadb/view-algorithms/

CREATE VIEW view_loans ALGORITHM = MERGE AS SELECT ...

View Algorithm Controls Query Execution Method

TEMPTABLE View Data put in Temporary Table, then Query Executed Possibly Slower since No Index for Temporary Table

MERGE Query and Original View SELECT are Merged Possibly Faster since Underlying Table Index Used

UNDEFINED The Optimizer choose what’s Best (Default)

https://mariadb.com/kb/en/mariadb/view-algorithms/

MariaDB for DBAs

Partitioning

Introduction

Architecture

Installation

Client Utilities

Configuration

Resource Usage

Storage Engines

Table Schema

Views

PartitioningUser & Server Security

Log Files



Customizing

MariaDB Replication

High Availability

Optimization

[email protected]


Value of Partitions

91

Documentation for Partitions: https://mariadb.com/kb/en/mariadb/create-table/#partitions

table1.frm table1.par

table1#P#p0.MYD

table1#P#p0.MYI

table1#P#p1.MYD

table1#P#p1.MYI

table1#P#p2.MYD

table1#P#p2.MYI

table1#P#p3.MYD

table1#P#p3.MYI

Partitioned Table *

* Files Generated for a MyISAM Table with Four Partitions

For Tables Larger than Memory Easier Maintenance with a Single

Partition Instead of Entire Table (Less Data, Fewer Locks, Faster Operations)

Partition Pruning — Better Performance for Queries Involving only Specific Partitions

Bulk Management of Data — Archiving & Fast Elimination of Data

https://mariadb.com/kb/en/mariadb/create-table/#partitions

[email protected]


Factors in Partitioning

92

Custom Partitioning Function is defined for a Table to return a Partitioning Key

Key Determines which Partitions Rows are Stored Key must be Integer or NULL Stored Procedures, Stored Functions, UDFs, Plug-ins, User

Variables & Sub-Queries Not Allowed in Function

A Column may be Used Directly as Key without a Function Maximum of 1024 Partitions per Table Foreign Keys are Disabled in an InnoDB Table FULLTEXT and Spatial Data Types are Disabled in MyISAM

CONNECT Storage Engine Doesn't Support Partitioning

[email protected]


RANGE Partitions

93

MAXVALUE indicates the Maximum Value Possible

Documentation for Partitions: https://mariadb.com/kb/en/mariadb/create-table/#partitions

Partition p0

Distribution of Rows

(100, 'Bob Smith', 4),(102, 'Wendy Jones', 6),(105, 'Sarah Jane', 4),(109, 'Bill Johnson', 7)

(101, 'Tim Schmidt', 16),(103, 'John Wells', 12),(107, 'Cindy James', 18)

(104, 'Lena Denison', 25),(106, 'Tom Smith', 20),(108, 'Elaine Wilson', 30),(110, 'Susan Hill, 120)

Partition p1

Partition p2

employees tableCREATE TABLE employees ( id INT, name VARCHAR(20), dept_id INT NOT NULL) ENGINE = MyISAM PARTITION BY RANGE(dept_id) (

PARTITION p0 VALUES LESS THAN (10), PARTITION p1 VALUES LESS THAN (20), PARTITION p2 VALUES LESS THAN MAXVALUE);

Partition by a Range of Column Values

https://mariadb.com/kb/en/mariadb/create-table/#partitions

[email protected]


RANGE Columns

94

Integer, Date, and String Data Types Allowed for Partition Keys

TEXT, BLOB, DECIMAL, and FLOAT Not Allowed

Partition p0


(40, 'Bob Smith', 4),(85, 'Wendy Joans', 25),(99, 'Sarah Jane', 30),(120, 'Bill Johnson', 8)

(125, 'Tim Schmidt', 16),(100, 'John Wells', 15),(150, 'Cindy James', 40)

(210, 'Lena Denison', 25),(200, 'Tom Smith', 50)

Partition p1

Partition p2

employees tableCREATE TABLE employees ( id INT AUTO_INCREMENT KEY, name VARCHAR(20), dept_id INT) ENGINE = MyISAM PARTITION BY RANGE COLUMNS(id, dept_id) (

PARTITION p0 VALUES LESS THAN (100,10), PARTITION p1 VALUES LESS THAN (200,20), PARTITION p2 VALUES LESS THAN (MAXVALUE,MAXVALUE));

Partition by Column Tuples within a Range Tuples are like Sets of Values for Columns

[email protected]


LIST Partitions

95

Values Not Listed are Rejected No MAXVALUE

Partitionactive_clients


(110, 'Harold Wilson', 1),(116, 'Leslie Haynes', 1),(120, 'Mike Brown', 1),(126, 'Tim Simms', 1)

(115, 'Jim Berns', 0),(120, 'Ed Gibson', 0),(122, 'Sally King', 0)

(106, 'Betty Mills', -1),(125, 'Joe Cohen', 2)

Partition inactive_clients

Error

clients tableCREATE TABLE clients (

client_id INT,

name VARCHAR(20), active INT NOT NULL)

ENGINE = MyISAM

PARTITION BY LIST(active) (

PARTITION active_clients VALUES IN (1),

PARTITION inactive_clients VALUES IN (0));

Partition by a List of Specific Column Values

[email protected]


LIST COLUMNS Partitions

96

Partition employees_east


(110, 'Harold Wilson', 'Toykyo'),(116, 'Leslie Haynes', 'Tokyo'),(120, 'Mike Brown', 'Sydney'),(126, 'Tim Simms', 'Sydney')

(115, 'Jim Berns', 'London'),(120, 'Ed Gibson', 'New York'),(122, 'Sally King', 'New York)

(106, 'Betty Mills', 'Londonn'),(125, 'Joe Cohen', 'Chicago')

Partition employees_west

Error

employees table

CREATE TABLE employees (

id INT AUTO_INCREMENT KEY,

name VARCHAR(20),

office VARCHAR(20) NOT NULL)

ENGINE = MyISAM

PARTITION BY LIST COLUMNS(office)

(PARTITION employees_west

VALUES IN ('London', 'New York'),

PARTITION employees_east

VALUES IN ('Tokyo', 'Sydney'));

Partition by List of Specific String, Integer or Date Values

One or More Columns Allowed

[email protected]


HASH Partitions

97

Partitions contain Same Number of Rows Use COALESCE to Reduce Number of

Partitions

Distribution of Rowsemployees table

Partition p0

Partition p1

Partition p2

Partition p3

(100, 'Bob Smith', '2014-05-15'),(101, 'Tim Schmidt', '2014-05-20'),(102, 'Wendy Jones', 2014-06-01')

(103, 'John Wells', '2014-09-15'),(104, 'Lena Denison', '2014-12-01'),(105, 'Sarah Jane', '2015-01-15')

(106, 'Tom Smith', '2015-02-14'),(107, 'Cindy James', '2015-05-01'),(108, 'Elaine Wilson', '2015-09-20')

(109, 'Bill Johnson', '2016-02-24'),(110, 'Susan Hill, '2016-03-15'),(111, 'Rob Kane, '2016-08-03')

ALTER TABLE employees

COALESCE PARTITION 3;

CREATE TABLE employees (

id INT AUTO_INCREMENT KEY,

name VARCHAR(20), dept_id INT,

join_date DATE)

ENGINE = MyISAM

PARTITION BY HASH(id) PARTITIONS 4;

Partition Determined by a Hash from a Given Expression

[email protected]


KEY Partitions

98

Primary Key or Specified Columns Used for Partition Key

CREATE TABLE employees ( id INT AUTO_INCREMENT KEY, name VARCHAR(20), dept_id INT,

join_date DATE) ENGINE = MyISAM PARTITION BY KEY(dept_id, join_date) PARTITIONS 4;

Partition with a Hash, Using an Expression Generated Automatically

[email protected]


Composite or Sub-Partitioning

99

CREATE TABLE patients ( id INT,

name VARCHAR(20), birth_date DATE) ENGINE = MyISAM PARTITION BY RANGE( YEAR(birth_date) ) SUBPARTITION BY HASH( QUARTER(birth_date) ) SUBPARTITIONS 4 ( PARTITION elderly VALUES LESS THAN (1950),

PARTITION middle_age VALUES LESS THAN (1975), PARTITION young VALUES LESS THAN MAXVALUE );

patients table

elderly#p0

elderly#p1

elderly#p2

elderly#p3

middle_age#p0

middle_age#p1

middle_age#p2

middle_age#p3

young#p0

young#p1

young#p1

young#p3

Used to further Divide Partitions Each Partition Contains same Number of

Sub-Partitions Partitions and Sub-Partitions may use

different Methods

[email protected]


Partition Specific Queries

100

Supports Queries and Data Manipulation Statements SELECT, DELETE, INSERT, REPLACE, UPDATE, LOAD DATA, and LOAD XML

SELECT COUNT(*) FROM employees

PARTITION(employees_west);

Partition may be Specified in SQL Statements

[email protected]


Partition Maintenance

101

ALTER TABLE patients REBUILD PARTITION elderly;

Use ALTER TABLE to Perform Specific Partition Maintenance (ANALYZE, CHECK, REBUILD, REPAIR, and OPTIMIZE) Hard Disk Space is Reclaimed

REPAIR TABLE clients; +--------------+--------+----------+----------+

| Table | Op | Msg_type | Msg_text | +--------------+--------+----------+----------+ | test.clients | repair | status | OK | +--------------+--------+----------+----------+

Use Normal Full Table Maintenance Statements (ANALYZE, CHECK, OPTIMIZE, and REPAIR TABLE)

[email protected]


Partition Changes

102

ALTER TABLE clients

ADD PARTITION (PARTITION suspended_clients VALUES IN (-1) );

Adding Partitions Provide Full Definition of New

Partition

ALTER TABLE clients DROP PARTITION inactive_clients;

Dropping Partitions Data Contained is Dropped

Partition is Lost

ALTER TABLE clients REMOVE PARTITIONING;

Removing Partitioning Data is Not Lost

[email protected]


Partition Changes

103

Alterations are Allowed on RANGE, LIST Partitions

ALTER TABLE table1

REORGANIZE PARTITION partition_a, partition_b

INTO(PARTITION partition_ab

VALUES LESS THAN (1000));

ALTER TABLE table1

REORGANIZE PARTITION p0 INTO

(PARTITION n0 VALUES LESS THAN (1000),

PARTITION n1 VALUES LESS THAN (2000));

Splitting Partitions

Merging Partitions

ALTER TABLE employees_test

REORGANIZE PARTITION middle_age

INTO(PARTITION older VALUES LESS THAN (1975));

Renaming a Partition Use REORGANIZE PARTITION with

New Name, but Same Parameters

[email protected]


Exchanging Partitions with Tables

104

Both Partition and Table need Same Structure No Foreign Keys in Table, Nor Other Referencing

Tables Data is Moved; Partition is Preserved

Use DROP PARTITION to Remove PartitionALTER TABLE clients DROP PARTITION inactive_clients;

ALTER TABLE clients EXCHANGE PARTITION inactive_clients

WITH TABLE clients_archive;

Move a Partition or Sub-Partition to an Unpartitioned Table

[email protected]


Storage Engine Considerations

105

Storage Engine can be Specified for a Partitioned Table

ENGINE Clause precedes PARTITION Clause Partitions in Table must use Same Storage Engine MERGE, CSV and FEDERATED Don't Support

Partitioning DATA DIRECTORY and INDEX DIRECTORY Clauses Don't

Effect Partitioned InnoDB Tables

MariaDB for DBAs


Introduction

Architecture

Installation

Client Utilities

Configuration

Resource Usage

Storage Engines

Table Schema

Views

Partitioning

User & Server SecurityLog Files



Customizing

MariaDB Replication

High Availability

Optimization

[email protected]


Security — a Multi-Layered Challenge

107

Other Processes

mysqld Data Storage

Network Considerations Private Network Wifi Access SSL for Data In Motion DNS Controlled

ClientsServer

Port 3306

Server Considerations Data Location Non-Root Users Passwords Other Processes

Apps

[email protected]


MariaDB User Accounts

108

Documentation for CREATE USER: https://mariadb.com/kb/en/mariadb/create-user/

Privileges are Based on User and Host Combined

CREATE USER 'ted'@localhost

IDENTIFIED BY 'rover123';

Authentication Based on User, Host, and Password Empty String is User Wildcard Percent Sign (%) is a Host Wildcard

Host is an IP or Host Name — DNS, /etc/hosts localhost is used by Local Socket on Linux Systems

https://mariadb.com/kb/en/mariadb/create-user/

[email protected]


Granting Privileges — Databases and Tables

109

Documentation for GRANT: https://mariadb.com/kb/it/grant/

GRANT SELECT, DELETE,

UPDATE(col2,col3) ON demo.tester TO 'bob'@'localhost';

Users given Permission with GRANT Statement

Privilege Levels — Global, Database, Table, Column, or Routine ON *.* — Access to All Databases and Tables

ON database_name.* — Access to Database Named

ON database_name.table_name — Access to Table Named

privilege(column) ON TABLE db_name.table_name — Column by Privilege

ON PROCEDURE database_name.routine_name — Access to Routine

https://mariadb.com/kb/it/grant/

[email protected]


More User SQL Statements

110

Documentation for RENAME USER: https://mariadb.com/kb/en/rename-user/ Documentation for SET PASSWORD: https://mariadb.com/kb/en/set-password/ Documentation for DROP USER: https://mariadb.com/kb/en/drop-user/

SET PASSWORD FOR 'bob'@'localhost' = PASSWORD('fido456');

User Names and Hosts can be Changed

RENAME USER 'robert'@'127.0.0.1' TO 'bob'@'localhost';

Users Passwords can be Changed

DROP USER 'bob'@'localhost';

Users can be Deleted

https://mariadb.com/kb/en/rename-user/

https://mariadb.com/kb/en/set-password/

https://mariadb.com/kb/en/drop-user/

[email protected]


Permissions Tables

111

Permissions Checked when Client Connects and Query Received

mysql Database contains Permissions or Grants Tables

— Restrict Access

Grants Tables Loaded into Memory at Start Possible to INSERT, UPDATE, and DELETE Manually Tables

in mysql FLUSH PRIVILEGES Required to Reload Manual Changes

mysqldump will Copy mysql Database by Default

[email protected]


Privileges Available (SHOW PRIVILEGES)

112

Documentation on User Privileges: https://mariadb.com/kb/en/mariadb/grant/#privilege-levels

Basic Privileges

USAGE

SELECT

INSERT

UPDATE

DELETE

SHOW DATABASES

Developer Privileges

CREATE

ALTER

INDEX

DROP

Customizing Privileges

CREATE ROUTINE

ALTER ROUTINE

EXECUTE

EVENT

TRIGGER

Special Privileges

CREATE TEMPORARY TABLES

CREATE VIEW

SHOW VIEW

LOCK TABLES

Replication Privileges

REPLICATION CLIENT

REPLICATION SLAVE

Administrator Privileges

ALL [PRIVILEGES]

SUPER

CREATE USER

GRANT OPTION

PROCESS

FILE

RELOAD

SHUTDOWN

https://mariadb.com/kb/en/mariadb/grant/#privilege-levels

[email protected]


Checking User Privileges

113

Documentation for SHOW GRANTS: https://mariadb.com/kb/en/show-grants/

SHOW GRANTS FOR 'bob'@'localhost'; SHOW GRANTS FOR 'bob'@'home';

Check Regularly user Table in mysql Database for User Accounts

Check for Obsolete Accounts Check for Blank Passwords

SELECT User, Host, Password FROM mysql.user;

Use the SHOW GRANTS statement for User-Host Account

https://mariadb.com/kb/en/show-grants/

[email protected]


Revoking User Privileges

114

Documentation for REVOKE: https://mariadb.com/kb/en/revoke/

REVOKE DELETE,UPDATE ON db1.table1 FROM 'bob'@'localhost';

Use REVOKE Statement to Revoke Privileges — Need GRANT OPTION

List Privileges to Revoke or ALL PRIVILEGES REVOKE Doesn’t Remove a User-Host Account — Use DROP

USER

Check Results with SHOW GRANTS after Revoking

REVOKE ALL PRIVILEGES ON db1.table1 FROM 'bob'@'localhost';

GRANT SELECT ON database_name.table_name TO 'bob'@'localhost';

https://mariadb.com/kb/en/revoke/

[email protected]


Limiting Users

115

Documentation for User Account Resource Limits: http://dev.mysql.com/doc/refman/5.6/en/user-resources.html

GRANT SELECT ON db_name.table_name TO 'bob'@'home_bob' WITH MAX_QUERIES_PER_HOUR 20 MAX_CONNECTIONS_PER_HOUR 10

MAX_USER_CONNECTIONS 2 MAX_UPDATES_PER_HOUR 5;

User-Host Accounts may be Limited by Resources

Usage Related to Limits are kept in mysql Database Counters Reset when Server Starts or FLUSH PRIVILEGES Executed

User Counters may be Reset Specifically — Not MAX_USER_CONNECTIONS

FLUSH USER_RESOURCES;

http://dev.mysql.com/doc/refman/5.6/en/user-resources.html

[email protected]


Pluggable Authentication Module

116

Documentation on PAM: https://mariadb.com/kb/en/mariadb/pam-authentication-plugin/

[mysqld] plugin-load=authentication_pam.so

Uses Plug-in to Authenticate Client Connections Allows Credentials from Other Sources — Linux Passwords,

LDAP, Kerberos, Active Directory

CREATE USER user IDENTIFIED WITH authentication_pam AS 'mysql, root=developer, users=data_entry';

Specify PAM when Creating Users


https://mariadb.com/kb/en/mariadb/pam-authentication-plugin/

[email protected]


Proxy Users

117

GRANT PROXY ON 'manager'@'localhost' TO 'bob'@'localhost';

SELECT USER(), CURRENT_USER();

+---------------+-------------------+

| USER() | CURRENT_USER() |

+---------------+-------------------+

| bob@localhost | manager@localhost |

+---------------+-------------------+

MariaDB Perceives User as Proxied User

Documentation on Proxy Users: http://dev.mysql.com/doc/refman/5.5/en/proxy-users.html

REVOKE PROXY ON 'manager'@'localhost' FROM 'bob'@'localhost';

PROXY Permission give a User-Account Permission of Another Account

http://dev.mysql.com/doc/refman/5.5/en/proxy-users.html

[email protected]


User Roles

118

Documentation on User Roles: https://mariadb.com/kb/en/roles-overview/

User Role is a Set of Privileges available for a Group An Alternative to Assigning Privileges Individually to

Each User Account Useful for Assigning Privileges to Many Users or to

External Users

https://mariadb.com/kb/en/roles-overview/

[email protected]


Creating a User Role

119

Documentation on CREATE ROLE: https://mariadb.com/kb/en/mariadb/create-role/

CREATE ROLE 'sales_staff';

GRANT SELECT, INSERT, UPDATE ON sales.* TO 'sales_staff';

Use CREATE ROLE to Create Group Privileges

Use GRANT to Grant Privileges to Role

No Host is given for a Role

https://mariadb.com/kb/en/mariadb/create-role/

[email protected]


Listing & Showing Privileges of Roles

120

Query information_schema to Get List of Roles Created

Use SHOW GRANTS to Check Privileges for Role

SHOW GRANTS FOR 'admin_remote';

+------------------------------------------------------+

| Grants for admin_remote |

+------------------------------------------------------+

| GRANT USAGE ON *.* TO 'admin_remote' |

| GRANT INSERT, UPDATE ON `admin`.* TO 'admin_remote' |

+------------------------------------------------------+

No Host is Given for a Role

SELECT * FROM

information_schema.APPLICABLE_ROLES \G

********* 1. row ********

GRANTEE: root@localhost

ROLE_NAME: admin_remote

IS_GRANTABLE: YES

IS_DEFAULT: NO

There is No SHOW ROLES Statement

[email protected]


Granting a Role to a User

121

Use GRANT to Designate Users for Role

GRANT 'sales_staff' TO 'bob'@localhost;

SELECT * FROM mysql.roles_mapping;

+-----------+-------+--------------+--------------+ | Host | User | Role | Admin_option | +-----------+-------+--------------+--------------+ | localhost | bob | admin_remote | N | | localhost | root | admin_remote | Y |

+-----------+-------+--------------+--------------+

Check mysql.roles_mapping for Roles Granted

Executed while Logged in as an Administrator


GRANT 'admin_remote' TO 'bob'@localhost WITH ADMIN OPTION;

Use WITH ADMIN OPTION so Grantee may Grant Role

[email protected]


Assuming and Relinquishing a Role

122

Documentation on CURRENT_ROLE( ): https://mariadb.com/kb/en/mariadb/current_role/ Documentation on SET ROLE: https://mariadb.com/kb/en/mariadb/set-role/

SET ROLE sales_staff;

User Assumes a Role for Session with SET ROLE SELECT CURRENT_ROLE();

+----------------+ | CURRENT_ROLE() | +----------------+

| sales_staff | +----------------+

Executed while Logged in as bob@localhost

Executed while Logged in as user bob@localhost

User Relinquishes a Role by Setting it to None — Or by Ending Session

SET ROLE NONE;

Executed while Logged in as bob@localhost

SET DEFAULT ROLE sales_staff;

User can Set Default Role to NONE or Preferred Role

Executed while Logged in as user bob@localhost

https://mariadb.com/kb/en/mariadb/current_role/

https://mariadb.com/kb/en/mariadb/set-role/

[email protected]


Revoking a Role

123

Documentation on Revoking Roles: https://mariadb.com/kb/en/mariadb/revoke/#roles Documentation on DROP ROLE: https://mariadb.com/kb/en/mariadb/drop-role/

REVOKE 'admin_remote' FROM 'bob'@localhost;

Check mysql.roles_mapping for a List of Roles Granted

Use REVOKE to take Role Option from User



SELECT * FROM mysql.roles_mapping;

+-----------+---------+--------------+--------------+ | Host | User | Role | Admin_option | +-----------+---------+--------------+--------------+

| localhost | ted | admin_remote | N | | localhost | bob | admin_remote | Y | | localhost | root | admin_remote | Y | +-----------+---------+--------------+--------------+

https://mariadb.com/kb/en/mariadb/revoke/#roles

https://mariadb.com/kb/en/mariadb/drop-role/

[email protected]


Dropping a Role

124

Documentation on Revoking Roles: https://mariadb.com/kb/en/mariadb/revoke/#roles Documentation on DROP ROLE: https://mariadb.com/kb/en/mariadb/drop-role/

SELECT User FROM mysql.user

WHERE is_role='Y';

DROP ROLE 'sales_staff';

Check mysql.user for List of Roles

Use DROP ROLE to take Eliminate a Role



https://mariadb.com/kb/en/mariadb/revoke/#roles

https://mariadb.com/kb/en/mariadb/drop-role/

MariaDB for DBAs

Log Files

Introduction

Architecture

Installation

Client Utilities

Configuration

Resource Usage

Storage Engines

Table Schema

Views

Partitioning


Log Files



Customizing

MariaDB Replication

High Availability

Optimization

[email protected]


MariaDB Logs

126

Error Log

General Query Log

SQL Error Log

Slow Query Log

Binary Log

mysqld

Error Log General Query Log SQL Query Log Slow Query Log Binary Log

[email protected]


Error Log

127

Set log-warnings to 2 for VerboseDocumentation on Error Log: https://mariadb.com/kb/en/mariadb/error-log/

Error Log

General Query Log

SQL Query Log

Slow Query Log

Binary Log

mysqld

[mysqld]

log-error = /path/file_name log-warnings = 1

Contain Startup, Shutdown, Error Messages

Unix uses stderr, Sent to host_name.err in datadir

Some Unix systems Redirect to System Log

Windows uses host_name.err in datadir, or System Event Log


https://mariadb.com/kb/en/mariadb/error-log/

[email protected]


General Query Log

128

Documentation on General Query Log: https://mariadb.com/kb/en/mariadb/general-query-log/

Error Log

General Query Log

SQL Error Log

Slow Query Log

Binary Log

mysqld


[mysqld] general-log

general_log_file='/path/host.log'

All Queries Received from All Clients Order Received — Not Execution Potential Problems with Log

Possible Disk I/O bottleneck on High-Traffic Servers

Log can Quickly become Huge Contains Queries in Plain Text — a

Security Risk

https://mariadb.com/kb/en/mariadb/general-query-log/

[email protected]


SQL Error Log

129

Documentation on SQL Query Log: https://mariadb.com/kb/en/mariadb/sql_error_log-plugin/

Error Log

Slow Query Log

Binary Log

mysqld SQL Error Log

General Query Log

Logs SQL Errors Error Messages SQL Statement

Part of MariaDB Audit Plugin Used for Detecting SQL Injections

Logs User, Host, & Time

INSTALL PLUGIN sql_error_log SONAME 'sql_errlog';

Requires INSERT Privilege on mysql.plugin Table

https://mariadb.com/kb/en/mariadb/sql_error_log-plugin/

[email protected]


Slow Query Log

130

Documentation on Slow Query Log: https://mariadb.com/kb/en/slow-query-log/

Error Log

General Query Log

SQL Error Log

Slow Query Log

Binary Log

mysqld

Relevant Entries from my.cnf or my.ini configuration file

[mysqld] slow_query_log = ON

slow_query_log_file = /path/file long_query_time = 0.5 log_queries_not_using_indexes log_slow_admin_statements

Queries which take more seconds to execute than long_query_time

Contains Queries in Plain Text Use mysqldumpslow to Read

Log

https://mariadb.com/kb/en/slow-query-log/

[email protected]


Binary Log

131

Documentation on Binary Log: https://mariadb.com/kb/en/binary-log/

Error Log

General Query Log

Binary Log

Slow Query Log

SQL Error Logmysqld

[mysqld] log-bin

expire_logs_days = 7

Writes in the Order Executed Useful for Recovering Lost Data Used for Replication Move from to Separate Disk for

Security Set Expiration or Purge

Occasionally


https://mariadb.com/kb/en/binary-log/

[email protected]


Managing Log Files

132

Documentation on FLUSH: https://mariadb.com/kb/en/mariadb/flush/ Documentation on mysqladmin: https://mariadb.com/kb/en/mariadb/mysqladmin/

Rotated Regularly the Log Files Move manually the Existing Log Files

and then Flush Logs

FLUSH LOGS;

Back-Ups Include Logs in Back-ups Synchronize Binary Logs with Back-ups

Error Log Stop mysqld, Move the Log, then Restart mysqld

mysqladmin flush-logs

An Alternative Method from the Command-Line

https://mariadb.com/kb/en/mariadb/flush/

https://mariadb.com/kb/en/mariadb/mysqladmin/

[email protected]


Managing Binary Logs

133

Documentation on PURGE BINARY LOGS: https://mariadb.com/kb/en/mariadb/sql-commands-purge-logs/

Inspect Current Binary Logs FLUSH LOGS triggers a Binary Log

Rotation Binary Log Files use Incrementally

Numbered Extensions Binary Log Files can be Purged

explicitly Binary Logs can be set to Expire in

Configuration File (expire_logs_days) Allow Time for all Slaves to Read the

Binary Log Files

PURGE BINARY LOGS BEFORE '2016-06-01 10:00:00';

SHOW BINARY LOGS;

PURGE BINARY LOGS TO 'mariadb-bin.000010';

Purges All Binary Logs Before One Named

Purges All Binary Logs Before One which Contains Entries for Date Given

https://mariadb.com/kb/en/mariadb/sql-commands-purge-logs/

[email protected]


Audit Plugin Log

134

Documentation on Audit Plugin: https://mariadb.com/kb/en/mariadb/about-the-mariadb-audit-plugin/

Includes Table Event Logging (Triggers, Stored Procedure Calls) Optional Field Substitution of Placeholders in Query

Log to Improve Security Filtering Audit Logs by Role & User Accounts Records Privilege Changes Password Change Logging

https://mariadb.com/kb/en/mariadb/about-the-mariadb-audit-plugin/

MariaDB for DBAs


Introduction

Architecture

Installation

Client Utilities

Configuration

Resource Usage

Storage Engines

Table Schema

Views

Partitioning


Log Files



Customizing

MariaDB Replication

High Availability

Optimization

[email protected]


Command-Line Monitoring Tools

136

Site for mytop: http://jeremy.zawodny.com/mysql/mytop/ Site for innotop: https://code.google.com/p/innotop/

mytop -u user -p passwordmytop

A Clone of top for Monitoring MariaDB Threads

Works any Shell, SSH, or Shell Script

innotop

Tool for Monitoring MariaDB Threads

Includes InnoDB Specific Information

http://jeremy.zawodny.com/mysql/mytop/

https://code.google.com/p/innotop/

[email protected]


MonYog

137

Site for MonYog: https://www.webyog.com/product/monyog

User-Friendly, Cross Platform, GUI

No Server-Side Agent Required Manages Easily Multiple Servers Track and Alert many Metrics

Monitor Error Log and Security Vulnerabilities

Replication Lag and Excessive Resource Usage

Slow Queries and Deadlocks Detailed History and Trend

Analysis

https://www.webyog.com/product/monyog

[email protected]


Automated Monitoring Utilities

138

Automated Monitoring can poll MariaDB and Send Alerts

CA-Unicenter Cacti Ganglia HP Openview Hyperic IBM Tivoli

Munin Nagios NewRelic OpenNMS PMM Zabbix

Some are Generic Network Monitors with Plug-ins for MariaDB

[email protected]


Custom Monitoring Solutions

139

Custom Monitoring Solution or Method is sometimes Required

Check if mysqld is active with mysqladmin ping or C function, mysql_ping()

Look for Performance Hot Spots with SHOW GLOBAL STATUS

Parse the mysqld Error Log for “Error” or “Warning” of other Keywords

Use a Scheduler (e.g., cron) to run Monitor Tools — Email Results or Create SNMP Alerts

Kristian Koehntopp (http://vvv.koehntopp.de/rrd/)

http://vvv.koehntopp.de/rrd/

[email protected]


Error Log

140

Error Log is Stored by Default in Data Directory (datadir)

Error Log Contains Information about Errors from Queries

Shows Warnings - Depends on --log-warnings

Shows mysqld Stops, Starts, and Error Events Crashes will Appear as Abrupt Stop without Shutdown

Messages Check System Log for Information on Shutdown or

Crash

[email protected]


MariaDB Status

141

SHOW PROCESSLIST;

SHOW GLOBAL STATUS;

SHOW ENGINE INNODB STATUS \G

Documentation on SHOW PROCESSLIST: https://mariadb.com/kb/en/mariadb/show-processlist/ Documentation on SHOW STATUS: https://mariadb.com/kb/en/show-status/ Documentation on SHOW ENGINE INNODB STATUS: https://mariadb.com/kb/en/show-engine-innodb-status/

Check Global Status Unusually Large Values - Note Typical Results

Check Processes Long Running Queries - Lock & Load Problems

Intermittent or Short Term Problems - Try Multiple Snapshots or Polling

Check Status of Storage Engines Used Useful for InnoDB, though Returns Complex Results

https://mariadb.com/kb/en/mariadb/show-processlist/

https://mariadb.com/kb/en/show-status/

https://mariadb.com/kb/en/show-engine-innodb-status/

[email protected]


Server Status

142

See if Reached Limits of CPU, Disk, Memory, or Network Usage

Track System Statistics to Identify Trends or Patterns - Use a Monitoring Program

Watch for Unpredictable and Reoccurring Problems Check What is Running on Server — Check cron Check if Other Servers Affected, if more than One

[email protected]


Application Status

143

Check if New Applications Installed Recently Code Changes or Updates Applied Recently — If it

Handles Deadlocks Check Logs for Applications

MariaDB Connection or Disconnection Problems Unexpected Error Codes from MariaDB

Check if Application Traffic and Dataset within Expected Bounds

[email protected]


MariaDB Support

144

Web Page for MariaDB Support: http://www.mariadb.com/products/mysql-support

24 x 7 On-Line, Live Assistance General Consulting Performance Tuning Code Review Login Support Bug and Hot fixes On-Site Health Check

http://www.skysql.com/products/mysql-support

MariaDB for DBAs


Introduction

Architecture

Installation

Client Utilities

Configuration

Resource Usage

Storage Engines

Table Schema

Views

Partitioning


Log Files


Back-Ups & RestorationCustomizing

MariaDB Replication

High Availability

Optimization

[email protected]


Backup Principles

146

High Availability Can’t Prevent User Errors Copy Data, Logs, and Configuration Files Daily Snapshots (mysqldump, InnoDB Hot Backup, LVM) and

Real-Time Backups (Replication)

Store Backups in Multiple Locations - Onsite for fast access; Offsite for security

Synchronize Binary Logs with Backup Files

Back-Up Methods Physical Back-Up Logical Back-Up Binary Back-Up

Test Backup and Recovery

[email protected]


Making Physical Back-Ups

147

Produces a Binary Copy of Data Faster than Dumping to SQL Text File Recovered Only to the Same Storage Engine Not Useful for Migrations Corrupted Files are Undetected Multiple Options

Copy Manually Data Directory (Stopping mysqld is Necessary) LVM on Linux for Volume Snapshot (Data Complete and Consistent, but still Trigger InnoDB Recovery after

Restore) InnoDB Hot Backup tool, or XtraBackup

[email protected]


Logical Backups

148

Documentation on mysqldump: https://mariadb.com/kb/en/mariadb/mysqldump/ Documentation on SELECT…INTO OUTFILE: https://mariadb.com/kb/en/mariadb/select-into-outfile/

Produce Text Files with SQL statements to Replay to Rebuild Database

Allows Backup of Databases or Specific Tables — Schema or only Data

An SQL dump is independent of Storage Engine, and can be Restored to a different Storage Engine, or Used for Migration

Process can be Slow and Requires Locks Use a Local Drive, not across Network Multiple Options (mysqldump, "SELECT INTO OUTFILE", etc.)


https://mariadb.com/kb/en/mariadb/select-into-outfile/

[email protected]


Advantages & Factors of mysqldump

Documentation on mysqldump: https://mariadb.com/kb/en/mariadb/mysqldump/

mysqldump -u backup_user -p --all-databases > backup.sql mysqldump -u backup_user -p db1 table1 > table.sql mysqldump -u backup_user -p -h production | mysql -u dev_user -h development

Exports Schema and Data (SQL text file, tab-delimited, or XML) Good for Backups & Transferring Data between Servers Requires Locks for Consistency —May Affect Traffic Dump all Databases, or Specific Databases or Tables

Optimize for Importing (extended INSERT, temporarily disable indexes, and ignore foreign key checks)

149


[email protected]


Create a Back-Up User

150

CREATE USER 'admin_backup'@localhost IDENTIFIED BY 'rover123';

GRANT SELECT, RELOAD, LOCK TABLES, REPLICATION CLIENT, SHOW VIEW, EVENT, TRIGGER ON *.* TO 'admin_backup'@'localhost';

Create a Special User for Making Back-ups with mysqldump

[email protected]


Recovery with Binary Logs

151

Documentation on mysqlbinlog: https://mariadb.com/kb/en/mariadb/mysqlbinlog/

SHOW BINARY LOGS; SHOW MASTER STATUS;

Restore Databases from Dump file Use the Binary Logs to Execute

Remaining SQL Statements

mysqlbinlog binlog.000005 binlog.000006 | mysql -u admin_user -p

Use mysqlbinlog to Convert Binary Log and Pipe to mysql Client

https://mariadb.com/kb/en/mariadb/mysqlbinlog/

[email protected]


Point-in-Time Recovery

152

Documentation on Point-in-Time Recovery: http://dev.mysql.com/doc/refman/5.5/en/point-in-time-recovery.html

mysqlbinlog --stop-datetime='2017-02-10 11:59:59' \ /var/log/mysql/bin.000006 | mysql -u admin_user -p

Give a Stop Time to mysqlbinlog to Recover Until Then (e.g., Noon)

mysqlbinlog --start-datetime='2013-11-03 12:01:00' \ /var/log/mysql/bin.000006 | mysql -u admin_user -p

Give a Start Time to mysqlbinlog to Recover From Then

http://dev.mysql.com/doc/refman/5.5/en/point-in-time-recovery.html

[email protected]


Point-in-Time Recovery — Event Positions

153

mysqlbinlog --start-datetime='2013-11-03 11:55:00' \ --stop-datetime='2013-11-03 12:05:00' \ /var/log/mysql/bin.000001 > /tmp/mysql_restore.sql

Determine Event Position Numbers (log_pos)

mysqlbinlog --stop-position=720 /var/log/mysql/bin.0000006 \ | mysql -u admin_user -p

mysqlbinlog --start-position=828 /var/log/mysql/bin.000006 \

| mysql -u admin_user -p

Run mysqlbinlog on Binary Log until Stop Position Execute Binary Log again from a Start Position

[email protected]


Replication as a Back-up Method

154

Use MariaDB Replication to copy Data to Slave (close to real time, but technically asynchronous)

Recovery can be Fast and Simple Switch Traffic to Slave, Restore Master, Switch Traffic

back to Master Alternatively, Replay Binary Logs with mysqlbinlog tool

Allows other Methods to be executed on Slave without increasing Load on Master

[email protected]


XtraBackup (Percona)

155

Documentation on XtraBackup: http://www.percona.com/doc/percona-xtrabackup Installing XtraBackup: http://www.percona.com/doc/percona-xtrabackup/2.1/installation.html

Fast Back-Up Process Transactions Processed during Back-Up Process Low Drain on System Resources Back-Up Files Verified Automatically Faster Restore Process

http://www.percona.com/doc/percona-xtrabackup

http://www.percona.com/doc/percona-xtrabackup/2.1/installation.html

[email protected]


Develop a Back-Up PlanTake Inventory of Databases Write a Back-Up Schedule Write a Verification Schedule

Check and Test Back-Ups Practice Restoring

156

[email protected]


Take Inventory

157

List of Databases & Tables Number of Rows Frequency of Changes How Active Sensitivity of Data

Assessment of company DatabaseTables Rows Changing Active Sensitive

clients 1,159,913 ✓ ✓ ✓

clients_addresses 1,159,913 ✓ ✓ ✓clients_email 1,159,913 ✓ ✓ ✓clients_telephone 1,159,913 ✓ ✓ ✓employees 5,042 ✓ ✓employees_email 5,042 ✓employees_salaries 1,628 ✓ ✓employees_telephones 5,042 ✓org_departments 18

org_divisions 5

org_warehouses 5

ref_job_titles 240

ref_name_titles 9

ref_states 56

[email protected]


Back-Up Schedule

158

Back-Up Tables Frequency Schedule Secure Off-Site

company — full back-up (company-yyyy-mmm-dd.sql)

all tables weekly Sunday, 1:00 p.m.

✓ ✓

company — client tables (company-clients-yyyy-mmm-dd.sql)

clients, clients_addresses, client_telephones

daily every day, 1:00 a.m.

✓

company — employee tables (company-employees-yyyy-mmm-dd.sql)

employees, employees_email, employees_salaries, employees_telephones

daily every day, 1:15 a.m.

✓

Which Databases & Tables Labels and File Naming Patterns

Frequency — Days & Times Location — Security & Off-Site

[email protected]


Verification Schedule

159

Regularly Verify Back-Ups Practice Restoring Databases, Tables, Rows — Frequency Retention Periods

Restoration TestsBack-Up Verify Tables Rows Retention

company — full back-up weekly monthly two months

company — client tables weekly semi-monthly semi-monthly one month

company — employee tables weekly semi-monthly semi-monthly one monthBack-ups will be verified on a regular basis. For testing and practicing purposes, databases, tables, and rows will be restored regularly in a test environment.

MariaDB for DBAs

Customizing

Introduction

Architecture

Installation

Client Utilities

Configuration

Resource Usage

Storage Engines

Table Schema

Views

Partitioning


Log Files



Customizing

MariaDB Replication

High Availability

Optimization

[email protected]


Method and Value of Stored Routines

161

Documentation on CREATE PROCEDURE: https://mariadb.com/kb/en/create-procedure/ Documentation on CREATE FUNCTION: https://mariadb.com/kb/v/create-function/

Procedures and Functions are Supported Database Level Functions — Isolating Certain Functionality

Databases Used as Namespaces (e.g., CALL db_name.my_proc())

Routines are Dropped with Database

Library of Common Functions to make Complex Logic Accessible

https://mariadb.com/kb/en/create-procedure/

https://mariadb.com/kb/v/create-function/

[email protected]


Procedures vs. Functions

162

Stored Procedures Called Directly (e.g., CALL show_user()) Can Replace SQL statements, Encapsulate Complex Logic Can Recurse (see, max_sp_recursion_depth and thread_stack) Returns One or More Results Sets

User Defined Functions Called within other SQL Statements (e.g., SELECT DELTA_PCT(n, n)) Performs Smaller Data Manipulation, Calculations, or Conversion Cannot Recurse Returns a Single Scalar Value

[email protected]


User Defined Function Example

163

CREATE FUNCTION DELTA_PCT (old_nbr INT, new_nbr INT) RETURNS DECIMAL(6,2)

NO SQL COMMENT 'Fills need for percentage change function.' RETURN (new_nbr - old_nbr) / old_nbr;

SELECT DELTA_PCT(100, 125);

Identify Need: No Function for

Percentage of Change

Determine Any Input and Output

Develop SQL Statements

Stored Procures: https://mariadb.com/kb/v/stored-procedure-overview/ User Defined Functions: https://mariadb.com/kb/v/user-defined-functions/

GRANT EXECUTE ON my.* TO 'tim'@localhost;

https://mariadb.com/kb/v/stored-procedure-overview/

https://mariadb.com/kb/v/user-defined-functions/

[email protected]


Stored Procedure Example

164

DELIMITER // CREATE PROCEDURE SHOW_USERS()

CONTAINS SQL SQL SECURITY DEFINER COMMENT 'Fills need for SHOW USERS statement.' BEGIN SELECT User AS 'user',

SUBSTRING(Host, 1, 16) AS 'host', IF(Password != '', '','no') AS 'no pwd', IF(is_role = 'Y', 'yes', '') AS 'role', default_role AS 'default role'

FROM mysql.user ORDER BY User, Host; END // DELIMITER ;

CALL my.show_users;Stored Procures: https://mariadb.com/kb/v/stored-procedure-overview/ User Defined Functions: https://mariadb.com/kb/v/user-defined-functions/

Identify Need: No SHOW USERS Statement

Determine Any Input and Output Develop SQL Statements

https://mariadb.com/kb/v/stored-procedure-overview/

https://mariadb.com/kb/v/user-defined-functions/

[email protected]


Block and Scope

165

Block of SQL Statements Entered Between BEGIN and END

BEGIN SET max_connections = 10; ... END

SET for Non-Local Variables (i.e., external, user, system)

BEGIN DECLARE variable1 INT DEFAULT 100 ... END

CREATE PROCEDURE procedure1()

BEGIN SET @variable1 = CURDATE(); UPDATE table1 SET col1 = @variable1; END

Local Variables Declared Between BEGIN and END

Variable Scope can be Local or Outside the Block

Documentation on BEGIN…END: https://mariadb.com/kb/v/begin-end/

https://mariadb.com/kb/v/begin-end/

[email protected]


Flow Control — Conditions & Switches

166

Documentation on IF statement: https://mariadb.com/kb/v/if-statement/

Conditional BranchingDECLARE value1 INT

CASE value1 WHEN 10 THEN statements WHEN 20 THEN statements ELSE statements

END CASE

IF condition THEN statements

ELSEIF condition THEN statements ELSE statements

END IF

Switches or Case Statements

https://mariadb.com/kb/v/if-statement/

[email protected]


Flow Control — Loops

167

Documentation on LOOP: https://mariadb.com/kb/v/loop/

Simple LOOP Syntax Example

CREATE PROCEDURE procedure1 BEGIN LOOP

statements END LOOP END //

https://mariadb.com/kb/v/loop/

[email protected]


Flow Control — Loop Methods

168

Documentation on REPEAT: https://mariadb.com/kb/v/repeat-loop/ Documentation on WHILE: https://mariadb.com/kb/v/while/ Documentation on ITERATE: https://mariadb.com/kb/v/iterate/ Documentation on LEAVE: https://mariadb.com/kb/v/leave/

CREATE PROCEDURE procedure3

DECLARE c1 INT; label3: LOOP SET c1 = c1 +1; IF c1 < 5 ITERATE label3;

END IF; LEAVE label3; END LOOP label3 END //

ITERATE & LEAVE — Label Required

CREATE PROCEDURE procedure2

label1: WHILE condition DO statements END WHILE label1

label2: REPEAT statements

UNTIL condition END REPEAT label2

END //

REPEAT & WHILE Syntax— Label Optional

https://mariadb.com/kb/v/repeat-loop/

https://mariadb.com/kb/v/while/

https://mariadb.com/kb/v/iterate/

https://mariadb.com/kb/v/leave/

[email protected]


Error Handling

169

Documentation on DECLARE HANDLER: https://mariadb.com/kb/v/declare-handler/ Documentation on SIGNAL: https://mariadb.com/kb/v/signal/

DECLARE condition_name CONDITION FOR condition_value

DECLARE is Used also for Conditions

SIGNAL condition_value SET signal_information

SIGNAL for Complete Error Handling

DECLARE handler_type HANDLER FOR condition_value statement

DECLARE can Declare Handlers

https://mariadb.com/kb/v/declare-handler/

https://mariadb.com/kb/v/signal/

[email protected]


Cursors

170

Documentation on DECLARE CURSOR: https://mariadb.com/kb/v/declare-cursor/

BEGIN <variable_declarations> DECLARE cursor1 CURSOR FOR

SELECT col1,col2 FROM table1; <handler_declarations>

OPEN cursor1; read_loop: LOOP

FETCH cursor1 INTO a, b; ... END LOOP; CLOSE cursor1;

END //

DECLARE Used to Define a Cursor

OPEN, FETCH, CLOSE, then Operate the Cursor

Cursors are also Supported in Triggers and Events

https://mariadb.com/kb/v/declare-cursor/

[email protected]


Triggers

171

Documentation on Triggers: https://mariadb.com/kb/v/trigger-overview/

Executes Automatically SQL Statements — Not for Views or Temporary Tables

Useful for Checking, Filtering, Adjusting Data before Writing

Maintaining Related Data (History Tables, Materialized Views) Mimics Foreign Keys and Constraints for All Storage

Engines

Executes Once per Row, Not Once per SQL Statement Separate Triggers Allowed for Before and After SQL

Statements Stored in .TRG Files in the Database Folder

https://mariadb.com/kb/v/trigger-overview/

[email protected]


Trigger Example

172

Documentation on CREATE TRIGGER: https://mariadb.com/kb/en/create-trigger/ Documentation on DROP TRIGGER: https://mariadb.com/kb/v/drop-trigger/

CREATE TRIGGER salary_change

BEFORE UPDATE ON company.employees_salaries FOR EACH ROW INSERT INTO admin_log.co_salary_chg (user_info, salary_id, amount_before, amount_after, chg_date)

VALUES(USER(), OLD.salary_id, OLD.amount, NEW.amount, NOW());

Action to Trigger: an SQL Statement of Multiple Statements with BEGIN…END

Can't Create Triggers on mysql Database

Timing of Trigger: BEFORE or AFTER

Type of SQL Statement to Start Trigger: INSERT, UPDATE or DELETE

CREATE TABLE admin_log.co_salary_chg

(change_id INT AUTO_INCREMENT KEY, user_info VARCHAR(141), salary_id INT, amount_before DECIMAL(9,2), amount_after DECIMAL(9,2), chg_date DATETIME);

May want to surpress warnings (e.g., binary_format = 'mixed')

https://mariadb.com/kb/en/create-trigger/

https://mariadb.com/kb/v/drop-trigger/

[email protected]


Event Scheduler

173

Documentation on Event Scheduler: https://mariadb.com/kb/v/events/

Executes Tasks based on Predefined Schedule Alternative to cron or similar Utilities — Portable with

Database

Enable and Activate the Event Scheduler

event_scheduler = ON

Entry in Configuration File (e.g., my.cnf)

https://mariadb.com/kb/v/events/

[email protected]


Creating & Monitoring Events

174

Documentation on CREATE EVENT: https://mariadb.com/kb/v/create-event/ Documentation on SHOW EVENTS: https://mariadb.com/kb/v/show-events/

CREATE EVENT cleanup_log ON SCHEDULE EVERY 1 HOUR DO DELETE FROM admin_log.user_logins WHERE login_time < DATE_SUB(NOW(), INTERVAL 1 HOUR);

Schedule One-Time or Reoccurring Events Specify Optionally Start and End Dates and Times

Check and Monitor Regularly EventsSHOW EVENTS; SHOW SCHEDULER STATUS;

https://mariadb.com/kb/v/create-event/

MariaDB for DBAs

MariaDB Replication

Introduction

Architecture

Installation

Client Utilities

Configuration

Resource Usage

Storage Engines

Table Schema

Views

Partitioning


Log Files



Customizing

MariaDB ReplicationHigh Availability

Optimization

[email protected]


Purpose of MariaDB Replication

176

Load Balancing (Scaling SELECT Queries) Move Slow, Heavy Queries to Slave Take Slave Off-Line to Make Back-ups

Multiple Data Centers Need Fast Reads Gain Redundancy (High Availability)

Fail Over - Promote Quickly a Slave to Master Fail Over Isn’t Automatic — Requires External

Monitoring Minimal Downtime for Upgrades or Schema Changes

Apply Changes to a Slave Promote Slave to Master and Redirect Traffic Apply Changes to Master and Switch

[email protected]


Replication Terrain

177

mysqld Data Storage

Master

Binary Log

mysqld Data Storage

Slave 1

Relay Log

Data Storage

Slave 2/Master 2

Relay Log

Binary Log

mysqldSlave 2A

Slave 2B

Slave 2C

INSERT UPDATE DELETE

CREATE ALTER DROP

Dump Thread

Client Threads

SQL Thread

IO Thread

IO Thread

IO Thread

[email protected]


MariaDB Replication Factors

178

One Master, Multiple Slaves No true Multi-Master Solution, but Circular Replication

Close to Real Time, but Asynchronous Semi-Synchronous Replication Mode Crash-Safe Slaves with Transactional Storage Engines

A Slave may also be a Master Set log_slave_updates in Configuration File Apply Optionally Replication Filtering Rules or Storage

Engine Changes on Intermediate Slaves

[email protected]


Replication — Threads

179

Master binlog Dump Thread Pushes binlog Events to Slave Visible in SHOW PROCESSLIST as "Binlog Dump"

Slave IO Thread — Visible in SHOW SLAVE STATUS Requests and receives binlog events from the Master Writes them to the local relay log

Slave SQL Thread — Visible in SHOW SLAVE STATUS

Reads the Relay Log and Executes Queries on Local Data Checks the Query Result Codes Match those Recorded by Master

Slave Multiple Execution Threads Multi-Threaded Slave separates events based on Database Names Updates are Applied in Parallel, Not Sequence

[email protected]


Parallel Replication

180

Documentation on Parallel Replication: https://mariadb.com/kb/en/mariadb/documentation/replication/standard-replication/parallel-replication/

Replication Process on Slaves Events Received from Master by IO Thread and Queued

in Relay Log Each Relay Log Entry is Retrieved by the SQL Thread Each Transaction is Applied to the Slave

On Non-Parallel Systems, Application Performed Sequentially by SQL Thread

On Parallel Systems, Application Performed in Pool of Separate Replication Worker Threads

https://mariadb.com/kb/en/mariadb/documentation/replication/standard-replication/parallel-replication/

[email protected]


Topologies

181

Master to Slave Simplest Solution and Used Most Widely Allows Off-Loading of SELECT Traffic to Slave

Master1 to MasterN ... to Master1 (circular) Servers Replicate in a Circle, with binlog Events

Traversing the ring until Originating Server Does Not Alleviate Heavy Write Load Needs Careful setup of server-id and

auto_increment_offset, auto_increment_increment settings

Master to Slave to Slaves Can build Complex Trees Useful for Replication Rules or Storage Engine Changes

[email protected]


Master Configuration

182

GRANT REPLICATION SLAVE ON *.* TO 'maria_replicator'@'52.34.19.24' IDENTIFIED BY 'rover123';

Enable Binary Log — Choose Binary Log Format Set server-id in Configuration File to Unique Value Create Replication User Account on Master

Make a Consistent Snapshot of Data on Master

mysqldump -p -u admin_backup --master-data --flush-logs \ --all-databases > full-dump.sql

[email protected]


Slave Configuration

183

Documentation on Slave Options: https://mariadb.com/kb/en/mariadb/replication-and-binary-log-server-system-variables/ Documentation on CHANGE MASTER TO: https://mariadb.com/kb/en/mariadb/change-master-to/

CHANGE MASTER TO

MASTER_HOST='35.161.145.71',

MASTER_PORT=3306,

MASTER_USER='maria_replicator',

MASTER_PASSWORD='rover123';

Set server-id in Configuration File to Unique Value Add read-only in Configuration File to Prevent Writes Set Optionally Replication Rules — Covered Later in Class

Restart MariaDB Load Data from Master

Execute START SLAVE on Slave

mysql -p -u root < full-dump.sql

scp -i ./.ssh/mariadb.pem [email protected]:/home/ec2-user/full-dump.sql .

https://mariadb.com/kb/en/mariadb/replication-and-binary-log-server-system-variables/

https://mariadb.com/kb/en/mariadb/change-master-to/

[email protected]


Monitoring Replication

184

Documentation on SHOW MASTER STATUS: https://mariadb.com/kb/en/show-master-status/ Documentation on SHOW SLAVE STATUS: https://mariadb.com/kb/en/mariadb/show-slave-status/

SHOW MASTER STATUS;

Check Regularly Status on Master — Includes binlog number and position

SHOW SLAVE STATUS \G

Slave_IO_State: Waiting for master to send event Slave_IO_Running: Yes

Slave_SQL_Running: Yes Last_Errno: 0 Last_Error: Seconds_Behind_Master: 300

Check More Often Status of Replication on Slave

https://mariadb.com/kb/en/show-master-status/

https://mariadb.com/kb/en/mariadb/show-slave-status/

[email protected]


Replication Files

185

Documentation mysqlbinlog: https://mariadb.com/kb/en/mariadb/using-mysqlbinlog/

Binary Log Files (Master) Master Records Write-Queries to File Rotated when Flushed or Periodically to New Log File — File

Name Pattern (.000001)

Relay Log File (Slave) Record of Master binlog Events Rotated when Flushed or Periodically — File Name Pattern (.000001)

Replication Configuration Recorded in master.info (Slave)

Name of Relay Log File Recorded in relay-log.info (Slave)

https://mariadb.com/kb/en/mariadb/using-mysqlbinlog/

[email protected]


Slave Configuration Files

186

52.89.128.176 maria_replicator rover123 3306 60

0 ...

master.info

196803 mariadb-bin.000040

196513 30217 9 ...

relay-log.info

[email protected]


Replication File Maintenance & Back-Ups

187

Replication Files Updated & Purged Automatically — Don’t Edit or Move Manually

Include Replication Files when Making Binary Back-ups

Use --raw option with mysqlbinlog to Back-up Binary Log

[email protected]


Binary Log Format

188

Documentation on Binary Log Format: https://mariadb.com/kb/en/mariadb/binary-log-formats/

Statement Based (SBR) — Original Queries are Replicated

Least Data Sent over Wire and Tested for Years Non-Deterministic Statements Executed on Slave — Slave Load is Increased vs. RBR

Row Based (RBR) - Table Rows are Replicated Only Non-Deterministic Statements Executed on Master — Slave Load is Reduced vs. SBR More Data sent over Wire — Not Supported by All Engines

Mixed (default) - Smart Switching between SBR and RBR[mysqld]

binlog-format=MIXED

Checksum (--binlog-checksum) in Binary and Relay Logs to detect Errors

Includes Errors in Memory, Disk, Network and Database Can be Implemented for each Slave

https://mariadb.com/kb/en/mariadb/binary-log-formats/

[email protected]


Slave Filtering Rules — Database Level

189

Documentation on Slave Options: https://mariadb.com/kb/en/mariadb/replication-and-binary-log-server-system-variables/

Exclude Specific Databases (e.g., mysql)

CHANGE REPLICATION FILTER REPLICATE_IGNORE_DB = (), REPLICATE_DO_DB = (sales,inventory);

CHANGE REPLICATION FILTER REPLICATE_IGNORE_DB = (mysql);

Include Specific Databases

Excluding can Cause Problems with Joins

SET GLOBAL replicate_ignore_db = ''; SET GLOBAL

replicate_do_db = 'sales,inventory';

SET GLOBAL replicate_ignore_db = 'mysql';


[email protected]


Slave Filtering Rules — Table Level

190

Documentation on Slave Options: https://mariadb.com/kb/en/mariadb/replication-and-binary-log-server-system-variables/

Ignore Specific Tables

CHANGE REPLICATION FILTER REPLICATE_IGNORE_DB = (employees), REPLICATE_DO_TABLE = (employees.names, employees.contacts);

CHANGE REPLICATION FILTER REPLICATE_IGNORE_TABLE = (employees.salary);

CHANGE REPLICATION FILTER

REPLICATE_IGNORE_DB = (sales), REPLICATE_DO_TABLE = (sales.europe_%), REPLICATE_IGNORE_TABLE = (sales.europe_uk_%);

Include Specific Tables

Wildcards for Multiple Tables


[email protected]


MariaDB Replication - Asynchronous

191

Master Doesn’t Wait for Slaves IO Thread may be Slow to Receive binlog Packets

Network Congestion or Disconnects

SQL Thread may be Slow in Processing Relay Log Events

Load on Slave or Network Problems

[email protected]


Semi-Synchronous

192

Documentation on Semi-Synchronous Replication: https://mariadb.com/kb/en/mariadb/semisynchronous-replication/

Implemented with an optional Plug-In A COMMIT on Master can Wait for a Slave to

Acknowledge it’s Received Transaction Master Waits for Slave to Write

Transaction to Relay Log, Not to Execute Transaction — Slave SQL Thread may still Lag

One Slave Response needed for Master to Continue (Semi-Synchronous, Not Synchronous)

Can Affect Significantly Performance of Master

INSTALL PLUGIN rpl_semi_sync_master SONAME 'semisync_master.so';

INSTALL PLUGIN rpl_semi_sync_slave SONAME 'semisync_slave.so';

https://mariadb.com/kb/en/mariadb/semisynchronous-replication/

[email protected]


Lagging Slave

193

Documentation on Time-Delayed Replication: http://dev.mysql.com/doc/refman/5.6/en/replication-delayed.html

When Slave SQL Thread is Slow or Disabled due to Errors, Slave is said to Lag behind Master

Slave SQL Thread must Execute Serially Queries that were Executed in Parallel on the Master

Slave Multiple Execution Threads Multi-Threaded Slave separates events based on

Database Names Updates are Applied in Parallel, Not Sequence

Time-Delayed Replication - Set Provides a Buffer to Stop Replication of Mistakes

CHANGE MASTER TO MASTER_DELAY = seconds;

http://dev.mysql.com/doc/refman/5.6/en/replication-delayed.htm

[email protected]


Troubleshooting Problems with Replication

194

Check Slave Error Log for Errors affecting Replication Look for Disconnects from Network Problems Binary or Relay Log Event Corruption will cause Slave

SQL Thread to Stop Different Query Error Codes on Slave indicate it’s Not

Synchronized with Master

Tools like Maatkit can help with Replication Troubleshooting and Recovery

May Need to Rebuild Slave from a fresh Snapshot (Back-up of Master or Another Slave)

MariaDB for DBAs

High Availability

Introduction

Architecture

Installation

Client Utilities

Configuration

Resource Usage

Storage Engines

Table Schema

Views

Partitioning


Log Files



Customizing

MariaDB Replication

High AvailabilityOptimization

[email protected]


High Availability Goals and Concepts

196

Never Lose Data because of a Crash — Users Should Never Notice

Remain Operational Despite Unforeseen Problems

System Redundancy of Software and Hardware Write Data to Multiple Devices and Locations

(e.g., RAID, Replication, Galera, DRBD) A Second Database Server Designated as a Hot

Spare

[email protected]


High Availability Goals — Reality Check

197

Availability Down-Time90% 1 Nine 36.5 days per year 99% 2 Nines 3.65 days per year 99.9% 3 Nines 8.76 hours per year99.99% 4 Nines 52 minutes per year 99.999% 5 Nines 5 minutes per year 99.9999% 6 Nines 31 seconds per year

Availability = Up-Time / (Up-Time + Down-Time)

Availability = Mean Time Before Failure / (Mean Time Before Failure + Mean Time To Recovery)

[email protected]


MariaDB Replication

198

Close to Real Time — Asynchronous or Semi-Synchronous

Mono-Threaded on Slaves Semi-Synchronous Replication Mode

A Slave may also be a Master Set log_slave_updates in Configuration File

Master High Availability Manager Monitors the Master Automates Master Fail-Over and Slave Promotion Integrates with Pacemaker & Heartbeat

[email protected]


MariaDB MaxScale

199

Tutorials on MaxScale: https://github.com/mariadb-corporation/MaxScale/tree/develop/Documentation/Tutorials Documentation on MaxScale: https://github.com/mariadb-corporation/MaxScale/blob/develop/Documentation/Getting-Started/MariaDB-MaxScale-Installation-Guide.md

Application-to-Database Dynamic Data Routing Platform

Insulates Client Applications from Complexities of Backend Database Cluster

Improves Database Availability, Security and Scalability

Database-to-Database Dynamic Data Routing Platform

Simplifies Interoperability across Databases

MaxScale

MariaDB Master

MariaDBSlave

MariaDBSlave

Client Client Client

https://github.com/mariadb-corporation/MaxScale/tree/develop/Documentation/Tutorials

https://github.com/mariadb-corporation/MaxScale/blob/develop/Documentation/Getting-Started/MariaDB-MaxScale-Installation-Guide.md

[email protected]


MariaDB Replication Manager

200

Download MariaDB Replication Manager: https://github.com/mariadb-corporation/replication-managerl

High Availability Solution Monitor and Administer MariaDB Replication and

MariaDB Enterprise Clusters On-Demand Slave to Master Promotion (i.e.,

Switchover) Electing a New Master on Failure Detection — MaxScale

will Route Queries to New Master

https://github.com/mariadb-corporation/replication-manager

[email protected]


MariaDB Galera Cluster

201

Documentation for Galera: http://galeracluster.com/documentation-webpages/

Virtually Synchronous Replication Uses Only InnoDB Storage Engine True Multi-Master Solution Conflict Detection & Resolution upon Commit Automatic Provisioning Nodes Isolated Easily for Maintenance

http://galeracluster.com/documentation-webpages/

[email protected]


Distributed Replicated Block Device

202

DRBD User Guide: http://www.drbd.org/en/doc/users-guide-84/p-introl

DRBD is a Linux Kernel Module, Providing Synchronous Replication of a Block Device between Two Servers — Hot Spare Server

If Primary Server Fails, Secondary Server is Used — Immediately and Seamlessly

DRBD Disk Writes over Network slows MariaDB

http://www.drbd.org/en/doc/users-guide-84/p-intro

[email protected]


Shared-Disk Architecture

203

More information on Shared Storage: https://mariadb.com/sites/default/files/SharedStorage.pdfl

Active – Passive replication Failover requires MariaDB Crash

Recovery — Often File System Crash Recovery Non-Transactional — Only MyISAM Storage

Engine Combined with Pacemaker/Heartbeat

for Auto Failover Virtual IP often used to Fail Over

In theory the SAN is a SPOF

https://mariadb.com/sites/default/files/SharedStorage.pdf

MariaDB for DBAs

Optimization

Introduction

Architecture

Installation

Client Utilities

Configuration

Resource Usage

Storage Engines

Table Schema

Views

Partitioning


Log Files



Customizing

MariaDB Replication

High Availability

Optimization

[email protected]


Optimization Concepts & Considerations

205

Performance Optimizing relates to Resolving Bottlenecks to Handle More Traffic or Data at a Time

Efficiency Optimizing has to do with the Best Use of Hardware, Bandwidth, and Budget

Consider Carefully Schema and Indexes Don’t Spend too much Time on

Optimizing without Data — Need Real Data and Traffic to Optimize Effectively

[email protected]


Query Tuning

206

Don’t Tune Queries on Production Server Tune Queries on a Test Server Use a Complete Copy of Data Use Indexes to Improve Read Performance

[email protected]


Query Analysis

207

Documentation on EXPLAIN: https://mariadb.com/kb/en/mariadb/explain/ Documentation on EXPLAIN EXTENDED: https://mariadb.com/kb/en/mariadb/explain/#explain-extended Documentation on mysqldumpslow: https://mariadb.com/kb/en/mariadb/mysqldumpslow/

EXPLAIN SELECT * FROM employees WHERE MONTH(birth_date) = 8 \G id: 1 select_type: SIMPLE

table: employees type: ALL possible_keys: NULL key: NULL

key_len: NULL ref: NULL rows: 299587 Extra: Using where

Use the Slow Log to find Problem Queries (--log-slow-queries)

Use mysqldumpslow Utility for Manageable Reports

Use EXPLAIN to see how MariaDB Executes a Troublesome Query and if Indexes are Used

Use EXPLAIN EXTENDED and SHOW WARNINGS to see how MariaDB Rearranges a Query before Execution

EXPLAIN EXTENDED SELECT …; SHOW WARNINGS;

https://mariadb.com/kb/en/mariadb/explain/

https://mariadb.com/kb/en/mariadb/explain/#explain-extended

https://mariadb.com/kb/en/mariadb/mysqldumpslow/

[email protected]


Fields from EXPLAINField Descriptionid Query identifierselect_type Type of SELECT performed

table The table in use (each table in a JOIN will use one row)

*type The table access strategy

possible_keys Any available indexes that could resolve the query

key Index MariaDB chose from possible_keyskey_len The portion of the chosen index to be used (width in bytes)ref Columns from the index, or a constant, usedrows Number of rows the query is expected to touchExtra Extra information relating to the access strategy

208

Documentation on EXPLAIN: https://mariadb.com/kb/en/mariadb/explain/ Documentation on EXPLAIN EXTENDED: https://mariadb.com/kb/en/mariadb/explain/#explain-extended

* See Next Slide for List of Types

https://mariadb.com/kb/en/mariadb/explain/

https://mariadb.com/kb/en/mariadb/explain/#explain-extended

[email protected]


Scan or Index Method TypesField Description

const A constant value can be read once and cached

eq_ref One index access per outer query row

ref Multiple index accesses per outer query row

index_merge Multiple indexes used, merging into a single result set

range Multiple index accesses to return all rows within a range

index Full index scan (every index entry is read sequentially)

all Full table scan (every record is read sequentially)

209

[email protected]


Choosing Data Types

210

Documentation for PROCEDURE ANALYSE( ): https://mariadb.com/kb/en/mariadb/procedure-analyse/

Appropriate Type (e.g., INT for numbers not VARCHAR)

Accurate Type (e.g., DECIMAL for currency, not DOUBLE)

Smallest Useful Type (e.g., CHAR for known text length, not VARCHAR)

Use NOT NULL, if practical

Analyze Columns with PROCEDURE ANALYSE()

SELECT col1 FROM table1

WHERE col1 = 'test' PROCEDURE ANALYSE() \G

Field_name: db1.table1.col1 Min_value: 7450554

Max_value: 8592336 Min_length: 7 Max_length: 7 Empties_or_zeros: 0

Nulls: 0 Avg_value_or_avg_length: 7955635.7647 Std: 342328.4478 Optimal_fieldtype: MEDIUMINT(7) UNSIGNED NOT NULL

https://mariadb.com/kb/en/mariadb/procedure-analyse/

[email protected]


Optimize Tables

211

Appropriate Data Types - Smaller is Better Shorten Variable Length Columns (i.e., VARCHAR)

— May Expand to Full Width in Memory Use PROCEDURE ANALYSE() for Data Type

Recommendations

DESCRIBE employees;

+------------+---------------+------+

| Field | Type | Null |

+------------+---------------+------+

| emp_no | int(11) | NO |

| first_name | varchar(255) | YES |

| last_name | varchar(255) | YES |

+------------+---------------+------+

SELECT first_name, last_name FROM employees PROCEDURE ANALYSE() \G *************************** 1. row *********************

Field_name: employees.employees.first_name

Min_value: Aamer

Max_value: Zvonko

Min_length: 3

Max_length: 14

Empties_or_zeros: 0

Nulls: 0

Avg_value_or_avg_length: 6.2157

Std: NULL

Optimal_fieldtype: VARCHAR(14) NOT NULL

*************************** 2. row *********************

Field_name: employees.employees.last_name

Min_value: Aamodt

Max_value: Zykh

Min_length: 4

Max_length: 16

Empties_or_zeros: 0

Nulls: 0

Avg_value_or_avg_length: 7.1539

Std: NULL

Optimal_fieldtype: VARCHAR(16) NOT NULL

[email protected]


Indexes for Performance Optimization

212

MariaDB can Resolve Queries Faster with Indexes Faster Retrieval of Matching Rows, Faster Sorting of

Result Sets Without Indexes, MariaDB does Full Table Scan

Indexes can be for a Column or Multiple Columns (i.e., Composites)

Index for Speed, but Avoid Indexing Excessively or Arbitrarily

Remove Unused or Redundant Indexes

[email protected]


Table Indexes

213

SHOW CREATE TABLE clients \G

*************************** 1. row ***********************

Table: clients

Create Table: CREATE TABLE 'clients' (

'client_id' int(11) NOT NULL AUTO_INCREMENT,

'name_first' varchar(255) DEFAULT NULL,

'name_last' varchar(255) DEFAULT NULL,

PRIMARY KEY ('client_id'),

KEY 'client_names'

('name_last'(10),'name_first'(5)))

ENGINE=InnoDB DEFAULT CHARSET=latin1

Use SHOW CREATE TABLE to see a Table's Indexes

Documentation on CREATE INDEX: https://mariadb.com/kb/en/mariadb/create-index/

Two Methods for Creating an Index for an Existing Table

ALTER TABLE employees ADD INDEX emp_names (name_last, name_first);

CREATE INDEX client_names ON clients (name_last(10), name_first(5));

Use ALTER TABLE or CREATE INDEX to add an Index

https://mariadb.com/kb/en/mariadb/create-index/

[email protected]


Creating Indexes

214

CREATE TABLE people

(id INT, name CHAR(20), age TINYINT UNSIGNED, KEY aged_name(age, name));

ALTER TABLE employees

ADD INDEX full_name (name_last, name_first);

CREATE INDEX short_name

ON employees (name_last(8), name_first(5));

Indexes are Created with CREATE TABLE and ALTER TABLE

Create an Index for a new Table with CREATE TABLE

Add an Index to a Table with ALTER TABLE or CREATE INDEX

Keep Indexes as Small as Practical For Indexing Strings, Use Prefix

[email protected]


Better JOIN Construction

215

SELECT last_name, city, country_name FROM clients

JOIN country_names USING(country_id) WHERE clients.country_id IN('de','fr','it','uk') ORDER BY country_name, city, last_name;

Foreign Keys used with Joins should be Indexed Index Columns in ON and USING Clauses

Keep GROUP BY and ORDER BY columns in one Table for better Indexing

[email protected]


Optimize Tables

216

Minimize Table Size on Disk and in Memory Use Partitioning Spread Large Tables across Multiple Servers if Application

Permit Archive Table Data when Possible and Appropriate

Index Appropriately Remove Duplicate or Unused Indexes Don’t Index a Column if First Field in Another Index

[email protected]


Buffer Pool

217

Use the SHOW STATUS statement to see Variables related to Buffer Pool

SHOW VARIABLES LIKE '%innodb_buffer_pool%';

+---------------------------------------+-----------+


+---------------------------------------+-----------+

| innodb_buffer_pool_instances | 1 |

| innodb_buffer_pool_populate | OFF |

| innodb_buffer_pool_restore_at_startup | 0 |

| innodb_buffer_pool_shm_checksum | ON |

| innodb_buffer_pool_shm_key | 0 |

| innodb_buffer_pool_size | 134217728 |

+---------------------------------------+-----------+

SHOW STATUS LIKE '%innodb_buffer_pool%';

+-----------------------------------------+-----------+


+-----------------------------------------+-----------+

| Innodb_buffer_pool_pages_data | 8032 |

| Innodb_buffer_pool_bytes_data | 131596288 |

| Innodb_buffer_pool_pages_dirty | 0 |

| Innodb_buffer_pool_bytes_dirty | 0 |

| Innodb_buffer_pool_pages_flushed | 850451 |

| Innodb_buffer_pool_pages_LRU_flushed | 119 |

| Innodb_buffer_pool_pages_free | 1 |

| Innodb_buffer_pool_pages_made_not_young | 0 |

| Innodb_buffer_pool_pages_made_young | 207678 |

| Innodb_buffer_pool_pages_misc | 158 |

| Innodb_buffer_pool_pages_old | 2944 |

| Innodb_buffer_pool_pages_total | 8191 |

| Innodb_buffer_pool_read_ahead_rnd | 0 |

| Innodb_buffer_pool_read_ahead | 0 |

| Innodb_buffer_pool_read_ahead_evicted | 0 |

| Innodb_buffer_pool_read_requests | 215938795 |

| Innodb_buffer_pool_reads | 37986 |

| Innodb_buffer_pool_wait_free | 0 |

| Innodb_buffer_pool_write_requests | 4817642 |

+-----------------------------------------+-----------+

[email protected]


Optimize Hardware

218

Benchmark Systems using Real Data and Real Traffic

Search for Performance Limits Related to Equipment Upgrade, Fine Tune, or Buy More Hardware Buy Fast Disks for Large Databases More RAM for Faster Query Processing - Enough Memory

for Most or All Results Sets More CPU Cores Helps High Traffic Sites

Ideally, Run Only MariaDB on Server (e.g., Web, Email Service on Separate Server)

MariaDB for DBAs

Conclusion

Introduction

Architecture

Installation

Client Utilities

Configuration

Resource Usage

Storage Engines

Table Schema

Views

Partitioning


Log Files



Customizing

MariaDB Replication

High Availability

Optimization

[email protected]


On-Line Resources

220

MariaDB Documentation and Knowledge Base: http://mariadb.com/kb

Documentation Knowledge Base Open-Source Community

Post Questions on Forums Ask Questions on IRC — FreeNode

http://mariadb.com/kb

[email protected]


Other Training Courses

221

Training Courses: https://mariadb.learnupon.com/store Upcoming Courses: https://mariadb.learnupon.com/store/sessions

Public In-Person Courses Private In-Person Courses Live Virtual Training Courses Self-Paced On-Line Courses

https://mariadb.learnupon.com/store

https://mariadb.learnupon.com/store/sessions

MariaDB for DBAs

Thanks for Participating

Documents

MariaDB for DBAs · PDF fileMariaDB for DBAs Installation Introduction Architecture Installation Client Utilities Conﬁguration Resource Usage Storage Engines Table Schema