New data dictionary an internal server api that matters

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

New Data Dictionary:An Internal Server APIThat Matters

Alexander Nozdrin, Principle Software Developer

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2

Safe Harbor StatementThe following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.


MySQL Community Reception @ Oracle OpenWorldMingle with the MySQL community and the MySQL team from Oracle for a fun and informative evening!

• Time: September 30 (Tue) @ 7pm• Jillian’s at Metreon 175 Fourth Street, San Francisco, CA

At the corner of Howard and 4th st.; only 2-min walk from Moscone Center


Agenda

What is a Data Dictionary?

The MySQL Traditional Data Dictionary

New Data Dictionary

Benefits for users

Q & A

1

2

3

4

5


Data Dictionary

What is it


Data Dictionary

• Metadata is information about user data– User table structure– Column definitions– Index definitions– Foreign key definitions– Stored program definitions...

• Data Dictionary collects all metadata in RDBMS

Definition


CREATE PROCEDURE p1(v INT)SQL SECURITY INVOKERBEGIN...END

Data DictionaryOverview

Data Dictionary

Table Definitions SP Definitions

View Definitions Plugins

Time zones

Privileges

CREATE TABLE customers( id INT AUTO_INCREMENT ... PRIMARY KEY (id), INDEX ... FOREIGN KEY ...)


MySQL Server

Data Dictionary DefinitionThe ecosystem

Query Executor

Optimizer

PerformanceSchema InnoDBSE

SQLstatement

Client Parser

Result

Information Schema

DataDictionary



MySQL 5.6 and earlier


The MySQL Traditional Data Dictionary• A mix of files and tables:– File based• Tables: FRM• Triggers: TRN, TRG...

– Table based• mysql.time_zone...

• InnoDB has a separate data dictionary


The MySQL Traditional Data DictionaryData Dictionary

Files

FRM TRG OPT

System tables (mysql.)

user proctime_zone

InnoDB internal data dictionary

CSV

Archive

InnoDB

INFO

RMATIO

N_SCH

EMA

File Scan

Table Scan

Intern.Access



• Poor INFORMATION_SCHEMA performance• Makes crash-safe / transactional DDL impossible• Inconsistencies between files and tables• Inconsistencies between DD in InnoDB and the server• File-system dependency (lower-case-table-names)• Makes replication of DDL statements difficult• Too difficult to extend

Problems


New Data Dictionary

A great leap forward


• Stored in InnoDB tables• Reliable & crash-safe• Single repository of metadata– for the MySQL server – for Storage Engines– for Plugins

• Redundancy• Data Dictionary API

• INFORMATION_SCHEMA SQL VIEWs–Queries can be optimized– Improved performance

• Metadata versioning• Extendable– Simplify metadata upgrades– Designed with plugins in mind

New Data Dictionary : Main Features


New Data DictionaryTransition

Data DictionaryFiles

FRM TRG OPT

System tables (mysql.)

user proctime_zone

InnoDB internal dictionary

CSV

Archive

InnoDBINFO

RMAT

ION

_SCH

EMA

DD TableSQL VIEW

File Scan

Table Scan

Intern.Access


New Data DictionaryOverview

InnoDB

Data Dictionary

DD Table User Table

INFORMATIONSCHEMA

Views

Archive

User Table

CSV

User Table


InnoDB

New Data DictionaryArchitecture

Query Executor

Parser Optimizer

Data Dictionary Tablespace

Data Dictionary Internal API

Data Dictionary External API

Plugin Plugin

Plugin Plugin Plugin

Archive

User TableTablespace Storage Engine

Storage Engine


New Data Dictionary

• WL#6379: Schema definitions for new DD• InnoDB Data Dictionary Tablespace• Designed with INFORMATION_SCHEMA in mind• Ability to store SE-specific data• Use PK / FK to ensure consistency

Data Dictionary tables


New Data Dictionary API

• The only way to access Data Dictionary– For the server core– For Storage Engines– For plugins

• Hard to misuse• Internal API (non-stable) and external API (stable)• Provide a way to handle SE specific data

Design goals


New Data Dictionary

SDI : Serialized Dictionary Information

Redundancy

InnoDB

Single User TS

User Table

SDI

General TS

User Table

SDI

System TS

User Table

SDI

Data Dictionary

User Table Definition

Stored Program

Privileges


New Data DictionaryFRM shipping for MySQL Cluster?

SELECT ... FROM t1

FRMSDI


• No FRM files• New INFORMATION_SCHEMA• Migrated to InnoDB:– time zone tables– help tables–mysql.plugins–mysql.servers

• Draft of Data Dictionary API

• http://labs.mysql.com– Do NOT use it in production– Install on a spare server

• MTR can be run

New Data Dictionary : Labs Release

http://labs.mysql.com/

http://labs.mysql.com/


New Data Dictionary

Why does it matter for YOU?


INFORMATION_SCHEMA performance improvementsGet per table size Blog post by Shlomi Noach: http://tinyurl.com/y8cnj7o

SELECT TABLE_SCHEMA, TABLE_NAME, ENGINE,SUM(DATA_LENGTH+INDEX_LENGTH) AS size,SUM(INDEX_LENGTH) AS index_size

FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA NOT IN ('mysql', 'INFORMATION_SCHEMA') AND ENGINE IS NOT NULLGROUP BY TABLE_SCHEMA, TABLE_NAME;

Version Time

5.7.5-m15 0.38 sec

Labs Release 0.08 sec

http://tinyurl.com/y8cnj7o




Case: server crash

• There are some data files• No / outdated backup• FRM files lost• How to use those data files?

The problem


Traditional Data Dictionary• “Move FRM files around”• CREATE TABLE t1 (...)

• ALTER TABLE t1 DISCARD TABLESPACE

• ALTER TABLE t1 IMPORT TABLESPACE ...

• Easy to make mistakes

New Data Dictionary• Self-descriptive tablespaces (SDI)• Dedicated IMPORT statement• Goal: error-proof procedure

Case: server crash


For Plugin Developers• A way to access Data Dictionary• Persistent Storage for plugins– Store/restore custom data

• Plugins can extend INFORMATION_SCHEMA & PERFORMANCE_SCHEMA– Add new tables– Add new columns to existing tables


Data Dictionary

Summary


Data Dictionary : Takeaways• Fundamental component in RDBMS• Critical for performance• Critical for reliability• Critical for scalability


The MySQL TraditionalData Dictionary

• Mix of files and tables• Server DD and InnoDB DD• Inefficient INFORMATION_SCHEMA• Difficult to extend

New Data Dictionary

• Crash-safe InnoDB tables• Single repository• INFORMATION_SCHEMA as VIEWs• Designed to be extendable• Aims for backward compatibility• Huge reengineering

Data Dictionary : Takeaways


Questions?


MySQL Community Reception @ Oracle OpenWorldMingle with the MySQL community and the MySQL team from Oracle for a fun and informative evening!

• Time: September 30 (Tue) @ 7pm• Jillian’s at Metreon 175 Fourth Street, San Francisco, CA

At the corner of Howard and 4th st.; only 2-min walk from Moscone Center


Oracle University MySQL Training ServicesPrepare Your Organization to Enable Reliable and High-Performance Web-Based Database Applications

“Training and team skill have the most significant impact on overall performance of technology and success of technology projects.” - IDC, 2013

Premier Support customers eligible to save 20% on learning credits.

Benefits

Expert-led training to support your MySQL learning needs Flexibility to train in the classroom or online Hands-on experience to gain real world experience Key skills needed for database administrators and developers

•MySQL for BeginnersMySQL for Database AdministratorsMySQL Performance TuningMySQL Cluster – NEW - Register Your Interest!MySQL and PHP - Developing Dynamic Web ApplicationsMySQL for DevelopersMySQL Developer Techniques

MySQL 5.6 Database Administrator MySQL 5.6 Developer

To find out more about available MySQL Training & Certification offerings, go to: education.oracle.com/mysql

Top Courses for Administrators and Developers

Top Certifications

RECENTLY RELEASEDALL NEW! MySQL Cluster Training To Register your interest to influence the

schedule on this newly released course – go to education.oracle.com/mysql and click on the

MySQL Cluster Course

http://education.oracle.com/pls/web_prod-plq-dad/db_pages.getpage?page_id=3

http://education.oracle.com/pls/web_prod-plq-dad/db_pages.getpage?page_id=609&get_params=dc:D62018GC10,p_preview:N&sc=WWOU14047594MPP002C005


Thank You!

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.