Embed Size (px)
Citation preview
MySQL 8 New Features, Updates and Changes
Michael Messina
Senior Managing Consultant, Rolta AdvizeX
Introduction
• Michael Messina
• Senior Managing Consultant Rolta-AdvizeX,
• Working with Oracle Approximately 25 years, MySQL about 12 years.
• Background includes Performance Tuning, High Availability and Disaster Recovery
• MySQL Cloud Service 2018 Implementation Specialist
• Oracle Database OCP
• Oracle RAC Certified Expert
• Oracle Exadata Implementation Specialist
• Oracle ACE
• www.advizex.com
Who’s Using MySQL
3
Cloud
Web & Enterprise
OEM & ISVs
Still a Community Driven Open Source Database
• MySQL 8 Update Contributors based on conversions at Open World
▪ Linked-in
▪ Percona
▪ These are just the main recognizable contributors
NoSQL + SQL = MySQL 8
Agenda
• MySQL History
• Why MySQL
• MySQL 8 So far……
• MySQL Changes
• MySQL Deprecated/Removed
• Data Dictionary
• Atomic Data Definition Statement Support
• Security and User Management
• Persistent Runtime Configuration Changes
• Invisible Indexes
• WITH
• SQL Windowing Functions
Agenda
• TmpTable storage engine
• Optimizer Hints
• JSON (Document Store)
• Performance
• Group Replication
• Other Updates
• New – Works 5.7 and AboveData Masking and De-identification
• Coming Up
• Questions/Discussion
MySQL History
• MySQL was created by a Swedish company, MySQL AB
• First release on 05/23/1995, making a 30 year History for MySQL
• Version 3.20 released January 1997
• Windows Version of MySQL Release January 1998
• Version 3.21 production release in 1998
• Version 4.0 Beta release August 2002
• Version 4.0 Production release March 2003
▪ Unions
• Version 5.0 Beta release March 2005
• Version 5.0 Production release October 2005
▪ cursors, stored procedures, triggers, views, XA transactions
MySQL History
• Sun Acquires MySQL AB in 2008
• Version 5.1 Production Release November 2008
▪ event scheduler, partitioning, row-based replication, server log to tables
• Oracle Acquires Sun Microsystems January 2010
• MySQL Server 5.5 December 2010
▪ Unicode character sets utf16, utf32, and utf8mb4
▪ New Partitioning options
• Version 5.6 February 2013
▪ full-text search for innodb
▪ NoSQL memcache
• Version 5.7 October 2015
▪ Group Replication
• MySQL 8
▪ Development Milestone 2016
▪ Release Candidates 2017
Why MySQL
• Long History of a Solid Scalable Database (Proven over 30 year)
• Great Proven Performance
• Worlds Most Popular Open Source Database
• Open Source and run Community Editions for Free
• Large Database Company Oracle more then 20,000 developers
• Enterprise Options for Support by Oracle at very low cost
• Rich tool set to manage and monitor available
• Enterprise Version has Database Firewall a security option from Oracle that only the Oracle Database can complete with no other open source database has this.
• Large Support Community
Why MySQL
• Run in Cloud / Virtualized / Dedicated Hosts
▪ Cloud Infrastructure as a Service - IaaS
▪ Cloud Database as a Service - DBaaS
• Full and Incremental Database Backup Capability
• Point in Time Recovery
• Highly Scalable
▪ Many scalability options
▪ NDB Cluster
▪ Sharding
• Can Be Made Highly Available
▪ Many Options to Provide Redundancy for availability based on needs
▪ NDB Cluster
▪ Group Replication
▪ Async Replication
MySQL 8 Solid Development Path
• 09/2016 MySQL 8.0.0 Development Milestone
• 04/2017 MySQL 8.0.1 Development Milestone
• 07/2017 MySQL 8.0.2 Development Milestone
• 09/2017 MySQL 8.0.3 Release Candidate 1
• 01/2018 MySQL 8.0.4 Release Candidate
• 04/2018 MySQL 8.0.11 GA
• 10/20/18 Current MySQL 8.0.13
MySQL 8 Upgrade Hints
• Move all user tables to innodb before doing upgrade, not required, but recommended for sure
• errors on older table types have been known to happen due to conflict with default SQL_MODE.
▪ When you have queries on older type tables, you have to do
▪ mysql> set global sql_mode = ‘’;
▪ otherwise you’ll get mysql errors for stuff like (select * from table where id = <id>)
▪ This appears to be a bug, but still RC releases, but thought we would share
MySQL 8 Changes
• tx_isolation and tx_read_only system removed, use transaction_isolationand transaction_read_only
• log_warnings system variable and --log-warnings server parameter removed, use log_error_verbosity system variable instead
▪ log_error_verbosity = (1- Errors, 2- Errors and Warnings, 3-Errors, Warnings and Notes)
• sql_log_bin server parm removed, sql_log_bin has session scope only
▪ applications that rely on accessing @@global.sql_log_bin should be adjusted.
• encryption-related items have been removed
▪ The ENCODE() and DECODE() functions.
▪ The ENCRYPT() function.
▪ The DES_ENCRYPT(), and DES_DECRYPT() functions, the --des-key-file option, the have_crypt system variable, the DES_KEY_FILE option for the FLUSH statement, and the HAVE_CRYPT CMake option.
▪ Use ENCRYPT(), consider using SHA2() instead for one-way hashing or others use AES_ENCRYPT() and AES_DECRYPT() instead.
MySQL Changes
• The client-side --ssl and --ssl-verify-server-cert removed.
▪ Use --ssl-mode=REQUIRED instead of --ssl=1 or --enable-ssl.
▪ Use --ssl-mode=DISABLED instead of --ssl=0, --skip-ssl, or --disable-ssl.
▪ Use --ssl-mode=VERIFY_IDENTITY instead of --ssl-verify-server-cert options. (The server-side --ssl option remains unchanged.)
• mysql_install_db program has been removed, use myqld with the --initialize or --initialize-secure for new database creation.
• No Direct upgrade for non-innodb partitioned tables, change all partitioned tables to innodb prior to upgrade
• InnoDB no longer supports compressed temporary tables.
MySQL 8 Deprecated/Removed
• Deprecated
▪ innodb_undo_tablespaces which controls the number of tablespaces to divide rollback segments into is deprecated
▪ log_syslog for write error log to syslog deprecated
• Removed
▪ sync_frm as .frm files are obsolete
▪ group_replication_allow_local_disjoint_gtids_join
▪ Group Replication will no longer allow any disjointed GTIDs
▪ InnoDB file format configuration parameters were removed:
▪ innodb_file_format
▪ innodb_file_format_check
▪ innodb_file_format_max
▪ innodb_large_prefix
▪ innodb_checksums
▪ show_compatibility_56
MySQL Deprecated/Removed
• The MySQL Query Cache has been Removed in MySQL 8
▪ No more FLUSH QUERY CACHE and RESET QUERY CACHE statements
▪ System variables around query cache removed
▪ query_cache_limit
▪ query_cache_min_res_unit
▪ query_cache_size
▪ query_cache_type
▪ Status Variables removed
▪ qcache_free_blocks
▪ qcache_free_memory
▪ qcache_hits
▪ qcache_inserts
▪ qcache_lowmem_prunes
▪ qcache_not_cached
▪ qcache_queries_in_cache
▪ qcache_total_blocks
Data Dictionary
• MySQL Now has an actual Data Dictionary.
• All privilege tables are now Innodb not MyISAM
• Renamed InnoDB Information Schema Views
Old Name New Name
INNODB_SYS_COLUMNS INNODB_COLUMNS
INNODB_SYS_DATAFILES INNODB_DATAFILES
INNODB_SYS_FIELDS INNODB_FIELDS
INNODB_SYS_FOREIGN INNODB_FOREIGN
INNODB_SYS_FOREIGN_COLS INNODB_FOREIGN_COLS
INNODB_SYS_INDEXES INNODB_INDEXES
INNODB_SYS_TABLES INNODB_TABLES
INNODB_SYS_TABLESPACES INNODB_TABLESPACES
INNODB_SYS_TABLESTATS INNODB_TABLESTATS
INNODB_SYS_VIRTUAL INNODB_VIRTUAL
Data Dictionary
• Uniformly stores database dictionary data (metadata)
• Removes file based metadata storage
• Makes data dictionary transactional, making data dictionary crash safe
▪ Commit, rollback
• Central caching of data dictionary metadata.
▪ Dictionary Object Cache
▪ shared global cache that stores previously accessed data dictionary objects in memory
▪ LRU-based
• Access via the information_schema and “show” statements
• Data dictionary metadata all in innodb tables
• Contained in a single innodb tablespace (mysql.ibd)
Atomic Data Definition Statement Support
• With Data Dictionary we get Atomic DDL
• Combines the data dictionary updates, storage engine operations, and binary log writes associated with a DDL operation into a single, atomic transaction
• Makes so any DDL operation and associated operations are all committed or all rolled back this creates more consistent and stable DDL operations.
• Innodb DDL only
• Partial execution of account management statement will no longer be permitted.
• Uses mysql.innodb_ddl_log table to record the operations for the DDL statement allows statement recovery.
• enable innodb_print_ddl_logs to have MySQL write the DDL logs to stderr, for most systems this will be the error log file for the database.
Resource Management
• Resource Groups
▪ permits assigning threads to particular groups
▪ threads execute according to the resources available to the group
▪ modify these attributes as appropriate for workloads
• Types – Once Group created, Type for Group can not be changed
▪ System
▪ User
• Resources
▪ CPU Affinity – no affinity = All CPUs otherwise the CPU Ids Allowed to be used
▪ OS Thread priority – execution priority range from -20 highest priority to lowest priority of 19
• 2 Groups by default▪ SYS_Default
▪ USR_Default
▪ Priority 0 and No CPU Affinity for default groups
▪ Can not be altered or changed
▪ New User Threads go to USR_Default and SYS Threads to SYS_Default
Resource Groups
Example: RESOURCE_GROUP_NAME: USR_default
RESOURCE_GROUP_TYPE: USER
RESOURCE_GROUP_ENABLED: 1
VCPU_IDS: 0-3
THREAD_PRIORITY: 0
CREATE RESOURCE GROUP 'name'
TYPE=SYSTEM|USER [VCPU=num|start-end[,num|start-end]*]
[THREAD_PRIORITY=num] [ENABLE|DISABLE] ;
ALTER RESOURCE GROUP 'name'
[VCPU=num|start-end[,num|start-end]*]
[THREAD_PRIORITY=num] [ENABLE|DISABLE] [FORCE] ;
DROP RESOURCE GROUP 'name' [FORCE];
SELECT * from INFORMATION_SCHEMA.RESOURCE_GROUPS;
Security and User Management
• Roles
▪ Implements roles to group privileges
▪ Grant roles to accounts to standardize privileges
▪ Control privileges to many uses through the privileges of roles
• Password History
▪ Keeps password history
▪ Allows for password reuse rules and restrictions on pervious passwords
▪ Can apply globally or per account
• caching_sha2_password plugin
▪ Improvement on the sha2_password plugin
▪ Implements sha2 password hashing like the sha2_password plugin
▪ Improves on sha2_password plugin by using caching to address latency issues at connect time
▪ Change mysql_native_password to caching_sha2_password is recommended
Persistent Runtime Configuration Changes
• SET PERSIST allows runtime configuration to be kept and survive a restart. In previous versions changes made using SET GLOBAL would be discarded upon restart.
• Adds setting for the parameter variable to mysqld-auto.cnf
• PERSIST – Set variable for current runtime and through restart
• PERSIST_ONLY – Set variable for persist to next restart does not change current runtime.
• RESET_PERSIST – Reset and remove the persist value setting.
• Be Careful this will take priority over you my.cnf parameters
• Can now change things and take affect at runtime
▪ SET PERSIST innodb_buffer_pool_size=8G ;
▪ SET PERSIST max_connections = 1000 ;
Invisible and Descending Indexes
• MySQL now supports invisible indexes.
▪ not used by the optimizer at all
▪ maintained normally with DML operations.
▪ Indexes visible by default.
▪ Invisible indexes make it possible to test the effect of removing an index on query performance, without making a destructive change that must be undone should the index turn out to be required.
• MySQL now supports descending indexes
▪ DESC in an index definition is no longer ignored but causes storage of key values in descending order.
▪ Prior to this indexes could be scanned in reverse order but it took a performance hit”.
WITH Now Exists in MySQL
• The with clause has been in use for quite some time to offer both simple query builds and better performance for some queries. MySQL now has the capability for using WITH.
WITH tab1 (select * from table1 where columna = ‘My Value’),
tab2 (select * from table2 where mycol = ‘My Value too’)
select * from tab1 a, tab2 b where a.keycol = b.keycol ;
SQL Windowing Functions
• SQL aggregate functions support windowing
▪ COUNT, SUM, AVG, MIN, MAX, BIT_OR, BIT_AND, BIT_XOR,
STDDEV_POP (and its synonyms STD, STDDEV), STDDEV_SAMP,
VAR_POP (VARIANCE synonym for VAR_POP), VAR_SAMP
• Specialized window functions
▪ RANK, DENSE_RANK, PERCENT_RANK, CUME_DIST, NTILE,
ROW_NUMBER, FIRST_VALUE, LAST_VALUE, NTH_VALUE, LEAD, LAG
SQL Windowing Functions
Simple Example using and emp_sales table:
SELECT empid, date, sale, SUM(sale) OVER (PARTITION BY
employee) AS sum FROM emp_sales;
+-------+------------+------+------+
| empid | date | sale | sum |
+-------+------------+------+------+
| 1 | 2018-03-01 | 200 | 900 |
| 1 | 2018-04-01 | 300 | 900 |
| 1 | 2018-05-01 | 400 | 900 |
| 2 | 2018-03-01 | 400 | 1200 |
| 2 | 2018-04-01 | 300 | 1200 |
| 2 | 2018-05-01 | 500 | 1200 |
+-------+------------+------+------+
NOWAIT and SKIP LOCKED
• Default behavior is for a transaction to wait for a lock.
• NOWAIT
▪ Forces transaction to fail if lock can not be obtained immediately.
▪ select * from my_table where my_key BETWEEN 2 AND 3 AND reserved = 'NO’ FOR UPDATE NOWAIT ;
• SKIP LOCKED
▪ Transactions will skip rows where a lock can not be obtained.
▪ Example Using a SELECT… FOR UPDATE
▪ START TRANSACTION;
▪ select * from my_table where my_key BETWEEN 2 AND 3 AND reserved = 'NO’ FOR UPDATE SKIP LOCKED;
▪ Execute the actual update
▪ update my_table set reserved = ‘YES’ where my_key between 2 and 3;
▪ commit ;
TmpTable Storage Engine
• Efficient Storage engine for Internal Temporary Tables.
• Allows larger sort operations that flow over to disk to be faster
• Allows more efficient index builds
• No Blob/Text Column Support
Optimizer Hints
• Introduced the hint syntax /*+ hint */ in 5.7
• Look Familiar to all you Oracle Folks out there?????
• New Hints
▪ INDEX_MERGE and NO_INDEX_MERGE
▪ control index merge
▪ JOIN_FIXED_ORDER, JOIN_ORDER, JOIN_PREFIX, and JOIN_SUFFIX
▪ control table order for joins
▪ SET_VAR
▪ Sets variable to be used at that value for the statement only after statement completes variable reverts to prior value.
Instant Add Column
• Add Column instantly
▪ alter table add column i, ALGORITHM=INSTANT ;
• Alter table with ALGROITHM INSTANT supports the following operations so it is a bit more then just instant add column:
▪ Change index option
▪ Rename table (in ALTER way)
▪ SET/DROP DEFAULT
▪ MODIFY COLUMN
▪ Add/drop virtual columns
▪ Add columns(non-generated) – We call this instant ADD COLUMN
Instant Add Column
• Can not add a column to table with default value with LOCK NONE
▪ ALTER TABLE t1 ALTER COLUMN i SET DEFAULT 11, ALGORITHM=INSTANT, LOCK=NONE;
▪ ERROR HY000: Incorrect usage of ALGORITHM=INSTANT and LOCK=NONE/SHARED/EXCLUSIVE
• You can set default value when LOCK is DEFAULT
▪ ALTER TABLE t1 ALTER COLUMN i SET DEFAULT 13, ALGORITHM=INSTANT, LOCK=DEFAULT;
JSON (Document Store)
• Native JSON Document Storage
▪ JSON data type
▪ Ability to create virtual columns that can be indexed
• Extended Syntax for Ranges in JSON path expressions
• JSON Table Functions
▪ JSON_TABLE() - creates a relational view of JSON data
• JSON Aggregation Functions
▪ JSON_ARRAYAGG() - generate JSON arrays
▪ JSON_OBJECTAGG() - generate JSON objects
JSON (Document Store)
• JSON Merge Functions
▪ JSON_MERGE_PATCH() - Merger multiple JSON documents by replacing any match in first document with second document
▪ JSON_MERGE_PRESERVE() - Merge multiple JSON documents
• JSON Update
▪ JSON_REMOVE() - Removes values from a JSON document
▪ JSON_SET() - can replace existing values and add new values for JSON document
▪ JSON_REPLACE() - Replace existing values in a JSON document
▪ JSON Improved Sorting
▪ 1.2 to 1.8 times performance improvement
MySQL Document Store (High Level Summary)
• Work with SQL Relational Tables and schema less JSON collections
• Eliminates need for a separate NoSQL solution
• Innodb Based
• Highly reliable
• Fully consistent
• Works in Group Replication for Availability and redundancy
• Online backup with the mysql database
• With Enterprise Transparent Data Encryption for higher security
• Works with MySQL Auditing
Performance
• Improved the prefetching by taking the query optimizer’s estimates into account when allocating the buffer
• Makes scans faster (index scan, range scan, table scan)
• For replication Reduce contention between IO and SQL threads
• Performance improvement for large LOBs
▪ Up to 14x in internal tests at Oracle
• Optimizer cost model
▪ Take into account whether data is already in memory or need to be read from disk
• Optimizer Histograms on (detailed stats on columns)
▪ Better Join Order Compare and Index Selection.
• Scalable Redo log with dedicated redo log threads
▪ log_writer writes from log buffer to file
▪ log_flusher executes fsync()
▪ log_notifier notifies user threads finish fsync()
▪ log_checkpointer writes checkpoints
Performance
• Innodb Performance improves around 30% of Mysql 5.7 and 50% over MySQL 5.6, Sysbench OLTP-RW 48 cores trx_commit=1
** Chart from Dimitri KRAVTCHUK, MySQL Performance Architect Oracle
Performance with SSD
• Sysbench OLTP_RO 48 cores 2 SSD Flash Drives
** Chart from Dimitri KRAVTCHUK, MySQL Performance Architect Oracle
Performance with SSD
• OLTP_RW Update-NoKey Test, 48 cores 2 SSD Flash Drive
** Chart from Dimitri KRAVTCHUK, MySQL Performance Architect Oracle
Group Replication
• Better event recording for mutex and condition synchronization objects in GCS are recorded
▪ can been seen through Performance Schema tables.
▪ Give MySQL GCS developers a look for abnormal wait events like bottlenecks or deadlocks. MySQL XCom is currently single threaded and not affected by this work.
• Transaction SAVEPOINT support in Group Replication. Transaction savepoints are now supported in Group Replication
• More date for the replication applier
▪ New fields were added to the existing performance schema tables.
▪ Example: introduced immediate and original commit timestamps are exposed. This feature will greatly help users to profile the applier execution and conduct analysis when they find that for some reason there is replication lag.
▪ Now see at which time the last transaction in the applier was queued in the relay log, when it was processed and when it was applied.
• No More disjointed GTIDs, all must match or database can not join group.
• DDL Recovery in Group Replication
Other Updates
• RESTART command
▪ Command the can restart MySQL without logging into server
• Double write penalty eliminated (8.0.14 and above)
• Binary Logging on by default
• UTF8mb4 default characterset
• Redolog and undolog can not be encrypted with AES 256 with TDE
▪ Innodb_undo_log_encrypt=ON/OFF
▪ Innodb_redo_log_encrypt=ON/OFF
• Audit Logs can be encrypted
• Can now control the time waited for group replication communication before others in the group decide to evict the mysql from the group
▪ 0 to 1 hour, default 5 seconds (8.0.13 and above)
• Automatically reclaim space after large transactions
New – Works 5.7 and AboveData Masking and De-identification
• Mask sensitive data for compliance
▪ GDPR
▪ PCI DSS
▪ HIPAA
▪ HiTECH
▪ SOX
• Great for you non production environments when refreshing data from production.
▪ Allow Production volume data in pre-production test environments
▪ Load Testing
▪ Volume Testing
New – Works 5.7 and AboveData Masking and De-identification
• Masking Functions
▪ Selective - Portion of numbers or strings for like card numbers, SSN, etc
▪ Strict or Relaxed
▪ Random Data Substitution – Replace Real with Random Values
▪ Blurring – Random variance applied to existing values great for randomizing numeric ranges
▪ Dictionary Substitution- Randomly replace values from task specific dictionaries
▪ Blacklisting and Substitution – Replace specific blacklist data only leave all other data
Coming Up
• RAPID
▪ MySQL Analytics
▪ Offered as Part of the MySQL Cloud Service
▪ Oracle Cloud Only
▪ Can Create a slave in the cloud for an on prem environment and use MySQL Analytics with the Slave.
• When member leaves group in replication mysql is restarted automatically to attempt to rejoin the group.
46
Any Further Questions?
