Embed Size (px)
Citation preview
Preparse Query Rewrite PluginsNew SQL syntax for fun & performance
Sveta SmirnovaPrincipal Support Engineer
January, 30, 2016
Table of Contents
•Introducing new SQL syntax
•Working with results
•Variables
•Summary
2 www.percona.com
Introducing newSQL syntax
3 www.percona.com
MySQL often receives blames• From mascots and from humans
• It cannot make a toast• It does not support some syntax
4 www.percona.com
MySQL often receives blames
• From mascots and from humans
• It cannot make a toast
• It does not support some syntax
4 www.percona.com
MySQL often receives blames• From mascots and from humans• It cannot make a toast• It does not support some syntax
4 www.percona.com
Or does it?
• FILTER clause in MySQL on my homemachine
5 www.percona.com
Or does it?
• FILTER clause in MySQL on my homemachine
5 www.percona.com
Or does it?
• FILTER clause in MySQL on my homemachine
• But not in the user manual
5 www.percona.com
How is it done?
• With 181 lines of code
• Including comments!
• And new Query Rewrite Plugin interface
6 www.percona.com
A little bit of history
• First introduced in version 5.7.5
• Was available at MySQL Labs
• Two types of plugins
• Pre-parse
• Post-parse
7 www.percona.com
Today
• Part of Audit plugin interface• Step in at
• MYSQL AUDIT GENERAL ALL• MYSQL AUDIT CONNECTION ALL• MYSQL AUDIT PARSE ALL
• MYSQL AUDIT PARSE PREPARSE
• MYSQL AUDIT PARSE POSTPARSE
• MYSQL AUDIT AUTHORIZATION ALL• ...
8 www.percona.com
Plugin skeleton
#include <mysql/plugin.h>
#include <mysql/plugin_audit.h> - Audit plugin declaration
...
static MYSQL_PLUGIN plugin_info_ptr; - Pointer to the plugin
...
static int filter_plugin_init(MYSQL_PLUGIN plugin_ref); - Plugin initialization
...
static int filter(MYSQL_THD thd, mysql_event_class_t event_class,
const void *event); - Entry point for MYSQL_AUDIT_PARSE_PREPARSE
...
static st_mysql_audit filter_plugin_descriptor;
...
mysql_declare_plugin(filter_plugin);
9 www.percona.com
Plugin descriptor
static st_mysql_audit filter_plugin_descriptor= {
MYSQL_AUDIT_INTERFACE_VERSION, /* interface version */
NULL,
filter, /* implements FILTER */
// You can also use MYSQL_AUDIT_PARSE_ALL
{ 0, 0, (unsigned long) MYSQL_AUDIT_PARSE_PREPARSE,}
};
10 www.percona.com
Plugin declaration
mysql_declare_plugin(filter_plugin)
{
MYSQL_AUDIT_PLUGIN,
&filter_plugin_descriptor,
"filter_plugin",
"Sveta Smirnova",
"FILTER SQL:2003 support for MySQL",
PLUGIN_LICENSE_GPL,
filter_plugin_init,
NULL, /* filter_plugin_deinit - TODO */
0x0001, /* version 0.0.1 */
NULL, /* status variables */
NULL, /* system variables */
NULL, /* config options */
0, /* flags */
}
mysql_declare_plugin_end;
11 www.percona.com
Memory management for plugins#include <my_thread.h> // my_thread_handle needed by mysql_memory.h
#include <mysql/psi/mysql_memory.h>
...
static PSI_memory_key key_memory_filter;
static PSI_memory_info all_rewrite_memory[]=
{
{ &key_memory_filter, "filter", 0 }
};
static int filter_plugin_init(MYSQL_PLUGIN plugin_ref)
{
plugin_info_ptr= plugin_ref;
const char* category= "sql";
int count;
count= array_elements(all_rewrite_memory);
mysql_memory_register(category, all_rewrite_memory, count);
return 0; /* success */
}
12 www.percona.com
SQL:2003
<filter clause> ::=
FILTER <left paren> WHERE <search condition> <right paren>
(10.9 <aggregate function>, 5WD-02-Foundation-2003-09.pdf, p.505)
Only for aggregate functions:
<computational operation> ::=
AVG | MAX | MIN | SUM | EVERY | ANY
| SOME | COUNT | STDDEV_POP | STDDEV_SAMP
| VAR_SAMP | VAR_POP | COLLECT | FUSION | INTERSECTION
<set quantifier> ::=
DISTINCT
| ALL
MySQL only supports
COUNT | AVG | SUM | MAX | MIN
| STDDEV_POP | STDDEV_SAMP
| VAR_SAMP | VAR_POP
13 www.percona.com
Implementing FILTER clause
• FILTER is practically
CASE WHEN foo THEN bar ELSE NULL
• So we only need to catch
FUNCTION(var) FILTER(WHERE foo)
• And replace it with CASE
14 www.percona.com
Catching up the querystatic int filter(MYSQL_THD thd, // MySQL Thread object
mysql_event_class_t event_class, // Class of the event
const void *event // Event itself
)
{
const struct mysql_event_parse *event_parse=
static_cast<const struct mysql_event_parse *>(event);
if (event_parse->event_subclass != MYSQL_AUDIT_PARSE_PREPARSE)
return 0;
string subject= event_parse->query.str; // Original query
string rewritten_query;
//requires std::regex and GCC 4.9+
regex filter_clause_star("(COUNT)\((\s*\*\s*)\)\s+"
+ "FILTER\s*\(\s*WHERE\s+([^\)]+)\s*\)",
ECMAScript | icase);
rewritten_query= regex_replace(subject, filter_clause_star,
"$1(CASE WHEN $3 THEN 1 ELSE NULL END)");
...
15 www.percona.com
Rewritten query
void _rewrite_query(const void *event,
const struct mysql_event_parse *event_parse,
char const* new_query
)
{
char *rewritten_query= static_cast<char *>(my_malloc(
key_memory_filter, strlen(new_query) + 1, MYF(0)));
strncpy(rewritten_query, new_query, strlen(new_query));
rewritten_query[strlen(new_query)]= ’\0’;
event_parse->rewritten_query->str= rewritten_query; // Rewritten query
event_parse->rewritten_query->length=strlen(new_query);
// You must set this flag to inform MySQL Server what query was rewritten
*((int *)event_parse->flags)|=
(int)MYSQL_AUDIT_PARSE_REWRITE_PLUGIN_QUERY_REWRITTEN;
}
16 www.percona.com
Working withresults
17 www.percona.com
Can we do better?
• Playing with syntax is fun
• But can we introduce something moreMySQL-ish?
18 www.percona.com
Custom hint plugin
• MySQL 5.7 has Optimizer HintsSELECT /*+ NO_RANGE_OPTIMIZATION(t3 PRIMARY, f2_idx) */ f1
FROM t3 WHERE f1 > 30 AND f1 < 33;
SELECT /*+ BKA(t1) NO_BKA(t2) */ * FROM t1 INNER JOIN t2 WHERE ...;
SELECT /*+ NO_ICP(t1, t2) */ * FROM t1 INNER JOIN t2 WHERE ...;
SELECT /*+ SEMIJOIN(FIRSTMATCH, LOOSESCAN) */ * FROM t1 ...;
EXPLAIN SELECT /*+ NO_ICP(t1) */ * FROM t1 WHERE ...;
• But sometimes thread-specific buffersaffect query execution
• Workaround requires processing result setof each of these statements
• This is why I extended optimizer hintsyntax
19 www.percona.com
Custom hint plugin
• MySQL 5.7 has Optimizer Hints• But sometimes thread-specific buffers
affect query execution• Common workaround exists:
SET tmp_table_size=1073741824;
SELECT * FROM t1 INNER JOIN t2 WHERE ...;
SET tmp_table_size=DEFAULT;
• Workaround requires processing result setof each of these statements
• This is why I extended optimizer hintsyntax
19 www.percona.com
Custom hint plugin
• MySQL 5.7 has Optimizer Hints• But sometimes thread-specific buffers
affect query execution• Workaround requires processing result set
of each of these statements• This is why I extended optimizer hint
syntaxSELECT /*+ join_buffer_size=16384 */ f1 FROM t3 WHERE f1 > 30 AND f1 < 33;
SELECT /*+ tmp_table_size=1073741824 BKA(t1) NO_BKA(t2) */ *
FROM t1 INNER JOIN t2 WHERE ...;
19 www.percona.com
New features
• For Custom Hints we need to:
• Store previous values of threadvariables we are going to modify
• Modify variables
• Revert them back before sending result
20 www.percona.com
Store previous values
// map to store modified variables
static map <my_thread_id, map<supported_hints_t, ulonglong> > modified_variables;
...
/* The job */
static int custom_hint(MYSQL_THD thd, mysql_event_class_t event_class,
const void *event)
{
...
// If we have a match store create map of thread variables
std::map<supported_hints_t, ulonglong> current;
...
After processing variables store them in modified_variables map
modified_variables[thd->thread_id()]= current;
...
21 www.percona.com
Modify variables
• Since we have access to MYSQL THDthis is easy:
switch(get_hint_switch(ssm[1]))
{
case JOIN_BUFFER_SIZE:
current[JOIN_BUFFER_SIZE]= thd->variables.join_buff_size;
thd->variables.join_buff_size= stoull(ssm[2]);
break;
case TMP_TABLE_SIZE:
current[TMP_TABLE_SIZE]= thd->variables.tmp_table_size;
thd->variables.tmp_table_size= stoull(ssm[2]);
break;
...
22 www.percona.com
Revert variables back
• First we need to tell plugin descriptor whatwe needMYSQL AUDIT GENERAL RESULTstatic st_mysql_audit custom_hint_plugin_descriptor= {
MYSQL_AUDIT_INTERFACE_VERSION, /* interface version */
NULL,
custom_hint, /* implements custom hints */
{ (unsigned long) MYSQL_AUDIT_GENERAL_RESULT, 0,
(unsigned long) MYSQL_AUDIT_PARSE_PREPARSE,
}
};
• Then revert variables before sending result• And, finally, erase stored values for current
thread:
23 www.percona.com
Revert variables back• First we need to tell plugin descriptor what
we needMYSQL AUDIT GENERAL RESULT
• Then revert variables before sending resultif (event_general->event_subclass == MYSQL_AUDIT_GENERAL_RESULT)
{
map<my_thread_id, map<supported_hints_t, ulonglong> >::iterator
current= modified_variables.find(thd->thread_id());
if (current != modified_variables.end())
{
for (map<supported_hints_t, ulonglong>::iterator it=
current->second.begin(); it!= current->second.end(); ++it)
{
switch(it->first)
{
case JOIN_BUFFER_SIZE:
thd->variables.join_buff_size= it->second;
break;
• And, finally, erase stored values for currentthread:
23 www.percona.com
Revert variables back
• First we need to tell plugin descriptor whatwe needMYSQL AUDIT GENERAL RESULT
• Then revert variables before sending result
• And, finally, erase stored values for currentthread:
modified_variables.erase(current);
23 www.percona.com
Before Custom Hint Pluginmysql> flush status;
Query OK, 0 rows affected (0.00 sec)
mysql> select count(*), sum(c) from
-> (select s, count(s) c from joinit where i < 1000000 group by s) t;
+----------+--------+
| count(*) | sum(c) |
+----------+--------+
| 737882 | 737882 |
+----------+--------+
1 row in set (24.70 sec)
mysql> show status like ’Created_tmp_disk_tables’;
+-------------------------+-------+
| Variable_name | Value |
+-------------------------+-------+
| Created_tmp_disk_tables | 2 | -- 2 temporary tables on disk
+-------------------------+-------+
1 row in set (0.00 sec)
24 www.percona.com
Custom Hint Plugin at workmysql> flush status;
Query OK, 0 rows affected (0.00 sec)
mysql> select /*+ tmp_table_size=134217728 max_heap_table_size=134217728 */
-> count(*), sum(c) from
-> (select s, count(s) c from joinit where i < 1000000 group by s) t;
+----------+--------+
| count(*) | sum(c) |
+----------+--------+
| 737882 | 737882 |
+----------+--------+
1 row in set, 2 warnings (6.21 sec) -- 4 times speed gain!
mysql> show status like ’Created_tmp_disk_tables’;
+-------------------------+-------+
| Variable_name | Value |
+-------------------------+-------+
| Created_tmp_disk_tables | 0 | -- No disk-based temporary table!
+-------------------------+-------+
1 row in set (0.00 sec)
25 www.percona.com
Variables
26 www.percona.com
BACKUP DATABASE plugin
• Very simple syntax
• mysql> BACKUP SERVER;
+---------------------------------+
| Backup finished with status OK! |
+---------------------------------+
| Backup finished with status OK! |
+---------------------------------+
1 row in set, 1 warning (42.92 sec)
• Supports many tools
• Needs to pass options
27 www.percona.com
BACKUP DATABASE plugin
• Very simple syntax• Supports many tools
• mysqldump• mysqlpump• mysqlbackup• XtraBackup - Planned!
• Needs to pass options
27 www.percona.com
BACKUP DATABASE plugin
• Very simple syntax
• Supports many tools
• Needs to pass options
• Credentials
• Backup directory
• Custom
27 www.percona.com
Customization: credentials
• We have access to• MYSQL_THR->security_context
• thd->security_context()->user().str
• Password still has to be in the configuration file, under
[client]
or
[toolname]
section
• System variables• Since we are interested in backing up local server we will use
mysqld_unix_port
28 www.percona.com
Customization: variables
• Global variables - Example only!
static MYSQL_SYSVAR_STR(backup_dir, backup_dir_value, PLUGIN_VAR_MEMALLOC,
"Default directory...", NULL, NULL, NULL);
static MYSQL_SYSVAR_ENUM(backup_tool, backup_tool_name,
PLUGIN_VAR_RQCMDARG, "Backup tool. Possible values:
mysqldump|mysqlbackup", NULL, NULL,
MYSQLDUMP, &supported_tools_typelib);
• Thread variables
• Add to plugin declaration
29 www.percona.com
Customization: variables
• Global variables - Example only!
• Thread variables
static MYSQL_THDVAR_STR(backup_dir, PLUGIN_VAR_MEMALLOC,
"Default directory...", NULL, NULL, NULL);
static MYSQL_THDVAR_ENUM(backup_tool, PLUGIN_VAR_RQCMDARG,
"Backup tool. Possible values:
mysqldump|mysqlbackup|mysqlpump", NULL, NULL,
MYSQLDUMP, &supported_tools_typelib);
...
• Add to plugin declaration
29 www.percona.com
Customization: variables• Global variables - Example only!• Thread variables• Add to plugin declaration
static struct st_mysql_sys_var *mysqlbackup_plugin_sys_vars[] = {
MYSQL_SYSVAR(backup_dir),
MYSQL_SYSVAR(backup_tool),
MYSQL_SYSVAR(backup_tool_basedir),
MYSQL_SYSVAR(backup_tool_options),
NULL
};
mysql_declare_plugin(mysqlbackup_plugin)
{
MYSQL_AUDIT_PLUGIN,
&mysqlbackup_plugin_descriptor,
"mysqlbackup_plugin",
...
NULL, /* status variables */
mysqlbackup_plugin_sys_vars, /* system variables */
...
29 www.percona.com
Summary
30 www.percona.com
More possibilities
• Custom locks
• Access to thread- and server-specificvariables
• Fine control at multiple steps of queryexecution
• More
31 www.percona.com
Code
• https://github.com/svetasmirnova/
• filter plugin
• custom hint plugin
• mysqlbackup plugin
32 www.percona.com
More information
• MySQL source dir/plugin• rewriter• rewrite example
• Writing Audit Plugins manual• MySQL Services for Plugins manual
33 www.percona.com
Place for your questions
???
34 www.percona.com
Thank you!
http://www.slideshare.net/SvetaSmirnova
https://twitter.com/svetsmirnova
35 www.percona.com
