MySQL creative programming

Creative programming with MySQL

Giuseppe Maxia, MySQL Community Team Lead

Database Group, Sun Microsystems Inc

[email protected] The Data Charmer

mailto:[email protected]

mailto:[email protected]

2008 CommunityOne Conference | developers.sun.com/events/communityone | 2

Creative programming?



WTF?



WTF?

bright ideas



WTF?

hacking!

bright ideas


About me

22 years in ITconsultantMySQL communityQA developerMySQL Community Team Lead


About me

database hackercreative programmer


Does it look familiar?



SELECT * FROM table_name;



SELECT * FROM table_name;UPDATE table_name SET x=1 WHERE y=10;




DELETE FROM tab_name WHERE z=1;




DELETE FROM tab_name WHERE z=1;make coffee;




DELETE FROM tab_name WHERE z=1;make coffee;play movie;




DELETE FROM tab_name WHERE z=1;make coffee;play movie; ??


Agenda

Database programming blues Stored routinesTriggersViewsEventsFederatedBlackholeCreative loopsMySQL Proxy


Agenda



Agenda



Agenda

Database programming blues Stored routinesTriggersViewsEventsFederatedBlackholeCreative loopsRevisiting TriggersMySQL Proxy


What average programmers dislike

regular expressions



regular expressions

"I (love|hate)\s*([Ff]\w*)\s*REGEXP"



regular expressionsdatabase backend



regular expressionsdatabase backend

SELECT t.id FROM (SELECT id2 FROM other WHERE b = 2) AS tWHERE a > (SELECT MAX(x) FROM z);


Why many programmers hate database handling?

SQL != JavaSQL != PerlSQL != PHPSQL != C++I want to write in my language of choice


database handling

bring the logic at application level


database handling

database

C/C++Java

Perl

PHP

.NET


database handlingtoo many languages

database

C/C++Java

Perl

PHP

.NET

Perl library

Java class

PHP library


database handling

set the logic at server level (stored routines)


database handling

C/C++Java

Perl

PHP

.NET

database

stored routineslibrary


database handling

set the logic at protocol level (proxy)


database handling

C/C++Java

Perl

PHP

.NET

database

Proxylibrary


Agenda



Stored routines - the boring stuff

SQL standardheavy syntaxtricky performanceno debug - limited exception handlingprocedures • can return multiple record sets• can execute dynamic code

functions• must return one value• no dynamic code


Stored routines - the boring stuff

SQL statementsIF THEN ELSE .. END IFWHILE .. DO .. END WHILELOOP .. END LOOPCURSORCONDITIONHANDLER


Stored routines - some hacks

enhance the language• arrays• easy loops• global variables• see MySQL general purpose routine library

create routines on-the-fly• undocumented hack!• see http://datacharmer.org

http://datacharmer.org

http://datacharmer.org


Agenda



Triggers - the boring stuff

BEFORE or AFTER eventBEFORE INSERT/UPDATE/DELETEAFTER INSERT/UPDATE/DELETENo dynamic codeNo operations on the same tableNo multiple triggers per event


Triggers - what for

dedicated loggingdata aggregationtransfer of data to other tables/databasescalculated fields


Triggers - HACKS!

wait.Strong combination with FEDERATED and BLACKHOLE


Agenda



Views

no storagelook like tablescan aggregate datacan insert datacan save timebut can also become horribly inefficient


Views - the cool stuff

can run functionscan create sort of triggers on select


Views with functions

CREATE FUNCTION f(V VARCHAR(20))RETURNS INTBEGIN INSERT INTO log_table VALUES (V, NOW()); RETURN 1;END


Views with functions

CREATE VIEW v1 ASSELECT x FROM table_nameWHERE f('v1') = 1;


Agenda



Events - the not-so-boring facts

Temporal triggersOperating System independentexecutes SQL command or routine• AT timestamp• EVERY X time_interval

can execute dynamic SQL


Events

CREATE EVENT e1ON SCHEDULE AT NOW() + INTERVAL 10 minuteDO CALL clean_employee_recs();


Events

CREATE EVENT e1ON SCHEDULE EVERY 5 minuteDO CALL import_recs_from_file();


Agenda

Database programming blues Stored routinesTriggersEventsFederatedBlackholeCreative loopsRevisiting TriggersMySQL Proxy


Federated storage engine

no storageconnects to a table on a remote serversince version 5.1.22 can connect to a table on the same server

table t1ffederated

table t1MyISAM


Federated storage engine - caveats

no drop-in replacement for regular tablesinefficient aggregateslimited push-down conditions


Federated storage engine limited pushdown conditions

base table

federated table

SELECT ID FROM table_nameWHERE x = 10 SELECT ID

FROM table_name

expected~ 100 records returns

1 million recordsfilter


Federated storage engine workaround pushdown conditions

base table

federated table

SELECT ID FROM table_nameWHERE x = 10 SELECT ID

FROM table_nameWHERE x = 10

expected~ 100 records returns

100 records

add fake indexon column xto Federated table


Agenda



Blackhole

The NON-STORAGE enginelike /dev/null in Unixused for triggers and replication relay

recordrecordrecordrecord binary log

triggers


Agenda



More exciting triggers

Put them together• triggers• Federated engine• Blackhole engine


cascade triggers

BASE TABLEbefore insert

after insertbefore updateafter updatebefore deleteafter delete

t1


cascade triggers

BASE TABLEbefore insert

after insertbefore updateafter updatebefore deleteafter delete

t1

Federated TABLE

before insert

after insertbefore updateafter update

before deleteafter delete

t1_f1


triggers on request

BASE TABLE

Federated TABLE

t1

t1_f1

Federated TABLE

before insert

after update

after delete

before insert

after insert

before update

before deletet1_f2

either

or


blackhole triggersBASE TABLE

blackhole TABLE

before insert

t1_b1

BASE TABLE

t1

t2


remote executionblackhole

tableFederated

TABLE

t1 t1_f1

before insert


remote executionVIEW Federated

TABLE

v1 t1_f1

function


Agenda



loops or why do we have to endure this?

loops with CURSORS• dreadful syntax• INEFFICIENT (temporary table created)

loops alternatives ... coming soon


loops with cursors :(

temporary table

data to process

crc

crc

loop

handler

flagcursor


loops with cursors (I) :(CREATE PROCEDURE p(j INT)

BEGIN DECLARE done BOOLEAN default 'false'; DECLARE crc VARCHAR(42) default ''; DECLARE cid INT; DECLARE x CURSOR FOR SELECT id FROM table_name WHERE b < j; -- to be continued


loops with cursors (II) :(( DECLARE CONTINUE HANDLER

FOR NOT FOUND set done = true; OPEN CURSOR x; search: LOOP FETCH x into cid; IF done THEN LEAVE search; END IF; -- do something with cid SET crc = MD5(CONCAT(crc,cid))END LOOP;SELECT crc;


loops with blackhole :)

blackhole table

data to process crc

crc


loops with blackhole (I) :)

CREATE TABLE t1 ( id VARCHAR(200)) ENGINE = BLACKHOLE;

SET @crc = '';SET @j = 10;


loops with blackhole (II) :)

INSERT INTO t1SELECT @crc := MD5(CONCAT(@crc, id)) FROM table_name WHERE b < @j;

SELECT @crc;

IT'S FASTER!!!


more loops alternatives

use the General Purpose Routine Library• https://sourceforge.net/projects/mysql-sr-lib/

use events! (ON SCHEDULE EVERY 1 SECOND ENDS NOW() + INTERVAL 10 SECOND)

use MySQL Proxy (later)

https://sourceforge.net/projects/mysql-sr-lib/

https://sourceforge.net/projects/mysql-sr-lib/


loops with MySQL stored routines librarycall for_each_table_value_complete(

'db_name', 'table_name', 'id', null, null, '', 'set @crc=MD5(concat(@crc,$I1))', 'set @crc=""', 'select @crc','', 'once');


Agenda



MySQL ProxySo, what about "make coffee" ?





MySQL Proxy


MySQL Proxy principles - hooks



PROXY CORE

connection hook

read query hook

read result hook



PROXY CORE

connection hook

read query hook

read result hook

function

Lua script

function

functionfunctionfunction



PROXY CORE

connection hook

read query hook

read result hook

function

Lua script

function

functionfunctionfunction


MySQL Proxy principles - Lua

??Why not ...{Perl ?

PHP?Javascript?[whatever]?





• SMALL ( < 200 KB)• DESIGNED for

EMBEDDED systems• Widely used (lighttpd)



• SMALL ( < 200 KB)• DESIGNED for

EMBEDDED systems• Widely used (lighttpd)

lighttpd, like MySQL Proxy, was created by Jan Kneschke



Very popular among game writers








injecting queries


injecting queries


Debugging

server

client

proxy

diagnosticstext


Debugging scripts

server

client

proxy

diagnosticstext

proxy

diagnosticstext


Chained proxies - double features

server

proxy

pivot tablesproxy

loopsclient


Testing - customized packages

server

client

proxy

fake packetse.g.

connectors


MySQL Proxy - loops revisited

# get the script from MySQL Forge# http://forge.mysql.com/tools/tool.php?id=96

mysql-proxy \ --proxy-lua-script=loop.lua

http://forge.mysql.com/tools/tool.php?id=96

http://forge.mysql.com/tools/tool.php?id=96


MySQL Proxy - loops revisited

mysql -h 127.0.0.1 -P 4040

FOR VAR 1 3 CREATE TABLE t_$VAR (id int);

# creates t_1, t_2, t_3


MySQL Proxy - loops revisitedmysql -h 127.0.0.1 -P 4040

FOR VAR ( aa, bb cc) CREATE TABLE t_$VAR (id int);

# creates t_aa, t_bb, t_cc


Summary

MySQL is open to development creativityKnow your bricks• stored routines, triggers, views, events• dynamic engines (Federated, Blackhole)• MySQL Proxy

Put them together : be creative


For More Information

MySQL Forge ( http://forge.mysql.com )my blog ( http://datacharmer.blogspot.com )MySQL blogs ( http://planetmysql.org )

http://forge.mysql.com

http://forge.mysql.com

http://datacharmer.blogspot.com

http://datacharmer.blogspot.com

http://planetmysql.org

http://planetmysql.org

80

Q&A

Let's talk!

Technology

MySQL creative programming