Embed Size (px)
Citation preview
MySQL Plugins —why should you bother
Sergei GolubchikMariaDB
Hi, my name is Sergei Golubchik. I'm one of the core MariaDB developers, work full-time on MariaDB. If you don't know what MariaDB is – it's like MySQL but better, but this talk is not about MariaDB, the talk about MariaDB will be at 2pm, this talk is about Plugins. MariaDB plugins, Percona Server plugins, even MySQL plugins. The talk equally applies to all of them. So, you might have heard these words – “MySQL Plugins” but what are they? Should you care? Or is it something that only hard-core C/C++ programmers (like myself) need to know about?
Plugins are not just some our weird feature. In fact, lots of products use plugins. Plugins are great – users can extend the functionality of the core product without bothering the vendor (and the vendor is big it may take a lot of bothering to have something implemented for you). And there are lots of benefits for the vendor too. No wonder plugins are so popular.
storage engine
information schema
daemon authentication
auditfull-text parser
But we're talking not about any plugins, but only about MySQL plugins (again, when I say MySQL, I mean MySQL, MariaDB, Percona Server). There are many different “kinds” or “types” of them. Most known are storage engines – yes, every storage engine, even MyISAM, is a plugin. But there are also I_S tables, authentication and audit plugins, full-text parsers and daemons. And soon we'll see what they can do.
ClientServer
But first, let me tell you a bit or two about plugins in MySQL. There can be server plugins and client plugins. That is, plugins that are loaded into the server, and extend the server. Or plugins are loaded into the client and extend, respectively, the client.
dynamically loaded
statically linked
Plugins – both client and server – can be dynamically loaded into the server or a client library, or they can be statically linked into it, and thus they'll be always present, becoming part of the binary.
versioning&
metadata
API changes
user mistakes
loading non-plugins
One of the main goal I had when creating Plugin API was the safety and user friendliness. Plugin stores a certain metadata that describe what it is, what it does, and it needs, and so on. Which means, the user does not need to specify any of that – no chance to make a mistake here. Plugin knows what version – or what range of versions – of MySQL it is compatible with. And it won't load into a server of a wrong version. Also, because of that checks it is impossible to load as a plugin something that is not. Which fixes a whole range of security attacks that were possible with user defined functions, plugin predecessors.
There are lots of plugins in MySQL. Even by default, if you didn't install any. They're all listed in the I_S table PLUGINS. And there's a corresponding SHOW command too. SHOW PLUGINS. This is how it looks. As you can see, every plugin is of a specific type – storage engine, authentication, and so on. Some plugins were loaded dynamically from a dll, others have no corresponding dll name – they are part of the server, statically linked. Every plugin has a readable description, and other fields, that didn't fit into the window, are plugin author, plugin version and maturity, plugin license, plugin status – enabled or disabled, and so on. This is how one can see all installed plugins.
INSTALL SONAME "foobar.so"
--plugin-load=foo=foobar.so
INSTALL PLUGIN foo SONAME "foobar.so"
--plugin-load=foobar.so
Now, to install a plugin you can use a special command: install plugin. As I said earlier, plugin knows its type, and all requirements, and so on. You only need to specify what plugin to install, and from what file. Because one .so file may have many plugins in it. Sometimes it's useful to install all plugins from a given .so file. To do it one can use a variant of install plugin without a plugin name – a useful shortcut. To uninstall a plugin there's uninstall plugin command. And uninstall soname too. Alternatively, one can install plugin from a command line of from a my.cnf file using plugin-load option.
--foo-option=value --disable-foo
SET GLOBAL foo_option=value
SHOW STATUS LIKE "foo_%"
Once installed, plugin is pretty much indistinguishable from the core MySQL functionality. It can be configured from the server command line or my.cnf file, it can have server variables and status variables – all that works exactly the same for plugins and the server core. And any plugin – independently from its type – can have that. But, of course, there's also something that makes plugins of different types different.
storage engines
Let's start from storage engine plugins. I'm sure you know what they are. MyISAM is a storage engine plugin, and InnoDB too. Storage engine is responsible for storing data. Basically, anytime when you do INSERT, MySQL asks storage engine to store the data, and when you do SELECT it reads the data from the engine. But MySQL doesn't really care what the engine does with the data, how it is stored, and how it is searched and retrieved. And there are lots and lots of interesting things that the engine can do with the data. That's why there are many storage engines to choose from. First, there are general purpose storage engines. Like MyISAM and Aria, InnoDB and XtraDB, PBXT, TokuDB. They have their strengths and weaknesses, but generally they're designed to be used in a wide range of applications. Then, there are specialized engines.
analytics
storage engines
For example, engines for data analysys. They can store huge amounts of data. Bulk-load it very quickly too. But constantly update it... hmm, not necessarily. Users can run different queries, complex queries, ad-hoc queries. Which means, indexes won't be of much help, because for ad-hoc queries you don't know what indexes you'll need. These engines usually store data column-oriented, compress the data, and kind of automatically index everything that's possible. Examples of analytical engines are InfoBright and InfiniDB.
analytics
high availability
storage engines
Then, there are engines designed to be highly available. Clusters. NDB cluster, for example. But there's also ScaleDB – which is less different from NDB than one might think based on white-papers Those clustering engines have multiple nodes – computers storing the data, with some redundancy. And multiple MySQL servers connecting to those nodes. Any node or MySQL server can fail, but the cluster will still work as a whole. New nodes or new servers can be added or removed any time. Furthermore, a cluster can execute part of the query in parallel, on different nodes, which can provide significant speedups too.
analytics
high availability
data formats
storage engines
There's also a group of engines that reads foreign data formats. They can be used to import legacy data, to migrate from old database products. Or to cooperate with them. Examples would be a CSV engine or an engine that reads Microsoft Access files.
analytics
high availability
data formats
special
storage engines
And also there are engines with very unique features, engines that are just... different. SphinxSE that talks to a full-text search daemon, and understands its own search query language. OQGraph, the engine that stores and searches within trees and more complex graphs. Revision engine that remembers all versions of all rows, and allows to query them. A queue engine (Q4M) – a message queue that pretends to be a table. One can push messages into it, wait for messages, subscribe to messages, and so on. And that's not all, there are many more engines available. But now, let's see what other plugins can do.
information schema
For example, information schema plugins. Every such a plugin represent one table in the information_schema. When it is installed and enabled, a new table appears in the information_schema, and this plugin defines what would be in this table. There are different tasks that information schema plugins can solve.
supporting other plugins
information schema
Most often they are simply used to support other plugins. For example, a storage engine plugin may be accompanied with few information schema plugins, that provide various information about the engine. InnoDB, XtraDB, and PBXT engines do that. And others probably too.
supporting other plugins
OS information
information schema
But information schema plugins can be used independently too. For example, they can show the information about the operating system, the environment that the server is running in. These are plugins that show the information about the resource usage, cpu, io, memory, disks, and so on. And it's not just hypothetical possibility – these plugins already exist. You can download and start using them.
supporting other plugins
OS information
server information
information schema
And, of course, there are information schema plugins that provide access to the information about the server. There's a plugin that shows all savepoints you've set in a transaction. Or a plugin that shows all user variables you've created. Or all table locks. Or all queries in a query cache. This information may be useful to know, but MySQL server won't tell you that. But these plugins will.
authentication
Authentication plugins. Unlike storage engines and information schema plugins, you don't use them directly. Instead, MariaDB server invokes them automatically when you're connect. What plugin will be used for every specific user can be specified with the CREATE USER or GRANT statement, different users may have different authentication plugins. The authentication plugin defines what information you need to provide to prove that it's really you. A password, a hardware token, you grandmother's maiden name – whatever. By default MariaDB uses a built-in authentication plugin that implements conventional MySQL authentication with a password hash and random scramble. But another authentication plugin may do something completely different.
convenience
authentication
For example, you can implement a single sign on within your organization, and with an authentication plugin this single sign on will work for MariaDB too. Thus users won't need to remember another username/password pair, and won't need to login to the database server specifically. As long as they're in the intranet, or on the VPN they will be logged into the database automatically. This both adds convenience and improves security. You can do single sign on with pam, windows, or unix_socket plugins.
convenience
security
authentication
There are a lot of ways to improve login security. For example, you may want to implement one-time passwords, or two-step verification, like your gmail account has, or logging in with hardware tokens. There are plugins for that – for example, skey plugin will do one-time passwords. Pam plugin can do that too, and two-step verification as well (note that here I mean MariaDB pam plugin or Percona pam plugin. Oracle's closed source pam plugin can not do that). I've shown a working example of two-phase verification in our blog. Usbsn plugin (from our book) uses usb serial numbers to login, so that's how one can use hardware tokens.
convenience
security
user management
authentication
Another important use case for authentication plugins is the centralized user management. With a plugin you no longer need to have all your users in the user table of the mysql database. You can store them in the LDAP, in the active directory, NIS, or whatever. Even in a plain-text file. This can significantly reduce the administration overhead.
full-text parser
What do we have now? Full-text parsers. The very first plugin type in MySQL. These plugins sit before the full-text search and work on the text before it gets indexed.
data formats
full-text parser
This way, they can allow full-text search to work with anything, not just plain-text data. With pdfs, open document files, urls, and so on. Our book, for example, features a plugin that works on exif tags (those found in jpeg files of your digital camera). You insert file names – not files themselves – into a table, and then you can search for a specific camera model, or an exif comment. Isn't that cool?
data formats
cjk, expression syntax
full-text parser
But that's not all that parser plugins can do. They can allow searching within, for example, Chinese texts – where people have a habit of not using the space character, so a default parser cannot split the text into words. But there are quite a few parser plugins that can. Download and use them, if you need this kind of functionality. Also a full-text parser plugin can implement its own syntax for Boolean searches – like in our book, where we have a parser that understands operators like AND, OR, and NOT instead of cryptic plus and minus signs.
data formats
cjk, expression syntax
stemming, fuzzy search
full-text parser
And full-text parsers can also apply stemming, soundex, and all kinds of word transformations, to make searches less dependent on the word form and insensitive to typos.
audit
Unlike full-text parsers, audit plugins are relatively new. These plugins are told about what's happening in the server, and they can do with this information what they want – usually they write it to some kind of a log.
security
audit
This can be very useful from security point of view. Take, for example, a recent security problem, where an attacker could login with an invalid password. This is all fixed by now, of course. But if somebody would be trying to exploit this bug – you'd never know about it. MySQL does not log connection attempts. Well, an audit plugin can – and you would immediately spot a large number of failed connection attempts in the audit log.
security
monitoring
audit
Audit plugins can be used for general monitoring too, not necessarily security related. In MariaDB there's a plugin that logs all sql errors to a file. It's not the normal error log – it logs errors like unique constraint violation, syntax error, deadlock, and so on. This way you can see if something's is wrong in an application, even if this application doesn't tell you that.
security
monitoring
statistics
audit
But audit plugins can do more than just logging. There is an audit plugin that calculates some statistics one the statements that were executed. And shows this statistics via an information schema plugin.
daemon
And the last plugin type that I'll talk about is the daemon. Daemon plugins cannot do anything that other plugins can – every other plugin type has something special, daemon plugins don't. So it's useful either for features that don't need the server or for deep server hacking. Here's what they do.
heartbeat
daemon
Hearbeat is a simple plugin that periodically says that the server is still alive. The process may still be present in the process list, but if it hanged it's of no use, right? With a heartbeat plugin you can detect it and restart the server.
heartbeat
HandlerSocket
daemon
HandlerSocket plugin. You all have heard about it right? It is a daemon plugin that adds a new client-server protocol to MariaDB.
heartbeat
HandlerSocket
thread pool
daemon
Thread pool. Not our built-in MariaDB thread pool, but Oracle closed-source thread pool plugin – it is implemented as a daemon plugin too.
only in MariaDB
INSTALL SONAME
SphinxSE, OQGraph, handlersocket
plugin metadata: version and maturity
So, this was an overview of what plugins can do, and what plugins exists. Now these are plugin-related extensions that we have in MariaDB, as compared to vanilla MySQL. We have community plugins – SphinxSE, OQGraph engine, handlersocket plugins. But also federatedx, pbxt, and even more are coming. A useful convenience statement INSTALL SONAME that allows you to install innodb or xtradb engine and all its information schema tables with one statement, not twenty. And in MariaDB you can see how mature every plugin is, and there is a command line switch that can prevent loading, for example, alpha- and beta-quality plugins into a production server.
only in MariaDB
more server services
custom attributes in CREATE TABLE
simplifications of storage engine API
There are also changes without direct user-visible effects, they are more interesting for developers. We do work on making plugin api better, easier and safer to user.
plans and ideas
show plugin soname
pluggable replication
misc. storage engine API features
even more server services
ideas → [email protected]
What we will do next? Oh, we have lots of ideas. Many more than I could fit on the slide. If you do too – feel free to write to [email protected].
