Out-of-the-box MySQL 8.0 for Modern Node.js Applications

Copyright © 2019 Oracle and/or its affiliates.

Out-of-the-box MySQL 8.0 for

Modern Node.js Applications

Principal Software Developer

Oracle, MySQLSeptember 18, 2019

Rui Quelhas

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation.

Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website at http://www.oracle.com/investor. All information in this presentation is current as of September 2019 and Oracle undertakes no duty to update any statement in light of new information or future events.

Safe Harbor


http://www.oracle.com/investor


$ whoami

Rui QuelhasMySQL Middleware and ClientsConnector/Node.js lead developer

[email protected]@ruiquelhas

mailto:[email protected]

https://twitter.com/ruiquelhas

Part 1 Part 2 Part 3 Part 4


Outline

MySQL 8.0 is great The community conundrum

Up and running with Connector/Node.js

Closing thoughts

WARNING

This deck contains code (a lot of it)!


MySQL 8 is great


• MySQL 8.0 brings more than 250 new features across the board

• Fancy things like Window functions and CTEs, proper GIS support are finally available

• The practice of backporting stuff to 5.7 has seen better days

• For a rundown, check https://lefred.be/content/top-10-mysql-8-0-features-for-developers/

• For the complete list, check https://mysqlserverteam.com/the-complete-list-of-new-features-in-mysql-8-0/


Content

https://lefred.be/content/top-10-mysql-8-0-features-for-developers/

https://mysqlserverteam.com/the-complete-list-of-new-features-in-mysql-8-0/

From an application standpoint, there are some interesting additions

Improved security with OpenSSL and new authentication mechanisms

Unlimited languages and emojis with the utf8mb4 charset

The X DevAPI and the MySQL document store


Modern MySQL stack

Client App MySQL

Router

X DevAPI X Plugin

X Protocol

https://dev.mysql.com/doc/refman/8.0/en/document-store.html

Classic Protocol


Modern MySQL stack

Client App

Router

X DevAPI X Plugin

X Protocol



Modern MySQL stack

Client App

X DevAPI



• High-level database API to develop MySQL-based applications

• Gateway for the underlying document store infrastructure

• Unified interface for handling all kinds of data

• Fluent CRUD, NoSQL and raw SQL APIs

• Standard specification for all official client implementations

• Connectors for multiple platforms and languages and the MySQL Shell

• DEV2366 👀

X DevAPI

• Client-server protocol using Google Protocol Buffers

• Easier for ANYONE to extend, improve or build tools and interfaces on top

• Placeholder binding

• Pipelining and feature detection via expectations

• Client-side pluggable authentication mechanisms

• Open source at https://dev.mysql.com/doc/internals/en/x-protocol.html

X Protocol

https://dev.mysql.com/doc/internals/en/x-protocol.html

Enabling the MySQL Document Store

• Unstructured data using a MySQL database and InnoDB

• Abstraction over the JSON data type (document)

• A document maps to a proper data structure in the language of the application

• Schemas, strict data types or SQL become optional

• Logical consistency and ACID compliance are retained

• Traditional tables can still be mixed within the same schema


The community conundrum


• Overall, at Oracle, we are not really that bad (e.g. Fn, GraalVM, etc.)

• In MySQL, we take it even further

• Everything you will hear about in this talk is available on GitHub

• You are free to contribute either via bug reports, pull requests, or just plain feedback

• You can also build stuff on top of MySQL (e.g. ProxySQL, Vitesse, etc.)


Open Source at MySQL

• MySQL is a community-friendly project

• Multiple successful open-source projects build on top of it

• Particularly the case in the Node.js ecosystem

• We don’t compete (community is mostly focused on the classic protocol, we are focused on the X Protocol)

• We plan to support them to the best of our ability with our own limited resources


Supporting the community

• With the current rolling release model, MySQL 8 is an ever-evolving beast

• Breaking changes are expected

• Versioning is still a bit awkward (it’s just not semver), so it’s not easy to handle them

• Corporate policies sometimes prevent earlier outside communication

• Open source maintainers usually have other full-time jobs


Keeping up the pace

Reaching out


Reaching out


Reaching out


Reaching out


Reaching out


Reaching out


Reaching out


Damage control


Damage control


Damage control


• Not really a trademark

• Informal internal initiative

• All products are released under the MySQL server version umbrella

• Compatibility between products in the same version is guaranteed

• Breaking changes and security issues are handled promptly


One Product™

Up and running with Connector/Node.js


• Only Node.js driver for MySQL with out-of-the-box support for MySQL 8.0 series

• Open source, available on GitHub and on npm

• Modern Node.js asynchronous API using Promises

• X DevAPI implementation on top of the X Protocol

• Document store and relational CRUD API as well as plain old SQL

• Secure sessions and multiple authentication mechanisms

• Smart connection lifecycle management


Overview

• Smart enough to react with changes in the server setup

• Graceful degradation in the presence of more limited or older servers

• Autonomous features that are enabled and disabled under the right conditions under the hood

• Enhanced observability and troubleshooting tools

• Placeholder assignment at the protocol-level


Not just a driver

Smart things we do

Autonomous server-side prepared statements

Seamless connection pool management

Automatic connection failover and timeout handling

Server-side instrumentation with connection attributes


await collection.find('name = :n').bind('n', 'foo').fields('name').limit(1)

.execute()

await table.select('name').where('name = :n').bind('n', 'foo').limit(1)

.execute()

await session.sql("select name from ourTable WHERE name = ? LIMIT ?").bind('foo')

.bind(1)

.execute()


Programming style

Asynchronous flow in composable operations

let ctx

ctx = collection.add({ name: 'foo' }).execute()

console.log(ctx instanceof Promise) // true

ctx = table.select().execute()


ctx = session.sql("select 'foo'").execute()



Asynchronous one-off operations

let ctx

ctx = collection.getOne('1')


ctx = collection.removeOne('1')


ctx = collection.dropIndex('name_idx')



var docs = []

await collection.find().execute(doc => docs.push(doc))

// result sets from tables can contain column metadata

var rows = [], columns = []

await table.select()

.execute(r => rows.push(r), c => { columns = columns.concat(c) })


Handling result sets (the old way)

With a single result set, things are still not that weird

var rows = [], columns = []

await session.sql("select 'foo' as c")

.execute(r => rows.push(r), c => { columns = columns.concat(c) })

console.log(rows) // [['foo']]

console.log(columns[0].getColumnLabel()) // ['c']


What happens when there are multiple result sets?

DELIMITER //

CREATE PROCEDURE proc()

BEGIN

SELECT '1' AS s1c1, 'foo' AS s1c2;

SELECT '2' as s2c1, 'bar' AS s2c2;

END//


Painful and error prone since it involves shared state

var rows = [], metadata = [], i = 0

var result = await session.sql('CALL proc').execute(row => {

rows[i - 1] = rows[i - 1] || []

rows[i - 1].push(row)

}, columns => {

metadata[i] = metadata[i] || []

metadata[i] = metadata[i].concat(columns)

i += 1

})


“Keep in mind that .execute() doesn’t return a promise, but

rather receives a callback function to do your data processing of

each individual row. This could be a bit annoying for you.


https://gabi.dev/2018/07/18/a-small-dive-into-the-mysql-8-0-x-devapi/

Gabriela Ferrara

https://gabi.dev/2018/07/18/a-small-dive-into-the-mysql-8-0-x-devapi/

• Weird mix of Node.js asynchronous execution patterns

• Breaks the consistency promised by the X DevAPI

• Totally different from the JavaScript API of the MySQL Shell

• Does not annoy only one person

• We acknowledge this https://insidemysql.com/working-with-result-sets-in-connector-node-js/

• But there’s good news!


Rethinking the interface

https://insidemysql.com/working-with-result-sets-in-connector-node-js/

var result = await collection.find().execute()

var docs = result.fetchAll()

result = await table.select().execute()

var row = result.fetchOne()

// extracting column metadata

var columns = result.getColumns()

columns[0].getColumnLabel()


Coming soon (in 8.0.18)

Handling multiple result sets is not a big deal anymore

var cursor = await session.sql('CALL proc').execute()

console.log(cursor.fetchAll()) // [['1', 'foo']]

console.log(cursor.getColumns()[0].getColumnLabel()) // 's1c1'

cursor.nextResult()

console.log(cursor.fetchAll()) // [['2', 'bar']]

console.log(cursor.getColumns()[1].getColumnLabel()) // 's2c2'


# install from npm (no semver)

$ npm install @mysql/xdevapi --save --save-exact

// import the module

const mysqlx = require('@mysql/xdevapi')

// create an X DevAPI session over a server connection

const session = await mysqlx.getSession('mysqlx://root@localhost:33060')


Starting from a blank slate

Create a new schema with a collection and add some documents

const schema = await session.createSchema('oco19')

const collection = await schema.createCollection('ourCollection')

const result = await collection.add({ name: 'foo' }).add({ name: 'bar' })

.execute()

console.log(result.getAffectedItems()) // 2

const ids = result.getGeneratedIds()

console.log(ids) // ['00005d79ffcb0000000000000001', '00005d79ffcb0000000000000002']


Retrieve data from documents in the collection

// get all documents that match a certain filtering criteria

const docs = [];

await collection.find('name = :value').bind('value', 'foo')

.execute(doc => docs.push(doc))

console.log(docs) // [{ _id: '00005d79ffcb0000000000000001', name: 'foo' }]

// fetch a single document with a given id

const doc = await collection.getOne('00005d79ffcb0000000000000002')

console.log(doc) // { _id: '00005d79ffcb0000000000000002', name: 'bar' }


Update properties of existing documents

await collection.modify('_id = :id').bind('id', ids[0])

.set('name', 'baz')

.execute()

await collection.modify('name = :n').bind('n', 'bar').patch({ name: 'qux', age: 42 })

.execute()

await collection.replaceOne(ids[0], { name: 'foo' })

await collection.addOrReplaceOne(ids[1], { name: 'bar' })


Remove documents from a collection

await collection.remove('name = :n1 or name = :n2')

.bind('n1', 'foo')

.bind('n2', 'bar')

.execute()

await collection.removeOne('00005d79ffcb0000000000000003')

// something like this would empty the collection

await collection.remove('true').execute()


Cleaning up the mess

// you can remove just the collection

await schema.dropCollection('ourCollection')

// or you can remove the entire database

await session.dropSchema('oco19')

// then you can exit gracefully

await session.close()


Closing thoughts


• Now is the time to start using or switch to MySQL 8.0

• The pace of change is ever-increasing

• It’s not easy to keep up with the latest and greatest

• Open source, community, trustworthiness and feedback loops are important pieces of the puzzle

• Connector/Node.js provides a robust and modern MySQL experience and ticks all these boxes

• Give it a try and leave us your feedback


Takeaways

• https://www.npmjs.com/package/@mysql/xdevapi

• https://github.com/mysql/mysql-connector-nodejs

• https://dev.mysql.com/doc/dev/connector-nodejs/8.0/

• https://dev.mysql.com/doc/x-devapi-userguide/en/

• https://dev.mysql.com/doc/refman/8.0/en/document-store.html

• https://forums.mysql.com/list.php?44

• https://insidemysql.com/category/mysql-development/connectors/


Resources

https://www.npmjs.com/package/@mysql/xdevapi

https://github.com/mysql/mysql-connector-nodejs

https://dev.mysql.com/doc/dev/connector-nodejs/8.0/

https://dev.mysql.com/doc/x-devapi-userguide/en/


https://forums.mysql.com/list.php?44

https://insidemysql.com/category/mysql-development/connectors/

• MySQL 8.0: The New Replication Features [DEV3013]

• Building Carefree Data-Driven Applications with the MySQL X DevAPI [DEV2366] 👀

• MySQL InnoDB Cluster: Management and Troubleshooting with MySQL Shell [DEV2066]

• What’s New in MySQL 8.0 Security? [DEV2034]


MOAR MySQL

Thank You


The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation.

Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website at http://www.oracle.com/investor. All information in this presentation is current as of September 2019and Oracle undertakes no duty to update any statement in light of new information or future events.

Safe Harbor


http://www.oracle.com/investor

Documents

Out-of-the-box MySQL 8.0 for Modern Node.js Applications