Upload
itamar-haber
View
89
Download
2
Embed Size (px)
Citation preview
Developing a Redis ModuleItamar Haber
2
Who We Are
The open source home and commercial provider of Redis
Open source. The leading in-memory database platform, supporting any high performance OLTP or OLAP use case.
Chief Developer Advocate at Redis [email protected]@itamarhaber
3
Redis Modules Hackathon
http://bit.ly/redishack• Opens today, ends on November 12th• Global & local events (SF and TLV)• Submit your Redis module to participate• Win $10K in cash prizes! Gain world fame!
~10 Things About Redis
5
1. Redis: REmote DIctionary Server2. / rɛdɪs/: “red-iss”3. OSS: http://github.com/antirez/redis4. 3-clause BSD-license: http://redis.io5. In-memory: all in RAM (Flash option)6. A database for: ~9 data structures7. And: 4 (+1) more specialized ones
6
8. Developed & maintained: (mostly) Salvatore Sanfilippo (a.k.a. @antirez) and his OSS team at @RedisLabs
9. History:v1.0 August 9th, 2009… v3.2.4 September 26th, 2016
10.“The Leatherman™ of Databases”: mostly used as a DB, cache & broker
7
11.A couple or so of extra features:(a) atomicity; (b) blocking wait;(c) configurable persistence;(d) data expiration and (e) eviction; as well as transactions, PubSub, Lua scripts, high availability & clustering
12.Next version (v4.0): MODULES!
8
Redis 1011. Redis is “NoSQL”0. No (explicit) schema, access by key1. Key -> structure -> data
9
Redis data stratav1.0 Strings
Lists
Setsv1.2 Sorted Setsv2.0 Hashes
v2.2 Bit arraysv2.8.9HyperLogLogv3.2 Geo Sets
Bit fieldsv4 Neural Network
MODULES!
10
How to Redis in 3 steps:1. 150 OSS clients in 50 languages, e.g:
Java, Node.js, .NET, Python, Ruby…2. You make a request, i.e.:
PING3. The server replies, i.e.g:
PONG
11
~$ redis-cli127.0.0.1:6379> SET counter 1OK127.0.0.1:6379> GET counter"1"127.0.0.1:6379> INCRBY counter 1(integer) 2127.0.0.1:6379> APPEND counter b||!2b(integer) 7127.0.0.1:6379> GETSET counter "\x00Hello\xffWorld""2b||!2b"127.0.0.1:6379>
DEMOTIME!
The Evolution of Versatility
13
Flexibility: model (almost) anything with basic “building blocks” and simple rules (v0.0.1)Composability: transactions (v1.2) and server-embedded Lua scripts (v2.6)Extensibility: modules (v4) for adding custom data structures and commands
MODULES! (a.k.a plugins)
15
First mentioned in release v1.0
https://groups.google.com/forum/#!msg/redis-db/Z0aiVSRAnRU/XezAFFtgyPUJ
“Another interesting idea is to add support for plugins implementing specific commands and associated data types, and the embedding of a scripting language.”
16
Redis before modules:1. Redis is ubiquitous for fast data, fits
lots of cases (Swiss™ Army knife)2. Some use cases need special care3. Open source has its own agenda
So what can you do? FR, PR or fork
17
Redis with modules:1. Core still fits lots of cases2. Module extensions for special cases3. A new community-driven ecosystem4. “Give power to users to go faster”
What to expect? Nothing’s impossible!
18
Redis modules are:1. Dynamically (server-)loaded libraries2. Future-compatible3. (will be mostly) written in C4. (nearly) as fast as the core5. Planned for public release Q3 2016
19
Modules let you:1. Process: where the data is at2. Compose: call core & other modules3. Extend: new structures, commands
20
redis> ECHO "Alpha""Alpha"redis> MODULE LOAD example.soOKredis> EXAMPLE.ECHO "Bravo""Bravo"redis> ^C
~$ wc example.c 13 46 520 example.c~$ gcc -fPIC -std=gnu99 -c -o example.o example.c~$ ld -o example.so example.o -shared -Bsymbolic -lc
core command
module library“new”
command
21
Some exemplary modules:• Neural Redis – a neural network• RediSearch – full text search• countminsketch – streaming counter• redablooms – Bloom filters• redis-tdigest – rank estimator• More at the hub: redismodules.com
Redis Modules API
23
The API1. Where most of the effort was made2. Abstracts & isolates Redis’ internals3. The server’s (C-) binding contract4. Will not be broken once released5. Exposes three conceptual layers
24
Modules API layers1. Operational: admin, memory, disk,
replication, arguments, replies…2. High-level: client-like access to core
and modules’ commands3. Low-level: (almost) native access to
core data structures memory
~$ cat example.c: operational-API-only example
26
#include "redismodule.h"int Echo(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { if (argc != 2) return RedisModule_WrongArity(ctx); return RedisModule_ReplyWithString(ctx,argv[1]); }
int RedisModule_OnLoad(RedisModuleCtx *ctx) { if (RedisModule_Init(ctx, "example", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; if (RedisModule_CreateCommand(ctx, "example.echo", Echo, "readonly", 1, 1, 1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; }
27
#include "redismodule.h"int Echo(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { if (argc != 2) return RedisModule_WrongArity(ctx); return RedisModule_ReplyWithString(ctx,argv[1]); }
int RedisModule_OnLoad(RedisModuleCtx *ctx) { if (RedisModule_Init(ctx, "example", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; if (RedisModule_CreateCommand(ctx, "example.echo", Echo, "readonly", 1, 1, 1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; }
#include "redismodule.h"
int RedisModule_OnLoad(RedisModuleCtx *ctx) {
MUST:API definitions
MUST:is called when module is loaded
pointer to context
28
RedisModuleCtx *ctx1. The module’s call execution context2. Used by most calls to the API, just
pass it along3. Exposes the client id if needed
29
#include "redismodule.h"int Echo(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { if (argc != 2) return RedisModule_WrongArity(ctx); return RedisModule_ReplyWithString(ctx,argv[1]); }
int RedisModule_OnLoad(RedisModuleCtx *ctx) { if (RedisModule_Init(ctx, "example", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; if (RedisModule_CreateCommand(ctx, "example.echo", Echo, "readonly", 1, 1, 1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; }
if (RedisModule_Init(ctx, "example", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; if (RedisModule_CreateCommand(ctx, "example.echo", Echo, "readonly", 1, 1, 1) == REDISMODULE_ERR) return REDISMODULE_ERR;
register thecommand
register the moduleor die trying
30
#include "redismodule.h"int Echo(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { if (argc != 2) return RedisModule_WrongArity(ctx); return RedisModule_ReplyWithString(ctx,argv[1]); }
int RedisModule_OnLoad(RedisModuleCtx *ctx) { if (RedisModule_Init(ctx, "example", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; if (RedisModule_CreateCommand(ctx, "example.echo", Echo, "readonly", 1, 1, 1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; }
int Echo(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) if (argc != 2) return RedisModule_WrongArity(ctx); return RedisModule_ReplyWithString(ctx,argv[1]); validate
number of arguments
&err if needed
arguments
&countsend back
the argument
31
RedisModule_ReplyWith• Error – duh• Null – no words• LongLong – integer• String – also Simple and Buffer• Array – Redis array (can be nested)• CallReply – High-Level API reply
High-Level API
33
RedisModule_Call(…)• Does: runs a command • Expects: context, command name,
printf-like format and arguments• Returns: RedisModuleCallReply *
• Not unlike: Redis’ Lua redis.call
35
int Educational_HighLevelAPI_Echo(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { if (argc != 2) return RedisModule_WrongArity(ctx); RedisModule_AutoMemory(ctx); RedisModuleCallReply *rep = RedisModule_Call(ctx, "ECHO", "s", argv[1]); return RedisModule_ReplyWithCallReply(ctx, rep);}
Using the High-Level API to call the Redis core ‘ECHO’ command...
...is impractical but educational :)
36
RedisModule_AutoMemory(…)Automagically manages memory• RedisModuleCallReply *• RedisModuleString *• RedisModuleKey *• RedisModule_Alloc() and family
High-Level Visualization Of The Low-Level API
38
user
app
Redis client
Redis cor
e
data
GET foo
"bar"101010010101101010
39
user
101010010101101010
High
level
API
app
module
40
user
101010010101101010
app
Low
level
API
41
With the low-level API you can:• Manage keys: open, close, type,
length, get/set TTL, delete…• Manipulate core data structures:
e.g. RedisModule_StringSet(…), RedisModule_ListPop(…) and RedisModule_Zset*Range(…)
42
• Directly access String memory: RedisModule_StringDMA(…)
• Register custom data types: RedisModule_CreateDataType(…)
• Asynchronous (blocking) commands: RedisModule_BlockClient(…)
• And much more…
Redis Modules Hackathon
44
Redis Modules Hackathon
http://bit.ly/redishack• Learn new stuff, help the community• Have an amazing idea? Start coding!• Open to individuals and teams globally• Win $10K in cash prizes! Gain world fame!
45
Developing your module• Tools: a text editor and a compiler• Get Redis unstable branch• Docs and code are at
src/modules• Quick jumpstart & SDK: https://
github.com/RedisLabs/RedisModulesSDK
46
Guidelines for a module’s API• Keep it simple• Make it generic (and composable)• Implicitly manage data structures• Commands have predetermined
space/time cost• Inspiration: Redis’ core API
47
A few more pointers• Top performance is key• RAM consumption is also important• Code isn’t everything: docs, tests,
benchmarks and examples of use• Make it useful and awesome!
Thank you