KDB+/R Integration

KDB+ AND R INTEGRATIONRORY WINSTON

KDB+ AND R INTEGRATION

AGENDA

▸ Why integrate R/kdb+

▸ Basic R integration via CSV import/export

▸ R integration via qserver

▸ Integrating R math functionality into kdb+

▸ Generating kdb+ data with R


WHAT IS R?

▸ DSL For Statistical Techniques and Data Analysis

▸ Integrated programming and EDA (Exploratory Data Analysis) environment

▸ Open-Source

▸ Thousands of contributed packages


WHY USE R?

▸ Comprehensive Environment

▸ Tooling

▸ Command-Line

▸ IDE

▸ Web

▸ Thousands of Packages

▸ Rich Reporting/Graphing/Analysis Capabilities


PART ONE: CSV IMPORT/EXPORT

▸ CSV import/export

▸ In kdb+:

▸ save`:table.csv

▸ In R:

▸ read.csv(‘table.csv’,header=TRUE)

EXAMPLE #1CSV IMPORT/EXPORT


CSV IMPORT / EXPORT

▸ Simplest approach

▸ Interchange with any other system

▸ No type information

▸ Manual overhead

▸ Can be a slow/laborious process


PART TWO: THE QSERVER LIBRARY

▸ Available from

▸ http://code.kx.com/wsvn/code/cookbook_code/r/

▸ Consists of:

▸ A native library containing kdb+ connectivity

▸ An R wrapper

▸ Binaries for Windows/Linux/OSX

http://code.kx.com/wsvn/code/cookbook_code/r/


THE QSERVER LIBRARY

qserver

kdb+

laye

r

R la

yer


BASIC QSERVER OPERATIONS

▸ Connect to a kdb+ instance:

▸ open_connection(host,port,credentials)

▸ Returns a handle

▸ Execute a query:

▸ execute(handle,query)

▸ Close connection:

▸ close_connection(handle)

EXAMPLE #2QSERVER


QSERVER INTEGRATION

▸ Most Efficient

▸ Can Manage Multiple Connections

▸ Can Be Wrapped Into An R Library

▸ Easy to Extend With Custom R Code


PART THREE: LEVERAGING THE R MATH LIBRARY

▸ R Comes With A Standalone Math Library

▸ Contains A Large Range Of Base Statistical/Math Routines

▸ E.g. Base Probability Functions/RNG

▸ Can Be Packaged Into a Library And Reused in kdb+

▸ I have packed a library and a set of routines on:

▸ https://github.com/rwinston/kdb-rmathlib

https://github.com/rwinston/kdb-rmathlib


PART THREE: LEVERAGING THE R MATH LIBRARY

▸ Example In R:

▸ rnorm(1000,5,2)

▸ Generates 1000 random variables with a mean of 5 and standard deviation of 2

▸ In q, using the rmathlib integration:

▸ norm[1000;5;2]

▸ R’s math library contains hundreds of useful base functions

EXAMPLE #3RMATHLIB


USING R TO GENERATE Q CODE

▸ Generate q code from R

▸ Similar in principle to CSV export

▸ See

▸ http://www.theresearchkitchen.com/archives/776

http://www.theresearchkitchen.com/archives/776

EXAMPLE #4Q CODE GENERATION

THANKS

Software

KDB+/R Integration