Pragmatic Software Development in q/kdb+

Ajay Rathore

My preferred tools for kdb+/q development

● Sublime Text3● Maven● Git● Jenkins● Docker

Sublime Text3

● Light on the CPU, faster startup● Symbol indexing across projects, which allows you to

use Goto Definition and Goto Symbol to jump to related code in other files. Pretty huge

● Lots of open source plugins to enhance and customize● Kdb+ syntax highlighting sublime-q

Screenshot of my IDE below

MavenMaven’s primary goal is to allow a developer to comprehend the complete state of a development effort in the shortest period of time. In order to attain this goal there are several areas of concern that Maven attempts to deal with:

● Making the build process easy● Providing a uniform build system● Providing quality project information● Providing guidelines for best practices development● Allowing transparent migration to new features

Apart from the above benefits, one of the biggest advantage of using maven is that it allows certain level of modularity in big kdb+ projects. Using maven we can break kdb+ libraries and services into smaller, manageable artifacts and using nexus repositories we can publish and share those artifacts across different teams.

Let me give a simple example.

Suppose we want to create a logging library which can be shared across different kdb services without duplicating the code.

Maven project layout will look something like below

Assembly.xml (This will create a zip archive maintaining the dir structure relative to src/main/kdb)

pom.xml

When we run “mvn install” on above project, maven will create a zip artifact “logging-library-1.0-SNAPSHOT-dist.zip” and push that into nexus repository.In order for other projects to use this logging library, they will need to declare a maven dependency in their project pom.

<dependency> <groupId>com.kdb.ajay</groupId> <artifactId>logging-library</artifactId> <version>1.0-SNAPSHOT</version> <classifier>dist</classifier> <type>zip</type></dependency>

Provided those other projects are also assembled as zip distribution, maven dependency plugin will pull any dependencies

and package everything together in a single artifact while maintaining the individual project’s directory structure. Now for using the functions in the logging library, other q files will need to simply load the library using

\l code/logging.q

For deployment purpose, we can use maven-antrun-plugin to scp the final zip archive to any target server and unzip it and initialize the service.

GitGit is probably the most popular distributed version control system out there, it can be used as a server out of the box. Dedicated Git server software helps, amongst other features, to add access control, display the contents of a Git repository via the web, and help managing multiple repositories.For hosting the code, there are couple of options for free web-based Git repository hosting service like github and bitbucket. I prefer using bitbucket which allows private repositories.Git can be integrated with maven using <scm> tag, this allows maven to automatically push any tagged artifacts when user runs mvn release:prepare and release:perform.

JenkinsJenkins is an open source continuous integration server. Basically Continuous Integration is the practice of running your tests on a non-developer machine automatically every time someone pushes new code into the git repository. It gets tightly integrated with maven and git. There are multiple ways of setting up jenkins git repository monitoring, whenever new code is pushed to git either master or feature branch, jenkins can download any changes and build the branch and can deploy the artifact to the dev or uat server. All of this can be automated and speeds up develop-test-release cycles.

DockerDocker is an open source technology and provides a convenient way to ship any infrastructure code and supporting libraries and tools along with the application code. You can look into references section for more resources on docker.

We can make use of couple of maven plugins for creating docker images of our kdb services and push those into docker registries via jenkins job.

Project structure will look something like

For building docker images and integrating into maven lifecycle we can create a maven docker profile

To invoke the above docker profile we can simply callmvn clean install -P docker

This will result in following sequence of events, first of all maven will build project zip artifact and put it inside project build directory which by default is project-root/target/After that maven-resource plugin will run in the prepare-package phase and will copy all files inside src/main/docker dir to target/docker and project build artifact from target to target/docker dir, it will also replace any property place holders via resource filtering. Now everything we need to build docker image is inside target/docker dirNext we invoke maven-antrun-plugin in package phase. It will perform two tasks, first it sets up appropriate file permission on target/docker/build.sh and then run that script to produce docker image.

build.sh

The above script simply invokes docker build command inside target/docker/ dir which has dockerfile with all the instructions for building the kdb service image.

Dockerfile

Once the docker image is produced, we can simply launch the container using container port mapping and docker volumes for loading data from SAN or NAS.

docker run -v /Users/ajay/deploy/kdb-tick/data/marketdata:/home/ajay/services/kdb-tick/data/marketdata --name marketdata -td -p 5000:5000 -p 5001:5001 -p 5002:5002 ajayrathore/kdb-tick-marketdata

The above container will launch a standard kdb tick triplet with tickerplant on port 5000, realtime on port 5001 and hdb on port 5002. Historic data is mounted from localhost using docker -v flag.

References

Maven

Git

Jenkins

Docker