Jsonnet, Terraform & Packer

Dave CunninghamSoftware Engineer, Google Cloud

Devops Days Denver2015-04-23

Overview

1. Intro to Terraform

2. Intro to Packer

3. Intro to Jsonnet

4. Cloud App Management Methodology

(Fractal Demo)}

myservice.tf

apply

Resources

Forwarding Rule

ForwardingRule

Address

Disk

Route Firewall

Health Check

TargetPool

Network

Address

Instance

TargetPool

Address

Instance

Instance

InstanceInstanceInstance

Address

The first time:1. Builds plan

○ Ordered by Dependency

○ Parallelized2. Executes plan3. Writes local state file

Subsequent changes:1. Examine & refresh state2. Diff, build plan

○ Ordered by Dependency

○ Parallelized○ Minimally disruptive

3. Executes plan4. Updates local state file

Terraform By Hashicorphttp://www.terraform.io/

Provider & credentialsProvider & credentials

Build images, content defined by a JSON configuration file:

{ "builders": [{ "type": "googlecompute", "source_image": "debian-7-wheezy-v20140718", .. credentials .. }], "provisioners": [ { "type": "shell", "inline": [ "sudo apt-get update", "sudo apt-get install -y redis-server" ] }, ... ]}

Packer By Mitchell Hashimoto (Hashicorp founder)http://www.packer.io

Jsonnet https://google.github.io/jsonnet/doc/

Addresses the config language problem:

Write application

Simple config file

+ Comments + vars

+ String arith

+ conditionals+ repetition

+ int arith

Turing completeness!!1

+ templates

+ closures

+ user def. functions

Typical config language

Hazards of ad-hoc language design:

Jsonnet https://google.github.io/jsonnet/doc/

Complex / surprising behaviorNo specification:

difficult to develop tools

Feature creep(overlapping

features)

Ugly implementation

Hard to improve / replace

implementation with same semantics

Hard to port implementation (e.g.

to Javascript)

Use existing general purpose scripting language?

Write application

Simple config file

Python / Go / Ruby / Lua / etc.

Jsonnet https://google.github.io/jsonnet/doc/

Not hermetic: Can yield different config in

different environment

Designed for specifying

behavior, not data

Code cannot be substituted with data

(side effects)

Heavyweight implementations

Jsonnet https://google.github.io/jsonnet/doc/

// Trivial Example{ person1: { name: "Alice", welcome: "Hello " + self.name + "!", }, person2: self.person1 { name: "Bob" },}

{ "person1": { "name": "Alice", "welcome": "Hello Alice!" }, "person2": { "name": "Bob", "welcome": "Hello Bob!" }}

➡

A configuration language designed like a programming language

● Simple: Just 9 features, (3 are from JSON)

○ Literals, arrays, objects, variables, conditionals, arithmetic, closures, mixins, errors

● Powerful: Multi-paradigm (supports OO and functional)

● Hermetic: Repeatable evaluation, code/data interchangeable

● Familiar: All syntax and semantics compatible with JSON / Python

● Concise: Code / data interleaving, prototype inheritance

● Formal: Complete operational semantics

Application ServerApplication ServerApplication Server

Application ServerApplication ServerTile Generation Service

CassandraCassandra

Cassandra

HTTP

HTTP

Cassandra protocol

Live version

Fractal Application Architecture

Jsonnet configurationJsonnet

configurationJsonnet configuration

One logical configuration, perhaps broken into several files by import constructs.

appserv.packer.json

cassandra.packer.json

tilegen.packer.json

{ "appserv.packer.json": ..., "cassandra.packer.json": ..., "tilegen.packer.json": ..., "terraform.tf": ...}

Multiple output mode:Configuration defines several files

terraform.tf

Jsonnet evaluation

Centralize ALL configuration

Use a makefile: Invoke Jsonnet, run Packer and Terraform on JSON output:

● Build multiple images in parallel, only if config changed

● Just run make -j to build images and deploy / update cloud application

infrastructure, system, daemons, databases, builds, ...

● Abstraction - say less

○ Build template libraries, factor out repetitive code

■ Both tilegen and appserv use Nginx + uWSGI + Flask

○ Override bits of default Nginx / uWSGI / Cassandra configs as needed

○ Higher level templates allow listing of apt packages, repos, keys, etc

● Synchronize details

○ Backend endpoints / credentials feature in

■ frontend / backend application config (packer configs)

■ infrastructure (metadata, firewalls, health checks, load balancer...)

Advantages of this methodology

Packer + Terraform + Jsonnet methodology:

● Simple, powerful, centralized, unopinionated

● Straightforward combination of tools

● Arbitrarily complex use cases

Conclusion

Questions?