Continuous integration is not a solved problem

https://cyclid.io @cyclidci github.com/Cyclid/

Continuous Integration is not a solved problem

Config Management Camp, Gent 2017


Kristian van der Vliet

● DevOpsing hard for the previous 7+ yearsPuppet 0.24Chef 0.9

● Gainfully employed as an SRE for a CDN company

● You may remember me from such projects as Syllable

● Don’t be fooled by the name, mijn Nederlands is niet goed!


Why isn’t CI a solved problem?

● I’ve seen many CI pipelines for things like Chef; I’ve never met one I liked● Continuous Integration tools make assumptions

○ You’re “building” software○ It will result in an “artifact”○ Your software is one of a known set of languages○ The build process is well defined

● None of these are true for “non-traditional” software○ Configuration Management/Infrastructure as Code○ Network configuration○ Database schemas



“How hard can it be?”


My ideal Continuous Integration

● Unopinionated Not tied to any workflow or toolset.

● Agnostic Tools change constantly.

● Flexible ...so do functional requirements.

● Operable I’m an Operations person!

● Open Source I’m also an Open Source person.



Unopinionated? Agnostic? Flexible? Operable? Open Source?

Jenkins Y Y Y N Y

Travis N N N N Note 1

CircleCI N N N - N

Codeship Y N N - N

GoCD Y N Y Note 2 Y

Semaphore Y N N - N

Concourse Y N Note 3 Note 4 Y

Drone Note 5 N N Y Y

1. Mostly Open Source but key components are not.2. YUM/APT packages, but UI or giant XML configuration file of Doom configuration.3. Concourse supports plugins but the plugin interface is basic.4. Basic setup is simple (Go binary) but it gets complex, fast E.g. BOSH clusters. Lots of components.5. Drone jobs are single scripts, so yes in the sense that you can anything you can do in a script.



● Written in Ruby, REST API○ Boring technologies, because boring means well understood○ I wanted plugins!

● “Everything is a plugin” philosophy○ Even where something isn’t a plugin, it’s highly decoupled

● Keep the component & dependency footprint as small as possible○ Cyclid, Nginx or Apache, Sidekiq, Redis, “a SQL database”


Everything is a plugin

● Builder Create build instances to run jobs● Transport Communicate with build instances● Provisioner Configures build instances● Action Do things when running a job● Source Get the source(s) to build● API Extend the core REST API● Dispatcher Run a job somewhere


Builders● Google● Digitalocean● LXD

Provisioners● Ubuntu● Debian

Transports● SSH● LXD

Actions● Command● Script● Log● Slack● Email

Sources● Git

API● Github

Dispatchers● “Local” (Sidekiq)


{ "name" : "minimal-example", "environment": {

}, "stages" : [ { "name" : "hello-world", "steps" : [ { "action" : "log", "message" : "hello from %{name}" } ] } ], "sequence" : [ { "stage" : "hello-world" } ]}

● Jobs can be JSON or YAML

● Single file, with the source

● Defines the Environment,

Stages that make up the

Sequence and the Sequence

itself (pipeline)


---name: minimal-exampleenvironment: {}stages:- name: hello-world steps: - action: log message: Hello from %{name}sequence:- stage: hello-world


2017-01-07 12:49:58 +0000 : Obtaining build host...2017-01-07 12:51:38 +0000 : Preparing build host...==============================================================================================================================================================2017-01-07 12:51:38 +0000 : Job started. Context: {"job_id"=>88, "job_name"=>"minimal-example", "job_version"=>"1.0.0", "organization"=>"test", "name"=>"buildhost-a2713b62c93f7fd828c35261b1535614", "host"=>"104.197.96.230", "username"=>"build", "workspace"=>"/home/build", "password"=>nil, "key"=>"/etc/mist/id_rsa_build", "server"=>"prod-euw-build01", "distro"=>"ubuntu", "release"=>"trusty"}-------------------------------------------------------------------------------2017-01-07 12:51:39 +0000 : Running stage hello-world v1.0.0Hello from buildhost-a2713b62c93f7fd828c35261b1535614


- name: bundle-install steps: - action: command cmd: sudo gem install bundler --no-ri --no-doc - action: command cmd: bundle install --path vendor/bundle path: '%{workspace}/Cyclid'

sequence:- stage: bundle-install on_success: lint on_failure: failure- stage: lint on_success: rspec on_failure: failure...


Even more jobs

● Environments○ Operating System/Distro & version○ Additional package repositories○ Additional packages

● Sources● Secrets

○ Tokens, passwords, keys etc.

● Stages can also be defined on the server


$ cyclid org showName: testOwner Email: [email protected] Key: -----BEGIN PUBLIC KEY-----...-----END PUBLIC KEY-----Members:

vanders$ cyclid job submit job.yamlJob: 89$ cyclid job status 89Status: Waiting$ cyclid job log 892017-01-07 13:38:04 +0000 : Obtaining build host......



The future!


What we’ve learned from Config Mgmt

1. Flexibility is good2. It’s fine to have more than one tool

Competition is good3. Different tools and workflows suit different users4. Composability is good5. Data driven is good

DRY


Composable CI

● Every CI job is a snowflake● Cyclid is built with Composability in mind

○ Every Job & Stage is versioned○ Stages can be defined on the server and used across different jobs

● Next: shareable Stages○ We’ve already done it with Config Management (Forge, Supermarket, Modules

etc.)○ Just another server with an API○ ...but will require dependency management


Data driven

● Data Driven is a natural requirement of Composability● What would it even look for for Continuous Integration?

○ What data sources would be useful?○ How do we organize that data?○ How do we expose that data to both jobs & users?

● Jenkins is the closest with Parameterized Builds○ Limited in scope

● I genuinely don’t know what this looks like○ Let’s start the conversation!


Logic

● Let’s stop pretending CI jobs don’t require logic○ Stop externalizing the logic into shell scripts

● only_if/not_if○ Same as conditionals on resources in Configuration Management tools

sequence:- stage: release only_if: %{branch} eq ‘master’


Event Framework

● Cyclid jobs are run by a simple state machine● Originally I tried to make “Notifiers” Action plugins

○ Whoops!

● Events will be emitted during the job run, and consumed by plugins etc.○ Fan-out & Fan-in (Pipelines!)○ Proper Notifications, and Notification chains○ Deep job introspection & profiling○ Monitoring & Statistics


Lots more

● More Plugins○ AWS Builder○ Redhat-ish Provisioner(s)○ Not-Linux○ Deployment

● Docker?● Vagrant?● Qemu?


Questions?


Thank you!

Website cyclid.io

Twitter @cyclidci

Github github.com/cyclid

Email [email protected]@cyclid.io

https://github.com/cyclid

mailto:[email protected]

mailto:[email protected]

Software

Continuous integration is not a solved problem